Backed by a16z.
About the role
We're looking for an engineer to own how we know whether our robots are actually getting better. As we move from research to production, we need evaluation that's rigorous enough to trust and fast enough to keep up. You'll build the harnesses, task suites, and regression infrastructure that turn "the demo looked good" into a number we can act on. Own the evaluation strategy for our policies in simulation and on real hardware. Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- Built evaluation, validation, or benchmarking systems that a team made real decisions from
- Fluency with model training and evaluation workflows: comfortable reading a training curve, reasoning about generalization, and telling a real improvement from an overfit one
- Strong software engineering and programming skills
- Practical statistics for experiment design: sample sizing, significance, and knowing when a result is noise
- Demonstrated skepticism: a track record of finding the flaw in a result that everyone else was ready to accept
More about this role
We're looking for an engineer to own how we know whether our robots are actually getting better. As we move from research to production, we need evaluation that's rigorous enough to trust and fast enough to keep up. You'll build the harnesses, task suites, and regression infrastructure that turn "the demo looked good" into a number we can act on.
Own the evaluation strategy for our policies in simulation and on real hardware.
Make evaluation results trustworthy and reproducible; the same evaluation, run by different people on different days, gives the same answer.
Ensure a checkpoint that improves one behavior can't silently break another, and that releases are gated on it.
Establish sim-to-real correlation, so we know which simulation results we can trust.
Own the internal benchmark our models are measured against, and keep results comparable as it evolves.
Know where we stand against the state of the art, benchmarked against external baselines and published results.
Make evaluation cheap enough to run continuously, without burning robot hours on every checkpoint.
Give the company clear answers to "is this checkpoint better than the last one" and "what is it failing at."
Built...
Browse similar: AI jobs · AI startup jobs · Startup jobs · San Francisco Bay Area