About Jarmin
AI Employees for human-level roles. Backed by Y Combinator.
About the role
As a Member of Technical Staff on Evals, you'll own how we measure AI Employee performance on the job.
What they're looking for
- Evaluation systems: datasets, offline replay, scorers, and regression alerts
- Feedback loops for models and agents
- Autonomous failure analysis
- Defining what good, degraded, and failed runs look like, and alerting on them
- Have built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality
- Can turn ambiguous quality questions into concrete metrics and decisions
More about this role
As a Member of Technical Staff on Evals, you'll own how we measure AI Employee performance on the job.
- Evaluation systems: datasets, offline replay, scorers, and regression alerts
- Feedback loops for models and agents
- Autonomous failure analysis
- Defining what good, degraded, and failed runs look like, and alerting on them
- Have built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality
- Can turn ambiguous quality questions into concrete metrics and decisions
- Have strong software engineering fundamentals and ship production systems
- Are comfortable where there are few established patterns
- Want to work in person in San Francisco
- You've shipped LLM systems to production and learned where they break
- You have strong data instincts and work well with researchers
- You contribute to open source projects
Browse similar: AI jobs · AI startup jobs · Startup jobs · San Francisco Bay Area