# AI Engineer, Evaluation at Distyl

- Company: Distyl
- What the company does: Architecting the AI-Native Enterprise. Distyl partners with the most ambitious enterprises to design and operationalize their AI transformations. Backed by Khosla, Lightspeed and Peak XV.
- Company website: https://distyl.ai/
- Type: Startups (AI role)
- Level: Mid level
- Location: San Francisco
- Work setup: Remote
- Pay: $150K to $250K base salary per year (USD)
- Posted: 2026-06-18
- Apply by: 2026-10-08
- Apply: https://jobs.ashbyhq.com/distyl/13aa2aa1-e5e4-4bab-b2ea-7d8ff5658076
- Page: https://www.1752.vc/careers/jobs/distyl-ai-engineer-evaluation/

## About the role

Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments Define how system quality is measured in each domain, ensuring that evaluation signals reflect real user needs, domain constraints, and business objectives

## What they're looking for

- At Distyl, we build AI systems using Evaluation-Driven Development —an approach where evaluation is not an afterthought, but the primary mechanism for iterating, improving, and trusting AI behavior in production
- This role is for engineers who believe that AI systems only improve when measurement is tightly coupled to development—and who want to apply that philosophy directly to systems that matter
- 5+ years of software engineering experience
- Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments. You treat evaluation code with the same rigor as application code
- Experience with Evaluation-Driven or Experiment-Driven Development: Experience using structured evaluation or experimentation frameworks to drive system iteration, and understand the pitfalls of overfitting to metrics that don’t reflect real outcomes
- Ability to Translate Human Judgment into Code: Work with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders that scale

Tags: Engineering