# Senior Software Engineer, AI Evals at Sentry.co

- Company: Sentry.co
- What the company does: Offline credential manager. Backed by Antler.
- Company website: https://sentry.co/
- Type: Startups (AI role)
- Level: Senior
- Location: San Francisco, California
- Work setup: Remote
- Pay: $155K to $400K base salary per year (USD)
- Posted: 2026-01-28
- Apply by: 2026-10-08
- Apply: https://jobs.ashbyhq.com/sentry/95d2eeab-291d-40ad-97a2-86b104f3c7ad
- Page: https://www.1752.vc/careers/jobs/sentry-co-senior-software-engineer-ai-evals/

## About the role

As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence.

## What they're looking for

- Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
- Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
- Comfort writing production-quality code (we use Python and TypeScript)
- Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
- Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
- Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools

Tags: Engineering