Backed by IVP.
About the role
As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivery of useful and credible evaluations of agents offered by software vendors. You’ll enhance the system that performs validation of agent performance, and lead the core team behind it. You will leverage “agent-first” ways of working, applying agentic engineering techniques to deliver your work to production. Build and enhance agent evaluations that run against live agents [80%]:
What they're looking for
- We realize applying for jobs can feel daunting at times. Even if you don’t check all the boxes in the job description, we encourage you to apply anyway
- 10+ years of professional programming experience in backend or full-stack environments
- 2+ years of experience directly managing engineers
- Expert-level proficiency in backend development using languages like Python, Java/Kotlin, Typescript/Javascript or Go, strong proficiency in associated backend frameworks like FastAPI or Node.js
- Direct experience creating evaluations (evals) against a customer-facing agent, leveraging agent trajectory trace data and rubrics to measure an agent’s task completion rate, accuracy, correctness, or policy adherence
- Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications
More about this role
About G2 - The Company
G2 is the world's largest and most trusted software marketplace. When you join G2, you’re joining the industry’s leading team that helps businesses reach their peak potential by powering decisions and strategies with trusted insights from real software users.
Now, we have joined forces with Capterra , SoftwareAdvice , and GetApp to create the largest source of online data and software insights to fuel intelligent buying in the age of AI. With 200M+ combined annual visitors and 6M verified reviews, we are now the centralized place to enable software buyers to make better and faster decisions with confidence.
And we are just getting started! We are setting out to transform the global B2B software industry and become the most trusted data foundation for buyers and sellers of software for the age of AI.
Does that sound exciting to you? Come join us as we try to reach our next PEAK!
At G2, everything we are and what we do is grounded in our PEAK values— ( P erformance + E ntrepreneurship + A uthenticity + K indness. Working at G2 means you are part of a value-driven, growing global community that climbs PEAKs together. We cheer for each other’s successes, learn...
Browse similar: Startup jobs · Remote jobs