Build custom software for your business without hiring a developer. Create internal tools, customer apps, and websites by chatting with AI. Backed by Andreessen Horowitz.
About the role
Your #1 and only goal is to make Hercules' AI coding agent cheaper, faster, and more accurate (our pareto frontier) You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to guide your eval creation Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- Hercules is looking for a senior or staff level product manager to own evals for our AI coding agent
- Evals. You have explicitly created and/or managed technical evals before. You should be able to work across a variety of different eval methodologies and deeply understand optimal reward modeling
- Product sense : strong intuition for the goals customers are trying to achieve and novel solutions to help them achieve them
- Design sense: Track record of building simple, delightful UX/UI that makes it easy for customers to achieve their goals
- Yes, we know this industry is evolving rapidly and the "best practices" change every week, but yes, we are still actively looking for someone who has wrestled with these challenges at scale and has strong opinions about what works and what doesn't
More about this role
Hercules is looking for a senior or staff level product manager to own evals for our AI coding agent
Your #1 and only goal is to make Hercules' AI coding agent cheaper, faster, and more accurate (our pareto frontier)
You will own Hercules Bench, our offline eval set
You will own Hercules online evals
You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to guide your eval creation
You will anonymize data to protect our customer's privacy
We expect candidates to be incredibly strong across all of the following areas:
Software fundamentals : You are building an AI coding agent. We expect you to be technical and able to code. We expect you to have a decent understanding of system design, architecture, data modeling, and API design. In saying this, you will work with an engineering counterpart to work on complex implementation
Evals . You have explicitly created and/or managed technical evals before. You should be able to work across a variety of different eval methodologies and deeply understand optimal reward modeling,
Product sense : strong intuition for the goals customers are trying to achieve and novel solutions to help them achieve them
Design sense:...
Browse similar: Startup jobs · San Francisco Bay Area