Private, domain-specific benchmarks in legal, tax, and finance. Backed by a16z and Pear VC.
About the role
Measuring intelligence is hard, and humans haven't been particularly good at it. The proxies we've used — IQ, standardized tests, credentials — have shaped how we develop intelligence and how we value it, often in ways we later regret. AI gives us a chance to do better. The field is young enough that the methodologies for measuring what these systems can actually do are still being written, and the answers we settle on will shape what gets built, what gets deployed, and which workflows get automated next.
What they're looking for
- A PhD in ML/NLP (in progress or completed), or equivalent industry research track record
- Deep familiarity with the LLM evaluation landscape: existing benchmarks, their failure modes, judge-model approaches, human-in-the-loop methodologies
- A bias toward research that affects what people actually deploy, rather than benchmarks that are easy to game
- Strong written and verbal communication. You'll publish, present, and talk to customers and labs
- Ability to work in-person, in San Francisco
- Learning velocity: The role encompasses a wide variety of tasks. Rather than expecting you to be an expert on Day 1, we are looking for someone who can learn new skills and technologies quickly
More about this role
Measuring intelligence is hard, and humans haven't been particularly good at it. The proxies we've used — IQ, standardized tests, credentials — have shaped how we develop intelligence and how we value it, often in ways we later regret. AI gives us a chance to do better. The field is young enough that the methodologies for measuring what these systems can actually do are still being written, and the answers we settle on will shape what gets built, what gets deployed, and which workflows get automated next.
Vals is building the measurement layer for the AI economy: the benchmarks, methodologies, and standards that determine which models ship and where they get trusted. We're hiring a Head of Research to lead it.
The hard research questions don't have textbook answers yet. How do you measure whether an LLM can actually do a real lawyer's contract review, a real underwriter's risk assessment, a real radiologist's read? How do you build evaluations that hold up as models get better at gaming them? You'll be the person setting the direction on how Vals — and by extension, much of the field — answers them.
Advance the science of evaluation. The methodologies the field uses today — judge...
Browse similar: AI jobs · AI startup jobs · Startup jobs · San Francisco Bay Area