Private, domain-specific benchmarks in legal, tax, and finance. Backed by a16z and Pear VC.
About the role
We are looking for strong engineers to join our team and own the leaderboards that appear on Vals AI. You'll be responsible for testing new models against our benchmarks as they're released; covering tasks in law, tax, coding, finance, social mobility, and more. You will analyze error modes of models, evaluate their strengths and weaknesses, and work with our communications team to release results.
What they're looking for
- Familiarity with the LLMs: You should already be familiar with the space - the current leading models, relative performance across them, how to use large language models in practice
- Strong engineering fundamentals : You can build and ship quickly with high quality. You should have a track record of building things of significant scope (at jobs, side projects, open source, etc.)
- Python expertise : Significant experience in Python, especially in a professional setting
- Team collaboration : Experience working in development sprints, Git workflows, and pull request reviews
- Strong work ethic: Willingness to work long hours during model releases and get high-quality results out under tight deadlines
- Location : We are an in-person team based in San Francisco. We will support your relocation or transportation as needed
More about this role
We are looking for strong engineers to join our team and own the leaderboards that appear on Vals AI.
You'll be responsible for testing new models against our benchmarks as they're released; covering tasks in law, tax, coding, finance, social mobility, and more. You will analyze error modes of models, evaluate their strengths and weaknesses, and work with our communications team to release results.
Our results are used by startups, enterprises, and research labs alike. We work with all the major foundation model labs, some of the largest financial institutions, and hospital systems in the world. Our work has been featured by the Wall Street Journal, Washington Post, and Bloomberg.
We are building the standard for evaluating the ability of LLMs to perform real-world tasks. You will contribute directly to the leaderboards that make this possible.
Evaluate new LLM model releases across the Vals AI suite of benchmarks
Work directly with both open-source and closed-source foundation model labs in evaluating model performance
Use tools like Docent to analyze common failure modes and patterns in model performance
Work directly with our social media team to post interesting findings and...
Browse similar: AI jobs · AI startup jobs · Startup jobs · San Francisco Bay Area