Firecrawl is the web data API to search, scrape, and interact with the web at scale. Turn any source into clean Markdown or structured data your agents can ship with. Backed by Y Combinator.
About the role
Design and run head-to-head benchmarks of data providers against verified ground truth. For example: FullEnrich vs. DataLegion on finding the right person and their current employer and title. Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- You've shipped evals or benchmarks, and you can explain exactly why your results were trustworthy
- You've done this somewhere that matters: a frontier lab, a data company like Scale, Surge, micro1 or Mercor, a third-party benchmark org, or a public benchmark project. OSS contributors very welcome
- You're strong in Python and API integrations, and comfortable with messy vendor APIs, rate limits, and inconsistent schemas
- You know how to build test sets and scoring methods: sampling, labeling, inter-rater agreement, and when to trust an LLM judge and when not to
- You use AI heavily and keep upgrading your own workflow. You ship without waiting for instructions
- You write findings clearly for engineers, marketers, and the vendors on the other side of the leaderboard
More about this role
Every week, someone asks which data provider is actually best: for company records, for finding the right person, for fresh job listings. Right now the answers come from the vendors themselves. You'll build the independent version. You'll run rigorous, automated, public benchmarks of the providers in Alexandria and beyond, and publish them weekly. The same results will feed straight back into Alexandria so it learns which provider to call for which job.
This isn't our internal evals role. You're measuring the market, in public, where every number will be challenged by the vendors it ranks. You'll own it end to end: the datasets, the ground truth, the scoring, the harness, the weekly release, and the loop into Alexandria's provider selection. No one hands you a methodology. You write it, defend it, and ship it every week.
Equity Range: Competitive equity. Details shared during the process.
Equity Range: Competitive equity. Details shared during the process.
Job Type: Full-Time
Experience: 4+ years in ML, research engineering, or data engineering, with evaluation or benchmark work you've shipped
Work Authorization: Open to candidates already authorized to work in the US or Canada....
Browse similar: AI jobs · AI startup jobs · Startup jobs · Remote jobs · San Francisco Bay Area