Profound is the AI marketing platform built for the agentic era. See what your customers ask AI, deploy agents to act on it, and measure the results in a collaborative workbench. Backed by Kleiner Perkins, Lightspeed and Sequoia.
About the role
Build Profound’s eval system: golden datasets, scorers and LLM judges, programmatic verifiers, regression alerts, and dashboards Write evals for every pipeline behind the product: brand extraction, visibility scoring, sentiment, FactCheck, and citation classification Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- 3+ years building evaluation or measurement systems, whether AI evals, experimentation, ranking, or search quality
- Strong Python and data skills, with a track record of shipping production systems
- Deep familiarity with LLM evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and significance testing
- Sound experimental design, and the instinct to separate signal from noise in non-deterministic systems
- Strong opinions about how models and agents behave, and you keep up with current research
- You believe a good eval is clear, reliable, and hard to game
More about this role
Profound is the fastest-growing AI marketing platform, used by 15% of the Fortune 500 including Comcast, Estée Lauder, and Walmart, and innovators like Ramp, MongoDB, and Figma. It is where marketing teams do the work: research, briefs, content, campaigns, reporting, all in one place, alongside agents that carry the load with them.
Built on 2B+ real user prompts and growing, Profound’s data shows you what people ask AI and what the models say back. Profound's AI Marketer joins that data with deep context about our customers to find the opportunities worth acting on, and then puts agents to work: automating reporting, drafting content, auditing pages for readability, running competitive research. It measures what works and gets better every day.
As an Evals Engineer, you will own how Profound knows its AI is right. At Profound, measurement is the product: when a Fortune 500 CMO sees their brand’s visibility drop in ChatGPT, they act on that number. No public benchmark measures whether a model attributed a citation correctly or ran a competitive analysis well, so we build our own. You’ll decide what gets measured, why, and how results shape what we ship. Every product runs through...
Browse similar: AI jobs · AI startup jobs · Startup jobs