Labeling, evaluation, and quality workflows that deploy where your data lives and scale with the world. Backed by 500 Global.
About the role
Every Data Services engagement succeeds or fails on two things: whether the team doing the work understands the spec, and whether we can prove the output meets it. This role owns both. Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- Turn customer spec documents into project-ready enablement packages: annotator guidelines, decision trees, edge-case libraries with worked examples, and onboarding walkthroughs
- Build qualification tests and gold-standard sets that confirm annotators understand the spec before they touch production work
- Run pilot and calibration rounds at project kickoff to surface spec ambiguities early. Resolve those ambiguities with the customer and delivery lead before scaling up
- Maintain versioned guidelines throughout the project. Roll out updates as new edge cases appear, and confirm the workforce has absorbed the changes
- Design the quality plan for every project. Choose the review structure (gold tasks, overlap/consensus, multi-stage review, expert adjudication) based on the task type, risk, and budget
- Set the sampling methodology. Size samples to hit target confidence levels, stratify by class and difficulty, and adjust review rates for each annotator based on performance
More about this role
Real-world data is the competitive edge in AI.
HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their researchers from real-world data creation to annotation to delivery.
We design and create datasets from scratch, recruit and manage the domain experts who evaluate model output, and run everything through our own platform, Label Studio, the open-source standard for data labeling and evaluation, used by over 1 million practitioners worldwide.
We specialize in the operationally complex: real-world data collection, multimodal pipelines, and multi-step workflows. Advanced ML and AI teams use our enterprise platform to run their own data factories, and our services team to extend their reach where in-house capacity runs out.
If you want to do work that materially shapes how the next generation of AI products gets built, we'd love to talk.
Every Data Services engagement succeeds or fails on two things: whether the team doing the work understands the spec, and whether we can prove the output meets it. This role owns both.
As Head of Quality & Project Enablement, you'll take each customer spec...
Browse similar: AI jobs · AI startup jobs · Startup jobs · San Francisco Bay Area