Our mission is to make the world more productive. To do this, we built the everything app for work - Tasks, Docs, Goals, and Chat. Backed by Lightspeed and a16z.
About the role
Design, build, and own ClickUp's AI evaluation system , including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features. Build and evolve the core agent harness , and the AI that performs well on those evals, partnering closely with product and engineering to define what "great" looks like. Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- 5+ years of software engineering experience, including 2+ years building production LLM or ML systems
- Hands-on experience building LLM evaluation systems in production (golden datasets, LLM-as-judge, judge calibration, offline and online evals), tied to real product quality
- Strong experiment design and statistical rigor, including A/B testing, offline vs. online metrics, and separating real regressions from noisy or variable results
- Proven track record of applying LLMs and AI models to real-world, customer-facing product features
- Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs
- Deep expertise in backend engineering, distributed systems, and cloud-native technologies (e.g., Kubernetes, Docker, AWS/GCP/Azure)
More about this role
At ClickUp, we're building the future of work: the first truly converged AI workspace unifying tasks, docs, chat, calendar, and enterprise search, all supercharged by context-driven AI. We are an AI-native company. Every team member is expected to leverage AI daily, and we evaluate AI fluency as part of our hiring process. Join us and help redefine what's possible. 🚀
We're looking for a highly skilled and experienced Senior AI Engineer to join our ClickUp Engineering team, with a focus on AI evaluations . You'll own how we measure and improve the quality of ClickUp's AI: building the eval systems that tell us how well our AI is performing, the core agent harness, and the AI that performs well on those evals. You'll work on backend systems that make AI-powered capabilities scalable, reliable, and secure, while working hands-on with LLMs to solve real user problems and drive product innovation.
Design, build, and own ClickUp's AI evaluation system , including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features.
Build and evolve the core agent harness , and the AI that performs well on those evals, partnering closely with...
Browse similar: AI jobs · AI startup jobs · Startup jobs · Remote jobs