Streamline your builds with Procore's construction project management software. Connect field and office teams, track project progress, and manage workflows in real time. Backed by Bessemer and ICONIQ.
About the role
We're looking for a Senior AI Engineer to join Procore's Evals & Goldsets team, which measures and improves the quality of the AI agents we ship to the construction industry. You'll build and run the systems that tell our AI teams how well their agents are doing: production tracing in Langfuse and Databricks, our online LLM judge, and the goldsets and benchmarks behind our offline evals. Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- 5+ years of experience in software, AI, or ML engineering
- Strong proficiency in Python, experience with TypeScript is a plus
- Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement
- Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar
- Experience building data pipelines, ETL processes, or streaming data systems
- Experience with analytics platforms and data warehouses such as Databricks or Snowflake
More about this role
We're looking for a Senior AI Engineer to join Procore's Evals & Goldsets team, which measures and improves the quality of the AI agents we ship to the construction industry. You'll build and run the systems that tell our AI teams how well their agents are doing: production tracing in Langfuse and Databricks, our online LLM judge, and the goldsets and benchmarks behind our offline evals.
As a Senior AI Engineer , you will partner with data and ML engineers to make our evals faster, cheaper, more trustworthy, and harder to saturate, and you'll turn production traces into feedback loops that improve agent behavior. We're a small, fast-moving team that pairs hands-on building with architecture and review, and we value engineers who are rigorous about measurement and comfortable with ambiguity. If you want to shape how AI in construction gets tested and improved, come join us.
This hybrid position reports to the Engineering Manager for Evals & Goldsets and will be based in our Austin, TX office.
- Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
- Build and maintain production trace infrastructure with Langfuse and...
Browse similar: AI jobs · AI startup jobs · Startup jobs