Session

How to Evaluate Your Agents From First Benchmark to CI/CD

Overview

ExperienceIn Person
TrackArtificial Intelligence & Agents
IndustryEnterprise Technology, Communications, Media & Entertainment
TechnologiesAI/BI, Databricks Agents, Lakebase
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES
If your team is building agents, evaluation is the difference between a demo that works and a product you can actually rely on. But most teams don't know where to start — which metrics matter, how to build a benchmark, how to automate tests, or how to catch regressions before they reach users.In this how-to session, we walk through agent evaluation end-to-end: defining your first benchmark, choosing the right judges and metrics for your use case, automating evaluation across prompts and workflows, and building a CI/CD loop that catches regressions before they hit production. We'll share practical patterns from real production teams and show how MLflow ties it all together.You'll leave with a concrete playbook for evaluating agents from first test to continuous quality at scale — and the confidence to ship agents that keep getting better with every iteration.

Session Speakers

Max Fisher

/Lead Solutions Architect
Databricks

Wenwen Xie

/GTM Staff Software Engineer
Databricks

Full Summary

Why agents stall before production: building trust through evaluation and CI/CD

Many agentic AI initiatives linger in prototypes, not because models are incapable, but because teams skip the disciplined evaluation, observability, and release practices that create trust. The session outlines a practical path from gut-feel checks to calibrated judges, production telemetry, and CI/CD gates that block regressions. The core idea is straightforward: trust is engineered, and it must be built into both development and deployment.

FAQ


Smaller than many expect. Focus on a targeted set that captures the most important, business-relevant failure modes. Production traces are best, SME-authored adversarial examples are next, and synthetic data should only fill remaining gaps.

Agents can fail in retrieval, tool selection, argument construction, generation, or end-to-end execution. A single score hides root causes. Layered metrics make failures diagnosable and improvements targeted.

Calibration against SME-labeled examples and a clear rubric aligned to application risks. Compare judge outputs to human labels, analyze disagreements, refine criteria, and repeat until the judge behaves consistently with expert expectations.

Fixed thresholds are early guesses that grow stale and are easy to game. Baselines derived from sampled production behavior evolve with the agent, so PRs are evaluated against current reality and meaningful deltas.

It serves as the centralized entry point for agent traffic, enabling consistent sampling, monitoring, and application of the same judges used in development. That single pane of glass ties production telemetry back to evaluation and CI/CD.