Session

How to Build and Manage the Full Agent Lifecycle

Overview

Experience	In Person, Virtual
Track	Artificial Intelligence & Agents
Industry	Enterprise Technology, Retail & Consumer Goods
Technologies	Genie
Skill Level	Intermediate

Taking agents into production—coordinated, measurable, and grounded in enterprise data—is where most teams hit the wall. Fragmented components, inconsistent evaluation, and brittle orchestration turn promising prototypes into projects that never ship. This hands-on session walks through the full agent lifecycle on Databricks. You'll build modular, production-ready agents using Agent Bricks, AI Functions, and Genie Code—then orchestrate them with a Supervisor Agent that dynamically routes tasks across tools and MCP servers. We'll deploy a complete agentic application on Lakebase, fully governed through Unity AI. You'll see how to measure quality, trace behavior, and build confidence before and after deployment. Through real-world examples, you'll leave with clear patterns to build, evaluate, deploy, and scale intelligent multi-agent applications on Databricks' latest agentic platform.

Session Speakers

Amber Roberts

/Staff Tech Marketing Engineer
Databricks

Craig Wiley

/VP of AI
Databricks

Full Summary

From prototype to production: the two questions every AI agent must answer

Impressive demos are easy, production is not. The session argues that the real blockers to shipping agents into real workflows are governance, evaluation, and cost.

The Databricks Platform is presented as a unified way to build, monitor, and control agents across their full lifecycle so organizations can answer leadership's fundamental questions before go‑live. For a broader view of how these capabilities come together, the Databricks Platform shows how data, AI, and governance are unified under one roof.

The conversation reframes hype around models and orchestration as implementation details. What matters for production, and for organizational trust, is the ability to control what an agent can do, prove it works, and manage spend.

The CFO test: control and reliability first

Every production-bound agent must pass two leadership questions. Can you control it, meaning what it can access, what actions it can take, and what the worst case looks like? Does it actually work, including how it fails and recovers gracefully? Cost sits underneath both, because scale, usage, and failure modes all have spend implications.

Rapid model turnover has not changed those fundamentals. Model, framework, and harness choices affect performance, but they do not determine whether an agent ships. Production readiness depends on clear guardrails, repeatable evaluation, and visibility into resource usage.

From agent sprawl to governed fleets

Enterprises have shifted from piloting one agent to running many, often without inventory, accuracy metrics, or consistent controls. That sprawl creates risk when business users can assemble agents across clouds and SaaS without enterprise governance.

A platform answer combines three ideas: agents that truly know your data through retrieval, reasoning, and business semantics, a full‑stack developer experience that supports preferred models and frameworks, and unified governance that treats every agent, tool, and data source as a managed object. Unity Catalog is the control plane for that approach, registering data, tools, and even MCP servers so access, auditing, and policy are enforced in one place. Permissions flow through on behalf of the invoking user, so an HR agent returns different results for an employee than for the head of HR. Fine‑grained safety policies can be applied consistently across LLMs, agent harnesses, and MCP servers to prevent issues such as PII leakage regardless of the path a request takes. A cautionary HR chatbot that exposed other employees' W‑2s shows why correct answers on a test set are not enough, and Fox Sports' "Cletus" assistant illustrates the need for tight topic guardrails and accuracy to preserve fan trust.

Evaluation as the performance lever

Evaluation is the most underused way to turn a good agent into a great one. Databricks captures trace logs for every call, applies AI‑assisted judges, and routes hard cases to human subject matter experts via a dedicated review UI, all backed by MLflow. When a case fails, engineers can click through to see likely causes, such as flawed content in a vector store, plus concrete suggestions to fix it. Flow Healthcare uses this pattern so doctors can review medically oriented answers without operating inside core developer tools. Treating evaluation as a first‑class workflow, rather than growing prompts endlessly, produces measurable quality gains and faster iteration.

Choice without chaos for builders

Teams want choice, and they can have it without sacrificing control. The platform supports many orchestration frameworks, access to newly added models like Grok and Kimmy, and orchestration across structured tables, unstructured documents, meeting transcripts, and slide decks. AI functions let teams run models and agents inside SQL to classify, extract, or summarize at table scale. Operational visibility comes through standardized tracing, health monitoring, and the AI Gateway's usage and performance dashboards, so leaders can see what is running, how it is behaving, and what it costs.

Operational trust beats clever prompts

Production agents do not win on the newest model or flashiest orchestration. They win on the boring, essential disciplines: governing access and safety, running rigorous evaluations with human oversight where needed, and managing cost. Treating those as first‑class concerns turns agent sprawl into a reliable, auditable fleet the business can trust.

FAQ

Leaders ask two questions before approving production use: can you control it, meaning governance, access limits, and worst‑case understanding, and does it work reliably, including graceful failure. Cost is the implicit third consideration tying both together.

Every asset an agent touches, including data, tools, and MCP servers, is registered and virtualized through Unity Catalog. That single control plane enforces access by user context, provides auditable tracing, and applies consistent safety policies across models, harnesses, and external systems.

Evaluations run with MLflow capture full trace logs, apply AI‑assisted judges, and optionally route cases to human experts. Failure analysis highlights likely root causes and offers concrete remediation steps, such as correcting content in a vector store, so teams move from vague complaints to actionable improvements.

No. The platform emphasizes choice across foundation models, including newly added options like Grok and Kimmy, and supports a broad set of orchestration frameworks. Governance, evaluation, and cost controls remain consistent regardless of the model or library in use.

Agent sprawl occurs when business units build and deploy agents faster than the organization can inventory or govern them. Leaders often cannot say how many agents run or how accurate they are, which creates risk exposure and wasted spend without a unifying governance and evaluation layer.