Agent Ops on Databricks, Powered by Omnigent

Agents are systems, not single models. They orchestrate tools, retrieval, memory, and reasoning, which blurs the line between model engineering and software engineering and expands the team to include software engineers, SMEs, and new evaluators such as LLM judges. Treating agents as governed systems is the lever that closes the gap between promising demos and production outcomes.

Why agents stall on the path to production

Prototypes fail without a per-action control plane. A model gateway can limit model access, yet cannot govern tool calls or sensitive actions inside long chains. Teams need runtime policies that allow, deny, or require approval before a tool runs, preventing prompt injection or risky actions instead of reporting them after the fact.

Reliability breaks under multi-step complexity. Compound and silent errors produce confident but wrong outcomes that vibe checks miss. Structured evaluation, including programmatic checks, LLM judges, and SME review, is required to trust behavior at scale.

Observability must unify traces and feedback. Teams need complete traces of every tool call and sub-agent handoff, plus attached feedback from users, SMEs, or judges, so failures can be reproduced and fixed. Maintenance costs spike when each use case rebuilds pipelines from scratch, and agents underperform without deep business context, from correct data sources to brand voice. Finally, crowded stakeholder rooms, each with different priorities, can stall progress without explicit alignment.

A composable AgentOps stack on the Databricks Platform

The speakers frame execution in three tempos inspired by Formula 1: laps for rapid dev loops, races for production releases, and seasons for longer-term strategy. Composability and policy control are central at every tempo.

Omnigent unifies multiple agentic systems with a single declarative agent YAML. Teams define sub-agents, function tools, MCP tools, models, harnesses, and runtime policies in one spec, then deploy across dev, staging, and prod using Databricks Asset Bundles. Resulting assets appear as governed objects in Unity Catalog. Changes to tools, models, or policies become configuration updates, not rewrites, which speeds iteration and reduces maintenance overhead.

Shared runtime guardrails operate per action. Policies can require user approval for sensitive steps, enforce organization-wide checks such as PII detection, and provide consistent behavior across all agents and use cases.

An evaluation flywheel across development, CI/CD, and production

MLflow anchors layered evaluation across environments. Teams define a consistent eval suite comprising programmatic checks, LLM judges, and optional human review, then run the same suite in development, on pull requests in CI/CD, and in production via scheduled monitoring jobs. That consistency helps catch regressions in safety, brand compliance, and clarity before they impact users.

Production quality improves through online and offline loops. Scheduled jobs apply known checks to fresh traces, while offline analysis uses coding agents with custom MLflow skills to scan thousands of traces, compare them to existing tests, and surface uncovered failure modes. Findings feed back into tests and policies, closing the loop between discovery and hardening.

Organizing for the long season

Operational rigor only works when matched with human alignment. Map stakeholders early, set shared success metrics, and move communications from education to metric-driven reporting as the project matures. Use a RACI-style role map, revisited periodically, to clarify ownership across product, engineering, compliance, and end users so quality scales across use cases rather than resetting each time.

Operational discipline, not model tuning, unlocks reliable agents

Successful production agents come from treating them as governed systems with evaluation, observability, and alignment built in. AgentOps turns ad hoc demos into durable services by unifying configuration, policy, and testing across the full lifecycle, enabling faster iteration without sacrificing control.