An Intro to Building and Scaling Agentic Apps

The approach pairs three complementary layers for intelligence, user experience, and persistence, then closes the loop with evaluation and monitoring. The goal is a repeatable pattern teams can use to move beyond a notebook chatbot and into a customer‑facing app with safety and confidence.

A production stack that assigns clear jobs to each layer

Production agents need more than a model and a prompt. They need governed access to models and tools, a system of record, durable memory, audit trails, and a place users actually work.

• Agent Bricks provides the intelligence and context layer: LLMs, MCP tool servers, and managed context sources.
• Databricks Apps packages the agent into a real application with both UI and backend APIs for end users.
• Lakehouse, presented here as managed Postgres, backs the app and the agent's memory. Postgres compatibility lets frameworks like LangGraph plug in their Postgres savers for session history and checkpointing. Scale to zero, branching, and Unity Catalog native zero copy integration support safe iteration.

Unity AI Gateway governs access to models and tools so enterprise controls are respected. MLflow for observability and evaluation underpins tracing, experiments, judges, and evals across the lifecycle.

Scope tools narrowly with MCP, and know when to go custom

Raw text-to-SQL across entire schemas creates risk, latency, and cost for customer-facing scenarios. The recommended pattern wraps narrowly scoped operations in Unity Catalog functions, then exposes them as managed MCP servers. Vector search via AI Search and external MCP servers for systems like Confluence, Slack, or Glean can augment context without widening access to sensitive data.

Genie remains the fastest way to start many agents. Custom builds still matter when teams need to reuse investments in frameworks like LangGraph, exert tight control over which tools an agent can invoke, or steer behavior precisely in higher risk use cases. The two approaches are complementary. A custom orchestrator can call Genie through MCP to blend control with conversational analytics.

Close the AgentOps loop with simulation, judges, and measurable iteration

Quality comes from a disciplined loop: explore data, build the agent, simulate conversations, judge performance, and iterate on prompts, tools, or models until the bar is met. MLflow provides two linchpins:

• Conversation Simulator creates realistic multi‑turn traces that target known failure modes, such as recommending discontinued products or conflating manufacturer warranties with store policy. These traces seed a durable eval dataset.
• LLM judges, scaffolded by coding agents, score traces against business‑defined concerns. Judges can be compared across prompt versions, turning prompt work into a measurable practice rather than guesswork.

Prompt versions live in a prompt registry and can be A/B compared with consistent judges. The same judges can be enabled on future traces with a sampling rate so production conversations are scored for ongoing monitoring. The Databricks AI DevKit, used with coding agents like Genie, accelerates setup, from provisioning Lakehouse instances and wiring LangGraph's Postgres saver to generating app templates and registering judges.

Ship with guardrails, not gut feel

Confidence comes from narrow tools, governed access, persistent memory, synthetic evaluation, and continuous monitoring. Teams can put agents in front of customers knowing risky behaviors have been stress‑tested, improvements are measured, and live traffic is watched with the same judges that shaped development. The workshop assets and AI DevKit referenced in the session make the pattern reproducible from day one.