Session

Building Higher Quality Genie Agents (fka Genie spaces): How to Ensure Accuracy and Trust

Overview

Experience	In Person
Track	Analytics & BI
Industry	Enterprise Technology
Technologies	Genie
Skill Level	Intermediate

High-quality Genie Agents (formerly known as Genie Spaces) are built on a foundation of clear metadata, curated semantics, and continuous iteration. New capabilities make it faster than ever to bootstrap and develop Genie Agents.

In this session, we'll show how to design, refine, and scale Genie Agents that consistently deliver accurate, trusted results. Topics include:

Using Unity Catalog Business Semantics (Metric Views) improve accuracy and consistency from the start
Leveraging Genie Code and knowledge mining to automate space setup
How to define metadata and context in the Genie knowledge store
How to use evaluation and Benchmarks to measure accuracy and guide improvements
Best practices for monitoring and optimizing Genie Agents at scale

Session Speakers

IMAGE COMING SOON

Hanlin Sun

/Product Manager
Databricks

Shah Amini

/Engineering Leader
Databricks

Full Summary

Building production-ready Genie Agents: a practical guide

Databricks' session at Data and AI Summit outlined a practical path for moving beyond text-to-SQL prototypes to production-grade Genie Agents. The core shift reframed agents as focused, evolving "mini-employees" that deliver reliable answers, learn from usage, and work across structured data, documents, and external tools.

For teams ready to ship AI agents that business users trust, the talk offered a crisp blueprint. The Databricks Platform brings together the data, governance, and AI capabilities that make this possible, and the AI/BI Dashboards and Genie product page explains how conversational analytics fits into that unified foundation.

Customers are asking for more than SQL generation. Support is expanding from Genie Spaces to Genie Agents so a single agent can reference metric definitions, query tables, analyze PDFs, and connect to third-party apps via MCP, all inside the Databricks Platform experience. The payoff is speed and accuracy on real workflows, not just query automation.

From text-to-SQL to focused business agents

Genie Agents extend Genie Spaces, not replace them. Spaces gain new capabilities such as document analysis and MCP tool connections, so existing deployments continue to work. Early adopters have already scaled agents widely, with examples like Meta's finance agents and General Motors deploying over 1,000 agents to compress hours of analysis into minutes.

Scope drives quality. Effective agents cover a single domain, for example finance, product usage, or sales, instead of answering everything for everyone. Narrow remit reduces ambiguity, improves retrieval, and simplifies maintenance.

Curate data, files, and instructions like an analyst would

Quality starts with semantic modeling. Define business metrics centrally with metric views in Unity Catalog so calculations stay consistent across agents. Provide clean, unambiguous datasets, add table and column descriptions, and model joins explicitly. Treat SQL logic as SQL, using SQL expressions and example queries to encode business rules that apply dynamically at runtime.

Author guidance works best as explicit rules, not vague nudges. Require filters or time windows where needed and avoid overloading instruction blocks that can conflict. Common pitfalls include duplicate metric definitions, missing joins, oversized or overlapping files, and embedding SQL logic as free text.

Files require the same discipline. Prefer digital-native PDFs over scanned versions to maximize machine readability, avoid very large files by splitting them, reduce overlapping versions, and assign clear purposes to each volume so the agent knows when to consult them.

Genie Code accelerates authoring. It can bootstrap a new agent from a natural language description, scan Unity Catalog to pull relevant tables, propose missing descriptions and joins, help set up metric views, and suggest targeted context improvements during debugging. Authors review and approve changes to keep control.

Prove quality with benchmarks and iterate with monitoring

Manual spot checks do not scale. Benchmarks in Genie Agents function like unit tests: define representative questions, expected answers or ground-truth SQL, and optional LLM-as-judge criteria. Run tests in chat or agent mode and schedule runs via API to catch regressions as context evolves.

Deployment meets users where they work. Agents can live in Databricks One and Genie Mobile, embed in Slack, Teams, and M365, or power custom apps via APIs, iframes, and MCP. Operational monitoring is built in, with a tab that lists conversations and feedback plus list-conversation APIs for programmatic access. Genie Code can analyze recent usage to surface top questions, highlight common issues such as undefined acronyms, and recommend concrete context updates. DABs and export or import APIs support promotion across dev, staging, and prod.

Unifying structured and unstructured analysis in one agent

A single Genie Agent can query tables and analyze PDF reports within the same conversation. Consolidating both data types under one agent often improves accuracy over routing through a separate vector search pipeline or a supervisor pattern, because tool selection remains precise and context stays coherent end to end.

Earning trust through scope, semantics, and steady iteration

Production success comes from disciplined scoping, clean semantic models, repeatable benchmarks, and ongoing monitoring treated as authoring. Teams that treat agents like evolving teammates and use Genie Code to tighten the loop from usage back to curation are the ones that scale adoption.

FAQ

No. Spaces are not being deprecated. Capabilities such as document analysis and MCP-based tool connections are being added to the experience you already use.

Use SQL expressions and example SQL for calculations, filters, and metric logic so rules apply dynamically per question. Reserve text instructions for behavior, and write them as explicit rules, for example always require a time range for sales questions.

Prefer digital-native PDFs over scanned files so text is machine-readable. Split files larger than 50 MB, reduce overlapping versions of the same content, and give each volume a clear purpose so the agent knows when to consult it.

Benchmarks are proactive tests that validate quality using curated questions, expected answers or SQL, and optional LLM-as-judge criteria, and can run on a schedule via API. Monitoring is reactive, surfacing real conversations and feedback, with Genie Code summarizing issues and recommending updates.

Yes. The demo showed a single agent querying an opportunities table and analyzing win-loss PDF reports in one flow. Keeping both sources under one agent often yields more granular and accurate answers than stitching separate pipelines.