Session

From Space to Scale: What to Do (and Not Do) for Secure, Fast, Production AI/BI

Overview

Experience	In Person
Track	Artificial Intelligence & Agents
Industry	Enterprise Technology
Technologies	AI/BI, Databricks Agents, Genie
Skill Level	Intermediate

Learn the essential, Databricks AI FDE -tested playbook for deploying production Genie. Start by scoping topic-focused Spaces that leverage Unity Catalog and knowledge-store metadata along with parameterized example SQL rather than free-text queries. This approach drives both clarity and speed.

Before scaling out, establish pragmatic, business-centric benchmarks using trusted assets and value dictionaries to measure what truly matters. Integrate Genie into multi-agent workflows through the Conversation API, Teams, and the integration into Databricks’ Agent Framework. Operationalize at scale with pre-joined and trimmed tables, audit-driven monitoring, and a lightweight review cadence.

We'll share proven do's and don'ts, common pitfalls to avoid, and a reusable checklist to accelerate your time-to-value, all grounded in Unity Catalog metadata and real-world enterprise deployments.

Session Speakers

Robert Martin-Short

/AI Engineer
Databricks

Yinxi Zhang

/Sr Staff Data Scientist
Databricks

Full Summary

From a single Genie space to a governed supervisor: best practices for scaling conversational analytics

Conversational analytics earns trust only when the underlying assistants are tightly scoped, well curated, and continually evaluated. The session explains how to design one high‑quality Genie space, how to govern many spaces under a supervisor, and how to automate best‑practice checks so quality does not drift as data and usage evolve.

The guidance applies directly to teams building self-service analytics on the Databricks Platform, where the gap between a prototype and a production assistant is usually disciplined curation and operations rather than model selection. Teams exploring the broader landscape of AI‑assisted querying will find useful context in what AI analytics is and why it only works with governed data, which explains the data governance foundations that make conversational assistants reliable at scale.

Start narrow and curate in a durable order

Begin with a focused business use case, roughly five tables per space, and instructions tailored to a single persona. Language and expectations differ across roles, so narrowing scope simplifies curation and improves accuracy.

Curate in priority order. First, strengthen table and column descriptions so the assistant can reliably select tables, choose columns, and infer joins from primary and foreign keys. Next, enrich the knowledge store with entity mappings, example joins, SQL expressions, and glossary terms. Then add trusted assets, including example SQL and callable SQL functions as deterministic tools. Treat free‑text instructions as a last resort, since they are general, occupy context every turn, and often mask weak metadata. When users change topics, start a fresh conversation to avoid stale context hurting quality and latency.

Make it production ready before you scale

Enforce production checkpoints. Benchmark in a lower environment and require a minimum accuracy threshold before rollout. If evaluation reveals dominant query patterns, stabilize them by creating materialized views so the assistant stops regenerating the same join logic on the fly.

Plan for throughput. Review current API rate limits, use an orchestrating agent with retries, and route known queries away from the live API path. Coordinate with the account team if you expect high concurrency and need quota increases. Deploy with a code‑first workflow: export space assets via the Genie export API, manage them in Git, and promote through CI/CD as Databricks Apps.

Supervise multiple spaces with clear routing and guardrails

GenAI 1 can automatically orchestrate across multiple spaces, but teams that need finer control can build a custom supervisor using Agent Bricks or a DIY orchestration. Clear tool descriptions drive accurate routing, and supervisors can combine Genie sub‑agents with SQL functions and MCP servers.

Protect sensitive data with on‑behalf‑of‑user authentication so row‑ and object‑level permissions in Unity Catalog apply per user end to end. Close the loop with tracing, evaluation, and monitoring, using MLflow to capture signals, run audits and regression tests, and sustain iterative improvements.

Automate best‑practice checks with an open‑source validator

Manual policy enforcement does not scale, especially as tables evolve and spaces drift. An open‑source Best Practices Evaluation Agent addresses this by pulling a space's exported configuration, a snapshot of referenced Unity Catalog tables, the Databricks best‑practices markdown, and an editable organization‑specific conventions file.

A reasoning model selected from Databricks Foundation model endpoints analyzes the configuration against these references, optionally runs the space's benchmarks, and produces a prioritized markdown report with critical issues and recommended fixes. Teams clone the repo into a Databricks Git folder, point it at a space ID, and iterate: apply changes, re‑run, and re‑check.

Treat conversational analytics like software you operate

High‑trust assistants come from software discipline: tight scope, strong metadata, reproducible deployment, governed supervision, and automated evaluation. Teams that adopt this loop can scale from one reliable Genie space to many without sacrificing accuracy or control.

FAQ

A tight scope keeps the space specialized to one use case and persona, which makes metadata, joins, and examples easier to curate accurately. Wider scopes dilute the signals the assistant uses to pick tables and joins, which lowers accuracy and makes debugging harder.

Text instructions are general, not tied to specific tables or columns, and they are injected into every turn, consuming context. Fixes through clearer table metadata, knowledge store entries, and trusted SQL assets are usually more durable and performant.

GenAI 1 can orchestrate multiple spaces automatically, which often suffices. A custom supervisor makes sense when you need finer routing control, specialized instructions, or integration with additional tools such as SQL functions or MCP servers.

It passes the end user's credentials through the supervisor to the data layer so Unity Catalog enforces row‑ and object‑level permissions per user. That prevents shared orchestration from exposing data a user is not allowed to see.

You need access to the Genie space you want to assess, its space ID, and the underlying Unity Catalog tables it queries. Adding benchmarks to the space is strongly recommended so the validator can run them and factor the results into its prioritized report.