Session

Behind the curtain: Key learnings from building Genie

Overview

Experience	In Person
Track	Analytics & BI
Industry	Enterprise Technology
Technologies	Genie
Skill Level	Advanced

As organizations scale, the real challenge isn't just single talk AI — but designing a system like Genie that can coordinate multiple agents, domains, MCP connections, structured data, unstructured data, etc. while keeping the user experience effortless.

We've introduces a unified, multi-agent interface that allows anyone to ask complex questions across their entire data and agent estate and get answers without needing to understand how the system works underneath.

In this session, we’ll pull back the curtain on how Genie was designed and share the best practices we developed to make multi-agent analytics usable, reliable, and scalable.

In this session, we’ll cover:

How to design a unified experience across multiple agents and data domains
How to route, orchestrate, and combine results from different agents
How to balance flexibility with simplicity for business users
Key lessons learned building and deploying multi-agent systems in production

Session Speakers

IMAGE COMING SOON

Chung Wu

/Principal Software Engineer
Databricks

Yi Liu

/Engineer
Databricks

Full Summary

Inside Genie One: how Databricks is building an AI coworker that actually understands your business

Genie One is a new Databricks agent built to act as an AI coworker that works across data, AI/BI Dashboards, documents, and workflows. The promise is not chat for chat's sake, but an assistant that can reason over your actual enterprise context so questions like "What was revenue last quarter?" map to the right tables, definitions, and calculations.

The conversation offers a clear look at how conversational analytics with AI/BI Dashboards is moving beyond curated Q&A toward open-ended, grounded enterprise reasoning.

The core insight is that language models do not know your business semantics. Genie One addresses that gap by automatically reconstructing context from governed metadata and real user activity, then using a flat-first agent harness with strong grounding, authority signals, and strict access controls.

Rebuilding business context from your workspace activity

Traditional Genie Agents work well because a subject matter expert hand-crafts high-recall, high-precision, high-authority guidance. Genie One flips that model. It aims to understand a workspace out of the box, without upfront curation.

Genie One draws from two complementary sources. Unity Catalog provides governed definitions and metadata. The Databricks Platform contributes a richer layer of tacit knowledge encoded as people build AI/BI Dashboards, write in Databricks Notebooks, run Lakeflow Declarative Pipelines, and chat with agents. Genie One indexes these assets, enriches them with generated summaries and keywords, and distills them so retrieval is precise and token efficient. Lineage is tracked end to end so distilled knowledge inherits permissions from its sources.

Authority is established using a graph of authors, viewers, and assets, then propagating signals similar to PageRank. Popular assets tend to be built and consumed by experts, and experts prefer authoritative assets. Combined with Unity Catalog governance, this yields a practical notion of trust. The team refers to this governed and discovered knowledge layer as Genie Ontology.

A flat-first agent harness, with hierarchy only when it helps

With context in hand, the harness favors a flat architecture by default. A single agent keeps routing simple, concentrates state, is easier to debug, and is more token efficient with prompt caching. Recent models handle long flat traces increasingly well.

Hierarchy is introduced selectively when tasks grow complex or when an existing Genie Agent can be reused as a specialized sub-agent. Skills and tools are organized into bundles and loaded on demand, keeping prompts lean while preserving deep capability. Tool calls run with non-blocking parallelism, so fast results are used immediately while slow SQL queries continue in the background, maintaining responsiveness during exploration.

Agentic search and responsive execution across messy data

Finding the right assets in an open-ended workspace is pivotal. Genie One combines parallel exploration of asset types, type-specific ranking, iterative agentic search that adapts its focus as it learns, and a usage graph to understand relationships. Index quality is improved with LLM-generated summaries and usage-driven embeddings, turning search into guided discovery rather than basic retrieval.

Beyond search, exploration often fans out across metadata reads, table inspections, SQL runs, and side-by-side comparisons. Non-blocking parallelism lets the agent progress with partial results and incorporate slower outputs as they arrive, reducing end-to-end latency without sacrificing thoroughness.

Personalization, grounding, and auditability by design

Genie One personalizes retrieval and ranking based on recent activity, with personal memory features coming soon. MCP connections to systems like documents, email, calendar, chat, and tickets can further enrich context, while all ACLs remain in force.

Grounding is treated as mandatory. Aggregation, filtering, joins, and numeric reasoning are executed in Databricks SQL or Python, not inferred in the model's head. Conclusions and visualizations cite source tables, documents, and query results. Inputs and identifiers are validated against real assets. Authority and collaboration signals weight trustworthy sources more heavily, and injection-resistant sanitization fences untrusted content. Answers are reproducible, traceable, and auditable.

Enterprise agents win by system design, not model size

A capable enterprise agent depends less on a larger model and more on the surrounding system: how it discovers knowledge, adjudicates authority, manages context, and refuses to guess. Genie One's choices around ontology, harness design, search, personalization, and grounding point to an assistant that can be trusted with real work, not just fluent chat.

FAQ

Genie Agents are curated by subject matter experts for a specific topic, which delivers high-recall, high-precision, high-authority guidance. Genie One is a general-purpose workspace agent that reconstructs business context automatically from Unity Catalog and ongoing platform activity, removing the need for upfront manual curation.

Genie One blends Unity Catalog governance with a graph-based authority model similar to PageRank. Popularity, expert authorship, and expert consumption propagate through the graph so that certified or widely trusted assets rise above ad hoc or rarely used ones.

Yes. As knowledge is indexed and distilled, Genie One tracks lineage back to underlying assets. Users only see distilled knowledge derived from sources they are entitled to access, and personal MCP connections continue to respect ACLs.

A flat design is faster, simpler to debug, and more token efficient, especially with prompt caching, and it avoids lossy context handoffs. Sub-agents are added when task complexity demands isolation or when an existing Genie Agent can be reused as a specialist.

Genie One executes computations in Databricks SQL or Python, requires citations to source data and documents, validates inputs and identifiers, weights authoritative assets more heavily, and sanitizes untrusted content. These measures make answers reproducible and auditable.