Building Query Expert MCP: Using RAG to Build an Analytics Agent That Goes Beyond Text-to-SQL
Overview
| Experience | In Person |
|---|---|
| Track | Artificial Intelligence & Agents |
| Industry | Financial Services |
| Technologies | Databricks SQL, Unity Catalog, Databricks Apps |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
LLMs are fast and confident SQL writers that can unlock self-serve analytics across your business. But actionable insights require more than SQL syntax—they demand business context, domain expertise and nuances that live outside your schema docs.
In this talk, we'll deep-dive into how Block built Query Expert MCP, a scalable AI agent that captures and distributes institutional data knowledge using Databricks. You'll learn how we:
- Automated documentation at scale—using query_ai sub-agents to generate standardized table documentation across 11,000+ tables
- Captured SQL nuances—Databricks sub-agents categorize and label domain-specific query patterns across 1M sample queries
- Human in the loop—Databricks Apps allow users to directly edit, remove or upload context to the agent’s memory
- Enabled intelligent retrieval—Databricks Vector Search indexes for RAG-powered query generation that scales to a 12,000 person org, while supporting domain-specific scoping for teams' use cases
Session Speakers
Daniel Gole
/Data Scientist
Block
Philip Azar
/Data Scientist
Block
Full Summary
Inside QueryXpert: How Block built a company-wide data agent on Databricks
Block faced a familiar enterprise problem: vital data knowledge lived across business units, BI tools, and tribal Slack threads, which turned simple questions into week-long hunts for the right table, owner, or metric definition. QueryXpert, a company-wide data agent, tackles that bottleneck by codifying how a data scientist searches, validates, and explains data, so employees can self-serve trusted answers in the agent harness they already use.
FAQ
QueryXpert focuses on exploratory and analytical questions where discovery and context matter. Governed metrics and executive reporting, such as weekly revenue, flow through a separate metric store MCP so that high-stakes numbers remain deterministic and consistent.
Retrieval is biased toward verified tables and labeled queries, with time-decay penalties for older information. The agent validates joins using observed probability distributions between tables, and progressive disclosure hides low-confidence context, which reduced hallucinations in evaluations.
Lakeflow Jobs regenerate table and column descriptions from recent query history using LLM calls inside SQL, then open GitHub pull requests for review. Deprecations cascade to remove or flag affected queries and docs so stale context does not linger.
They add a queryxpert comment label directly inside SQL to describe what a query measures. Pipelines ingest these labels, embed the descriptions, and give them higher weight during re-ranking, which lets contributions happen within normal workflows.
No. The tool is treated as a non-deterministic coworker that accelerates discovery and first drafts. Humans review the analysis plan and final synthesis, and some data literacy is still required for nuanced or small-sample questions.