Scaling Databricks Genie 10x at GSK Global Supply Chain: Building a Production AI/BI Factory

Designing focused Genie spaces that scale

Grounding agents on a vast data estate required a repeatable approach. The team centered on Genie, Databricks' managed semantic layer for structured data, delivered through narrowly scoped Genie for conversational analytics. Quality improved when every space followed four rules:

• Scope to one business domain and target persona.
• Push business logic into the data layer, using materialized views and gold tables.
• Teach through parameterized SQL examples, not prose instructions.
• Benchmark changes against a curated gold set of 20 to 50 SQL queries.

Lifecycle was treated as a loop: define business context and jargon, prepare a semantic model on well-documented Unity Catalog tables, contextualize with instructions and SQL exemplars, benchmark iteratively, then expose the space to agents via APIs only after it meets the gold standard.

Seeing and fixing latency at scale

User expectations were set by consumer chatbots, so minute-long responses were unacceptable. Engineers instrumented Genie end to end, tracking SQL generation time, warehouse execution time, and internal retry counts to locate bottlenecks such as poor SQL, warehouse queuing, or fragile space setup.

A global view came from dashboards built on Databricks audit tables. Queries per minute highlighted rate limits and attribution to specific spaces and users. Per-message latency was approximated by differencing first and last polling events with window functions, then correlated back to request-level traces for root cause analysis and tuning.

Routing around LLM cost with a tiered execution path

Many questions were repetitive, yet still paid the full natural-language-to-SQL round trip. A three-tier routing layer in front of Genie reduced compute and latency:

• SQL tools for the highest frequency asks, where an LLM fills parameters into vetted SQL templates and executes directly.
• A semantic cache, a vector database of question-to-SQL pairs, that routes by similarity. At 0.99 or higher, it replays SQL. Between roughly 0.8 and 0.99, it feeds top neighbors to a general-purpose LLM that is grounded in the same semantic model. Below 0.8, the query goes to Genie.
• Continuous instrumentation of path selection, execution time, cache rate, and correctness supported threshold tuning per use case.

Benchmarks showed median latency dropping from 34 seconds to 22 seconds, a 33 percent improvement, and a 64 percent reduction in calls to Genie. Correctness dipped by 3 to 4 percent, which the team accepted for the faster, more scalable experience. An initial latency bump appeared when adding the semantic cache, reflecting lookup overhead that subsides as cache hit rate grows.

Operating like software: CI/CD and trust at enterprise scale

Genie spaces moved from manual promotion to versioned code. Export APIs serialized spaces with ACLs and per-environment transforms like warehouse IDs and catalog names. A dispatch workflow applied transforms and imported to higher environments, enabling review, testing, rollback, and promotion alongside the agents published to a shared registry.

Adoption hinged on people. Spaces were co-created with subject matter experts who help train and refine them, feedback buttons were embedded in the UI to close the loop, and ongoing training built literacy. Company-wide sponsorship, including an internal assistant rolled out broadly, shifted sentiment from skepticism to demand for more agents across the supply chain.

The boring work that makes GenAI valuable in production

Enterprise GenAI depends on pragmatic rigor: tight semantic scopes, deep observability, smart routing and caching, CI/CD, and human-centered rollout. Those foundations are enabling the shift from asking questions to diagnosing and acting, with the end goal of a more autonomous supply chain that gets medicines to patients faster.