Session

Atlassian's Journey of Using AI/BI Genie From Pilot to Production and What's Next?

Overview

Experience	In Person
Track	Analytics & BI
Industry	Enterprise Technology
Technologies	Genie
Skill Level	Beginner
DOWNLOAD SESSION SLIDES

This session is not a generic “AI is the future” talk. It is a practical, experience‑based guide to taking Databricks AI/BI Genie from a promising pilot to a trusted, production‑grade capability. Dashboards answer “what,” but not “why” or “what if.” Here is where Genie comes in.

We’ll walk through our journey at Atlassian from pilot to production—designing Genie spaces on curated tables and metrics, building an evaluation framework for accuracy and coverage, and integrating Genie into our everyday tools and Rovo AI agent. We will touch upon how Databricks and Atlassian partnered to bring the power of Genie to our Rovo Marketplace and built the ability to route a user question to the right Genie space.

In the end attendees will have a concrete “0‑to‑95*” playbook for unleashing their data with Genie to reach a point where roughly 95% of everyday questions can be answered accurately, safely, and self‑service freeing data teams to focus on higher‑value work and empowering faster decisions.

Session Speakers

Manav Trivedi

/Senior Product Manager
Atlassian

Prakash Reddy

/Head of Data Engineering
Atlassian

Full Summary

Inside Atlassian's playbook for production-grade self-serve analytics with Genie

Atlassian set out to let any employee ask data questions in plain English and get reliable answers. The hardest work was not picking a large language model. It was making data trustworthy, scoping problems tightly, and building organizational trust.

The session details how Atlassian took conversational analytics built on AI/BI Dashboards and Genie from a pilot to production use by hundreds of employees, and offers a repeatable operating model. Organizations evaluating similar investments will find the tradeoffs examined in depth in this complete guide to business intelligence and analytics in the AI era.

The story spans four moves: fix the lake before adding chat, learn from failed patterns, run Genie spaces like products, and remove adoption friction by embedding analytics where work happens. It ends with a forward look at governed context as core infrastructure.

Why governance and scope come before any chat interface

Years of growth and acquisitions left Atlassian with valuable but fragmented data: duplicative tables, conflicting metrics, tribal knowledge, and unclear ownership. A chat interface on top would have amplified bad data behind a friendly UI. The team first modernized the lake on the Databricks Platform with Unity Catalog governance, standardized ingestion, modular dbt pipelines, and a medallion architecture. Certified gold tables became the only substrate for self-serve analytics.

A tight scope rule emerged early. Even on a governed lake, broad or ambiguous domains multiplied disambiguation errors. Small, well-bounded use cases consistently outperformed expansive ones.

What broke in early patterns, and why Genie fit

Industry patterns the team tested each had ceilings. Intent-to-template routing was fast but brittle. RAG-driven SQL hinged entirely on metadata quality. Agentic loops were more robust but slower and hard to debug. Benchmarks like BirdBench showed accuracy plateauing near 70 percent without rich domain context. An internal hybrid, Data360, confirmed three lessons: metadata quality moved accuracy more than model choice, business definitions did not scale across domains, and maintenance exploded as upstream schema changes silently broke agents.

Genie addressed the parts Atlassian did not want to keep rebuilding. Curated Genie spaces provided governed, scoped containers for tables, metric definitions, sample questions, and instructions. Native text-to-SQL ran directly on Databricks SQL and honored Unity Catalog permissions out of the box. No separate context or plumbing layer was required, which let engineers focus on enriching business context.

From pilot to scale: treat spaces like products

The pilot targeted four domains with the most ad hoc demand, engaged stakeholders, and the richest metadata. Each space sat on certified gold tables, fewer than ten per space, with clear success criteria for accuracy, adoption, and ticket deflection. Out of the box, Genie delivered roughly 60 to 70 percent accuracy. After expanding table and column descriptions, clarifying metric glossaries, and adding few-shot examples, accuracy rose into the 80 to 90 percent range, depending on question complexity.

Three adoption insights mattered. Transparency built trust, since users liked seeing the generated SQL even if they could not read every line. Scope discipline beat comprehensiveness, with narrowly defined spaces outperforming broad ones. Hard cases remained multi-step joins and advanced analytical queries. Session memory for learned preferences was unreliable unless stored as instructions, and context-window limits forced ruthless selectivity in what to include.

Demand surged, so Atlassian shifted to a hub-and-spoke program. A small hub owned standards, tooling, governance, shared playbooks, and benchmarks. Domain teams owned their spaces and outcomes. Champions stress-tested spaces before broad rollout, and biweekly show-and-tells routed reusable patterns back into the playbook.

Meeting users in their flow: smart routing and distribution

Choosing the right space became a new friction point as spaces grew. Atlassian embedded a Databricks query runner agent into its Rivian platform and Teamwork Graph, so employees could ask questions directly inside Confluence and other work surfaces. The agent routed questions to the correct Genie space using user and document context, then called Genie conversation APIs to return answers. The Databricks query runner agent is available on the Atlassian Marketplace for organizations using both Atlassian and Databricks.

Where it is heading: governed context as shared infrastructure

Manually curating instructions, descriptions, and examples can work for dozens of spaces, not hundreds. Atlassian is investing in a context compiler that lets data owners author, govern, certify, and activate business context across surfaces, with a feedback loop tied to benchmarks and user flags. Treating context as a governed, machine-readable layer, not scattered prompt text, is the path to sustainable, production-grade AI analytics.

Data discipline beats prompt tricks when the stakes are trust

Production self-serve analytics is a data and operating-model problem first. Invest in governed gold data and precise metadata, scope spaces narrowly, benchmark continuously, scale with a hub-and-spoke model, and deliver answers where people already work. Model choice matters, but context quality and governance move accuracy and trust the most.

FAQ

Genie achieved about 60 to 70 percent accuracy on governed gold tables with no tuning. After enriching table and column descriptions, clarifying metric glossaries, and adding few-shot examples, accuracy rose to roughly 80 to 90 percent depending on question complexity.

Tight scope reduces ambiguity and disambiguation errors, keeps prompts within context limits, and makes evaluation practical. Spaces with fewer than ten tables and a single use case consistently outperformed broader ones.

Yes. Visibility into the generated SQL increased trust, even among non-technical users. Transparency signaled that a real system executed a real query rather than a black box making a guess.

A hub-and-spoke model. The hub owned standards, tooling, governance, benchmarks, and the playbook. Domain teams owned their Genie spaces and outcomes. Champions and regular show-and-tells accelerated reuse and quality.

Metadata quality. Rich, precise, governed table and column descriptions, clear metric glossaries, and business context moved accuracy far more than model selection or prompt tweaks. A benchmark evaluation loop made improvements measurable and durable.