Bulk Metadata Generation To Bring Genie to Life

The centerpiece is a bulk, human-in-the-loop metadata pipeline that generates, deduplicates, reviews, and deploys table and column descriptions at scale for Genie Agents on the Databricks Platform. The conversational analytics patterns described here build on the broader shift in how AI is transforming data analytics, where the quality of context and metadata determines whether natural language interfaces can be trusted at enterprise scale.

Genie assembles descriptions, relationships, metric views, and instructions into governed SQL. Without clear, consistent descriptions, even strong models guess. Optus faced roughly 15,000 tables with 40 to 80 columns each and cryptic names like CUST_PSTP_MBB_FLG. Manual stewardship was estimated at 6,000 hours, and per-table AI suggestions produced inconsistent results across bronze, silver, and gold layers.

Turning metadata into a scalable, reviewable workflow

The solution treats metadata generation as a pipeline with three core components: inventory, generation, and deduplication. Unity Catalog's information schema provides a complete inventory. A configurable LLM step applies industry-specific prompts, abbreviations, and synonym guidance, with optional value sampling to improve accuracy. A fingerprinting layer finds repeated column names across layers so a single reviewed description is applied consistently, which also contains LLM cost.

The production workflow ships as a Databricks Apps app with a React front end and a FastAPI back end, backed by a Postgres database for state, audit, and prompt libraries. Review tasks can be allocated by email to subject matter experts, edits are tracked, and only approved descriptions flow out as DDL for deployment via CI or CD. A specialized table-description prompt aligns to Genie best practices, producing richer multi-paragraph context about entities, grain, and relationships. It remains opt-in because longer prompts can hallucinate on noisier tables.

A configurable dictionary of abbreviations and suffixes reflects Optus-specific vocabulary. The app flags ambiguous tokens for closer human review and suggests expansions for new terms detected during scans. The generation step is PII-aware, skipping sampling on columns protected by row filters, masks, or governance tags, with tag-driven exclusions configurable to local policy.

AI-assisted build, production-grade in days

The demoed app evolved from an internal script into a production-grade tool built quickly with AI-assisted development. Using Databricks' internal coding workflow around Claude, the team turned screenshots and requirements into a working app in roughly three days and 52 commits. The app queries and applies metadata through Databricks SQL, and accesses models through Unity AI Gateway. Teams can follow the same pattern with Genie Code and Databricks Apps to move from prototype to shareable solution rapidly.

Impact at Optus and how it compounds

Generation across the estate now runs in roughly two weeks of compute and prompting, shifting the critical path to review rather than authoring. Optus prioritizes heavily used and heavily duplicated columns first, improving Genie quality where it matters most. The team targets about 80 percent coverage within one to two quarters, then plans to extend the review app to federated business teams working in their own sandboxes. A related field initiative, Genie Workbench, benchmarks a Genie space, proposes patches, and rescoring improvements. Better metadata directly lifts those measures, making the two efforts complementary.

Why the win is really about context operations

Enterprise-grade conversational analytics is won in the metadata layer. Consistent generation, confident deduplication, governed review, and full audit make Genie trustworthy and repeatable. Treating context as a product turns tuning from guesswork into an operational loop any large organization can sustain. The Databricks Free Edition is a pragmatic place to trial the pattern before scaling to a full catalog.