Automating Data Documentation with AI: How 7-Eleven Bridged the Metadata Gap

7-Eleven’s Data Documentation Dilemma

7-Eleven’s data ecosystem is massive and complex, housing thousands of tables with hundreds of columns across our Databricks environment. This data forms the backbone of our operations, analytics and decision-making processes. Traditionally, 7-Eleven’s data dictionary and documentation lived in Confluence pages, meticulously maintained by our data team members who would manually document table and column definitions.

We faced a critical roadblock as we began exploring the AI-powered features on the Databricks Data Intelligence Platform, including AI/BI Genie, intelligent dashboards and other applications. These advanced tools rely heavily on table metadata and comments embedded directly within Databricks to generate insights, answer questions about our data, and build automated visualizations. Without proper table and column comments in Databricks itself, we were essentially leaving powerful AI capabilities on the table. For example, when Genie lacks column definitions, it can misinterpret the meaning of bespoke columns, requiring end users to clarify. Once we enriched our metadata, Genie’s contextual understanding improved dramatically—accurately identifying column purposes, surfacing the right tables in response to natural language queries, and generating far more relevant and actionable insights. Simply put, Genie, like all AI agents, gets more thoughtful and more helpful when it has better metadata to work with.

The gap between our well-documented Confluence pages and our “metadata-light” Databricks environment was preventing us from realizing the full potential of our data platform investment.

Manual Migration’s Impossible Scale

When we initially considered migrating our documentation from Confluence to Databricks, the scale of the challenge became immediately apparent. With thousands of tables containing hundreds of columns each, a manual migration would require:

• Time-intensive labor: Hundreds of person-hours to copy and paste documentation
• Manual metadata updates: Crafting thousands of individual SQL statements to update metadata or going to each table UI
• Project oversight: Implementing a tracking system to ensure all tables were properly updated
• Quality assurance: Creating a validation process to catch inevitable human errors
• Ongoing upkeep: Establishing an ongoing maintenance protocol to keep both systems in sync

Human error would be unavoidable even if we dedicated significant resources to this effort. Some tables would be missed, comments would be incorrectly formatted, and the process would likely need to be repeated as documentation evolved. Moreover, the tedious nature of the work likely leads to inconsistent quality across the documentation.

Most concerning was the opportunity cost. While our data team focused on this migration, they couldn’t work on higher-value initiatives. Every day, we faced delays in strengthening our Databricks metadata, leaving untapped potential in the AI/BI capabilities already at our fingertips.

The Intelligent Document Processing Pipeline

To solve this challenge, 7-Eleven developed a sophisticated agentic AI workflow powered by Llama 4 Maverick, deployed through Databricks Model Serving, that automated the entire documentation migration process through an intelligent multistage pipeline:

1. Discovery phase: The agent uses Databricks APIs to get all tables, table names and column structures.
2. Document retrieval: The agent pulls all relevant data dictionary documents from Confluence, creating a corpus of potential documentation sources.
3. Reranking and filtering: Implementing advanced reranking algorithms, the system prioritizes the most relevant documentation for each table, filtering out noise and irrelevant content. This critical step ensures we match tables with their proper documentation even when naming conventions aren’t perfectly consistent.
4. Intelligent matching: For each Databricks table, the AI agent analyzes potential documentation matches, using contextual understanding to determine the correct Confluence page even when names don’t match exactly.
5. Targeted extraction: Once the correct documentation is identified, the agent intelligently extracts relevant descriptions for both tables and their columns, preserving the original meaning while formatting appropriately for Databricks metadata.
6. SQL generation: The system automatically generates properly formatted SQL statements to update the Databricks table and column comments, handling special characters and formatting requirements.
7. Execution and verification: The agent runs the SQL updates and, through MLflow tracking and evaluation, verifies that metadata was applied correctly, logs results, and surfaces any issues for human review.
8. Monitoring and insights: The team also uses the AI/BI Genie Dashboard to track project metrics in real time, ensuring transparency, quality control, and continuous improvement.

This intelligent pipeline transformed months of tedious, error-prone work into an automated process that completed the initial migration in days. The system’s ability to understand context and make intelligent matches between differently named or structured resources was key to achieving high accuracy.

Since implementing this solution, we plan to migrate documentation for over 90% of our tables, unlocking the full potential of Databricks’ AI/BI features. What began as a lightly used AI assistant has evolved into an everyday tool in our data workflows.. Genie’s ability to understand context now mirrors how a human would interpret the data, thanks to the column-level metadata we injected. Our data scientists and analysts can now use natural language queries through AI/BI Genie to explore data, and our dashboards leverage the rich metadata to provide more meaningful visualizations and insights.

The solution continues to provide value as an ongoing synchronization tool, ensuring that as our documentation evolves in Confluence, those changes are reflected in our Databricks environment. This project demonstrated how thoughtfully applied AI agents can solve complex data governance challenges at enterprise scale, turning what seemed like an insurmountable documentation task into an elegant automated solution.

Want to learn more about AI/BI and how it can help unlock value from your data? Learn more here.