Session

Sponsored by: Collibra | From Pilot to Production: Governing AI Agents on Databricks with Collibra

Overview

ExperienceIn Person
TrackGovernance & Security
IndustryHealthcare & Life Sciences, Retail & Consumer Goods, Financial Services
TechnologiesAI/BI, Unity Catalog, Databricks Agents
Skill LevelIntermediate

Most enterprise AI agents never make it past pilot – stalled by fragmented data, missing context, and governance that can't keep up. As organizations scale AI agents and semantic search on Databricks, the gap between data that exists and data that AI can safely use is widening. Learn how Collibra acts as the enterprise context layer across your Databricks ecosystem, bridging structured and unstructured data. We'll explore how Collibra integrates with Unity Catalog and Agent Bricks to deliver semantic interoperability, AI governance, and trusted data products end-to-end. See how Collibra’s MCP Server empowers AI agents with compliant context. Walk away with a blueprint for moving AI from pilot to production – faster, with less risk.

Session Speakers

Jeff Koss

/Staff Customer Engineer
Collibra

Full Summary

From pilot to production: Governing AI agents with Databricks and Collibra

Enterprises stall when AI agents leave the lab without trusted context about data, models, and the rules that govern both. The session explored how combining Collibra with the Databricks Platform creates the shared metadata, governance signals, and curated unstructured content agents need to operate reliably at scale.

FAQ


Unity Catalog governs assets inside Databricks. Collibra adds curated business context, policies, standards, contracts, ownership, and cross-platform lineage, and can push select attributes back to Unity Catalog as tags. The systems complement each other so agents and users see one governed view.

It routes each question to the right tool. Genie handles data analysis and querying, while the Collibra MCP server provides governance answers such as definitions, policies, data product membership, lineage, and SLAs. It can chain tools within one response.

The platform combines AI with pattern-based detection and contextual cues to suggest tags and classifications. That approach reduces the need to send every document to a model, which matters when targeting very large repositories.

Governed data slices can be scheduled to refresh automatically. When new files arrive or old ones are removed in a source like SharePoint, the slice updates so agents and chatbots always reference current content.

No. Spinning up a Genie agent with full semantic context via Collibra and Databricks APIs was described as roadmap, with no committed release date.