Session

Agent Ready Data Governance: Databricks on Databricks

Overview

ExperienceIn Person, Virtual
TrackGovernance & Security
IndustryEnterprise Technology, Public Sector
TechnologiesAI/BI, Unity Catalog
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES

When your data consumers shift from 10,000 humans to 100,000 high-speed agents, “governance” can no longer mean a wiki page or a PDF policy document. Agents don’t read wikis; they need semantic, machine-readable guardrails. In this talk, the Data Platform team discusses our pivot to “agent-ready governance.” We are moving beyond simple access control lists (ACLs) to a model where Unity Catalog provides deep semantic context, usage constraints and “rules of the road” that our 100,000 agents can natively understand. Join us to see how we ensure that our digital workforce operates safely, preventing the chaos of 100,000 bots making bad decisions at machine speed.

Session Speakers

Amit Pahwa

/Sr. Staff Software Engineer
Databricks

Prashant Gupta

/Senior Software Engineer
Databricks

Full Summary

Governing 100,000 agents: how Databricks rebuilt its own data platform for the agentic era

Enterprises are racing to infuse AI into every function. When agents inherit human permissions, move at machine speed, and optimize only for the prompt, legacy governance crumples. After six months of real-world experimentation across 100-plus workspaces and 500 SaaS systems, the team found a clear mandate: move governance into the platform, ahead of access, because audits and training cannot keep up. The enterprise AI agent trends shaping governance, use cases, and evaluations make clear that prevention—not better warnings—is the only durable answer.

FAQ


3PI is an internal, no-code ingestion platform Databricks built for its own operations. It composes Lakeflow Connect, Unity Catalog, and Lakeflow Jobs with a thin configuration layer. Customers can assemble similar patterns using the same building blocks.

Agents end up joining and filtering in their context window instead of a query engine. That leads to whole-table pulls, token burn, rate limits, and occasional fabrications when work seems slow or costly. Moving data into the lakehouse first avoids these pitfalls.

Unity Catalog data classification tags sensitive columns as soon as data lands. ABAC policies, defined in the pipeline config, automatically mask those columns on every read. Agents receive masked values without any change to their query.

3PI uses declarative configs to apply residency filters, redact sensitive fields, and aggregate locally. Only then is a scrubbed view shared via Delta Sharing into a central workspace, so raw regulated data never crosses compliance borders.

Exclusive groups replace permissions when a role is assumed, preventing agents from holding two sensitive keys at once and blocking risky joins. Unity AI Gateway provides on-behalf-of tracing, plus contextual policies, budgets, and smart routing for governing how LLMs and agents access the lakehouse.