Session

How Data Warehousing Changes in the World of Agents

Overview

ExperienceIn Person
TrackData Warehousing
IndustryEnterprise Technology
TechnologiesDatabricks SQL
Skill LevelBeginner
The rise of agentic applications is reshaping what we demand from data warehouses. Agents don't operate on yesterday's batch — they require fresh, consistent data with sub-second latency to reason, decide, and act in real time. In this session, we examine why traditional data warehousing patterns, built around scheduled ETL and human-speed concurrency and latency, fall short when the consumer of your data is an army of agents making thousands of decisions per minute. We address the challenges this shift introduces: tighter freshness SLAs, evolving governance when agents —not people — query your data, and the complexity of always-on pipelines. We then explore how real-time lakehouse capabilities — streaming ingestion, incremental materialization, high concurrency and low-latency query serving, and unified governance — support agentic applications at scale. This session offers a framework to make your lakehouse real-time and agent-ready.

Session Speakers

Ruwen Henning Hess

/Director, Product Management
Databricks

Full Summary

How data warehousing must evolve for the age of agents

Agent-driven applications are reshaping enterprise data workloads. Concurrency is exploding, query patterns are shifting, and many organizations are letting agents act on data directly. A warehouse designed for human analysts struggles under that pressure. The message is clear: the shortcuts and side stacks that once worked now add cost, delay, and risk.

FAQ


Zerobus Ingest is a serverless connector that lets producer applications write directly into governed Unity Catalog tables in Delta Lake or Iceberg. It removes the separate message bus, auto-scales, and requires no cluster setup or maintenance. Apps integrate via SDKs or APIs, reducing complexity and cost.

Materialized views are incrementally refreshed by Enzyme, an AI-powered optimizer that processes only changed data. Because views are declarative and governed, a SQL definition produces a maintained pipeline with Unity Catalog lineage and access control, not just a cached result.

Lakehouse RT, powered by the Raiden engine, delivers tens-of-milliseconds latency at high concurrency and reads directly from open lakehouse tables. No serving copies are required, which preserves freshness and simplifies operations for large numbers of agents and users.

When agents move from informing to making decisions, the risk of silent failure grows. Unity Catalog applies one policy across ingestion, transformation, and serving, providing consistent access control, complete lineage, and auditability for agent activity.

No. The architecture runs natively on open Delta Lake and Iceberg tables with zero data duplication. Open formats preserve flexibility while enabling real-time ingestion, transformation, and serving under a single governance model.