Native Streaming-to-Lakehouse Governance with Unity Catalog
Overview
| Experience | In Person |
|---|---|
| Track | Governance & Security |
| Industry | Enterprise Technology, Communications, Media & Entertainment, Financial Services |
| Technologies | Unity Catalog |
| Skill Level | Intermediate |
Real-time data powers modern AI and analytics yet governing high-velocity streaming data in the lakehouse remains a challenge. In this session, StreamNative and Databricks showcase a native integration that brings governance to streaming pipelines through Databricks Unity Catalog without sacrificing performance or simplicity.
We introduce catalog-based commits, a new approach where streaming data written in Delta Lake and Apache Iceberg formats is registered and governed in Unity Catalog at commit time. This eliminates post-processing, manual metadata synchronization and governance gaps, while enabling consistent access control, lineage and auditing across streaming and batch workloads.
Attendees will learn how this integration reduces operational complexity, accelerates AI and real-time analytics and delivers a seamless path from event streams to governed, AI-ready lakehouse tables.
Session Speakers
Benjamin Mathew
/Product Manager
Databricks
KUNDAN VYAS
/Director,Product & Partnerships
StreamNative
Full Summary
Streaming Kafka and Pulsar topics directly into governed Delta Lake tables
Enterprises want streaming data in the lakehouse without losing governance, lineage, or simplicity. The session introduces catalog commits, an open standard that lets external streaming engines write directly into Unity Catalog managed Delta Lake tables with full policy enforcement and audit in place. The approach removes the long-standing gap where streaming pipelines wrote to Delta Lake by storage path, bypassing the catalog and creating governance blind spots. With catalog commits now generally available, teams can connect Kafka and Pulsar topics to governed Delta Lake tables on the Databricks Platform and preserve a unified control plane for access, audit, and schema management across engines.
FAQ
No. The session describes enabling catalog commits on existing Unity Catalog managed Delta Lake tables via a simple ALTER TABLE statement, which switches the table to use the catalog as its commit coordinator without recreating it.
Latency-optimized clusters use persistent storage to deliver ultra-low-latency behavior common in Kafka workloads. Cost-optimized clusters skip persistent storage and write directly to object storage, trading some latency for significant cost savings on analytics-oriented streaming pipelines.
Streaming generates many small files. StreamNative runs a compaction service that consolidates these into larger Parquet files as part of the managed table lifecycle, so downstream readers get efficient file sizes without custom maintenance jobs.
Catalog commits brings catalog-coordinated writes to Delta Lake tables, aligning Delta Lake with Iceberg's catalog-oriented design. StreamNative supports landing topics into either Delta Lake or Iceberg managed tables within Unity Catalog using a similar operator experience.
On Databricks, create or identify a Unity Catalog managed Delta Lake table and enable catalog commits with an ALTER TABLE statement. In StreamNative Cloud, create a cluster, choose the latency or cost profile, register your Unity Catalog, and start streaming Kafka or Pulsar topics into governed Delta Lake tables.