Session

Delta Kernel <> Catalog-Managed Tables

Overview

ExperienceIn Person
TrackGovernance & Security
IndustryEnterprise Technology
TechnologiesUnity Catalog
Skill LevelBeginner

With the introduction of Catalog-Managed Tables, Delta Lake has unlocked unified governance, enforceable constraints, and a path towards multi-table updates.As a result, Delta connectors must adapt to properly load, read, and write Delta Tables, as the Catalog is now the authoritative broker of commits, not the filesystem.In this talk, learn how we designed Catalog Managed Tables support in Delta Kernel—the one right Delta abstraction to rule them all. We’ve built a unified catalog-agnostic API against which connectors can develop, enabled deep and powerful engine integrations for DuckDB, Delta-Spark, and Delta-Flink, all while keeping the complexity of catalog support out of Kernel.

Session Speakers

Speaker placeholderIMAGE COMING SOON

Scott Sandre

/Sr Software Engineer
Databricks

Full Summary

How Delta Kernel brings catalog-managed tables to any engine

Delta Kernel has become a central integration layer for Delta Lake tables, letting engines and connectors read and write Delta Lake without reimplementing the protocol. The talk advances that story by showing how Kernel now works with catalog-managed tables, so reads and commits can pass through a catalog for governance, auditability, and multi-table transactions, even when engines operate directly against cloud storage.

FAQ


No. The Delta Lake spec defines catalog commits without tying to a specific catalog. Kernel contains no Unity Catalog-specific code, and the Unity Catalog client ships separately as a crate or jar. Connectors can implement their own client if they follow the spec.

The flows remain similar. Load the table through a catalog to obtain the table path and a log tail of ratified commits, then read the main log plus those ratified files. On write, return generated commit actions to the engine or catalog client instead of issuing a direct put-if-absent.

The catalog tells it which commits are ratified. Kernel reads only those files from the underscore staged commits directory alongside the regular log and ignores unratified, failed, or aborted entries.

Yes. Because the catalog is in the commit path, the same write pattern can be replicated across multiple tables before ratification, enabling atomic multi-table operations that were not possible with single-table put-if-absent.

A Delta Kernel user guide is available and covers engine APIs, read and write paths, and how connectors integrate with file I/O. Searching for "Delta Kernel user guide" is a good starting point.