Advancing the Lakehouse with Apache Iceberg v3 on Databricks

Databricks supports Apache Iceberg v3 in the Data Intelligence Platform, giving customers a unified and open data layer with best-in-class performance, interoperability, and governance.

With this release, customers running Iceberg workloads can now take advantage of Databricks features, including Predictive I/O, Lakeflow Spark Declarative Pipelines, and row-level concurrency, which leverage deletion vectors, row-level lineage, and the Variant data type. This allows teams to run modern workloads efficiently and consistently. With Unity Catalog, these features work seamlessly across both Delta and Iceberg tables, enabling interoperability without rewriting data.

This release strengthens Databricks’ commitment to open standards and helps customers build on the lakehouse foundation of Delta Lake, Apache Iceberg, Apache Parquet, and Apache Spark, all with full governance and flexibility.

In this blog, we’ll explore:

• A unified data layer with Iceberg v3
• Efficient Iceberg v3 workloads on Databricks
• Advancing open table formats

A unified data layer with Iceberg v3

Delta Lake and Apache Iceberg have become the foundation of the modern lakehouse, each with strong capabilities for reliability, governance, and scalable data management. They both use metadata files to track Parquet data files and row-level deletes. However, minor differences between the formats in these data and delete files often forced organizations to pick a format and its features, usually based on which data platform they use. This choice was often irreversible, since rewriting petabytes of data is impractical.

Iceberg v3 closes this gap. It introduces features that align closely with Delta and the broader open ecosystem, such as Parquet and Spark, allowing teams to use a single copy of data with consistent behavior and performance across formats.

Databricks has long believed that the future of the lakehouse is optionality without fragmentation. Our contributions to Iceberg v3 reflect that commitment: helping unify core table behaviors so customers can use the engines and tools they prefer while governing everything consistently with Unity Catalog.

Efficient Iceberg v3 workloads on Databricks

With Iceberg v3, Databricks brings the features of the Data Intelligence Platform to all Unity Catalog managed tables.

Deletion vectors for faster updates with Predictive I/O

Deletion vectors enable deleting or updating rows without rewriting Parquet files. Instead, deletes are stored as separate files and merged during reads. Most data engineering workloads modify only a few rows at a time, making this a critical feature for efficient writes.

With Databricks, customers have the best read and write experience with deletion vectors. While deletion vectors improve write performance, they add a read penalty for the engine to filter deleted rows. Databricks intelligently balances read and write performance by applying and removing deletion vectors as needed using Predictive I/O. Compared to classic MERGE statements, Predictive I/O can speed up updates by up to 15x.

Customers also have the flexibility to use any engine with their Unity Catalog managed tables that have deletion vectors. This is the power of open standards: any engine across the Delta or Iceberg ecosystems can read and write to these tables using the Unity Catalog APIs. As Geodis notes:

“Now that Deletion Vectors have come to Iceberg, we can centralize our Iceberg data estate in Unity Catalog, while leveraging the engine of our choice and maintaining best-in-class performance.”  — Delio Amato, Chief Architect & Data Officer, Geodis 

Row lineage for incremental processing with Lakeflow Spark Declarative Pipelines

Row lineage gives each row a unique ID, making it easy to track changes over time. That means you process only what has changed instead of reprocessing everything, which improves efficiency and lowers costs. Row lineage is enabled on all Iceberg v3 tables.

Databricks leverages row lineage for incremental updates across the platform, using Lakeflow Spark Declarative Pipelines to create materialized views and streaming tables, Vector Search, Lakehouse Monitoring, and more. Additionally, with deletion vectors and row lineage, Databricks can reconcile concurrent modifications across the same files with row-level concurrency. Databricks remains the only lakehouse engine that brings this capability to open table formats.

Variant data type for flexible ingestion with Auto Loader

Modern data rarely fits neatly into rows and columns. Logs, events, and application data often arrive in JSON format. The Variant data type stores semi-structured data directly, offering excellent performance without the need for complex schemas or brittle pipelines.

Using the Variant data type in Databricks, you can land raw JSON data directly into their lakehouse tables using ingestion functions or Auto Loader to incrementally process semi-structured data files. Variant supports shredding, which extracts common fields into separate chunks to provide columnar-like performance. This speeds up queries for low-latency BI, dashboards, and alerting pipelines.

Variant works across both Delta and Iceberg. Teams using different engines can query the same table, including the Variant columns, without any conversion or data loss

“Gone are the days of simple scalar data, particularly for use cases that require security and application logs. Unity Catalog and Iceberg v3 unlock the power of semi-structured data through Variant and efficient data processing through row lineage. This enables interoperability and cost-effective, petabyte-scale log collection.”  —  Russell Leighton, Chief Architect, Panther 

Advancing open table formats

Iceberg v3 marks a major step toward unifying open table formats across the data layer. The next frontier is improving how formats manage and synchronize metadata at scale. Community efforts, such as the adaptive metadata tree first introduced at the Iceberg Summit, can reduce metadata overhead and accelerate table operations at scale.

As these ideas mature, they bring the Delta and Iceberg communities closer together, with shared goals around faster commits, efficient metadata management, and scalable multi-table operations. Databricks continues to contribute to this evolution, enabling customers to get performance and interoperability without being constrained by format-level differences.

Try Iceberg v3 Today with Databricks

These Iceberg v3 features are now available on Databricks, providing customers with the most comprehensive and future-ready implementation of the standard, backed by the governance and performance of Unity Catalog. While the broader ecosystem adopts v3, Databricks brings its value to customers immediately, leading adoption throughout the lakehouse across Delta and Iceberg tables.

Creating a Unity Catalog managed table with Iceberg v3 is easy:

Get started with Unity Catalog and Iceberg v3 and join us at upcoming Open Lakehouse + AI events to learn more about our work across the open ecosystem.