Transactions on the Open Lakehouse Unlock Mission-Critical Warehousing Workloads

Historically, organizations running transactional workloads were forced to maintain separate data warehouses alongside their lakehouses, because lakehouse architectures lacked native transaction support. These fragmented architectures lead to drastic increases in total-cost-of-ownership (TCO), operational complexity, data duplication, and data staleness.

That changes now. Today, multi-statement, multi-table transactions are available in Public Preview. With this launch, Databricks becomes the first lakehouse architecture to support multi-statement, multi-table transactions across both Delta and Iceberg. This enables data teams to build their transactional warehousing workloads alongside their analytics and AI use cases, all under the governance of Unity Catalog. Databricks’ SQL features, including Transactions, Stored Procedures, and Scripting, make it easier than ever to migrate traditional warehousing workloads to the Databricks Lakehouse with confidence.

In this blog, we’ll cover:   

• The core problems that transactions solve on the lakehouse
• A practical example using Transactions, Stored Procedures, and SQL Scripting together
• How to get started using transactions on Databricks today

How do transactions on the lakehouse work? 

Transactions can write to any Unity Catalog managed table using industry-standard syntax. Performing a transaction is as simple as: 

Transactions can be executed on any SQL surface, including the SQL Editor, Notebooks, native JDBC/ODBC/PyDBC APIs, the Statement Execution API, and Job tasks. Tables involved in transactions can also be Delta Shared with your customers and partners.

Non-interactive transactions (BEGIN ATOMIC … END; shown above) are ideal for most use cases - and should be used for scheduled workloads. Interactive transactions (BEGIN TRANSACTION; … COMMIT; or ROLLBACK;) are useful for experimenting with transaction logic before going to production, debugging data across multiple tables, and building applications where database drivers perform conditional checks prior to committing transactions.

Why transactions on lakehouses used to be complex

Supporting transactions on a lakehouse was historically difficult because lakehouse architectures run on object storage, which is not designed for database-style transactional guarantees. Without native support, teams had to choose between limiting their workloads or engineering custom solutions to approximate atomicity, isolation, and rollback.

In practice, this led to the following common workarounds:

These approaches are brittle and expensive to maintain. Native transactions on the lakehouse remove the need for these workarounds, allowing teams to apply familiar warehouse patterns directly on open tables while preserving consistency at scale.

Fivetran is building ingestion connectors that leverage Databricks’ native transactions to reliably land data in customers’ lakehouses: 

“Databricks bringing native transactions to the lakehouse removes a long-standing source of complexity for data ingestion at Fivetran. It allows us to build connectors that can reliably land data into Unity Catalog–managed tables while preserving consistency, isolation, and correctness at scale.”  — Anjan Kundavaram, Chief Product Officer, Fivetran

Key performance benefits of Databricks transactions

Industry-leading concurrency support and transaction throughput

Modern data teams need to run many transactions at the same time, without slowing down as concurrency increases.

Databricks is the only lakehouse to support transactions with row-level concurrency without locks on open table formats. This means that concurrent transactions will only conflict if they are updating the same row. On the other hand, traditional warehouses guarantee atomicity by performing table-level locking, so transactions that write to the same table must be performed serially. 

Databricks supports concurrent transactions at volumes, throughputs, and latencies that traditional warehouses struggle to match, as shown in the comparative study below. 

Multi-table transactions on open table formats

With transaction support natively available in both Delta and Iceberg, customers retain maximum flexibility and performance across their entire data stack without sacrificing consistency. Since transactional guarantees are implemented natively using open table metadata, any engine or tool accessing Unity Catalog managed tables sees a consistent, committed view of data, even when there are ongoing concurrent transactions.

Key benefits of transactions on the lakehouse architecture

Unblocks migrations and ETL use cases

Native transaction support on the lakehouse lets customers confidently build warehousing workloads on the lakehouse architecture. It simplifies migrations from legacy warehouses and allows data teams to add transactional guarantees to their existing SQL-based lakehouse use cases.

Lower total-cost-of-ownership

As transactional workloads move onto the lakehouse, organizations no longer need to maintain complex ingestion pipelines between warehouses and their lakehouse. Unifying warehousing, analytics, and AI on a single architecture delivers a single source of truth, centralized governance, and reduced end-to-end TCO.

Databricks’ support of Transactions, Stored Procedures, and SQL Scripting helps AnyClip build its key warehousing workloads on Databricks: 

“Transactions, combined with all of the new SQL features like SQL Scripting and Stored Procedures, enable us to confidently migrate our most critical warehousing workloads to Databricks. These workloads underpin essential analytics across our business, and having robust transactional guarantees on the lakehouse is a game-changer.”  —Gal Doron, Head of Data, AnyClip

Example use case: ETL on the lakehouse with Transactions, Stored Procedures, and SQL Scripting

Let’s dive into a real-world example that requires core transactional functionality built into the lakehouse. We will consider a product company running a batch ingestion pipeline that must update both orders and total_sales atomically to keep analytics consistent.

Historically, many lakehouse deployments provided strong single-table ACID guarantees, but lacked multi-table ACID guarantees and support for transactional workflows. As a result, teams performed multi-table updates using a warehouse, and then replicated the results into their lakehouse for analytics and ML. That split increases complexity: duplicative copies of critical tables, potentially diverging sources of truth, siloed governance, and complicated ingestion pipelines to maintain.

Now, these transactional warehousing use cases can be built directly on the lakehouse without sacrificing consistency. In this example, we will write a transaction that leverages a stored procedure and SQL scripting to atomically update orders and total_sales together.

Step 1: Define the stored procedure.

Step 2: Define the transaction.

The stored procedure standardizes how orders are applied, and a variable is used to track staged rows. When the procedure is executed inside a transaction, the updates to the gold tables occur with full transactional guarantees.

Get started with transactions today

Transactions on managed Delta tables are now in Public Preview. Get started with transactions today using Databricks’ documentation or by prompting the in-product Assistant. 

Additionally, Lakebridge makes it seamless to convert transactions from other data warehouses to Databricks SQL.

Transactions on managed Iceberg tables are in Private Preview – fill out this form to express interest in the preview, and your account team will follow up with you once you’re enrolled.

Stay tuned for our follow-up post on Transactions, where we’ll walk through use case examples and best practices!