Clean, Correct Change Data Capture with Spark Declarative Pipelines
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Enterprise Technology, Consulting & Services |
| Technologies | Lakeflow |
| Skill Level | Intermediate |
Learn how to construct clean, correct, cost-conscious CDC pipelines that conquer chaotic change, column creep and complicated SCD calculations. Today, implementing CDC typically means writing complex, expensive merge logic. Handling straightforward cases is manageable — but complexity escalates quickly when you introduce out-of-order data, duplicates, schema evolution and slowly changing dimensions. What works in development often breaks under real production conditions.
Spark Declarative Pipelines (SDP) addresses this with Auto CDC, a declarative API that abstracts away merge orchestration and correctness guarantees. Instead of implementing imperative patterns, you declare intent — such as applying SCD Type 2 — and SDP executes it with built-in consistency, incremental processing and resilience.
In this session, we’ll walk through the Auto CDC API, examine how it automates complex merge operations and show how declarative CDC improves reliability across batch and streaming workloads.
Session Speakers
Andreas Neumann
/Senior Staff Software Engineer
Databricks
Joseph Torres
/Software Engineer
Databricks
Full Summary
Inside AutoCDC: How Apache Spark handles change data without the merge headaches
Change data capture is easy in theory and unforgiving in practice. Traditional streaming models treat a stream as an ever-growing prefix of a table, which works for append-only data but fails when updates, deletes, and late events enter the picture. AutoCDC in Apache Spark introduces a CDC-native ingestion model that preserves event-time semantics, handles deletes, and eliminates most hand-written merge logic. For teams replicating databases or rebuilding state from events on the Databricks Platform, it offers a simpler, more correct foundation.
FAQ
Update mode cannot represent deletes and has no event-time sequencing, so results depend on processing timing. Complete mode emits the entire result each batch, which is often impractical. AutoCDC brings explicit delete handling, version tracking, and event-time ordering to produce order-independent results.
No. Users declare target, source, keys, delete identification, event-time sequence, and storage. The engine manages auxiliary state, reconciliation, and tombstones that are typically implemented as complex and error-prone MERGE clauses.
No. Inserts and updates collapse into a single upsert behavior because AutoCDC reconciles by event-time sequence. If a workload has no deletes, users do not need to specify operation types at all.
AutoCDC is included in the current release candidate for Apache Spark 4.2. It has not yet been formally voted on, so it is not officially part of 4.2 until that process completes.
AutoCDC writes deletes to auxiliary state and retains them for a period so late inserts at older versions are correctly suppressed. Keeping tombstones forever would block physical deletion, while expiring them too early risks resurrecting removed rows, so the engine manages this tradeoff.