Spark DSV2: Growing Up Fast
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Enterprise Technology |
| Technologies | Databricks SQL |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
Spark’s DataSource V2 integration has taken a major step forward, beginning with the addition of a procedure catalog and row identifier support to enable richer table management and row-level operations across modern data sources. Building on this foundation, recent updates improve MERGE INTO with safer schema evolution and enhance partition filtering for more efficient query planning and execution. DML summaries now provide clearer visibility into write behavior, while key cache fixes resolve long-standing correctness issues in Spark execution. DataSource V2 has also been extended with first-class SQL features such as table constraints, complex default values, and generated columns, laying the groundwork for more advanced table semantics. We’ll also look ahead to deeper Change Data Feed (CDF) integration in Spark to support robust incremental processing, highlighting what’s available today and what’s coming next as Spark continues to close the gap with traditional warehouse systems.
Session Speakers
Anton Okolnychyi
/Senior Staff Software Engineer
Databricks
Szehon Ho
/Software Engineer
Databricks
Full Summary
Inside Spark DSV2: How a Unified Data Source API Is Reshaping Apache Spark
Apache Spark has powered large-scale data processing for a decade, but table formats and connectors have evolved faster than its original data source API. Data Source V2, or DSV2, is Spark's answer, creating a stable, consistent contract between the engine and modern storage systems such as Apache Iceberg and Delta Lake, plus a long tail of connectors.
FAQ
Usually not. Many capabilities reuse existing DSV2 interfaces, such as ALTER TABLE and the write API. Transactional commits and type-widening negotiation add small new hooks that connectors must implement to participate fully.
They target open source Apache Spark 4.1 and 4.2. Iceberg and Delta Lake are adapting to the new APIs so users of either format on open source Spark can benefit.
Transactions are scoped to a single DML statement and a single catalog. Explicit multi-statement blocks and cross-catalog transactions are not supported yet.
The connector receives a predicate as a black box with an eval function. It evaluates partition values without understanding the UDF, enabling pruning driven by arbitrary expressions.
CDC execution involves complex logic, for example removing no-op rows, stitching delete and insert pairs, and computing net changes across versions. Centralizing in Spark ensures consistent behavior and avoids reimplementing complex execution in multiple communities.