Session
Read-Time CDF in Delta Lake
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Enterprise Technology |
| Technologies | Databricks SQL, Lakeflow |
| Skill Level | Intermediate |
Traditionally, enabling Change Data Feed (CDF) in Delta Lake incurs a "write tax"—increasing storage costs and latency to materialize changes during ingestion.In this session, we introduce Read-Time CDF, a new architecture that unlocks zero-overhead writes by deferring change computation to query time. By leveraging the new unified CDC interface in Spark Data Source V2 and Delta’s Row Tracking, users can now query row-level changes without ever explicitly enabling delta.enableChangeFeed.
Session Speakers
Gengliang Wang
/Staff Software Engineer
Databricks
Johan Lasperas
/Staff Software Engineer
Databricks