Rahul Mahadev is a Software Engineer at Databricks. He is a developer focusing on building Delta Lake and Structured Streaming. Rahul received his MS in Computer Science from the University of Illinois at Urbana-Champaign.
May 28, 2021 11:05 AM PT
Change Data Feed is a new feature of Delta Lake on Databricks that is available as a public preview since DBR 8.2. This feature enables a new class of ETL workloads such as incremental table/view maintenance and change auditing that were not possible before. In short, users will now be able to query row level changes across different versions of a Delta table.
In this talk we will dive into how Change Data Feed works under the hood and how to use it with existing ETL jobs to make them more efficient and also go over some new workloads it can enable.