How to Modernize Your ETL with Declarative Pipelines
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Energy & Utilities, Manufacturing, Financial Services |
| Technologies | Lakeflow |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
Did you ever wonder why Lakeflow Spark Declarative Pipelines (SDP) do not allow dynamic parameters or partial recalculations for specific dates? Or why for-loops are restricted in table definitions? Furthermore, how do you choose between a Streaming Table and a Materialized View?
These are common struggles users face when starting. While often perceived as tool limitations, these hurdles usually stem from attempting to apply imperative habits to a declarative framework.
In this session we will discuss real-world customer examples to demonstrate how to transition from imperative to declarative approach.Stop fighting the framework and start leveraging the declarative paradigm for more resilient data engineering.
Session Speakers
Aleksandra Chashchina
/Specialist Solutions Architect
Databricks
Tomasz Bacewicz
/Product Specialist - Lakeflow
Databricks
Full Summary
Modernizing ETL with declarative pipelines: how data engineers can build faster on Databricks
Data engineering teams are hitting the limits of imperative ETL, where every pipeline encodes its own orchestration, incremental logic, and error handling. A declarative model, centered on Lakeflow Declarative Pipelines on the Databricks Platform, shifts focus to business intent while the framework standardizes execution. For organizations already investing in modern data engineering on the Databricks Platform, the approach promises simpler development, fewer defects, and easier scale. Teams looking to understand the broader shift can explore what data engineering for AI looks like in practice as context for why declarative approaches are gaining ground.
FAQ
You trade some low-level execution control for a framework that handles orchestration, checkpointing, and incremental logic. Business logic remains fully customizable. Highly specialized execution patterns may still warrant imperative code, but most ETL benefits from the standardization.
Yes. Lakeflow Declarative Pipelines supports scheduled batch and continuous streaming. Many components are incremental by design, and a real-time mode enables sub-second latencies for operational use cases such as fraud detection.
AUTO CDC uses a sequence_by field to order events. In SCD Type 1 it can ignore stale updates, and in SCD Type 2 it records late changes as inactive historical rows. The engine optimizes reads so merges touch only relevant target data.
Materialized views update incrementally, reflecting only the parts of sources that changed while still handling late data. Replace Where flows refresh a defined window, push filters to sources, and update only the days that actually changed.
Lakeflow Declarative Pipelines infers dependencies, validates them with dry runs, and emits an event log that can be queried for monitoring and alerting. The enhanced autoscaler reacts to backlog as well as compute utilization, improving both performance and cost control.