Session

Building Production-Grade SQL ETL on the Lakehouse

Overview

ExperienceIn Person, Virtual
TrackData Warehousing
IndustryEnterprise Technology
TechnologiesDatabricks SQL
Skill LevelBeginner
DOWNLOAD SESSION SLIDES

ETL pipelines are often the most brittle part of the analytics stack — spread across tools, written in multiple languages, and hard to operate at scale. This session shows how teams build production-grade SQL ETL on Databricks Lakehouse using a unified, SQL-first approach. We’ll cover how to ingest, transform, and manage data using declarative pipelines, apply governance and quality controls consistently, author and optimize with AI, and deliver reliable datasets directly to analytics and downstream workloads. The focus is on simplifying architecture, reducing operational overhead, and enabling faster iteration without sacrificing correctness or scale.

Session Speakers

Mengfei Ren

/Engineering Manager
Databricks

Speaker placeholderIMAGE COMING SOON

Shanelle Roman

/Senior Product Manager
Databricks

Full Summary

Modernizing SQL ETL on the Lakehouse: From brittle pipelines to declarative flows

Handwritten SQL often powers enterprise ETL, but the real drag on reliability and speed sits in the plumbing around it: watermarks, schema evolution, backfills, and error recovery. A Data and AI Summit session shows how a fictional travel company, Wanderbricks, modernized SQL ETL by shifting from imperative scripts to declarative primitives that run natively on the Databricks Platform. Teams looking to understand the broader shift can explore data engineering for AI, which explains why declarative, AI-ready pipelines are becoming the standard. The result is simpler code, faster refreshes, and in-place adoption that preserves existing tables and consumers.

FAQ


No. Flows write to standard Delta Lake tables and managed Iceberg tables. Existing Unity Catalog grants, lineage, and downstream consumers remain intact, since you swap the job that updates a table rather than the table itself.

Yes. AutoCDC is generally available directly in the lakehouse and can be run from the SQL editor and from Databricks Notebooks connected to warehouse compute, using a streaming table with an AutoCDC flow.

Choose it when you routinely overwrite a time window, for example the last seven days, even though only a fraction changed. Enzyme updates only changed rows, delivering significant gains in speed and cost, and it supports native backfill predicates.

Flows and AutoCDC include managed checkpoints. The engine tracks offsets and state, resumes after failures, and supports rewinding a table to a prior point and replaying without custom truncate-and-reload logic.

The programming model is the same. Databricks SQL fits individual steps and smaller workloads, while Lakeflow Declarative Pipelines adds pipeline-level orchestration, graph visualization, multi-file project management, and data quality for full lifecycle management.