eBook
A Guide to Building ETL Pipelines With SQL
Own your ETL from development to production, using only SQL

SQL practitioners can now build and run production ETL pipelines end to end without leaving SQL. With Apache Spark™ Declarative Pipelines (SDP), you describe the result you want and Databricks handles the incremental processing, orchestration and data quality. Common ETL patterns are expressed as brief declarative flows, such as APPEND, AUTO CDC and REPLACE WHERE. This guide shows SQL analysts, analytics engineers and data warehouse teams how to get pipelines into production faster, with fewer handoffs.
In this eBook, you'll learn:
- Why declarative beats procedural for SQL ETL, and how to stop maintaining scheduling, MERGE logic and orchestration glue by hand
- How to express common ETL patterns in a few lines of declarative SQL, from appending new records to applying change data capture and running incremental batch updates
- How to keep data fresh without reprocessing everything, so pipelines stay fast and cost-efficient as data grows
- Two paths to production: open source SDP, or the fully managed Lakeflow experience on Databricks
- How to modernize legacy warehouse ETL by moving existing SQL to Databricks, then evolving it into declarative pipelines at your own pace
This eBook is for SQL analysts, analytics engineers and data warehouse engineers who want to own more of the pipeline without waiting on data engineering.