Building ETL Pipelines with SQL
This course teaches how to build production-ready ETL pipelines using pure SQL on the Databricks Data Intelligence Platform. Students learn Streaming Tables with Auto Loader for incremental ingestion, Materialized Views with incremental refresh for Silver-to-Gold transformations, AUTO CDC (FLOW AUTO CDC) for declarative SCD Type 1 and Type 2 dimension management, and Apache Spark Declarative Pipelines (SDP) to group these objects into production multi-table pipelines with data quality expectations, PRIVATE tables, and unified monitoring. The course follows a realistic retail dataset through the medallion architecture (Bronze → Silver → Gold). Orchestrating with Lakeflow Jobs is available as a bonus module.
Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.
In this course, the content was developed for participants with these skills/knowledge/abilities:
• Navigating the Databricks workspace (sidebar, Catalog Explorer, SQL Editor)
• Unity Catalog basics (catalogs, schemas, tables, volumes)
• Intermediate SQL (SELECT, JOIN, GROUP BY, CAST, COALESCE, CREATE TABLE)
• Data warehousing concepts (fact/dimension tables, star schemas, medallion architecture)
• Basic understanding of ETL workflows
Outline
1. SQL ETL on Databricks
SQL ETL on Databricks: The Big Picture
Demo: Exploring the Course Dataset and SQL Editor
Lab: Using the SQL Editor and Genie Code
2. Streaming Tables and Materialized Views
Building SQL ETL Pipelines
Demo: Building a Bronze-to-Silver Pipeline
Lab: Building a Customer Feedback Pipeline
3. Auto CDC
AUTO CDC Streaming Dimension Updates
Demo: Building Slowly Changing Dimensions with AUTO CDC
Lab: Building Slowly Changing Dimensions
4. Building Pipelines with Spark Declarative Pipelines
From Standalone SQL to Production Pipelines
Demo: Building an SDP Pipeline in the Multi-File Editor
5. (Bonus) Orchestrating with Lakeflow Jobs
Orchestrating SQL Pipelines with Lakeflow Jobs
Demo: Building a Lakeflow Job for the ETL Pipeline
Lab: Orchestrating SQL Pipelines with Lakeflow Jobs
Upcoming Public Classes
Date | Time | Your Local Time | Language | Price |
|---|---|---|---|---|
Oct 16 | 09 AM - 01 PM (Asia/Kolkata) | - | English | $750.00 |
Oct 16 | 01 PM - 05 PM (America/New_York) | - | English | $750.00 |
Nov 17 | 09 AM - 01 PM (America/Los_Angeles) | - | English | $750.00 |
Nov 24 | 01 PM - 05 PM (Australia/Sydney) | - | English | $750.00 |
Dec 15 | 01 PM - 05 PM (Europe/Paris) | - | English | $750.00 |
Jan 26 | 09 AM - 01 PM (Asia/Singapore) | - | English | $750.00 |
Jan 26 | 01 PM - 05 PM (America/New_York) | - | English | $750.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
