Databricks Streaming and Lakeflow Declarative Pipelines

This course provides a comprehensive understanding of Spark Structured Streaming and Delta Lake, including computation models, configuration for streaming read, and maintaining data quality in a streaming environment.

Note:

1. Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!

2. This course is the first in the 'Advanced Data Engineering with Databricks' series.

Skill Level

Professional

Duration

Prerequisites

The content was developed for participants with these skills/knowledge/abilities:

⇾ Ability to perform basic code development tasks using the Databricks Data Engineering and Data Science workspace (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)

⇾ Intermediate programming experience with PySpark

⇾ Extract data from a variety of file formats and data sources

⇾ Apply a number of common transformations to clean data

⇾ Reshape and manipulate complex data using advanced built-in functions

⇾ Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)

⇾ Beginner experience configuring and scheduling data pipelines using the Lakeflow Declarative Pipelines UI

⇾ Beginner experience defining Lakeflow Declarative Pipelines using PySpark

⇾ Ingest and process data using Auto Loader and PySpark syntax

⇾ Process Change Data Capture feeds with APPLY CHANGES INTO syntax

⇾ Review pipeline event logs and results to troubleshoot Declarative Pipeline syntax

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Customer registration Partner registration

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Machine Learning Practitioner

Feature Engineering at Scale

In this course, you will gain a comprehensive understanding of how to design, scale, and operationalize end-to-end feature engineering pipelines on the Databricks platform. The curriculum is structured across three progressive modules: mastering the fundamentals of Spark’s distributed execution and optimization, implementing scalable data ingestion with Auto Loader and declarative Lakeflow pipelines, and advancing to production-grade MLOps with the Databricks Feature Store.

You will engage in hands-on learning experiences such as debugging Spark performance with the Catalyst Optimizer and Spark UI, building robust Bronze-Silver-Gold medallion architectures with automated quality checks, and implementing scalable feature transformations using SparkML. The course culminates in deploying real-time feature serving through Online Feature Stores, defining FeatureSpecs with on-demand transformations, and applying governance and lineage tracking with Unity Catalog.

By the end of the course, you will be equipped to manage the complete feature lifecycle for machine learning at enterprise scale—designing pipelines that are performant, reliable, governed, and production-ready.

Paid & Subscription

Lab

Professional

Data Warehousing Practitioner

Data Warehousing with Databricks

This course is designed for data professionals who want to explore the data warehousing capabilities of Databricks. Assuming no prior knowledge of Databricks, it provides an introduction to leveraging Databricks as a modern cloud-based data warehousing solution. Learners will explore how use the Databricks Data Intelligence Platform to ingest, transform, govern, and analyze data efficiently using the industry-standard TCP-DI dataset as a reference. Learners will also explore Genie, an innovative Databricks feature that simplifies data exploration through natural language queries. By the end of this course, participants will be equipped with the foundational skills to implement and optimize a data warehouse using Databricks.

Note: Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!

Languages Available: English | 日本語 | Português BR | 한국어

Deploy Workloads with Lakeflow Jobs

Deploy Workloads with Lakeflow Jobs course teaches how to orchestrate and automate data, analytics, and AI workflows using Lakeflow Jobs as a unified orchestration platform within the Databricks ecosystem.

⇾ You will learn to design and implement data workloads using Directed Acyclic Graphs (DAGs), configure various scheduling options, and implement advanced workflow features such as conditional task execution, run-if dependencies, and for each loops.

⇾ The course covers best practices for creating robust, production-ready pipelines with proper compute selection, modular orchestration, error handling techniques, and fault-tolerant design-all natively integrated within the Databricks Data Intelligence Platform.

Paid & Subscription

Lab

Associate