Advanced Data Engineering with Databricks

In this course, participants will build upon their existing knowledge of Apache Spark, Delta Lake, and Delta Live Tables to unlock the full potential of the data lakehouse by utilizing the suite of tools provided by Databricks. This course places a heavy emphasis on designs favoring incremental data processing, enabling systems optimized to continuously ingest and analyze ever-growing data. By designing workloads that leverage built-in platform optimizations, data engineers can reduce the burden of code maintenance and on-call emergencies, and quickly adapt production code to new demands with minimal refactoring or downtime. The topics in this course should be mastered prior to attempting the Databricks Certified Data Engineering Professional exam.

Skill Level

Professional

Duration

12h

Prerequisites

Ability to perform basic code development tasks using the Databricks Data Engineering & Data Science workspace (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc)
Intermediate programming experience with PySpark
Extract data from a variety of file formats and data sources
Apply a number of common transformations to clean data
Reshape and manipulate complex data using advanced built-in functions
Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions etc.)
Beginner experience configuring and scheduling data pipelines using the Delta Live Tables (DLT) UI
Beginner experience defining Delta Live Tables pipelines using PySpark
Ingest and process data using Auto Loader and PySpark syntax
Process Change Data Capture feeds with APPLY CHANGES INTO syntax
Review pipeline event logs and results to troubleshoot DLT syntax

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Generative AI Engineer

New Capability Overview: DBRX

This course provides an in-depth overview of the new capability, DBRX, introduced in the Databricks Data Intelligence Platform. DBRX stands as a cornerstone of innovation within the Databricks Data Intelligence Platform. It represents a significant leap forward in language modeling technology, leveraging state-of-the-art techniques to deliver unparalleled performance and efficiency. In this course, we'll dive deep into how DBRX works, focusing on its architecture, and hands-on demonstrations illustrating its value across diverse use cases. By the end, you will have a solid understanding of DBRX and how it can be applied in real-world scenarios.

Data Engineering with Databricks

This course prepares data professionals to leverage the Databricks Lakehouse Platform to productionalize ETL pipelines. Students will use Delta Live Tables to define and schedule pipelines that incrementally process new data from a variety of data sources into the Lakehouse. Students will also orchestrate tasks with Databricks Workflows and promote code with Databricks Repos.