Advanced Data Engineering with Databricks

In this course, participants will build upon their existing knowledge of Apache Spark, Delta Lake, and Delta Live Tables to unlock the full potential of the data lakehouse by utilizing the suite of tools provided by Databricks. This course places a heavy emphasis on designs favoring incremental data processing, enabling systems optimized to continuously ingest and analyze ever-growing data. By designing workloads that leverage built-in platform optimizations, data engineers can reduce the burden of code maintenance and on-call emergencies, and quickly adapt production code to new demands with minimal refactoring or downtime. The topics in this course should be mastered prior to attempting the Databricks Certified Data Engineering Professional exam.

Skill Level

Professional

Duration

12h

Prerequisites

Intermediate programming experience with PySpark
Extract data from a variety of file formats and data sources
Apply a number of common transformations to clean data
Reshape and manipulate complex data using advanced built-in functions
Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)
Beginner experience configuring and scheduling data pipelines using the Delta Live Tables (DLT) UI
Beginner experience defining Delta Live Tables pipelines using PySpark
Ingest and process data using Auto Loader and PySpark syntax
Process Change Data Capture feeds with APPLY CHANGES INTO syntax
Review pipeline event logs and results to troubleshoot DLT syntax

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Engineer

Deploy Workloads with Databricks Workflows

By scheduling tasks with Databricks Jobs, applications can be run automatically to keep tables in the Lakehouse fresh. Using Databricks SQL to schedule updates to queries and dashboards allows quick insights using the newest data. In this course, students will be introduced to task orchestration using the Databricks Workflow Jobs UI. Optionally, they will configure and schedule dashboards and alerts to reflect updates to production data pipelines.

In this course, you’ll learn how to orchestrate data pipelines with Databricks Workflow Jobs and schedule dashboard updates to keep analytics up-to-date. We’ll cover topics like getting started with Databricks Workflows, how to use Databricks SQL for on-demand queries, and how to configure and schedule dashboards and alerts to reflect updates to production data pipelines.

Note: This course is part of the 'Data Engineering with Databricks' course series.

Free

Associate

Machine Learning Practitioner

Machine Learning Operations

This course will guide participants through a comprehensive exploration of machine learning model operations, focusing on MLOps and model lifecycle management. The initial segment covers essential MLOps components and best practices, providing participants with a strong foundation for effectively operationalizing machine learning models. In the latter part of the course, we will delve into the basics of the model lifecycle, demonstrating how to navigate it seamlessly using the Model Registry in conjunction with the Unity Catalog for efficient model management. By the course's conclusion, participants will have gained practical insights and a well-rounded understanding of MLOps principles, equipped with the skills needed to navigate the intricate landscape of machine learning model operations.

Free

Associate

Data Engineer

Databricks Streaming and Delta Live Tables

This course provides a comprehensive understanding of Spark Structured Streaming and Delta Lake, including computation models, configuration for streaming read, and maintaining data quality in a streaming environment.

Note: This course is part of the 'Advanced Data Engineering with Databricks' course series.

Free

Professional

Career Workshop/

March 20

Careers at Databricks

We're on a mission to help data teams solve the world's toughest problems. Will you join us?

Advance my career now

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.

FAQ