Data Engineering with Databricks

This course prepares data professionals to leverage the Databricks Lakehouse Platform to productionalize ETL pipelines. Students will use Delta Live Tables to define and schedule pipelines that incrementally process new data from a variety of data sources into the Lakehouse. Students will also orchestrate tasks with Databricks Workflows and promote code with Databricks Repos.

Skill Level

Associate

Duration

12h

Prerequisites

Prerequisities for both versions of the course (Spark SQL and PySpark):

Beginner familiarity with cloud computing concepts (virtual machines, object storage, etc.)
Production experience working with data warehouses and data lakes
Familiarity with basic SQL concepts (select, filter, group by, join, etc)

Additional prerequisites for the Python version of this course (PySpark):

Beginner programming experience with Python (syntax, conditions, loops, functions)
Beginner programming experience with the Spark DataFrame API:
Configure DataFrameReader and DataFrameWriter to read and write data
Express query transformations using DataFrame methods and Column expressions
Navigate the Spark documentation to identify built-in functions for various transformations and data types

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Generative AI Engineer

New Capability Overview: DBRX

This course provides an in-depth overview of the new capability, DBRX, introduced in the Databricks Data Intelligence Platform. DBRX stands as a cornerstone of innovation within the Databricks Data Intelligence Platform. It represents a significant leap forward in language modeling technology, leveraging state-of-the-art techniques to deliver unparalleled performance and efficiency. In this course, we'll dive deep into how DBRX works, focusing on its architecture, and hands-on demonstrations illustrating its value across diverse use cases. By the end, you will have a solid understanding of DBRX and how it can be applied in real-world scenarios.

Paid & Subscription

Lab

Introductory

Apache Spark Developer

Apache Spark™ Programming with Databricks

Welcome to the Apache Spark™ Programming with Databricks course. This course is part of the Apache Spark™ Developer learning pathway and was designed to help you prepare for the Apache Spark™ Developer Certification exam. Please note - this is the self-paced version of the Apache Spark™ Programming with Databricks instructor-led course

In this course, you will explore the fundamentals of Apache Spark™ and Delta Lake on Databricks. You will learn the architectural components of Spark, the DataFrame and Structured Streaming APIs, and how Delta Lake can improve your data pipelines. Lastly, you will execute streaming queries to process streaming data and understand the advantages of using Delta Lake.

Machine Learning Practitioner

Scalable Machine Learning with Apache Spark™ (V2)

This course teaches you how to scale ML pipelines with Spark, including distributed training, hyperparameter tuning, and inference. You will build and tune ML models with SparkML while leveraging MLflow to track, version, and manage these models. This course covers the latest ML features in Apache Spark, such as Pandas UDFs, Pandas Functions, and the pandas API on Spark, as well as the latest ML product offerings, such as Feature Store and AutoML.

Career Workshop/

March 20

Careers at Databricks

We're on a mission to help data teams solve the world's toughest problems. Will you join us?

Advance my career now

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.

FAQ