Scalable Machine Learning with Apache Spark™

This course teaches you how to scale ML pipelines with Spark, including distributed training, hyperparameter tuning, and inference. You will build and tune ML models with SparkML while leveraging MLflow to track, version, and manage these models. This course covers the latest ML features in Apache Spark, such as Pandas UDFs, Pandas Functions, and the pandas API on Spark, as well as the latest ML product offerings, such as Feature Store and AutoML.

Skill Level

Associate

Duration

16h

Prerequisites

Intermediate experience with Python, Experience building machine learning models, Beginner experience with PySpark DataFrame API

Outline

Day 1

Spark / ML overview
Exploratory data analysis (EDA) and feature engineering with Spark
Linear regression with SparkML: transformers, estimators, pipelines, and evaluators
MLflow Tracking and Model Registry

Day 2

Tree-based models: Hyperparameter tuning and parallelism
HyperOpt for distributed hyperparameter tuning
Databricks AutoML and Feature Store
Integrating 3rd party packages (distributed XGBoost)
Distributed inference of scikit-learn models with pandas UDFs
Distributed training with pandas function API
Pandas API on Spark for data manipulation

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

Request Private Training

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Analyst

Data Analytics and Warehousing with Databricks SQL

This course provides an introduction to Databricks SQL. Learners will ingest data, write queries, produce visualizations and dashboards, and configure alerts using Databricks SQL.

Advanced Data Engineering with Databricks

In this course, students will build upon their existing knowledge of Apache Spark, Structured Streaming, and Delta Lake to unlock the full potential of the data lakehouse by utilizing the suite of tools provided by Databricks. This course places a heavy emphasis on designs favoring incremental data processing, enabling systems optimized to continuously ingest and analyze ever-growing data. By designing workloads that leverage built-in platform optimizations, data engineers can reduce the burden of code maintenance and on-call emergencies, and quickly adapt production code to new demands with minimal refactoring or downtime.

The topics in this course should be mastered prior to attempting the Databricks Certified Data Engineer Professional exam.

Languages Available: English | 日本語 | Português BR