Skip to main content

Advanced Machine Learning with Databricks

This course is aimed at data scientists and machine learning practitioners and consists of two, four-hours modules. 


Machine Learning at Scale

In this course, you will gain theoretical and practical knowledge of Apache Spark’s architecture and its application to machine learning workloads within Databricks. You will learn when to use Spark for data preparation, model training, and deployment, while also gaining hands-on experience with Spark ML and pandas APIs on Spark. This course will introduce you to advanced concepts like hyperparameter tuning and scaling Optuna with Spark. This course will use features and concepts introduced in the associate course such as MLflow and Unity Catalog for comprehensive model packaging and governance.


Advanced Machine Learning Operations

In this course, you will be provided with a comprehensive understanding of the machine learning lifecycle and MLOps, emphasizing best practices for data and model management, testing, and scalable architectures. It covers key MLOps components, including CI/CD, pipeline management, and environment separation, while showcasing Databricks’ tools for automation and infrastructure management, such as Declarative Automation Bundles (DABs), Lakeflow Jobs, and Model Serving. You will learn about monitoring, custom metrics, drift detection, model rollout strategies, A/B testing, and the principles of reliable MLOps systems, providing a holistic view of implementing and managing ML projects in Databricks.


Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.

Skill Level
Professional
Duration
8h
Prerequisites

The content was developed for participants with these skills/knowledge/abilities: 

• Basic understanding of DS/ML concepts (e.g. classification and regression models), common model metrics (e.g. F1-score), and Python libraries (e.g. scikit-learn and XGBoost).

• The user should have intermediate-level knowledge of traditional machine learning concepts, development, and the use of Python and Git for ML projects.

• It is recommended that the user has intermediate-level experience with Python. 

• Access to a Databricks workspace with administrator permissions and familiarity with basic Databricks operations (create clusters, run notebooks, basic notebook operations)

• Intermediate experience with Git version control, including repository management and Personal Access Token (PAT) configuration for GitHub integration

• Basic knowledge of CI/CD workflows, pipeline configurations, and DevOps concepts for automated deployment processes

• Intermediate programming experience with Python, MLflow for model tracking and management, and Unity Catalog for data governance

• Familiarity with machine learning model development lifecycle, including feature engineering, model training, validation, and deployment concepts

• Experience with command line interfaces, particularly Databricks CLI installation, configuration, and authentication using personal access tokens

• Understanding of model deployment strategies, including A/B testing, traffic distribution, and real-time inference concepts

• Basic knowledge of Lakeflow Jobs for job creation, task dependencies, and workflow orchestration

Outline

1. Machine Learning at Scale

Model Development with Spark

• Spark Architecture Overview

• Introduction to Spark ML for Model Development

• Model Tracking and Packaging with MLflow

• Demo: Model Development with Spark

• Lab: Model Development with Spark

Model Tuning with Optuna on Spark

• Overview of Hyperparameter Tuning

• Scalable HPO Frameworks on Databricks 

• Demo: Hyperparameter Tuning with SparkML

• Demo: HPO with Ray Tune

Model Deployment with Spark

• Deployment with Spark

• Inference with Spark

• Demo: Model Deployment with Spark

• Demo: Optimization Strategies with Spark and Delta Lake

• Lab: Model Deployment with Spark

Pandas APIs

• Scaling with Pandas APIs

• Pandas UDFs and Function APIs

• Demo: Pandas APIs

• Lab: Pandas APIs


2. Advanced Machine Learning Operations

Overview of Machine Learning Operations

• Review of MLOps

• Streamlining Development to Deployment

Streamlining MLOps with Databricks

• Streamlining MLOps

• Streamlining MLOps with Databricks

• Demo: Building a CI/CD Pipeline with Databricks CLI

• Lab: Building a CI/CD Pipeline with Databricks CLI

Model Rollout Strategies with Databricks

• Automate Comprehensive Testing

• Demo: Common Testing Strategies

• Demo: Integration Tests with Lakeflow Jobs

• Model Rollout Strategies with Databricks

• Demo: Model Rollout Strategies with Databricks AI Model Serving

• Lab: Rollout Strategies with Lakeflow Jobs

Data Profiling

• Data Profiling

• Demo: Data Profiling Model Quality

• Lab: Monitoring Drift with Data Profiling

Build ML Assets as Code

• Build ML Assets as Code

• Demo: Working with Declarative Automation Bundles

Upcoming Public Classes

Date
Time
Your Local Time
Language
Price
Oct 28 - 29
01 PM - 05 PM (America/New_York)
-
English
$1000.00
Jan 05
09 AM - 05 PM (Europe/Paris)
-
English
$1000.00
Jan 12
10 AM - 06 PM (Asia/Kolkata)
-
English
$1000.00
Jan 27 - 28
09 AM - 01 PM (Asia/Kolkata)
-
English
$1000.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Engineer

Databricks Performance Optimization - Mandarin Chinese

Databricks Performance Optimization 课程教导数据工程师和分析师如何在 Databricks Data Intelligence Platform 上诊断、衡量和修复性能瓶颈,以及如何将性能提升与成本关联起来。本课程遵循"先衡量、先平台"的工作流:让平台的自动优化功能承担繁重工作,通过 Query Profile 和系统表验证已应用的优化,再在必要时进行手动调优。

学员将以 Query Profile、Performance Insights 和查询历史记录系统表为基础,学习如何衡量性能。在此基础上,他们将探索关键优化技术,包括数据布局与自调优托管表、liquid clustering、缓存与中间结果、shuffle、数据倾斜、溢出、行爆炸、驱动程序性能、Python UDF、serverless compute、Photon 以及成本归因。

在整个课程中,学员将针对合成零售数据中刻意设计的慢查询进行探索,观察不同优化技术对性能的影响。他们将利用文件裁剪、任务执行时间、Photon 覆盖率和成本等依据来评估优化效果。两个基于场景的实验室将强化从诊断到验证的完整优化工作流。

注意:Databricks Academy 正在将 Databricks 环境中的课堂教学转为基于 notebook 的形式,不再使用幻灯片进行授课。您可以在 Vocareum 实验环境中访问课程 notebook。

Languages Available: English | 日本語 | Português BR | 한국어 | Français | Español | 普通话

Paid
4h
Lab
instructor-led
Professional

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.