Skip to main content

Databricks Performance Optimization

In this course, you’ll learn how to optimize workloads and physical layout with Spark and Delta Lake and and analyze the Spark UI to assess performance and debug applications. We’ll cover topics like streaming, liquid clustering, data skipping, caching, photons, and more.

Note: This course is part of the 'Advanced Data Engineering with Databricks' course series.


Languages Available: English | 日本語 | Português BR | 한국어

Skill Level
Professional
Duration
4h
Prerequisites

The content was developed for participants with these skills/knowledge/abilities:  

  • Ability to perform basic code development tasks using Databricks (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)
  • Intermediate programming experience with PySpark
  • Extract data from a variety of file formats and data sources
    • Apply a number of common transformations to clean data
  • Reshape and manipulate complex data using advanced built-in functions
  • Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)


Outline

  • Spark Architecture

Spark UI Introduction

  • Designing the Foundation File Explosion 

 Introduction to Designing Foundation

 File Explosion

 Data Skipping and Liquid Clustering

 Lab: Data Skipping and Liquid Clustering 

  • Code Optimization

 Skew

 Shuffle 

 Spill

 Lab: Exploding Join 

 Serialization 

 User-Defined Functions

  • Fine-Tuning - Choosing the Right Cluster

 Fine-Tuning: Choosing the Right Cluster

 Pick the Best Instance Types

Upcoming Public Classes

Date
Time
Language
Price
Aug 11
09 AM - 01 PM (America/New_York)
English
$750.00
Aug 13
09 AM - 01 PM (Europe/London)
English
$750.00
Aug 15
01 PM - 05 PM (Asia/Singapore)
English
$750.00
Sep 09
09 AM - 01 PM (Asia/Singapore)
English
$750.00
Sep 10
01 PM - 05 PM (America/New_York)
English
$750.00
Sep 11
01 PM - 05 PM (Europe/London)
English
$750.00
Oct 08
09 AM - 01 PM (Asia/Kolkata)
English
$750.00
Oct 09
09 AM - 01 PM (America/New_York)
English
$750.00
Oct 10
09 AM - 01 PM (Europe/London)
English
$750.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Analyst

Data Analysis with Databricks

This course provides a comprehensive introduction to Databricks SQL. Learners will ingest data, write queries, produce visualizations and dashboards, and configure alerts. This course will prepare you to take the Databricks Certified Data Analyst Associate exam.

This course consists of two four-hour modules. 

SQL Analytics on Databricks

In this course, you'll learn how to effectively use Databricks for data analytics, with a specific focus on Databricks SQL. As a Databricks Data Analyst, your responsibilities will include finding relevant data, analyzing it for potential applications, and transforming it into formats that provide valuable business insights. 

You will also understand your role in managing data objects and how to manipulate them within the Databricks Data Intelligence Platform, using tools such as Notebooks, the SQL Editor, and Databricks SQL. 

Additionally, you will learn about the importance of Unity Catalog in managing data assets and the overall platform. Finally, the course will provide an overview of how Databricks facilitates performance optimization and teach you how to access Query Insights to understand the processes occurring behind the scenes when executing SQL analytics on Databricks.

AI/BI for Data Analysts

In this course, you’ll learn how to use the features Databricks provides for business intelligence needs: AI/BI Dashboards and AI/BI Genie. As a Databricks Data Analyst, you will be tasked with creating AI/BI Dashboards and AI/BI Genie Spaces within the platform, managing the access to these assets by stakeholders and necessary parties, and maintaining these assets as they are edited, refreshed, or decommissioned over the course of their lifespan. This course intends to instruct participants on how to design dashboards for business insights, share those with collaborators and stakeholders, and maintain those assets within the platform. Participants will also learn how to utilize AI/BI Genie Spaces to support self-service analytics through the creation and maintenance of these environments powered by the Databricks Data Intelligence Engine.

Languages Available: English | 日本語 | Português BR | 한국어
Paid
8h
Lab
instructor-led
Associate

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.