Skip to main content

Optimizing Apache Spark™ on Databricks

In this course, you will explore the five key problems that represent the vast majority of performance issues in an Apache Spark application: skew, spill, shuffle, storage, and serialization. With examples based on 100 GB to 1+ TB datasets, you will investigate and diagnose sources of bottlenecks with the Spark UI and learn effective mitigation strategies. You will also discover new features introduced in Spark 3 that can automatically address common performance problems. Lastly, you learn how to design and configure clusters for optimal performance based on specific team needs and concerns.

Skill Level
Professional
Duration
12h
Prerequisites
Hands-on experience developing Apache Spark applications (6+ months), Intermediate experience in Python or Scala

Outline

Day 1

  • Review of Spark architecture and Spark UI    
  • Skew    
  • Spill    
  • Shuffle    
  • Storage    
  • Serialization


Day 2

  • Ingestion basics    
  • Predicate push downs    
  • Disk partitioning    
  • Z-ordering    
  • Bucketing    
  • Optimization with Adaptive Query Execution (AQE)    
  • Designing and configuring clusters for high performance

Upcoming Public Classes

Date
Time
Language
Price
Apr 22
02 PM - 06 PM (America/New_York)
English
$1500.00
May 20
10 AM - 02 PM (Asia/Singapore)
English
$1500.00
May 20
02 PM - 06 PM (America/New_York)
English
$1500.00
May 27
09 AM - 05 PM (Europe/Paris)
English
$1500.00
Jun 24
09 AM - 05 PM (Europe/London)
English
$1500.00
Jul 15
09 AM - 05 PM (Europe/Paris)
English
$1500.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Career Workshop

Career Workshop/

March 20

Careers at Databricks

We're on a mission to help data teams solve the world's toughest problems. Will you join us?
Advance my career now

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.