Optimizing Apache Spark™ on Databricks

Description

In this course, you will explore the five key problems that represent the vast majority of performance issues in an Apache Spark application: skew, spill, shuffle, storage, and serialization. With examples based on 100 GB to 1+ TB datasets, you will investigate and diagnose sources of bottlenecks with the Spark UI and learn effective mitigation strategies. You will also discover new features introduced in Spark 3 that can automatically address common performance problems. Lastly, you learn how to design and configure clusters for optimal performance based on specific team needs and concerns.

Duration

2 full days or 4 half days

Objectives

• Articulate how the five most common performance problems in a Spark application can be mitigated to achieve better application performance

• Summarize the most common performance problems associated with data ingestion and how to mitigate them

• Articulate how new features in Spark 3.x can be employed to mitigate performance problems in your Spark applications

• Configure a Spark cluster for maximum performance given specific job requirements

Prerequisites

• Hands-on experience developing Apache Spark applications (6+ months). We recommend the Apache Spark Programming course to get started working with Spark.

• Intermediate experience in Python or Scala

Outline

Day 1

• Review of Spark architecture and Spark UI

• Skew

• Spill

• Shuffle

• Storage

• Serialization

Day 2

• Ingestion basics

• Predicate push downs

• Disk partitioning

• Z-ordering

• Bucketing

• Optimization with Adaptive Query Execution (AQE)

• Designing and configuring clusters for high performance

Upcoming Public Classes

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.