HomepageData + AI Summit 2022 Logo
Watch on demand

How To Make Apache Spark on Kubernetes Run Reliably on Spot Instances

On Demand

Type

  • Session

Format

  • Hybrid

Track

  • Data Engineering

Difficulty

  • Intermediate

Room

  • Moscone South | Upper Mezzanine | 155

Duration

  • 35 min
Download session slides

Überblick

Since the general availability of Apache Spark’s native support for running on Kubernetes with Spark 3.1 in March 2021, the Spark community is increasingly choosing to run on k8s to benefit of containerization, efficient resource-sharing, and the tools from the cloud-native ecosystem.



Data teams are faced with complexities in this transition, including how to leverage spot VMs. These instances enable up to 90% cost savings but are not guaranteed to be available and face the risk of termination. This session will cover concrete guidelines on how to make Spark run reliably on spot instances, with code examples from real-world use cases.



Main topics:

• Using spot nodes for Spark executors

• Mixing instance types & sizes to reduce risk of spot interruptions - cluster autoscaling

• Spark 3.0: Graceful Decommissioning - preserve shuffle files on executor shutdown

• Spark 3.1: PVC reuse on executor restart - disaggregate compute & shuffle storage

• What to look for in future Spark releases

Session Speakers

Jean-Yves Stephan

Senior Product Manager

Spot by NetApp

Hudson Buzby

Solutions Architect

Spot.io

Das Beste des Data+AI Summits anzeigen

Watch on demand