Session

Breaking With Spark Versions: A Better Way to Manage Workload Compatibility + Dependency Management

Overview

ExperienceIn Person
TypeBreakout
TrackData Engineering and Streaming
IndustryEnterprise Technology
TechnologiesApache Spark
Skill LevelAdvanced
Duration40 min

This session explains how we’ve made Apache Spark™ versionless for end users by introducing a stable client API, environment versioning and automatic remediation. These capabilities have enabled auto-upgrade of hundreds of millions of workloads with minimal disruption.

 

We’ll also introduce a new approach to dependency management using environments. Admins will learn how to speed up package installation with Default Base Environments, and users will see how to manage custom environments for their own workloads.

Session Speakers

IMAGE COMING SOON

Justin Breese

/Staff Product Manager - serverless jedi
Databricks