HomepageData + AI Summit 2022 Logo
Watch on demand

Tools for Assisted Apache Spark Version Migrations, From 2.1 to 3.2+

On Demand

Type

  • Session

Format

  • Hybrid

Track

  • Data Engineering

Industry

  • Media and Entertainment

Difficulty

  • Intermediate

Room

  • Moscone South | Upper Mezzanine | 155

Duration

  • 35 min
Download session slides

Overview

This talk will look at the current state of tools to automate library and language upgrades in Python and Scala and apply them to upgrading to new version of Apache Spark. After doing a very informal survey, it seems that many users are stuck on no longer supported versions of Spark, so this talk will expand on the first attempt at automating upgrades (2.4 -> 3.0) to explore the problem all the way back to 2.1.

Session Speakers

Headshot of Holden Karau

Holden Karau

Engineer

Netflix

See the best of Data+AI Summit

Watch on demand