Tools for Assisted Apache Spark Version Migrations, From 2.1 to 3.2+
On Demand
Type
- Session
Format
- Hybrid
Track
- Data Engineering
Industry
- Media and Entertainment
Difficulty
- Intermediate
Room
- Moscone South | Upper Mezzanine | 155
Duration
- 35 min
Overview
This talk will look at the current state of tools to automate library and language upgrades in Python and Scala and apply them to upgrading to new version of Apache Spark. After doing a very informal survey, it seems that many users are stuck on no longer supported versions of Spark, so this talk will expand on the first attempt at automating upgrades (2.4 -> 3.0) to explore the problem all the way back to 2.1.
Session Speakers
See the best of Data+AI Summit
Watch on demand