Advanced Migrations: From Hive to SparkSQL
- 데이터 엔지니어링
- Moscone South | Upper Mezzanine | 160
- 35 min
Learn how Pinterest moved over 6000 Hive queries to SparkSQL, achieved a 2x runtime-weighted speed up and made significant savings in compute resources. In order to do migrations at this scale. Companies often take one of two approaches, either employ hundreds of engineers to manually migrate or completely change the query engine to be compatible with Hive both of which take significant engineering time.
In this session you will learn how Pinterest took a hybrid approach and the tools and tricks Pinterest used to safely migrate thousands of queries at scale.