ホームData + AI Summit 2022 のロゴ
Watch on demand

Apache Spark on Kubernetes—Lessons Learned from Launching Millions of Spark Executors

On Demand

Type

  • Session

フォーマット

  • Hybrid

Track

  • データエンジニアリング

Difficulty

  • Intermediate

Room

  • Moscone South | Upper Mezzanine | 160

Duration

  • 35 min
Download session slides

概要

At Apple, data scientists and engineers are running enormous Spark workloads to deliver amazing cloud services. Apple Cloud Service supports the ever-increasing scale of Spark workloads and resource requirements with great user experience: from code to deployment management, one interface for all compute backends.



In this talk, Aaruna and Zhou would walk through the lessons we learnt and pitfalls encountered for supporting the service at Apple scale - we would share how Apple Cloud Services effectively orchestrate Spark applications, as well as the seamless switchover among different resource managers - be it in Mesos or Kubernetes, private or on-premise infrastructure. We will also cover the monitoring system and how it helps tuning Spark resource requirements with actual execution analysis.

Session Speakers

Zhou JIANG

ソフトウェアエンジニア

Apple

Aaruna Godthi

Software Engineer Manager

Apple

Data+AI サミットの様子をご覧いただけます

Watch on demand