Scaling Apache Spark on Kubernetes at Lyft

Download Slides

Lyft is on the mission to improve people’s lives with the world’s best transportation. As part of this mission Lyft invests heavily in open source infrastructure and tooling. At Lyft Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark at Lyft has evolved to solve both Machine Learning and large scale ETL workloads. By combining the flexibility of Kubernetes with the data processing power of Apache Spark, Lyft is able to drive ETL data processing to a different level. In this talk, Li Gao and Rohit Menon will talk about challenges the Lyft team faced and solutions they developed to support Apache Spark on Kubernetes in production and at scale.

Topics Include: – Key traits of Apache Spark on Kubernetes. – Deep dive into Lyft’s multi-cluster setup and operationality to handle petabytes of production data. – How Lyft extends and enhances Apache Spark to support capabilities such as Spark pod life cycle metrics and state management, resource prioritization, and queuing and throttling. – Dynamic job scale estimation and runtime dynamic job configuration. – How Lyft powers internal Data Scientists, Business Analysts, and Data Engineers via a multi-cluster setup.


Try Databricks
See More Spark + AI Summit in San Francisco 2019 Videos

« back
About Li Gao

Li Gao is the tech lead in the cloud native data compute team at Lyft that powers many of Lyft's critical data and ML compute pipelines. Prior to Lyft, Li worked at Salesforce, Fitbit, and a few startups etc. on various technical leadership positions on cloud native and hybrid cloud data platforms at scale. Besides Spark, Li has scaled and productionized other open source projects, such as Presto, Apache HBase, Apache Phoenix, Apache Airflow, Apache Hive. Spark Summit 2019 - Scaling Spark on K8s

About Rohit Menon

Rohit Menon is a Software Engineer on the Data Platform team at Lyft. Rohit's primary area of focus is building and scaling out the Spark and Hive Infrastructure for ETL and Machine learning use cases. Previously he was one of the early engineers on the Data Platform team at Electronic Arts (EA, Inc) focussing on Data Compute and Security. Before EA, Rohit was a Software Engineer at VMware working on Application Deployment Framework on Virtualized environments.