As a core component of data processing platform, scheduler is responsible for schedule tasks on compute units. Built on a Directed Acyclic Graph (DAG) compute model, Spark Scheduler works together with Block Manager and Cluster Backend to efficiently utilize cluster resources for high performance of various workloads. This talk dives into the technical details of the full lifecycle of a typical Spark workload to be scheduled and executed, and also discusses how to tune Spark scheduler for better performance.
Xingbo Jiang is a software engineer at Databricks, where he investigates the use cases on Spark Core and Spark SQL. Xingbo is an active contributor to Apache Spark. His areas of interest include distributed system, database, and data warehouse.