Auto-Pilot for Apache Spark Using Machine Learning

Download Slides

At Qubole, users run Spark at scale on cloud (900+ concurrent nodes). At such scale, for efficiently running SLA critical jobs, tuning Spark configurations is essential. But it continues to be a difficult undertaking, largely driven by trial and error. In this talk, we will address the problem of auto-tuning SQL workloads on Spark. The same technique can also be adapted for non-SQL Spark workloads. In our earlier work[1], we proposed a model based on simple rules and insights. It was simple yet effective at optimizing queries and finding the right instance types to run queries.

However, with respect to auto tuning Spark configurations we saw scope of improvement. On exploration, we found previous works addressing auto-tuning using Machine learning techniques. One major drawback of the simple model[1] is that it cannot use multiple runs of query for improving recommendation, whereas the major drawback with Machine Learning techniques is that it lacks domain specific knowledge. Hence, we decided to combine both techniques. Our auto-tuner interacts with both models to arrive at good configurations.

Once user selects a query to auto tune, the next configuration is computed from models and the query is run with it. Metrics from event log of the run is fed back to models to obtain next configuration. Auto-tuner will continue exploring good configurations until it meets the fixed budget specified by the user. We found that in practice, this method gives much better configurations compared to configurations chosen even by experts on real workload and converges soon to optimal configuration.

In this talk, we will present a novel ML model technique and the way it was combined with our earlier approach. Results on real workload will be presented along with limitations and challenges in productionizing them. [1] Margoor et al,’Automatic Tuning of SQL-on-Hadoop Engines’ 2018,IEEE CLOUD


Try Databricks
See More Spark + AI Summit Europe 2019 Videos

« back
About Amogh Margoor


Amogh Margoor is currently leading the Spark efforts at Qubole. His area of interests are Compilers, Big Data Engines and SQL Optimizers. At Qubole, prior to Spark he has worked on the internals of Presto and Hive to help them scale to thousands of nodes on cloud. Specifically, he has worked on join optimizations like Dynamic Filtering and Cost based Join Reordering, Join Distribution types, SQL Optimizer work for supporting Materialized Views and OLAP Cubes, Auto-Tuning Big Data Stack, infrastructure for AIR (data intelligence and QDS internal Data warehouse), Object Store (S3) optimizations for cloud etc.

Urvashi Kohli
About Urvashi Kohli


Urvashi Kohli works as a Big Data Support Engineer at Qubole. She is a recent graduate from Carnegie Mellon University where she pursued Master in Information Systems. Prior to this she pursued her undergrad in Computer Science and worked as a Systems Engineer in India. She is passionate about leveraging Statistical Modeling and Machine Learning as a tool to solve complex optimization problems

Mayur Bhosale
About Mayur Bhosale


Mayur Bhosale is a member of Technical Staff at Qubole, where he works on Spark. He is a recent graduate of BITS Pilani, Goa where he pursued a Bachelor's in Computer Science. Previously he has worked as a summer intern with CERN-HSF developing computing infrastructure for processing 3D data sets using Spark.