Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters

Download Slides

Adding nodes at runtime (Upscale) to already running Spark-on-Yarn clusters is fairly easy. But taking away these nodes (Downscale) when the workload is low at some later point of time is a difficult problem. To remove a node from a running cluster, we need to make sure that it is not used for compute as well as storage.

But on production workloads, we see that many of the nodes can’t be taken away because:

  1. Nodes are running some containers although they are not fully utilized i.e., containers are fragmented on different nodes. Example. – each node is running 1-2 containers/executors although they have resources to run 4 containers.
  2. Nodes have some shuffle data in the local disk which will be consumed by Spark application running on this cluster later. In this case, the Resource Manager will never decide to reclaim these nodes because losing shuffle data could lead to costly recomputation of stages.

In this talk, we will talk about how we can improve downscaling in Spark-on-YARN clusters under the presence of such constraints. We will cover changes in scheduling strategy for container allocation in YARN and Spark task scheduler which together helps us achieve better packing of containers. This makes sure that containers are defragmented on fewer set of nodes and thus some nodes don’t have any compute. In addition to this, we will also cover enhancements to Spark driver and External Shuffle Service (ESS) which helps us to proactively delete shuffle data which we already know has been consumed. This makes sure that nodes are not holding any unnecessary shuffle data – thus freeing them from storage and hence available for reclamation for faster downscaling.


Try Databricks
See More Spark + AI Summit Europe 2019 Videos

« back
Prakhar Jain
About Prakhar Jain


Prakhar Jain is a member of the technical staff at Qubole, where he works on Spark. Prakhar holds a bachelor of computer science engineering from the Indian Institute of Technology, Bombay, India.

About Venkata Krishnan Sowrirajan


Venkata is a Senior Software Engineer at LinkedIn working on a big data platform based on Apache Spark. Previously worked at Qubole building big data platforms primarily on public service clouds using Apache Spark. Before starting to work in the industry, graduated from Arizona State University with an MS in Computer Science.