Deep Dive: Apache Spark Memory Management

Download Slides

Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.

Learn more:

  • Learn about Apache Spark’s Memory Model and Spark’s State in the Cloud
  • Understanding Memory Management In Spark For Fun And Profit

    « back