Kazuaki Ishizaki

This talk explains how to enable a vectorized engine in Apache Spark to accelerate Apache Spark programs. Vectorization is an exciting approach to maximize performance as Delta Lake and other commercial database use. On the other hand, the current Apache Spark does not use the vectorization technique yet because it is not easy to use vector instructions in the current Java language.

First, this talk reviews Vector API for ease of use of the vector instructions in Java 16. Then, this talk discusses three possible approaches to vectorize Apache Spark Engine by using Vector API: 1) replace external libraries such as BLAS library, 2) use a vectorized runtime such as a sort routine, and 3) generate vectorized Java code by Catalyst from a given SQL query. Finally, this talk shares analysis and performance results by these approaches.

Here are takeaways of this talk:

1. Overview of Vector API to vectorize Java programs
2. Multiple approaches to use a vectorized engine in Apache Spark
3. Analysis and performance results by these vectorization approaches

[daisna21-sessions-od]

This talk explains how Spark 3.0 can improve the performance of SQL applications. Spark 3.0 provides many performance features such as dynamic partitioning and enhanced pushdown. Each of them can improve the performance of a different type of SQL application. Since the number of features is large, it is not easy for application developers to understand these features at a glance. This talk gives a brief explanation of these features with an example program and explains how it works and how we can improve the performance.

Here are takeaways of this talk:

1. What optimization features for SQL Spark 3.0 support.
2. Which programs Spark 3.0 can accelerate.