Skip to main content
Page 1

PySpark in 2023: A Year in Review

With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use...

Python Dependency Management in Spark Connect

November 14, 2023 by Hyukjin Kwon and Ruifeng Zheng in
Managing the environment of an application in a distributed computing environment can be challenging. Ensuring that all nodes have the necessary environment to...