Apache Spark™

Spark is used for large-scale ETL and data engineering, real-time stream processing, SQL analytics, machine learning, and preparing data for AI and LLM applications. It reads and writes open table formats such as Delta Lake and Apache Iceberg, so it fits into existing data architectures without lock-in.

Yes. Apache Spark is 100% open source under Apache License 2.0, hosted at the vendor-independent Apache Software Foundation. You can download and run it for free, including on a single laptop in local mode.

Both process streaming data. Spark is a unified engine: the same APIs support batch, streaming, SQL, and machine learning workloads, so teams don’t need to maintain separate systems or skill sets for real-time and batch work. Spark Structured Streaming reuses existing batch code and the broader Spark ecosystem (SQL, MLlib, open table formats), and Real-Time Mode (introduced in Spark 4.1) adds a lower-latency streaming trigger. See Apache Spark versus Apache Flink for a side-by-side comparison of streaming use cases and Breaking the microbatch barrier for how Real-Time Mode closes the latency gap.

Yes. Spark Structured Streaming provides end-to-end exactly-once, fault-tolerant stream processing, and Real-Time Mode (a streaming trigger introduced in Spark 4.1) targets lower-latency processing for real-time dashboards, fraud detection, and real-time AI inference. The streaming API matches the batch API, so one codebase covers both. See the Real-Time Mode documentation.

Spark supports Python (PySpark), SQL, Scala, Java, and R. The DataFrame and SQL APIs behave consistently across languages, and pandas users can use the pandas API on Spark.

Yes. Spark runs in local mode on a single machine with no cluster required, which is ideal for development, testing, and learning. The same code can run unchanged on a cluster of thousands of nodes.

Spark Connect is a lightweight client-server architecture that decouples an application from the Spark cluster. Applications talk to Spark through a thin remote client (SparkSession.builder.remote("sc://..."), default port 15002), so Spark can be embedded in any app, notebook, or IDE, the server can be upgraded independently of clients, and workloads can run from anywhere. See the Spark Connect overview.

Spark Declarative Pipelines let you declare the tables and transformations you want, and Spark manages dependencies, orchestration, and incremental processing. In Python, you declare a table with the @dp.table decorator (from pyspark.pipelines) and describe the result instead of hand-coding the execution flow. See the Spark Declarative Pipelines programming guide.

Yes. Spark reads and writes Parquet, ORC, JSON, CSV, and Avro out of the box and connects to Kafka, and it works with the open table formats Delta Lake and Apache Iceberg through their Spark connectors. It is not tied to a single storage format, so it fits open lakehouse architectures. See Spark SQL data sources.

Yes. Spark includes MLlib for scalable machine learning, and it is widely used to prepare features and large-scale datasets for ML, AI, and LLM workloads. Combined with streaming and Real-Time Mode, Spark can also power real-time feature pipelines and inference.

You can run Apache Spark locally on a laptop, on your own cluster, or as a managed service on every major cloud and data platform, including Amazon Web Services (EMR), Google Cloud (Managed Service for Apache Spark), Microsoft Fabric, Cloudera, and Databricks. Because Spark is open source, the same application runs across all of them.

Thousands of organizations run Apache Spark in production, from startups to the largest internet and AI companies. Named users include Netflix, OpenAI, Yahoo, eBay, Amazon, and IBM, among others. See the Apache Spark Powered By Spark page for the full list.

Apache Spark is the open source engine. Databricks, founded by Spark’s original creators, offers a fully managed platform built on Spark with added performance (Photon), unified governance, collaborative notebooks, and enterprise security. Code written for open source Spark runs on Databricks. See Spark on Databricks for the managed platform.

Apache Spark began as a research project at UC Berkeley in 2009. The team that started it founded Databricks in 2013 and continues to contribute heavily to the open source project.