What is a Hosted Spark?
Unified data platform with REST API access to Spark clusters for remote applications, enabling interactive and batch data exploration in multiple languages
- Hosted Spark describes cloud services that run Apache Spark for you so teams can use Spark without installing or managing their own clusters.
- The provider handles provisioning, scaling, monitoring and upgrades, while users focus on writing jobs, notebooks and SQL queries.
- Databricks offers a fully hosted Spark environment with optimizations, security and collaboration features that simplify large scale data and AI workloads.
What is hosted Spark?
Apache Spark is a fast and general cluster computing system for Big Data built around speed, ease of use, and advanced analytics that was originally built in 2009 at UC Berkeley. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. In addition, it also supports several other tools such as Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Spark Streaming for stream processing.
What are the two modes for data exploration in Spark?
- Interactive
- Batch
For a simplified end-user interaction, Spark is also provided to organizations in a unified hosted data platform. In the absence of direct access to Spark resources by remote applications, the user had to face a longer route to production. In order to overcome this obstacle, there have been created services that enable remote apps to efficiently connect to a Spark cluster over a REST API from anywhere. These interfaces support the execution of snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN. Hosted Spark interfaces proved to be turnkey solutions as they facilitate the interaction between Spark and application servers, streamlining the architecture required by interactive web and mobile apps.
The agentic AI playbook for the enterprise
What features do hosted Spark services provide?
- Interactive Scala, Python, and R coverings
- Batch submissions in Scala, Java, Python
- Multiple users are able to share the same server
- Allows users to submit jobs from anywhere through REST
- No code change is required do be done to your programs
Organizations can now easily overcome the existing bottlenecks that impede their ability to operationalize Spark, and instead, focus on capturing the value promised by big data.
Additional resources on hosted Spark
- About Apache Spark
- Azure Databricks—Apache Spark as a Service
- Learning Apache Spark 2nd Edition eBook
Frequently asked questions
What problem does hosted Spark solve for a data team?
Hosted Spark removes the need for a team to install, configure or maintain its own Apache Spark clusters, since a cloud provider runs Spark on the team's behalf. This lets engineers and analysts focus on writing jobs, notebooks and SQL queries instead of managing infrastructure. It also shortens the path to production because remote applications no longer need direct, manual access to Spark resources. For background on the underlying engine, see About Apache Spark.
What tasks does a hosted Spark provider take care of for me?
A hosted Spark provider handles provisioning, scaling, monitoring and upgrades of the Spark environment. This means the operational work of keeping clusters running and up to date shifts away from the end user. Users are then free to concentrate on the logic of their jobs and queries rather than cluster administration.
Which programming languages can I use with hosted Spark services?
Hosted Spark services support interactive work in Scala, Python and R, plus batch job submissions in Scala, Java and Python. This lets teams write and run Spark programs in whichever of these languages fits their workflow. Because Spark itself exposes high-level APIs in these languages, no code changes are required to run existing programs through a hosted Spark service.
How do remote applications connect to a hosted Spark cluster?
Remote applications connect to a hosted Spark cluster over a REST API, allowing users to submit jobs from anywhere. These interfaces run snippets of code or full programs in a Spark context, either locally or on Apache Hadoop YARN. This REST-based access streamlines the architecture needed by interactive web and mobile apps, and it also allows multiple users to share the same server.
What does Databricks add on top of a hosted Spark environment?
Databricks provides a fully hosted Spark environment with added optimizations, security and collaboration features for large-scale data and AI workloads. This goes beyond basic hosted Spark by combining cluster management with a unified platform for interactive and batch data exploration in multiple languages. The result is a single environment where teams can operationalize Spark instead of being slowed down by infrastructure bottlenecks.
Get the latest posts in your inbox
Subscribe to our blog and get the latest posts delivered to your inbox.