What are Continuous Applications?

Continuous applications are an end-to-end application that reacts to data in real-time. In particular, developers would like to use a single programming interface to support the facets of continuous applications that are currently handled in separate systems, such as query serving or interaction with batch jobs. Below is an example of continuous applications can handle the following use cases.

• Updating data that will be served in real time. The developer would write a single Spark application that handles both updates and serving (e.g. through Spark’s JDBC server), or would use an API that automatically performs transactional updates on a serving system like MySQL, Redis or Apache Cassandra.
• Extract, transform and load (ETL). The developer would simply list the transformations required as in a batch job, and the streaming system would handle coordination with both storage systems to ensure exactly-once processing.
• Creating a real-time version of an existing batch job. The streaming system would guarantee results are always consistent with a batch job on the same data.
• Online machine learning. The machine learning library would be designed to combine real-time training, periodic batch training, and prediction serving behind the same API.

Additional resources on continuous applications

• Writing Continuous Applications with Structured Streaming PySpark API
• Headaches and Breakthroughs in Building Continuous Applications
• Continuous Applications at Scale of 100 Teams with Databricks Delta and Structured Streaming

Frequently asked questions

What is a continuous application?

A continuous application is an end-to-end software system that reacts to data in real time rather than processing it in scheduled batches. It continuously processes incoming events to update metrics, trigger actions or feed downstream services as new information arrives, which supports use cases like monitoring, personalization and fraud detection where timely responses matter.

How do continuous applications update data that gets served in real time?

A continuous application can update and serve data within a single Apache Spark™ application, for example through Spark's JDBC server, so developers don't need to stitch together separate update and serving systems. Alternatively, developers can use an API that automatically performs transactional updates on a serving system such as MySQL, Redis or Apache Cassandra, keeping the served data consistent as new events arrive.

How do continuous applications approach ETL (extract, transform and load) tasks?

For ETL, a developer simply lists the required transformations just as they would in a batch job, and the streaming system handles coordination with the underlying storage systems on its own. This coordination ensures exactly-once processing, so the same transformation logic can run continuously without the developer having to manage duplicate or missed records.

How can a continuous application create a real-time version of an existing batch job?

A continuous application can turn an existing batch job into a real-time one because the streaming system guarantees that its results stay consistent with what the equivalent batch job would produce on the same data. This means developers can move a job from batch to streaming without rewriting the underlying logic or worrying about the two producing different answers.

How do continuous applications support online machine learning?

For online machine learning, the machine learning library is designed to combine real-time training, periodic batch training and prediction serving behind the same API. This lets a single interface handle model updates as new data streams in alongside scheduled batch retraining and the serving of predictions, rather than requiring separate systems for each stage.