Adaptive Query Execution in Structured Streaming
In Databricks Runtime, Adaptive Query Execution (AQE) is a performance feature that continuously re-optimizes batch queries using runtime statistics during query execution. Starting...
Page 1
>
Latency goes subsecond in Apache Spark Structured Streaming
Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the...
How Collective Health uses Delta Live Tables and Structured Streaming for Data Integration
Collective Health is not an insurance company. We're a technology company that's fundamentally making health insurance work better for everyone— starting with the...
Scalable Spark Structured Streaming for REST API Destinations
Spark Structured Streaming is the widely-used open source engine at the foundation of data streaming on the Databricks Lakehouse Platform . It can...
Build Reliable and Cost Effective Streaming Data Pipelines With Delta Live Tables’ Enhanced Autoscaling
This year we announced the general availability of Delta Live Tables (DLT) , the first ETL framework to use a simple, declarative approach...
Python Arbitrary Stateful Processing in Structured Streaming
More and more customers are using Databricks for their real-time analytics and machine learning workloads to meet the ever increasing demand of their...
State Rebalancing in Structured Streaming
In light of the accelerated growth and adoption of Apache Spark Structured Streaming, Databricks announced Project Lightspeed at Data + AI Summit 2022...
Using Streaming Delta Live Tables and AWS DMS for Change Data Capture From MySQL
In this article we will walk you through the steps to create an end-to-end CDC pipeline with Terraform using Delta Live Tables, AWS...
Databricks at Current 2022
Current 2022 , organized by Confluent, is the first-ever data streaming industry event – and it's coming up soon! No matter where you...
Simplifying Streaming Data Ingestion into Delta Lake
Most business decisions are time sensitive and require harnessing data in real time from different types of sources. Sourcing the right data at...