Feature Deep Dive: Watermarking in Apache Spark Structured Streaming
Key Takeaways Watermarks help Spark understand the processing progress based on event time, when to produce windowed aggregates and when to trim the...
Low-latency Streaming Data Pipelines with Delta Live Tables and Apache Kafka
Delta Live Tables (DLT) is the first ETL framework that uses a simple declarative approach for creating reliable data pipelines and fully manages...
Using Spark Structured Streaming to Scale Your Analytics
This is a guest post from the M Science Data Science & Engineering Team. Modern data doesn't stop growing "Engineers are taught by...
Project Lightspeed: Faster and Simpler Stream Processing With Apache Spark
Streaming data is a critical area of computing today. It is the basis for making quick decisions on the enormous amounts of incoming...
How to Monitor Streaming Queries in PySpark
Streaming is one of the most important data processing techniques for ingestion and analysis. It provides users and developers with low latency and...
How I Built A Streaming Analytics App With SQL and Delta Live Tables
Planning my journey I'd like to take you through the journey of how I used Databricks' recently launched Delta Live Tables product to...
Build Scalable Real-time Applications on the Lakehouse Using Confluent & Databricks, Part 2
This is a collaborative post between Confluent and Databricks. We thank Paul Earsy Staff Solutions Engineer at Confluent, for their contributions. In this...
Streaming Windows Event Logs into the Cybersecurity Lakehouse
Streaming windows events into the Cybersecurity Lakehouse Enterprise customers often ask, what is the easiest and simplest way to send Windows endpoint logs...
Speed Up Streaming Queries With Asynchronous State Checkpointing
Background / Motivation Stateful streaming is becoming more prevalent as stakeholders make increasingly sophisticated demands on greater volumes of data. The tradeoff, however...
Databricks Ventures Invests in Arcion to Enable Real-Time Data Sync with the Lakehouse
Databricks customers, regardless of size and industry, are increasingly seeking to unify their data onto a single platform. To do this, they need...