Skip to main content
Page 1
Engineering blog

Project Lightspeed Update - Advancing Apache Spark Structured Streaming

June 29, 2023 by Karthik Ramasamy, Michael Armbrust, Matei Zaharia, Reynold Xin, Praveen Gattu, Ray Zhu, Shrikanth Shankar, Awez Syed, Sameer Paranjpye, Frank Munz and Matt Jones in Engineering Blog
In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance...
Engineering blog

Latency goes subsecond in Apache Spark Structured Streaming

May 15, 2023 by Jerry Peng, Pranav Anand, Sourav Gulati, Karthik Ramasamy, Michael Armbrust and Matei Zaharia in Engineering Blog
Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the...
Company blog

Databricks at Current 2022

September 28, 2022 by Matt Jones, Frank Munz, Emma Liu, Karthik Ramasamy and Riley Maris in Company Blog
Current 2022 , organized by Confluent, is the first-ever data streaming industry event – and it's coming up soon! No matter where you...
Engineering blog

Project Lightspeed: Faster and Simpler Stream Processing With Apache Spark

June 28, 2022 by Karthik Ramasamy, Matei Zaharia, Reynold Xin, Michael Armbrust, Awez Syed, Ray Zhu, Alexander Balikov, Jerry Peng, Shrikanth Shankar and Sameer Paranjpye in Engineering Blog
Streaming data is a critical area of computing today. It is the basis for making quick decisions on the enormous amounts of incoming...
Engineering blog

How to Monitor Streaming Queries in PySpark

May 27, 2022 by Hyukjin Kwon, Karthik Ramasamy and Alexander Balikov in Open Source
Streaming is one of the most important data processing techniques for ingestion and analysis. It provides users and developers with low latency and...