Introducing the Open Variant Data Type in Delta Lake and Apache Spark
We are excited to announce a new data type called variant for semi-structured data. Variant provides an order of magnitude performance improvements compared...
Databricks Assistant Tips & Tricks for Data Engineers
The generative AI revolution is transforming the way that teams work, and Databricks Assistant leverages the best of these advancements. It allows you...
Announcing the State Reader API: The New "Statestore" Data Source
Databricks Runtime 14.3 includes a new capability that allows users to access and analyze Structured Streaming 's internal state data: the State Reader...
PySpark in 2023: A Year in Review
With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use...
Simplify PySpark testing with DataFrame equality functions
The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set...