Introducing English as the New Programming Language for Apache SparkJune 29, 2023 by Gengliang Wang, Xiangrui Meng, Reynold Xin, Allison Wang, Amanda Liu and Denny Lee in Open Source Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™...
Announcing Delta Lake 3.0 with New Universal Format and Liquid ClusteringJune 29, 2023 by Ryan Johnson, Michael Armbrust, Reynold Xin, Denny Lee, Tathagata Das, Bart Samwel, Terry Kim, Sirui Sun, Himanshu Raja, Rahul Potharaju, Juan Yu and Susan Pierce in Engineering Blog We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in...
Project Lightspeed Update - Advancing Apache Spark Structured StreamingJune 29, 2023 by Karthik Ramasamy, Michael Armbrust, Matei Zaharia, Reynold Xin, Praveen Gattu, Ray Zhu, Shrikanth Shankar, Awez Syed, Sameer Paranjpye, Frank Munz and Matt Jones in Engineering Blog In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance...
Introducing Lakehouse Federation Capabilities in Unity CatalogJune 28, 2023 by Matei Zaharia, Andrew Li, Can Efeoglu, Cyrielle Simeone, Sachin Thakur and Daniel Tenedorio in Platform Blog Lakehouse Federation is now in public preview! Data teams face many challenges to quickly access the right data primarily due to data fragmentation...
Introducing Materialized Views and Streaming Tables for Databricks SQLJune 28, 2023 by Paul Lappas, Michael Armbrust, Yannis Papakonstantinou, Nitin Sharma and Andreas Neumann in Platform Blog We are thrilled to announce that materialized views and streaming tables are now publicly available in Databricks SQL on AWS and Azure. Streaming...