Make Your Data Lakehouse Run, Faster With Delta Lake 1.1
Delta Lake 1.1 improves performance for merge operations, adds the support for generated columns and improves nested field resolution With the tremendous contributions...
The Ubiquity of Delta Standalone: Java, Scala, Hive, Presto, Trino, Power BI, and More!
The Delta Standalone library is a single-node Java library that can be used to read from and write to Delta tables. Specifically, this...
Creating a Faster TAR Extractor
Tarballs are used industry-wide for packaging and distributing files, and this is no different at Databricks. Every day we launch millions of VMs...
Extending Delta Sharing for Azure
We are excited for the release of Delta Sharing 0.3.0, which introduces several key improvements and bug fixes, including the following features: Delta...
Log4j2 Vulnerability (CVE-2021-44228) Research and Assessment
This blog relates to an ongoing investigation. We will update it with any significant updates, including detection rules to help people investigate potential...
Scala at Scale at Databricks
With hundreds of developers and millions of lines of code, Databricks is one of the largest Scala shops around. This post will be...
The Foundation of Your Lakehouse Starts With Delta Lake
It’s been an exciting last few years with the Delta Lake project. The release of Delta Lake 1.0 as announced by Michael Armbrust...
Turning 2 Trillion Data Points of Traffic Intelligence into Critical Business Insights
This is a guest authored post by Stephanie Mak , Senior Data Engineer, formerly at Intelematics. This blog post offers my experience of...
Introducing Apache Spark™ 3.2
We are excited to announce the availability of Apache Spark™ 3.2 on Databricks as part of Databricks Runtime 10.0 . We want to...
Native Support of Session Window in Spark Structured Streaming
Apache Spark™ Structured Streaming allowed users to do aggregations on windows over event-time . Before Apache Spark 3.2™, Spark supported tumbling windows and...