Congratulations to Summer Hackathon Winners!
Earlier this year, Databricks launched Dolly 2.0: the world's first truly open instruction-tuned Large Language Model (LLM) . To build off this excitement...
Page 1
Introducing English as the New Programming Language for Apache Spark
Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™...
Announcing Delta Lake 3.0 with New Universal Format and Liquid Clustering
We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in...
Defining the Future of Data & AI: Announcing the Finalists for the 2022 Databricks Data Team OSS Award
The annual Databricks Data Team Awards recognize data teams who are harnessing the power of data and AI to deliver solutions for some...
Extending Delta Sharing to Google Cloud Storage
This blog article has been cross-posted from the Delta.io blog . We are excited for the release of Delta Sharing 0.4.0 for the...
Make Your Data Lakehouse Run, Faster With Delta Lake 1.1
Delta Lake 1.1 improves performance for merge operations, adds the support for generated columns and improves nested field resolution With the tremendous contributions...
The Ubiquity of Delta Standalone: Java, Scala, Hive, Presto, Trino, Power BI, and More!
The Delta Standalone library is a single-node Java library that can be used to read from and write to Delta tables. Specifically, this...
Extending Delta Sharing for Azure
We are excited for the release of Delta Sharing 0.3.0, which introduces several key improvements and bug fixes, including the following features: Delta...
The Foundation of Your Lakehouse Starts With Delta Lake
It’s been an exciting last few years with the Delta Lake project. The release of Delta Lake 1.0 as announced by Michael Armbrust...
Get Your Free Copy of Delta Lake: The Definitive Guide (Early Release)
At the Data + AI Summit, we were thrilled to announce the early release of Delta Lake: The Definitive Guide , published by...
How We Launched a Podcast: Lessons, (Minor) Mishaps & Key Takeaways
After six episodes featuring amazing leaders and practitioners in the data and AI community, we wrapped up season 1 of Data Brew by...
Attack of the Delta Clones (Against Disaster Recovery Availability Complexity)
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Notebook: Using Deep Clone...
Automatically Evolve Your Nested Column Schema, Stream From a Delta Table Version, and Check Your Constraints
We recently announced the release of Delta Lake 0.8.0 , which introduces schema evolution and performance improvements in merge and operational metrics in...
Natively Query Your Delta Lake With Scala, Java, and Python
Today, we’re happy to announce that you can natively query your Delta Lake with Scala and Java (via the Delta Standalone Reader )...
How Scribd Uses Delta Lake to Enable the World's Largest Digital Library
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Scribd uses Delta Lake...
Enabling Spark SQL DDL and DML in Delta Lake on Apache Spark 3.0
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Last week, we had...
Time Traveling with Delta Lake: A Retrospective of the Last Year
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Try out Delta Lake...
Schema Evolution in Merge Operations and Operational Metrics in Delta Lake
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Try this notebook to...
COVID-19 Datasets Now Available on Databricks: How the Data Community Can Help
Initially published April 14th, 2020; updated April 21st, 2020 With the massive disruption of the current COVID-19 pandemic, many data engineers and data...
Query Delta Lake Tables from Presto and Athena, Improved Operations Concurrency, and Merge performance
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. We are excited to...
Using AutoML Toolkit's FamilyRunner Pipeline APIs to Simplify and Automate Loan Default Predictions
Try this Loan Risk with AutoML Pipeline API Notebook in Databricks Introduction In the post Using AutoML Toolkit to Automate Loan Default Predictions...
Simple, Reliable Upserts and Deletes on Delta Lake Tables using Python APIs
We are excited to announce the release of Delta Lake 0.4.0 which introduces Python APIs for manipulating and managing data in Delta tables...
Diving Into Delta Lake: Schema Enforcement & Evolution
Try this notebook series in Databricks Data, like our experiences, is always evolving and accumulating. To keep up, our mental models of the...
Using AutoML Toolkit to Automate Loan Default Predictions
Download the following notebooks and try the AutoML Toolkit today: Evaluating Risk for Loan Approvals using XGBoost (0.90) | Using AutoML Toolkit to...
Productionizing Machine Learning with Delta Lake
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Try out this notebook...
Simplifying Streaming Stock Analysis using Delta Lake and Apache Spark: On-Demand Webinar and FAQ Now Available!
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. On June 13th, we...
Detecting Financial Fraud at Scale with Decision Trees and MLflow on Databricks
Try this notebook in Databricks Detecting fraudulent patterns at scale using artificial intelligence is a challenge, no matter the use case. The massive...
Applying your Convolutional Neural Network: On-Demand Webinar and FAQ Now Available!
Try this notebook in Databricks On October 25th, we hosted a live webinar— Applying your Convolutional Neural Network —with Denny Lee, Technical Product...
Simplifying Change Data Capture with Databricks Delta
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake Note: We also recommend...
Training your Neural Network: On-Demand Webinar and FAQ Now Available!
Try this notebook in Databricks On October 9th, we hosted a live webinar— Training your Neural Network —on Data Science Central with Denny...
MLflow v0.7.0 Features New R API by RStudio
Today, we’re excited to announce MLflow v0.7.0 , released with new features, including a new MLflow R client API contributed by RStudio...
Introduction to Neural Networks: On-Demand Webinar and FAQ Now Available!
Try this notebook in Databricks On September 27th, we hosted a live webinar— Introduction to Neural Networks —with Denny Lee, Technical Product Marketing...
Simplify Market Basket Analysis using FP-growth on Databricks
When providing recommendations to shoppers on what to purchase, you are often looking for items that are frequently purchased together (e.g. peanut butter...
Identify Suspicious Behavior in Video with Databricks Runtime for Machine Learning
With the exponential growth of cameras and visual recordings, it is becoming increasingly important to operationalize and automate the process of video identification...
MLflow On-Demand Webinar and FAQ Now Available!
On August 30th, our team hosted a live webinar— Introducing MLflow: Infrastructure for a complete Machine Learning lifecycle —with Matei Zaharia, Co-Founder and...
Building a Real-Time Attribution Pipeline with Databricks Delta
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. In digital advertising, one...
Loan Risk Analysis with XGBoost and Databricks Runtime for Machine Learning
Try this notebook series in Databricks For companies that make money off of interest on loans held by their customer, it’s always about...
MLflow 0.4.2 Released
Today, we’re excited to announce MLflow v0.4.0, MLflow v0.4.1, and v0.4.2 which we released within the last week with some of the recently...
Simplify Advertising Analytics Click Prediction with Databricks Unified Analytics Platform
Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...
Simplify Streaming Stock Data Analysis Using Databricks Delta
Traditionally, real-time analysis of stock data was a complicated endeavor due to the complexities of maintaining a streaming system and ensuring transactional consistency...
Make Your Oil and Gas Assets Smarter by Implementing Predictive Maintenance with Databricks
How to build an end-to-end predictive data pipeline with Databricks Delta and Spark Streaming Maintaining assets such as compressors is an extremely complex...
Analyze Games from European Soccer Leagues with Apache Spark and Databricks
Try this notebook series in Databricks Introduction The global sports market is huge, comprised of players, teams, leagues, fan clubs, sponsors, etc., and...
Build a Mobile Gaming Events Data Pipeline with Databricks Delta
How to build an end-to-end data pipeline with Structured Streaming Try this notebook in Databricks The world of mobile gaming is fast paced...
Announcing RStudio and Databricks Integration
At Databricks, we are thrilled to announce the integration of RStudio with the Databricks Unified Analytics Platform. You can try it out now...
Introducing Getting Started with Apache Spark on Databricks
We are proud to introduce the Getting Started with Apache Spark on Databricks Guide . This step-by-step guide illustrates how to leverage the...
Apache Spark Key Terms, Explained
This article was originally posted on KDnuggets The Spark Summit Europe call for presentations is open, submit your idea today As observed in...
Another Record-Setting Spark Summit
The lure of San Francisco is indisputable as is its position as the preeminent high-tech hub. On day one of Spark Summit 2016...
On-Time Flight Performance with GraphFrames for Apache Spark
Introduction Graph structures are a more intuitive approach to many classes of data problems. Whether traversing social networks, restaurant recommendations, or flight paths...
Findify’s Smart Search Gets Smarter with Apache Spark MLlib and Databricks
Spark Summit East is just around the corner! If you haven’t registered yet, you can get tickets here with this promo code for...
An Illustrated Guide to Advertising Analytics
To learn the latest developments in Apache Spark, register today to join the Spark community at Spark Summit in New York City! This...
Spark Summit East 2016 Agenda is now available
This February, join the Apache Spark community in New York City at the New York Midtown Hilton for the second annual Spark Summit...
Databricks 2015 Year In Review: Democratizing Access to Data
To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016 . 2015 has been a phenomenal year...
Databricks launches Meetup-in-a-box for Apache Spark Meetup Organizers
One of the most important reasons for the growth of Apache Spark is the amazing grassroots community interest and support to share, teach...
Spark Survey 2015 Results are now available
We ran the Spark Survey 2015 this summer to gain insights on how organizations are using Apache Spark. The results of this year’s...
Spark Summit Europe Full Agenda Available Online
This October, join the Apache Spark community in Amsterdam at the Beurs Van Berlage for the very first Spark Summit in Europe! We...
Announcing SparkHub: A Community Site for Apache Spark
Today, we are happy to announce SparkHub , a service for the Apache Spark community to easily find the most relevant Spark resources...
Databricks and IBM Collaborate to Enhance Apache Spark Machine Learning
At today’s Spark Summit , Databricks and IBM announced a joint effort to contribute key machine learning capabilities to the Apache Spark Project...
Simplify Machine Learning on Apache Spark with Databricks
As many data scientists and engineers can attest, the majority of the time is spent not on the models themselves but on the...