Introducing Apache Spark™ 3.2October 19, 2021 by Gengliang Wang, Wenchen Fan, Hyukjin Kwon, Xiao Li and Reynold Xin in Engineering Blog We are excited to announce the availability of Apache Spark™ 3.2 on Databricks as part of Databricks Runtime 10.0 . We want to...
MLflow for Bayesian Experiment TrackingOctober 18, 2021 by Srijith Rajamohan, Ph.D. in Engineering Blog This post is the third in a series on Bayesian inference ( [1] , [2] ). Here we will illustrate how to use...
Pandas API on Upcoming Apache Spark™ 3.2October 4, 2021 by Hyukjin Kwon and Xinrong Meng in Engineering Blog We're thrilled to announce that the pandas API will be part of the upcoming Apache Spark™ 3.2 release. pandas is a powerful, flexible...
Catalog and Discover Your Databricks Notebooks FasterSeptember 22, 2021 by Darin McBeath and Vuong Nguyen in Engineering Blog This is a collaborative post from Databricks and Elsevier. We thank Darin McBeath, Director Disruptive Technologies -- Elsevier, for his contributions. As a...
Managing Model Ensembles With MLflowSeptember 21, 2021 by Anindita Mahapatra, Rafi Kurlansik and Sri Tikkireddy in Engineering Blog In machine learning, an ensemble is a collection of diverse models that provide more predictive power together than any single model would on...