Voice from Facebook: Using Apache Spark for Large-Scale Language Model Training
This is a guest post from Facebook. Tejas Patil and Jing Zheng, software engineers in the Facebook engineering team, show how to use...
Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1
In part 1 of this series on Structured Streaming blog posts, we demonstrated how easy it is to write an end-to-end streaming ETL...
Processing a Trillion Rows Per Second on a Single Machine: How Can Nested Loop Joins be this Fast?
This blog post describes our experience debugging a failing test case caused by a cross join query running “too fast.” Because the root...
Intel’s BigDL on Databricks
Try this notebook on Databricks Intel recently released its BigDL project for distributed deep learning on Apache Spark. BigDL has native Spark integration...
Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1
Explore why lakehouses are the data architecture of the future with the father of the data warehouse, Bill Inmon. Try this notebook in...