Lakehouse Playhouse: Introducing Apache Spark for Kids

 

Happy April Fools' Day! There is currently no integration or partnership with Scratch to develop any Databricks-related offering for kids.

Are your child’s coding projects getting bottlenecked by scalability issues? Maybe your 8-year-old’s after-school assignment executes just fine on their toy data set but gets stuck when scaling to gigabytes, terabytes, or even petabytes of data. Apache Spark is the de facto open source standard tool for manipulating large data sets, but to date, kids have been largely unable to leverage this powerful technology. Luckily, there’s Scratch, an open source project that gives children a safe and playful programming environment to learn, create, and collaborate on coding projects such as games and animations.

Scratch strongly aligns with our own mission to simplify and democratize data and AI. Today, we’re thrilled to announce the end to scalability issues for budding child programmers everywhere through a new Scratch API for Spark!

Scaling your Scratch workloads

Scratch has taken the classroom by storm with 42 million people creating over 113 million projects. Aimed at children 8 and older, it’s been translated into 74 languages and is used all across the globe. Despite Scratch inspiring and training the next generation of computer scientists, there are limitations. As we’ve heard from children time and time again, Scratch’s single-threaded concurrency, paltry machine learning capabilities, and lack of distributed functionality make running production Scratch workloads a veritable nightmare. What’s a kid to do?

This is where Scratch on Spark comes in. Scratch’s colorful and intuitive drag-and-drop functionality provides kids everywhere a linearly scalable solution for their most data-intensive classroom applications. Need to connect back to the toy boats sitting at the dock in the data Lakehouse? Drag the blue tile and just like that, you’re connected to a highly-optimized, scalable, and ACID-compliant Delta table. Need to handle those pesky HIPAA requirements to encrypt your Operation board game data? That’s the red tile. Need to put it into production? Click that green flag in the Scratch UI to CI/CD your latest creation into your production Scratch environment.

Creating a Spark cluster is easy in the scratch UI: just drag and drop tiles to build out your workflow. It even adjusts the cluster size as the data scales!

Below shows how the code blocks above interface with the Scratch cat to process petabytes of data using sliders and text boxes. Now that’s a good kitty! 

Easily extend Scratch to address production security concerns by making custom code blocks. Gotta keep that homework safe! 

Early success with Scratch on Spark

Families everywhere are already seeing traction with Scratch on Spark. Take Little Bobby Tables, who wants to play with his mom, but she’s stuck in her text editor getting that last JIRA ticket across the board. Bobby thought: “If only I could solve the memory issue of that stream-to-stream join. Then Mom could play with me.” 

With a few clicks and drags in the Scratch editor, Bobby whipped up a solution. He made the Scratch Cat mascot provision Spark resources, connect to a Kafka cluster, and say “I love you, Mom.” Then it was just an inner join of the events stream with the cache of user data before dumping it into a Delta table. Once it was picked up by the real-time CEO dashboard, it was mission accomplished with no tables dropped! They were playing board games before the DevOps team got the notification for the code review.

A bright future for budding computer scientists

The Scratch API for Spark is revolutionizing the way children are writing their applications. And to make sure kindergarteners aren’t left scratching their heads with scalability bottlenecks, there are some exciting announcements in the pipeline for a Spark integration with Scratch Jr., a flavor of Scratch for kids 5-7 years old. To be competitive in today’s classrooms, kids today need deep knowledge of the industry standard distributed computing engine for big data. By integrating these two valuable open source technologies, Databricks is democratizing data science and engineering beyond just the citizen scientist by opening up these powerful tools to classrooms worldwide.

---

Frequently asked questions

Is the Scratch API for Spark a real Databricks product?

No — this is an April Fools' Day joke, and there is no actual integration or partnership between Databricks and Scratch. The article humorously imagines what it would look like if children could run Apache Spark™ workloads directly from the Scratch interface. Any features described, such as drag-and-drop Delta table connections or automatic Spark cluster provisioning, are fictional and not available in either product.

What is Scratch, and who uses it?

Scratch is an open source project that gives children a safe, playful programming environment for creating games and animations through drag-and-drop coding blocks. According to the article, more than 42 million people have created over 113 million projects on the platform, and it has been translated into 74 languages for use by children age 8 and older in classrooms worldwide.

Why does the article say kids would need Apache Spark instead of just Scratch?

The article jokes that Scratch's single-threaded concurrency, limited machine learning capabilities, and lack of distributed functionality make it unsuitable for running "production" workloads at scale. Apache Spark is described as the de facto open source standard for processing large datasets, so pairing the two is presented as a way to eliminate scalability bottlenecks in kids' coding projects.

What playful features does the imagined Scratch-Spark integration include?

The article describes several fictional capabilities, like dragging a blue tile to connect to a highly optimized, ACID-compliant Delta table, a red tile to encrypt data for "HIPAA requirements," and clicking a green flag to CI/CD a project into a production Scratch environment. It also imagines the interface automatically adjusting Spark cluster size as data scales and letting kids build custom code blocks for added security.

Is there a planned integration with Scratch Jr.?

The article jokes about "exciting announcements in the pipeline" for a Spark integration with Scratch Jr., a version of Scratch aimed at children ages 5 to 7. As with the rest of the piece, this is part of the April Fools' Day satire and does not represent an actual product roadmap or partnership.