Skip to main content

Databricks Announces DataFrames for Spark

Company Unveils Contributions to Spark Roadmap, Delivers Spark Education Sessions and Certification at Strata and Hadoop World

February 17, 2015
Share this post

SAN FRANCISCO, CA--(Marketwired - Feb 17, 2015) - Databricks, the company founded by the creators of the popular open-source Big Data processing engine Apache Spark with its flagship product, Databricks Cloud, today announced they have developed a new API for Spark 1.3 called DataFrames, aimed to simplify distributed data processing and enable data scientists who are familiar with single-machine tools to now immediately use Spark. DataFrames will support a variety of data sources and automatically optimize computations to make big data processing easier, accelerating the rate at which organizations can use big data.

The DataFrames API, inspired by data frames in R and Python (Pandas), provides a familiar and efficient interface for data scientists that they can now run on big data, making large-scale computing accessible to users with experience in these languages. The API builds on the Spark SQL query optimizer to automatically execute code efficiently on a cluster of machines. Finally, through Spark SQL's external data sources API, DataFrames can access a wide array of third-party data sources such as databases and NoSQL stores. DataFrames will be incorporated into Spark 1.3 and released in early March and is expected to play a major role in the R API for Spark coming later this year.

"There has always been a major gap in the skill sets required to work on 'small data' and 'big data'. By adding DataFrames to Spark, we are making the power of the Spark ecosystem available to the wide set of data scientists and analysts. Along with our other ongoing work in Spark and Databricks Cloud, we expect this feature to make big data accessible to a much wider range of users," said Matei Zaharia, CTO of Databricks and the creator of Apache Spark.

Matei Zaharia Keynote at Strata and Hadoop World
Databricks is also announcing today that its chief technology officer and co-founder, Matei Zaharia will keynote at Strata and Hadoop World. Zaharia's keynote, "New Directions for Spark in 2015," will address the significant momentum Spark is gathering in 2015, specifically how the platform is positioned for enterprise utilization, performance, and scalability with its recent improvements and additions in 2014. The session will highlight key framework enhancements related to Spark in 2015 and what these enhancements mean for pushing the envelope in the cloud analytics space.

What: "New Directions for Spark in 2015"
Who: Matei Zaharia, chief technology officer and co-founder at Databricks
When: Friday, February 20 at 9:15am PT
Where: Strata and Hadoop World (San Jose Convention Center, Grand Ballroom 220)

Zaharia will be joined at the conference by several Databricks colleagues who will also be presenting, including Paco Nathan, Patrick Wendell, Reynold Xin and more, to address Spark adoption and growth slated for 2015.

Spark at Strata and Hadoop World
For the first time ever, Strata and Hadoop World is including a Spark-focused track at the conference, "Spark in Action," where attendees can get deep dives of best practices, architectural considerations, and real-world case studies drawn from startups to large enterprises. Databricks will be training more than 300 attendees via interactive demos and tutorials, delivering an advanced training program for more than 40 people, in addition to running a Spark certification exam in partnership with O'Reilly. This certification exam is the industry's first and most comprehensive Spark certification.

Additional Resources:

About Databricks:

Databricks was founded by the team that created and continues to drive Apache Spark, the most active open source project in the Big Data ecosystem. Databricks' vision is to dramatically simplify big data processing and free users to focus on turning data into value. Databricks Cloud, a cloud platform built around Apache Spark, delivers on this vision by combining the power of Spark with a zero-management hosted platform and an initial set of applications built around common workflows. Databricks is venture-backed by Andreessen Horowitz and NEA. For more information, visit

Recent Press Releases

Databricks Announces Over 70% Annualized Growth in EMEA Fueled by Enterprise Demand for AI
Read Now
Databricks Launches Data Intelligence Platform for Energy, Bringing Generative AI Capabilities to the Energy Sector
Read Now
Databricks Sees Over 70% Annual Growth in the ANZ Market as Enterprise AI Booms
Read Now
Databricks Launches DBRX, A New Standard for Efficient Open Source Models
Read Now
Databricks Strengthens Presence in Latin America, Appointing Marcos Grilanda as Vice President and General Manager
Read Now
View All



For press inquires:

[email protected]

Stay connected

Stay up to date and connect with us through our newsletter, social media channels and blog RSS feed.
Subscribe to the newsletter

Get assets

If you would like to use Databricks materials, please contact [email protected] and provide the following information:

Your name and title
Company name and location
Description of request
View brand guidelines