Skip to main content

Databricks Enables Metacog to Accelerate Time to Market With Continuous Integration

Seamless Software Testing and Deployment Automation Made Possible by Enterprise Spark Platform

March 31, 2016
Share this post

SAN FRANCISCO, CA--(Marketwired - Mar 31, 2016) - Databricks, the company behind Apache Spark, today announced that Metacog, a cloud-based analytics platform for adaptive and competency-based learning, has selected Databricks as its enterprise Spark solution. Databricks affords Metacog the ability to automate Spark operations for numerous measureable benefits.

Metacog provides analytics to education institutions, corporations, and government entities by monitoring and analyzing how individuals tackle open-ended performance tasks to assess whether learning goals have been met. The company's business model hinges on accurately scoring hundreds of different assessments per customer in an automated fashion by machine learning. Given the scoring criteria for each assessment is unique to different learning goals, Metacog needed an efficient system to build and deploy production for a large number of machine learning models.

As result of the quantity of Metacog's data and speed at which it needs to be processed and analyzed, Metacog was attracted to Apache Spark as a big data engine for its flexibility in performing ETL and developing machine learning algorithms. Yet the company suffered release delays and software defects because thoroughly testing code on Spark clusters proved to be too complex and time-consuming.

With the implementation of Databricks, Metacog created an environment to fully automate their test and release processes, allowing them to achieve a number of benefits:

  • Doubled release cadence: Metacog accelerated their time-to-market by reducing release cycles from one month to, at most, two weeks with the continuous integration system.
  • Managing infrastructure costs: Metacog used Databricks to automatically shut down unused Spark clusters (e.g., over the weekend), achieving significant infrastructure savings of approximately 28 percent.
  • Improve developer productivity: Databricks enabled the Metacog team to free up 20 percent of their time, reallocating this to product development rather than upgrading and maintaining their Spark production environment.

"Metacog's success in using our platform validates two key Databricks values; not only are we simplifying operations to achieve measurable improvements in business metrics, but we are doing so while helping them to manage infrastructure complexity and costs. This kind of customer use case is encouraging and incredibly rewarding," said Kavitha Mariappan, Vice President of Marketing at Databricks.

"Not having a way to automate our Spark production environment resulted in many software defects and long release cycles. Databricks enabled us to build a continuous integration environment easily," said Doug Stein, Chief Technology Officer at Metacog. "Now every member of the team can automatically test code with Spark clusters, resulting in faster detection and remediation of bugs and much shorter release cycles."

Download the Metacog case study here:
For more information on Databricks visit:

About Databricks:

Databricks' vision is to make big data simple for the enterprise. The company was founded by the team that created and continues to drive Apache Spark, a powerful open source data processing engine built for sophisticated analytics, ease of use, and speed. Databricks is the largest contributor to the open source Spark project providing 10x more code than any other company. The company has also trained over 20,000 users on Apache Spark, and has the largest number of customers deploying Spark to date. Databricks is venture-backed by Andreessen Horowitz and NEA. For more information, contact [email protected].

Recent Press Releases

Databricks Announces Over 70% Annualized Growth in EMEA Fueled by Enterprise Demand for AI
Read Now
Databricks Launches Data Intelligence Platform for Energy, Bringing Generative AI Capabilities to the Energy Sector
Read Now
Databricks Sees Over 70% Annual Growth in the ANZ Market as Enterprise AI Booms
Read Now
Databricks Launches DBRX, A New Standard for Efficient Open Source Models
Read Now
Databricks Strengthens Presence in Latin America, Appointing Marcos Grilanda as Vice President and General Manager
Read Now
View All



For press inquires:

[email protected]

Stay connected

Stay up to date and connect with us through our newsletter, social media channels and blog RSS feed.
Subscribe to the newsletter

Get assets

If you would like to use Databricks materials, please contact [email protected] and provide the following information:

Your name and title
Company name and location
Description of request
View brand guidelines