Establishing the foundations for better business decisions
Fonterra uses Databricks to democratize data insights for decision-making
Reduction in data load times
Boost in engineering productivity
Fonterra is one of the largest dairy cooperatives in New Zealand, and they collaborate with EY to gain the data, insights and analytics necessary to run operations efficiently. Fonterra had several legacy analytics platforms that drove complexity and duplication, which hindered timely, smart decision-making. Having hit the inherent limits, Fonterra migrated to the Databricks Data Intelligence Platform, where they democratized data securely and enhanced speed considerably. Now, Fonterra enables their business users by unlocking new opportunities for scalable AI use cases, from optimizing supply chains and inventory management to helping farmers with on-farm compliance data.
Reaching the limits of legacy systems
For more than 150 years, Fonterra has brought the goodness of natural dairy to people worldwide, but relying on legacy systems was creating problems for both the business users and the business itself.
While Fonterra used some Databricks services, they were not taking advantage of the Databricks Data Intelligence Platform as an end-to-end solution. Individual BUs did not have tailored control over their datasets, nor the ability to operate in a fully self-service environment. Additionally, there was limited understanding of data lineage across the enterprise platform. Business users utilized Power BI for reporting and dashboards, but they needed SQL for functionality. This required Fonterra data teams to move the data from the Databricks layer to another layer, meaning additional engineering effort and time to enable reliable and accurate reporting.
Satya Thadiparthi, Data Platform Owner at Fonterra, explained, “Reporting around big datasets was time-consuming, which meant the business couldn’t make timely decisions and consume the data faster.” Data sharing was equally challenging and difficult without built-in data governance controls. Manual intervention was required to move data to third parties, adding cost and resourcing pressures, and creating a barrier to supporting and advancing Fonterra’s data maturity.
For Fonterra to deliver on their promises, they needed to centralize their scattered data infrastructure to remove the inefficiencies that were slowing delivery.
Simplifying data complexity with Databricks
Fonterra aims to democratize data for business users through self-service technologies, enabling fast, accurate decision-making. With the Databricks Data Intelligence Platform in place, the team first introduced Unity Catalog for unified, open data governance. This gave BU data practitioners access to data lineage, offering a clear view of where data originates and flows, as well as the capability to manage access to their datasets.
This data lineage was then made accessible through Genie, a conversational experience for business teams to engage with their data through natural language, and Databricks Assistant, which has boosted engineering productivity by automating code checks and verifications. Together, these tools offer much-needed support to both nontechnical users and experienced developers at Fonterra and EY looking to be more productive with the data at their disposal.
“We were noticing business value from the introduction of Unity Catalog, so the move to a simplified architecture within Databricks felt like the logical next move,” Satya said. Fonterra proceeded to migrate their data to Delta Lake on the Databricks Platform. This centralized environment allows business users to access and connect datasets efficiently using preconfigured data products that scale. Data practitioners can now also run and train models on large datasets, supporting AI and ML while meeting traditional business needs.
Fonterra also implemented Delta Sharing, leveraging Unity Catalog’s security to share data with third parties. This has provided a full end-to-end view of the delivery supply chain, including the third-party logistics partners.
Doubling time to value with speed, security and access
Since migrating to the Databricks Data Intelligence Platform three months ago, Fonterra has boosted engineer productivity by up to 20%, reduced data load times by more than 50% and reduced complex report response time from minutes to seconds. The engineering pipeline development time for new data products has decreased significantly, enabling the creation of twice the number of data products per week with the same number of resources. This streamlined approach has decreased time to value for business teams across the cooperative. From an IT perspective, the simplified landscape means better support and fewer potential breakpoints.
As an EY client, Fonterra also benefits from EY’s consolidation on Databricks, which allows for better support of Fonterra’s data-driven innovation. With Unity Catalog and Delta Sharing, real-time data from Fonterra’s third-party partners can be accessed for analytics. This also enables Fonterra to collect and process their farmers’ compliance data in an efficient and streamlined manner.
Fonterra and EY plan to continue utilizing the Databricks Data Intelligence Platform to accelerate innovations with GenAI. One project is to better support members of the cooperative by building a GenAI chatbot on top of their unstructured data to help people find the answers they need more easily. The next stage is to unify with Genie to provide the same level of chatbot capability across both unstructured and structured data. “Everything we’re doing with Databricks is aligned with helping Fonterra get the most out of their data,” Vincent Mutuswa, Director of AI and Data at EY, said. “This partnership enables us to deliver optimal solutions for Fonterra and positions us to seamlessly implement future AI use cases for them.”
