SQM: From Fragmented Data Silos to Real-Time ML Operations with Delta Sharing

SQM set out to increase production efficiency by improving how raw materials are converted into finished product. But, it faced a classic data engineering challenge: critical operational data was trapped in silos (plant historians, lab systems, simulation tools) with no scalable way to unify it with enterprise data in Databricks. Operating decisions relied on operator experience rather than predictive models, leading to excess raw material use and inconsistent output.

Instead of building custom ETL pipelines to move data across systems, SQM leveraged Delta Sharing to create a governed, real-time data backbone. AVEVA’s cloud-based industrial intelligence platform, CONNECT, seamlessly shared data with Databricks on Azure using zero-copy and no data movement. This enabled operational data to be unified and refined into consistent minute-level views, and used to develop machine learning and optimization models.