The connected power of unifying data and leveraging AI
In order to quickly provide customers with accurate insights into their photovoltaic plants, Solytic needed to address multiple challenges quickly. First, they required a way to clean, enrich and aggregate incoming data streams to enable rapid querying and analytics. With more than 10,000 events coming in from IoT panels per second, along with a large spread of granularity, varying data quality and delayed data, the company needed a unified solution that could resolve all these issues on one platform.
The Databricks Data Intelligence Platform on Azure did just that — quickly and efficiently. While running one big query on their old database would have completely overwhelmed the system, Delta Lake has allowed Solytic to unify all their data, scale without disruption, and build reliable and performant ETL pipelines to feed their analytics needs. Delta Lake is an open format storage layer that delivers reliability, security and performance to a data lake — for both streaming and batch operations.
A second goal at Solytic was to find a way for the data engineers and analytics teams to work together more efficiently. With Databricks fully managed cloud infrastructure, each team is now able to spin up and run their own clusters without tripping over each other, boosting their ability to collaborate. Even better? It took barely any time at all to implement.
“When we began migrating to the Databricks Data Intelligence Platform, we were concerned with having to create a working prototype in less than three months with new technology and an inexperienced team,” said Helge Brügner, Data Engineer at Solytic. “But Databricks stepped up and helped us to fully understand the architecture and how to avoid common pitfalls, and we succeeded without issue.”
Working with Databricks notebooks makes the environment even more collaborative. Whether team members know coding languages or not, analytics team members are able to go into the data with their SQL knowledge and execute their queries for quick insights. And in the collaborative notebooks, one team member can easily draw a plot, compare them to others, and quickly identify the pros and cons of various approaches.