Optimizing the stability of Europe’s energy grid
Sympower leverages Databricks to help stabilize Europe’s energy grid
Sympower empowers commercial and industrial energy users to build a truly sustainable energy system. They help unlock additional revenue streams for industrial customers by offering energy flexibility on the electricity balancing markets. With an energy grid that is becoming increasingly volatile with the rise of renewable energy, they provide the balancing capacity that is needed for a stable grid. Due to the complexities of balancing the grid and the need for precise forecasting, Sympower introduced the Databricks Data Intelligence Platform to better scale their existing forecasting models and make data insights more accessible in the company. The Databricks Platform transformed their capabilities, allowing for robust data management, streamlined workflows and advanced forecasting models. This shift enhanced data governance and real-time decision-making and unlocked the potential for deeper data exploration through AI-driven analytics, collaborative tools and serverless dashboards. With Databricks, Sympower can now prototype and deploy new forecasting models rapidly and collaboratively, setting a new standard in managing and optimizing energy stability.
Rising volatility challenges grid stability in renewable energy transition
Sympower, a pioneering force in the energy sector, employs around 200 professionals and operates across various European countries. The company’s mission is crucial: managing approximately 2 gigawatts of flexible capacity to stabilize the electricity grid amid increasing energy transitions. Rik van der Vlist, Senior Machine Learning Engineer at Sympower, is helping with the company’s evolution toward using advanced forecasting techniques to maintain grid stability.
Van der Vlist, with a background that bridges electrical engineering and data science, emphasizes the significance of their work. “We’re in the business of balancing the electricity grid by leveraging both production-side steering and demand-side flexibility, which is increasingly vital as we shift from traditional energy sources to more renewable ones.” Balancing supply and demand is one of the most challenging problems in the transition to a climate-neutral energy system.
The challenges are primarily rooted in the unpredictable nature of renewable energy sources and the rising demand for electric vehicles and heat pumps, which introduces greater volatility in both the supply and demand of energy. Sympower connects large industrial consuming and generating resources and steers the consumption or production of those resources to balance the grid in case it experiences a disturbance. Sympower has divided their data team into two main focus areas: forecasting and insights. Where the insights team focuses on providing internal dashboards and insights to the company, the forecasting team models the electricity markets and industrial resources. As van der Vlist, whose team is responsible for forecasting, explained, “Our goal is to predict electricity demand from our customers to estimate the available capacity for grid balancing for the upcoming days.”
Ultimately, they needed to make informed, data-backed bids in the electricity markets. There was also a need for better data governance, lineage and analytics capabilities — without sacrificing the ability to provide quick, serverless dashboards for real-time decision-making. “We don’t have any automated documentation for our data lineage, and it is tedious to keep the manual documentation up to date, especially across different projects,” added van der Vlist. This imperative for precise and scalable forecasting led Sympower to adopt Databricks.
Building better forecasting models with the Databricks Platform
Sympower’s implementation of Databricks showcases a sophisticated use of the platform’s features tailored to meet their specific needs in energy management. The platform’s versatility allows Sympower to address the challenges of big data and complex model requirements inherent in the energy sector. “We utilize Databricks for its robust capabilities in handling large datasets and complex workflows, which are integral to our forecasting models,” van der Vlist stated. “Our forecasting models are built on a medallion architecture, which includes stages like data ingestion from production databases, preprocessing, dataset preparation and pushing results to a microservice that handles our bidding platform.”
The Databricks Platform has streamlined their workflow significantly across data storage, pipeline management, and deployment and testing. Sympower’s data team employs Delta tables to efficiently manage data storage, ensuring that large volumes of data are handled with ease and reliability. Various stages of their data pipelines are composed together in Databricks Workflows, enhancing the modularity and manageability of these processes. “During acceptance and staging, our CI/CD pipelines sync this code to Databricks Repos and update the associated workflow,” van der Vlist explained, highlighting the seamless integration with their version control and deployment systems.
Databricks’ API-first approach has been particularly beneficial, simplifying the management and reproduction of acceptance and staging jobs. This feature aligns perfectly with Sympower’s need for flexibility and speed in developing and deploying new models. “The development experience in Databricks Notebooks has improved a lot over the years. It’s now critical for collaboration,” said van der Vlist. “Integrated versioning with Notebooks is a very useful feature.”
As part of their continuous improvement and adaptation to new technological advancements, Sympower has migrated their Hive metastore and externally managed tables to Databricks Unity Catalog. This migration has streamlined data governance and lineage, providing fine-grained control over data access and security while addressing the challenge of maintaining up-to-date documentation.
Stabilizing the electricity grid through improved data integration
The adoption of the Databricks Platform has not only streamlined their operational processes but also poised them for significant advances in how they manage and leverage data for electricity grid balancing. Their current migration to Unity Catalog emphasizes their commitment to leveraging cutting-edge technology to improve data governance and analytic capabilities.
Using the Databricks Platform has enabled the data team to scale their forecasting models and make data insights easily accessible within the company. “With Databricks, we are now able to rapidly and collaboratively prototype new forecasting models that benefit our customers. With self-service dashboards, notebook prototypes and SQL workspaces, everyone can participate in exploring and extracting insights from our data,” explained van der Vlist.
Looking ahead, Sympower plans to expand their use of Databricks’ capabilities to further areas of their business. One of the next steps involves enhancing their predictive capabilities and integrating more AI-driven analytics into their operations. Van der Vlist explained, “We’re exploring new AI/BI Genie features. We did some experiments with this a few months ago and think it has great value for self-service analytics.”
Sympower’s use of Databricks is central to their strategy for maintaining grid stability and responding to the dynamic needs of the energy market. The continued integration of advanced data analytics and governance tools signifies their proactive approach to embracing technological advancements, ensuring they remain at the forefront of the energy transition.
