Scaling for MHHS: Octopus Energy’s 50x Cost-Efficient Margin Data Engineering

As the global energy system moves to intermittent renewables, the challenge for data engineering is providing a clear signal to consumers for when the energy is at cheapest and cleanest. This talk is for data engineers and leaders navigating the UK’s Market-wide Half-Hourly Settlement (MHHS) transition. Facing a 48x surge in data volume for 8M+ customers, Octopus Energy re-engineered its margin systems to bridge the gap between monthly billing and half-hourly settlement.The Results:Data Efficiency: 98.8% reduction in rows processed (from 25B to 300M).Cost Savings: 50x cost reduction per settlement date, avoiding $1M in annual infrastructure increases.Velocity: Improved data freshness from weekly to daily.By decoupling the pipeline into grain-specific modules and applying Spark and Incremental optimisations, we transformed a regulatory hurdle into a scalable engine for margin visibility. Aligning your data architecture with the business logic enables a sustainable energy future.