Measuring the Success of Your Algorithm Using a Shadow System
- MLOps and DataOps
- 40 min
How to determine whether your new data product is a success if you cannot use A/B testing techniques? At Gousto we recently implemented our newest algorithm to route orders to sites. Comparing this to the previous algorithm using classic A/B testing techniques was not possible, because the algorithm requires a full set of orders to optimise and ensure the volume we send to sites remains stable. To ensure confidence in our algorithm before go-live, we came up with a different experimentation strategy. This included building a full-blown shadow system. For measuring its performance we built a set of data pipelines (including ETL) using DataBricks.
Routing orders at Gousto is a challenging task. We have 60 different recipes on the menu each week and customers are able to select 2 to 4 of those into their box. This means there are many unique recipe combinations to be fulfilled. Our routing algorithm optimises for various objectives, whilst keeping order volumes stable to allow for production and procurement planning. Given its high impact, putting this new system live required high confidence in the algorithm. Using the shadow system, we were able to prove that the algorithm was able to meet these objectives, by visualising and tracking the metrics relevant to the business and comparing them to the routing system in place at the time.
Sometimes an A/B test cannot do the job. This talk will outline challenges and benefits of building a shadow system, providing the audience with an A/B testing alternative and an overview of relevant considerations in terms of choosing and building this experiment design.