Auto-Train a Time-Series Forecast Model With AML + ADB

Moe Steller: Hello, everyone. Welcome to today’s session, The Better Together Story: Auto-train a time-series forecast model with AML and ADB. My name is Moe Steller. I am with Microsoft as a Senior Cloud Solution Architect, and I would like to guide everyone through this very exciting topic. Again, just a little bit about myself, working at Microsoft in the WCB, the Worldwide Commercial Business group, as the Senior Cloud Solution Architect in AI/ML, focusing really on NLP, deep learning, and time-series, as well as more realistic fields, such as explainable AI and AI ethics, as well as looking into scalable AI solutions, so there are many topics to cover in my favorite types of industries, healthcare, finance, supply chain, insurance, and health/life sciences.
Please provide feedback to this particular session. We appreciate it, and I would like to answer any type of questions after the session. Please send me an email or within the portal, reach out to me anytime.
All right, let’s kick it off for today’s session. So now everybody’s familiar with the title. In case you’ve just joined, I’m just going to repeat, Better Together: Auto-train a time-series forecast model with AML and ADB. And our goal here over the next 30 minutes is to look at time series forecasting, just to level-set a little bit so we have a mutual understanding. This may be a little boring for folks that have done this for a while, but for newcomers, basically, just to level-set the topic. We’re going to look at the product preview a little bit, advanced parameters, and then at the Better Together architecture. We’ll go through a diagram that logically makes sense for the use case demo I’m going to show towards the end of the session.
All right, let’s dive right in. Time series forecasting. So what does it mean? In time series Forecasting, we are looking usually at three particular use cases, a univariate time series, multivariate time series forecasting, and the DNNs, the deep neural networks. For univariate time series forecasting, you usually have a time feature, so a time column, and a target variable on what you’re trying to forecast. For the multivariate time series, you usually have a time column, multiple features or regressors you may want to build over time, and a target variable you may want to forecast. For the DNNs, the deep neural networks, it’s really about pattern understanding of a particular time series, usually on large data. So more statistical methods that exist in the space, such as ARIMA models, may not really make sense on very large amounts of data that have a lot of noise. The DNNs will help you to make you understand underlying patterns.
What does it mean in Azure ML? In Azure ML, we focus from an auto ML perspective on three particular use cases, regression, time series forecasting, and classification. For our use case here, we only want to look at the time series forecasting. The models that are provided are traditional models, such as elastic nets and so forth, but also tree models of any variety so light GBMs for example, random forest, XGBoost models, models everybody should be pretty familiar with in classical machine learning. On top of that, Azure ML enables Auto-ARIMA, which is still in preview. We’re going to discuss this here in a little bit, what that means. And also Prophet, a Facebook library that is provided within Azure Machine Learning AutoML, and the ForecastTCN, excuse me, which is a DNN enablement. So for deep neural networks, in case you have large amounts of data of time series that you need to forecast on.
Particularly from a product preview perspective, I mentioned Prophet is still in preview. It is fast and accurate, if you want to use it on usually medium amounts to small amounts of data. In my personal recommendation, it is really neat to forecast relatively quickly and also relatively accurate. It is a well-beloved library in the space. Many folks use it. There’s also a new Prophet recently came out, which is the DNN solution, currently not enabled in AutoML. But if you think about moving beyond your baseline model that you have built, where AutoML often is good at, then you may want to look into these advanced metrics and then run them in AML, in particular. Auto-ARIMA helps, which is also still in the preview state, helps me to forecast a univariate time series to predict basically the future. This is also more done on smaller data sets. So smaller data sets, let’s say, broken down up to 500 rows and whatnot really helps you to leverage this moving average technique. If you move towards larger data sets, and I just mentioned that in the prior slide, the ForecastTCN is what you need to go to. To enable that in AutoML forecasting, you need to enable a parameter of 24 hours minimum run time. As these DNNs have relatively heavy run times to even enable it within the engine. And we’ll look at the configuration parameters here in a little bit in the demo, where you can do that.
Furthermore, the AutoML config, which is basically the configuration to execute your model, comes with different types of advanced parameters. I picked a few that I think are important. There are others in different types of use cases that may appear as important. But for this use case here, let’s look at the first two. If you want to enable ensable models, stack or voting and ensable model, which is sort of like a blend model between many types of algorithms, you need to say that you want to enable them. That really helps you to build a better model than maybe a single algorithm ran model. The disadvantage you may want to keep in mind is that they’re really hard to enable, regarding explainability and maybe potential bias applications. Your customer may be very interested to understand how a model works under the hood. So I would not recommend to use them in that particular case. But if you’re only looking for accuracy, they’re a great models to go to, and they usually score high and win within the AutoML leaderboard.
Another parameter to look at is the onnx compatible model parameter. Here, you can build an onnx model. So in case you want to bring your model to another service within Microsoft, let’s say Synapse, for example, you can build an onnx model, export the onnx model, bring it to Synapse, and run inference in Synapse on the T-SQL predict function. What that enables for you is within, let’s say your star schema or your snowflake schema builds, or in case you have like [inaudible] architecture, you can have data engineers very easily run inference with AI ML on these data batches. I can personally recommend it in case you have some kind of reporting running and so forth. This comes up pretty much in handy here to enable it. Furthermore, the spark context, SC, we all know this within Spark has to be enabled when you want to run insight in Azure Databricks or Spark environment. Very important for our particular use case here, in case you want to choose the spark cluster. And furthermore, let me go to the last parameter here, the seasonality. I think this is like a binary value of minus one, one, where you can infer directly seasonality. So if you work in kind of like a retail pattern, where you have particular peaks in summer, winter, autumn, or whatnot, you can say, “Hey, there is seasonality.” And then predefine this because the business has that as identified from the get-go.
Moving on to the compute. From a compute perspective, we are looking here at the Databricks runtime for machine learning. Compute supported up from 5.5 LTS ML up to 8.1 and beyond, at this point, as well as the usage of attribute clusters is enabled. Especially when you have very large data, and you want to run your DNNs, this comes very neatly in handy. Otherwise, you can choose the M series or D series, basically, that have no restrictions on the AML side. Or if you want to go beyond, in more like HB type series, and sometimes that requires approval, usually those are pretty high compute nodes. Or sometimes they’re not available in the region so you require approval to get those type of VM series.
I want to dive in now into the architecture, before we go into the demo, just to get a little bit of an understanding how such forecast engine can sit within the larger pipeline. And you often start with data. So model data that is enterprise data, where you want to basically forecast on, but also third-party data, which is really, really important because outside data, outside of your enterprise, can inform your models, right? This is a logical correlation that exists. And just think about COVID. COVID numbers may drive, in supply chain, a loss of sales. So you want to bring COVID numbers in through third-party data to really understand why your time series acts the way it acts. Moving on. You want to stage the data. You can do this in an enterprise data like fashion for your enterprise data. Or in case it is third party data, our recommendation is to use Synapse, basically, where you have like an enterprise aggregated data pool that helps you basically to distribute a third-party data across the enterprise. Again, it really depends on your organization how you want to model this. These are two options that are available.
And then if you move on, thirdly, into the AI/ML life cycle, you bring both the assets together in a pre model pipeline, ingested into your actual model pipeline and train your model with AutoML forecasting on the particular data. This is an iterative approach that often starts with the POC or MVP, but nowadays, once you have established your baseline model, you want to go beyond that with hyper parameter tuning or exhaustive searches of your model. It makes often sense then to take the best trained algorithm in an AutoML and build on top with a customized model that then exhaustively searches for the best model within there. You can do this in AML with the same type of compute setup, as we have prior to discussed.
Fourthly, you have your analytical workload. You want to run inference somewhere. So let’s say you do it in Azure Synapse with an onnx model and T-SQL predict function, as we just prior discussed. It will really help you to set up your environment there efficiently. And in this particular scenario, let’s say you work in some kind of [inaudible] reporting fashion. You then have, fifthly, your front end reporting in Power BI or in a native web application, depending on how Orca Enterprise prefers to visualize your results from your forecasting, show to business stakeholders, executives, whomever’s interested into these forecast numbers. Again, this is more the traditional forecast approach. If you work in IOT, forecasting can look a little different because it is done usually on [inaudible] architectures or in near real time only. You don’t really have batches. So something to keep in mind.
There’s a particular use case I want to touch on when we were talking about forecasting. So especially in retail, where you have many, many stores globally or nationally, you want to break down your forecast to an individual store level. This concept is called many-models forecasting. The reason why you want to do it is because the individual forecast on a store gives you the accuracy of the overall aggregation of all stores. And you can imagine if you forecast on a national or on a state level, for example, the accuracy and the algorithm may not execute as well as you would do it on the individual store store level, where you could tweak the algorithm in a certain way. There’s a particular pipeline. You can enable this particular architecture with a parallel run step in Azure ML. So you can train up to N models, like hundreds of thousands of models, at the same time. And the inference then ties basically a model ID to your data that comes in and knows how to run inference on an individual store. After you have done your predictions on the individual level, you can aggregate it up. Again, show it in Power BI to business stakeholders and to get the best accuracy out of your individual store levels.
All right. This is a couple of examples in energy. It’s pretty good maintenance of thousands of oil wells, for example. Retail, we just talked about it. Restaurant chains, it’s really important for demand forecasting. If you think about these large chains, basically, they were especially impacted by COVID. You bring in third party data. You run the many-models approach. The accuracy overall goes up because you understand the correlations, and you run it on a particular store level. None of this architectural approach would be able to be enabled with all the Ops, I like to call it, so DataOps, MLOps or AIOps, and DevOps. The reason why, DataOps brings basically your governance together, keeps track of your data sets and so forth, and how you set up your data pipelines. MLOps takes care of the overall automatic deployment, takes care of model drift, data drift, and anything that goes into retraining your model, or your model registry. Especially the many-model use case, if you use ML flow, it is really important to have MLOps properly set up to keep track of your model changes over time. And DevOps automates the entire system and the entire environment. So all the Ops is really the key to success to build a production ready product.
I want to now jump into the demo. Just heads up, for the demo, we are looking into the data, trying to prepare it for time series modeling, very rudimentary. We’ll go into the specifics of the parameters with the AutoML config object. So I’ll point out a couple of things, what to look out for. We will not run inference. This will go beyond this use case. And quite frankly, it will take many hours to explain an intelligent system with MLOps around that. So let’s jump right into the demo.
All right. So within the demo, we will start just importing the data. We are working here on a beer forecast use case. So I hope everybody’s pleased with that example. But also the beer forecast is a pretty well-known, open dataset. So what we are doing first, we’re importing our data. This is our training, sorry, testing data. This is our training data. Overall, we are working only with 319 rows of data, consisting of a date column and a beer production column. We’re importing the data, setting it up according to the Databricks. I’m going to skip these steps. These are just for my exploratory data analysis to understand the data. We’ll import, this is important here, the Azure ML SDK notebook within Databricks, installing that properly, and then installing, under this particular TXT file, our AML. This errored out, as of this morning. I need to investigate why it worked in Jupiter Notebook in parallel. I apologize for that. This will work on your end. I promise.
And then you’ll basically just check what kind of version you have, as one of the main, major important steps to understand that everything is in sync. We’ll import particular libraries. So this block is really important, especially for the setup of our forecasting. And following on, we want to log into our environment. So this is my particular tenant ID for a secure environment. I’m going to scroll down a little bit because you want to disregard this. But other than that, you’ll basically set up your workspace in your individual environment. And then you’ll basically get this little output window here that just defines that you properly logged in. There will be a question that you have to log in with your email address and so forth and enter a security code that is tied to your particular email address to basically connect your notebook to your AML environment.
You can build individual compute within your environment. So this particular step ensures that. You can just build a compute. In this case, I’m building a DS3 V2 standard machine in there. If it already exists, it will tell you, “Found existing instance.” You can use it. So this is just pretty neat as a fail safe. Basically, does the VM already exist? So now where we have logged into AML and built our compute, we want to look at our data. You see that there is a pretty predominant pattern of swings, basically, between the years and up and down with even higher swings going forward. So as a prediction for moving on, as we want to predict the beer production, you can technically expect higher swings over the years but in initial growth, basically, right over the time. So this is our expectation.
Now, I want to set up basically my data set. I apologize for the very redundant code here. It’s not efficiently written, and that’s really not the goal. I just did it in a copy and paste type of fashion. I’m going to set up my three data sets, my training dataset, my validation dataset, and my testing dataset, which is split into 40 rows for training, 60 rows for validation, and 19 rows for testing. I am establishing all my datasets within the DBFS. This is all very standard practice, nothing out of the box. And one thing to point out is if your data set is basically in the DBFS, you want to derive it from there with the object datasets tabular from delimited files. This is the input format that is required for Azure AutoML for forecasting. And once that is done, we can really go into our setup of our algorithm.
So, first off, I want to define my target column. Where do you want to predict on, right, or be a production? I want to define my date column. This is basically, from a univariate perspective, the date that defines the production over time. And then I want to define the forecast horizon. So since we’re dealing with monthly data, I want to predict one year out. You can add different types of featurization of individual columns to define basically if they’re numeric, or if they’re daytime. This is a pretty important step, as you want to have the right data type going into the algorithm. And then we’re really going to set up our algorithm.
So there are two parts to it. First, our forecast parameters, consisting of our time column and our horizon, how far you want to forecast. And then on top of that, you can define, I haven’t done this in here. You see this. I’ve commented this out. But you can say, in case you have particular swings and holidays, or there are no sales during holidays. You can do it on a day by day level forecasting or weekly forecasting level. You can define holidays in the US, for example. You can define if there is a seasonal trend. You can infer the seasonal trend. We talked about this in the advanced parameters, in the beginning. You can infer particular feature lags, in case you want to more go into like a moving average type of prediction and so forth. There’s entire lists out there. I would recommend to go to docsmicrosoft.com. You can go through that list, basically see what fits your forecasting parameters the best.
Moving on. The AutoML config is really the most important part here. You define what type of task you want to execute, regression, classification or forecasting. Excuse me, in this case, it is forecasting. You want to enable DNNs, if your data sets is more than 500 rows. That is the minimum for an enabling DNNs. Mine is 319 so a DNN won’t run. So that is basically a check within your AutoML. You can define your primary metric, how to evaluate basically a model. In this case, I have a normalized RMSE, in this case. You can choose an R square, a MAPE, whatever you want to do in [inaudible]. I chose this particular metric just because it shows the ranking the best for this particular example. Again, this is up to your business and up to your industry and use case, how you want to evaluate your model. You can allow a block model in case [inaudible] model takes too long to run. You may want to eliminate it from further runs. Basically, all that good stuff.
A couple of things I want to point out. This particular parameter here, the experiment timeout hours. They have to be set to minimum to 24 hours, if you want to run DNNs. So if you have all 500 rows of data, you have to set it to 24. There’s also within details in your AML studio, you can see that as a detail, but for the SDK or if you want to code it out, 24 hours minimum, very important. And then you just really define your training set, your target column and so forth, your validation set. You define your compute cluster. In this case, I’m running it on an AML compute, not on the Databricks cluster. If I want to run it on the Databricks cluster, I need to enable the Spark context, the SC. Again, advanced parameters. I mentioned that earlier that you have to change your cluster names, basically, right?
Other than that, besides by inferring the forecast parameters up here, you’re good to go. And you are able to execute your run. Your run is executed with experiment.submit of your AutoML config. You can add some parameters here to this particular submission that define basically what it is. You can say it runs with DNN or so forth. So you can leave text comments there to basically get more control in your studio. And in the individual AML studio run, how it looks like, basically you’ll have all your models listed.
So let’s go one step back. So I can show you how this looks like. And you will see many failed models on my side because I was trying out a few things. But if you go in here, you can click on models. This takes a second to load. You see all your models that are currently running. Our current leader model with a normalized RMSE of 0.05 is the profit model. So the Facebook library leads currently. You see different types of models are ran currently. The list is long. I think I’m doing like an exhaustive run here to just run everything and see how it turns out on my side, what the leading model is. There are many different pages, and you see the normalized RMSE increases over time. I’m running four models at the same time. I also defined that in my parameters. And once, basically, your runs succeeded, it will say finished up here. You can click on your individual model.
In this case, let’s just pick the profit model. You can look at your run metrics over here. I’m not evaluating those. An R square a 0.71 may be bad. It may be good, depending on the data and how the current state of the models is. It actually just completed. Up here you can see that. You can look into the metrics. This takes a second to load, which gives you model insights, basically your predicted values versus your true value. So this is good for model evaluation, as well as your residual histograms and so forth. So very great. You can enable your model explainability. To build it out, you need to attach I think a compute test so I haven’t done this yet. You can try this out on your own time. Please ping me if you have questions about this. Again, fairly standard [inaudible]. After you’re done with your model and build selection, you can basically aggregate the results, and you can isolate your model. In this case, I ran again and isolate the model and basically take the best model and run inference on that. So this is the basic pipeline that leads you through this particular demo.
Well, so thanks so much for hanging in with me during the demo. I want to cover one last slide here before we want to close the session. So what are the next steps when thinking about inference or deploying a model? So you want to think about, from a Databricks perspective, about MLFlow. That is one recommendation I can make, especially in these many-model approaches, to build a model registry and track your models. It becomes important, right? You cannot keep track of hundreds of thousands of models often. So this is a recommendation. You also want to look maybe into Hyperopt if you want to automate and distribute your ML parameters and really do am exhaustive search of models, basically, especially when you go beyond the baseline model training. On the AML side, especially for very large data, consider using GPUs on AKS, basically, for highly paralyzed computation, as bigger your data, as heavier your workloads. It’s more complex in models. So you want to consider that if you want to think about to build more containerized environments. So Docker images are recommended. They are also easily to stand up within Azure to get more tighter control of your model. So in case you have just like a build and dump type of environment, you just want to abandon very quickly, this is one way to go and to consider for productionization.
I want to thank everyone to hang in here with me for Building the Better Together story for Auto-train a time-series model. I hope you had fun. Feel free to reach out if you have any questions on [inaudible] model build in this space for demand forecasting supply chain, for an AML, particularly how to build forecast models. Let’s discuss. And I hope everybody enjoyed the session. Thank you so much.