Predicting the movements of price action instruments such as stocks, ForEx, commodities, etc., has been a demanding problem for quantitative strategists for years. Simply applying machine learning to raw price movements has proven to yield disappointing results. New tools from deep learning can substantially enhance the quality of results when applied to traditional technical indicators rather than prices including their corresponding entry and exit signals. In this session Kris Skrinak and Igor Alekseev explore the use of Databricks analysis tools combined with deep learning training accessible through Amazon’s SageMaker to enhance the quality of predictive capabilities of two technical indicators: MACD and Slow stochastics. We use the S&P 500 as a baseline for prediction. Then we explore the capabilities of optimizing the statistical parameters of these indicators First, followed by hyper parameter optimization of the deep learning model deepAR. The session will illustrate how to build such indicators in Databricks notebooks and extend Databricks’ functionality to train deep learning models in the cloud via PySpark and Amazon SageMaker. No prior experience is required.
– Hello, My name is Kris Skrinak. I’m a partner solutions architect and the machine learning segment lead for the partner organization at AWS. Today I’m joined with Igor Alekseev, the data and analytics partner solutions architect, dedicated to Databricks and Rumi Olson, a machine learning specialist, also at AWS.
In this presentation we propose a method for interpreting equity price action as a conversation between two participants, bulls and bears. We provide a way to interpret the conversation using classical technical analysis, and predict what will happen next using machine learning. Nothing contained here is investment advice. Always check with your registered financial advisor before making personal investment decisions. The best opportunities that exist for investors are the misperceptions of others. When investors as a group lose their way and misprice an instrument, another group of investors rush in to correct that mistake, they beat the market. Market participants consist broadly of two groups, bulls who insist prices will continue to rise and bears who insist that prices will continue to fall. From their perspectives, and from the moment, they are both right. There’s only one thing that obscures their vision, the hard right edge. That is the right side of the chart right now. That’s where history ends. For fundamental investors, the hard right edge is the limit of historical data about a company, its supply chain, its ability to acquire and retain talent, the news. There’s another way to discover misperceptions. By listening to and interpreting the conversations of market participants. This conversation and its lexicon has existed as long as the markets. Japanese candlestick charts go back to the 17th century, the same century as the Dutch tulip crisis. Today we call that language technical analysis. Just as deep learning has given us the ability to predict how to finish our sentences when we type in Gmail, we can use similar techniques to look ahead and predict how bulls and bears will finish their sentences in the market, based on their sentiment. Tactical analysis has been extremely useful over the years to predict short-term behavior, but it often misfires, and those misfires can be costly. We’ll combine a Databricks notebook with powerful built-in Amazon SageMaker algorithms to make those predictions. Let’s start with the basics. There are always only two sides to any trade. A decision was made to buy and sell by two parties.
Sometimes prices move from lack of participation, that is buyers are not showing up. Therefore, only those most inclined to sell, or perhaps who have a directive to sell, can participate. Prices must fall low enough to find a bullish buyer. Of course the opposite is true when the sentiment is reversed. For any behavior to happen, three things need to occur at the exact same moment. There needs to be an ability to act. That could mean either having or needing liquidity. There must be motivation. That could be either need for capital or for investment returns. Importantly, no action will ever be made without a trigger, an event that says, “The time is now.” Technical analysis provides clear signs to what is behind these three essential dimensions of every trade. Rational market theory states that there is perfect information in the market, and that agents act logically with that information. The behavioral school takes a different approach, and “Extraordinary Popular Delusions and the Madness of Crowds” a book more than a 100 years old. The madness is the Dutch tulip buying mania. It’s the first use case of your rationality overcoming an otherwise rational market. You don’t need to be a psychologist to understand this. Some things are pretty obvious. People remember prices. If your favorite stock close to 20 consistently, and today you see it at 19, you think it’s a bargain. You’ll pick it up, you have the ability, you have the motivation. The lower price, whether justified or not, is the trigger. You’re ready to buy, and you act. To understand technical indicators, consider a pilot who can land her aircraft either with vision, or entirely based on her instrumentation. Using technical analysis to interpret group behavior, is very much like that pilot making an instrument only landing. Trendlines are linear equations that join highs to other highs and lows to other lows in a trend. That trend can be up, down, or sideways. What’s fascinating about trendlines is how simple and easily discerned they are. Remember a moment ago I said that people remember prices. This is reflected in a phenomenon where prices move in a channel for a period of time. Then there’s a breakout. A breakout could be on the up or downside. Soon prices will snap back to their former levels. People are habitual. There may be good reasons for the breakout, but it’s our nature to return to safety before stepping into the unknown. The hard right edge is the home of the unknown. Moving averages, especially front weighted exponential moving averages capture gradual changes in sentiment over time. Having worked on Wall Street for many years, I have some insights into block trading, large institutional behavior.
It’s much more difficult to buy a 100,000 shares of any company than a few 100. Institutional trends track over a longer period of time, there’s a wider band. Faster moving individual investors, versus institutional are relatively easily characterized by shorter and longer exponential moving averages. Finally, the granddaddy of all trend indicators, MACD is a highly effective gauge of trends strength. The oddly named stochastic is a hot trigger. It’s a common precursor to a trend, a change in the trend, but it frequently misfires.
Excuse me, it rarely misses the reversal, thank you, when the target is real. Looking at this chart from left to right, on the bottom orange and the middle green lines, Amazon traded between 1685 and 1815 from September, 2019 to the end of that year. This pattern is called range-bound. It’s a sideways trend. Then something happened. There was a Christmas breakout. Breakouts are important, perhaps the most important market signal. They indicate that the conversation has changed. Where it ends, no one really knows. Every breakout is driven by the strong conviction of the dominant group, in this case bulls. The trend ends when the bull gets tired. The bears who happen to own Amazon as well take profits as good bears do, and they become the dominant group for a short time. However, their end point is a bit more predictable. It’s called a pullback. Frequently to a well known price as we see at the end of January. That’s how resistance becomes support. Switch the rolls around, and that’s how support becomes resistance. People are habitual. In the case of an uptrend, bulls have full control. Prices of the 500 largest companies by market capitalization are represented here by the ETF SPY. They were on an uptrend from September of 2019 to mid-February, 2020. Bears had their moments. You can think of the bouncing around this channel as bear is saying, “Excuse me,” but they could not become the dominant group. Then something happened in mid-February. Bears took over, bulls lost all conviction. Notice that at the end of that month, the bulls said, “Excuse me,” but it didn’t last long. We fell to 2018, 2018 does now support a position that was resistance when Trump entered office. Drawing straight lines on a chart seems like a strange way to describe a fluid conversation. That’s where EMAs come in. I found over the decades then we’re looking at the broad market. The two most effective EMAs are the 13 and 50 day EMAs. You can adjust these for other instruments. However, their meaning is unchanged. The 13 day EMA represents the speculators, day traders, and individual investors.
They’re not in this conversation for the long haul. They just wanna see it move along. When it gets boring, either bullish or bearish, or if they’re punished for their misperceptions, they get out. The 50 day EMA is the conservative group, institutional investors, those seeking capital preservation and long-term gains. They too must assess their commitment daily, and adjust by becoming either bullish or bearish. When those lines cross, the conversation has changed. Think of a dog chasing a rabbit. It gets a hint and it runs. Initially the rabbit is fast, but the dog may catch up. The cross of the 13 and 50 day EMAs is a consistent sign that the conversation has changed. Let’s dive deeper into the anatomy of a trend. If EMAs are so powerful, then what would happen if we turned our entire attention to the distance between those two groups, the individual and institutional investors, or the rabbit and the dog?
If the rabbit is continually escaping the dog day after day, then that dog may have no chance of catching her prey. This divergence of moving averages is a strong trend indicator, so strong that it overpowers the day-to-day thinking of the participants. It turns bulls into bears and vice versa, and as it’s continuing strength snowballs as it did in early March, until we hit a climax.
Imagine you couldn’t see the prices of the instrument we were watching, we could only see MACD. Can you tell, looking at this chart, excuse me, when sentiment changed? Can you spot when the best time was to buy and sell? Now we’re flying by instruments only. When the fast line crosses the slow line, we have an opportunity to get ahead of the pack. In fact, that based on MACD alone, our first cross is a misfire. We shorted and had to cover quickly. However, second time was a charm and it was a doozy. MACD is built on historical prices, so it’s always late. In fact, it’s a conservative indicator of the conversation. What if we want to narrow in a bit and hit the exact peaks and troughs? Well, let’s mix a few more metaphors. Imagine two boxers entering a ring, it’s a well-matched fight, either fighter can win. However, they both have their limits. They will, if only for a moment, lose their momentum. They will need to recover their strength. That’s what stochastics measures. It’s what happens at the limits of exhaustion.
Stochastic is measured from zero to 100. It’s an average of the percentages to which prices close consistently, either at the higher low of the day. Over time, bulls and bears enter the exhaustion zone. Sometimes that’s called overbought or oversold. A cross in the top 20% of that range signifies a change. It means the bull is exhausted, it’s the bear’s time, and vice versa. In fact, stochastic frequently misfires. It tends to get pegged in a solid up or down trend. And that makes sense. The bull or bear has lots of energy, it doesn’t need a break. However, when it fires, it’s right. And it’s always right, right on time.
Now that we’ve introduced a new vocabulary of group psychology, and how quantitative derivatives can give us powerful insight, we’re going to propose a novel strategy, similar to ensemble models.
The first indicator will be the bullish or bearish cross of stochastic. We don’t mindlessly buy or sell based off of a single indicator. We look for confirmations. The second indicator may confirm that it’s time to act. If we’re conservative, we may wait for a third. We may sacrifice some profits to reduce our risk in the long run. The first mantra of every good wealth manager is capital preservation. This is great, but we’re left with a serious problem. We can’t see the future, nobody can. Using deep learning, we can make predictions with a degree of confidence, far beyond the coin toss of what’s coming next. With these predictions we can act with a greater degree of confidence than the lowly technical trader of former times. Let’s take a look at our options. The ARIMA algorithm is especially useful for data sets that can be mapped to stationary time series. The statistical properties of stationary time series, such as auto correlations are independent of time.
We can reformat our data to stationary eyes it, but trends are by nature not stationary, so we’re off to a bad start. NPTS generates predictions for each time series individually. This is getting better. The ETS algorithm is especially useful for datasets with seasonality and other prior assumptions about the data. ETS computes a weighted average over all observations in the input time series data set as its prediction. The waits are exponentially decreasing over time, rather than the constant waits and simple moving average methods. The waits are dependent on a constant parameter, which is known as a smoothing parameter. Facebook’s Prophet is especially useful for data sets that contain an extended period of time of detailed historical observations, have multiple strong seasonalities, include previously known important, but irregular events, have missing data points or large outliers, and have nonlinear growth trends that are approaching a limit. But all of the above are statistical methods that fall short of the power of deep learning. DeepAR is a supervised learning algorithm for forecasting one dimensional time series, using recurrent neural networks.
Classical forecasting methods, such as ARIMA and ETS, fit a single model in each individual time series, and then use that model to extrapolate the time series in the future. In many applications however, you have many similar time series across a set of cross sectional units, just like stocks in a portfolio. In our case it would be beneficial to train a single model jointly over all of the time series, DeepAR takes this approach. When your dataset contains hundreds of time series features, the DeepAR algorithm outperforms ARIMA and ETS methods. You can also use the trained model for generating forecasts for new time series that are similar to the ones that it has been trained on. It’s commonly said that 70% of any stock’s movement is thanks to the broad indices, 20% to the industry, and just 10% to company news. DeepAR looks like it’s the right choice for predicting a stock portfolio. Now, Rumi will introduce the architecture we’ve proposed for this forecast. – Thank you, Chris. So here’s our reference architecture with Databricks Unified Analytics platform in the center, Amazon S3 and Amazon Kinesis are shown to be the data sources from where Databricks gets data. Databricks prepares the data to be useful, building a machine on your model. Once the training and test datasets are ready, you upload them to S3 bucket. You can upload the datasets and kick off the training all from that notebook running on the Databricks platform. As you can see in the diagram, Databricks can write data also to Amazon Redshift, which is a data warehouse service and Amazon RDS, relational database service, and Amazon DynamoDB, no sequel database, after you have done processing the raw data. You can also set the data for BI or reporting tool, straight from Databricks. Towards the end of last year we have launched our new service, AWS Data Exchange.
Data Exchange makes it easy to find, subscribe to, and use third party data in the cloud. Qualified data providers include category leading brands, such as Reuters who kill rate data from over 2.2 million unique news stories per year in multiple languages, and Foursquare, whose location data is derived from 220 million unique customers and includes more than 16 million global commercial venues. Xignite is one of such data providers and are provided the data to build a stock price protection model that you will see later. They are the market data industry leader with its AWS-based financial data distribution and the market data management solutions Xignite’s market data cloud platform provides market data managers, fully transparency on the usage of our firm’s first level, real time and our reference data, making it easy for data owners to entitle control, control consumption, and comply with licensing requirements. Please go next.
Here is a solution architecture or building the market sentiment production model. So we get a portfolio historical closing prices provided by Xignite via AWS Data Exchange. You can export to S3 bucket from the data exchange console, and data bricks picks up the data and compute various indicators that notebook running on the Databrick’s platform kicks off the SageMaker model training. Once a training completes, SageMaker saves the model artifacts in S3 bucket. The algorithm built into SageMaker have a number of advantages, and I encourage you to follow up after this webinar to understand the advantages. The particularly when it comes to large scale and cost optimization. We are using DeepAR, because it is designed specifically to handle very large datasets of multiple items, such as tracking inventory or web service statistics. In this case, we combined two years of historical prices for a stock portfolio which can contain any number of symbols. Igor will now guide us through that build and execution of the notebook on Databricks and Amazon SageMaker. – Thank you, Rumi. Hi, my name is Igor Alekseev. I’m a partner solution architect at AWS. In my role I work extensively with Databricks.
I’m going to talk now about the mechanics of how we went about figuring out market sentiment. And Chris at the end is going to talk about what we found. On this slide, so first things I had there was some prerequisites, and I had to set up an instant profile. This is the future of Databricks. So this instant profile corresponds to a role on AWS. In that role I had to give access to SageMaker. When Databricks launches a cluster, it attaches this role to the nodes running the cluster. So this role is needed to be able to run SageMaker. And when I launch the cluster, I attach that role, I specify this role in a cluster, I’ll show it to you in a second. And then when I can, on the right, at the bottom right you can see when the snippet of code, when I run, I can now establish a SageMaker session and specify my buckets and run the code. So at this point I’m going to switch to my notebook and show you the code we’ve created. Great, thank you. So, now we have this cluster. This is the cluster I’ve configured to run SageMaker. So this is the instant profile I’ve selected. Some additional configuration was needed. I had to install extra libraries. I had to install SageMaker, Plotly, S3FS, Spark Excel,
this is for reading the Excel data, but in the end we didn’t use it, it was a CSV. So we started with data exploration.
First, so as Rumi mentioned, data arrives in S3. We were not sure if the data is going to be incremental or full load, so yeah, I coded for incremental. If it takes a full load, you just drop it in this directory, it will take it. So this is a standard spark code by spark code.
It reads first the schema, it reads the CSV. With this code I was able to look at the data and have a preview. And because I was thinking about potentially working with updates come in, so I had to merge it. I was playing with one ticker, but for multiple tickers, you just need to supply also symbol. So I had, since I have multiple symbols here, I added extra tickets.
I was able to build the charts. This is example of a chart that is built in using built-in Databricks Visualizations. Here I used Matplotlib to construct the same.
I drew candlestick diagrams, which were useful in terms with the calculations. Next I’m gonna flip over to the notebook where we actually used… We interfaced it with SageMaker.
So SageMaker, let me just scroll to the top. So we are interactive with SageMaker using Python APIs.
The notebook prepared the data on S3 as we discussed before. I wanna show you here, have a look at the tail of the data. I wanna show you, in here Chris I built some diagrams based on the data that was loaded.
I wanna show you how the interaction with actual SageMaker happened. So yes, here we created the session. We have to specify the region, we got the role. So that’s the role that got passed to the cluster. And then we specified what kind of hyper-parameters we’re going to run with. Then we created estimators to specify the size of the image. And there’s a code, here it fit. So at this point, a call was made to the SageMaker site, and this is coming from SageMaker now. So SageMaker’s running the job. I’m gonna slip into SageMaker UI for a second, just to show you what happened on the SageMaker site when we run this code in SageMaker,
if you go to SageMaker jobs, you will see that
on the training job you’ll see all the jobs that started running. You’ll see their status, you will see when they start and how long it took. So this is the interface between Databricks
and SageMaker. After model got trained, here we create estimators
and sorry, here we create end points. I wanna show you how endpoints look like, the code.
Yup, yup, yes. So here are the end points, deploy it. So this results on the SageMaker side, on the endpoint screen you can see all the endpoints that got deployed. So here are the corresponding endpoints. So there all we could see all this end points that got created for different estimators. And before I hand over to Chris to talk about the results, I would like to point out what happens if something goes wrong. So you’re running something in Databricks, interacting with SageMaker, you got an error. So here is an example. Something went wrong, there was an error. Right here you’ll see the URL. So this is the URL to CloudWatch trails. You’re just gonna create that URL, copy here and go to the CloudWatch.
And you can see, so yeah, immediately you see here are the messages that were listened. And I know this one was related to this expected object of value. And the expectation was that, expected logic, expectation was that there was a value supplied, but it was not supplied, so this is the value error. So in summary, you can see how easy it was to interact with the SageMaker while running in Databricks. This gives you the ability, so this setup how we describe it. It gives ability to choose the best model or the best algorithm that you want to do model architecture, the best model architecture you want to do. You have choices of both Databricks and SageMaker, being the same environment. And I think this is a powerful story. And I’d like to hand now over to Chris to talk about the actual results. Chris, just let me know what you wanna look at. I wanna start with this diagram. – Sure, well, so we have actually those,
that’s not the newest plot. – Oh yeah, hold on, so this the DeeperAR. – Yeah, it’s a possible- – These are the newest post, yes. These are the one that we had errors, that’s correct. – Yes, yes, yes. Oh, there we go, beautiful. Okay, excellent, yeah. All right, so here’s the state of things today. If you take a look at this chart, we actually have the closing prices, the blue line, but what we’re really interested here is… What we’re really interested in is the R signals, R indicators. So first of all, the 13 day EMA, the green line, is well above the 50. However, if we scroll down just a touch, we could see that MACD is actually in bearish neighborhood. Now that doesn’t in its own say that the uptrend that began just a few weeks ago is going to end. So let me just remind everyone from the previous charts that we looked at that we entered an uptrend. So we’re actually looking at today’s prices as of June 4th, 2020. So it’ll be interesting to see how our prediction pans out when this airs, and what happened was we reached a limit at 2525, but we ran a bullish trend, right? We got that correct bullish signal back in mid-March that we needed to get back in. There’s a wide disparity between the 13, 50 day EMA. And we had a really strong MACD bullish signal. Now, MACD went bearish here, but look at stochastic. Stochastic is well in the center, it’s well in the mid-zone, it’s not at any of the extremes. So as we read this chart today and looking into the next two weeks, which should cover the time of this broadcast, we’re basically saying, “Hold on, this is a hold.” In fact, if we back out and look at a little bit more history, it may be that this equity is trading between 2250 and 2525, 2250 and 2525.
Now we know that Amazon tends to channel. So that may be the new support and resistance for this, but you know, we’re not gonna rely on our intuition here. We’re just gonna go with the conversation. The conversation is bulls are still in control, showing a little bit of fatigue, and this is a hold essentially.
So I just like to point out one or two other features from this chart. Oh, thank you. These are the actual results looking forward.
So sorry about that, Igor. Actually the chart we were looking at was the history, here’s looking forward. Thank you, I just was reading that as if it were the future. So what we’re seeing though is still somewhat identical. The 13 EMA remains consistently above the 50 days. So we’re still in that bullish uptrend, MACD looks really strong, but we’re drifting into negative territory 14 days out. Okay, fantastic. Sorry about that misreading there. I wanna point out one or two other features of this particular notebook, which is available online. It’s actually in my GitHub repo. We’ll post the link, and I’ll put post-it right here. So there’s a lot of features here. The first feature is that you can put any number of symbols into your portfolio, and they and only they, will be the symbols for which you’re training. So this is a highly customizable model to your portfolio. We’re not just looking at the S&P 500 et cetera, and pricing forward based on a single ticker. Secondly, we can choose any date period to create our trained model. Now that’s key, we’re right now at the end of, you could either characterize it as an eight or 10 year bull run. It doesn’t make any sense to look at the last two years of pricing to project what’s gonna happen in the next 14 days. In fact, what would be a better choice is with the last time we had a recession, which was say 2007 to 2009. For this notebook and the demo I chose 2014 and 2016, just for ease of access for folks to get other prices. So train your model on the model that represents the market conditions which you think you’re in, right? This is not a black box, you need to use judgment. And this is probably the most important piece. Like I said, just doesn’t make sense to train on the past two years when you can go back and get a crisis period. The third bit is how quickly this model trains and deploys, the way we achieve that is when we’re training our model, We specify in the fifth statement not to wait. We set wait to false, and Igor pointed this out when he pointed to the SageMaker control panel, that when we launch all of these training jobs, those training jobs are actually launched simultaneously. So you have five independent models training simultaneously. We wait on the last one so that we can go back and get that so that we can sort of pause and let everything catch up. Then when we deploy the endpoints, we do the exact same thing. So we’re training and deploying five completely independent time series models at the exact same time. Now there’s a couple of other features that we could put in here that are pretty cool. But at the end of the day, this is a pretty robust notebook for beginning a predictive technical analysis view
of market sentiment. So, consider the big dipper for a moment, because we are so far away from it that every evening when we look into the sky, it looks to be a fixed shape. It is unchanging, even though we ourselves are spitting on a globe that’s moving more than 8,000 miles or 12,000 kilometers from where we were in the morning to the evening. And in fact, the sun is similarly in motion. So we in fact are never in a fixed position day-to-day. However, the big dipper seems unchanged, constant, and permanent, and this is how most people view financial markets. It’s fixed, and it can be viewed through a lens of fundamental or technical analysis. It could be anchored with modern portfolio theory or rational market theory. The basic assumption is that the market is always right. In fact, if we were able to leave our planet, leave the solar system and travel in a cosmic circle around its seven stars its shape would change. The Milky Way would change. It might become a perfect septogram, in which case in another world, we would call it the great septogram. Similarly, the market itself does not exist per se. It is a reflection of the relationship of all of its participants. The conversation about the market is the actual reality. It is fleeting, momentary and mercurial. The quantitative analysis of that conversation can give us powerful insights into how we should act around the market. In fact, it’s the conversation that’s more interesting than the specific points in three dimensional space, because that reality is constantly changing as well. The technique presented here to describe that conversation is having two participants, bulls and bears, and their sentiment at any given time does little to explain the fundamental reality of the market. However, it does give us a powerful metaphor for participating in the conversation. This conversation like many is predictable. We can predict how the market will end its sentence, but we can’t predict what it will say next. Thank you very much for joining us. Thank you, Rumi, and thank you Igor. It was our pleasure to present this, and at Amazon and AWS we’re continually seeking to improve these tools and make our partners such as Databricks the best place to do deep learning and data science.
AWS
Kris Skrinak is the Global Machine Learning Technical Lead for the Amazon Partner Network (APN). He co-founded the Machine Learning group for the APN in 2017. Kris started his career as a Quantitative Strategist at Goldman Sachs, developed predictive maintenance apps as an AI Engineer at ATT, and prior to Amazon was Computer Vision Architect at GoPro. He founded and sold 2 Silicon Valley startups in Finance and Network Monitoring: ClearStation and SiteRock. As the technical lead for horizontal support of AI/ML he's the 1st line of support for AWS Partners looking to develop or expand their use of these.
AWS
Igor Alekseev is a Partner Solution Architect at AWS in Data and Analytics domain. In his role Igor is working with strategic partners helping them build complex, AWS-optimized architectures. Prior joining AWS, as a Data/Solution Architect he implemented many projects in Big Data domain, including several data lakes in Hadoop ecosystem. As a Data Engineer he was involved in applying AI/ML to fraud detection and office automation. Igor's projects were in variety of industries including communications, finance, public safety, manufacturing, and healthcare. Earlier, Igor worked as full stack engineer/tech lead.