Geospatial Analytics at Scale: Analyzing Human Movement Patterns During a Pandemic

Video Transcript

– All right, welcome to Spark Summit. First of all, I guess welcome to this session. A couple of intros. My name is Jim Young. I’m a business development lead and partnership lead for Esri in our commercial sector. I’m based here in Portland, Oregon and with me virtually today we’ve got Joel McCune, who represents our GOAI team and builds solutions that combine geography with artificial intelligence. So today we’re gonna talk to you about how we’ve combined the power of Databricks with Esri to build a COVID risk index based on human movement. So let’s get started and jump in.

Geography Matters,

So, we all know that location data’s everywhere. Whether you’re building a simple app that just finds coffee on a map or you’re trying to do something a little bit more sophisticated, like keeping a plane in the air, geography matters. A growing number of data scientists like you all are starting to see the value in applying that geographic lens onto your analysis, which is great. Because geography is really what brings context and understanding to your data, whether that’s through visualizing it on a map, whether that’s through building up location data to power explanatory variables, or even just bringing in contextual data to help put your data in perspective. It’s all geography and it’s sort of this additional lens.

And so although you don’t need a map to take advantage of geography, the map is a powerful metaphor for understanding because our brains evolutionarily are hardwired to understand complex 2D and 3D datasets spatially and physically. It’s very natural to consume data this way. So whether you’re trying to understand the impact of the weather on sales or trying to figure out where to put the next cell tower based on RF propagation or even how to plan a logistics suite, maps help you understand and decide and as Joel likes to say, the map is the original infographic.

Geography Matters (in a pandemic)

– So when we look at this idea of geography matters, particularly in the context of a pandemic, the current crisis we have over COVID, COVID can’t be transmitted unless there’s person to person contact. Jim currently sitting in Portland and myself sitting in Olympia, Washington, there can’t be a disease transmission unless we are physically in each other’s presence. Obviously this is the reason why social distancing works but ultimately if we want to be able to understand how risky an area is, either in terms of going there or in terms of where people are coming from, we need to understand a couple of factors about this and so, we’ve been looking at this and examining existing social distancing metrics and one of the things that is challenging about this is many of these have been normalized. So what this means is it allows us to understand how well people are social distancing in say, the middle of western Kansas, as compared to New York City. But ultimately from a risk perspective, those are two very fundamentally different places because of the fact of population density and the number of interactions that are occurring. So when we look at wanting to quantify risk in the lens of a pandemic, we need to take into consideration a couple of different things. We want to calculate this risk index by taking into account the volume, the number of interactions, as well as the distance somebody has gone for that interaction because the farther away it is, there’s a higher likelihood of connecting two geographies that would not otherwise be related. – So when we started thinking about this risk index, our research showed that the current social distancing metrics don’t fully consider geography. In many cases, they’re normalized by population, which removes what Joel said describing this volume, it removes that population density from the equation. And also as this metric of distance, the further that someone travels, the riskier that is. So if I go to downtown Portland, that’s sort of one level of risk. If I travel to Sao Paolo, influencing that population that is distinct and unique from my population, that connectivity represents a whole higher level of risk. And most models also don’t even cover this idea of significant group clusters. So we’ve been focused on building this risk index that considers distance and volume of people moving. – So when we talk about these movement risk factors, this idea that there’s two things we want to consider.

Quantifying Risk

Both how far away something has happened, in addition to the volume in which that interaction is happening. What we’ve polled on or looking at to be able to quantify this is human movement data.

More simply, we can call this just simply cell phone tracking data. Everybody has a cell phone and what you can do is get data that tracks the location of these devices. Now the way this occurs, and especially in the case of what we’re using here, Veraset data, this is background application tracking. So when you install, let’s say, a weather application, and it asks you, do you allow the application to track location? This location tracking then is what Veraset is using. Now as you can well imagine, this doesn’t represent everybody. Based on where it’s at, we can get around 8% market penetration. Now this varies a little bit depending on where you’re looking at and what time frame it is. And so, the best that we can call this is a representative sample, but as a representative sample it can convey a tremendous amount of information. And even though we’re looking at less than 10% of market penetration, this is individual records showing where the cell phone location is at and this is a lot of data. Of the magnitude of billions of records per day. So, with this in mind, we wanted to be able to understand where people are going from and where they’re going to, we wanted to be able to put it on a map so that we can understand it, but will billions and billions of records per day, we were trying to do analysis from the beginning of March. You can imagine, hundreds of billions of records. There was no really other way to do it than using Spark in a scaled environment in which case we’re using Databricks to be able to do this. – Okay, so this is essentially the general workload that we took, the approach that we took. The top three items here are powered by Databricks and our data toolkit, which again, sits natively in the cluster. The jar that gets loaded there is similar to our open source engine that powers things like Athena and AWS Athena and Presto but it’s been enhanced to be even more perform inside of data risk. And the bottom three here are powered by RJS. Essentially, we start from raw data, we apply this hexagon based index to generalize the data, we build up the summarized origin destination pairs for hexagon and then we bring that much smaller data set into Esri and we have pinned our demographic data, visualize, and ultimately publish an interactive dashboard. – So when we’re looking at this panel data, the raw data, what we’re looking at, as we were talking about before, is each one of these records is really nothing more than when did it happen, a unique identifier, the location of the data, and then finally, how accurate that location is. Because ultimately, there is a margin of error for how precise you know where a device is, so what this allows us to do is this gives us a starting point. Ultimately though, there’s a lot of work that has to be done to be able to understand the relationship of this data so that we can then ultimately get our index. And since there’s so many records, the first step that we want to do is to be able to understand the data in some sort of generalizable form. What we used is a hexagon index for this. This enables us to be able to group them based on an area that’s roughly the size of 2/3 of a city block in New York City, just to give you a rough idea of what we’re looking at and then from there, what this allows us to do is understand the relationship based on the origin and the destination. And in this case, what we refer to as the origin is where the device, and by proxy, a person resides during the night time hours and then everywhere that they go that is not during the night time hours, this then becomes a trip that they venture to. Specifically, we also examine how fast the device is moving because we don’t want to be looking at people driving down the interstate. Ultimately what we want to do is we want to understand the location of people that is relatively static because that’s when people are at rest and have the potential to be interacting with other humans in a different location other than their home. – Okay, so let’s get to the good stuff. Here’s what’s happening inside of Databricks really and our workflow in order to build up that risk index. Essentially, we take the raw data and we filter it by significant dwells where a device is seen multiple times in a given location. We bend those into these hexagons as Joel said and we’re doing this at level nine which is about a city block. We take those hexagons and we build it up, an origin cell, based on where they sleep and a destination cell and we total those up, so that at each hexagon, we have a cumulative trip the destination paired for all permutations. And from there, we calculate that risk index which is simply the number of trips times distance and finally, we output this much smaller reduced cleaning data set as a process table for use by the GIS and now, let’s take a look at the actual notebook.

So again, here is that multidimensional data in this interactive dashboard and I can see this is Detroit for instance, and I can see areas of risk, I can see contributions over here in terms of which of these different tapestry segments are contributing and I can just bounce around to a couple cities. I’ll go to Boston next. And we see a very different pattern. Each city is unique. Here are some clusters. There’s a large cluster down here which could be a risk. Let’s jump over to New York here. And here we see a story about Manhattan and we see these different contributors, but what’s interesting is let’s take these high rise renters and filter on that. So again, I can just filter again on the dashboard and show just different segments. Up here in the northern part of the city we see these high rise renters and there’s a large cluster up here. I may want to think about who are these people and as a policy maker or decision maker, how might I message to them? So we just look at this chart here or this little infographic shows who those high rise renters are from the segment. So we see median age 32, we see that they are relatively low income, much of which goes to rent. We see many single parents and we can just sort of explore who those people are that are occupying that cell and then as I said, make certain decisions or policies or messaging about how to reduce that risk.

Next Wave High Rise Renters TAPESTRY

So here we are back on the workflow and I guess what I wanna say is this analysis would have not been possible without that combination, magic combination really, or distributed processing on the Databricks front and the enrichment and visualization capabilities on the GIS. That’s really allowed us to take very raw, high volume data, and bring some meaning and understanding to the data and then we’re allowed that to be easily shared in a consumable form inside of a community. And that same work flow though or going from raw data to these visualizations, using the same methodology could be applied to a ton of industries, whether that’s telecom data or looking at movement data as Joel has done for things like retail openings and site selections. It’s really almost like a human weather pattern that you’re analyzing and I gotta say, I love the collaboration capability of Databricks and being able to build these notebooks up.

But if I’m honest, Joel’s the one that spent most of the time in the notebook and so, how was your experience Joel?

– So I think in closing, one of the more important things to emphasize, this is what Jim was alluding to is, and he always likes to laugh at me about this, is that I didn’t get into this because I was that interested in big data, what really interested me was I’m a geographer. But before that, the emphasis I always like to make is that if I can do this, you can do it, and here’s the reason why, is that I have a degree in parks, recreation, and tourism. I discovered geography almost by accident and then I kinda fell sideways into big data analysis because ultimately, what I do the most of, is I’m a geographer. I had a problem I needed to solve. I needed to understand where people were going from, where they were going to, in what magnitude, in a way that we could quantify risk. I was able to frame a geographic problem and I needed to solve it. The only real way to do this really was taking advantage of the scalable architecture, in concert with Esri’s technology to be able to distill this down into something meaningful that we can then take and put in the GIS to be able to add even more context to it. Cause ultimately the name of game is being able to take data and making information out of it and that really starts with being able to first understand the problem. So the emphasis I’m really putting on here is that I was able to do this. I’m not a Databricks expert. I will freely concede that. I started doing this about five weeks ago. And ultimately, I was able to put this all together and get something up and running. I didn’t do it alone. Jim obviously helped me a lot, but this really was an idea that came to fruition because I had a need and then reached out and found the right technologies and ultimately, the people to help me get over the humps to be able to do this. So really, this was the type of thing where the combination of these two is really greater than the sum of the parts because we have a scalable ability with the context of geography to be able to understand this problem in a very meaningful way. So, with that, thank you so much for your time.