Databricks での ArcGIS GeoAnalytics Engine

 

Advances in big data have enabled organizations across all industries to address critical scientific, societal, and business issues. Big data infrastructure development assists data analysts, engineers, and scientists address the core challenges of working with big data - volume, velocity, veracity, value, and variety. However, processing and analyzing massive geospatial data presents its own set of challenges. Every day, hundreds of exabytes of location-aware data are generated. These data sets contain a wide range of connections and complex relationships between real-world entities, necessitating advanced tooling capable of effectively binding these multifaceted relationships through optimized operations such as spatial and spatiotemporal joins. The numerous geospatial formats that must be ingested, verified and standardized for efficient scaled analysis add to the complexity.

Some of the difficulties of working with geographical data are addressed by the recently announced support for built-in H3 expressions in Databricks. However, there are many geospatial use cases, some of which are more complex or centered on geometry rather than grid indices. Users can work with a range of tools and libraries on the Databricks platform while taking advantage of numerous Lakehouse capabilities.

Esri, the world's leading GIS software vendor, offers a comprehensive set of tools, including ArcGIS Enterprise, ArcGIS Pro, and ArcGIS Online, to solve the aforementioned geo-analytics challenges. Organizations and data practitioners using Databricks need access to tools where they do their day-to-day work outside of the ArcGIS environment. This is why we are excited to announce the first release of ArcGIS GeoAnalytics Engine (hereafter called GA Engine), which allows data scientists, engineers, and analysts to analyze their geospatial data within their existing big data analysis environments. Specifically, this engine is a plugin for Apache Spark™ that extends data frames with very fast spatial processing and analytics, ready to run in Databricks.

Benefits of the ArcGIS GeoAnalytics Engine

Esri's GA Engine allows data scientists to access geoanalytical functions and tools within their Databricks environment. The key features of GA Engine are:

• 120+ spatial SQL functions—Create geometries, test spatial relationships, and more using Python or SQL syntax
• Powerful analysis tools—Run common spatiotemporal and statistical analysis workflows with only a few lines of code
• Automatic spatial indexing—Perform optimized spatial joins and other operations immediately
• Interoperability with common GIS data sources —Load and save data from shapefiles, feature services, and vector tiles
• Cloud-native and Spark-native—Tested and ready to install on Databricks
• Easy to use—Build spatially-enabled big data pipelines with an intuitive Python API that extends PySpark

SQL functions and analysis tools
Currently GA Engine provides 120+ SQL functions and 15+ spatial analysis tools that support advanced spatial and spatiotemporal analysis. Essentially, GA Engine functions extend the Spark SQL API by enabling spatial queries on DataFrame columns. These functions can be called with Python functions or in a PySpark SQL query statement and enable creating geometries, operating on geometries, evaluating spatial relationships, summarizing geometries, and more. In contrast to SQL functions which operate on a row-by-row basis using one or two columns, GA Engine tools are aware of all columns in a DataFrame and use all rows to compute a result if required. These wide arrays of analysis tools enable you to manage, enrich, summarize, or analyze entire datasets.

The GA Engine is a powerful analytical tool. Not to be overshadowed, though, is how easy the GA Engine makes working with common GIS formats. The GA Engine documentation includes multiple tutorials for reading and writing to and from Shapefiles and Feature Services. The ability to process geospatial data using GIS formats provides great interoperability between Databricks and Esri products.

GA engine for different use cases

Let's go over a few use scenarios from various industries to show how the ESRI's GA Engine handles large amounts of spatial data. Support for scalable spatial and spatiotemporal analysis is intended to assist any company in making critical decisions. In three diverse data analytics domains—mobility, consumer transaction, and public service—we will concentrate on revealing geographical insights.

Mobility data analytics

Mobility data is constantly growing and can be divided into two categories: human movement and vehicle movement. Human mobility data collected from smartphone users in mobile phone service areas provide a more in-depth look at human activity patterns. Millions of connected vehicles' movement data provide rich real-time information on directional traffic volumes, traffic flows, average speeds, congestion, and more. These data sets are typically large (billions of records) and complex (hundreds of attributes). These data require spatial and spatiotemporal analysis that goes beyond basic spatial analysis, with immediate access to advanced statistical tools and specialized geoanalytics functions.

Let's start by looking at an example of analyzing human movement based on Cell Analytics™ data from Esri partner Ookla^®. Ookla^® collects big data on global wireless service performance, coverage, and signal measurements based on the Speedtest^® application. The data includes information about the source device, mobile network connectivity, location, and timestamp. In this case, we worked with a subset of data containing approximately 16 billion records. With tools not optimized for parallel operations in Apache Spark(™), reading this high-volume data and enabling it for spatiotemporal operations could incur hours of processing time. Using a single line of code with GeoAnalytics Engine, this data can be ingested from parquet files in a few seconds.

To start deriving actionable insights, we'll dive into the data with a simple question: What is the spatial pattern of mobile devices over the conterminous United States? This will allow us to begin characterizing human presence and activity. The FindHotSpots tool can be used to identify statistically significant spatial clusters of high values (hot spots) and low values (cold spots).

The resulting DataFrame of hot spots was visualized and styled using Matplotlib (Figure 2). It showed many records of device connections (red) compared to locations with low density of connected devices (blue) in the conterminous United States. Unsurprisingly, major urban areas indicated a higher density of connected devices.

Next, we asked, does mobile network signal strength follow a homogeneous pattern across the United States? To answer that, the AggregatePoints tool was used to summarize device observations into hexagonal bins to identify areas with particularly strong and particularly weak cellular service (Figure 3). We used rsrp (reference signal received power) – a value used to measure mobile network signal strength – to calculate the mean statistic over 15km bins. This analysis illuminated that cellular service signal strength is not consistent - instead it tends to be stronger along the major road networks and urban areas.

In addition to plotting the result using st_plotting, we used the arcgis module, published the resulting DataFrame as a feature layer in ArcGIS Online, and created a map-based, interactive visualization.

Now that we understand the broad spatial patterns of mobile devices, how can we gain deeper insight into human activity patterns? Where do people spend time? To answer that, we used FindDwellLocations to look for devices in Denver, CO that spent at least 5 minutes in the same general location on May 31, 2019 (Friday). This analysis can help us understand locations with more prolonged activity, i.e., consumer destinations, and separate these from general travel activity.

The result_dwell data frame provides us with devices or individuals that dwelled at different locations. The dwell duration heatmap in Figure 4 provides an overview about where people spend their time around Denver.

We also wanted to explore the locations people visit for longer durations. To accomplish that, we used Overlay to identify which points-of-interest (POI) footprints from SafeGraph Geometry data intersected with dwell locations (from result_dwell DataFrame) on May 31, 2019. Using groupBy function, we counted the connected device dwell times for each of the top POI categories. Figure 5 highlights that a few urban POIs in Denver coincided with longer dwell times including office supplies, stationery and gift stores, and offices of trade contractors.

This sample analytical workflow with Cell Analytics^TM data could be applied or repurposed to characterize people's activities more specifically. For example, we could utilize the data to gain insights into consumer behavior around retail locations. Which restaurants or coffee shops did these devices or individuals visit after shopping at Walmart or Costco? Additionally, these datasets can be useful for managing pandemics and natural disasters. For example, do people follow public health emergency guidelines during a pandemic? Which urban locations could be the next COVID-19 or wildfire-induced poor air quality hot spots? Do we see disparities in human mobilities and activities due to income inequality at a broader geographic scale?

Transaction data analytics

Aggregated transaction data over points of interest contains rich information about how and when people spend their money at specific locations. The sheer volume and velocity of these data require advanced spatial analytical tools to understand consumer spending behavior clearly: How does consumer behavior differ by geography? What businesses tend to co-locate to be profitable? What merchandise do consumers buy at a physical store (e.g., Walmart) compared to the products they purchase online? Does consumer behavior change during extreme events such as COVID-19?

These questions can be answered using SafeGraph Spend data and GeoAnalytics Engine. For instance, we wanted to identify how people's travel patterns were impacted during COVID-19 in the United States. To accomplish that, we analyzed nationwide SafeGraph Spend data from 2020 and 2021. Below, we show yearly spend (USD) by consumers for enterprise rental cars, aggregated to U.S. counties. After publishing the DataFrame to ArcGIS Online, we created an interactive map using the Swipe widget from ArcGIS Web AppBuilder to quickly explore which counties showed change over time (Figure 6).

次に、年間でオンライン支出が最も多かった米国郡と、人口や農産物販売パターンの類似性を考慮した類似のオンラインショッピング支出パターンを持つ他の郡を調査しました。支出DataFrameの属性フィルタリングに基づいて、2020年のオンラインショッピング支出でサクラメントがトップであることがわかりました。類似地域を調べるために、FindSimilarLocationsツールを使用して、オンラインショッピングと支出に関してサクラメントに最も類似または類似しない郡を特定しました。これは、人口と農業（耕作地の総面積と農産物の平均販売額）の類似性との相対的なものです（図7）。

公共サービスデータ分析

311通報記録などの公共サービスデータセットには、住民に提供される緊急でないサービスに関する貴重な情報が含まれています。このデータの時空間パターンのタイムリーな監視と特定は、地方自治体が効率的な311通報解決のためのリソースを計画および割り当てるのに役立ちます。

この例では、ニューヨーク市の約2700万件の311サービスリクエスト（2010年から2022年2月まで）を迅速に読み込み、処理/クリーニングし、フィルタリングして、ニューヨーク市エリアに関する次の質問に答えることを目標としました。

• 平均311応答時間が最も長いエリアはどこですか？
• 平均応答時間が長い苦情の種類にパターンはありますか？

最初の質問に答えるために、最も応答時間が長い通報が特定されました。次に、データは、平均期間プラス3標準偏差より長い期間のレコードを含めるようにフィルタリングされました。

2番目の質問である重要な苦情グループを見つけるために、GroupByProximityツールを利用して、500フィートおよび5日以内にある同じタイプの苦情を探しました。次に、10件を超えるグループをフィルタリングし、各苦情グループの凸包を作成しました。これは、それらの空間パターンを視覚化するのに役立ちます（図8）。ArcGIS GeoAnalytics Engineに含まれる軽量なプロットメソッドであるst.plot()を使用すると、DataFrameに格納されているジオメトリを即座に表示できます。

このマップにより、ニューヨーク市のさまざまな苦情タイプの空間分布を簡単に特定できました。たとえば、マンハッタン中南部にはかなりの数の騒音苦情がありましたが、ブルックリンとクイーンズでは歩道の状態が主な懸念事項でした。これらの迅速なデータに基づいた洞察は、意思決定者が実行可能な対策を開始するのに役立ちます。

ベンチマーク
パフォーマンスは、多くの顧客が分析ソリューションを選択する際に決定的な要因となります。Esri のベンチマーク テストでは、ビッグ データ空間分析を実行する際に、オープンソース パッケージと比較して GA Engine が大幅に優れたパフォーマンスを提供することが示されています。データ サイズが増加するにつれてパフォーマンスの向上も増加するため、ユーザーはより大きなデータセットでさらに優れたパフォーマンスを確認できます。たとえば、下の表は、最大数百万のデータ レコードを持つさまざまなサイズの 2 つの入力データセット (ポイントとポリゴン) を結合する空間交差タスクの計算時間を示しています。各結合シナリオは、単一およびマルチ マシンの Databricks クラスターでテストされました。

アーキテクチャとインストール
最後に、GeoAnalytics Engine のアーキテクチャの内部を覗いて、その仕組みを探ってみましょう。クラウドネイティブで Spark ネイティブであるため、クラウドベースの Spark 環境で GeoAnalytics ライブラリを簡単に使用できます。Databricks 環境に GeoAnalytics Engine をデプロイするには、最小限の設定が必要です。JAR ファイルを介してモジュールをロードすると、クラスターによって提供されるリソースを使用して実行されます。

インストールには 2 つの基本的な手順があり、AWS、Azure、GCP で共通です。

1. ワークスペースの準備
   • Databricks ワークスペースを作成または起動する
   • GeoAnalytics JAR ファイルを DBFS にアップロードする
   • 初期化スクリプトを追加して有効にする
2. クラスターの作成

インストール後、ユーザーは Spark 環境にアタッチされた Python ノートブックを使用して分析します。Databricks Lakehouse Platform データにすぐにアクセスし、分析を実行できます。分析後、結果をデータ レイク、SQL Warehouse、BI (Business Intelligence) サービス、または ArcGIS に書き戻すことで永続化できます。

今後の展望

このブログでは、Databricks 上の ArcGIS GeoAnalytics Engine のパワーを紹介し、最も困難な地理空間ユースケースをどのように協力して解決できるかを示しました。上記で示した例の詳細な参照については、この Databricks Notebook を参照してください。今後、GeoAnalytics Engine は、GeoJSON エクスポート、H3 ビニング サポート、K 最近傍法などのクラスタリング アルゴリズムを含む追加機能で強化される予定です。

GeoAnalytics Engine は、Azure、AWS、および GCP 上の Databricks で動作します。お好みの Databricks 環境に GeoAnalytics ライブラリをデプロイする詳細については、Databricks および Esri のアカウント チームにお問い合わせください。GeoAnalytics Engine について詳しく知り、この強力な製品へのアクセス方法を調べるには、Esri の ウェブサイト をご覧ください。

(このブログ記事はAI翻訳ツールを使用して翻訳されています) 原文記事