地理空間データプロダクトの構築

Geospatial data has been driving innovation for centuries, through use of maps, cartography and more recently through digital content. For example, the oldest map has been found etched in a piece of mammoth tusk and dates approximately 25000 BC. This makes geospatial data one of the oldest data sources used by society to make decisions. A more recent example, labeled as the birth of spatial analysis, is that of Charles Picquet in 1832 who used geospatial data to analyze Cholera outbreaks in Paris, a couple of decades later John Snow in 1854 followed the same approach for Cholera outbreaks in London. These two individuals used geospatial data to solve one of the toughest problems of their times and in effect save countless lives. Fast tracking to the 20th century, the concept of Geographic Information Systems (GIS) was first introduced in 1967 in Ottawa, Canada by the Department of Forestry and Rural Development.

Today we are in the midst of the cloud computing industry revolution - supercomputing scale available to any organization, virtually infinitely scalable for both storage and compute. Concepts like data mesh and data marketplace are emerging within the data community to address questions like platform federation and interoperability. How can we adopt these concepts to geospatial data, spatial analysis and GIS systems? By adopting the concept of data products and approaching the design of geospatial data as a product.

In this blog we will provide a point of view on how to design scalable geospatial data products that are modern and robust. We will discuss how Databricks Lakehouse Platform can be used to unlock the full potential of geospatial products that are one of the most valuable assets in solving the toughest problems of today and the future.

What is a data product? And how to design one?

The most broad and the most concise definition of a "data product" was coined by DJ Patil (the first U.S. Chief Data Scientist) in Data Jujitsu: The Art of Turning Data into Product: "a product that facilitates an end goal through the use of data". The complexity of this definition (as admitted by Patil himself) is needed to encapsulate the breadth of possible products, to include dashboards, reports, excel spreadsheets, and even csv extracts shared via emails. You might notice that the examples provided deteriorate rapidly in quality, robustness and governance.

What are the concepts that differentiate a successful product versus an unsuccessful one? Is it the packaging? Is it the content? Is it the quality of the content? Or is it only the product adoption in the market? Forbes defines the 10 must haves of a successful product. A good framework to summarize this is through the value pyramid.

The value pyramid provides a priority on each aspect of the product. Not every value question we ask about the product carries the same amount of weight. If the output is not useful none of the other aspects matter - the output isn't really a product but becomes more of a data pollutant to the pool of useful results. Likewise, scalability only matters after simplicity and explainability are addressed.

How does the value pyramid relate to the data products? Each data output, in order to be a data product:

• Should have clear usefulness. The amount of the data society is generating is rivaled only by the amount of data pollutants we are generating. These are outputs lacking clear value and use, much less a strategy for what to do with them.
• Should be explainable. With the emergence of AI/ML, explainability has become even more important for data driven decision making. Data is as good as the metadata describing it. Think of it in terms of food - taste does matter, but a more important factor is the nutritional value of ingredients.
• Should be simple. An example of product misuse is using a fork to eat cereal instead of using a spoon. Furthermore, simplicity is essential but not sufficient, beyond simplicity the products should be intuitive. Whenever possible both intended and unintended uses of the data should be obvious.
• Should be scalable. Data is one of the few resources that grows with use. The more data you process the more data you have. If both inputs and outputs of the system are unbounded and ever-growing then the system has to be scalable in compute power, storage capacity and compute expressive power. Cloud data platforms like Databricks are in a unique position to answer for all of the three aspects.
• Should generate habits. In the data domain we are not concerned with customer retention as is the case for the retail products. However, the value of habit generation is obvious if applied to best practices. The systems and data outputs should exhibit the best practices and promote them - it should be easier to use the data and the system in the intended way than the opposite.

The geospatial data should adhere to all the aforementioned aspects, any data products should. On top of this tall order, geospatial data has some specific needs.

Geospatial data standards

Geospatial data standards are used to ensure that geographic data is collected, organized, and shared in a consistent and reliable way. These standards can include guidelines for things like data formatting, coordinate systems, map projections, and metadata. Adhering to standards makes it easier to share data between different organizations, allowing for greater collaboration and broader access to geographic information.

The Geospatial Commision (UK Government) has defined the UK Geospatial Data Standards Register as a central repository for data standards to be applied in the case of geospatial data. Furthermore, the mission of this registry is to:

• "Ensure UK geospatial data is more consistent and coherent and usable across a wider range of systems." - These concepts are a call out for the importance of explainability, usefulness and habit generation (possibly other aspects of the value pyramid).
• "Empower the UK geospatial community to become more engaged with the relevant standards and standards bodies." - Habit generation within the community is as important as the robust and critical design on the standard. If not adopted standards are useless.
• "Advocate the understanding and use of geospatial data standards within other sectors of government." - Value pyramid applies to the standards as well - concepts like ease of adherence (usefulness/simplicity), purpose of the standard (explainability/usefulness), adoption (habit generation) are critical for the value generation of a standard.

A critical tool for achieving the data standards mission is the FAIR data principles:

• Findable - The first step in (re)using data is to find them. Metadata and data should be easy to find for both humans and computers. Machine-readable metadata are essential for automatic discovery of datasets and services.
• Accessible - Once the user finds the required data, she/he/they need to know how they can be accessed, possibly including authentication and authorisation.

私たちは、FAIR原則が、信頼できるスケーラブルなデータ製品の設計にとって非常に重要であるという信念を共有しています。公平であるために、FAIRは常識に基づいています。では、なぜそれが私たちの考慮事項の鍵となるのでしょうか？ 「私がFAIRに見ているものは、それ自体新しいものではありませんが、それがうまくやっているのは、データの改善に対する全体的なアプローチの必要性を、アクセスしやすい方法で明確にすることです。このコミュニケーションの容易さが、FAIRが地理空間コミュニティだけでなく、データ改善のアンブレラとしてますます広く使用されている理由です。」 - A FAIR wind sets our course for data improvement。

このアプローチをさらにサポートするために、Federal Geographic Data Committeeは、2021年から2024年を対象とし、2020年11月に承認されたNational Spatial Data Infrastructure (NSDI) Strategic Planを開発しました。NSDIの目標は、本質的にFAIR原則であり、共通の標準に従って組織間を流れるデータ製品であるデータの循環経済を促進するシステムを設計するという同じメッセージを伝え、データサプライチェーンの各ステップで新しい価値と新しい機会を解き放ちます。これらの原則がさまざまな管轄区域に浸透し、さまざまな規制当局に採用されているという事実は、アプローチの堅牢性と健全性の証です。

FAIRの概念は、データ製品の設計と非常によく組み合わさっています。実際、FAIRは製品価値ピラミッド全体を横断し、価値サイクルを形成します。価値ピラミッドとFAIR原則の両方を採用することにより、内部と外部の両方の視点を持つデータ製品を設計します。これは、データの蓄積ではなく、データの再利用を促進します。

なぜFAIR原則が地理空間データと地理空間データ製品にとって重要なのでしょうか？FAIRは地理空間データを超越し、実際にはデータを超越しています。それは、何にでも適用できる良い設計のための、シンプルでありながら一貫した原則のシステムであり、その良い設計は地理空間データや地理空間システムを含む何にでも適用できます。

Grid index systems

従来のGISソリューションでは、空間操作のパフォーマンスは通常、ツリー構造（KDツリー、ボールツリー、Quadツリーなど）を構築することによって達成されます。ツリーアプローチの問題は、最終的にスケーラビリティの原則を破ることです。データが大きすぎてツリーを構築するために処理できず、ツリーを構築するために必要な計算が長すぎて目的を損なう場合です。これはデータのアクセシビリティにも悪影響を与えます。ツリーを構築できない場合、完全なデータにアクセスできず、結果を再現できません。この場合、グリッドインデックスシステムがソリューションを提供します。

グリッドインデックスシステムは、地理空間データのスケーラビリティの側面を念頭に置いて最初から構築されています。ツリーを構築する代わりに、関心のある領域をカバーする一連のグリッドを定義します。H3（Uberが先駆者）の場合、グリッドは地球の表面をカバーします。ローカルグリッドインデックスシステム（例：British National Grid）の場合、関心のある特定の領域のみをカバーする場合があります。これらのグリッドは、一意の識別子を持つセルで構成されています。場所とグリッド内のセルとの間には数学的な関係があります。これにより、グリッドインデックスシステムは非常にスケーラブルで並列な性質を持ちます。

グリッドインデックスシステムのもう1つの重要な側面は、オープンソースであるため、インデックス値をデータプロデューサーとコンシューマーの両方が普遍的に活用できることです。データは、データサプライチェーンを通過するどの段階でもグリッドインデックス情報で強化できます。これにより、グリッドインデックスシステムはコミュニティ主導のデータ標準の例となります。コミュニティ主導のデータ標準は、本質的に強制を必要としないため、価値ピラミッドの習慣生成の側面を完全に遵守し、FAIRの相互運用性とアクセシビリティの原則を意味のある方法で満たします。

Databricksは最近、同じ価値提案に従って、H3グリッドシステムネイティブサポートを発表しました。コミュニティによって推進されている共通の業界標準を採用することが、習慣生成と相互運用性を適切に推進する唯一の方法です。この声明を強化するために、CARTO、ESRI、Googleなどの組織は、スケーラブルなGISシステム設計のためにグリッドインデックスシステムの利用を推進してきました。さらに、Databricks LabsプロジェクトMosaicは、英国政府で広く使用されている標準グリッドインデックスシステムとしてBritish National Gridをサポートしています。グリッドインデックスシステムは、地理空間データ処理のスケーラビリティと、複雑な問題（例：図5-H3を使用したフライト保持パターンの例）のソリューションを適切に設計するために不可欠です。

Geospatial data diversity

地理空間データ標準は、データフォーマットの標準化にかなりの労力を費やしており、フォーマットは相互運用性と再現性に関して最も重要な考慮事項の1つです。さらに、データの読み取りが複雑な場合、単純さについて話すことはできますか？残念ながら、地理空間データのフォーマットは通常複雑です。データは、オープンソースとベンダー固有のフォーマットの両方を含む多数のフォーマットで生成される可能性があるためです。ベクトルデータのみを考慮すると、データはWKT、WKB、GeoJSON、Web CSV、CSV、Shape File、GeoPackageなど、さまざまな形式で提供されることが予想されます。一方、ラスタデータを考慮する場合、GeoTiff、netCDF、GRIB、GeoDatabaseなどの多数のフォーマットでデータが提供されることが予想されます。フォーマットの包括的なリストについては、このブログを参照してください。

Geospatial data domain is so diverse and has organically grown over the years around the use cases it was addressing. Unification of such a diverse ecosystem is a massive challenge. A recent effort by the Open Geospatial Consortium (OGC) to standardize to Apache Parquet and its geospatial schema specification GeoParquet is a step in the right direction. Simplicity is one of the key aspects of designing a good scalable and robust product - unification leads to simplicity and addresses one of the main sources of friction in the ecosystem - the data ingestion. Standardizing to GeoParquet brings a lot of value that addresses all of the aspects of FAIR data and value pyramid.

Why introduce another format into an already complex ecosystem? GeoParquet isn't a new format - it is a schema specification for Apache Parquet format that is already widely adopted and used by the industry and the community. Parquet as the base format supports binary columns and allows for storage of arbitrary data payload, at the same time the format supports structured data columns that can store metadata together with the data payload. This makes it a choice that promotes interoperability and reproducibility. Finally, Delta Lake format has been built on top of parquet and brings ACID properties to the table. ACID properties of a format are crucial for reproducibility and for trusted outputs. In addition, delta is the format used by scalable data sharing solution Delta Sharing. Delta sharing enables enterprise scale data sharing between any public cloud using Databricks (DIY options for private cloud are available using open source building blocks). Delta sharing completely abstracts the need for custom built Rest APIs for exposing data to other 3rd parties. Any data asset stored in delta (using GeoParquet schema) automatically becomes a data product that can be exposed to external parties in a controlled and governed manner. Delta sharing has been built from the ground up with security best practices in mind.

Circular data economy

Borrowing the concepts from the sustainability domain, we can define a circular data economy as a system in which data is collected, shared, and used in a way that maximizes its value while minimizing waste and negative impacts, such as unnecessary compute time, untrustworthy insights, or biased actions based data pollutants. Reusability is the key concept in this consideration, how can we minimize the "reinvention of the wheel". There are countless data assets out in the wild that represent the same area, same concepts with just ever slight alterations to better match a specific use case. Is this due to the actual optimizations or due to the fact it was easier to create a new copy of the assets than to reuse the existing ones? Or was it too hard to find the existing data assets, or maybe it was too complex to define data access patterns.

Data asset duplication has many negative aspects in both FAIR considerations and data value pyramid considerations - having many disparate similar (but different) data assets that represent the same area and same concepts can deteriorate simplicity considerations of the data domain - it becomes hard to identify the data asset we actually can trust. It can also have very negative implications towards habit generation, many niche communities will emerge that will standardize to themselves ignoring the best practices of the wider ecosystem, or worse yet they will not standardize at all.

In a circular data economy, data is treated as a valuable resource that can be used to create new products and services, as well as improving existing ones. This approach encourages the reuse and recycling of data, rather than treating it as a disposable commodity. Once again, we are using the sustainability analogy in a literal sense - we argue that this is the correct way of approaching the problem. Data pollutants are a real challenge for organizations both internally and externally. An article by The Guardian states that less than 1% of collected data is actually analyzed. There is too much data duplication, the majority of data is hard to access and deriving actual value is too cumbersome. Circular data economy promotes best practices and reusability of existing data assets allowing for a more consistent interpretation and insights across the wider data ecosystem.

Interoperability is a key component of FAIR data principles, and from interoperability a question of circularity comes to mind. How can we design an ecosystem that maximizes data utilization and data reuse? Once again, FAIR together with the value pyramid holds answers. Findability of the data is key to the data reuse and to solving for data pollution. With data assets that can be discovered easily we can avoid the recreation of same data assets in multiple places with just slight alteration, instead we gain a coherent data ecosystem with data that can be easily combined and reused. Databricks has recently announced the Databricks Marketplace. The idea behind the marketplace is in line with the original definition of data product by DJ Patel. The marketplace will support sharing of datasets, notebooks, dashboards, and machine learning models. The critical building block for such a marketplace is the concept of delta sharing - the scalable, flexible and robust channel for sharing any data - geospatial data included.

Designing scalable data products that will live in the Marketplace is crucial. In order to maximize the value add of each data product one should strongly consider FAIR principles and the product value pyramid. Without these guiding principles we will only increase the issues that are already present in the current systems. Each data product should solve a unique problem and should solve it in a simple, reproducible and robust way.

You can read more on how Databricks Lakehouse Platform can help you accelerate time to value from your data products in the eBook- A New Approach to data sharing.

(このブログ記事はAI翻訳ツールを使用して翻訳されています) 原文記事