Accelerating Provider MDM in Healthcare with Databricks and AI

Healthcare operations and patient care depends on accurate, complete, and unified data. From ensuring timely claims processing and efficient referral routing to delivering insightful performance analytics and maintaining regulatory compliance, a reliable single source of truth is paramount.

Provider information remains one of the most complex and challenging datasets for healthcare organizations, creating barriers to a single source of truth. Provider data is managed in many disparate sources: Electronic Medical Records (EMRs), the National Plan and Provider Enumeration System (NPPES), claims systems, credentialing databases, external directories, and more. All of these systems represent providers slightly differently and create numerous challenges in interoperability that serve as a barrier to valuable healthcare analytics and insights.

The opportunity with Master Data Management (MDM) to address this challenge

Master Data Management (MDM) solutions tackle these problems by moving data out of source systems and analytical systems, process it, and then move it back. This "move-first" approach introduces significant challenges: complex data pipelines, increased latency, governance hurdles, and substantial infrastructure costs. It's a model that struggles to keep pace with the volume, velocity, and variety of modern healthcare data.

That’s where the Databricks Data Intelligence Platform built on lakehouse architecture can help. By bringing data and processing together, Databricks enables organizations to overcome the limitations of traditional architectures and unlock new possibilities for data management. Leveraging the principle of "data gravity," Databricks enables you to process data where it lives, reducing costly and complex data movement.

To help healthcare organizations accelerate their journey on Databricks and tackle the provider MDM problem we're excited to introduce a product from Frisco Analytics LakeFusion and an accompanying Provider 360 Accelerator. Built natively on Databricks, this AI-powered tool represents a significant step to achieving comprehensive Provider MDM.

The Persistent Challenge of Provider Data

Traditional MDM systems often struggle with the inherent ambiguity and variability in provider data. Plugging in new sources of provider information and permutations of provider representation become increasingly difficult, time-consuming, and costly. Relying solely on exact matches, rigid rules, or fuzzy algorithms like Levenshtein distance (the distance between 2 phrases) can miss many duplicates (e.g., variations in name spelling, address formatting) and requires constant maintenance as data sources change and doesn’t scale to enterprise levels.

Accelerating Provider Data Quality with Databricks and AI

Whether organizations are consuming provider directory information or price transparency from CMS-9115-F mandate, build attribution models for Value Based Care (VBC) initiatives, drive better quality and utilization metrics through a golden provider record, or cleanup internal system representations of provider data, Lakefusion AI-powered entity resolution on Databricks shines. Instead of relying on brittle rules, we can leverage advanced techniques like embedding models and vector search to understand the semantic similarity between provider records. This allows us to identify records that are similar, even if they don't match exactly on traditional identifiers.

LakeFusion's core capabilities include:

• Advanced AI-Powered Entity Resolution: Building upon the concepts of embedding models and vector search, LakeFusion leverages large language models (LLMs) and sophisticated matching algorithms for highly accurate and scalable entity resolution, even for complex provider hierarchies and relationships.
• Robust Data Quality Framework: Profile, cleanse, validate, and monitor data quality using configurable rules and automated processes.
• Configurable Survivorship: Define rules to automatically determine the "golden record" attributes when merging duplicate records from multiple sources.
• Graphical & Intuitive Data Stewardship: Provide data stewards with a user-friendly interface to review potential matches, resolve exceptions, and manage data quality issues.
• Seamless Data Governance Integration: Fully leverages Databricks Unity Catalog for centralized data governance, lineage tracking, access control, and auditing across your mastered data.

The Provider 360 Accelerator is open source and demonstrates this capability in action. Its core function is to apply AI-powered record deduplication to your provider data using Vector Search and cutting-edge embedding models available on the Databricks. The set of open-source notebooks include:

1. Notebook 1 - Duplicate Candidate Generation: Performs the AI-powered fuzzy matching across your data, leveraging Vector Search to find potential duplicates for each record.
2. Notebook 2 - Duplicate Candidate Analysis: Provides analytical insights into the similarity scores of the candidate pairs, helping you understand the extent of duplicates and determine the right confidence thresholds for your data.
3. Notebook 3 - Deduplication Based on Threshold: Applies your chosen thresholds to filter the original data, generating a cleaner dataset by removing likely duplicates.

The challenge of managing complex provider data in healthcare is real, but the solution is within reach. By leveraging the power of Databricks and the latest advancements in AI, organizations can significantly accelerate their journey towards trusted provider data.

For organizations ready to unlock the full potential of a comprehensive, end-to-end Provider MDM solution, LakeFusion MDM, natively built on the Databricks, offers the capabilities needed to master provider data at scale, drive operational excellence, and enable advanced analytics.

Ready to accelerate your Provider MDM journey?

• Explore the open-source Provider 360 Deduplication Accelerator notebooks on GitHub/Databricks Repo
• Learn more about LakeFusion MDM on the Databricks Marketplace
• Connect with Frisco Analytics to discuss your healthcare data challenges and how LakeFusion can help via https://www.lakefusion.ai/ or https://www.friscoanalytics.com/