Empowering smarter decisions to transform healthcare outcomes
Abacus Insights uses Databricks to enhance health insurance member data management
health insurance members managed
to launch an AI-enabled data management platform

Abacus Insights manages healthcare data for over 23 million U.S. health insurance members, aiming to reduce costs, enhance patient outcomes and improve decision-making by unifying fragmented data. To help healthcare companies overcome challenges with data quality and process inefficiency, Abacus created a secure data infrastructure built on the Databricks Data Intelligence Platform that transformed their internal operations, while accelerating the launch of new proprietary generative AI (GenAI) products to help healthcare payers improve outcomes and generate greater value across their ecosystems.
Challenged by data quality and security issues
Abacus Insights is a healthcare data usability company dedicated to transforming how clients leverage data to drive improved outcomes. The company was founded in 2017 to help healthcare payers deal with “messy” structured and semi-structured data from various sources, such as clinical notes and EMR records. These healthcare payers faced challenges in managing a mix of structured and unstructured data from multiple sources. Additionally, governance and security requirements, including HIPAA compliance, further complicated matters. Before Abacus, bringing this data together — cleansed, organized and curated for usability — was an expensive and highly variable process. “It was not just about the completeness and correctness of the data but also its robustness,” Navdeep Alam, Chief Technology Officer at Abacus Insights, explained. With a mission to break down data silos, Abacus aimed to provide a secure and unified data infrastructure to their customers.
To expand their impact, Abacus Insights also wanted to build upon managing data for healthcare payers with capabilities to deliver insights. The company recognized the power of AI to help their customers extract valuable insights from their data and ultimately improve outcomes. With the recent breakthroughs of GenAI, their customers have been eager to explore the potential of GenAI to improve the member experience. As an example, a member service representative might have to scan through a 500-page document to answer questions — slowing their ability to deliver timely answers. A chatbot interface that could function within a highly regulated environment without exposing sensitive healthcare data online would help healthcare payers make more informed decisions in less time.
Abacus had to balance performance and scalability with cost efficiency when developing an AI solution that would address the needs of their clients. High costs of infrastructure, coupled with the need for a cautious and phased approach that would mitigate risks and ensure data governance, added to the operational challenges. To address these roadblocks, Abacus became a “Built on Databricks” partner, integrating Databricks Mosaic AI tools directly into their platform to securely enhance its AI capabilities.
Streamlining data management for actionable insights
Ready to expand their solutions, Abacus Insights turned to the Databricks Data Intelligence Platform to reimagine their data management approach. Leveraging Delta Lake, a storage layer that brings reliability and performance to data lakes, Abacus gained the ability to manage structured, semi-structured and unstructured data more effectively. By utilizing the medallion architecture, which organizes data into bronze, silver and gold layers, Abacus enabled structured and incremental data processing that enhanced data quality and readiness for analytics. The extract, transform, load (ETL) processes within Delta Lake also allowed Abacus to efficiently prepare and transform data, ensuring consistency and accuracy throughout the data pipeline.
Abacus integrated Databricks tools like Databricks SQL and Vector Search to enhance their AI-enabled solutions. Databricks SQL provided advanced querying capabilities for structured healthcare data, such as claims, patient records and provider information, allowing Abacus to analyze complex datasets accurately and support critical business decisions with real-time insights. Vector Search improved the querying of unstructured data, such as policy documents, clinical guides and internal records, by finding relevant information and delivering precise, context-aware answers to healthcare providers and payers.
To improve usability for customer support representatives and clinicians while maintaining data security, Abacus also adopted a retrieval augmented generation (RAG) approach, which leveraged internal data sources to generate accurate responses through a secure, chatbot interface. This use of GenAI enhanced how information was presented, returning answers in natural language. Altogether, these advancements optimized data querying and analysis, improved decision-making accuracy and reduced errors, leading to better member outcomes and operational improvements. Better yet, these innovations helped Abacus expand their product offerings, unlock new revenue opportunities and strengthen their market position.
Finally, Abacus relied on Unity Catalog for governance, access control and data lineage to ensure that advanced data management and AI deployments remained compliant. Unity Catalog provided comprehensive oversight, including the auditing and logging necessary to meet stringent healthcare standards like HIPAA. Abacus utilized HITRUST and SOC2-compliant implementations within Databricks to maintain the highest levels of data security and privacy, ensuring their platform adhered to industry standards for managing sensitive healthcare information. MLflow, an open source platform for managing the end-to-end machine learning lifecycle, enabled Abacus to track model performance and versioning while streamlining model integration with Abacus’ secure environment. “The Databricks Platform allowed us to deploy AI models like Llama 3.1 within our own environment, without exposing sensitive data. This level of control was essential for maintaining data privacy and meeting healthcare standards,” Alam added. By adopting the Databricks Data Intelligence Platform, Abacus scaled operations, accelerated innovation and delivered compliant AI solutions tailored to the healthcare industry’s needs.
Achieving rapid AI deployment and unmatched data speed
The partnership with Databricks enabled Abacus Insights to move from concept to a beta release of their secure, AI-enabled healthcare data management platform in just four months, showcasing Databricks’ ability to accelerate development and drive rapid innovation. “Databricks allowed us to bring a level of speed and efficiency that we hadn’t been able to achieve before,” Alam concluded. This allowed Abacus to efficiently handle semi-structured clinical and claims data, reducing lag time in data analysis and improving decision-making speed for healthcare providers, healthcare payers and their members.
Databricks also enhanced data usability by improving access to structured data, empowering Abacus to offer new data products that solve real business problems for payers, including compliance with CMS interoperability mandates, optimized risk adjustment, improved quality performance and achievement of value-based care goals. Cost efficiency and scalability are additional benefits of the partnership, as healthcare payers are under tremendous cost pressures while simultaneously trying to leverage ever-increasing amounts of data. Databricks’ scalable architecture helped Abacus optimize operational costs by efficiently utilizing cloud resources and scaling AI workloads according to demand, supporting sustainable growth. The platform’s modular design allowed Abacus to introduce AI capabilities incrementally without disrupting existing operations. Databricks’ capabilities in tracking model performance and versioning updates helped streamline the deployment process, maintaining a high level of control and security over sensitive data.
Looking ahead, Abacus plans to continue leveraging Databricks to drive innovation, with GenAI and machine learning remaining at the core of their evolving product strategy.