Better financial clarity helps secure investments for the future

Outgrowing Cloudera’s on-premises platform for a holistic view of data

Thrivent’s mission is to help people achieve financial clarity by providing advice, investments, insurance, banking and generosity programs and solutions so clients can make the most of all they’ve been given. In order to guide people on the right financial path, Thrivent advisors need as much information about clients as they can get. Unfortunately, Thrivent’s on-premises system failed to grow with the organization’s desire for a deeper understanding of clients. With the overarching goal of increasing client volume, Thrivent focused on enhancing the customer experience with one-on-one personalization strategies that go beyond the basic insights they were receiving.

Sravan Kasarla, Chief Data Officer at Thrivent, says, “Our vision was to attain a more holistic and personal view of our clients — not just what we’ve sold them, but who they are. What do they say in digital interactions with advisors and contact center employees? What are their offline conversations at properties? Plus, what are they not telling us? We need this information to stay informed of new life events and life stages to make next action recommendations and ultimately provide a better service to every customer.”

Thrivent’s scattered data across the cloud, on-premises warehouses and the Hadoop infrastructure in Cloudera made it impossible to centralize data, increase management efficiency and scale according to capacity needs. Without ACID transactions, data teams had to create cumbersome engineering frameworks that increased latency and prevented real-time streaming. Additionally, to circumvent data availability and collaboration challenges, teams had to copy data into different locations, creating data silos, which made analytics and reporting difficult.

“We needed to migrate to an enterprise-grade cloud ecosystem that would allow us to do multiple things in one place. Machine learning, data governance and having that integrated view of our data were the driving forces behind our migration to Databricks,” explains Satish Bandapati, Director of Data and ML Engineering and Platforms at Thrivent.

Migrating to the lakehouse paves the way for reporting-ready data and AI

With 16 nodes in their Cloudera cluster powering hundreds of jobs across data transformations, streaming, analytics and AI, Thrivent initiated a company-wide migration to the Databricks Data Intelligence Platform on AWS, starting with the data in Cloudera. While Thrivent primarily relied on internal resources (three engineers) for the migration, they also turned to Databricks as subject matter experts for architecture guidance and input on the approach and to consulting partner RCG Global Services to help execute the migration over a period of 11 months.

In preparation for migration, data teams built modifier pipelines and decoupled some existing pipelines for easy data transfer using Apache Spark™. From there, they examined the existing models in Cloudera to see how they could facilitate a seamless and collaborative migration process in the most cost-efficient manner. Bandapati says, “We created some templates instead of trying to migrate data one by one, and allowed the data scientists to complete their own migration. The actual code and all the parameters were configured, so that was easy.”

Thrivent selected the Databricks Data Intelligence Platform and Delta Lake for their built-in ACID transactions, AI and streaming capabilities, long-term scalability, and integration with tools for reporting and deploying models. With the Databricks Data Intelligence Platform, Thrivent is now creating and organizing raw and curated data — providing data users across the organization access to the right level of data they need for analytics, reporting and AI. Using MLflow, Thrivent’s data scientists are able to more effectively collaborate on models and version experiments, and to ultimately build a culture of model and feature sharing to accelerate the delivery of new AI innovations to market. And with Unity Catalog, Thrivent is looking to resolve reporting challenges, streamline categorization and governance, and improve data monitoring.

Data integration improves client experiences and fuels data-based business outcomes

Since migrating their customer analytics workloads to the lakehouse, Thrivent’s data science team has been able to make an impact on the business outcomes as a result of accessible and clean data now being applied via machine learning and reporting capabilities. Using the built-in ACID transactions in Delta Lake, team members are more productive, with fewer processing steps across multiple workflows. And with Delta Live Tables, Thrivent shifted from file-based thinking in Cloudera to table-based thinking, which has expanded collaboration and data availability among team members.

Now Thrivent can predict and recommend the most appropriate products and services based on a client’s financial situation and needs. For example, one prediction model identifies clients most likely to sign up for mobile or digital products, and another model uses past purchases to suggest the next best action step.

Although Thrivent’s motivation for the migration was primarily data-focused, the organization anticipates cost savings as well. Over the next three years, Thrivent projects total cost of ownership (TCO) to decrease by 46% with Databricks Data Intelligence Platform compared to Cloudera.

Thrivent will continue building out their new data ecosystem in the cloud while taking the steps to complete the company-wide data migration to Databricks. Kasarla says, “The Databricks Data Intelligence Platform gives us the single, unified, modern data platform in the cloud. By the end of 2024, we want to be completely cloud-based to get the data access we need for reporting, dashboards, insights and analytics.”