Providing critical mental health support to those in crisis

Real-time crisis intervention requires a scalable, secure data infrastructure

Crisis Text Line is a nonprofit organization dedicated to providing free, 24/7, confidential, text-based mental health support and crisis intervention. Crisis Text Line’s live volunteer crisis counselors help texters sort through anything they’re struggling with — from bullying, stress/anxiety, loneliness/isolation, difficult relationships or self-harm to suicidal thoughts. Their mission is to offer support to individuals in need, promoting mental well-being across the U.S. and internationally with affiliates in Canada, the UK and Ireland. This is achieved through a robust crisis care management platform that leverages advanced data and AI technologies. 

The organization addresses the global mental health crisis through tactical and strategic approaches. Tactically, they offer support to individuals in crisis, having pioneered tech-based mental health support for over a decade. Strategically, they aim to use insights gained from their operations to influence change. 

As CTO Matthew Vanderzee noted, “We think about how we take all of the learnings from the knowledge we’re gaining around mental health and crisis response, and see how we can use that to support more people while influencing public policy and system change.”

As Crisis Text Line sought to support a growing user base and expand high-quality crisis interventions, it became clear that a transition from legacy infrastructure to a more advanced and scalable solution was required. 

Principal Product Manager Kristi Lui explained, “At the time, our data landscape was very siloed in terms of record reporting. A lot of business rules and context remained inconsistent or lost in data silos. That meant our downstream operational teams struggled to make more efficient or aware decisions in real time.”

The limitations of their legacy systems included simple queries timing out and multiple points of failure due to batch scripts used for data transformation. This fragmented infrastructure hindered real-time decision-making and efficient operations. To overcome these challenges, Crisis Text Line sought a modern tech stack that allowed for agile development, quick iteration and better collaboration across teams.
 

Get live support to those who need help faster using generative AI

The Databricks Data Intelligence Platform helped solve these challenges, enabling Crisis Text Line to centralize its data and create a single source of truth. 

“Without that single source of truth, it was difficult to build a cohesive data culture across the organization,” Lui highlighted. The siloed nature of the infrastructure led to duplicated efforts across teams. “Given the sensitive nature of our data, we required an ecosystem with strong governance, security and transparency. Databricks allows us to ingest and process data in a much more reliable and secure manner. We’ve dramatically reduced the amount of time required to make critical datasets available for our clinical teams, our analytics groups and future ML work.”

Databricks empowered Crisis Text Line to create a federated data store that facilitated efficient and secure data handling and allowed teams to work independently while maintaining appropriate access controls. Unity Catalog has been instrumental in managing data access at granular levels, ensuring compliance and enabling faster data-driven decisions.  

“Being able to manage access at the table and column level for all these different data profiles has been vital for our teams to move quickly, but with the proper oversight,” Lui said.

Crisis Text Line leverages several components of Databricks to enhance their data and AI workflows. MLflow is used for managing the full model lifecycle, from training to deployment. It tracks experimentation and model versioning, ensuring that the right models are released in the correct capacities across workspaces. Delta Live Tables supports the post-processing pipeline infrastructure. This enables data scientists to define business entities without requiring data engineering support, thus accelerating the iteration and production processes. Notebooks facilitate cross-functional collaboration, active troubleshooting and rapid experimentation, all within a secure and compliant environment. Vanderzee emphasized the importance of these collaboration tools, saying, “It’s accelerated a lot of that research and impact work — the necessary R&D work.”

Additionally, two game-changing generative AI use cases have taken shape. A large language model (LLM)–powered conversation simulator allows new crisis counselors to practice engagement strategies in difficult scenarios, boosting their confidence without real-world repercussions. Meanwhile, a conversation phase classifier under development will help the team level up conversation assessment to maintain high quality and ensure appropriate responses.

Databricks has helped the team securely and efficiently fine-tune and deploy the models used in these workloads. “We created different fine-tuned versions of Llama 2, utilizing synthetic conversation role-plays created by our clinical staff as an initial training set,” Lui explained. 

This method ensures that the models are tailored to Crisis Text Line’s specific needs without compromising the security of sensitive data. Additionally, the conversation simulator has been used by over 50 real volunteers and clinicians to enhance their training and readiness.
 

Onboarding volunteer crisis counselors more confidently and quickly

Over the past year, crisis counselors have facilitated over 1.3 million conversations initiated by texters. Given this volume, Crisis Text Line’s highest priority was to centralize and support all of their reporting needs to properly launch and scale existing and new crisis interventions, using a suite of data product dashboards to monitor program health and quality. 

“Databricks has helped us surface the right data at the right time to the right individual in support of those in need,” Lui said. “And that’s been seamlessly managed from the data lifecycle perspective end to end — from all the standard reporting work that we’re doing to the novel AI application work.”

“Databricks helped us to do this in three ways. First, it allowed us to increase user adoption and ease of use for dashboard artifacts,” Lui said. “Second, it increased our program ROI as our customers and affiliates learned to better trust our operational and analytical expertise. And third, it helped with reducing our infrastructure and engineering overhead.” 

What’s next for the team? It’s simple: exploring more ways to expand their reach and support. Lui explained, “Databricks has accelerated our work by enabling us to focus on the problems that we alone can solve. The Databricks Data Intelligence Platform helps streamline and standardize our data work so that we can more effectively take on mental health challenges.”

In that vein, Senior Principal Research Scientist Elizabeth Olson added, “Databricks has been empowering and has unlocked a lot of flexibility that was previously not available to us.”

Crisis Text Line’s technology currently supports four global organizations. If you or someone you know is struggling, please reach out to:

Crisis Text Line (U.S.) Text HELLO or HOLA to 741741 or via web chat
Kids Help Phone (CA) Text 686868
Mental Health Innovations — Shout (UK) Text 85258
Text About It (IE) Text 50808