Product descriptions:
Lumen Technologies, a global digital network services company, modernized its data infrastructure by migrating critical workloads from a legacy on‑premises big data analysis platform to Databricks SQL. The effort focused on two mission‑critical systems—the Call Detail Record Warehouse (CDRW) and Realtime Analysis and Data‑Driven Insights (RADD)—which process terabytes of telecom data daily for billing accuracy, network optimization, and regulatory compliance.
Modernizing legacy systems for scalability and speed
Lumen’s legacy data warehouse was supported by manual processes that made it difficult to scale or adapt to changing business needs. The CDRW and RADD systems were central to the company’s operations, providing data for billing, call routing, and customer analytics. However, the on-premises environment suffered from fragile pipelines, limited data governance, and rigid retention policies that made it challenging to keep pace with growing data volumes. Routine maintenance and troubleshooting consumed a significant amount of engineering time, while hardware costs continued to rise. These challenges limited agility at a time when network operations demanded speed, accuracy, and flexibility. Teams could not easily modify data pipelines or support new use cases without extensive rework, and outages in one system cascaded to others. Lumen needed a modern platform that could process high-throughput network data reliably, support real-time analytics, and ensure governance across sensitive datasets.
As part of its modernization initiative, Lumen evaluated several cloud data warehouse options. A proof of concept with one provider demonstrated strong performance but projected significantly higher costs for large-scale workloads. Databricks SQL offered comparable performance with better cost efficiency, tighter governance, and native integration with Azure and Lumen’s existing Spark-based workflows. After a two-month evaluation involving IT leadership, enterprise architects, and finance teams, Lumen selected Databricks SQL as the strategic platform for its data warehouse modernization. Databricks SQL provided the scalability and simplicity to achieve that goal, combining unified data access with a secure, high-performance environment that supports both engineers and analysts.
Migrating from Cloudera to Databricks SQL
Lumen partnered with Databricks Professional Services to migrate its CDRW and RADD systems to Databricks SQL. The migration involved transferring 133 terabytes of historical data from the legacy vendor to Azure Data Lake Storage. Using Azure Data Box and AzCopy, the team reduced transfer times from months to just days, ensuring minimal disruption to business operations. Once in Azure, the data was processed using Databricks Notebooks and converted into Delta format tables, enabling consistent and reliable updates.
Databricks SQL now serves as the compute engine for both CDRW and RADD, processing approximately 7 gigabytes of telecom data every 10 minutes. Built-in capabilities such as ACID transactions and schema enforcement provide consistency and reliability, while Unity Catalog ensures secure, role-based access for global teams. To optimize performance, Lumen adopted Databricks SQL Serverless, which automatically scales resources based on workload demand. This approach improved job performance and eliminated cluster management overhead. “Serverless Databricks SQL automatically scales to our busiest periods and has reduced compute costs by up to 40 percent. It has become the backbone of our real-time reporting,” says Richard Henning, Data Platform and Architecture, Lumen Technologies.
The migration also introduced automation and quality assurance improvements. Lumen implemented continuous integration and deployment pipelines, enabling developers to update and test ETL workflows with minimal manual effort. Custom validation frameworks were developed to ensure data integrity between the legacy and new environments, providing business stakeholders with confidence in the accuracy and completeness of the migrated datasets. The 9-month migration project, which included contract and administrative tasks, was completed on time and on budget. Databricks Professional Services and the Lumen Technologies team collaborated on every step of the project, including scoping, migration, and risk assurance.
The Databricks and SAP partnership has also enabled further opportunities for Lumen, focusing on zero-copy capabilities for better end-to-end analytical reporting. “The integration of SaaS platforms with the data intelligence platform has been difficult in the past, so this partnership is very exciting for Lumen,” says Eric Mahaffey, Senior Director of Data Governance and Enablement, Lumen Technologies.
Driving cost efficiency and agility across telecom operations
Today, Databricks SQL powers near real-time analytics across Lumen’s operations. Data from the migrated systems fuels dashboards used by engineers, network operators, and finance teams to monitor call volumes, service performance, and billing accuracy. Databricks SQL enables teams to identify issues more quickly, reduce downtime, and respond proactively to customer and network demands. The platform also supports critical reporting for compliance and regulatory oversight, ensuring data remains auditable, secure, and accessible at scale. “Queries that once took hours in our old warehouse now complete in minutes, a 90% improvement. It has completely changed how quickly our teams can respond to network events,” says Joe Laird, Senior Manager of Data Hub Strategies, Lumen Technologies.
By unifying its network data in Databricks SQL, Lumen has unlocked a new level of efficiency and agility. Automated workflows and serverless compute have freed engineers from manual tuning and maintenance, while improved data quality and governance have accelerated decision-making across teams. The result is a more responsive organization that can act on trusted insights in real-time.
Since completing the migration, Lumen has reduced compute costs by 30-40% and significantly accelerated the delivery of insights across its telecom data ecosystem. With a single source of truth on Databricks, the company is now expanding into advanced analytics and AI-driven initiatives, such as predictive maintenance, network optimization, and intelligent capacity planning, to further enhance reliability and customer experience. By retiring legacy infrastructure and adopting a modern lakehouse architecture, Lumen has transformed how data drives its business and positioned the company to innovate faster in an increasingly data-driven network connectivity landscape. “Databricks SQL has become our engine for real-time insight. It powers everything from daily operations to long-term AI initiatives and continues to deliver measurable value to the business,” says Rafi Tzadikario, SVP of Engineering, Lumen Technologies.