YipitData turns to their Data Team to transform financial market information overload into insights


YipitData turns to their Data Team to transform financial market information overload into insights

The Data Team at YipitData is turning financial market information overload into meaningful insights. What’s their secret? Creating data sets at unprecedented scale and speed — from multiple sources that have never been combined before — with Databricks. As volatile markets stump investors everywhere during the COVID-19 global crisis, the YipitData team is delivering fresh insights to help their customers keep up.

Meet the Data Team

In a world turned upside down by COVID-19, hedge fund managers and institutional investors need sound, data-driven insights more than ever to help guide their investment decisions. While data is everywhere, it doesn’t get collected and transformed into insights without help from quick-thinking Data Teams. And with volatile financial markets changing so fast — and the stakes high for so many people’s finances — these insights have to be delivered to investors even faster. That’s where the YipitData data team comes in.

Each month, Khalid, Bill and Anup, along with the rest of the YipitData Data Team, send billions of API requests to collect data from hundreds of websites and use it to build a massive information library they mine for insights. This will be the first time these groupings of data, made up of app downloads, email receipts scraped from the web, credit card transactions data, among other sources, will be unified to create one massive data set, before giving an entire team of great experimental minds access to it. The YipitData Data Team adds value by articulating a story you couldn’t see by looking at one of the data sets alone.

Being able to house all the data in one place on the Databricks platform and integrate it as smoothly as possible has made the Data Team’s job of analyzing it much easier.

“YipitData’s sole mission is to answer investor questions by the data we collect and the data we source. Being able to access data and manage it efficiently is our product,” Khalid, a YipitData data analyst, said.

Bill, also on the data analytics team, spoke to the way the Data Team has been able to set YipitData apart for investors, adding a note about the originality of the work YipitData does.

“We’re constantly trying to come up with new creative ways to combine these data sets in ways that haven’t been done before, and sometimes we’re not sure if they’re even possible. In order to estimate company performance we have to think outside the box a lot.”

It’s a fast-paced environment that merges traditional roles of data science, data engineering and data analytics, allowing Bill, Khalid and Anup to be more productive. Using the Databricks platform, the YipitData Data Team has eradicated the need to ask each other to prep data or walk someone through a file. The democratization of data has mitigated the issue of information silos. Direct access to each other’s work and information shaves off time.

But one of the biggest time savers this team has come to rely on is the ability to make a query in 7 seconds. Before investing in the Databricks platform, their queries took 6 hours. Automated monitoring has impacted results directly as well, allowing the team to spot inflections quickly and publish on them, tripling YipitData’s publishing frequency. That means a whole lot more data-backed information to their investor clients, a whole lot sooner.

Anup mentioned how speed helps the internal business as well as the investors they serve.

“As a business, we are much more focused on profitability and becoming much more efficient with our resource usage. As data engineers, we’re constantly thinking: how do we do this whole process in a very effective and efficient manner? And with Databricks, it’s very much metered usage, so you can really hone in on what is consuming a lot of resources, what is suboptimal…to rethink how we do a lot of the data transformation work that is our product.”

With Databricks, it’s very much metered usage, so you can really hone in on what is consuming a lot of resources, what is suboptimal…

—Anup Segu, Data engineering

However the Data Team chooses to react to their own findings will have ripple effects for the entire company, the investors beyond, and even the markets themselves.

Which means it’s good news that the Data Team is a well-oiled machine, with teammates working from different computers in different locations, out of one notebook simultaneously. During the pandemic, this has been especially important, just as their investment advice has become especially important in the fluctuating markets. It’s the speed, the openness and the collaboration that has allowed them to flourish even in the new world. And with the Databricks platform delivering a 90% reduction in data processing time, YipitData’s Data Team is helping investment institutions use relevant, timely data insights to make informed decisions in a volatile market.