Databricks has enabled many organizations to harness the power of their data for machine learning and data science. But while Databricks enables collaboration across Data Scientists, Data Engineers, and Analysts,, there must be a paradigm shift to bring domain experts into the process too, to directly make discoveries by manipulating, analyzing and visualizing data with the team. Successfully achieving this requires a rethinking of the classic analytics user interfaces, towards interactive systems with highly collaborative visual interfaces.

Current visualization and workflow tools are ill-suited to bringing the full team together. They were not interactive to support teams to actually work together in real time. Similarly, most machine learning algorithms are not able to provide initial answers at "human speed" (i.e., seconds), and prevent exploration and iteration from happening at the speed of conversation. Finally, most visual data tools still fail when used over large datasets or require horrendous loading times before any real-work can begin.

In this talk, I will present Northstar, a novel system we developed for Interactive Data Exploration at MIT and Brown University and which is now commercialized by einblick analytics, inc. I will explain why Northstar required us to completely rethink the entire analytics stack, from the interface to the “guts” and highlight a few selected techniques we developed to provide a truly novel user-interface (see http://www.einblick.ai/ for a video demonstration) and interactive speeds even over the largest datasets and complex ML operations. This will allow organizations to enable the accessibility of Databricks to a broader audience.