Collaborative Data Science
A unified experience to boost data science productivity and agility
Data Scientists face numerous challenges throughout the data science workflow hindering productivity. As organizations continue to become more data-driven, a collaborative environment for easier access and visibility into the data, models trained against the data, reproducibility, and insights uncovered within the data is critical.
The Challenge
Data exploration at scale is difficult and costly
Spending too much time managing infrastructure and DevOps
Need to stitch together various open source libraries and tools for further analytics
Multiple handoffs between data engineering and data science teams are error prone and increase risks
Hard to transition from local to cloud-based development due to complex ML environments and dependencies
The Solution
Quick access to clean and reliable data for downstream analytics
One click access to pre-configured clusters from the data science workspace
Bring your own environment and multi-language support for maximum flexibility
A unified approach to streamline the end-to-end data science workflow from data prep to modelling and insights sharing
Migrate or execute your code remotely on pre-configured and customizable ML clusters
Databricks for Data Science
An open and unified platform to collaboratively run all types of analytics workloads, from data preparationto exploratory analysis and predictive analytics, at scale.
Workspace
One central place to store and share notebooks, experiments, and projects backed with role-based access control.
Collaborative Data Science at Scale
Collaboration across the entire data science workflow, and more
Collaboratively write code in Python, R, Scala, SQL, explore data with interactive visualizations, and discover new insights with Databricks notebooks.
Confidently and securely share code with co-authoring, commenting, automatic versioning, Git integrations, and role-based access controls.
Keep track of all experiments and models in one place, capture knowledge, publish dashboards, and facilitate hand-offs with peers and stakeholders across the entire workflow, from raw data to insights.
Focus on the data science, not the infrastructure
You don’t have to be limited by how much data fits on your laptop anymore, or how much compute is available to you.
Quickly migrate your local environment to the cloud with Conda support, and connect notebooks to auto-managed clusters to scale your analytics workloads as needed.
Use PyCharm, Jupyter Lab or RStudio with scalable compute
We know how busy you are… you probably already have hundreds of projects on your laptop, and are accustomed to a specific toolset.
Connect your favorite IDE to Databricks, so that you can still benefit from limitless data storage and compute. Or simply use RStudio or Jupyter lab directly from within Databricks for a seamless experience.
Get data ready for data science
Clean and catalog all your data in one place with Delta Lake: either batch, streaming, structured or unstructured, and make it discoverable to your entire organization via a centralized data store.
As data comes in, quality checks ensure data is ready for analytics. As data evolves with new data and further transformations, data versioning ensures you can meet compliance needs.
Discover and share new insights
You’ve done all the work and identified new insights with built-in interactive visualizations or any other supported library like matplotlib or ggplot.
Easily share and export results by quickly turning your analysis into a dynamic dashboard. The dashboards are always up to date, and can run interactive queries as well.
Cells, visualizations, or notebooks can also be shared with role-based access control and exported in multiple formats including HTML and IPython Notebook.
Simple access to the latest ML frameworks
Get going fast with one-click access to ready-to-use and optimized Machine Learning environments including the most popular frameworks like scikit-learn, XGBoost, TensorFlow, Keras and more. Or effortlessly migrate and customize ML environments with Conda. Simplified scaling on Databricks helps you go from small to big data effortlessly, so that you don’t have to be limited with how much data fits on your laptop anymore.
The ML Runtime provides built-in AutoML capabilities, including hyperparameter tuning, model search, and more to help accelerate the data science workflow. For example, accelerate training time with built-in optimizations on the most commonly used algorithms and frameworks, including Logistic Regression, Tree-based Models, and GraphFrames.
Automatically track and reproduce results
Automatically track experiments from any framework, and log parameters, results, and code version for each run with managed MLflow.
Securely share, discover, and visualize all experiments across workspaces, projects, or specific notebooks across thousands of runs and multiple contributors.
Compare results with search, sort, filter, and advanced visualizations to help find the best version of your model, and quickly go back to the right version of your code for this specific run.
Operationalize at scale
Schedule notebooks to automatically run data transformations, modelling, and share up to date results.
Set up alerts and quickly access audit logs for easy monitoring and troubleshooting
Saving millions in inventory management
Shell has deployed a data science tool globally to help it manage and optimise the $1 billion in spare part inventory it holds in case something breaks on its assets.
