Azure Databricks
Azure Databricks runs the Databricks Data + AI Platform on Azure: an Apache Spark lakehouse with Delta Lake tables, Unity Catalog governance, SQL warehouses, MLflow and model serving. It suits teams that need Spark-scale data engineering, machine learning and AI agents over their own cloud storage.1
- Intended for
- Data engineers, Data scientists and ML engineers, SQL analysts, Data platform administrators
- Runs on
- Azure regions (serverless compute in the Azure Databricks account), Classic compute in your Azure subscription and virtual network
At a glance
- What is it?
- Azure Databricks is a unified, open analytics platform for building and running data, analytics and AI solutions on Azure. It is built around a lakehouse that combines data lake storage with warehouse features, and it ships managed versions of open source projects first created by Databricks employees: Apache Spark and Structured Streaming, Delta Lake, MLflow, Unity Catalog, OpenSharing and Redash. Microsoft's Azure page describes it as an Apache Spark-based data and AI platform optimized for Azure, with Delta Lake as the default table format, the Photon query engine and serverless compute.12
- What does it do?
- Teams ingest data with Auto Loader and Lakeflow pipelines, transform it in notebooks (Python, SQL, Scala, R), schedule work with Jobs and deploy with Declarative Automation Bundles and Git folders. Analysts query through SQL warehouses, build AI/BI dashboards and ask questions in natural language with Genie Agents. Data scientists use Databricks Runtime for Machine Learning, MLflow and Model Serving, and AI functions let SQL users call LLMs from pipelines. Unity Catalog governs tables, models and functions, and Lakebase adds a managed Postgres database for transactional workloads.1
- Who needs it?
- Data engineering, data science and ML teams that process very large datasets on Spark, want one lakehouse for engineering, feature work and model development, and need production MLOps with managed serving endpoints. Microsoft's architecture guide also points to Azure Databricks when governance of data, features, models and functions in Unity Catalog is a requirement.3
- What does it need?4
- An Azure subscription
- An Azure Databricks workspace (classic or serverless) in a supported region
- Cloud storage for your data, or the default storage of a serverless workspace
- A Unity Catalog metastore attached to the workspace
- For classic compute: a virtual network in your subscription for the compute plane
- What it is not
- Azure Databricks is not a database or a storage system; it runs analytics over cloud storage. It is not Microsoft Fabric: Fabric is SaaS licensed by capacity, while Databricks bills Databricks Units, plus Azure VMs, storage and networking for classic compute. Microsoft says it is not the right fit when a team does not need Spark-scale processing, or when a scenario only calls AI models through APIs, where Foundry Tools or Microsoft Foundry are more direct.23
Availability and licensing. Generally available. Billing is usage based: Databricks Units (DBUs), a normalized hourly unit of processing capability billed per second, plus Azure VM, storage and networking costs for classic compute. One- or three-year pre-purchase plans of Databricks commit units discount DBU usage only. Microsoft's Azure product page shows no list prices for Azure Databricks (checked 26 Oct 2026).2
The problem it solves
Large-scale data engineering and machine learning need distributed compute, a reliable table format over low-cost object storage, and governance that covers both data and models. Assembling Spark, Delta Lake and MLflow on raw virtual machines takes ongoing operations work.
Azure Databricks delivers that stack as a managed service: Databricks runs the control plane, compute runs either serverless or in your own Azure subscription, and Unity Catalog applies one permission model across workspaces.4
How it works
An Azure Databricks account holds workspaces and Unity Catalog metastores, and manages identity, billing and policies. Each workspace runs on two planes:
- Control plane: backend services and the web application, run by Databricks in the Azure Databricks account.
- Compute plane: serverless compute runs in the Azure Databricks account in the workspace's region; classic compute runs in a virtual network in your Azure subscription.
Classic workspaces (shown as Hybrid in the Azure portal) keep a workspace storage account in your subscription; serverless workspaces use managed default storage and can connect to your own storage. Unity Catalog organizes data as catalog.schema.object, and one metastore can serve several workspaces in the same region.4
Diagram as a list
Copilot & business apps
- Notebooks, SQL editor, AI/BI dashboards and GenieHow engineers, analysts and business users work with the platformConnects to Control plane
- Microsoft Fabric and Power BIRead lakehouse data through metadata mirroring or the Power BI connectorConnects to Unity Catalog metastore
AI models & platform
- Model Serving and AI GatewayReal-time model endpoints and governed model accessConnects to Unity Catalog metastore
Data & analytics
- Cloud storage with Delta Lake tablesHolds the lakehouse data
Security, identity & governance
- Unity Catalog metastorePermissions, lineage and the catalog.schema.object namespaceConnects to Cloud storage with Delta Lake tables
Cloud & hybrid infrastructure
- Control planeWeb application and backend services run by Databricks
- Serverless compute planeCompute in the Azure Databricks account, in the workspace region
- Classic compute planeClusters in a virtual network in your Azure subscription
Capabilities
Lakehouse on Spark and Delta Lake13
Combines data lake and warehouse features over Delta tables, with Spark for batch processing and Structured Streaming for incremental and streaming data.
Why it matters: Engineers, analysts and data scientists work from one copy of the data.
Limits: Spark cluster overhead may not pay off for small or non-Spark workloads.
Unity Catalog governance14
One governance model for tables, volumes, features, models and functions, with access control lists managed in the UI or SQL, lineage, and managed OpenSharing for external sharing.
Why it matters: Access rules and lineage apply across workspaces instead of per cluster.
Limits: A metastore can only be linked to workspaces in the same region.
Machine learning and Model Serving3
Databricks Runtime for Machine Learning (including Hugging Face Transformers), MLflow tracking, models in Unity Catalog and managed real-time Model Serving with autoscaling and traffic controls. AI Gateway governs access to served and external models.
Why it matters: Covers the model lifecycle from feature work to production endpoints on one platform.
Limits: Spark-based data preparation and training costs remain significant; teams needing Azure Machine Learning endpoint standards can add its managed endpoints.
SQL warehouses, AI/BI dashboards and Genie15
SQL warehouses for analysts, metric views in Unity Catalog as a shared semantic layer, AI/BI dashboards and Genie Agents for natural language questions.
Why it matters: Business users can query lakehouse data without managing clusters.
Limits: Microsoft removed the Metric View BI Compatibility Mode from the Power BI connector; metric views now need native queries in DirectQuery mode.
Lakeflow pipelines, Auto Loader and Jobs1
Auto Loader loads files incrementally from object storage, Lakeflow pipelines manage dependencies and scaling, and Jobs schedule notebooks, SQL and other code.
Why it matters: Reduces hand-written orchestration for production ETL.
Limits: Pipelines are defined in Databricks-specific tooling, which adds platform coupling.
Lakebase Postgres1
A fully managed Postgres OLTP database integrated with the platform and stored in Azure Databricks-managed storage.
Why it matters: Applications can serve transactional data next to the lakehouse.
Limits: It is a Databricks service, separate from Azure Database for PostgreSQL.
Fabric and Power BI interoperability56
Microsoft Fabric mirrors Unity Catalog metadata and reads the data through OneLake shortcuts; Power BI Desktop connects to clusters and SQL warehouses, and data can be published to the Power BI service.
Why it matters: Microsoft 365 users can work with lakehouse data without copying it.
Limits: Metadata mirroring depends on source access time and shortcut performance rather than replication.
Practical use cases
Data engineers process large daily event volumes and need reliable tables for analytics and ML.
- Approach
- Load files with Auto Loader into Delta tables, transform them with Lakeflow pipelines and govern them in Unity Catalog.
- Role of Azure Databricks
- Azure Databricks provides ingestion, processing and governance on one lakehouse.
- Data, infrastructure and skills
- A workspace, cloud storage and a Unity Catalog metastore.
- Type of benefit
- Reliability at scale
- Caveats
- Size compute deliberately; Spark costs grow with data volume.
- First step
- Point Auto Loader at one landing folder and build one Delta table from it.
Data scientists need distributed training and a governed path to production models.
- Approach
- Train with Databricks Runtime for Machine Learning, track runs in MLflow, register models in Unity Catalog and deploy with Model Serving.
- Role of Azure Databricks
- Azure Databricks covers training, registry and serving.
- Data, infrastructure and skills
- ML runtime clusters or serverless compute and Unity Catalog.
- Type of benefit
- Faster path to production
- Caveats
- Add Azure Machine Learning managed endpoints if your organization standardizes on them.
- First step
- Log one existing training notebook to MLflow and register the resulting model.
Business users who live in Microsoft 365 need lakehouse data in their reports.
- Approach
- Mirror Unity Catalog metadata into Microsoft Fabric, or connect Power BI Desktop to a Databricks SQL warehouse.
- Role of Azure Databricks
- Azure Databricks stays the system of record; Fabric and Power BI read from it.
- Data, infrastructure and skills
- A SQL warehouse or Unity Catalog access for Fabric, and Power BI licenses.
- Type of benefit
- Broader data access
- Caveats
- Check which Unity Catalog features the Power BI connector supports.
- First step
- Create a mirrored Azure Databricks catalog in a Fabric workspace for one schema.
Who uses it
Novo Nordisk A/S
ScalingNovo Nordisk: a governed reasoning agent on clinical trial data
50+ now, up from 5 to 10 (measured)
Life sciences, Denmark
Premier League
In productionPremier League: an AI fan companion built on Microsoft Foundry
up approximately 20% (measured)
Media and entertainment, United Kingdom (global audience)
Air India
In productionAir India: AI.g customer service agent on Azure OpenAI
about 40,000 (2026), up from an average of 30,000 (2024) (measured)
Travel and hospitality, India
Works with
Optional integration
- Power BIPower BI connects to Databricks clusters and SQL warehouses.
- Microsoft FabricFabric mirrors Unity Catalog metadata and reads data through shortcuts.
Complementary tools
- Azure Machine LearningMicrosoft suggests Azure Machine Learning managed endpoints alongside Databricks for cross-team endpoint standards.
Alternative approaches
- Microsoft FabricMicrosoft's guide presents Fabric and Azure Databricks as alternative starting points for analytics-integrated ML.
- Azure Machine LearningFor lakehouse ML with MLflow and Unity Catalog, Microsoft points to Azure Databricks.
Optional integration for
- Microsoft FabricFabric mirrors Azure Databricks Unity Catalog metadata into OneLake through shortcuts.
- Power BIPower BI Desktop connects to Databricks SQL warehouses, and data can be published from Databricks to the Power BI service.
Relationship labels follow Microsoft's documentation. "Alternative approaches" does not mean one is better: each profile says when it fits.
Compared with
Getting started
Create a workspace
Create an Azure Databricks workspace from the Azure portal and choose serverless or classic (Hybrid) workspace type.
Check: The workspace opens and shows its region and workspace type.
Confirm Unity Catalog
Check that a metastore is attached; workspaces enabled automatically include a default workspace catalog.
Check: You can create a table in the default schema of the workspace catalog.
Ingest data
Use Auto Loader or a file upload to create a Delta table from sample data.
Check: The table is visible in Catalog Explorer with its schema.
Query and visualize
Start a SQL warehouse, query the table in the SQL editor and build an AI/BI dashboard.
Check: The dashboard refreshes against the warehouse.
Connect Microsoft tools
Connect Power BI Desktop to the SQL warehouse, or mirror the catalog into Microsoft Fabric.
Check: Power BI or Fabric lists the Databricks tables.
Official resources
Deciding whether you need it?
Three free tools apply Microsoft's own published guidance to your situation, and say when Azure Databricks or any new technology is not needed.
Sources
Each statement above links to the source it comes from. Labels say who reported it.
- What is Azure Databricks? (opens in a new tab)
- Azure Databricks product page (opens in a new tab)
- AI and machine learning products (Azure Architecture Center) (opens in a new tab)
- Azure Databricks high-level architecture (opens in a new tab)
- Power BI with Azure Databricks (opens in a new tab)
- Mirroring in Microsoft Fabric (opens in a new tab)
Thank you. Your correction was sent.
The editors check it against the sources. If you left an email address, they may reply about it.