Data Engineer Data Architect Platform Team — Use this page to select OneLake, deployment, workload, medallion, and real-time architecture patterns. Follow the linked specialist guides for implementation details.
OneLake Architecture
Designing a unified, governed data foundation for analytics and AI in Microsoft Fabric.
Architecture Overview
Microsoft Fabric combines specialized analytics workloads with OneLake, a tenant-wide logical data lake. A Fabric deployment follows four levels: tenant → capacity → workspace → item. OneLake provides the shared data foundation beneath these levels, while domains, the OneLake catalog, security, and governance support distributed ownership.
Key Concepts
Capacities
A capacity provides regional compute and a billing boundary. Fabric workloads share its pool of Capacity Units (CUs), so architecture decisions must account for workload isolation, concurrency, geography, and cost—not only total data volume.
Workspaces
Workspaces are the primary collaboration, lifecycle, and access boundary. They contain Fabric items, belong to one capacity at a time, and define the scope for Git integration and deployment pipelines. Fabric domains can group workspaces across business areas and capacities for delegated governance.
Items
Items are the deployable artifacts inside a workspace, including lakehouses, warehouses, databases, eventhouses, notebooks, pipelines, semantic models, and reports. Select item types according to access pattern and workload needs rather than forcing every use case into one engine.
OneLake
OneLake is the unified logical data lake automatically included with every Fabric tenant. It is built on Azure Data Lake Storage and supports open table formats including Delta Parquet and Iceberg. Shortcuts virtualize data in place, while mirroring creates platform-managed synchronized copies when analytics needs a query-optimized layout.
Prefer virtualization through shortcuts when data can remain at its source. Use mirroring when a synchronized analytics-ready copy is needed, and use pipelines or other movement tools when orchestration, transformation, streaming, or unsupported sources require them.
OneLake Architecture Patterns
These patterns are complementary building blocks rather than mutually exclusive choices. Most enterprise implementations combine several of them.
Unified Access, Minimal Replication
Expose data across clouds, external lakes, catalogs, and operational systems through shortcuts or mirroring before creating custom movement pipelines.
Choose an integration method →Medallion Architecture
Move data through raw, enriched, and curated quality layers with explicit contracts and ownership between each stage.
Review the Medallion pattern →Domain-Oriented Data Mesh
Let domains own discoverable data products while OneLake, shared policies, and the catalog provide a governed platform foundation.
Explore Data Mesh →Analytics and AI Consolidation
Use open OneLake data across engineering, warehousing, BI, data science, and AI workloads instead of maintaining engine-specific copies.
Prepare data for AI →External Data Sharing
Share governed OneLake data with external organizations without relying on unmanaged exports and duplicate files.
Review security boundaries →Workload Map
| Workload | Purpose | Key Items |
|---|---|---|
| Data Factory | Data ingestion and orchestration | Pipelines, Dataflows Gen2 |
| Data Engineering | Big data transformation with Spark | Lakehouse, Notebooks, Spark Jobs |
| Data Science | Machine learning and experimentation | Notebooks, Experiments, Models |
| Data Warehouse | Enterprise data warehousing with T-SQL | Warehouse, SQL Queries |
| Databases | Operational and transactional applications | SQL Database, Mirrored Data |
| Real-Time Intelligence | Streaming and time-series analytics | Eventhouse, KQL Queryset |
| Power BI | Business intelligence and reporting | Semantic Models, Reports, Dashboards |
| Fabric IQ | Semantic and operational intelligence | Ontology, Graph, Data Agents |
Fabric Deployment Patterns
Choosing tenant, capacity, and workspace boundaries for governance, isolation, lifecycle, and cost.
Start with the simplest topology that satisfies your security, performance, geography, and lifecycle requirements. Add boundaries deliberately: every additional capacity or workspace improves some forms of isolation but also increases administration and cost-management complexity.
| Pattern | Use When | Main Trade-off |
|---|---|---|
| Single workspace, single capacity | Proofs of concept or small, tightly coupled workloads | Simple, but weak lifecycle and workload isolation |
| Multiple workspaces, shared capacity | Teams need separate access and deployment boundaries while sharing compute | Better governance, but workloads can still contend for capacity |
| Multiple workspaces and capacities | Production workloads require performance, regional, billing, or criticality isolation | Strong isolation with higher operational and cost overhead |
| Multiple Fabric tenants | Legal, sovereignty, merger, or hard identity-boundary requirements demand separation | Maximum isolation, but fragmented governance and cross-tenant collaboration |
Define the tenant boundary first, place capacities according to region and isolation needs, use workspaces for ownership and application lifecycle, then organize related workspaces into domains. Avoid using separate tenants or capacities as substitutes for a clear governance model.
Workload Architecture Choices
Selecting the primary Fabric item for each data access and processing pattern.
| Choose | Best Fit | Primary Interface |
|---|---|---|
| Lakehouse | Open-format data engineering, Spark transformations, data science, and mixed structured/unstructured data | Spark, SQL analytics endpoint, OneLake APIs |
| Warehouse | T-SQL-first dimensional modeling, governed relational serving, and BI-focused engineering | T-SQL |
| SQL Database | Operational applications and transactional workloads that also need near-real-time analytics integration | T-SQL application access |
| Eventhouse | High-volume event, log, telemetry, and time-series workloads requiring low-latency ingestion and queries | KQL |
| Semantic Model | Reusable business metrics, governed BI semantics, and Direct Lake reporting | DAX, Power BI |
| Fabric IQ | Business concepts, relationships, operational context, graph scenarios, and agent grounding | Ontology, Graph, AI experiences |
A production solution commonly uses several item types: a Lakehouse for engineering, a Warehouse or curated Lakehouse for serving, a semantic model for business logic, and Eventhouse for hot event data. OneLake allows these workloads to cooperate without treating each engine as a separate data platform.
Medallion Architecture
The proven data organization pattern for building reliable and scalable lakehouses and warehouses.
What is the Medallion Architecture?
The medallion architecture (also known as the "multi-hop" architecture) is a data design pattern that organizes data into three logical layers: Bronze, Silver, and Gold. Each layer represents an increasing level of data quality and business readiness.
Layer Details
Implement each medallion layer as a separate Lakehouse or Warehouse in OneLake. Microsoft recommends placing each Lakehouse in its own workspace for stronger layer-level governance and access control.
- Bronze: Preserve source formats where practical. For data already in OneLake, ADLS Gen2, Amazon S3, or Google Cloud Storage, prefer a shortcut instead of copying it.
- Silver and Gold: Use Delta tables for ACID reliability, history, and performance across Fabric engines.
- Lakehouse pattern: Use a Lakehouse for every layer and expose curated data through the SQL analytics endpoint.
- Hybrid pattern: Use Lakehouses for Bronze and Silver, then a Warehouse for Gold when the serving team and consumers prefer T-SQL and warehouse semantics.
Read the official implementation guidance on Microsoft Learn ↗
🥉 Bronze Layer (Raw)
- Stores data exactly as received from source systems
- Append-only ingestion — never modify or delete raw records
- Include metadata columns:
_ingestion_timestamp,_source_system,_batch_id - Keep source formats when fidelity matters; use Delta tables when relational source data or downstream processing benefits from ACID transactions
- Use shortcuts instead of copying data that can remain in OneLake, ADLS Gen2, Amazon S3, or Google Cloud Storage
- Retain raw data for compliance, auditing, and reprocessing
🥈 Silver Layer (Cleansed & Conformed)
- Apply data quality rules: deduplication, null handling, type casting
- Standardize column names and data types across sources
- Join and enrich data from multiple Bronze tables
- Apply slowly changing dimensions (SCD Type 1/2) where needed
- This layer is the "single source of truth" for your organization
🥇 Gold Layer (Business-Ready)
- Build star/snowflake schemas with facts and dimensions
- Pre-aggregate KPIs and business metrics
- Optimized for Direct Lake mode in Power BI
- Apply column-level and row-level security as needed
- This layer serves dashboards, reports, and ad-hoc analysis
Fabric Data Warehouse Operating Rules
A production medallion architecture depends on keeping each layer operationally predictable. Apply these Microsoft-recommended practices when implementing Bronze, Silver, and Gold with Fabric Data Warehouse.
1. Batch the writes
Favor set-based operations over row-by-row processing. Use COPY INTO or Fabric pipelines for Bronze ingestion and CTAS, INSERT...SELECT, or MERGE for transformations. When practical, consolidate source files toward roughly 100 MB to 1 GB each.
2. Make Silver rerunnable
Design transformations to be idempotent so the same input can be processed again without duplicates or corruption. Use staging for complex logic and record quality failures, rejected rows, and rejection reasons.
3. Design Gold for consumption
Shape Gold around business access patterns: star schemas, data marts, wide tables, or aggregations. Build it from trusted Silver data; direct Gold-to-Bronze dependencies should be exceptional.
4. Start with Fabric defaults
Do not carry traditional warehouse tuning habits forward automatically. First optimize data layout, set-based transformations, data types, and table design, then tune further only when workload metrics justify it.
5. Monitor by layer
Separate ingestion, transformation and quality, and serving signals. Combine pipeline monitoring with Query Insights and Warehouse monitoring views to isolate failures and performance regressions quickly.
If Power BI still needs to clean source data, Silver is incomplete. If Gold becomes a staging area or routinely reads Bronze directly, layer responsibilities have drifted and should be corrected before the pipeline grows.
Naming Conventions
Lakehouses: lh_bronze — Raw ingestion lakehouse lh_silver — Cleansed and conformed data lh_gold — Business-ready consumption layer Tables (Bronze): bronze_crm_customers — Source: CRM, Table: customers bronze_erp_sales_orders — Source: ERP, Table: sales_orders Tables (Silver): silver_dim_customer — Conformed customer dimension silver_fact_sales — Conformed sales fact Tables (Gold): gold_sales_summary_daily — Daily sales aggregation gold_customer_360 — Customer 360 view
Don't skip the Silver layer. Going directly from Bronze to Gold creates brittle pipelines and makes it harder to add new consumers later. The Silver layer provides a stable contract between producers and consumers.
When to Use Medallion vs. Other Patterns
| Pattern | Best For | Considerations |
|---|---|---|
| Medallion (Bronze/Silver/Gold) | Most Fabric implementations; batch and micro-batch workloads | Clear separation of concerns; well-understood pattern |
| Data Mesh | Large orgs with domain-oriented teams | Combine with medallion within each domain |
| Lambda / Kappa | Dual batch + real-time pipelines | Use Real-Time Intelligence alongside medallion |
Real-Time Intelligence
Streaming analytics, event processing, and time-series workloads in Microsoft Fabric — from ingestion to live dashboards in seconds.
Use Real-Time Intelligence when you need sub-second latency on streaming data — IoT telemetry, clickstreams, fraud detection, operational monitoring, or log analytics. For batch/micro-batch workloads, the Lakehouse + medallion pattern is more appropriate.
Core Components
📡 Eventstreams
No-code event ingestion from 30+ sources — Azure Event Hubs, Kafka, IoT Hub, custom apps, CDC streams. Transform events in-flight with built-in processors (filter, aggregate, union).
🏠 Eventhouse
The primary database for real-time data. Built on Azure Data Explorer (Kusto) engine — optimized for append-heavy, time-series workloads with automatic indexing and compression.
🔍 KQL Queryset
Kusto Query Language for exploring streaming data. Purpose-built for time-series: summarize, make-series, render timechart, anomaly detection, and pattern matching.
📊 Real-Time Dashboards
Live dashboards with auto-refresh down to 1-second intervals. Pinned KQL visuals, parameters, and cross-filtering — no import or refresh schedule needed.
🔔 Activator
No-code trigger engine — monitor streaming data and fire actions (emails, Teams messages, Power Automate flows) when conditions are met. Think "alerts as a service."
🌐 Real-Time Hub
Centralized catalog of all streaming data in your organization. Discover, subscribe to, and share real-time event streams across workspaces and domains.
Architecture Pattern: Event-Driven Pipeline
Eventhouse vs. Lakehouse: When to Use What
| Aspect | Eventhouse (KQL) | Lakehouse (Spark/SQL) |
|---|---|---|
| Data pattern | Append-heavy, time-series, streaming | Batch, micro-batch, full reloads |
| Latency | Sub-second ingestion to query | Minutes (Spark jobs) to seconds (Direct Lake) |
| Query language | KQL (Kusto Query Language) | Spark SQL, PySpark, T-SQL |
| Best for | Logs, IoT, clickstream, monitoring, fraud | Data warehousing, ML feature stores, reports |
| Retention | Hot/warm caching with auto-purge policies | Persistent Delta tables in OneLake |
| Integration | Eventstreams, Real-Time Hub, Activator | Data Factory, notebooks, Power BI Direct Lake |
| OneLake | Can mirror data to OneLake as Delta for cross-engine access | Native OneLake storage |
Use both together when the scenario needs hot and historical paths: route event data to Eventhouse for low-latency dashboards and actions, and retain it in OneLake for historical analytics, engineering, and machine learning.