What Are Shortcuts?

Shortcuts are pointers to data in external or internal locations. They appear as folders in your lakehouse but don't copy data — queries read directly from the source. This enables unified data access across clouds, storage accounts, and Fabric workspaces with no ETL overhead.

Shortcut Types

Architecture Patterns with Shortcuts

Security & Access Control

• Authentication: Internal shortcuts use Fabric identity. External shortcuts (ADLS, S3, GCS) use connection credentials configured by an admin.
• Authorization: Access to the shortcut is governed by workspace permissions. The user must also have access to the underlying data source.
• Row-Level Security: RLS defined on the source data applies when queries traverse shortcuts.
• Governance: Shortcuts respect Microsoft Purview sensitivity labels and Information Protection policies.

Best Practices

✅ Use shortcuts for read-heavy, cross-domain sharing (Gold layer consumption)
✅ Use shortcuts to avoid duplicating Bronze data already in ADLS Gen2
✅ Apply consistent naming: sc_<source>_<dataset> (e.g., sc_sales_orders)
✅ Document shortcuts in a central catalog — they're invisible in lineage views

⚠️  Monitor source latency — external shortcuts add network round-trip time
⚠️  Test Direct Lake mode compatibility — not all shortcut types support it

❌ Don't use shortcuts for write-back scenarios — they're read-only
❌ Don't chain shortcuts (shortcut → shortcut) — performance degrades
❌ Don't shortcut to volatile/temporary storage locations

What Is Mirroring?

Mirroring continuously replicates data from external databases into OneLake as Delta tables using Change Data Capture (CDC). Unlike traditional ETL, mirroring is managed by Fabric — you configure the source, and data flows automatically with near real-time latency.

Supported Sources

How Mirroring Works

Architecture Patterns

Best Practices

✅ Start with high-value tables — mirror what's needed, not entire databases
✅ Use the SQL analytics endpoint for ad-hoc queries on mirrored data
✅ Combine mirroring with medallion architecture: mirror → Bronze, transform → Silver/Gold
✅ Monitor replication lag in the Fabric portal and set up alerts

⚠️  Plan for initial snapshot time — large tables may take hours on first sync
⚠️  Test schema evolution scenarios before going to production

❌ Don't use mirroring for sub-second latency requirements — use Eventstreams instead
❌ Don't mirror tables with frequent schema changes without a re-init plan
❌ Don't skip capacity planning — continuous CDC consumes CUs 24/7

What Are Dataflows Gen2?

Dataflows Gen2 provide a Power Query-based, low-code interface for connecting to data sources, applying transformations, and loading data into Fabric destinations (lakehouses, warehouses, or KQL databases). They run on a managed Spark backend and support 150+ connectors.

Key Capabilities

When to Use Dataflows Gen2

What Are Data Pipelines?

Data Pipelines in Fabric are the orchestration engine (based on Azure Data Factory) for scheduling and coordinating data movement and transformation activities. They connect all other integration components — triggering Dataflows, Spark notebooks, stored procedures, and copy operations in sequence or parallel.

Core Activities

Orchestration Patterns

Why Airflow in Fabric?

Microsoft Fabric offers managed Apache Airflow jobs as a first-class Data Factory capability. This replaces the deprecated Azure Data Factory Airflow integration and provides a fully managed, auto-scaling orchestration service — no infrastructure to maintain.

Airflow vs. Data Pipelines — Decision Matrix

Key Capabilities

Example: Orchestrating a Medallion Pipeline

from airflow import DAG
from airflow.providers.microsoft.fabric.operators.run_item import FabricRunItemOperator
from datetime import datetime

with DAG(
    "medallion_etl",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
    catchup=True,          # enables backfill for historical dates
) as dag:

    bronze_ingest = FabricRunItemOperator(
        task_id="bronze_ingest",
        fabric_conn_id="fabric_prod",
        workspace_id="ws-abc-123",
        item_id="notebook-bronze-ingest",
        job_type="RunNotebook",
        wait_for_termination=True,
        deferrable=True,   # frees Airflow worker while waiting
    )

    silver_transform = FabricRunItemOperator(
        task_id="silver_transform",
        fabric_conn_id="fabric_prod",
        workspace_id="ws-abc-123",
        item_id="notebook-silver-transform",
        job_type="RunNotebook",
        wait_for_termination=True,
        deferrable=True,
    )

    gold_publish = FabricRunItemOperator(
        task_id="gold_publish",
        fabric_conn_id="fabric_prod",
        workspace_id="ws-abc-123",
        item_id="pipeline-gold-publish",
        job_type="Pipeline",
        wait_for_termination=True,
        deferrable=True,
    )

    # DAG dependency chain
    bronze_ingest >> silver_transform >> gold_publish

Migration from ADF Airflow

Azure Data Factory's Apache Airflow integration was deprecated in 2025. Microsoft recommends migrating to Fabric's managed Airflow:

• DAGs are compatible: Move your existing Python DAG files with minimal changes (update connection IDs and operator imports)
• Connections migrate: Reconfigure connections to use Fabric's Entra-based authentication
• Cost benefit: Fabric's auto-pause reduces baseline costs compared to ADF's always-on Airflow environments

What Is Data Wrangler?

Data Wrangler is a visual, code-generation tool embedded directly in Fabric notebooks. You open a DataFrame in Wrangler, explore column distributions, apply transformations through a point-and-click interface, and it generates clean PySpark or Pandas code that you paste back into your notebook. It's designed for interactive profiling and prototyping — not for scheduled execution.

When to Use Data Wrangler

Best Practices

• Use Wrangler for exploration, not execution. Generate code in Wrangler, copy it into your notebook cell, then run the notebook in a Pipeline for production.
• Work on samples first. Wrangler loads data into memory — start with .limit(10000) or a filtered subset for large tables, then apply transforms to the full dataset in Spark.
• Export as functions. Use "Export to code" to get a clean transformation function. Wrap it in a reusable module for your team.
• Pair with Dataflows Gen2 for end-to-end flow: Wrangler prototypes the logic → you implement it in a notebook → Pipeline orchestrates the notebook on schedule.

Which Tool Should I Use?

Common Combinations