Development Lifecycle

A well-structured deployment strategy ensures reliability and consistency across environments. Microsoft Fabric supports a full Dev → Test → Prod lifecycle with built-in tools.

Fabric Deployment Pipelines

Deployment pipelines are Fabric's built-in tool for managing the lifecycle of your artifacts (lakehouses, warehouses, reports, notebooks). They let you promote artifacts through stages with comparison and selective deployment.

• Up to 10 stages per pipeline (typically Dev → Test → Prod)
• Compare artifact differences between stages before deploying
• Deploy specific items or all items at once
• Configure deployment rules to parameterize connections per environment
• Supports backward deployment (e.g., rolling back from Prod to Test)

Git Integration

Fabric supports direct integration with Azure DevOps and GitHub repositories. This enables version control, code review, and collaboration workflows for Fabric artifacts.

Git Integration Setup

• Connect workspace to a Git repo (Azure DevOps or GitHub)
• Each artifact is serialized into JSON/PBIR definitions in the repo
• Use feature branches for development, merge to main via PR
• Auto-sync: changes in Git can auto-update the workspace (and vice versa)

CI/CD Patterns

Python CI/CD with fabric-cicd

The fabric-cicd library is Microsoft's open-source Python SDK for automating Fabric workspace deployments. It provides deterministic, code-first deployments that integrate seamlessly with GitHub Actions and Azure DevOps pipelines.

Installation

pip install fabric-cicd

Key Capabilities

• Deterministic deployments: Full, predictable workspace deployments every time — no manual workspace operations needed
• Source-control integration: Deploy directly from Git repository directories (GitHub, Azure DevOps) into target workspaces
• Environment parameterization: Use parameter.yml files to manage environment-specific values (connection strings, workspace IDs, lakehouse names)
• Flexible authentication: Supports all Azure TokenCredential methods — user identity, service principal, and managed identity
• Supported item types: Notebooks, Data Pipelines, Semantic Models, Reports, Lakehouses, Environments, and more

Deployment Example

from fabric_cicd import FabricWorkspace, publish_all_items
from azure.identity import ClientSecretCredential

# Authenticate with service principal
credential = ClientSecretCredential(
    tenant_id="your-tenant-id",
    client_id="your-client-id",
    client_secret="your-client-secret"
)

# Define the target workspace
workspace = FabricWorkspace(
    workspace_id="your-workspace-id",
    environment="PROD",                    # Maps to parameter.yml values
    repository_directory="./fabric-items", # Local repo directory with item definitions
    item_type_in_scope=["Notebook", "DataPipeline", "SemanticModel", "Report"],
    credential=credential
)

# Deploy all items
publish_all_items(workspace)

GitHub Actions Integration

name: Deploy to Fabric
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install fabric-cicd
        run: pip install fabric-cicd

      - name: Deploy to Production
        env:
          AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
          AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
          AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
        run: python scripts/deploy.py --env PROD

Environment Parameterization

find_replace:
  - find: "your-dev-lakehouse-id"
    replace_with:
      DEV: "dev-lakehouse-id"
      TEST: "test-lakehouse-id"
      PROD: "prod-lakehouse-id"
  - find: "your-dev-connection"
    replace_with:
      DEV: "dev-connection-string"
      TEST: "test-connection-string"
      PROD: "prod-connection-string"

Environment Separation

• Use separate workspaces per environment (ws-dev, ws-test, ws-prod)
• Use separate capacities for production vs. non-production workloads
• Apply deployment rules to swap connection strings per environment
• Restrict production workspace access to service principals or limited admins

Capacity Units (CUs)

Microsoft Fabric uses a universal compute unit called a Capacity Unit (CU). All workloads — Spark, SQL, Power BI, Data Factory — consume from the same CU pool. This simplifies capacity planning compared to provisioning separate services.

SKU Options

*Approximate pricing (USD, East US region). Check the official pricing page for current rates. 3-year reservations offer even deeper discounts. F64+ includes free Power BI viewer access (no Pro license needed).

Pricing Models

🧮 Capacity Cost Calculator

Estimate your Fabric capacity needs based on your expected workload mix. Adjust the sliders to see a real-time SKU recommendation.

Cost Optimization Strategies

• Pause/Resume: Pause non-production capacities after business hours and on weekends — savings up to 70%
• Right-size capacity: Use the Capacity Metrics App to monitor utilization and resize accordingly
• Smoothing & Bursting: Fabric smooths CU usage over 24-hour windows, allowing short bursts without throttling
• Optimize Spark jobs: Use V-Order optimization, partition pruning, and right-size Spark sessions
• Use Direct Lake: Avoid import-mode datasets that consume memory — Direct Lake reads from OneLake directly
• Monitor with Capacity Metrics: Install the Microsoft Fabric Capacity Metrics app to track CU consumption by workload

How Capacity Works: Smoothing, Bursting & Throttling

Fabric doesn't enforce CU limits on a per-second basis. Instead, it uses a smoothing mechanism that spreads your CU consumption over time windows, allowing temporary bursts above your SKU limit. Understanding these mechanics is essential for right-sizing and avoiding unexpected throttling.

📈 Smoothing (24-Hour Window)

Every Fabric operation consumes CU-seconds. Instead of evaluating consumption instantly, Fabric smooths it over a rolling 24-hour window. This means a heavy job at 2 AM can be offset by idle time at 3 AM — your effective utilization is the average, not the peak.

As long as the green smoothed line stays below the blue SKU limit, your capacity is healthy — even if individual spikes (orange) exceed the limit temporarily.

⚡ Bursting

Fabric allows your workloads to burst above your SKU's CU allocation for short periods. This is not extra capacity you pay for — it's borrowed from your future idle time. Bursting is automatic and requires no configuration.

🚫 Throttling & Rejection

When your smoothed CU consumption exceeds the SKU limit for too long, Fabric begins throttling. There are two levels of enforcement:

🧮 Worked Example: F64 Capacity — A Day in the Life

Here's a concrete scenario for a team running on F64 (64 CUs) to illustrate how smoothing and bursting work together:

Migration Decision Framework

Migration is not one-size-fits-all. Use a phased approach to reduce risk and ensure continuity:

🔄 Interactive Migration Path Recommender

Select your current platform to see a step-by-step tailored migration path to Fabric.

Migration Paths

From Azure Synapse Analytics

• Dedicated SQL Pools → Fabric Warehouse: Migrate T-SQL scripts and stored procedures. Most T-SQL is compatible; review unsupported features (e.g., some system views)
• Serverless SQL Pools → Fabric Lakehouse: Replace OPENROWSET queries with Lakehouse SQL endpoints
• Synapse Spark → Fabric Notebooks: PySpark/Scala code largely compatible; update library references and session configs
• Use Shortcuts to connect Fabric to existing ADLS Gen2 storage during transition

From Azure Data Factory (ADF)

• Fabric Data Factory supports most ADF pipeline activities natively
• Use copy activity equivalents and Dataflows Gen2 for transformation
• Some connectors may differ — validate your source/sink connectors
• Consider running ADF and Fabric in parallel during migration

From Power BI Premium

• Power BI workspaces can be reassigned to Fabric capacity (F64 or higher)
• Existing datasets, reports, and dashboards continue to work unchanged
• Migrate import-mode datasets to Direct Lake for better performance and cost savings
• P SKUs map to equivalent F SKUs (P1 ≈ F64, P2 ≈ F128, etc.)

From On-Premises (SQL Server / SSIS)

• Use Data Factory pipelines with on-premises data gateway for initial data movement
• Migrate SSIS packages to Dataflows Gen2 or Fabric notebooks
• Use Fabric mirroring to replicate SQL Server databases to OneLake in near real-time
• Plan for a hybrid phase where on-prem and Fabric run in parallel

Sizing Approach

Capacity sizing is not a one-time exercise — it's an iterative process. Start with an estimate, deploy, monitor, and adjust. The key principle: size for your typical load, not your peak. Fabric's bursting and smoothing mechanisms handle short spikes automatically.

Sizing for New Workloads

When starting fresh with Fabric, follow this framework:

Step 1: Inventory Your Workloads

Catalog what you plan to run and the expected scale:

Step 2: Use the SKU Estimator

Microsoft provides an official Fabric SKU Estimator tool. Input your expected user count, data volumes, refresh rates, and workload mix to get a recommended starting SKU.

Step 3: Start with Trial or Dev Capacity

Sizing for Migrations

When migrating from an existing platform, you have historical usage data to guide your sizing:

From Power BI Premium (P-SKU)

You can enable Fabric on your existing P-SKU capacity — no need to purchase a new one. The same capacity pool now supports all Fabric workloads in addition to Power BI.

From Azure Synapse / Databricks

• Inventory current compute: Document your Spark pool sizes, SQL DWU usage, and pipeline activity hours
• Map to CUs: There's no exact 1:1 mapping. Run representative workloads on a Fabric trial to benchmark actual CU consumption
• Start parallel: Run Fabric and legacy platform side-by-side during migration. Compare performance and CU usage before cutting over
• Use Shortcuts: Point Fabric to your existing ADLS storage via shortcuts — this lets you test Fabric compute without moving data

From On-Premises (SQL Server / SSIS)

• Measure peak CPU and memory on existing SQL Servers during ETL windows
• Start with F8-F16 for most departmental SQL Server migrations
• Test with mirroring: Use Fabric mirroring to replicate your SQL databases to OneLake and measure CU impact before full migration
• Account for concurrency: Cloud workloads often see higher concurrency than on-prem — size accordingly

Ongoing Optimization