CI/CD Platform Deep Dive: Azure DevOps Pipelines

Introduction

Azure DevOps is Microsoft's complete DevOps platform — a unified suite covering project planning, source control, CI/CD pipelines, testing, and artifact management. While GitHub Actions dominates the open-source world, Azure DevOps remains the enterprise workhorse — especially for organizations deep in the Microsoft ecosystem with complex compliance requirements, on-premises infrastructure, and multi-team governance needs.

Azure Pipelines, the CI/CD engine within Azure DevOps, stands apart from competitors through its hybrid agent architecture (seamlessly mixing cloud and on-premises agents), native Azure ARM integration, and enterprise-grade features like approval gates, variable groups linked to Azure Key Vault, and template-based pipeline governance.

When to Choose Azure DevOps

Choose Azure DevOps over GitHub Actions when you need: work item tracking tightly integrated with deployments, fine-grained access control across dozens of teams, on-premises agent pools behind corporate firewalls, classic release pipelines with graphical deployment gates, or deep Azure Resource Manager integration with managed identity and service connections. Azure DevOps is also preferred when regulatory compliance requires separation between code hosting and CI/CD execution.

Organizations with hundreds of microservices, multiple Azure subscriptions, and complex approval chains find Azure DevOps' project-level isolation and role-based access control superior to repository-level scoping in GitHub Actions.

Azure DevOps Ecosystem

Azure DevOps consists of five integrated services, each independently usable but most powerful when combined:

flowchart TD
    ORG[Organization] --> P1[Project A]
    ORG --> P2[Project B]
    
    P1 --> B[Azure Boards]
    P1 --> R[Azure Repos]
    P1 --> PL[Azure Pipelines]
    P1 --> TP[Test Plans]
    P1 --> AF[Azure Artifacts]
    
    B -->|Work Items Link| PL
    R -->|Triggers| PL
    PL -->|Publishes| AF
    PL -->|Reports| TP
    AF -->|Consumed by| PL
    
    style ORG fill:#132440,color:#fff
    style PL fill:#3B9797,color:#fff
    style R fill:#16476A,color:#fff
    style B fill:#BF092F,color:#fff
    style AF fill:#16476A,color:#fff
    style TP fill:#16476A,color:#fff
                            

• Azure Boards — Agile planning with work items, sprints, Kanban boards. Integrates with pipelines via AB# commit links.
• Azure Repos — Git repositories with branch policies, pull request workflows, and code search.
• Azure Pipelines — CI/CD engine supporting YAML and classic pipelines. 1,800 free minutes/month for public projects.
• Azure Test Plans — Manual and exploratory testing with test case management and automated test integration.
• Azure Artifacts — Package management for npm, NuGet, Maven, Python, and Universal Packages.

Organizations & Projects

The organizational hierarchy is: Organization → Project → Repository/Pipeline/Feed. Organizations map to companies or divisions. Projects provide isolation boundaries — teams within a project share repos, pipelines, boards, and artifacts. Cross-project references are possible but require explicit permissions.

Pipeline Types

Azure DevOps supports two fundamentally different pipeline types: YAML pipelines (code-first) and classic pipelines (GUI-based). Understanding when to use each — and how to migrate from classic to YAML — is critical.

YAML Pipelines are the modern standard. Pipeline definition lives in the repository as code — versioned, reviewed in PRs, and portable across projects. YAML pipelines support multi-stage builds, template reuse, conditional logic, and everything classic pipelines can do.

Classic Pipelines use a graphical editor in the Azure DevOps portal. They were the original pipeline system and are still used by organizations that haven't migrated. Microsoft has signaled that classic pipelines will eventually be deprecated, making YAML migration a priority.

Key differences between classic and YAML:

YAML Pipeline Syntax

Azure Pipelines YAML has a hierarchical structure: trigger → pool → stages → jobs → steps. For simple pipelines, stages and jobs can be implicit.

Here's a comprehensive example demonstrating the full YAML syntax:

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main
      - release/*
    exclude:
      - feature/experimental/*
  paths:
    include:
      - src/**
      - tests/**
    exclude:
      - '*.md'

pr:
  branches:
    include:
      - main
  paths:
    include:
      - src/**

pool:
  vmImage: 'ubuntu-latest'

variables:
  - name: buildConfiguration
    value: 'Release'
  - group: 'production-secrets'
  - name: isMain
    value: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]

parameters:
  - name: environment
    displayName: 'Deploy Environment'
    type: string
    default: staging
    values:
      - staging
      - production

stages:
  - stage: Build
    displayName: 'Build & Test'
    jobs:
      - job: BuildApp
        displayName: 'Build Application'
        steps:
          - checkout: self
            fetchDepth: 0

          - task: UseDotNet@2
            displayName: 'Install .NET SDK'
            inputs:
              packageType: 'sdk'
              version: '8.x'

          - script: |
              dotnet restore
              dotnet build --configuration $(buildConfiguration)
            displayName: 'Restore & Build'

          - script: dotnet test --configuration $(buildConfiguration) --collect:"XPlat Code Coverage"
            displayName: 'Run Tests'

          - task: PublishTestResults@2
            inputs:
              testResultsFormat: 'VSTest'
              testResultsFiles: '**/*.trx'

          - task: PublishCodeCoverageResults@2
            inputs:
              summaryFileLocation: '**/coverage.cobertura.xml'

          - task: PublishBuildArtifacts@1
            inputs:
              pathToPublish: '$(Build.ArtifactStagingDirectory)'
              artifactName: 'drop'

  - stage: Deploy
    displayName: 'Deploy to ${{ parameters.environment }}'
    dependsOn: Build
    condition: and(succeeded(), eq(variables.isMain, true))
    jobs:
      - deployment: DeployWeb
        displayName: 'Deploy Web App'
        environment: '${{ parameters.environment }}'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'production-arm'
                    appName: 'myapp-$(parameters.environment)'
                    package: '$(Pipeline.Workspace)/drop/**/*.zip'

Key syntax elements to understand:

• trigger — Branch and path filters for CI triggers. Supports include/exclude patterns.
• pr — Pull request validation trigger. Runs on PR creation and updates.
• pool — Specifies the agent pool. Can be set at pipeline, stage, or job level.
• variables — Inline values, variable group references, or runtime expressions ($[]).
• parameters — Compile-time inputs with types and validation. Accessed via ${{ }}.
• stages — Top-level groupings with dependencies and conditions.
• deployment — Special job type that records deployments to environments.

Agent Pools

Agents are the compute that executes pipeline jobs. Azure DevOps provides three types: Microsoft-hosted agents, self-hosted agents, and scale set agents.

# Using Microsoft-hosted agent
pool:
  vmImage: 'ubuntu-latest'

# Using self-hosted agent with demands
pool:
  name: 'MyPrivatePool'
  demands:
    - docker
    - Agent.OS -equals Linux
    - myCustomCapability

# Using scale set agents
pool:
  name: 'MyScaleSetPool'
  demands:
    - Agent.OS -equals Linux

Agent capabilities and demands enable routing jobs to specific agents. Capabilities are key-value pairs on agents (set automatically or manually). Demands in the pipeline specify which capabilities are required.

Service Connections

Service connections are the bridge between Azure DevOps and external services — Azure subscriptions, Docker registries, Kubernetes clusters, AWS accounts, npm feeds, and more. They securely store credentials and manage authentication without exposing secrets in pipeline YAML.

The most important service connection type is Azure Resource Manager (ARM), which connects pipelines to Azure subscriptions:

# Using an ARM service connection in a pipeline
steps:
  - task: AzureCLI@2
    displayName: 'Deploy Infrastructure'
    inputs:
      azureSubscription: 'my-azure-connection'  # Service connection name
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: |
        az group create --name myapp-rg --location eastus
        az deployment group create \
          --resource-group myapp-rg \
          --template-file infrastructure/main.bicep \
          --parameters environment=production

Workload Identity Federation (OIDC) is the recommended authentication method for Azure connections — it eliminates stored secrets entirely:

# Service connection with workload identity federation
# No secrets stored — uses OIDC token exchange
# Azure DevOps presents a signed token to Azure AD
# Azure AD validates the token and returns an access token
# Pipeline authenticates without any stored credentials

# Configuration steps:
# 1. Create App Registration in Azure AD
# 2. Configure Federated Credential with:
#    - Issuer: https://vstoken.dev.azure.com/{organization-id}
#    - Subject: sc://{org}/{project}/{service-connection-name}
# 3. Assign RBAC roles to the App Registration
# 4. Create service connection referencing the App Registration

Variable Groups & Library

Variable groups centralize configuration values that are shared across multiple pipelines. They can contain plain text values or link directly to Azure Key Vault secrets. The Library also stores secure files (certificates, signing keys) that pipelines can download at runtime.

# Referencing variable groups in pipeline YAML
variables:
  - group: 'shared-settings'           # Plain text variable group
  - group: 'production-keyvault'       # Key Vault-linked variable group
  - name: localVariable
    value: 'inline-value'

# Variable scoping at different levels
stages:
  - stage: Build
    variables:
      - name: stageVar
        value: 'only-in-build-stage'
    jobs:
      - job: Compile
        variables:
          - name: jobVar
            value: 'only-in-compile-job'
        steps:
          - script: echo "$(localVariable) $(stageVar) $(jobVar)"

Key Vault Integration — Link a variable group to Azure Key Vault and secrets become pipeline variables automatically. The pipeline requests only the secrets it needs, and values are never written to logs:

# Key Vault-linked variable group (configured in UI)
# Maps Key Vault secrets to pipeline variables:
#   Key Vault Secret Name     →  Pipeline Variable
#   database-connection-string →  $(database-connection-string)
#   api-signing-key            →  $(api-signing-key)
#   storage-account-key        →  $(storage-account-key)

steps:
  - script: |
      # Secrets are automatically masked in logs
      echo "Connecting to database..."
      dotnet run --connection "$(database-connection-string)"
    displayName: 'Run with Key Vault secrets'

Templates & Extends

Templates are Azure Pipelines' most powerful feature for standardization and governance. They allow you to define reusable pipeline fragments (stages, jobs, steps, or even entire pipelines) that teams consume. The extends keyword enforces mandatory templates — pipelines must extend from an approved template.

# templates/build-dotnet.yml (reusable job template)
parameters:
  - name: dotnetVersion
    type: string
    default: '8.x'
  - name: projects
    type: string
    default: '**/*.csproj'
  - name: testProjects
    type: string
    default: '**/*Tests.csproj'
  - name: publishArtifact
    type: boolean
    default: true

jobs:
  - job: Build
    displayName: 'Build .NET Application'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - task: UseDotNet@2
        inputs:
          packageType: 'sdk'
          version: ${{ parameters.dotnetVersion }}

      - script: dotnet restore ${{ parameters.projects }}
        displayName: 'Restore packages'

      - script: dotnet build ${{ parameters.projects }} --configuration Release --no-restore
        displayName: 'Build solution'

      - script: dotnet test ${{ parameters.testProjects }} --configuration Release --no-build --collect:"XPlat Code Coverage"
        displayName: 'Run tests'

      - ${{ if eq(parameters.publishArtifact, true) }}:
        - task: PublishBuildArtifacts@1
          inputs:
            pathToPublish: '$(Build.ArtifactStagingDirectory)'
            artifactName: 'app'

# azure-pipelines.yml (consuming the template)
trigger:
  branches:
    include: [main]

stages:
  - stage: Build
    jobs:
      - template: templates/build-dotnet.yml
        parameters:
          dotnetVersion: '8.x'
          projects: 'src/**/*.csproj'
          testProjects: 'tests/**/*.csproj'

  - stage: Deploy
    dependsOn: Build
    jobs:
      - template: templates/deploy-webapp.yml
        parameters:
          environment: production
          appName: myapp-prod

The extends keyword provides governance by forcing all pipelines to inherit from an organizational template:

# azure-pipelines.yml (team pipeline — MUST extend)
extends:
  template: templates/secure-pipeline.yml@templates-repo
  parameters:
    stages:
      - stage: Build
        jobs:
          - job: BuildApp
            steps:
              - script: npm ci && npm run build
              - script: npm test

# templates/secure-pipeline.yml (org-approved template)
# Enforces: credential scanning, container scanning,
# required approvals, audit logging, allowed agent pools

Environments

Environments represent deployment targets and enable approval gates, deployment history, and resource tracking. When a deployment job targets an environment, Azure DevOps records what was deployed, when, and by whom.

flowchart LR
    subgraph runOnce["runOnce Strategy"]
        A1[Deploy] --> A2[Verify]
    end
    
    subgraph rolling["Rolling Strategy"]
        B1[Batch 1: 25%] --> B2[Batch 2: 25%] --> B3[Batch 3: 25%] --> B4[Batch 4: 25%]
    end
    
    subgraph canary["Canary Strategy"]
        C1[Deploy 10%] --> C2[Monitor] --> C3[Deploy 50%] --> C4[Monitor] --> C5[Deploy 100%]
    end
    
    style A1 fill:#3B9797,color:#fff
    style B1 fill:#16476A,color:#fff
    style C1 fill:#BF092F,color:#fff
                            

# Deployment job with environment and strategy
stages:
  - stage: DeployStaging
    jobs:
      - deployment: DeployToStaging
        displayName: 'Deploy to Staging'
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'staging-arm'
                    appName: 'myapp-staging'
                    package: '$(Pipeline.Workspace)/drop/*.zip'

  - stage: DeployProduction
    dependsOn: DeployStaging
    jobs:
      - deployment: DeployToProduction
        displayName: 'Deploy to Production'
        environment: 'production'  # Has approval gate configured
        strategy:
          rolling:
            maxParallel: 2
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'production-arm'
                    appName: 'myapp-prod'
                    package: '$(Pipeline.Workspace)/drop/*.zip'
            on:
              failure:
                steps:
                  - script: echo "Deployment failed - initiating rollback"
              success:
                steps:
                  - script: echo "Deployment succeeded"

Environment checks and approvals are configured in the Azure DevOps portal (not in YAML). Available checks include: manual approval, business hours gate, invoke Azure Function, invoke REST API, evaluate artifact policy, exclusive lock, and required template.

Azure Integration

Azure Pipelines has first-class integration with Azure services. Built-in tasks handle authentication, resource provisioning, and deployment without manual credential management.

# Deploy infrastructure with Bicep
steps:
  - task: AzureCLI@2
    displayName: 'Deploy Bicep Infrastructure'
    inputs:
      azureSubscription: 'production-arm'
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: |
        az deployment sub create \
          --location eastus \
          --template-file infrastructure/main.bicep \
          --parameters \
            environment=production \
            appName=$(appName) \
            sqlAdminPassword=$(sqlPassword)

# Deploy to Azure Kubernetes Service (AKS)
steps:
  - task: KubernetesManifest@1
    displayName: 'Deploy to AKS'
    inputs:
      action: 'deploy'
      connectionType: 'azureResourceManager'
      azureSubscriptionConnection: 'production-arm'
      azureResourceGroup: 'myapp-rg'
      kubernetesCluster: 'myapp-aks'
      namespace: 'production'
      manifests: |
        kubernetes/deployment.yml
        kubernetes/service.yml
      containers: |
        myregistry.azurecr.io/myapp:$(Build.BuildId)
      strategy: canary
      percentage: 25

# Deploy Azure Function
steps:
  - task: AzureFunctionApp@2
    displayName: 'Deploy Azure Functions'
    inputs:
      azureSubscription: 'production-arm'
      appType: 'functionAppLinux'
      appName: 'myapp-functions'
      package: '$(Pipeline.Workspace)/functions/*.zip'
      runtimeStack: 'DOTNET-ISOLATED|8.0'
      deploymentMethod: 'zipDeploy'

Multi-Stage Pipelines

Multi-stage pipelines consolidate build, test, and deploy into a single YAML file with explicit dependencies and conditions. This replaces the old pattern of separate build pipelines and release pipelines.

# Complete multi-stage pipeline
trigger:
  branches:
    include: [main]

variables:
  - group: 'shared-config'
  - name: imageTag
    value: '$(Build.BuildId)'

stages:
  - stage: Build
    displayName: 'Build & Unit Test'
    jobs:
      - job: Build
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: dotnet build --configuration Release
          - script: dotnet test --no-build --configuration Release
          - task: Docker@2
            inputs:
              command: buildAndPush
              repository: 'myregistry.azurecr.io/myapp'
              tags: $(imageTag)

  - stage: IntegrationTest
    displayName: 'Integration Tests'
    dependsOn: Build
    jobs:
      - job: IntTest
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: |
              docker compose -f docker-compose.test.yml up -d
              dotnet test tests/Integration/ --configuration Release
              docker compose -f docker-compose.test.yml down

  - stage: DeployStaging
    displayName: 'Deploy to Staging'
    dependsOn: IntegrationTest
    jobs:
      - deployment: Staging
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'staging-arm'
                    appName: 'myapp-staging'

  - stage: DeployProduction
    displayName: 'Deploy to Production'
    dependsOn: DeployStaging
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: Production
        environment: 'production'  # Requires manual approval
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: 'production-arm'
                    appName: 'myapp-prod'

Security & Governance

Azure DevOps provides enterprise-grade security controls for pipeline governance:

• Pipeline Permissions — Control which pipelines can use specific service connections, agent pools, variable groups, and environments.
• Branch Policies — Require PR validation builds, minimum reviewers, linked work items, and comment resolution before merge.
• Required Templates — Force all pipelines to extend from approved organizational templates.
• Approval Gates — Multi-stage approvals with timeout, business hours restrictions, and exclusive locks.
• Audit Logs — Complete audit trail of pipeline runs, permission changes, and service connection usage.
• Retention Policies — Configure how long build artifacts and logs are retained.

# Security-hardened pipeline pattern
trigger:
  branches:
    include: [main]

# Extend from org-approved template (enforced via settings)
extends:
  template: templates/secure-ci.yml@org-templates
  parameters:
    pool: 'approved-linux-pool'  # Only allowed pools
    serviceConnection: 'production-arm'
    stages:
      - stage: Build
        jobs:
          - job: BuildApp
            steps:
              - script: npm ci
              - script: npm run build
              - script: npm test
              # Credential scanning runs automatically (from template)
              # Container scanning runs automatically (from template)
              # SBOM generation runs automatically (from template)

Artifacts & Feeds

Azure Artifacts provides integrated package management supporting npm, NuGet, Maven, Gradle, Python (pip), and Universal Packages. Feeds can be project-scoped or organization-scoped, with upstream sources connecting to public registries.

# Publish NuGet package to Azure Artifacts
steps:
  - task: DotNetCoreCLI@2
    displayName: 'Pack NuGet'
    inputs:
      command: 'pack'
      packagesToPack: 'src/MyLibrary/*.csproj'
      versioningScheme: 'byBuildNumber'

  - task: NuGetCommand@2
    displayName: 'Push to Feed'
    inputs:
      command: 'push'
      packagesToPush: '$(Build.ArtifactStagingDirectory)/**/*.nupkg'
      publishVstsFeed: 'my-project/my-nuget-feed'

# Publish npm package to Azure Artifacts
steps:
  - task: Npm@1
    displayName: 'npm publish'
    inputs:
      command: 'publish'
      publishRegistry: 'useFeed'
      publishFeed: 'my-project/my-npm-feed'

# Universal Packages (arbitrary files)
steps:
  - task: UniversalPackages@0
    displayName: 'Publish Universal Package'
    inputs:
      command: 'publish'
      publishDirectory: '$(Build.ArtifactStagingDirectory)/release'
      feedsToUsePublish: 'internal'
      vstsFeedPublish: 'my-project/my-universal-feed'
      vstsFeedPackagePublish: 'my-application'
      versionOption: 'patch'

Exercises