Module 4: Azure Pipelines — YAML Fundamentals

Introduction to Azure Pipelines

Azure Pipelines is the CI/CD automation engine within Azure DevOps. It takes your code, builds it, tests it, and deploys it — automatically, reliably, and repeatably. Every time a developer pushes code, Pipelines can validate it within minutes, catching bugs before they reach production.

Think of a YAML pipeline as a recipe — triggers are "when to cook" (on push? on schedule? when another pipeline finishes?), stages are the courses (appetizer → main → dessert), jobs are individual dishes within each course, and steps are the cooking instructions for each dish. Variables are your ingredient measurements — you define them once and reference them throughout the recipe.

Classic vs YAML Pipelines

Azure Pipelines offers two authoring experiences:

For CI/CD concepts and theory, see Part 14: Continuous Integration and Part 15: CI/CD Pipeline Architecture. This module focuses on the YAML syntax — the hands-on "how" of writing pipelines.

Pipeline Structure (The Big Picture)

Every YAML pipeline follows a nested hierarchy: Pipeline → Stages → Jobs → Steps. Understanding this hierarchy is the single most important concept for writing pipelines.

flowchart TD
    A[Pipeline] --> B[Stage: Build]
    A --> C[Stage: Test]
    A --> D[Stage: Deploy]
    B --> E[Job: compile]
    B --> F[Job: lint]
    C --> G[Job: unit-tests]
    C --> H[Job: integration-tests]
    D --> I[Job: deploy-staging]
    D --> J[Job: deploy-prod]
    E --> K[Step: checkout]
    E --> L[Step: dotnet build]
    E --> M[Step: publish artifact]
                            

The simplest possible pipeline skips stages and jobs entirely — Azure Pipelines creates implicit ones for you:

# The simplest possible pipeline — no explicit stages or jobs.
# Azure Pipelines wraps your steps in an implicit job and stage.
trigger:
  - main                    # Run when code is pushed to 'main' branch

pool:
  vmImage: 'ubuntu-latest'  # Use a Microsoft-hosted Ubuntu agent

steps:
  - script: echo "Hello, Azure Pipelines!"  # A single shell command
    displayName: 'Run a one-line script'     # Friendly name in the UI

When you need multiple stages (e.g., Build → Test → Deploy with approval gates between them), you spell out the full hierarchy:

# Full hierarchy — Pipeline with explicit Stages, Jobs, and Steps.
# Use this structure when you need approval gates between stages,
# different agent pools per job, or parallel jobs within a stage.
trigger:
  - main

stages:
  # ─── Stage 1: Build ───────────────────────────────────────
  - stage: Build
    displayName: 'Build & Package'
    jobs:
      - job: CompileApp
        displayName: 'Compile Application'
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - checkout: self           # Clone the repository
          - script: dotnet build     # Compile the code
            displayName: 'Build solution'

  # ─── Stage 2: Test ────────────────────────────────────────
  - stage: Test
    displayName: 'Run Tests'
    dependsOn: Build               # Only runs after Build succeeds
    jobs:
      - job: UnitTests
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: dotnet test
            displayName: 'Run unit tests'

  # ─── Stage 3: Deploy ──────────────────────────────────────
  - stage: Deploy
    displayName: 'Deploy to Production'
    dependsOn: Test                # Only runs after Test succeeds
    jobs:
      - job: DeployApp
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: echo "Deploying..."
            displayName: 'Deploy application'

Triggers & Scheduling

Triggers tell Azure Pipelines when to run your pipeline. Without triggers, your pipeline would never execute. Azure Pipelines supports five trigger types:

CI Triggers (Push)

The most common trigger — run the pipeline whenever code is pushed to specific branches:

# CI trigger — runs on push to these branches.
# Branch filters control WHICH pushes trigger the pipeline.
trigger:
  branches:
    include:
      - main               # Production branch
      - develop            # Integration branch
      - release/*          # All release branches (wildcard)
    exclude:
      - feature/experimental*  # Skip experimental features

  paths:
    include:
      - src/**             # Only trigger when source code changes
      - tests/**           # Or when tests change
    exclude:
      - docs/**            # Don't trigger for documentation-only changes
      - '*.md'             # Don't trigger for README updates

  batch: true              # If pushes come faster than builds finish,
                           # batch them into a single run (prevents queue flood)

PR Triggers (Pull Request Validation)

Run the pipeline when a pull request targets specific branches — this is your quality gate before merge:

# PR trigger — validates pull requests before they merge.
# This is SEPARATE from the CI trigger above.
pr:
  branches:
    include:
      - main               # Validate PRs targeting main
      - release/*          # Validate PRs targeting release branches
    exclude:
      - feature/draft-*    # Skip draft feature branches

  paths:
    include:
      - src/**             # Only validate when source changes
    exclude:
      - docs/**            # Skip docs-only PRs

  drafts: false            # Don't run on draft PRs (save agent time)

Scheduled Triggers (Cron)

Run the pipeline on a schedule — useful for nightly builds, weekly security scans, or periodic cleanup:

# Scheduled triggers — cron syntax (minute hour day-of-month month day-of-week).
# All times are UTC unless you specify a timezone.
schedules:
  - cron: '0 2 * * 1-5'          # Weekdays at 2:00 AM UTC
    displayName: 'Nightly build'
    branches:
      include:
        - main
    always: false                  # Only run if there are new changes
                                   # (set to true to run even without changes)

  - cron: '0 8 * * 0'            # Sundays at 8:00 AM UTC
    displayName: 'Weekly security scan'
    branches:
      include:
        - main
    always: true                   # Always run (security scans should be regular)

Pipeline Triggers (Chaining)

Trigger one pipeline when another completes — perfect for separating build from deploy or chaining microservice builds:

# Pipeline resource trigger — run THIS pipeline when ANOTHER finishes.
# Common pattern: separate build pipeline feeds into deploy pipeline.
resources:
  pipelines:
    - pipeline: buildPipeline           # Internal alias (your choice)
      source: 'MyApp-CI'                # Name of the triggering pipeline
      trigger:
        branches:
          include:
            - main                       # Only trigger from main branch builds
        stages:
          - Build                        # Only trigger when Build stage completes

Disabling Triggers

# Disable all automatic triggers — pipeline runs only manually.
# Useful for deployment pipelines that need human approval to start.
trigger: none
pr: none

Variables & Variable Groups

Variables store values you use throughout your pipeline — connection strings, version numbers, feature flags, file paths. Azure Pipelines offers multiple ways to define and consume variables, each with different scoping and security characteristics.

Inline Variables

The simplest approach — define variables directly in your YAML file:

# Inline variables — defined directly in the pipeline YAML.
# These are visible in version control (don't put secrets here!).
variables:
  buildConfiguration: 'Release'       # Simple name/value pair
  dotnetVersion: '8.0.x'              # .NET SDK version to install
  projectPath: 'src/MyApp/MyApp.csproj'  # Path to the project file

steps:
  - script: dotnet build $(projectPath) --configuration $(buildConfiguration)
    displayName: 'Build in $(buildConfiguration) mode'

Variable Groups (from Library)

Variable groups live in Azure DevOps Library and can be shared across multiple pipelines. They're ideal for environment-specific configuration:

# Variable groups — defined in Azure DevOps Library, referenced here.
# Go to Pipelines → Library → + Variable group to create them.
variables:
  - group: 'Production-Settings'     # Links to a variable group in Library
  - group: 'Shared-Credentials'      # Can link multiple groups
  - name: localVar                    # You can mix groups with inline vars
    value: 'hello'

Secret Variables

# Secret variables — marked as secret in Library or pipeline settings.
# They're masked in logs (replaced with ***) and can't be printed.
# You CANNOT define secrets inline in YAML — use Library or pipeline UI.
variables:
  - group: 'My-Secrets'              # Contains: dbPassword (secret), apiKey (secret)

steps:
  - script: |
      # Secret is available as environment variable but MASKED in logs.
      # This will print: "Password is ***"
      echo "Password is $(dbPassword)"
    displayName: 'Use secret variable'
    env:
      DB_PASSWORD: $(dbPassword)      # Map to environment variable for scripts

Predefined System Variables

Azure Pipelines provides dozens of built-in variables automatically. These are the most commonly used:

Variable Scoping

Variables can be scoped at different levels. A more specific scope overrides a broader one:

# Variable scoping — pipeline > stage > job.
# More specific scopes override broader ones.
variables:
  environment: 'development'          # Pipeline-level (default)

stages:
  - stage: Build
    variables:
      environment: 'ci'               # Stage-level overrides pipeline-level
    jobs:
      - job: CompileApp
        variables:
          environment: 'build'        # Job-level overrides stage-level
        steps:
          - script: echo $(environment)  # Prints: "build"
            displayName: 'Show environment'

  - stage: Deploy
    jobs:
      - job: DeployApp
        steps:
          - script: echo $(environment)  # Prints: "development" (pipeline-level)
            displayName: 'Show environment'

The Three Variable Syntaxes

This is the most confusing part of Azure Pipelines for newcomers. There are three different syntaxes, each evaluated at a different time:

# Demonstrating all three variable syntaxes:
variables:
  myVar: 'hello'

steps:
  # 1. Macro syntax — $(varName) — most common, evaluated at runtime
  - script: echo $(myVar)
    displayName: 'Macro syntax'

  # 2. Template expression — ${{ }} — evaluated at COMPILE time
  #    (before the pipeline even starts running)
  - ${{ if eq(variables.myVar, 'hello') }}:
    - script: echo "Compile-time check passed"
      displayName: 'Template expression'

  # 3. Runtime expression — $[ ] — used in conditions
  - script: echo "This runs conditionally"
    displayName: 'Runtime condition'
    condition: eq(variables.myVar, 'hello')

Parameters

Parameters are compile-time inputs that users provide when queueing a pipeline. Unlike variables (which can change at runtime), parameters are resolved before the pipeline starts — they become fixed values baked into the compiled YAML.

Why Parameters Instead of Variables?

• Type safety — parameters have types (string, number, boolean, object) and validation
• Allowed values — you can restrict inputs to a predefined list
• Default values — provide sensible defaults so users don't have to fill in everything
• UI integration — parameters appear as form fields when queueing manually

# Parameters — typed inputs resolved at queue/compile time.
# Users see these as form fields when they manually run the pipeline.
parameters:
  # String parameter with allowed values (dropdown in UI)
  - name: environment
    displayName: 'Target Environment'
    type: string
    default: 'staging'
    values:
      - development
      - staging
      - production

  # Boolean parameter (checkbox in UI)
  - name: runTests
    displayName: 'Run test suite?'
    type: boolean
    default: true

  # Number parameter
  - name: parallelism
    displayName: 'Test parallelism level'
    type: number
    default: 4

  # Object parameter (complex structured data)
  - name: deployRegions
    displayName: 'Regions to deploy to'
    type: object
    default:
      - eastus
      - westeurope
      - southeastasia

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  # Reference parameters with ${{ parameters.name }} syntax
  - script: echo "Deploying to ${{ parameters.environment }}"
    displayName: 'Deploy to ${{ parameters.environment }}'

  # Use boolean parameter in condition
  - ${{ if eq(parameters.runTests, true) }}:
    - script: dotnet test --parallel ${{ parameters.parallelism }}
      displayName: 'Run tests with parallelism=${{ parameters.parallelism }}'

  # Iterate over object parameter
  - ${{ each region in parameters.deployRegions }}:
    - script: echo "Deploying to ${{ region }}"
      displayName: 'Deploy → ${{ region }}'

Expressions & Functions

Expressions let you compute values, evaluate conditions, and transform data within your pipeline. Azure Pipelines provides a rich library of built-in functions:

Expression Syntax

• ${{ expression }} — Compile-time (template expressions) — evaluated when YAML is parsed
• $[ expression ] — Runtime — evaluated during pipeline execution

Key Functions

Status Check Functions

These functions check the outcome of previous steps/jobs — essential for cleanup and notification logic:

# Status functions — control flow based on previous step outcomes.
steps:
  - script: dotnet build
    displayName: 'Build application'

  - script: dotnet test
    displayName: 'Run tests'

  # succeeded() — only runs if ALL previous steps passed (DEFAULT behavior)
  - script: echo "Everything passed!"
    displayName: 'Success notification'
    condition: succeeded()

  # failed() — only runs if a previous step FAILED
  - script: |
      echo "Build or tests failed — sending alert"
      curl -X POST $(slackWebhook) -d '{"text":"Pipeline failed!"}'
    displayName: 'Failure notification'
    condition: failed()

  # always() — runs regardless of success/failure (cleanup, logging)
  - script: |
      echo "Cleaning up temporary files..."
      rm -rf ./temp
    displayName: 'Cleanup (always runs)'
    condition: always()

  # canceled() — runs only if the pipeline was manually canceled
  - script: echo "Pipeline was canceled by user"
    displayName: 'Cancellation handler'
    condition: canceled()

Conditions & Control Flow

Conditions let you control which steps, jobs, or stages execute based on variables, parameters, branch names, or previous outcomes. This is how you build pipelines that behave differently for feature branches vs main, or skip expensive steps during PR validation.

Conditions on Steps

# Conditions — control which steps execute.
variables:
  isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]

steps:
  # Always runs
  - script: dotnet build
    displayName: 'Build'

  # Only runs on main branch
  - script: dotnet publish -o ./output
    displayName: 'Publish artifacts'
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')

  # Only runs on feature branches (NOT main)
  - script: echo "Feature branch build — skipping publish"
    displayName: 'Feature branch notice'
    condition: ne(variables['Build.SourceBranch'], 'refs/heads/main')

  # Combining conditions with and() / or()
  - script: ./deploy.sh
    displayName: 'Deploy to production'
    condition: |
      and(
        succeeded(),
        eq(variables['Build.SourceBranch'], 'refs/heads/main'),
        ne(variables['Build.Reason'], 'PullRequest')
      )

Conditional Insertion with ${{ if }}

Template expressions let you conditionally include or exclude entire blocks of YAML at compile time. This is different from runtime conditions — the YAML is literally different depending on the parameter:

# Conditional insertion — compile-time if/else.
# The YAML is different depending on the parameter value.
parameters:
  - name: environment
    type: string
    default: 'staging'
    values: [development, staging, production]

stages:
  - stage: Build
    jobs:
      - job: Compile
        steps:
          - script: dotnet build
            displayName: 'Build'

  # This stage ONLY EXISTS in the compiled YAML if environment is 'production'
  - ${{ if eq(parameters.environment, 'production') }}:
    - stage: Approval
      jobs:
        - job: WaitForApproval
          pool: server            # Agentless job (runs on server)
          steps:
            - task: ManualValidation@1
              inputs:
                notifyUsers: 'release-managers@company.com'
                instructions: 'Please approve production deployment'

  - stage: Deploy
    jobs:
      - job: DeployApp
        steps:
          # Different script based on environment
          - ${{ if eq(parameters.environment, 'production') }}:
            - script: ./deploy.sh --env prod --region all
              displayName: 'Deploy to production (all regions)'
          - ${{ if eq(parameters.environment, 'staging') }}:
            - script: ./deploy.sh --env staging --region eastus
              displayName: 'Deploy to staging (single region)'
          - ${{ if eq(parameters.environment, 'development') }}:
            - script: ./deploy.sh --env dev --region local
              displayName: 'Deploy to development'

Resources

Resources let your pipeline access external items — other repositories, Docker containers, other pipelines, or packages. The most common use case is multi-repo checkout (when your code spans multiple repositories).

Repository Resources (Multi-Repo Checkout)

# Multi-repo checkout — access code from multiple repositories.
# By default, pipelines only check out the repo where the YAML lives.
resources:
  repositories:
    # Reference another repo in the same Azure DevOps project
    - repository: shared-templates     # Alias (your choice)
      type: git                        # Azure Repos Git
      name: MyProject/shared-templates # Project/RepoName
      ref: refs/heads/main             # Branch to check out

    # Reference a GitHub repository
    - repository: open-source-lib
      type: github
      name: myorg/open-source-lib      # GitHub org/repo
      endpoint: 'GitHub-ServiceConnection'  # Service connection name

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  # Check out the primary repo (where this YAML lives)
  - checkout: self
    path: 'main-app'

  # Check out the shared templates repo
  - checkout: shared-templates
    path: 'templates'

  # Check out the GitHub repo
  - checkout: open-source-lib
    path: 'lib'

  # Now you can access files from all three repos
  - script: |
      ls $(Pipeline.Workspace)/main-app
      ls $(Pipeline.Workspace)/templates
      ls $(Pipeline.Workspace)/lib
    displayName: 'List all checked-out repos'

Container Resources

Run your job inside a Docker container — useful when you need specific tools, runtimes, or a reproducible environment:

# Container resources — run jobs inside Docker containers.
# The job steps execute inside the container, not on the host agent.
resources:
  containers:
    - container: node-env
      image: 'node:20-alpine'          # Docker Hub image
    - container: dotnet-env
      image: 'mcr.microsoft.com/dotnet/sdk:8.0'  # Microsoft Container Registry

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: BuildNode
    container: node-env                 # This job runs inside node:20-alpine
    steps:
      - script: |
          node --version
          npm ci
          npm run build
        displayName: 'Build Node.js app'

  - job: BuildDotNet
    container: dotnet-env               # This job runs inside dotnet/sdk:8.0
    steps:
      - script: |
          dotnet --version
          dotnet build
        displayName: 'Build .NET app'

Your First Complete CI Pipeline

Let's build a production-ready CI pipeline for a .NET application from scratch. This pipeline restores dependencies, builds, runs tests, and publishes artifacts — everything a real project needs.

# ═══════════════════════════════════════════════════════════════
# Complete CI Pipeline — .NET Application
# File: azure-pipelines.yml (place in repository root)
# ═══════════════════════════════════════════════════════════════

# ─── Trigger: Run on push to main or release branches ────────
trigger:
  branches:
    include:
      - main
      - release/*
  paths:
    exclude:
      - docs/**              # Don't rebuild for docs-only changes
      - '*.md'               # Don't rebuild for README changes

# ─── PR Validation: Run on PRs targeting main ────────────────
pr:
  branches:
    include:
      - main
  drafts: false              # Skip draft PRs

# ─── Variables: Shared configuration ─────────────────────────
variables:
  buildConfiguration: 'Release'
  dotnetVersion: '8.0.x'
  solution: '**/*.sln'
  testProjects: '**/*Tests/*.csproj'
  publishProject: 'src/MyApp.Api/MyApp.Api.csproj'

# ─── Pool: Microsoft-hosted Ubuntu agent ─────────────────────
pool:
  vmImage: 'ubuntu-latest'

# ─── Steps: The actual work ──────────────────────────────────
steps:
  # Step 1: Check out source code
  - checkout: self
    fetchDepth: 0            # Full history (needed for versioning tools)
    clean: true              # Start with clean workspace every time

  # Step 2: Install the correct .NET SDK version
  - task: UseDotNet@2
    displayName: 'Install .NET SDK $(dotnetVersion)'
    inputs:
      packageType: 'sdk'
      version: '$(dotnetVersion)'

  # Step 3: Restore NuGet packages (downloads dependencies)
  - task: DotNetCoreCLI@2
    displayName: 'Restore NuGet packages'
    inputs:
      command: 'restore'
      projects: '$(solution)'
      feedsToUse: 'select'   # Use configured package feeds

  # Step 4: Build the solution in Release mode
  - task: DotNetCoreCLI@2
    displayName: 'Build solution ($(buildConfiguration))'
    inputs:
      command: 'build'
      projects: '$(solution)'
      arguments: '--configuration $(buildConfiguration) --no-restore'
      # --no-restore: Skip restore (we already did it in step 3)

  # Step 5: Run unit tests with code coverage
  - task: DotNetCoreCLI@2
    displayName: 'Run unit tests'
    inputs:
      command: 'test'
      projects: '$(testProjects)'
      arguments: >-
        --configuration $(buildConfiguration)
        --no-build
        --collect:"XPlat Code Coverage"
        --logger trx
        --results-directory $(Agent.TempDirectory)/TestResults
      # --no-build: Don't rebuild (we built in step 4)
      # --collect: Generate code coverage data
      # --logger trx: Output test results in TRX format for Azure DevOps

  # Step 6: Publish code coverage results to Azure DevOps
  - task: PublishCodeCoverageResults@2
    displayName: 'Publish code coverage'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/TestResults/**/coverage.cobertura.xml'

  # Step 7: Publish the application (create deployment package)
  # Only runs on main branch (not on PR builds — saves time)
  - task: DotNetCoreCLI@2
    displayName: 'Publish application'
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    inputs:
      command: 'publish'
      projects: '$(publishProject)'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'
      zipAfterPublish: true  # Zip the output for easier deployment

  # Step 8: Upload build artifacts to Azure DevOps
  # Only runs on main branch (artifact is only useful for deployment)
  - task: PublishBuildArtifacts@1
    displayName: 'Upload build artifacts'
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)'
      ArtifactName: 'drop'   # Name of the artifact (used in release pipeline)
      publishLocation: 'Container'

Debugging Pipelines

When pipelines fail (and they will), you need tools to diagnose what went wrong. Here are the most effective debugging techniques:

Enable Verbose Logging

# Method 1: Set System.Debug variable to get verbose logs.
# Add this temporarily when troubleshooting, then remove it.
variables:
  System.Debug: true         # Enables detailed diagnostic logging
                             # Shows every variable, every file path,
                             # every environment variable — very noisy!

steps:
  - script: dotnet build
    displayName: 'Build (with verbose logging enabled)'

# Method 2: Enable debug on a single step (less noisy).
steps:
  - script: |
      echo "Build.SourceBranch = $(Build.SourceBranch)"
      echo "Build.Reason = $(Build.Reason)"
      echo "Agent.OS = $(Agent.OS)"
      echo "Pipeline.Workspace = $(Pipeline.Workspace)"
      pwd
      ls -la
    displayName: 'Debug — print all context'

Common YAML Errors and Fixes

Validate YAML Before Pushing

# Validate your pipeline YAML locally before pushing.
# Option 1: Use the Azure DevOps REST API (requires PAT token)
az pipelines validate \
  --organization https://dev.azure.com/myorg \
  --project MyProject \
  --yaml-path azure-pipelines.yml

# Option 2: Use the VS Code extension "Azure Pipelines"
# It provides real-time YAML validation, IntelliSense,
# and schema checking as you type.

# Option 3: Use the "Validate" button in Azure DevOps UI
# Edit pipeline → click "Validate" in top-right → shows errors

Exercises