Module 8: Azure Test Plans & Quality Gates

Why Structured Testing Matters

In Modules 4–7, you built pipelines that compile, test, and deploy code automatically. But here's the gap those automated pipelines can't fill alone: how do you know the application actually works from a user's perspective? How do you prove to auditors that every requirement was tested? How do you capture the bugs that automated tests never anticipated?

Azure Test Plans bridges this gap — it provides structured test management that complements your automated test suites with manual verification, exploratory discovery, and complete traceability.

Manual vs Automated Testing — Both Have a Place

The Traceability Chain

The ultimate goal of structured testing is traceability — the ability to answer "was this requirement tested, and did it pass?" for any requirement at any time:

flowchart LR
    A[User Story] --> B[Test Case]
    B --> C[Test Run]
    C --> D{Pass?}
    D -->|Yes| E[Verified ✓]
    D -->|No| F[Bug Created]
    F --> G[Fix Committed]
    G --> H[Re-test]
    H --> E
                            

This chain is critical in regulated industries (healthcare, finance, government) where you must demonstrate that every requirement has been verified. But even for startups, traceability means fewer "I thought someone tested that" incidents.

For testing theory and methodology fundamentals, see Part 18: Testing Fundamentals.

Test Plans, Suites & Cases

Azure Test Plans organizes testing into a three-level hierarchy. Understanding this hierarchy is essential before you create anything:

The Hierarchy

Test Suite Types

Azure DevOps offers three types of test suites, each serving a different purpose:

• Static Suite — Manually curated collection of test cases. You drag-and-drop cases into the suite. Best for ad-hoc groupings or regression packs.
• Requirement-based Suite — Automatically linked to a work item (User Story, Feature). Every test case in the suite traces back to that requirement. This is the most powerful type for traceability.
• Query-based Suite — Populated dynamically by a work item query. For example, "all test cases tagged 'smoke-test'" or "all test cases modified this sprint." Great for creating a nightly regression suite that auto-updates.

flowchart TD
    TP[Test Plan: Sprint 14] --> RS1[Requirement Suite: US-101 Login]
    TP --> RS2[Requirement Suite: US-102 Payment]
    TP --> SS[Static Suite: Regression Pack]
    TP --> QS[Query Suite: Smoke Tests]
    RS1 --> TC1[TC: Valid login]
    RS1 --> TC2[TC: Invalid password]
    RS1 --> TC3[TC: Account locked]
    RS2 --> TC4[TC: Credit card payment]
    RS2 --> TC5[TC: PayPal checkout]
    SS --> TC6[TC: Homepage loads]
    SS --> TC1
    QS --> TC6
    QS --> TC1
                            

Test Case Anatomy

A well-written test case has clear steps, expected results, and optional parameters for data-driven testing:

# Test Case: Verify login with valid credentials (TC-001)
# This is a conceptual representation — you create these in the Azure DevOps UI

Title: Verify login with valid credentials
Area Path: MyProject\Login Module
Priority: 1 (Critical)
State: Ready

Steps:
  1. Navigate to https://myapp.com/login
     Expected: Login page displays with email and password fields

  2. Enter valid email address
     Expected: Email field accepts input without error

  3. Enter valid password
     Expected: Password field masks input

  4. Click "Sign In" button
     Expected: User is redirected to dashboard within 3 seconds

  5. Verify user name appears in top-right corner
     Expected: "Welcome, [User Name]" text is displayed

# Parameters (for data-driven execution):
Parameters:
  - @email: admin@contoso.com, user@contoso.com, new.user@contoso.com
  - @password: P@ssw0rd123, UserPass!456, NewUser789!
  - @expectedName: Admin User, Standard User, New User

# Shared Steps reference (reusable across test cases):
Shared Step "Navigate to Login":
  1. Open browser
  2. Navigate to $(BaseUrl)/login
  3. Wait for page to fully load

Configurations

Configurations let you run the same test case across multiple OS/browser/device combinations without duplicating test cases:

A single test case with 3 OS × 4 browser configurations generates 12 test points — 12 unique combinations to execute and track independently.

Manual Test Execution

Once your test plan structure is in place, it's time to execute. Azure DevOps provides two runners for manual test execution:

• Web Runner — Browser-based, works everywhere, no installation needed
• Desktop Runner — Windows application with richer capture capabilities (video recording, advanced annotations)

Test Outcomes

Each test step (and the overall test case) receives one of four outcomes:

Creating Bugs from Failed Steps

The killer feature of the test runner: when a step fails, you click "Create bug" and Azure DevOps automatically populates:

• Repro steps — All steps executed so far (with pass/fail status)
• System info — Browser, OS, screen resolution
• Attachments — Screenshots captured during execution
• Links — Back to the test case, test run, and user story

This means developers receive perfectly documented bugs with exact reproduction steps — no more "it doesn't work" tickets with zero context.

Managing Test Runs with Azure CLI

# List test plans in a project
az devops invoke \
  --area testplan \
  --resource plans \
  --org https://dev.azure.com/myorg \
  --project MyProject \
  --http-method GET

# Query test results for a specific test run
az devops invoke \
  --area test \
  --resource runs \
  --route-parameters runId=12345 \
  --org https://dev.azure.com/myorg \
  --project MyProject \
  --http-method GET

# Get test points (which test cases to run in which configuration)
az devops invoke \
  --area testplan \
  --resource points \
  --route-parameters planId=100 suiteId=200 \
  --org https://dev.azure.com/myorg \
  --project MyProject \
  --http-method GET

Test Progress Charts

Azure Test Plans provides built-in charts showing test progress over time — how many test cases are passing, failing, or not yet run. Add these to your dashboard to give stakeholders visibility into testing progress without needing Test Plans access.

Exploratory Testing

Scripted tests verify what you expect. Exploratory testing discovers what you didn't expect. It's session-based, unscripted testing where a skilled tester explores the application with a mission but without predetermined steps.

Test & Feedback Extension

The Azure Test & Feedback browser extension (Chrome/Edge) is the primary tool for exploratory testing. It captures everything you do during a session:

• Screenshots — Annotated with drawing tools and text
• Screen recordings — Video of your exploration session
• Notes — Free-text observations as you explore
• Bugs — Created directly with captured evidence auto-attached
• Tasks — Follow-up actions ("investigate this edge case further")
• Timeline — Automatic log of pages visited and actions taken

Session Management

A well-structured exploratory session has three elements:

1. Charter — "Explore the checkout flow focusing on error handling when payment fails"
2. Timebox — 30–90 minutes (prevents fatigue and scope creep)
3. Stakeholder — Link the session to a Feature or User Story for traceability

When to Use Exploratory vs Scripted

Test Automation Integration

Azure Test Plans doesn't replace automated testing — it integrates with it. Your pipelines run automated tests (unit, integration, E2E) and publish results back to Azure DevOps, creating a unified view of both manual and automated test health.

Publishing Test Results from Pipelines

# Publishing test results from different frameworks
# Each framework produces results in a specific format

# .NET — produces TRX format (VSTest)
steps:
  - script: dotnet test --logger trx --results-directory $(Agent.TempDirectory)
    displayName: 'Run .NET tests'

  - task: PublishTestResults@2
    displayName: 'Publish .NET test results'
    inputs:
      testResultsFormat: 'VSTest'          # Format: VSTest, JUnit, NUnit, xUnit, CTest
      testResultsFiles: '**/*.trx'         # Glob pattern to find result files
      searchFolder: '$(Agent.TempDirectory)'
      mergeTestResults: true               # Merge results from multiple files
      testRunTitle: 'Unit Tests (.NET)'    # Display name in Test Runs tab
    condition: always()                    # Publish even if tests fail

# Java/Maven — produces JUnit XML format
  - script: mvn test
    displayName: 'Run Java tests'

  - task: PublishTestResults@2
    displayName: 'Publish Java test results'
    inputs:
      testResultsFormat: 'JUnit'
      testResultsFiles: '**/surefire-reports/TEST-*.xml'
      testRunTitle: 'Unit Tests (Java)'
    condition: always()

# JavaScript/Node — Jest with JUnit reporter
  - script: npx jest --ci --reporters=default --reporters=jest-junit
    displayName: 'Run Jest tests'
    env:
      JEST_JUNIT_OUTPUT_DIR: '$(Agent.TempDirectory)/jest-results'

  - task: PublishTestResults@2
    displayName: 'Publish Jest test results'
    inputs:
      testResultsFormat: 'JUnit'
      testResultsFiles: '$(Agent.TempDirectory)/jest-results/*.xml'
      testRunTitle: 'Unit Tests (Jest)'
    condition: always()

# Python — pytest with JUnit XML output
  - script: pytest --junitxml=$(Agent.TempDirectory)/pytest-results.xml
    displayName: 'Run Python tests'

  - task: PublishTestResults@2
    displayName: 'Publish Python test results'
    inputs:
      testResultsFormat: 'JUnit'
      testResultsFiles: '$(Agent.TempDirectory)/pytest-results.xml'
      testRunTitle: 'Unit Tests (Python)'
    condition: always()

Linking Automated Tests to Test Cases

You can link an automated test method to a manual test case in Azure Test Plans. This means:

• The test case shows "Automated" status (no manual execution needed)
• Pipeline test results automatically update the test case outcome
• Requirements traceability works seamlessly for automated tests too

To link: open the test case → Associated Automation tab → select the automated test method by its fully-qualified name (e.g., MyProject.Tests.LoginTests.ValidLogin_ShouldRedirectToDashboard).

Flaky Test Detection

Azure DevOps automatically detects flaky tests — tests that pass and fail intermittently on the same code. When a test is marked flaky:

• It's flagged in the test results with a ⚠️ icon
• Pipeline pass/fail decisions can optionally exclude flaky test failures
• A "Flaky Tests" report shows which tests need stabilization

Enable flaky test detection in Project Settings → Pipelines → Test Management → Flaky test detection.

Code Coverage & Quality Gates

Code coverage measures what percentage of your source code is exercised by automated tests. It's not a measure of test quality, but it's a useful indicator of test completeness.

# Publishing code coverage from different frameworks
# Coverage data is displayed on the pipeline run summary

# .NET — Cobertura format via coverlet
steps:
  - script: |
      dotnet test \
        --collect:"XPlat Code Coverage" \
        --results-directory $(Agent.TempDirectory)/coverage
    displayName: 'Run tests with coverage'

  - task: PublishCodeCoverageResults@2
    displayName: 'Publish code coverage'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/coverage/**/coverage.cobertura.xml'
      # Supported formats: Cobertura, JaCoCo
      # Cobertura is the most widely supported (C#, Python, JS, Go)

# Java — JaCoCo format via Maven
  - script: mvn verify  # JaCoCo plugin runs during verify phase
    displayName: 'Run tests with JaCoCo coverage'

  - task: PublishCodeCoverageResults@2
    displayName: 'Publish Java coverage'
    inputs:
      summaryFileLocation: '**/site/jacoco/jacoco.xml'

# Python — Cobertura via pytest-cov
  - script: |
      pytest \
        --cov=src \
        --cov-report=xml:$(Agent.TempDirectory)/coverage.xml \
        --cov-report=html:$(Agent.TempDirectory)/htmlcov
    displayName: 'Run Python tests with coverage'

  - task: PublishCodeCoverageResults@2
    displayName: 'Publish Python coverage'
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/coverage.xml'

Setting Coverage Thresholds as Quality Gates

# Pipeline that fails if code coverage drops below threshold
# This prevents merging PRs that reduce overall coverage

trigger:
  branches:
    include: [main]

pool:
  vmImage: 'ubuntu-latest'

steps:
  - script: dotnet test --collect:"XPlat Code Coverage" --results-directory $(Agent.TempDirectory)/coverage
    displayName: 'Run tests with coverage'

  - task: PublishCodeCoverageResults@2
    inputs:
      summaryFileLocation: '$(Agent.TempDirectory)/coverage/**/coverage.cobertura.xml'

  # Parse coverage and enforce threshold
  - script: |
      # Extract line coverage percentage from Cobertura XML
      COVERAGE=$(python3 -c "
      import xml.etree.ElementTree as ET
      tree = ET.parse('$(Agent.TempDirectory)/coverage/*/coverage.cobertura.xml')
      root = tree.getroot()
      line_rate = float(root.attrib['line-rate']) * 100
      print(f'{line_rate:.1f}')
      ")
      echo "Code coverage: ${COVERAGE}%"
      echo "##vso[task.setvariable variable=codeCoverage]${COVERAGE}"

      # Fail if below threshold
      THRESHOLD=80
      if (( $(echo "$COVERAGE < $THRESHOLD" | bc -l) )); then
        echo "##vso[task.logissue type=error]Coverage ${COVERAGE}% is below threshold ${THRESHOLD}%"
        echo "##vso[task.complete result=Failed;]Coverage gate failed"
        exit 1
      fi
      echo "✅ Coverage ${COVERAGE}% meets threshold ${THRESHOLD}%"
    displayName: 'Enforce coverage threshold (80%)'

Branch Policy Integration

For maximum enforcement, combine coverage thresholds with branch policies (covered in Module 3):

1. Create a build validation policy on your main branch
2. The validation pipeline includes the coverage threshold check above
3. PRs that reduce coverage below the threshold cannot be merged

This creates a ratchet effect — coverage can only go up, never down.

Quality Gates in Pipelines

Quality gates are automated checkpoints that prevent bad code from progressing through your pipeline. They transform your pipeline from "build and deploy" into "build, prove quality, then deploy."

Types of Quality Gates

Complete Pipeline with Quality Gates

# Production-grade pipeline with multiple quality gates
# Each gate must pass before the next stage proceeds

trigger:
  branches:
    include: [main]

variables:
  - name: minCoverage
    value: 80
  - name: maxCriticalIssues
    value: 0

stages:
  # ─── STAGE 1: BUILD & UNIT TEST ─────────────────────────
  - stage: Build
    displayName: 'Build & Unit Test'
    jobs:
      - job: BuildAndTest
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: dotnet build --configuration Release
            displayName: 'Build'

          - script: |
              dotnet test --configuration Release \
                --logger trx \
                --collect:"XPlat Code Coverage" \
                --results-directory $(Agent.TempDirectory)/results
            displayName: 'Run unit tests with coverage'

          - task: PublishTestResults@2
            inputs:
              testResultsFormat: 'VSTest'
              testResultsFiles: '**/*.trx'
              searchFolder: '$(Agent.TempDirectory)/results'
            condition: always()

          - task: PublishCodeCoverageResults@2
            inputs:
              summaryFileLocation: '$(Agent.TempDirectory)/results/**/coverage.cobertura.xml'

          # QUALITY GATE: Coverage threshold
          - script: |
              COVERAGE=$(python3 -c "
              import xml.etree.ElementTree as ET, glob
              files = glob.glob('$(Agent.TempDirectory)/results/**/coverage.cobertura.xml', recursive=True)
              tree = ET.parse(files[0])
              print(f\"{float(tree.getroot().attrib['line-rate'])*100:.1f}\")
              ")
              echo "Coverage: ${COVERAGE}%"
              if (( $(echo "$COVERAGE < $(minCoverage)" | bc -l) )); then
                echo "##vso[task.logissue type=error]Coverage ${COVERAGE}% < $(minCoverage)%"
                exit 1
              fi
            displayName: 'Gate: Coverage ≥ $(minCoverage)%'

  # ─── STAGE 2: STATIC ANALYSIS (SonarCloud) ──────────────
  - stage: StaticAnalysis
    displayName: 'Static Analysis'
    dependsOn: Build
    jobs:
      - job: SonarScan
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: SonarCloudPrepare@2
            inputs:
              SonarCloud: 'SonarCloud-Connection'
              organization: 'myorg'
              scannerMode: 'dotnet'
              projectKey: 'myorg_myproject'
              extraProperties: |
                sonar.cs.opencover.reportsPaths=$(Agent.TempDirectory)/results/**/coverage.cobertura.xml

          - script: dotnet build
            displayName: 'Build for analysis'

          - task: SonarCloudAnalyze@2
            displayName: 'Run SonarCloud analysis'

          # QUALITY GATE: SonarCloud quality gate status
          - task: SonarCloudPublish@2
            displayName: 'Publish SonarCloud results'

          - script: |
              # Check SonarCloud quality gate via API
              STATUS=$(curl -s -u "$SONAR_TOKEN:" \
                "https://sonarcloud.io/api/qualitygates/project_status?projectKey=myorg_myproject" \
                | python3 -c "import sys,json; print(json.load(sys.stdin)['projectStatus']['status'])")
              echo "SonarCloud Quality Gate: $STATUS"
              if [ "$STATUS" != "OK" ]; then
                echo "##vso[task.logissue type=error]SonarCloud quality gate FAILED"
                exit 1
              fi
            displayName: 'Gate: SonarCloud quality gate'
            env:
              SONAR_TOKEN: $(SonarToken)

  # ─── STAGE 3: DEPLOY (only if all gates pass) ───────────
  - stage: Deploy
    displayName: 'Deploy to Staging'
    dependsOn: [Build, StaticAnalysis]
    condition: succeeded()
    jobs:
      - deployment: deployApp
        pool:
          vmImage: 'ubuntu-latest'
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "All quality gates passed — deploying!"
                  displayName: 'Deploy application'

End-to-End Traceability

Traceability is the thread that connects every piece of your delivery pipeline — from the initial requirement through development, testing, building, and deploying. Azure DevOps provides this traceability automatically when you link work items, test cases, commits, and pipelines correctly.

flowchart TD
    F[Feature: User Authentication] --> US[User Story: Login with Email]
    US --> TC[Test Case: Verify login]
    US --> C[Commit: Implement login API]
    C --> PR["Pull Request #142"]
    PR --> BV[Build Validation Pipeline]
    BV --> TR[Test Run: 45 passed, 0 failed]
    TC --> TR
    PR --> B["Build #891"]
    B --> D[Deploy to Staging]
    D --> MT[Manual Test Run: 12 passed]
    MT --> P[Deploy to Production]
                            

Requirements Coverage Report

The Requirements Quality widget (available on dashboards) shows how many of your requirements have associated test cases, and of those, how many are passing:

• Requirements with tests — How many user stories have linked test cases
• Requirements passing — Of those with tests, how many have all tests passing
• Requirements failing — User stories where at least one test is failing
• Requirements not tested — Stories with zero linked tests (the dangerous gaps)

Test Gap Analysis

A test gap is a requirement with no associated test cases. Azure DevOps makes these visible through:

• Requirement-based suites — Empty suites highlight untested stories
• Board annotations — Work items show test status directly on the board
• Queries — "User stories with zero test cases" query to find gaps

Compliance Evidence for Audits

For regulated industries, traceability isn't optional — it's a legal requirement. Azure DevOps provides audit-ready evidence:

• Test run results — Timestamped records of who ran which tests, when, with what outcome
• Change history — Every modification to test cases is logged
• Deployment history — Which build deployed to production, linked to which test runs
• Export capabilities — Export test plans/results to PDF/Excel for external auditors

Inline Tests & Quick Testing

Not every team needs the full Azure Test Plans experience. For teams on the Basic plan (without Test Plans license), Azure DevOps offers inline tests — lightweight test tracking directly on the board.

Creating Inline Tests from User Stories

On the Kanban board, hover over a User Story card and click "+ Add Test". This creates a simple test case linked to that story — no Test Plans license required. You can mark tests as passed/failed directly from the board.

When Inline Tests Are Sufficient

Recommendation: Start with inline tests. When you find yourself needing test runners, exploratory sessions, or compliance reports, upgrade to the full Test Plans license. The upgrade is additive — your inline tests remain linked.

Case Study: Regulated Industry Testing

Exercises