Module 5: Data Sharing and Optimization

Job Artifacts — Uploading and Downloading

Artifacts are the primary mechanism for persisting data after a workflow run completes and for sharing files between jobs within the same workflow. Unlike the filesystem (which is wiped after each job), artifacts are uploaded to GitHub's storage and can be downloaded by subsequent jobs, other workflows, or manually via the UI.

The two core actions are actions/upload-artifact@v4 and actions/download-artifact@v4. Version 4 introduced immutable artifacts, faster uploads via concurrent chunking, and improved naming with automatic deduplication.

flowchart LR
    A[Job: Build] -->|upload-artifact| S[(GitHub Storage)]
    S -->|download-artifact| B[Job: Test]
    S -->|download-artifact| C[Job: Deploy]
    S -->|Manual Download| D[Developer via UI]
    A -->|build output| A1[dist/]
    A -->|test results| A2[coverage/]
    style S fill:#3B9797,color:#fff
    style A fill:#132440,color:#fff
    style B fill:#16476A,color:#fff
    style C fill:#16476A,color:#fff
                            

# Upload artifacts after a build
name: Build and Share Artifacts

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build application
        run: |
          npm ci
          npm run build

      - name: Upload build artifacts
        uses: actions/upload-artifact@v4
        with:
          name: dist-files
          path: dist/
          retention-days: 5
          if-no-files-found: error

      - name: Upload test coverage
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: |
            coverage/
            !coverage/.cache
          retention-days: 14

Key parameters for upload-artifact:

• name — Unique identifier for this artifact (must be unique within a workflow run)
• path — File, directory, or glob pattern to upload (supports multi-line for multiple paths)
• retention-days — Days to keep the artifact (1-90, default varies by plan: 90 for public, 30 for private)
• if-no-files-found — Behavior when no files match: warn (default), error, or ignore
• compression-level — 0 (no compression) to 9 (max), default 6. Use 0 for pre-compressed files like ZIPs
• overwrite — Whether to overwrite an existing artifact with the same name (default: false)

Downloading Artifacts

# Download artifacts in a subsequent job
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: app-build
          path: dist/

  test:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Download build artifacts
        uses: actions/download-artifact@v4
        with:
          name: app-build
          path: dist/

      - name: Run integration tests
        run: npm run test:integration

  deploy:
    needs: [build, test]
    runs-on: ubuntu-latest
    steps:
      - name: Download all artifacts
        uses: actions/download-artifact@v4
        # Omitting 'name' downloads ALL artifacts into separate dirs

      - name: List downloaded files
        run: |
          echo "Build artifacts:"
          ls -la app-build/

Artifact Naming Patterns and Retention

# Pattern: Named artifacts per matrix combination
jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node: [18, 20, 22]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci && npm test

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results-${{ matrix.os }}-node${{ matrix.node }}
          path: test-results/
          retention-days: 7

  report:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Download all test results
        uses: actions/download-artifact@v4
        with:
          pattern: test-results-*
          merge-multiple: true
          path: all-results/

      - name: Generate combined report
        run: npx test-reporter merge all-results/ --output summary.html

The merge-multiple: true option in v4 downloads all matching artifacts and merges their contents into a single directory — essential for collecting outputs from matrix jobs.

Caching Dependencies and Build Artifacts

While artifacts persist files for human consumption or cross-job data transfer, caching is designed for one purpose: speed. Caches store dependencies and build outputs that rarely change, eliminating redundant downloads and compilations across workflow runs.

The actions/cache@v4 action saves and restores a directory (or set of directories) keyed by a hash of your dependency files. On cache hit, the restore is near-instant (compared to minutes of npm install or pip install). On cache miss, the directory is saved at the end of the job for future runs.

Artifacts vs Cache — When to Use Each

Real-World Caching Examples

Node.js — Caching node_modules

# Node.js caching with npm
name: Node.js CI with Cache

on: push

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'  # Built-in caching! Handles keys automatically

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

Many setup-* actions have built-in caching (setup-node, setup-python, setup-java, setup-go). Use them when possible — they handle key generation automatically. For more control, use actions/cache@v4 directly:

# Manual cache configuration for Node.js
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache node_modules
        id: cache-npm
        uses: actions/cache@v4
        with:
          path: node_modules
          key: node-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            node-${{ runner.os }}-

      - name: Install dependencies
        if: steps.cache-npm.outputs.cache-hit != 'true'
        run: npm ci

      - name: Build
        run: npm run build

Python — Caching pip packages

# Python caching with pip
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: 'pip'  # Built-in pip caching

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest

# Advanced Python caching — cache the virtual environment
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Cache virtual environment
        id: cache-venv
        uses: actions/cache@v4
        with:
          path: .venv
          key: venv-${{ runner.os }}-py3.12-${{ hashFiles('requirements.txt', 'requirements-dev.txt') }}
          restore-keys: |
            venv-${{ runner.os }}-py3.12-

      - name: Create venv and install
        if: steps.cache-venv.outputs.cache-hit != 'true'
        run: |
          python -m venv .venv
          source .venv/bin/activate
          pip install -r requirements.txt -r requirements-dev.txt

      - name: Run tests
        run: |
          source .venv/bin/activate
          pytest --cov=src tests/

Docker — Caching Build Layers

# Docker build with layer caching
jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Cache Docker layers
        uses: actions/cache@v4
        with:
          path: /tmp/.buildx-cache
          key: docker-${{ runner.os }}-${{ hashFiles('Dockerfile', 'package-lock.json') }}
          restore-keys: |
            docker-${{ runner.os }}-

      - name: Build Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: false
          tags: myapp:latest
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

      # Workaround: move new cache to expected location
      - name: Rotate cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Cache Keys, Restore Keys, and Cache Management

The cache key is the identity of a cache entry. A well-designed key ensures cache hits when appropriate and cache misses when dependencies change. The hashFiles() function is the workhorse — it computes a SHA-256 hash of one or more files, creating a deterministic fingerprint of your dependency state.

# Cache key patterns — from most specific to broadest
- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/requirements*.txt', '**/setup.py', '**/pyproject.toml') }}
    restore-keys: |
      pip-${{ runner.os }}-${{ matrix.python-version }}-
      pip-${{ runner.os }}-
      pip-

How Key Matching Works

flowchart TD
    A[Cache Restore Step] --> B{Exact key match?}
    B -->|Yes| C[✅ Cache Hit - exact]
    B -->|No| D{Any restore-key prefix match?}
    D -->|Yes| E[⚠️ Partial Hit - stale cache]
    D -->|No| F[❌ Cache Miss]
    C --> G[Skip install step]
    E --> H[Restore partial + run install]
    F --> I[Full install from scratch]
    H --> J[Save new cache with exact key]
    I --> J
    style C fill:#3B9797,color:#fff
    style E fill:#BF092F,color:#fff
    style F fill:#132440,color:#fff
                            

The restore algorithm:

1. Exact match — If the key matches an existing cache entry exactly, restore it. The step output cache-hit is 'true'.
2. Prefix match via restore-keys — If no exact match, try each restore-keys entry (in order) as a prefix. The most recent cache matching that prefix is restored. cache-hit is 'false'.
3. Cache miss — No matches at all. The path is empty. At the end of the job, a new cache entry is saved with the exact key.

Restore Key Fallback Chains

# Layered restore keys — progressively broader
- uses: actions/cache@v4
  with:
    path: |
      ~/.gradle/caches
      ~/.gradle/wrapper
    key: gradle-${{ runner.os }}-jdk${{ matrix.java }}-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
    restore-keys: |
      gradle-${{ runner.os }}-jdk${{ matrix.java }}-
      gradle-${{ runner.os }}-

The strategy is specificity layering:

• Level 1 (exact key): OS + JDK version + dependency hash — perfect match, no install needed
• Level 2 (first restore-key): OS + JDK version — same environment, but dependencies changed. Restore old cache, then incremental install adds/updates only changed packages
• Level 3 (second restore-key): OS only — different JDK version, but most Gradle wrapper/plugin caches still valid

Cache Size Limits and Eviction Policy

GitHub enforces a 10 GB total cache limit per repository. Understanding the eviction policy prevents unexpected cache misses:

• 10 GB total per repository — across all branches and workflows
• Individual cache entry limit — no hard limit per entry, but entries over 1 GB are slow to restore
• Eviction: LRU after 7 days — entries not accessed in 7 days are eligible for eviction. When the 10 GB limit is reached, the least recently used caches are deleted first
• Branch deletion — caches scoped to a deleted branch are removed

# Monitor cache usage with the GitHub CLI
# List all caches for a repository
# gh actions-cache list --repo owner/repo --sort size

# Delete specific caches by key pattern
# gh actions-cache delete "node-Linux-" --repo owner/repo

# Workflow to clean old caches on PR close
name: Cache Cleanup
on:
  pull_request:
    types: [closed]

jobs:
  cleanup:
    runs-on: ubuntu-latest
    permissions:
      actions: write
    steps:
      - name: Cleanup branch caches
        uses: actions/github-script@v7
        with:
          script: |
            const branch = context.payload.pull_request.head.ref;
            const caches = await github.rest.actions.getActionsCacheList({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: `refs/heads/${branch}`
            });
            for (const cache of caches.data.actions_caches) {
              await github.rest.actions.deleteActionsCacheById({
                owner: context.repo.owner,
                repo: context.repo.repo,
                cache_id: cache.id
              });
            }
            console.log(`Deleted ${caches.data.total_count} caches for branch ${branch}`);

Dynamic Matrices

Static matrices work great when your test dimensions are fixed. But real-world scenarios often require dynamic configuration — generating the matrix at runtime based on changed files, API responses, or conditional logic. The pattern uses fromJSON() to parse a JSON string from a step output into a matrix array.

# Dynamic matrix from step output
name: Dynamic Matrix

on: push

jobs:
  prepare:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4

      - name: Determine matrix
        id: set-matrix
        run: |
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            # Full matrix on main
            echo 'matrix={"node-version": [18, 20, 22], "os": ["ubuntu-latest", "windows-latest", "macos-latest"]}' >> $GITHUB_OUTPUT
          else
            # Minimal matrix on feature branches
            echo 'matrix={"node-version": [20], "os": ["ubuntu-latest"]}' >> $GITHUB_OUTPUT
          fi

  test:
    needs: prepare
    strategy:
      matrix: ${{ fromJSON(needs.prepare.outputs.matrix) }}
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci && npm test

The fromJSON() function converts a JSON string into a GitHub Actions object. The JSON must be a valid matrix definition — an object where each key maps to an array of values.

Dynamic Matrix Based on Changed Files

# Only test packages that changed (monorepo pattern)
name: Selective Testing

on:
  pull_request:
    paths:
      - 'packages/**'

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      packages: ${{ steps.changes.outputs.packages }}
      has-changes: ${{ steps.changes.outputs.has-changes }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Detect changed packages
        id: changes
        run: |
          CHANGED=$(git diff --name-only origin/main...HEAD | grep '^packages/' | cut -d/ -f2 | sort -u | jq -R -s -c 'split("\n") | map(select(. != ""))')
          echo "packages=${CHANGED}" >> $GITHUB_OUTPUT
          if [ "$CHANGED" = "[]" ]; then
            echo "has-changes=false" >> $GITHUB_OUTPUT
          else
            echo "has-changes=true" >> $GITHUB_OUTPUT
          fi

  test:
    needs: detect-changes
    if: needs.detect-changes.outputs.has-changes == 'true'
    strategy:
      matrix:
        package: ${{ fromJSON(needs.detect-changes.outputs.packages) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Test ${{ matrix.package }}
        run: |
          cd packages/${{ matrix.package }}
          npm ci && npm test

Dynamic Include and Exclude

# Generate matrix with include entries dynamically
name: Dynamic Include Matrix

on:
  workflow_dispatch:
    inputs:
      environments:
        description: 'Target environments (JSON array)'
        default: '["staging", "production"]'

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.build-matrix.outputs.matrix }}
    steps:
      - name: Build deployment matrix
        id: build-matrix
        uses: actions/github-script@v7
        with:
          script: |
            const envs = JSON.parse('${{ github.event.inputs.environments }}');
            const matrix = { include: [] };

            for (const env of envs) {
              matrix.include.push({
                environment: env,
                url: env === 'production'
                  ? 'https://app.example.com'
                  : `https://${env}.app.example.com`,
                replicas: env === 'production' ? 3 : 1
              });
            }

            core.setOutput('matrix', JSON.stringify(matrix));

  deploy:
    needs: setup
    strategy:
      matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}
    runs-on: ubuntu-latest
    environment:
      name: ${{ matrix.environment }}
      url: ${{ matrix.url }}
    steps:
      - name: Deploy to ${{ matrix.environment }}
        run: |
          echo "Deploying to ${{ matrix.url }} with ${{ matrix.replicas }} replicas"

Matrix Strategies: Include, Exclude, and Handling Failures

The matrix strategy generates a Cartesian product of all defined arrays, then allows fine-tuning with include (add specific combinations), exclude (remove specific combinations), and controls for failure handling.

# Complete matrix strategy example
name: Full Matrix CI

on: [push, pull_request]

jobs:
  test:
    strategy:
      fail-fast: false
      max-parallel: 6
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node: [18, 20, 22]
        exclude:
          # Node 18 EOL soon, skip on macOS to save resources
          - os: macos-latest
            node: 18
        include:
          # Add extra config for specific combinations
          - os: ubuntu-latest
            node: 22
            coverage: true
            experimental: false
          # Test nightly Node on ubuntu only
          - os: ubuntu-latest
            node: 23
            experimental: true

    runs-on: ${{ matrix.os }}
    continue-on-error: ${{ matrix.experimental || false }}

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: 'npm'

      - run: npm ci
      - run: npm test

      - name: Upload coverage
        if: matrix.coverage
        uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.os }}-node${{ matrix.node }}
          path: coverage/

Fail-Fast and Max-Parallel

Two strategy-level controls affect matrix execution behavior:

• fail-fast: true (default) — If any matrix job fails, all other in-progress jobs in the matrix are cancelled immediately. Fast feedback, but you might miss failures on other platforms.
• fail-fast: false — All matrix jobs run to completion regardless of individual failures. Essential for comprehensive test reports.
• max-parallel — Maximum number of matrix jobs running simultaneously. Use to limit resource consumption or avoid overwhelming external services (e.g., a shared test database).

# Controlled parallelism for database-dependent tests
jobs:
  integration-tests:
    strategy:
      fail-fast: false
      max-parallel: 2  # Only 2 jobs hit the shared DB at once
      matrix:
        test-suite: [auth, billing, inventory, notifications, reporting, analytics]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run ${{ matrix.test-suite }} tests
        run: npm run test:integration -- --suite=${{ matrix.test-suite }}
        env:
          DATABASE_URL: ${{ secrets.SHARED_TEST_DB_URL }}

Complex Matrix Examples

Multi-Dimensional with Include Overrides

# Cross-platform build with architecture-specific settings
jobs:
  build:
    strategy:
      matrix:
        include:
          - os: ubuntu-latest
            target: x86_64-unknown-linux-gnu
            artifact-name: myapp-linux-amd64
            cross: false
          - os: ubuntu-latest
            target: aarch64-unknown-linux-gnu
            artifact-name: myapp-linux-arm64
            cross: true
          - os: windows-latest
            target: x86_64-pc-windows-msvc
            artifact-name: myapp-windows-amd64.exe
            cross: false
          - os: macos-latest
            target: x86_64-apple-darwin
            artifact-name: myapp-macos-amd64
            cross: false
          - os: macos-latest
            target: aarch64-apple-darwin
            artifact-name: myapp-macos-arm64
            cross: false

    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4

      - name: Install cross-compilation tools
        if: matrix.cross
        run: cargo install cross

      - name: Build
        run: |
          if [ "${{ matrix.cross }}" = "true" ]; then
            cross build --release --target ${{ matrix.target }}
          else
            cargo build --release --target ${{ matrix.target }}
          fi
        shell: bash

      - name: Upload binary
        uses: actions/upload-artifact@v4
        with:
          name: ${{ matrix.artifact-name }}
          path: target/${{ matrix.target }}/release/myapp*

Service Container Matrix

# Test against multiple database versions
jobs:
  test:
    strategy:
      matrix:
        db:
          - engine: postgres
            version: '14'
            port: 5432
          - engine: postgres
            version: '16'
            port: 5432
          - engine: mysql
            version: '8.0'
            port: 3306

    runs-on: ubuntu-latest
    services:
      database:
        image: ${{ matrix.db.engine }}:${{ matrix.db.version }}
        env:
          POSTGRES_PASSWORD: ${{ matrix.db.engine == 'postgres' && 'testpass' || '' }}
          MYSQL_ROOT_PASSWORD: ${{ matrix.db.engine == 'mysql' && 'testpass' || '' }}
        ports:
          - ${{ matrix.db.port }}:${{ matrix.db.port }}
        options: >-
          --health-cmd "${{ matrix.db.engine == 'postgres' && 'pg_isready' || 'mysqladmin ping' }}"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v4
      - name: Run tests against ${{ matrix.db.engine }} ${{ matrix.db.version }}
        run: npm run test:db
        env:
          DB_ENGINE: ${{ matrix.db.engine }}
          DB_PORT: ${{ matrix.db.port }}
          DB_PASSWORD: testpass

Exercises