Prometheus Deep Dive Part 10: Remote Storage Systems

Remote Storage Overview

Why Remote Storage?

Prometheus’s local TSDB is optimized for recent data queries. It excels at last-few-hours dashboards but has inherent limitations for enterprise use cases:

Remote Write vs Remote Read

flowchart LR
    subgraph Prometheus
        TSDB[Local TSDB]
        WAL[WAL]
    end

    subgraph RemoteWrite["Remote Write (push)"]
        RW["Samples pushed
as they're ingested"]
    end

    subgraph RemoteRead["Remote Read (pull)"]
        RR["Queries proxied
at query time"]
    end

    subgraph LTS["Long-Term Store"]
        MIMIR[Grafana Mimir]
    end

    WAL -->|"continuous push"| RW --> MIMIR
    TSDB -.->|"query-time proxy"| RR -.-> MIMIR
                            

Solution Landscape

Grafana Mimir

Architecture & Components

flowchart TD
    subgraph Write["Write Path"]
        DIST[Distributor]
        ING[Ingester
x3 replicas]
    end

    subgraph Read["Read Path"]
        QF[Query Frontend]
        QS[Query Scheduler]
        Q[Querier]
    end

    subgraph Storage["Storage"]
        OBJ[(Object Store
S3/GCS/Azure)]
        SC[Store Gateway]
    end

    subgraph Ops["Operations"]
        COMP[Compactor]
        RUL[Ruler]
    end

    P[Prometheus
remote_write] -->|"push"| DIST
    DIST -->|"hash ring"| ING
    ING -->|"flush blocks"| OBJ
    GF[Grafana] --> QF --> QS --> Q
    Q --> ING
    Q --> SC --> OBJ
    COMP --> OBJ
    RUL --> Q
                            

Deployment (Monolithic & Microservices)

# Mimir monolithic mode — simplest deployment for <1M active series
# All components in a single binary
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mimir
  namespace: monitoring
spec:
  replicas: 3
  serviceName: mimir
  template:
    spec:
      containers:
        - name: mimir
          image: grafana/mimir:2.13.0
          args:
            - -target=all
            - -config.file=/etc/mimir/mimir.yaml
          ports:
            - containerPort: 8080    # HTTP API
            - containerPort: 9095    # gRPC (internal)
            - containerPort: 7946    # Memberlist gossip
          volumeMounts:
            - name: config
              mountPath: /etc/mimir
            - name: data
              mountPath: /data
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
            limits:
              memory: 12Gi
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi

# mimir.yaml — monolithic configuration
multitenancy_enabled: true

server:
  http_listen_port: 8080
  grpc_listen_port: 9095

distributor:
  ring:
    kvstore:
      store: memberlist

ingester:
  ring:
    kvstore:
      store: memberlist
    replication_factor: 3

blocks_storage:
  backend: s3
  s3:
    endpoint: s3.us-east-1.amazonaws.com
    bucket_name: mimir-blocks-prod
    region: us-east-1
  tsdb:
    dir: /data/tsdb
  bucket_store:
    sync_dir: /data/tsdb-sync

compactor:
  data_dir: /data/compactor
  sharding_ring:
    kvstore:
      store: memberlist

store_gateway:
  sharding_ring:
    kvstore:
      store: memberlist

ruler:
  alertmanager_url: http://alertmanager:9093
  rule_path: /data/rules

limits:
  # Per-tenant limits
  ingestion_rate: 100000                    # samples/sec per tenant
  ingestion_burst_size: 200000
  max_global_series_per_user: 5000000       # 5M series per tenant
  max_global_series_per_metric: 100000
  compactor_blocks_retention_period: 365d   # 1 year retention

memberlist:
  join_members:
    - mimir-0.mimir:7946
    - mimir-1.mimir:7946
    - mimir-2.mimir:7946

Multi-Tenancy

# Prometheus remote_write with tenant header
remote_write:
  - url: https://mimir.internal/api/v1/push
    headers:
      X-Scope-OrgID: payments-team    # Tenant identifier
    queue_config:
      max_samples_per_send: 2000
      batch_send_deadline: 5s

# Grafana datasource configuration per tenant
# Each team queries only their own data
apiVersion: 1
datasources:
  - name: Mimir-Payments
    type: prometheus
    url: https://mimir.internal/prometheus
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: payments-team

Configuration Deep Dive

# Per-tenant overrides (runtime configuration)
# File: /etc/mimir/runtime.yaml — hot-reloaded without restart
overrides:
  # Default limits for all tenants
  __default__:
    ingestion_rate: 50000
    max_global_series_per_user: 2000000
    compactor_blocks_retention_period: 90d

  # Override for high-volume tenant
  platform-team:
    ingestion_rate: 200000
    max_global_series_per_user: 10000000
    compactor_blocks_retention_period: 365d

  # Restricted tenant (dev environment)
  dev-team:
    ingestion_rate: 10000
    max_global_series_per_user: 500000
    compactor_blocks_retention_period: 14d

VictoriaMetrics

Architecture (Single & Cluster)

flowchart TD
    subgraph Insert["Insert Path"]
        VMI[vminsert
Stateless]
    end

    subgraph Store["Storage"]
        VMS1[vmstorage-0]
        VMS2[vmstorage-1]
        VMS3[vmstorage-2]
    end

    subgraph Query["Query Path"]
        VMSL[vmselect
Stateless]
    end

    P[Prometheus
remote_write] --> VMI
    VMI -->|"consistent hashing"| VMS1 & VMS2 & VMS3
    GF[Grafana] --> VMSL
    VMSL --> VMS1 & VMS2 & VMS3
                            

Deployment

# VictoriaMetrics single-node — handles up to 10M active series
# Simplest possible deployment
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: victoriametrics
  namespace: monitoring
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: victoriametrics
          image: victoriametrics/victoria-metrics:v1.101.0
          args:
            - -storageDataPath=/storage
            - -retentionPeriod=12     # 12 months
            - -httpListenAddr=:8428
            - -search.maxUniqueTimeseries=10000000
            - -dedup.minScrapeInterval=15s    # Dedup HA pairs
          ports:
            - containerPort: 8428
          volumeMounts:
            - name: storage
              mountPath: /storage
          resources:
            requests:
              cpu: "4"
              memory: 16Gi
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 500Gi

# Prometheus remote_write to VictoriaMetrics
remote_write:
  - url: http://victoriametrics:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 20000
      max_shards: 30

Unique Features (MetricsQL, Downsampling)

# MetricsQL — VictoriaMetrics' PromQL superset
# Supports additional functions not in standard PromQL

# Range over range — rate calculation that handles counter resets better
rate(http_requests_total[5m])        # Standard PromQL
increase(http_requests_total[5m])     # VM handles resets more accurately

# Rollup functions with explicit window
rollup_rate(http_requests_total, 5m)

# Label manipulation
label_set(metric, "env", "prod")
label_del(metric, "instance")
label_copy(metric, "pod", "pod_name")

# Running aggregations
running_sum(rate(requests_total[1m])[1h:])
running_avg(cpu_usage[1h:5m])

# Built-in downsampling at query time
# No pre-configuration needed — VM automatically uses appropriate resolution
# Query a year of data efficiently:
avg_over_time(cpu_usage[1y:1h])    # 1-hour step over 1 year

Remote Write Configuration

Queue Tuning & Reliability

# Production remote_write configuration
remote_write:
  - url: https://mimir.internal/api/v1/push
    headers:
      X-Scope-OrgID: production

    # Queue tuning for high-volume (>500K samples/sec)
    queue_config:
      capacity: 10000              # Per-shard buffer (default: 2500)
      max_shards: 50               # Max parallel writes (default: 200)
      min_shards: 10               # Min parallel writes (faster startup)
      max_samples_per_send: 5000   # Batch size (default: 2000)
      batch_send_deadline: 5s      # Max wait before partial batch send
      min_backoff: 30ms            # Initial retry backoff
      max_backoff: 5s              # Max retry backoff
      retry_on_http_429: true      # Retry on rate limiting

    # Metadata configuration
    metadata_config:
      send: true
      send_interval: 5m

    # TLS for encrypted transport
    tls_config:
      cert_file: /etc/certs/client.crt
      key_file: /etc/certs/client.key
      ca_file: /etc/certs/ca.crt

Write-Path Relabeling

# Selective remote write — reduce cost by filtering
remote_write:
  - url: https://mimir.internal/api/v1/push
    write_relabel_configs:
      # Only send recording rules and critical raw metrics
      - source_labels: [__name__]
        regex: '(namespace|job|cluster|slo):.*'
        action: keep

      # Drop high-cardinality debug metrics
      - source_labels: [__name__]
        regex: 'go_(gc|memstats)_.*'
        action: drop

      # Remove labels that only matter locally
      - regex: '__replica__|prometheus_replica'
        action: labeldrop

Exemplars Support

# Enable exemplars in remote write (links metrics to traces)
remote_write:
  - url: https://mimir.internal/api/v1/push
    send_exemplars: true    # Forward exemplars to backend

# Mimir configuration to accept exemplars
limits:
  max_exemplars_per_user: 100000    # Per-tenant exemplar limit

Mimir vs VictoriaMetrics

Feature Comparison

Operational Comparison

Decision Guide

Conclusion