Prometheus Deep Dive Part 15: OpenTelemetry & the Future of Prometheus

The Convergence

Prometheus and OpenTelemetry started as separate projects with different models — pull vs push, metrics-focused vs multi-signal. Today, they’re converging:

timeline
    title Prometheus + OpenTelemetry Convergence
    2012 : Prometheus created at SoundCloud
    2015 : Prometheus open-sourced
    2019 : OpenTelemetry formed (OpenTracing + OpenCensus merge)
    2021 : OTel Metrics GA
         : Prometheus remote_write receiver
    2023 : Native Histograms (experimental)
         : OTLP write receiver (experimental)
    2024 : Prometheus 3.0 released
         : Native OTLP ingestion (stable)
         : UTF-8 metric names
    2025 : OpenTelemetry Collector replaces custom receivers
         : Exemplars link metrics to traces
    2026 : Native histograms widespread
         : Unified observability standard
                            

OTLP Native Ingestion

Prometheus 3.x supports native OTLP ingestion — applications can push metrics directly to Prometheus using the OpenTelemetry Protocol without needing a Collector or remote_write adapter:

# prometheus.yml — Enable OTLP receiver (Prometheus 3.x)
otlp:
  # Enable OTLP gRPC endpoint
  protocols:
    grpc:
      endpoint: 0.0.0.0:4317
    http:
      endpoint: 0.0.0.0:4318

  # Translation settings
  resource_to_telemetry_conversion:
    enabled: true    # Map OTel resource attributes to Prometheus labels

  # Promote resource attributes to metric labels
  promote_resource_attributes:
    - service.name
    - service.namespace
    - deployment.environment

# Application instrumented with OpenTelemetry SDK
# Sends metrics directly to Prometheus via OTLP

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

# Configure OTLP exporter pointing at Prometheus
exporter = OTLPMetricExporter(
    endpoint="http://prometheus:4317",
    insecure=True
)

reader = PeriodicExportingMetricReader(exporter, export_interval_millis=15000)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)

# Create metrics
meter = metrics.get_meter("payment-service", "1.0.0")

request_counter = meter.create_counter(
    "http.server.request.count",
    description="Total HTTP requests",
    unit="1"
)

request_duration = meter.create_histogram(
    "http.server.request.duration",
    description="HTTP request duration",
    unit="s"
)

# Use in application code
request_counter.add(1, {"http.method": "POST", "http.status_code": "200"})
request_duration.record(0.045, {"http.method": "POST", "http.route": "/api/payments"})

Native Histograms

Native histograms (also called exponential histograms) solve the classic Prometheus histogram problem — bucket explosion. Instead of pre-defined buckets that multiply cardinality, native histograms store a single time series with dynamic resolution:

# Enable native histograms in Prometheus 3.x
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'my-service'
    # Scrape native histograms from /metrics endpoint
    scrape_classic_histograms: false    # Don't also scrape classic format
    native_histogram_bucket_limit: 256  # Max bucket count per histogram
    static_configs:
      - targets: ['my-service:8080']

// Go application exposing native histograms
package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var requestDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
    Name: "http_request_duration_seconds",
    Help: "HTTP request duration",
    // Native histogram config (replaces Buckets field)
    NativeHistogramBucketFactor:     1.1,   // ~10% bucket width
    NativeHistogramMaxBucketNumber:  160,
    NativeHistogramMinResetDuration: 1 * time.Hour,
})

func init() {
    prometheus.MustRegister(requestDuration)
}

func handler(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    // ... handle request ...
    requestDuration.Observe(time.Since(start).Seconds())
}

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

OpenTelemetry Collector

The OpenTelemetry Collector acts as a vendor-agnostic telemetry pipeline — receiving, processing, and exporting metrics from any source to any destination:

flowchart LR
    subgraph Sources["Sources"]
        A1[App OTLP]
        A2[Prometheus
Scrape]
        A3[StatsD]
        A4[Host Metrics]
    end

    subgraph Collector["OTel Collector"]
        R[Receivers]
        P[Processors]
        E[Exporters]
        R --> P --> E
    end

    subgraph Destinations["Destinations"]
        D1[Prometheus]
        D2[Grafana Mimir]
        D3[Datadog]
        D4[OTLP Backend]
    end

    A1 & A2 & A3 & A4 --> R
    E --> D1 & D2 & D3 & D4
                            

# otel-collector-config.yaml
receivers:
  # Receive OTLP from applications
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape Prometheus endpoints (replaces Prometheus server for collection)
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true

  # Host metrics (replaces Node Exporter)
  hostmetrics:
    collection_interval: 15s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      network: {}
      filesystem: {}

processors:
  # Add Kubernetes metadata
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name

  # Batch for efficiency
  batch:
    send_batch_size: 10000
    timeout: 10s

  # Memory limiter (prevent OOM)
  memory_limiter:
    check_interval: 1s
    limit_mib: 2048
    spike_limit_mib: 512

  # Filter unwanted metrics
  filter:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - "go_.*"
          - "process_.*"

exporters:
  # Send to Prometheus via remote write
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
    resource_to_telemetry_conversion:
      enabled: true

  # Or send to Prometheus via OTLP (Prometheus 3.x)
  otlp/prometheus:
    endpoint: prometheus:4317
    tls:
      insecure: true

  # Also send to Grafana Cloud
  otlphttp/grafana:
    endpoint: https://otlp-gateway-prod-us-east-0.grafana.net/otlp
    headers:
      Authorization: "Basic ${GRAFANA_CLOUD_TOKEN}"

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [memory_limiter, k8sattributes, filter, batch]
      exporters: [prometheusremotewrite, otlphttp/grafana]

Prometheus 3.x Features

# UTF-8 metric names — OTel conventions work natively
# Before (Prometheus convention): http_server_request_duration_seconds
# After (OTel convention): http.server.request.duration

# Prometheus 3.x accepts both formats
# PromQL works with either:
# rate(http.server.request.duration_count[5m])
# rate(http_server_request_duration_seconds_count[5m])

# Exemplars — linking metrics to traces
# When querying, exemplars show trace_id for specific samples

# In Grafana: click a data point → "View trace" jumps to Tempo/Jaeger
# In PromQL: query returns exemplars alongside regular data

# Application exposes exemplars with metrics
# Go example:
# httpDuration.WithLabelValues("200").
#     (Exemplar(prometheus.Labels{"trace_id": traceID}, duration))

# Query with exemplars API:
curl 'http://prometheus:9090/api/v1/query_exemplars' \
  --data-urlencode 'query=http_request_duration_seconds_bucket' \
  --data-urlencode 'start=2026-06-15T00:00:00Z' \
  --data-urlencode 'end=2026-06-15T01:00:00Z'

Migration Strategies

# Phase 1: Dual collection — Collector mirrors Prometheus scraping
# Both Prometheus and Collector scrape the same targets
# Compare results to validate before switching

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'dual-read-validation'
          static_configs:
            - targets: ['service-a:8080', 'service-b:8080']
          # Add a label to distinguish collector-scraped data
          relabel_configs:
            - target_label: __collector__
              replacement: otel

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus-validation:9090/api/v1/write

The Future Landscape

Conclusion & Series Wrap-Up