Running OpenTelemetry at Scale: Architecture Patterns for 100s of Services

The combination that works at scale

Parent-based sampling means the sampling decision is made once at the root span — the first service that receives the request — and every downstream service in that trace inherits the same decision automatically, so you never end up with a trace where some spans were kept and others were dropped by different services making independent choices.

Use parent-based sampling at the SDK level to reduce overall span volume before it even reaches the collector, then use tail-based sampling at the gateway tier to make intelligent keep-or-drop decisions on what makes it through. Two passes of selection — aggressive on volume, smart about what survives.

A concrete example: set parent-based sampling at 10 percent for general traffic at the SDK. At the gateway, keep 100 percent of error traces, 100 percent of traces exceeding your latency SLO, and 10 percent of everything else. You end up storing roughly 11 to 12 percent of total trace volume, but with near-complete coverage of the production incidents you actually need to investigate.

processors:
  tail_sampling:
    decision_wait: 10s      # wait for all spans before deciding
    num_traces: 100000      # traces held in memory simultaneously
    expected_new_traces_per_sec: 1000
    policies:
      # always keep error traces
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}

      # always keep slow traces (adjust threshold to your SLO)
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}

      # keep 100% of checkout and payment — business critical
      - name: keep-critical-services
        type: string_attribute
        string_attribute:
          key: service.name
          values: [checkout-api, payment-service]

      # probabilistic baseline for everything else
      - name: baseline-sample
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

Memory sizing for tail-based sampling

The num_traces parameter is the one that will bite you if you undershoot it. It controls how many traces the gateway holds in memory simultaneously while waiting for all their spans to arrive. A rough formula: multiply your expected traces per second by your decision_wait value, then add 20 percent headroom. For 1,000 traces per second with a 10 second wait, you need at least 12,000 slots — not the 1,000 that most tutorial configs show.

The tail sampling processor documentation has the full parameter reference including the memory limiter integration, which you absolutely want enabled at the gateway tier to prevent OOM kills during traffic spikes.

Multi-Cluster Setups: When One Pipeline Is Not Enough

At some point, a single OTel pipeline stops being the right model. Maybe you operate in multiple regions with data residency requirements. Maybe you have a mix of Kubernetes clusters running different workloads with different SLOs. Whatever the reason, multi-cluster OTel setups introduce a layer of complexity that single-cluster thinking does not prepare you for.

The fundamental question is where aggregation happens. Aggregate within each cluster and forward summarized telemetry to a global backend, and you keep cross-region bandwidth low but lose the ability to do cross-cluster trace correlation. Forward raw telemetry to a central aggregation layer, and you get full correlation capability at significantly higher egress cost. Most organizations end up with a hybrid: metrics and logs aggregate locally, traces are forwarded to a central tier for correlation.

Getting trace context across cluster boundaries

Cross-cluster trace correlation only works if your services propagate the W3C traceparent header across cluster boundaries. Internal service mesh traffic usually handles this correctly. However, cross-cluster calls that pass through an API gateway, CDN, or any reverse proxy that strips unknown headers will break trace continuity at that boundary.

Diagnosing this is straightforward: if you see a trace starting at an API gateway span and the first downstream service shows a different root span with no parent, there’s a propagation break. To fix it, add traceparent and tracestate to your proxy’s header allowlist.

Here is what that looks like in the two most common cases:

# nginx — add inside your proxy_pass block
proxy_set_header traceparent $http_traceparent;
proxy_set_header tracestate  $http_tracestate;

---

# Envoy — request_headers_to_add in HttpConnectionManager
route_config:
  request_headers_to_add:
    - header: { key: traceparent }
    - header: { key: tracestate }

Data residency and the GDPR headache

If you operate in the EU, forwarding raw traces containing user identifiers to a central tier outside the EU can be a compliance problem. The practical solution is to run attribute redaction in your regional gateway before any data leaves the region. The OTel Collector’s transform processor lets you hash, mask, or drop specific attributes before export.

processors:
  transform/redact_pii:
    trace_statements:
      - context: span
        statements:
          # hash user IDs rather than drop
          - set(attributes["user.id"], SHA256(attributes["user.id"]))
          # drop email entirely
          - delete_key(attributes, "user.email")
          # truncate IP to /24 for geo without individual tracking
          - replace_pattern(attributes["net.peer.ip"], "\\d+$", "0")

Keeping the Pipeline Itself Observable

It would be funny if the  observability tools couldn’t be observed. The OTel Collector exposes its own internal telemetry as a standard OTLP pipeline, which means you can route it to any backend or an observability solution that you are already using.

otelcol_processor_batch_timeout_trigger_send (gotta love this long property name!) tells you whether the batch processor is flushing because the timeout fired rather than because the batch was full. A high ratio of timeout-triggered flushes means your traffic volume is lower than your batch config expects, and you are adding unnecessary latency.

otelcol_exporter_queue_size is the canary for backpressure. When otelcol_exporter_queue_size climbs toward your configured maximum, your exporter is falling behind the ingest rate. If it hits the maximum, the collector starts dropping data. Set an alert at 80 percent of queue capacity and you will catch pressure building before it becomes data loss.

otelcol_processor_tail_sampling_sampling_decision_timer_latency (another awesome long name!) tells you how long the tail sampling processor is taking to make decisions. A sudden increase here usually means the number of active traces in memory has grown past what the processor can efficiently scan — either increase resources or tighten your sampling policy.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 15s
          static_configs:
            - targets: ["localhost:8888"]

# Expose collector's own telemetry via its service config
service:
  telemetry:
    metrics:
      level: detailed   # basic | normal | detailed
      address: 0.0.0.0:8888
    logs:
      level: warn       # keep collector logs quiet in production

Rolling This Out Without Breaking Everything

The migration path from a single collector to a tiered setup does not have to be a big-bang cutover. You could introduce the gateway tier first while keeping the existing single collector in place, route a small percentage of services to the new tier, and validate that data is flowing correctly before moving everything over.

I suggest you start with a non-critical service — one that has decent traffic but where gaps in telemetry during the migration window would not cause anyone to lose sleep. Verify spans arrive at the gateway, verify they arrive at the backend with the right resource attributes, and check that your tail sampling policies are making sensible decisions. That validation loop is worth running for a week before you touch any of your critical services.

The config change on the service side is usually just updating the OTLP endpoint to the new agent address. If you are using the OTel Operator for Kubernetes, you can inject the agent endpoint as an environment variable through the Instrumentation custom resource — no application code changes, no redeployment of service configs when the collector topology changes.

The pattern across all of this — tiered collectors, trace-aware load balancing, layered sampling strategies, regional pipelines — is that scaling OTel is fundamentally an architecture problem, not an instrumentation problem. The instrumentation is the relatively easy part. The hard part is building a pipeline that stays operational under load, degrades gracefully when individual components have problems, and gives you enough visibility into itself that you can tell when something is wrong before it starts affecting the data your engineers depend on during incidents.

Once your OpenTelemetry pipeline is running at scale, the next step is learning how to interpret the traces to identify performance bottlenecks and root causes. See Troubleshooting Microservices with OpenTelemetry Distributed Tracing for an in-depth and very practical guidance on that subject.