Distributed Observability Architecture with OpenTelemetry and Dynamic Sampling

The Operational Challenge of Data Growth in Distributed Systems

When modern applications move away from traditional monoliths—single software blocks where everything runs together—and split into dozens or hundreds of smaller services communicating over a network, a fascinating and costly problem arises: visibility. In microservices architectures, a single user action can traverse multiple independent systems, passing through message queues, databases, and external APIs. OpenTelemetry emerged as an industry-standard project to unify how we generate traces, metrics, and logs. In practice, it acts as a universal black box installed in every train car, recording exactly where the request went and how long it took at each step.

However, collecting absolutely everything that happens in a high-volume corporate environment creates a data tsunami. If a company processes tens of thousands of requests per second, storing every tiny telemetry detail requires massive storage infrastructure, driving up monthly cloud bills exponentially. This is where sampling comes in, which in practice means deciding which paths to record and which to discard to save resources without losing sleep at night. The eternal dilemma for engineers has always been finding the equilibrium point: collect too little, and silent failures slip by unnoticed; collect too much, and the company budget melts away on database servers and analysis tools.

Understanding Traditional Sampling and Its Limitations

Historically, monitoring data sampling was performed in a crude, static manner. The most common approach was head-based sampling, which happens right at the moment a request enters the system. Imagine a security guard at a building entrance who decides, upon seeing the first person in line, that only one in every hundred people will have their documents checked throughout their entire internal journey. In computing, this means a random number is generated at the start of a transaction; if the number meets the rule, the entire path of that request is recorded and sent to the observability tool. Otherwise, it is summarily ignored.

The flaw in this simplistic approach is that it is completely blind to the business value of what is being discarded. If the dropped request belonged to a VIP client executing a high-value financial transaction that failed due to an obscure bug, that error simply vanishes from diagnostic statistics. Conversely, millions of perfectly successful and trivial requests—such as a repeated query for static product catalogs—might be exhaustively collected, wasting precious space on redundant information. In practice, static sampling treats a failing login transaction exactly the same way it treats loading a decorative background image.

The Concept of Dynamic Sampling Based on Criticality

To solve this asymmetry between storage cost and data value, modern engineering has evolved toward dynamic and tail-based sampling. Unlike the security guard at the door, the tail mechanism acts as an insightful manager who observes the outcome of the entire process before deciding whether that service report should be permanently archived or discarded. In OpenTelemetry architecture, this is implemented using intermediate collectors that temporarily hold trace data in memory, evaluate the global behavior of the request, and apply intelligent preservation rules right before sending the data to long-term storage.

In practice, this means the system now understands the commercial context of what is happening. If a transaction took longer than normal, returned a server error code (like an HTTP 500 error), or involved a client categorized as priority, the collector ensures that 100% of those traces are saved, regardless of any random statistical rule. Simultaneously, if a routine request occurred within the expected timeframe and without technical hiccups, the system can decide to keep only a tiny fraction of it, such as 1% or 2% of the total. This approach guarantees that the rarest and most complex incidents never lack debugging evidence, while overall data volume drops drastically.

Practical Implementation Architecture with OpenTelemetry Collectors

Implementing this engineering in practice requires a well-designed network and processing topology, usually structured in two layers of OpenTelemetry collectors: local agents and the central collection cluster. Local agents run side-by-side with applications, whether on the same physical server, inside the same Kubernetes pod, or as an embedded library. The primary role of these edge agents is to perform basic instrumentation, inject tracing headers into HTTP requests, and perform light pre-filtering of noisy telemetry before it traverses the company's internal network.

Next, this data flows to the central cluster of collectors configured to support tail-based sampling. Because this central layer needs to retain traces for a few seconds in memory to correlate all microservices involved in a single transaction, it must be provisioned with nodes capable of handling this volatile retention. Below is a simplified example of a YAML configuration file for a central OpenTelemetry collector running tail-based sampling rules:

receivers:  otlp:    protocols:      grpc:      http:processors:  tail_sampling:    decision_wait: 10s    num_traces: 50000    expected_new_traces_per_sec: 2000    policies:      - name: errors-policy        type: status_code        status_code: {status_codes: [ERROR]}      - name: latency-policy        type: latency        latency: {threshold_ms: 500}      - name: probabilistic-policy        type: probabilistic        probabilistic: {sampling_percentage: 5}exporters:  otlp:    endpoint: