How to create Prometheus alerts for TCP latency spikes
Learn how to build composite metrics in Prometheus to spot packet retransmissions and connection degradation before outages occur.
Summary
- High TCP latency and silent retransmissions degrade distributed systems performance long before total outages happen.
- The Node Exporter utility exposes crucial Linux kernel network metrics directly for continuous Prometheus collection.
- Composite PromQL expressions combine sent packet volumes with active retransmissions to generate highly accurate alerts.
- Preventive monitoring of congestion windows prevents false alarms caused by brief traffic spikes in the infrastructure.
- Correlating packet loss and application metrics quickly isolates network bottlenecks in production environments.
Why monitor TCP latency and retransmissions
When a distributed application starts responding slowly, the blame usually falls on the database or the API code. However, the real bottleneck is often hidden in the network transport layer, specifically in TCP connections suffering from micro-drops, packet loss, and silent retransmissions. In practice, this means data packets get lost halfway due to router congestion or damaged cables, forcing the operating system to resend them and creating accumulated lag that ruins user experience.
For those seeking operational stability, relying solely on basic pings or checking if the HTTP port is open is insufficient. We need to look inside the operating system kernel and extract low-level metrics that reveal the true health of traffic. As we saw when measuring packet loss and jitter in Linux with iperf3, understanding the physical and logical behavior of traffic prevents unpleasant surprises during peak hours. Prometheus emerges as the exact tool to collect these variables and trigger automatic warnings.
Extracting kernel metrics with Node Exporter
Prometheus alone cannot guess what is happening in the Linux network stack; it requires a collector agent installed on the monitored machines. Node Exporter is the industry standard for this task, exposing hundreds of operating system statistics in a format that Prometheus can easily read via periodic HTTP requests.
Within the Node Exporter ecosystem, TCP-related statistics come directly from the Linux /proc/net/snmp virtual file. This file aggregates counters since the machine booted up. To monitor degradation, we must focus on metrics like TcpOutSegs, which indicates total sent segments, and TcpRetransSegs, which shows how many of those segments had to be resent because the recipient never acknowledged receipt.
Building the composite metric in Prometheus
Monitoring only the absolute number of retransmissions does not work well, because a server with high natural traffic will have more retransmissions than an idle server, without that indicating a serious problem. Reliability engineering requires creating composite metrics by calculating the proportional rate of retransmissions relative to total sent packets using PromQL's rate() function.
The mathematical expression divides the retransmission rate by the sent segment rate over a sliding time window, such as five minutes. If the proportion exceeds a safe threshold, say 3%, we know the network is degraded. This mathematical approach filters out transient noise and ensures the alert only triggers when there is a real impact on communication quality between microservices.
Writing the alert rule in Prometheus
With the validated PromQL expression, the next step is to structure the Prometheus alerting configuration file, known as Alerting Rules. This file defines the critical threshold, the persistence time needed to avoid false positives, and descriptive metadata helping the operator understand the issue quickly during an emergency call.
Below is a functional example of an alert rule configured in YAML format to detect anomalous TCP retransmission spikes:
groups:- name: tcp_alertsrules:
- alert: AltaTaxaRetransmissaoTCPSeveraexpr: (rate(node_netstat_Tcp_RetransSegs[5m]) / rate(node_netstat_Tcp_OutSegs[5m])) > 0.05for: 2minlabels:severity: criticalannotations:summary: