Slow Linux disk? How to spot IOPS bottlenecks and latency

What happens when your disk stalls under pressure

When a database application or web server starts responding slowly, the first suspicion usually falls on a lack of RAM or excessive CPU usage. However, in the vast majority of cases, the real invisible villain is the storage subsystem. Modern disk drives, especially PCIe-based NVMe solid-state drives, can perform thousands of operations per second. But when process concurrency spikes, internal queues form and I/O latency skyrockets, turning fast operations into severe operational bottlenecks.

To understand this behavior in practice, imagine a ten-lane highway that suddenly narrows down to a single track due to an invisible barrier. That is precisely what happens inside the Linux kernel when the number of read and write requests exceeds the physical capacity of the controller or storage medium. As we detailed in our article on analyzing IOPS bottlenecks in database servers, ignoring these latency symptoms can corrupt data or crash entire services due to connection pool exhaustion waiting on the disk.

How to use iostat to identify initial bottlenecks

The first step in any serious storage performance investigation is observing global system behavior through established tools. The iostat command, part of the sysstat package, is the perfect starting point for measuring CPU usage and block device I/O statistics. When running the command with a time interval, the main goal is not just looking at raw IOPS numbers, but monitoring critical metrics like average wait time and disk utilization percentage.

To initiate real-time monitoring every two seconds, execute the command in your terminal with the appropriate flags:

iostat -xz 2

In the output generated by this command, the await column indicates the average time in milliseconds that each request took to be serviced, including queue time and disk processing time. If this value consistently exceeds a few milliseconds on SSD or NVMe units, you have a clear queuing problem. Furthermore, the %util column shows the device saturation level; if it remains close to one hundred percent for long periods, the disk has reached its physical limit of simultaneous processing.

Tracing deep events with blktrace

When iostat merely points out that the disk is slow but fails to reveal which process or layer is causing the delay, we must turn to more advanced kernel instrumentation utilities. blktrace is a kernel-level tracing tool that captures detailed I/O events in real time, recording precisely when a data block enters the queue, when it is dispatched to the device driver, and when the hardware finally confirms operation completion.

This surgical granularity makes it possible to identify whether the issue lies in how the operating system batches blocks or if the hardware itself is taking too long to answer NVMe commands. To collect these traces on a specific disk, such as the /dev/nvme0n1 device, we use a command pipeline that records raw data and summarizes it with the blkparse utility:

blktrace -d /dev/nvme0n1 -o - | blkparse -i -

In practice, each line generated by this tool represents a step in the lifecycle of a disk request. By analyzing letters such as Q (queued), D (dispatched), and C (complete), we can calculate the exact time the request spent waiting in the operating system's internal queue before being physically dispatched to the storage hardware.

Optimizing NVMe schedulers for high concurrency

A common conceptual mistake in modern server administration is keeping the same I/O scheduling algorithms used back in the mechanical hard disk drive (HDD) era. Mechanical disks needed complex algorithms like CFQ to physically reorganize read and write head movements, minimizing time wasted spinning physical platters. However, solid-state drives and NVMe memories operate via electronic chips and feature native massive parallelism, rendering mechanical sorting completely unnecessary and counterproductive.

To extract maximum performance from high-concurrency environments, we must configure the block scheduler to none or kyber, depending on the workload. The none mode completely removes the overhead of unnecessary software queues, allowing requests to flow straight to the NVMe controller, which already possesses its own ultra-efficient hardware queue manager. As we explored in the guide on database optimization on dedicated hardware, this simple change reduces latency by precious microseconds.

Tuning queue parameters and depth in sysfs

Beyond choosing the right scheduler, the Linux kernel allows direct adjustment of queue depth limits through the sysfs virtual filesystem. Queue depth defines how many operations the operating system can submit simultaneously to the disk without waiting for prior responses. If this number is too low, the disk sits idle waiting for orders; if it is too high, requests accumulate and cause noticeable application delays.

To check and adjust the queue depth of an NVMe device dynamically, we can interact directly with the kernel configuration files exposed in the system directory:

cat /sys/block/nvme0n1/queue/nr_requests
echo 256 > /sys/block/nvme0n1/queue/nr_requests

This immediate change helps calibrate the storage subsystem's breathing room according to your infrastructure's actual load. On servers dealing with thousands of concurrent connections, raising this limit usually unblocks data flow, preventing database threads from getting stuck waiting for space inside the operating system's internal I/O queues. To explore these architectural concepts further in production, see our related technical analysis on Why use Row-Level Security in PostgreSQL to isolate tenants.

Final thoughts on storage resilience

Diagnosing and resolving IOPS bottlenecks in high-concurrency Linux environments requires a shift in mindset, moving away from guesswork toward rigorous metric observation. By mastering tools like iostat for macro visibility and blktrace for micro diagnosis, engineers gain the ability to see precisely where data flow faces resistance. Combining this surgical visibility with proper tuning of kernel schedulers and queue parameters ensures your NVMe hardware operates at peak potential, safeguarding your data and securing long-term operational stability.