Slow server? How to spot disk bottlenecks with iostat

What happens when storage becomes the bottleneck

When an application gets slow or the system feels unresponsive without a clear CPU spike, the primary suspect is usually the storage subsystem. The operating system constantly reads and writes data to mechanical hard drives or solid-state drives. When these devices hit their physical limits of operations per second, the processor enters a wait state. On database servers or busy file systems, finding the root cause of this lag is essential to restoring system stability.

To investigate Input/Output (I/O) activity, Linux provides the iostat tool. Part of the sysstat package, this utility details how the CPU and storage devices handle system requests. If you have investigated performance issues on high-performance storage drives, such as when learning how to spot I/O and IOPS bottlenecks on Linux NVMe drives, you will find iostat to be the fundamental tool for tracking performance across all storage layers.

Understanding CPU iowait metric

When running commands like top or htop, system administrators often notice a high CPU percentage labeled as wa or iowait. The term iowait represents the percentage of time the CPU spent idle while waiting for outstanding disk read or write requests to complete. High iowait does not mean the CPU is working hard; rather, it indicates the processor is blocked waiting for slower storage hardware.

It is crucial to monitor processes stuck in this waiting phase. Tasks delayed by slow disk responses enter an uninterruptible sleep state in the system kernel. To understand how stalled tasks relate to resource starvation, you can check how to find zombie processes on Linux and free up CPU to prevent task queues from degrading overall performance.

Installing and running iostat in real time

Installing iostat is simple on almost any Linux distribution via the sysstat package. To analyze live behavior rather than cumulative historical averages stored since system boot, you must pass an update interval in seconds to the command.

1. Install the sysstat package using your distribution package manager:

   sudo apt update && sudo apt install sysstat -y

2. Run iostat with extended statistics enabled in megabytes per second every 2 seconds:

   iostat -xz 2

3. Examine the generated columns in your terminal to pinpoint which disk is suffering high latency or accumulating requests.

Interpreting core iostat metrics

The output of the extended mode (-x) provides several columns that reflect disk health. Here is what the key fields mean in practice:

Device            r/s     w/s     rMB/s     wMB/s   r_await   w_await   aqu-sz  %util  sda             45.00  120.00      1.20     15.40      4.20     35.50     4.80  98.50  nvme0n1          12.00   85.00      0.40      8.10      0.30      0.80     0.10  15.00

The r/s and w/s columns show the number of read and write requests completed per second. The rMB/s and wMB/s fields show total data throughput. However, the most critical metrics for troubleshooting performance bottlenecks are r_await, w_await, aqu-sz, and %util.

The w_await metric measures average latency in milliseconds for write operations, from queue creation to hardware acknowledgment. If w_await or r_await exceeds 10 to 20 ms on SSDs or 50 ms on mechanical drives, applications will experience noticeable slowdowns. The aqu-sz column represents the average queue length of pending requests. Values consistently above 1 or 2 indicate that storage cannot process incoming requests fast enough.

The %util trap on modern drives

Historically, the %util column indicated the percentage of time the storage device was busy serving requests. A value near 100% on spinning hard drives meant complete actuator saturation. However, modern SSDs and NVMe drives process requests concurrently across multiple internal channels. As a result, %util can reach 100% while the drive still has spare processing capacity for parallel requests.

Because of this, never rely solely on %util. Always evaluate latency metrics (r_await and w_await) along with queue sizes (aqu-sz). If latency remains low and queues stay small, the storage system is performing well despite high %util numbers. To explore these architectural concepts further in production, see our related technical analysis on How to measure ripple in bench power supplies with oscilloscope.

Diagnosing and resolving I/O bottlenecks

Once you identify a disk with elevated w_await and growing queues, remediation depends on what generates the workload. On database servers like PostgreSQL or MySQL, unindexed queries might be triggering massive disk scans, or transaction log writes might be overwhelming the drive. In containerized environments, inspecting per-process I/O using tools like iotop helps pinpoint the exact application causing high activity.

If the hardware bottleneck is caused by legitimate traffic volume, effective strategies include splitting workloads across separate drives (such as isolating write-ahead logs), tuning kernel page cache parameters (like vm.dirty_ratio), or upgrading to storage with higher IOPS performance.