Slow server? How to find disk and IOPS bottlenecks in Linux

Why does your server lag even with low CPU usage?

When a production server starts responding with extreme sluggishness, the first instinct of most engineers is to inspect processor consumption and RAM utilization. In practice, the real culprit is frequently the storage subsystem, which struggles with I/O saturation, meaning the inability of hard drives or solid-state drives to keep up with incoming read and write requests. If the processor has to wait for data to arrive from the disk before proceeding, it sits idle, creating a false sense of headroom while the entire system grinds to a halt. This scenario is extremely common in database servers, virtualized environments, and applications handling heavy log volumes.

To avoid guesswork-driven diagnoses, we must understand the concept of IOPS, which stands for input/output operations per second. This is the fundamental metric measuring how many small reads or writes a storage device can perform in a given timeframe. Another critical factor is latency, the time a request takes from the moment the operating system dispatches it until the disk confirms completion. When the wait queue grows and latency spikes, the entire application bottlenecks. During severe space shortages and wear caused by excessive logs, administrators often resort to procedures like cleaning caches and logs in Linux, but when structural slowdowns occur, we must look far beyond simple file cleanup.

Monitoring real-time disk health with iostat

The first step in any storage performance investigation within the Linux ecosystem is extracting macroscopic metrics to identify which physical device is suffering. The iostat command, part of the sysstat package, is the ideal native tool for this initial task. In practice, it displays detailed statistics regarding CPU utilization, data transfer rates, and request rates for every partition or disk attached to the system. To monitor hardware behavior continuously every two seconds, we execute the command with specific parameters in the terminal.

iostat -xz 2 5

When analyzing the output of this command, we must pay close attention to crucial columns such as await, which indicates the average wait time in milliseconds for I/O requests to be serviced, and %util, showing the percentage of time the device spent processing requests. If the %util parameter remains near one hundred percent for extended periods and the await value spikes into dozens or hundreds of milliseconds, we have clear evidence of IOPS saturation. This means the disk has reached its physical limit and cannot process the current workload, requiring query optimization, load distribution, or migration to faster storage tiers.

Investigating block-level behavior with blktrace

When the high-level overview provided by iostat is insufficient to isolate the root cause of erratic behavior, we must drop down the abstraction layer and inspect traffic directly at the Linux kernel block layer. This is precisely where blktrace comes into play, a powerful tool capable of capturing and logging all I/O operations sent to block devices with microsecond precision. In practice, it acts like a high-speed camera pointed at the path data travels between the file system and the physical disk controller.

To extract useful data without overwhelming the system with massive log files, we typically combine it with the blkparse utility to generate a human-readable summary. The basic procedure involves starting the background capture for a specific disk, such as the device /dev/nvme0n1, reproducing the workload, and stopping collection immediately afterwards.

blktrace -d /dev/nvme0n1 -o - | blkparse -i -

This execution details every read and write request, showing precisely when the request was generated, when it entered the driver queue, and when it was completed by the hardware. If we observe recurring latency bottlenecks at this layer, it is worth investigating whether there are physical interference issues or electrical noise in the server infrastructure—a critical topic we address when analyzing how to eliminate EMI in PCB routing and grounding, ensuring the signal integrity of data traversing internal and external buses.

Simulating stress scenarios with fio

Often, the greatest challenge for an infrastructure engineer is discovering a storage array's limits before end users begin complaining about sluggishness in production. To solve this planning dilemma, we use fio (Flexible I/O Tester), widely considered the Swiss Army knife of storage stress testing. In practice, it allows simulating exact disk access patterns, such as massive sequential reads or thousands of concurrent small random writes, emulating the real behavior of relational databases.

We can configure fio directly from the command line to run a quick random write test using four-kilobase blocks, measuring the actual IOPS performance of the target disk.

fio --name=randwrite-test --filename=/tmp/testfile --size=4G --bs=4k --iodepth=64 --rw=randwrite --direct=1 --ramp_time=2

The direct=1 parameter is fundamental because it bypasses the operating system's RAM read and write cache, forcing the application to interact directly with the physical disk and revealing the true capability of the hardware. By analyzing fio results, we can accurately estimate whether our current infrastructure will support seasonal traffic spikes or if we need to resize storage before the next major traffic event.

Final thoughts on storage troubleshooting

Efficiently tracking I/O bottlenecks and disk latency requires a methodical approach spanning macroscopic observation down to microscopic kernel traffic analysis. Tools like iostat, blktrace, and fio form an indispensable trio in any systems reliability engineer's arsenal, allowing the transformation of subjective suspicions into precise, actionable technical diagnoses. Understanding storage subsystem behavior prevents unnecessary hardware investments and guarantees operational stability for the critical services supporting the business.

Ultimately, maintaining proactive monitoring and running periodic load tests prevents silent latency failures from surprising engineering teams during critical moments. By mastering these native Linux tools, you gain total autonomy to identify the root cause of any slowdown, optimize data flows, and ensure your technology infrastructure always operates at peak performance and reliability.