Slow disk on Linux? How to spot I/O bottlenecks with perf

What is the perf command and why storage performance matters

When a Linux server begins to respond with extreme sluggishness, the first instinct of many administrators is to check overall memory usage or central processing unit load. Often, however, the root of the problem lies in stuck or slow disk operations, commonly known as input/output or I/O bottlenecks. To investigate these scenarios that remain invisible to basic tools, engineers turn to the perf command, a powerful native diagnostic utility built directly into the operating system kernel.

In practice, perf acts like a black box capable of intercepting events in real time, from the moment a program requests data from the disk down to the hardware's final response. If you have ever tried diagnosing slowdowns using only traditional infrastructure monitoring commands, as we explain in our guide on how to spot I/O bottlenecks with iostat, you probably noticed they show aggregate averages but rarely reveal which exact process generates excessive wait times. It is precisely in this precision gap that perf becomes indispensable for any systems engineer.

How system calls and tracing work under the Linux hood

To understand how perf monitors storage, we need to decipher the path a file takes. When an application needs to read or write data, it does not talk directly to the physical hard drive or SSD controller. Instead, it makes what we call a system call, which is a formal request for the operating system kernel to interact with the hardware securely and in a controlled manner.

These system calls generate events that can be measured with surgical precision. The perf utility connects to specific points in the kernel code called tracepoints. A tracepoint acts like a sensor installed on a highway that logs the exact time every vehicle passes by. In our case, the vehicle is the block of data traveling between RAM and the storage device, enabling us to calculate latencies with minimal error margins.

Preparing the environment and installing diagnostic tools

Before we can start tracing disk behavior, we must ensure the utility is properly installed and has the necessary permissions to interact with processor and kernel performance counters. On Debian or Ubuntu-based distributions, installation typically occurs through the system's standard package manager.

To verify if the package matching your current kernel version is available and ready for use on the machine, run the command below in your terminal with administrative privileges:

sudo apt update && sudo apt install linux-tools-common linux-tools-$(uname -r)

Following the successful execution of the previous command, test the basic operation of the utility by requesting the help screen or listing available system events. If the command returns the list without permission errors or missing debug symbols, your environment is ready to capture real I/O performance data.

Tracing block I/O events with the perf command

The subsystem responsible for managing read and write operations on hard drives and SSDs in Linux is the block layer. The perf utility features specific tools to monitor this subsystem via the record option, which logs events to a local file for later detailed inspection on screen.

To initiate a recording focused on block request submission and completion events during a critical traffic interval, use the structured command below in your terminal:

sudo perf record -e block:block_rq_issue,block:block_rq_complete -a sleep 10

The command above instructs the system to collect all block dispatch and completion events across all server CPUs for exactly ten seconds. The -a parameter ensures global monitoring rather than restricting it to a single isolated process, capturing a complete overview of the machine's storage activity.

Analyzing latency reports and identifying culprits

Once the performance data recording finishes, perf generates a hidden file named perf.data in the current working directory. To transform this pile of binary logs into a readable report that points out which processes caused the longest queues, we use the interactive report command.

Execute the following instruction to open the summarized analysis dashboard directly in your text console:

sudo perf report --stdio

In this textual report, you will be able to observe the percentage distribution of CPU time spent in kernel functions related to disk management. If you notice that calls tied to storage driver wait routines occupy the top of the list, you have found mathematical proof that your I/O subsystem is choking overall server performance.

Correlating disk latencies with overall machine health

Identifying an isolated I/O bottleneck is only half of the engineering job. Often, the disk is slow not because the physical controller failed, but because background secondary processes are saturating the bus with unnecessary operations. Continuous monitoring helps prevent wasting time swapping hardware when the real culprit is a misconfigured database or a service generating excessive logs.

If disk space is compromised by accumulated temporary files from the operating system, it is worth checking complementary cleaning tips, such as the procedure we detail in our article on how to clean systemd journal cache and free up disk space. Ensuring adequate free space reduces fragmentation and relieves the block subsystem investigated with perf. To explore these architectural concepts further in production, see our related technical analysis on How to test motherboard charging circuit with multimeter.

Final considerations

Advanced use of the perf command transforms I/O bottleneck debugging from a guessing game into an exact science rooted in deep kernel telemetry. By tracing system calls and block subsystem events, engineers gain the ability to isolate millimeter latencies and resolve chronic performance issues in demanding production environments.

Maintaining this detailed inspection routine ensures greater stability for your applications, reduces costs associated with oversized infrastructure, and guarantees that any disk anomaly is addressed with surgical precision before impacting end users.