Slow NVMe disk? How to find bottlenecks on Linux

Why do NVMe disks slow down even under high speeds?

NVMe disks (Non-Volatile Memory Express, an ultra-fast protocol for solid-state drives connected directly to the main bus) promise incredible read and write speeds, but under heavy loads, they can still suffer from inexplicable slowdowns. When applications demand thousands of operations per second, the operating system creates waiting queues to organize the traffic of data. In practice, this means that if the incoming flow is greater than the device can process, requests start to pile up, generating what we call queue latency. It is similar to an automated toll booth receiving more cars than the gates can release per minute, creating a massive traffic jam even with modern lanes. Identifying where this traffic gets stuck requires going beyond traditional disk usage counters and directly analyzing how the operating system kernel interacts with the hardware. As we explain in our article on how to debug RAM memory bottlenecks with perf on Linux, deep tracing tools are essential to uncover hidden bottlenecks that impact overall application performance.

How I/O queues work inside the Linux kernel

The Linux kernel manages storage by dividing tasks into virtual queues that communicate directly with the physical NVMe channels. Each request generated by a database or an application passes through software layers before turning into an electrical signal on the disk's printed circuit board. When the volume of data explodes, these queues reach their maximum capacity, and the system must pause new requests until the hardware frees up space. To monitor this behavior in real time, we can rely on native utilities that capture kernel events without installing complex third-party packages. Tracking the evolution of these metrics helps understand whether the performance limiter is in the software, the disk controller, or the motherboard infrastructure, ensuring more assertive diagnostics and definitive fixes.

Tracking storage events with blktrace

The blktrace utility is a powerful tool in the Linux ecosystem designed to record in detail all I/O operations entering and leaving the system's block layers. To use it in practice and capture your NVMe disk's behavior under heavy load, you can run the following command in your terminal. Before running, replace nvme0n1 with your drive's correct identifier:

blktrace -d /dev/nvme0n1 -o - | blkparse -i -

This command monitors every request sent to the disk in real time, showing the exact moment it enters the queue, when it is dispatched to the hardware, and how long it takes to receive a response. In the generated output, you will see letters like Q (queue), D (dispatch), and C (complete), which form the complete timeline of the disk operation. If the time distance between the Q and D events is very high, it means the operating system is holding onto data due to a lack of resources in the internal queue. On the other hand, if the delay occurs between D and C, the bottleneck lies strictly in the processing capacity of the NVMe disk itself or the PCIe bus.

Interpreting hardware performance counters

In addition to tracking software flow, it is essential to evaluate the performance counters of the NVMe controller itself to check if the physical component is operating within safe temperature and bandwidth limits. Tools like nvme-cli allow extracting advanced telemetry directly from the storage unit's firmware, revealing vital information that the operating system often summarizes generically. To check the error log and device health status under stress, run the command below:

nvme smart-log /dev/nvme0n1

Careful analysis of these logs helps identify if the controller is suffering from thermal limitations, known as thermal throttling, where the disk drastically reduces read and write speed to prevent overheating. In high-density environments, monitoring hardware temperature and integrity prevents catastrophic failures and ensures the stability of critical services depending on continuous low latency. For scenarios where thermal and electrical stress tests are necessary, it is worth correlating this data with power consumption analyses, as discussed in our guide on how to measure real power consumption in server ATX power supplies.

Adjusting queue depths to mitigate latency

When we identify that high latency is caused by saturation in I/O queues, one of the most effective approaches is to adjust the queue depth parameter. By default, many systems use generic values that might not reflect the actual parallelism capacity of your specific NVMe model. We can check the current queue depth by navigating to the device directory in the kernel's virtual file system and adjusting the appropriate parameter. To check and dynamically modify the maximum number of simultaneous requests supported, use the following instructions in your terminal:

1. Navigate to your NVMe device configuration folder in the sysfs directory using the command cd /sys/block/nvme0n1/queue
2. Check the current queue request limit by running the command cat nr_requests
3. Change the value to an optimized level by writing the desired new number with the command echo 128 > nr_requests

This simple modification reduces the volume of data artificially accumulated by the operating system, allowing the NVMe controller to process each batch of requests with greater agility and less waiting time. It is important to emphasize that excessively high values in queue depth increase latency perceived by applications, while values that are too low can leave the hardware idle, requiring practical tests to find the ideal balance point for each workload.

Final considerations on storage diagnostics

Debugging I/O bottlenecks in high-performance Linux environments requires a balanced combination of software monitoring and hardware telemetry analysis. By mastering native tools like blktrace and nvme-cli, engineers and administrators gain the ability to look beyond superficial disk usage metrics, identifying the exact root of any slowdown. Understanding the life cycle of requests in kernel queues turns troubleshooting into an exact science, ensuring your applications achieve maximum efficiency and operational stability under any load volume.