Slow server? How to find NVMe disk bottlenecks on Linux

Why ultra-fast storage still feels sluggish under pressure

When migrating traditional mechanical hard drives to solid-state drives built on the NVMe bus, a massive leap in speed is expected. In practice, these devices read and write data at breathtaking speeds, measuring hundreds of thousands of operations per second. However, under heavy server loads, such as transactional databases or streaming platforms, the operating system can face hidden bottlenecks. The moment request queues pile up, the bottleneck stops being raw hardware and becomes how the kernel manages data flow.

To understand this behavior, think of NVMe as a ten-lane highway. If traffic is moderate, cars fly. But if there is a poorly configured toll booth at the entrance, congestion occurs even with open lanes. In Linux, this toll booth is managed by I/O queues. When applications demand more than the controller can dispatch, latency spikes. Just like investigating drops in network traffic monitored with eBPF, we need precise tools to open the storage hood and find where traffic is truly stalling.

Measuring system pulse with the iostat utility

The first step in any storage performance investigation is observing overall system behavior. The iostat utility, present in most Linux distributions, shows crucial statistics on CPU utilization and data transfer volume per device. When running the command during runtime, we must closely look at metrics like wait times in queues and disk utilization percentage.

To monitor a specific NVMe drive every two seconds, run the following command in the server terminal:

iostat -xz 2

In practice, the -x flag reveals advanced data, including the average service time per request, known as svctm, and total wait time in the queue, called await. If the await value is much higher than svctm, it means your data is getting stuck in an internal operating system wait queue. This is the first clear sign that the application is sending more commands than the NVMe controller can process simultaneously.

Simulating extreme loads and measuring limits with fio

Often, bottlenecks only appear when a server hits maximum effort. To diagnose disk behavior before issues impact real users, we use fio, considered the Swiss Army knife of storage stress testing. With it, we can simulate simultaneous reads and writes across various block sizes, perfectly mimicking heavy database workloads.

To run a quick random write test that stresses the NVMe subsystem, use the following command:

fio --name=randwrite-test --ioengine=libaio --rw=randwrite --bs=4k --numjobs=4 --size=4G --iodepth=64 --runtime=60 --time_based

In this practical example, we configured four parallel jobs executing random writes in 4-kilobyte blocks, maintaining an queue depth of 64 simultaneous operations. Observing latency behavior during this test helps determine if the device suffers performance degradation under heat or if the controller hits internal communication channel limits.

When general metrics and synthetic tests fail to reveal the root cause of slowness, we need an X-ray tool. blktrace is an advanced utility that captures kernel block-layer events, recording the exact microsecond a request enters the queue, when it is sent to the driver, and when the hardware confirms completion.

blktrace -d /dev/nvme0n1 -o - | blkparse -i -

In practice, this tool generates a massive volume of data mapping the complete lifecycle of every read or write operation. Analyzing the output helps identify whether delays occur at the filesystem layer, the kernel I/O scheduler, or directly within the physical SSD controller. This level of visibility is essential for diagnosing complex issues in mission-critical environments.

Final thoughts on storage performance diagnosis

Identifying bottlenecks in NVMe drives requires looking beyond superficial peak speed numbers provided by manufacturers. By combining the panoramic view of iostat, the controlled stress capability of fio, and the surgical precision of blktrace, engineers can isolate the exact source of hidden latency. Understanding these metrics ensures high-performance hardware investment truly translates into a fast, stable experience for application end-users.