Slow disk? How to debug NVMe I/O bottlenecks on Linux
Is your system lagging due to storage? Learn to debug latency and IOPS on NVMe drives using native Linux kernel monitoring tools.
Summary
- High IOPS usage can overwhelm the NVMe command queue and trigger noticeable system-wide latency.
- The iostat command effectively reveals device utilization and average request wait time.
- Monitoring via blktrace allows for tracking exactly which processes are creating I/O locks within the kernel.
- I/O scheduler adjustments are sometimes needed when hardware operates under extreme load limits.
- Identifying disk bottlenecks requires isolating hardware issues from processes that ignore concurrency limits.
What really causes NVMe slowdowns?
When discussing NVMe storage, the bottleneck is rarely the hardware itself, but rather how the Linux kernel manages the flood of read and write requests. Imagine a ten-lane highway where every car tries to squeeze into a single toll booth at once; that 'toll' is your disk's command queue. If you have dealt with performance issues before, you know that, just like checking for data integrity when cloning disks with dd, identifying latency requires looking at the time the OS spends waiting for the hardware.
Measuring I/O pressure with iostat
The fastest tool to get started is iostat. It shows not just how much the disk is being used, but the average service time. By running iostat -xz 1, you get a real-time view of the await column, which indicates the average wait time for each I/O operation. If this value is high, even with low CPU usage, your disk is literally being choked by requests from active processes.
# Monitoring the NVMe disk every second with detailed latency
iostat -xz 1Identifying who is lagging the system
If iostat shows high latency, the next step is finding the culprit. iotop is an excellent starting point, acting like a 'top' for storage. It displays in real-time which process is reading or writing the most data. This is crucial for preventing misconfigured logs or unexpected memory dumps from killing your server performance, which is as critical as debugging a container in a CrashLoopBackOff error.
Deep tracing with blktrace
For cases where latency is intermittent and hard to catch, we use blktrace. It acts as an 'event logger' for the kernel block subsystem. It records every move, from the moment the application makes the request until the moment the NVMe confirms the write. It is the most granular view you can get. Unlike basic tools, blktrace reveals if there is contention within the kernel's own execution queue.
Final thoughts on storage performance
Debugging high-performance disks requires discipline. Not every lag is failed hardware; often it is just a concurrency bottleneck that can be resolved by optimizing how your application handles files. Always remember to validate that the problem isn't physical before spending hours tuning kernel parameters. With these native tools, you will have the same precision a technician would have when diagnosing electronic failures with a multimeter, ensuring your system remains responsive under any load.