How to measure real GPU power consumption on Linux

Why monitor GPU power consumption on Linux?

Managing dedicated servers running artificial intelligence, rendering, or mining workloads requires absolute control over hardware. The power draw of a modern graphics card can exceed three hundred watts under heavy loads, generating intense heat and high electricity bills. In practice, monitoring these figures helps identify bottlenecks, prevent equipment shutdowns due to overheating, and predict failures before they happen.

When dealing with Linux infrastructure, the absence of a graphical user interface forces administrators to master text-based tools. Instead of relying on heavy visual software, we extract precise data directly from the operating system. This enables automated alerts and allows adjusting hardware performance according to the actual application demand.

How to use nvidia-smi for real-time watts and temperature

For those using NVIDIA cards, the nvidia-smi (System Management Interface) utility comes pre-installed with proprietary drivers. It acts as a quick control panel in the terminal, showing memory usage, graphics core utilization percentage, and instantaneous power consumption in watts. In practice, running a single command gives you a complete health check of your card.

To keep the reading continuously updated, we use the watch command along with the card utility. In practice, this means the screen refreshes every second, letting you observe how electrical draw spikes immediately when starting a heavy task. Before executing advanced commands, ensure your network and ports are secure by following guidelines such as How to unblock ports on Linux UFW firewall to prevent unauthorized access.

watch -n 1 nvidia-smi

This command outputs a detailed table containing the current temperature, power consumed relative to the maximum supported limit, and fan speeds. If you notice temperatures approaching critical thresholds, the driver automatically throttles processor speeds to prevent permanent damage, a mechanism known as thermal throttling.

Extracting granular metrics with targeted queries. The default nvidia-smi output shows plenty of visual information, but it is not always ideal for automation scripts. For systems that write logs to files or send metrics to monitoring software, we need to extract only raw numerical data. We can achieve this by using specific query parameters that filter the output to return just what matters.

The following command interrogates the driver to strictly return the current power draw in watts, without cluttering the screen with tables or decorative borders. In practice, this simplifies writing Bash scripts that check energy expenditure every minute and record historical data to disk.

nvidia-smi --query-gpu=power.draw --format=csv,noheader

Beyond power draw, you can query dedicated memory usage, current core frequency, and component temperature in the same command line. This versatility makes the tool indispensable for infrastructure engineers who need to audit electrical consumption across multiple servers simultaneously.

Accessing hardware readings directly through sysfs

Although nvidia-smi works perfectly for NVIDIA cards, modern servers also utilize Intel integrated graphics or AMD cards, requiring a universal approach. This is where sysfs comes in, a virtual filesystem in the Linux kernel that exposes information about physical computer components in standard text directories. In practice, the operating system treats hardware sensors as simple files that can be read with standard commands.

To find thermal and power data for your card in sysfs, we generally navigate to the kernel hardware directory. Reading can be done quickly using the cat command to display the contents of the corresponding temperature or voltage file, as shown in the example below.

cat /sys/class/drm/card0/device/hwmon/hwmon*/temp1_input

Because the exact hwmon folder name can vary depending on the PCI bus and loaded drivers, we typically use wildcards or create scanning scripts. This direct filesystem reading is extremely lightweight and does not require installing heavy proprietary packages, making it ideal for minimalist container environments and lean servers.

Automating energy monitoring in scripts

Collecting data manually does not solve long-term problems; we need to record GPU behavior over hours to identify consumption spikes. We can create a simple Bash script that performs periodic readings and saves the results with timestamps to a text file. In practice, this allows auditing whether a specific application is wasting energy over weekends.

The script below runs an infinite loop capturing the current power consumption every five seconds and appending the information to a log file. To stop execution, simply press the Ctrl and C keys on your keyboard.

while true; do date +'%Y-%m-%d %H:%M:%S'; nvidia-smi --query-gpu=power.draw --format=csv,noheader; sleep 5; done > gpu_power_log.txt

With this file generated, you can import data into spreadsheets or visualization tools to generate precise energy efficiency charts. This type of audit helps companies significantly reduce electricity bills in their dedicated data centers.

Final thoughts on efficiency and monitoring

Measuring real GPU power consumption on Linux is no longer a luxury, but an operational necessity for anyone managing high-performance hardware. Mastering nvidia-smi and understanding how the kernel exposes data via sysfs empowers administrators to keep servers secure, cool, and economically viable. By combining these monitoring practices with proper infrastructure planning, you ensure your computational resources deliver maximum performance without surprises on your monthly electricity bill.