Hardware Monitoring with IPMI and Prometheus: Collection and Custom Metrics

The Critical Need for Physical Visibility in the Data Center

When managing physical servers in a production environment, the software layer is merely the tip of the iceberg. Beneath the operating system and containers lies a silicon infrastructure composed of motherboards, redundant power supplies, high-RPM cooling fans, and processors dissipating hundreds of watts of heat. In practice, this means that ignoring the physical health of the machine is gambling with fate, as a single jammed cooler can melt expensive components before any traditional software alert is even triggered. This is where monitoring hardware independently and continuously becomes absolutely essential.

To solve this challenge without depending on an operating system that might be locked up, the industry standardized a technology called IPMI, which stands for Intelligent Platform Management Interface. Simply put, IPMI is like a secondary auxiliary minicomputer embedded directly into the server motherboard, powered by a secondary electrical line. It operates completely independently of the main CPU and installed operating system, allowing engineers to power the equipment on and off and check temperatures even if Linux or Windows has completely frozen.

Communication Architecture and the Role of the BMC

The heart of any IPMI-based implementation is the BMC, short for Baseboard Management Controller. Think of the BMC as a dedicated night watchman that never sleeps inside your server. It has its own network address, its own memory, and sensors distributed throughout the entire chassis. When we want to know the current processor temperature, our monitoring software does not ask the operating system directly; instead, it sends a network request straight to the BMC IP address using dedicated management protocols.

In practice, this separation of worlds brings a massive operational advantage to engineering teams. If the operating system kernel suffers a fatal crash or kernel panic, normal monitoring services stop responding, leaving the team blind. However, because the BMC runs on a separate chip and has direct access to hardware buses, it continues responding to external requests perfectly. This structural resilience ensures you can diagnose the exact cause of a crash even before physically rebooting the machine.

Extracting Data with Native Command-Line Tools

Before sending any metrics to our monitoring system, we need to be able to read this data directly from the server. In the Linux ecosystem, the industry-standard tool for interacting with the BMC is the ipmitool utility. In practice, it works as a translator that sends structured binary commands to the management chip and returns a readable table of sensors, voltages, and system event logs.

To verify if your operating system can talk to the BMC locally, you can run a simple sensor query through the terminal. This command interacts directly with the kernel driver that talks to the hardware. Here is the typical command used for this initial workbench verification:

sudo ipmitool sensor list

If the response returns a long list containing fan speeds in RPM, temperatures in Celsius, and operational statuses like 'ok', local communication is successfully established. Otherwise, you will need to check if the corresponding kernel module, such as ipmi_devintf, is properly loaded in the operating system.

Integrating IPMI into the Prometheus Metrics Ecosystem

Prometheus is one of the most popular time-series databases in the world for monitoring modern infrastructures. However, it does not understand native IPMI proprietary commands out of the box. To bridge this gap, we use an intermediate program called an exporter, specifically the ipmi_exporter. In practice, this exporter acts as a bilingual translator: it receives an HTTP request from Prometheus, talks to the server BMC using IPMI, translates the raw temperature and voltage data into standard Prometheus format, and returns the response in fractions of a second.

To get this machinery running in a containerized environment, we can use a structured Docker Compose configuration file. Below is a functional template to start the exporter and route data collections cleanly and in isolation:

version: '3.8'
services:
  ipmi-exporter:
    image: quay.io/prometheus/ipmi-exporter:latest
    container_name: ipmi-exporter
    restart: unless-stopped
    ports:
      - '9290:9290'
    volumes:
      - ./ipmi.yml:/config.yml
    command:
      - '--config.file=/config.yml'

In the associated configuration file, we map the IP addresses of each managed physical server and their respective BMC access credentials. It is crucial to ensure that the user configured for this reading has permissions strictly limited to sensor reading, drastically reducing the attack surface in case of any credential compromise on the management network.

Interpreting Metrics and Creating Pragmatic Alerts

With metrics flowing into Prometheus, the next essential step is to define efficient alerting rules that prevent false positives and catch real problems on time. Common exported metrics include ipmi_sensor_state, which returns numeric values indicating whether the sensor is in a nominal or critical state, and ipmi_temperature_celsius. In practice, setting up an alert to trigger when processor socket temperature exceeds a safe threshold for more than five consecutive minutes prevents unexpected downtime and premature silicon degradation.

Another vital indicator that deserves close attention is the status of redundant power supplies. In mission-critical servers, it is common to have two power supplies connected to different electrical circuits to ensure high availability. IPMI provides binary metrics indicating if one of the supplies has failed. Monitoring this metric ensures the team is notified immediately to replace the faulty power supply before a second power surge brings the entire server down due to lack of redundancy. To complement this implementation in production environments, see our related technical analyses on Container Infrastructure Monitoring with Prometheus Grafana Mimir and Long Term Metrics Collection and Ephemeral Infrastructure Monitoring with Push-Based Metrics Collection in Lightweight Agents.

Final Considerations on Operational Reliability

Implementing hardware monitoring using IPMI and integrating it into Prometheus transforms an team's operational posture, moving it from a reactive approach to a predictive one. Instead of discovering a server shut down because it overheated, engineers receive gradual warnings about rising temperatures or incipient fan failures weeks before disaster strikes. By mastering this physical layer, we ensure that the underlying infrastructure is just as resilient as the modern software running on top of it.