How to Automate NVMe-oF and DRBD Failover in Proxmox

The high availability challenge in NVMe storages

Maintaining fast and secure storage in virtualization environments is one of the toughest juggling acts an infrastructure engineer faces every day. When dealing with NVMe drives, which deliver blazing-fast read and write speeds, any network bottleneck or hardware failure can instantly take down dozens of virtual machines. In practice, this means having fast hardware isn't enough; we must ensure that data written to one server is instantly copied to another secure location without hiccups or packet loss.

In Proxmox clusters, the main goal is ensuring that if a physical server suddenly dies, the virtual machines running on it wake up on another hardware within seconds, with minimal impact on the end user. To achieve this magic without losing data, we combine two powerful technologies: NVMe-oF for fast block traffic and DRBD for synchronous data replication. Before diving into practical setup, keeping an eye on infrastructure health is vital, similar to practices discussed in our guide on building predictive infrastructure alerts with Prometheus.

Understanding NVMe-oF and DRBD in practice

NVMe-oF, or NVMe over Fabrics, is essentially the technology that allows NVMe disk commands to travel across computer networks instead of being restricted directly to the server's motherboard. In practice, it extends the ultra-fast NVMe bus to high-performance network cables, such as InfiniBand or RDMA-enabled Ethernet. This makes accessing a remote disk feel almost as fast as accessing a disk physically connected inside the computer itself.

On the other hand, DRBD acts as a network disk mirror. It intercepts every tiny piece of data the operating system tries to write to the primary disk and sends an exact copy across the network to a second server in real time. If the first server crashes or loses power, the second server takes over immediately because it already holds an identical, up-to-date copy of all data blocks. This structural redundancy mirrors the care we take in critical systems, analogous to principles of diagnosing physical bus faults in industrial hardware.

Configuring the NVMe-oF transport layer

To start structuring our data redundancy, we need to prepare the Proxmox nodes to communicate via NVMe-oF using TCP or RDMA protocols. The first step involves loading the required kernel modules across all cluster nodes and ensuring the discovery service maps available network storages.

The following commands should be executed directly in the terminal of each Proxmox node to enable transport drivers and connect the NVMe initiator to remote storages:

modprobe nvme
modprobe nvme-fabrics
modprobe nvme-tcp
nvme discover -t tcp -a 192.168.100.50
nvme connect-all -t tcp -a 192.168.100.50

With these commands, Proxmox treats the remote NVMe disk as if it were a local drive connected to the server's own controllers. Multipathing ensures that if one network interface fails, traffic automatically migrates to the secondary interface without dropping active connections.

Implementing synchronous mirroring with DRBD

With NVMe disks exposed and connected over the network, the next step is establishing synchronous replication using DRBD. This tool operates directly at the Linux block layer, ensuring that any modification made to the primary volume is confirmed only after successful writing on the secondary node.

Below is an example configuration file for the DRBD resource managing our shared storage volume between two Proxmox nodes:

resource r0 {
    protocol C;
    disk {
        c-plan-ahead 512;
        c-fill-target 24M;
        c-max-rate 720M;
    }
    net {
        allow-two-primaries no;
        rr-conflict disconnect;
    }
    on proxmox-node1 {
        device /dev/drbd0;
        disk /dev/nvme0n1;
        address 192.168.200.10:7788;
        meta-disk internal;
    }
    on proxmox-node2 {
        device /dev/drbd0;
        disk /dev/nvme0n1;
        address 192.168.200.20:7788;
        meta-disk internal;
    }
}

After creating this configuration file on both servers, we initialize the resource by executing activation and formatting commands for the resulting logical device:

drbdadm create-md r0
drbdadm up r0
drbdadm primary --force r0
mkfs.ext4 /dev/drbd0

This synchronous setup ensures the Proxmox cluster always maintains a perfectly intact data block device ready to be mounted in case of unexpected hardware failure.

To guarantee that the entire hardware ecosystem and sensors respond healthily under thermal or electrical stress, maintaining solid hardware engineering practices is essential, reviewing fundamental concepts such as those in our guide on monitoring VRM temperatures and telemetry via I2C bus.

Automating failover and preventing split-brain

The greatest hazard in block-replication high availability systems is split-brain—a scenario where both nodes lose network communication with each other and simultaneously assume the other has died, writing conflicting data to the same disk and corrupting storage. To prevent this disaster, we configure fencing and quorum mechanisms based on STONITH (Shoot The Other Node In The Head).

Proxmox manages this intelligence through its corosync cluster manager, which continuously monitors the heartbeat of each physical machine. When a node stops responding to the heartbeat for a few seconds, automation triggers immediate isolation of the faulty node and promotes the secondary node to absolute primary in DRBD, allowing virtual machines to keep running without human intervention.

Final considerations

Automating NVMe storage failover in Proxmox clusters using DRBD and NVMe-oF multipathing transforms home labs or corporate infrastructures into highly resilient environments prepared for the unexpected. Although initial configuration complexity demands careful attention to network details and fencing, operational peace of mind rewards every invested minute. With path redundancy, synchronous block mirroring, and quorum automation, your infrastructure gains the robustness needed to operate continuously, keeping your data secure and services always online.