Implementation of Secure Layer 2 Tunnels with VXLAN over Encrypted IPsec Overlay Networks

The Challenge of Connecting Distant Networks Without Losing Local Identity

In modern network engineering, we often need to make servers or virtual machines located in different buildings or cities believe they are plugged into the same physical network cable. Technically, we call this extending the Layer 2 domain. In practice, this means computers can communicate using MAC addresses directly, just as if they were in the same room, even when separated by thousands of miles. The major problem is that the public internet and modern routed networks work exclusively at Layer 3, focusing on IP addresses and traditional routing. To solve this dilemma without rewriting entire architectures, we turn to overlay networks, which act as virtual highways built on top of our existing basic infrastructure.

Understanding the Role of VXLAN in Creating Virtual Networks

VXLAN, or Virtual Extensible LAN, is an intelligent protocol that takes the original Layer 2 network packet, wraps it inside an ordinary UDP packet, and sends it across a standard IP network. In practice, it works like putting an old letter inside a modern express delivery package so it can cross the country without alterations. When the packet arrives at its destination, the outer packaging is discarded, and the original content is delivered exactly as it was generated. This allows the creation of up to 16 million isolated virtual networks on the same infrastructure, surpassing the historical four-thousand-network limit of the older VLAN technology. However, since traditional VXLAN was not designed with heavy encryption in mind, all this traffic travels completely in the clear unless additional precautions are taken.

To ensure no one can read or tamper with this data while it travels across the internet, we combine VXLAN with IPsec. IPsec is a widely tested suite of security protocols that applies end-to-end encryption to network packets. In practice, it turns any data connection into an armored vault, guaranteeing confidentiality, integrity, and authenticity of the exchanged messages. When we stack these two technologies, the process happens in two well-defined layers: first, the original packet gets the VXLAN wrapper to simulate the physical network cable; then, that same VXLAN packet is completely enveloped and encrypted by IPsec before touching the physical network interface. This engineering guarantees the best of both worlds: total Layer 2 flexibility combined with the impenetrable security of a modern corporate VPN.

Architecture Planning and Design Decisions

Before rolling up our sleeves and bringing up services, we need to design the topology considering operational trade-offs and the physical limits of the equipment. The most critical decision involves calculating the MTU, which represents the maximum size of the data packet the network can transmit without fragmenting it into smaller pieces. Because VXLAN adds a 50-byte header and IPsec adds a considerable amount more depending on the chosen encryption algorithm, the final packet becomes visibly bloated. If the intermediate physical network has a standard limit of 1500 bytes, the encapsulated packets will breach that limit, causing excessive fragmentation, high CPU usage, and chronic slowness. The definitive design solution is to configure the physical network MTU to 9000 bytes, a feature known as Jumbo Frames, or strictly adjust the MSS at the endpoints to force computers to negotiate smaller packets right at the source.

Practical Implementation with Reference Configurations

To bring this architecture to life in a modern Linux environment, we use native tools from the operating system kernel, avoiding expensive or complex commercial software. The first step involves configuring the IPsec tunnel interface using the strongSwan utility or the kernel's native XFRM subsystem, ensuring the encrypted channel is active and stable between the two data centers. Next, we create the VXLAN interface to encapsulate Layer 2 traffic inside this secure tunnel. Below is a practical command-line configuration example using standard Linux tools to bring up the VXLAN interface and bind it to the local physical network.

# Creates the VXLAN interface pointing to network identifier (VNI) 1005
ip link add vxlan0 type vxlan id 1005 dstport 4789 dev eth0 nolearning

# Assigns a temporary management IP address and brings up the interface
ip addr add 192.168.100.1/24 dev vxlan0
ip link set vxlan0 up

# Adds a static rule to the VXLAN forwarding table if necessary
bridge fdb append to 00:00:00:00:00:00 dst 10.0.0.2 dev vxlan0

The second step involves validating that the IPsec tunnel came up correctly and is negotiating cryptographic keys without authentication errors. To do this, we can inspect the current status of the IPsec service directly in the terminal using integrated diagnostic commands. Keeping monitoring active prevents unpleasant surprises during maintenance windows or sudden internet link drops.

# Checks the current status of active IPsec connections on the system
ipsec status

# Displays details about active Security Associations (SAs)
ipsec whack --trafficstatus

The third critical step is associating the newly created VXLAN interface with a local network bridge, allowing virtual machines or containers connected to that bridge to start exchanging Layer 2 packets across the encrypted tunnel. This integration transforms the virtual infrastructure into a transparent extension of the main physical local network.

# Creates a virtual network bridge named br-vxlan
ip link add name br-vxlan type bridge
ip link set br-vxlan up

# Connects the VXLAN interface to the created bridge
ip link set vxlan0 master br-vxlan

Common Pitfalls and Operational Best Practices

Even with everything configured correctly, engineers often run into subtle issues that drag down overlay network performance. A classic example is the blocking of specific UDP ports by intermediate firewalls or internet service providers, since VXLAN uses port 4789 by default. Ensuring security rules allow UDP traffic on this port and IPsec protocols is the first item on any troubleshooting checklist. Another delicate point concerns CPU consumption: because IPsec encryption and packet encapsulation demand heavy processing, servers handling gigabits of continuous traffic must feature hardware cryptographic accelerator cards or processors with dedicated AES-NI instructions enabled in the BIOS.

Final Considerations

Combining VXLAN and IPsec delivers a robust answer to the challenge of connecting data centers and corporate environments with strict isolation and security requirements. Although it demands careful planning of MTU and processing capacity, the flexibility to extend Layer 2 over public networks opens up a huge array of possibilities for modern infrastructure projects. Mastering this architecture puts engineers in a privileged position to design resilient systems prepared to scale without compromising data integrity.