How to block Layer 4 DDoS attacks with iptables and hashlimit

The silent danger of packet floods at the transport layer

Imagine that the front door of your business suddenly receives thousands of people at the same time, but none of them actually want to enter—they just block the doorway. That is exactly what happens in a distributed denial-of-service attack, known as a DDoS, focused on layer 4 of the OSI model, which handles data transport between networks. In this layer, malicious actors unleash floods of packets using the TCP protocol (responsible for reliable data delivery) or the UDP protocol (focused on speed without delivery confirmation) to choke the server until it stops responding. In practice, this means your website or system goes offline because the Linux operating system spends all its energy just trying to log every incoming fake connection, failing to serve those who actually matter.

When legitimate traffic blends with the noise generated by thousands of zombie machines spread across the globe, identifying the attacker manually becomes an impossible mission. It is in this critical scenario that the Linux networking ecosystem comes into play, allowing us to build smart barriers using native tools. If you have ever experienced the frustration of having your server freeze due to excessive traffic, you know that merely restarting the service does not solve the problem. To better understand how to isolate the environment and protect infrastructure, it is worth checking out the guide on how to block suspicious connections in Linux with iptables and fail2ban, which provides a solid foundation for perimeter security.

How the hashlimit module works to control packet flow

iptables, which acts as the digital doorman of Linux responsible for deciding who enters and leaves the machine, features several powerful extension modules. One of the most useful for containing network floods is hashlimit, a feature that creates dynamic tables in memory to track the rate of packets per second coming from specific IP addresses or network ports. Unlike traditional rate limiting that treats everyone the same, hashlimit remembers the behavior of each individual origin. In practice, it works like a smart flow meter that tightens the screws only on those sending data too fast, letting normal traffic flow smoothly without interruption and without penalizing legitimate users.

Configuring hashlimit requires understanding a few fundamental parameters, such as the maximum allowed rate and the burst bucket size, which tolerates minor natural traffic fluctuations. When an IP address exceeds this stipulated quota, the Linux kernel can drop excess packets or flag them for other actions. This surgical approach prevents you from having to shut down access to an entire network just because a single computer got infected and started hurling garbage at your server. For administrators managing servers exposed to constant scanning, combining this strategy with the techniques explained in how to block malicious traffic at the edge with eBPF and XDP ensures an extra layer of performance even before traffic reaches iptables.

Mitigating SYN packet floods in the TCP protocol

The classic Achilles' heel of any web server exposed to the internet is the so-called SYN flood, a dirty trick where an attacker sends thousands of TCP connection requests but never completes the final step of the digital handshake. The server, politely, reserves RAM space and waits for a response that never arrives, until all available descriptors are exhausted. To lock down this vulnerability with iptables, we create a rule that intercepts new connection requests and applies the hashlimit filter to block abusers. In practice, this means that if an IP address starts firing dozens of session opening requests per second, the firewall blocks further packets from that specific IP, saving server memory from an imminent collapse.

To put this protection into practice safely on your machine, follow the steps below in the terminal with administrator privileges:

1. Create a new rule in the input chain to identify initial SYN packets using the state module. The following command isolates the beginning of TCP communication:

   iptables -A INPUT -p tcp --syn -j SYN_FLOOD

2. Configure the hashlimit rule inside this custom chain to cap the rate at a maximum of fifty packets per second per source IP:

   iptables -A SYN_FLOOD -m hashlimit --hashlimit-above 50/sec --hashlimit-burst 100 --hashlimit-mode srcip --hashlimit-name syn_rate_limit -j DROP

3. Ensure that the rest of the legitimate traffic that passed through the filter is accepted normally by the default return rule:

   iptables -A SYN_FLOOD -j RETURN

With this structure in place, any attempt to exhaust connection resources through fake requests is halted right at the entrance. It is worth noting that adjusting limit values depends heavily on the real traffic of your site or application; high-volume portals may need higher margins to avoid disrupting legitimate simultaneous visits.

Containing UDP flood attacks and volumetric traffic

While the TCP protocol requires confirmation and maintains session control, the UDP protocol is entirely anarchic: it simply throws packets onto the network hoping they reach the destination, without asking if the server is ready to receive them. This characteristic makes UDP the favorite vehicle for volumetric amplification attacks, where the attacker spoofs the source IP address and makes thousands of servers respond simultaneously to the victim. On your Linux server, this translates into an avalanche of unwanted packets that clog the network card and consume all available bandwidth. In practice, defense against this type of flood requires dropping UDP packets on ports that do not run essential services and drastically limiting the rate on active ports.

To protect specific UDP ports against abuse using hashlimit, we can apply targeted rules that discard excess digital garbage before it consumes precious application processing power. Check the practical example for an application running on UDP port 53 or any other service port:

iptables -A INPUT -p udp --dport 53 -m hashlimit --hashlimit-above 100/sec --hashlimit-burst 200 --hashlimit-mode srcip --hashlimit-name udp_rate_limit -j DROP

This command line tells the firewall that any IP trying to send more than one hundred UDP packets per second to the chosen port will have the excess summarily dropped. This simple measure prevents automated scripts from crashing your infrastructure with useless data bursts. Monitoring hardware health during these events is essential to ensure the server does not suffer from overheating or physical bottlenecks, something that can be closely tracked with the guidance in how to monitor server temperature on Linux with lm-sensors.

Final considerations and validation of security rules

Implementing transport layer barriers with iptables and the hashlimit module transforms a vulnerable server into a resilient fortress against volumetric denial-of-service attacks and SYN packet floods. However, no security system should be put into production without rigorous testing in controlled environments to ensure legitimate users are not accidentally blocked during normal traffic peaks. In practice, network engineering requires constant monitoring of kernel logs and fine-tuning of rate thresholds as your user base behavior evolves over time. Combining these edge defenses with good monitoring and architecture practices guarantees the stability of your services even under heavy pressure on the internet.