How to block Layer 4 DDoS attacks with nftables

The invisible danger of denial of service attacks

Imagine a store entrance suddenly flooded by thousands of people trying to enter at once, not to buy anything, but simply to block the doorway. This is exactly how a Distributed Denial of Service (DDoS) attack works at layer 4 of the OSI model. Layer 4 handles data transport, managing the TCP and UDP connections that keep your services accessible on the internet. When a server is hit by a flood of forged packets or infinite simultaneous connections, the operating system exhausts its memory and processing resources, crashing for real users.

For system administrators and network engineers, defending exposed infrastructure requires fast, intelligent tools capable of inspecting traffic in real time without introducing latency. Historically, iptables fulfilled this role for decades, but its monolithic architecture bottlenecks when rule volumes grow significantly. This is where nftables shines as the natural evolution of the Linux firewall, offering a unified syntax, better performance, and advanced data structures like maps and dynamic sets, ideal for mitigating transport layer attacks.

Why choose nftables to protect exposed servers

Nftables was designed to replace iptables, ip6tables, arptables, and ebtables under a single logical core and an integrated virtual machine inside the Linux kernel. In practice, this means operations that once required dozens of repetitive rules can now be elegantly solved with typed tables and sets. If you have ever needed to troubleshoot connectivity issues or track bottlenecks, you know that speed when applying filters makes all the difference in maintaining operational network stability.

Beyond superior performance across large rule lists, nftables natively handles connection states and counters without overloading RAM. It allows you to create dynamic counters per source IP address, applying granular restrictions only to clients that exceed acceptable request limits. To better understand how the kernel filtering ecosystem evolved, check out complementary strategies in how to block suspicious traffic and attacks with nftables, combining packet inspection with smart countermeasures.

Setting up the environment and creating the base table

Before implementing rate-limiting rules, we must ensure nftables is installed and active on your Linux server. Most modern distributions include the utility by default, but you can easily manage it using the systemd service. The first practical step consists of structuring a dedicated packet filtering table, separating DDoS rules from the machine's other network configurations.

To get hands-on safely, let's create a table named 'filter' in the inet protocol family, which covers both IPv4 and IPv6 at once. Run the following commands in your server's command line to initialize the basic filtering structure and avoid accidental disruptions to your SSH connection:

1. Verify if the nftables service is enabled and running:

   sudo systemctl status nftables

2. Create the main inet table:

   sudo nft add table inet ddos_defense

3. Add an input chain for traffic arriving at the machine:

   sudo nft add chain inet ddos_defense input { type filter hook input priority 0 \; policy accept \; }

Implementing rate limiting and connection control

The most efficient Layer 4 defense against volumetric attacks involves limiting the number of new connections per second and restricting control packet volumes. In a SYN flood attack, for instance, the attacker sends thousands of TCP connection requests without ever completing the handshake process, exhausting the server's open connection table. To neutralize this threat, we can configure nftables to monitor the pace at which packets arrive from a single IP address.

The following command creates a rule that drops TCP packets if the source address exceeds a specific request limit within a short time window. This technique stops malicious scripts from crashing the application while preserving resources for legitimate accesses. If you want to deepen other operating system security layers, combine these rules with auxiliary tools detailed in the guide on how to block malicious IP addresses with Fail2Ban, creating a dual barrier against intrusions.

Advanced rules with dynamic sets and maps

One of nftables' greatest strengths is its ability to use dynamic sets that automatically store offending IPs for a set period. Instead of writing static rules requiring constant manual intervention, we can instruct the firewall to add the address of any IP exceeding concurrent connection limits to a temporary blacklist. This drastically reduces CPU usage during large-scale attacks.

Below we present the advanced rule structure using timeout sets to automate abusive IP blocking:

1. Create a dynamic set to store IPs temporarily blocked for 10 minutes:

   sudo nft add set inet ddos_defense blacklist { type ipv4_addr\; flags timeout\; }

2. Add the rule checking if the IP is already blacklisted and dropping it immediately:

   sudo nft add rule inet ddos_defense input ip saddr @blacklist drop

3. Insert the limit rule that automatically adds new offenders to the set:

   sudo nft add rule inet ddos_defense input tcp flags syn limit rate 20/second burst 5 packets add @blacklist { ip saddr timeout 10m } drop

Best practices and validating rules in production

Applying firewall rules directly to production servers requires extreme caution to avoid accidentally blocking legitimate clients or your own administrative team. Before applying restrictive policies with drop actions, use the log directive to monitor traffic behavior for a few hours. Analyzing kernel logs helps tune rate limits to values reflecting real application usage without false positives.

Additionally, keep your rule backups updated and use persistent configuration files in the system's standard directory, ensuring defenses remain active after reboots. Monitoring network performance with diagnostic tools, as discussed in how to measure real speed and packet loss, also helps ensure the firewall isn't introducing unnecessary latency to legitimate packets.

Final considerations

Protecting internet-facing servers against Layer 4 denial-of-service attacks is an essential requirement for any modern infrastructure operation. Nftables proves to be a powerful, flexible, and high-performance tool for surgically filtering malicious traffic. By combining state-based rate limiting, dynamic sets, and continuous monitoring, you build a robust defensive wall that keeps your services online even under extreme network stress conditions.

Remember that information security is a continuous and evolving process. Tracking traffic behavior, periodically reviewing firewall rules, and integrating multiple protection layers ensure your architecture remains resilient against growing digital threats. Implement the concepts presented carefully, validate each step, and keep your systems always prepared for the unexpected.