Firewalld

So, for some reason, this website has attracted attention of the unwanted kind. I see several Class B subnets are engaging in persistent SYN attack to overload the webserver and knock it unresponsive.

This is a Linux system, and it is running firewalld (nftables). So after some nosing around online today, I may have a solution.

# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 0 -p tcp -m multiport --dports 80,443 -m state --state NEW -m recent --set
# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 1 -p tcp -m multiport --dports 80,443 -m state --state NEW -m recent --update --seconds 60 --hitcount 30 -j REJECT --reject-with tcp-reset
# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 2 -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP
# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 3 -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix '[SYN Flood Detected] '
# firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 4 -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT

This is a solution that I will be monitoring, to see how well it works, and will potentially constrain it more if I have to. Reducing the srcmask to 24 will apply the limits to a whole Class C in one hit if one IP in the range is misbehaving for example. The solution can of course be applied to other ports than 80 (http) and 443 (https). Any service you have exposed can benefit from this as this is only about the establishing of the connection, not what happens afterwards. And to be honest, ten connection attempts per second with an initial burst of twenty is generous for a small server such as what I am running. The only thing that potentially could have come close to that was email.

Thanks to:

https://ivansalloum.com/preventing-syn-flood-attacks-on-your-linux-server

https://www.certdepot.net/rhel7-mitigate-http-attacks/

For the actual commands to use and the structure of the actual mitigation.

As for what it looks like being under attack, you can use netstat to see:

# netstat -planet|grep SYN_
tcp        0      0 79.136.7.42:443         45.166.59.153:20483     SYN_RECV    0          0          -
tcp        0      0 79.136.7.42:443         45.166.56.203:38975     SYN_RECV    0          0          -
tcp        0      0 79.136.7.42:443         45.166.56.191:46463     SYN_RECV    0          0          -
tcp        0      0 79.136.7.42:443         45.166.58.33:7709       SYN_RECV    0          0          -
tcp        0      0 79.136.7.42:443         52.204.81.148:4671      SYN_RECV    0          0          -
tcp        0      0 79.136.7.42:443         45.166.59.228:10201     SYN_RECV    0          0          -
tcp        0      0 79.136.7.42:443         45.166.58.55:30362      SYN_RECV    0          0          -

With no mitigation, there were hundreds of these. I ended up using a srcmask of 20 rather than 32, as the attacks comes specifically from a few Class B networks. 20 seemed a half decent tradeoff

One more thing, you will probably want to tweak /etc/sysctl.conf and add this:

net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_synack_retries = 2

The SYN backlog is normally quite small. 8192, if you have some spare memory, means you can handle more connection attempts. And setting the retries low, like 2, means you get rid of connection attempts faster when flooded with them. And you definitely want syncookies turned on.

ipv4 filter INPUT_direct 0 -p tcp -m multiport '!' --dports 22,25,53,70,80,88,119,433,443,465,587,749,993,9999 -m conntrack --ctstate NEW -j DROP
ipv4 filter INPUT_direct 1 -p tcp -m tcp '!' --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
ipv4 filter INPUT_direct 2 -p tcp --syn -m multiport --dports 22,25,53,70,80,88,119,433,443,465,587,749,993,9999 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 16 -m recent --name blacklist --set --rsource -j DROP
ipv4 filter INPUT_direct 3 -p tcp --syn -m multiport --dports 22,25,53,70,80,88,119,433,443,465,587,749,993,9999 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 16 -j LOG --log-prefix '[SYN Flood Detected] '
ipv4 filter INPUT_direct 4 -m recent --name blacklist --rcheck --seconds 900 --hitcount 1 --rsource -j DROP
ipv4 filter INPUT_direct 5 -p tcp --syn -m multiport --dports 22,25,53,70,80,88,119,433,443,465,587,749,993,9999 -m conntrack --ctstate NEW -j ACCEPT
ipv4 filter INPUT_direct 6 -p udp -m multiport '!' --dports 53,88,1194,51820 -m conntrack --ctstate NEW -j DROP

This also, apparently, needed some tweaking in /etc/nginx/nginx.conf.

http { server_tokens off; client_header_timeout 20s; client_body_timeout 20s; send_timeout 30s; keepalive_timeout 30s; client_max_body_size 10M; add_header X-Frame-Options "SAMEORIGIN" always; add_header X-XSS-Protection "1; mode=block" always; add_header X-Content-Type-Options "nosniff" always; add_header Referrer-Policy "strict-origin-when-cross-origin" always; add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always; limit_req_zone $binary_remote_addr zone=one:5m rate=9r/s; limit_req_zone $binary_remote_addr zone=two:15m rate=3r/s; }

And in the actual server definitions:

server { location / { limit_req zone=two burst=5; } }

Two-pronged approach. You filter out the SYN attacks in the firewall, but you also want to toughen up nginx (which in my case proxies for a container) so that the initial attack dies at the firewall, and if the unsubs change tactic, you basically frustrate the hell out of them.

If you want to really jerk their chain, you also do this:

-p tcp -m multiport ! --dports <the ports you have open> -m conntrack --ctstate NEW -j DROP
-p udp -m multiport ! --dports <the ports you have open> -m conntrack --ctstate NEW -j DROP

Why? It drastically slows down portscans. Like, makes them take many times longer. It can cause issues, so you might have to drop these if you start seeing ill side-effects, but this is the recommendation from nmap.org themselves.