How can I block a list of about 1.4 million IP addresses? I've already tried to do it with iptables PREROUTING, like:
-A PREROUTING -d IP_HERE/32 -j DROP
But with this many records, my bandwidth goes down like crazy when I do a speedtest.
Without blocked IPs in iptables:
1 Gb/s
With blocked IPs in iptables:
3 Mb/s at peak.
I want to use XDP_DROP like here (last step):
But I don't have an idea how to use this. :/ (I'm really bad at programing)
Are there alternatives to this approach?
117 Answers
You should have a look into ipset.
From the official website:
Ipset may be the proper tool for you [...] to store multiple IP addresses or port numbers and match against the collection by iptables.
[...] (Ipset) may store IP addresses, networks, (TCP/UDP) port numbers, MAC addresses, interface names or combinations of them in a way, which ensures lightning speed when matching an entry against a set.
To use it, you need to create an ipset, add the IPs and create an iptables rule to match with the ipset:
ipset create blacklist hash:ip hashsize 1400000
ipset add blacklist <IP-ADDRESS>
iptables -I INPUT -m set --match-set blacklist src -j DROPA real life example of usage can be found here. Notice that it uses ipset restore instead of going through each IP in a loop because it’s much more faster.
If your list of IPs has overlaps, you may want to preprocess it to convert to IP ranges where possible. Here is an example of a tool to do it. It won't get you better performances with ipset but it will reduce the size of your list.
On a side note, in term of performances, it is very fast and scale without penalty. As the Cloudflare's blog mention, there are faster low level approaches; but it's much more complex and only adds a few bytes per seconds, which, unless you have the scale and ambition of a cloud provider, are not worth the effort.
Frame challenge - what's the shorter list, authorised or blocked addresses?
Rather than denying 1.4 million, simply allow the perhaps ~dozen IPs you want to permit, and default-deny everything.
1If the IP addresses operate in a well-defined range, then you can use ufw like this to block traffic:
sudo ufw deny from 192.0.0.0/8 to anyThe example above blocks all traffic from 192.0.0.1 to 192.255.255.254, which works out to 16,777,214 addresses and this has zero (noticeable) effect on network throughput.
So long as your IP list is in a workable fashion to generate IP ranges, this may work for you.
9You can minimize look-ups to gain more speed by tree-structuring your rules. You can for example do it based on the first part of the IP i.e. /8 like so:
iptables -N rule8_192_0_0_0
iptables -N rule8_172_0_0_0
iptables -N rule8_10_0_0_0
iptables -A INPUT -s 192.0.0.0/8 -j rule8_192_0_0_0
iptables -A INPUT -s 172.0.0.0/8 -j rule8_172_0_0_0
iptables -A INPUT -s 10.0.0.0/8 -j rule8_10_0_0_0
iptables -A rule8_192_0_0_0 -s 192.168.2.3 -j DROP
iptables -A rule8_172_0_0_0 -s 172.16.2.3 -j DROP
iptables -A rule8_10_0_0_0 -s 10.10.2.3 -j DROPThere's another improvement that directly solves your 3 Mb/s problem:
iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPTThis allows established connections to traverse as few iptables rules as possible, although using ipset to improve the IP address lookup speed is still necessary for new connections to establish faster.
No matter how many other rules you have, this is a good one to deploy as the first rule.
0This doesn't use iptables but the ip kernel routing table, it may be worth trying it and check for performances:
ip route add blackhole IPv4/32
IIRC it's supposed to be faster than filtering with iptables, but I've never done a benchmark with 1.4 million IPs :)
XDP_DROP is probably overkill unless you plan on running these blocklists at extremely high packet speeds (Think >1mpps). As such i would recommend Cyrbil's anwser if you aren't that experienced with code.
If you nevertheless want to try with XDP you are looking for something called a bloom filter which is able to quickly check if a ip is "possibly in set" or "definitely not in set"
A example of a bloom filter in C: This blog post
