Published in · 3 min read · Oct 9, 2022
--
TLDR: IP Bans don’t work against bots because they just change their IP addresses. What does works? Bot whitelists. Let in only good bots and block all the rest.
A common technique in stopping bots is IP Bans.
If bot mitigation software sees a bot, it bans the IP address the bot comes from. Bot mitigation companies will then keep blacklists of IPs/CIDR ranges. Even more advanced companies will have a reputation score for IP addresses.
That used to work. But the Internet has changed alot in the last five years.
Three big changes have made IP bans ineffective.
It used to be you had a computer at home that had an IP address and that address changed infrequently. Now with cell phones, your IP address changes depending on the cell tower you are using.
Within some cellular carriers, a single IP address can cater to more than 4,000 devices per day, making cellular traffic an ideal location for bots to remain undetectable. As mobile devices move through different gateways, (based on device owners changing location throughout the day,) bots effectively change identities to make detection even more difficult.
IP addresses are shared by many tower users and reused all the time. So a bad actor could visit your site from an IP address and 3 minutes later, a legitimate person could do the same. If you block IP addresses, you risk blocking legitimate users especially cell phone users.
Bot mitigation companies evolved beyond simple IP blocks to blocking datacenter IPs and bad reputation IPs. The problem is bot makers and fraudsters realized this and moved onto the next strategy.
There are many companies that rent out residential IP addresses. They are from real ISPs and look like a regular user visiting from their home IP address.
I will say Datadome has done some good work with machine learning to ferret out residential proxies. However any bot mitigation companies are still easily fooled by residential proxies.
The problem only gets worse as bot makers have evolved even more and created a volume problem in the form of mobile proxies.
5G along with IPV6 has allowed for a much larger set of IP addresses to be used.
IPV4 had 4,294,967,296 IP addresses.
IPV6 has 340,282,366,920,938,463,463,374,607,431,768,211,456 IP addresses.
For $5-$10, a bot can rotate to a brand new mobile IP address every 30 seconds if they want.
Making IP bans all but useless against sophisticated bots.
Even the most sophisticated machine learning is going to run into trouble when a mobile IP address is used by both bots and humans. It won’t be able to easily divine patterns. This means alot of false positives and pissed humans. With the sheer volume of addresses with IPV6, storing reputation data on every IPV6 address isn’t feasible.
So bot mitigation companies will continue to play a game of whack a mole.
If IP bans don’t work, what should you do to stop bots? “Invert, always invert” as Charlie Munger says. Instead of doing IP bans aka using an IP blacklist, you should use an IP whitelist for bots.
It sounds impossible and crazy, but I’ll lay out in another article exactly how it works.
