@S0X

I do actually agree with a lot of that. The challenge is in finding the balance between different layers of security and dynamic firewalls vs the reality of what a server sees and can cope with, even with all the best optimisations already employed. You might say it's a constant phase of learning.

We don't have much of a whitelist approach on this server, the goal is to avoid that - partly for the reasoning you use above. But I do still need to see an IP address in order to know how the firewalls are reacting to specific connections or URLs, and adjust the rules if necessary.

If somebody is seeing a 403 from the server itself, then they're probably on an IP or closely associated range that the firewall determines has recently attacked the server (not just merely scraping botnets), or they're connecting from an address that is currently or was previously associated with several blacklisted countries or ASNs (you can probably guess the few that might fall into this bracket - some are notorious, but I'm not going to list details of our security policy for hopefully obvious reasons).

Most other times the 403 will only be short-term or medium-term dynamic restrictions, based on the firewall's own decision-making or because you've tried to access a restricted part of the site. There are actually four layers of firewall style security on servers like this, one of which is imposed by the hosting firm and not under my control. But two of those don't block IPs, they're more content specific filtering.

So to be clear, it's not a case of intentionally setting out to perma-block VPNs or anything like that. But the fact remains that a lot of attack traffic comes from them, you're just more likely to run into an obstruction today than a few years ago due to the 10,000x fold increase in such nefarious automated activity.

On the subject of hard captches. Yes I totally agree, although the only ones we use are the standard CloudFlare "click this box" ones (except on some sign-up forms, but that's a different thing), which are now common on many sites. This is a new normal. But those aren't imposed at individual IP level either, they're usually country or ASN level. I much prefer those to 403s by the server's own firewall, which is all actively part of that balancing work and learning I mentioned earlier.

On the subject of spamming. That specifically hasn't really been a big problem on this site for quite a few years. The modern systems and methods cope with that quite well, even without server side restrictions. The issue is more the bots that scrape and probe websites and then provide that data to the spammers.

Just for a little context. If I switched off all the security and filtering measures right now for this forum, you'd have about 20,000+ guests and server load making it sluggish for all - akin to a perpetual mild DDoS. There's never going to be a perfect balance, a small minority of users will always be inconvenienced to protect the majority. If you have a bank balance and server that can cope with that, then great - embrace the bots, but alas we do not.