Andrew Golding on Nostr: automatically redirect requests for robots.txt itself as well, maybe use fail2ban or ...

automatically redirect requests for robots.txt itself as well, maybe use fail2ban or something similar to send all future requests from something that tries to read robots.txt first straight up the hole. The implicit social contract for crawlers is broken, maybe it's time to stop embracing them.