Join Nostr
2026-07-26 21:22:17 UTC
in reply to

Alice Pea (she/her) 🏳️‍⚧️ on Nostr: The genie has been mass scraping my small personal web server at a rate of 167 ...

The genie has been mass scraping my small personal web server at a rate of 167 requests per minute from thousands of IP addresses over the past 4 weeks already.

The genie sometimes identifies it's user agent as 'GPTBot', and completely ignores robots.txt

The genie sometimes identifies it's user agent as 'meta-externalagent', and uses Irish IP addresses to scrape my server for some reason.

The genie would've DDoS'd my server if I didn't have my server rate-limit said genie.