Currently trying to deal with AI crawlers that:
1. Make requests every few seconds.
2. Don't respect crawl-delay directive in robots.txt.
3. Use a huge range of IPs, so can't block on a 'too many requests from this IP' fail2ban rule.
All to feed their LLMs which then hallucinate anyway.
One crawler alone made over 18,000 requests in 24 hours.
Paul
npub1ss…846m8
2026-05-26 15:21:58 UTC
Author Public Key
npub1ssvx8l4lq8wtyn66hkudzdllz4lh0y32qravynpw67srvt2g7f2qx846m8Seen on
wss://relay.ditto.pubPublished at
2026-05-26 15:21:58 UTCEvent JSON
{
"id": "5ce6e791cfb944074a02a80dc9e7e5d9480b194f423868385a50b0d47debb048",
"pubkey": "841863febf01dcb24f5abdb8d137ff157f77922a00fac24c2ed7a0362d48f254",
"created_at": 1779808918,
"kind": 1,
"tags": [
[
"proxy",
"https://social.xk7.net/users/pwaring/statuses/116641557250438908",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.ditto.pub"
]
],
"content": "Currently trying to deal with AI crawlers that:\n\n1. Make requests every few seconds.\n\n2. Don't respect crawl-delay directive in robots.txt.\n\n3. Use a huge range of IPs, so can't block on a 'too many requests from this IP' fail2ban rule.\n\nAll to feed their LLMs which then hallucinate anyway.\n\nOne crawler alone made over 18,000 requests in 24 hours.",
"sig": "e6a7321b18cdc9c279e98a343b8dc225c64fccded7c450b5262e3ba1b3df8ca6891f1f599d91dd76ed2a34a3a715a0c13a828fe35abaad546fa1ea8bd0b0dc2d"
}