Currently trying to deal with AI crawlers that:
1. Make requests every few seconds.
2. Don't respect crawl-delay directive in robots.txt.
3. Use a huge range of IPs, so can't block on a 'too many requests from this IP' fail2ban rule.
All to feed their LLMs which then hallucinate anyway.
One crawler alone made over 18,000 requests in 24 hours.
Paul
npub1ys…pt2sa
2026-05-26 15:21:58 UTC
Author Public Key
npub1yslhdmayhshjsyazvttrtrxcsy794p0c7s8vdl2j85v4unumh5kqspt2saSeen on
wss://relay.momostr.pinkPublished at
2026-05-26 15:21:58 UTCEvent JSON
{
"id": "d093e26fa91bc6124c20765aefd0fa6c5b087d549b06c2e8a44daa883928e68c",
"pubkey": "243f76efa4bc2f2813a262d6358cd8813c5a85f8f40ec6fd523d195e4f9bbd2c",
"created_at": 1779808918,
"kind": 1,
"tags": [
[
"proxy",
"https://social.xk7.net/@pwaring/116641557250438908",
"web"
],
[
"proxy",
"https://social.xk7.net/users/pwaring/statuses/116641557250438908",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://social.xk7.net/users/pwaring/statuses/116641557250438908",
"pink.momostr"
],
[
"-"
]
],
"content": "Currently trying to deal with AI crawlers that:\n\n1. Make requests every few seconds.\n\n2. Don't respect crawl-delay directive in robots.txt.\n\n3. Use a huge range of IPs, so can't block on a 'too many requests from this IP' fail2ban rule.\n\nAll to feed their LLMs which then hallucinate anyway.\n\nOne crawler alone made over 18,000 requests in 24 hours.",
"sig": "acb3aada240ec5a2175e4045e8ad642046c3f8b2e5b91d7e2eee59e36101cfda856787d627488bcefc3318abd548c80e9ee04764827f9c78c6fb1a89404b59c5"
}