We've entered the Era of the AI wars, AI scrapers are openly ignoring robots.txt and other measures designed to tell bot website visitors to behave in a certain way.
If that's how it's going to be let's get into an arms race then. There's open source tools like Anubis, Cloudflare announced AI Labyrinth. This won't be the end of AI scrapers, they will get better and so will these tools and so many resources will be wasted on a problem that was already resolved if the bots worked properly
Brodie Robertson
npub1zm…90dvj
2025-03-22 22:52:05 UTC
Author Public Key
npub1zmkqmumcuhnqc9pchq5rpafqwmtwmwmhd4j7llw9e6exuf5g9e7s090dvjSeen on
wss://relay.ditto.pubPublished at
2025-03-22 22:52:05 UTCEvent JSON
{
"id": "3593be6d7cc8926fa0fb4d57f5463dd8175aa30108a85091fe6926bba6967132",
"pubkey": "16ec0df378e5e60c1438b82830f52076d6edbb776d65effdc5ceb26e26882e7d",
"created_at": 1742683925,
"kind": 1,
"tags": [
[
"proxy",
"https://mstdn.social/users/BrodieOnLinux/statuses/114208533722920462",
"activitypub"
]
],
"content": "We've entered the Era of the AI wars, AI scrapers are openly ignoring robots.txt and other measures designed to tell bot website visitors to behave in a certain way. \n\nIf that's how it's going to be let's get into an arms race then. There's open source tools like Anubis, Cloudflare announced AI Labyrinth. This won't be the end of AI scrapers, they will get better and so will these tools and so many resources will be wasted on a problem that was already resolved if the bots worked properly",
"sig": "ce1a09d935cfcbb0f9492fba4e0f0cd667ce0d08688e23a372507e3e01d41163df7bac00f06ee92830b2cc6e34f184020da9dec0dad1a0292b59ca330d544c97"
}