After having read about aggressive AI bots scraping Open Source project websites, Git repositories (and other sites) [1] I decided to implement more aggressive blocking of such bots on metalhead.club.
So far I've only added their user agents to "robots.txt", but that is a very polite way of saying "don't scrape any content here".
Today I've implemented a more harsh way: If a bot user agent is detected by the web server, it blocks access right away!
My configuration is based on the ai-robots-txt generator by nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqu95u6yejxp0qwkkh9z6dpkptd8f0vudsxm4hkmyku4kljh5sy88qnu74gf (nprofile…74gf) , nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpq7gduh28layuz0qc9jjcda2794n2p3z8yfzyueank5f3t4whw226s3dz04q (nprofile…z04q) , nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqy9zwp8x30rmeptkk60t7m60fpmu9ddykf3m77wkcau3nusx5elsqg4q8jv (nprofile…q8jv) and others
I just added another "exporter" for Nginx config: https://github.com/ai-robots-txt/ai.robots.txt/pull/94
[1]: https://glm.io/194704
#ai
Tom :damnified:
npub1xc…59y48
2025-03-27 15:21:11 UTC
Author Public Key
npub1xcn37xn9rajvwjzdd4td468fnssj07xs9v29m5fw8xr52qxmexys259y48Published at
2025-03-27 15:21:11 UTCEvent JSON
{
"id": "da2cbf9c6b1ef90ef7be7dc760b97aaa32c4c2346485c8288fe4d354d7b34c75",
"pubkey": "36271f1a651f64c7484d6d56dae8e99c2127f8d02b145dd12e39874500dbc989",
"created_at": 1743088871,
"kind": 1,
"tags": [
[
"p",
"e169cd1332305e075ad728b4d0d82b69d2f671b036eb7b6c96e56df95e9021ce",
"wss://relay.mostr.pub"
],
[
"p",
"f21bcba8ffe93827830594b0deabc5acd41888e44889ccf676a262babaee52b5",
"wss://relay.mostr.pub"
],
[
"p",
"2144e09cd178f790aed6d3d7ede9e90ef856b4964c77ef3ad8ef233e40d4cfe0",
"wss://relay.mostr.pub"
],
[
"p",
"cb3b40e3f18a7a1dbf9d4fc44d99c2bc03041a29c2c695e145adb3169cdc1ce0",
"wss://relay.mostr.pub"
],
[
"p",
"ec9fca73c602a6e2a693a7fb18ead508a349a031572e4b8a636a7e364bb1bf9b",
"wss://relay.mostr.pub"
],
[
"p",
"65cd8b3ef65f48f32b01896d26bc0147b906a61b16bccd4891f83121fba31ea2",
"wss://relay.mostr.pub"
],
[
"t",
"ai"
],
[
"proxy",
"https://metalhead.club/users/thomas/statuses/114235072273303641",
"activitypub"
]
],
"content": "After having read about aggressive AI bots scraping Open Source project websites, Git repositories (and other sites) [1] I decided to implement more aggressive blocking of such bots on metalhead.club. \n\nSo far I've only added their user agents to \"robots.txt\", but that is a very polite way of saying \"don't scrape any content here\". \n\nToday I've implemented a more harsh way: If a bot user agent is detected by the web server, it blocks access right away!\n\nMy configuration is based on the ai-robots-txt generator by nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqu95u6yejxp0qwkkh9z6dpkptd8f0vudsxm4hkmyku4kljh5sy88qnu74gf , nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpq7gduh28layuz0qc9jjcda2794n2p3z8yfzyueank5f3t4whw226s3dz04q , nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqy9zwp8x30rmeptkk60t7m60fpmu9ddykf3m77wkcau3nusx5elsqg4q8jv and others\n\nI just added another \"exporter\" for Nginx config: https://github.com/ai-robots-txt/ai.robots.txt/pull/94\n\n[1]: https://glm.io/194704\n\n#ai",
"sig": "16826b5ea58269f07c50871260e1e2690406b835bc76378761868e42a53dfb5f276c969487cc3fe61146ef5ac86cde8bba4a00fbfd27925af1becdf3d8feedcf"
}