Wolf480pl on Nostr: Remember when most scrapers respected robots.txt, and even ones that didn't tried ...
Remember when most scrapers respected robots.txt, and even ones that didn't tried hard to keep their request rate low enough so as not to be noticable to the website's admins?
Published at
2026-09-03 09:17:49 UTCEvent JSON
{
"id": "5c2d499c4a1f5d8b9afbc92dd689aa82eea7e2c1cbf71dde635325090bbdb150",
"pubkey": "d903cc21d6cc3b8f84ae9c167792a1de95f938165aa47ed67735857e44908eac",
"created_at": 1788427069,
"kind": 1,
"tags": [
[
"proxy",
"https://mstdn.io/@wolf480pl/117206356395553860",
"web"
],
[
"proxy",
"https://mstdn.io/users/wolf480pl/statuses/117206356395553860",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mstdn.io/users/wolf480pl/statuses/117206356395553860",
"pink.momostr"
],
[
"-"
]
],
"content": "Remember when most scrapers respected robots.txt, and even ones that didn't tried hard to keep their request rate low enough so as not to be noticable to the website's admins?",
"sig": "a7ea94ad78e1e0fb9bcd66d1c24de97e6873b44ec65e988667e0ce07e7ff1f06c42b0587884b173108baca2cee2777965128f98f5193733ba584448350685316"
}
