K. Ryabitsev-Prime 🍁 on Nostr: Scrapers are out in force in the past week or so. I'm sorry if i've not been ...
Scrapers are out in force in the past week or so. I'm sorry if i've not been responsive to your emails -- I'm trying to figure a way to deal with that. We're currently sustaining about 10,000 requests per minute just to git.kernel.org per single Toronto node, or about 40,000 requests per minute across all git.kernel.org nodes. These are not light hits -- these are random commit-id requests that require running git, or diffs, or plain patch downloads.
Like I said, the bots are solving Anubis difficulty 5, though we manage to weed out about 45% of all requests through Anubis itself, plus some analysis of the requests (some cgit query string combinations are unlikely to be ever hit by humans). Still, we're looking at about 7M requests per day that are attributable to bots. That's a guesstimate, because these bots look and act just like real browsers.
If you answer with "why don't you just..." I am going to be cross with you. There is no "why don't you just" here.
Published at
2026-08-28 03:26:04 UTCEvent JSON
{
"id": "0c3ce25a850f1412fdd44c79868844e59db59e7ed5f0674c51fd9a0a1bcc5141",
"pubkey": "925b75eded4613c6575218045e2b1d85b92a7d1f7383382b7f657210b2d3e9f8",
"created_at": 1787887564,
"kind": 1,
"tags": [
[
"proxy",
"https://social.kernel.org/objects/40c4b99e-b6fc-4b92-8a6a-a6d3ff350842",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.ditto.pub"
]
],
"content": "Scrapers are out in force in the past week or so. I'm sorry if i've not been responsive to your emails -- I'm trying to figure a way to deal with that. We're currently sustaining about 10,000 requests per minute just to git.kernel.org per single Toronto node, or about 40,000 requests per minute across all git.kernel.org nodes. These are not light hits -- these are random commit-id requests that require running git, or diffs, or plain patch downloads.\n\nLike I said, the bots are solving Anubis difficulty 5, though we manage to weed out about 45% of all requests through Anubis itself, plus some analysis of the requests (some cgit query string combinations are unlikely to be ever hit by humans). Still, we're looking at about 7M requests per day that are attributable to bots. That's a guesstimate, because these bots look and act just like real browsers.\n\nIf you answer with \"why don't you just...\" I am going to be cross with you. There is no \"why don't you just\" here.",
"sig": "1e388530911ee9dad32283020bcfd30e0c6a7278b152b6d52e538b5819f3089e73ce703494035b1e7c980439d5589da4985259e382dc1191e0b27ec998a45ee5"
}