Large Heydon Collider on Nostr: "Now it’s LLMs. If you think these crawlers respect robots.txt then you are several ...

"Now it’s LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality."

I never updated my robots.txt to block LLMs--as many suggested--because, well... come on.

https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html