Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
I admire the optimism to see it this way and not “it’s still not worth it to them to bother blacklisting the domain”
I wonder too, why they didn’t, because they’re happily crawling domains that never had anything but junk on them. To me, that suggests they have no idea they’re trapped. Not at crawling time at least.