1129
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.
(i.imgur.com)
cross-posted from: https://programming.dev/post/35852706
The internet came together to define the robots file standard, it could just as easily come with a standard API for database dumps. But decided on war since the 2023 API wars and now we're going to see all the small websites die while facebook gets even more powerful.
Well there you have it. Although I still feel weird that it's somehow "the internet" that's supposed to solve a problem that's fully caused AI companies and their web crawlers.
If a crawler keeps spamming and breaking a site I see it as nothing short of a DOS attack.
Not to mention that
robots.txtis completely voluntary and, as far as I know, mostly ignored by these companies. So then what makes you think that any them are acting in good faith?To me that is the core issue and why your position feels so outlandish. It's like having a bully at school that constantly takes your lunch and your solution being: "Just bring them a lunch as well, maybe they'll stop."
The solution is breaking intellectual property and making sharing public data easy and efficient. A top-down imposition DESIGNED to crush the giants back down to the level playing field of the small players into a system where cooperation empower the small and place the burdens on the big with the understanding that all public data is "our" data and nobody, including its custodian should get between US and IT. Something designed by actually competent and clever politicians who will anticipate and counter all the dirty tricks big tech would try to regain the upper hand. I want big tech permanently losing on a field designed to disadvantage anything that accumulates power.