32
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 21 Jul 2024
32 points (100.0% liked)
TechTakes
1427 readers
106 users here now
Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.
This is not debate club. Unless it’s amusing debate.
For actually-good tech, you want our NotAwfulTech community
founded 1 year ago
MODERATORS
Not a sneer, but a mildly interesting open letter:
The basic idea is effectively an extension of robots.txt which attempts to resolve the issue by providing a means to politely ask AI crawlers not to scrape your stuff.
Personally, I don't expect this to ever get off the ground or see much usage - this proposal is entirely reliant on trusting that AI bros/companies will respect people's wishes and avoid scraping shit without people's permission.
Between OpenAI publicly wiping their asses with robots.txt, Perplexity lying about user agents to steal people's work, and the fact a lot of people's work got stolen before anyone even had the opportunity to say "no", the trust necessary for this shit to see any public use is entirely gone, and likely has been for a while.
OpenAI is creating the trustless economy that makes bitcoin necessary. taps forehead
The best proposal I've seen so far ~~short of destroying all AI scrapers~~, and essentially what anyone familiar with the specs would come up with.
The only thing I'd add is an analogue to
data-nosnippet
to exclude only specific sections of the HTML document (w/o needing to reach for an entire iframe); though that's harder to implement on the crawler end so maybe that's for the best.Google uses a second User-Agent directive; while Bing suggests using noarchive. Both of these are pretty hacky and not general, so it'd be good to see the industry standardize on the above proposal.
The proposal itself does still assume that AI scrapers are being run by decent human beings with functioning moral compasses, which is why I feel its inadequate.
This take might be overly harsh on AI/tech as a whole, but at this point I've run out of patience regarding this bubble and see no reason to believe anyone in the AI space is a decent human being, at least for the time being.
News article about how hard it is to block AI scraping: https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/
At this point, I wouldn't fault anyone for blanket-blocking all scrapers/robots - sure, doing that will make you unfindable by search engines, but search is basically useless nowadays for finding anything actually interesting, and trying to play whack-a-mole with AI scrapers just means you're gonna get your shit stolen.
Might as well go back to word-of-mouth.