this post was submitted on 02 Jun 2025
35 points (100.0% liked)

TechTakes

1946 readers
286 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago
MODERATORS
dgerard@awful.systems
35
Fighting the AI scraper bots at Pivot to AI and RationalWiki (pivot-to-ai.com)
submitted 1 week ago* (last edited 1 week ago) by dgerard@awful.systems to c/techtakes@awful.systems
7 comments fedilink hide all child comments
 

video version

that s3kr1t method: https://www.jwz.org/blog/2025/05/user-agent-blocking/#comment-259266

you are viewing a single comment's thread
view the rest of the comments
[–] HedyL@awful.systems 7 points 1 week ago* (last edited 1 week ago) (3 children)

Even if it's not the main topic of this article, I'm personally pleased that RationalWiki is back. And if the AI bots are now getting the error messages instead of me, then that's all the better.

Edit: But also - why do AI scrapers request pages that show differences between versions of wiki pages (or perform other similarly complex requests)? What's the point of that anyway?

[–] BlueMonday1984@awful.systems 14 points 1 week ago

Edit: But also - why do AI scrapers request pages that show differences between versions of wiki pages (or perform other similarly complex requests)? What’s the point of that anyway?

Gonna take a shot in the dark here and guess that the people behind these AI scrapers are incredibly stupid, and think Scrape More = More Data = More Growth = More Good^TM^. Given this bubble's already given us the goddamn scourge that is "vibe coding", and is headed by growth-obsessed fuckwits and True Believers^TM^, chances are I'm completely correct on this.

load more comments (2 replies)