this post was submitted on 23 Aug 2025
-2 points (47.5% liked)

Ask Lemmy

34180 readers
2355 users here now

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have funDoxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spamPlease do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reasonJust remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.
It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.


6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


founded 2 years ago
MODERATORS
 

Disclaimer: I don't have a background in computer science

I recently heard about a lightweight opensource software called Anubis.

Anubis is designed to stop AI crawlers that download a lot of data to train artificial intelligence models

https://anubis.techaro.lol/

Several websites have deployed Anubis:

https://gitlab.gnome.org/GNOME

10 websites have deployed. 10 websites. Out of millions of websites.

My question is extremely simple.

If this software is so damn great, why isn't it everywhere?

Seriously. Why isn't it used on Lemmy? On Wikipedia? On CBC?

top 16 comments
sorted by: hot top controversial new old
[–] Shadow@lemmy.ca 4 points 15 hours ago

It's not great, it's actually pretty ineffective.

https://lock.cmpxchg8b.com/anubis.html

[–] Luffy879@lemmy.ml 1 points 14 hours ago

10 websites have deployed.

Source?

Because many already have solved the Problem via cloudflare, etc.

[–] icystar@lemmy.cif.su 0 points 13 hours ago* (last edited 13 hours ago)

It's not that great, and the creator seems to have some weird fixation on little girls.

[–] slazer2au@lemmy.world 25 points 1 day ago

Because it's not a perfect solution and other sites have other solutions in place.

[–] traches@sh.itjust.works 9 points 1 day ago
  • not every website has the problem it solves
  • not everyone who does likes the solution it offers
  • web development moves fast but not that fast
[–] sbv@sh.itjust.works 11 points 1 day ago (3 children)

Weird that this is getting downvotes. It's a legit question.

[–] icystar@lemmy.cif.su 1 points 13 hours ago

I think we're probably going to see a schism at some point in the fediverse.

The fragile people who can't seem to tolerate anything they don't like will end up in their own bubble.

[–] underline960@sh.itjust.works 10 points 1 day ago

My guess is it's the tone/wording that implies that OP doesn't think it's great.

I agree that it's a legit question, though.

[–] ArgumentativeMonotheist@lemmy.world 6 points 1 day ago (1 children)

People get sensitive when religion is mentioned!

[–] stringere@sh.itjust.works 4 points 1 day ago (1 children)
[–] zloubida@sh.itjust.works 3 points 1 day ago

Pffff stop with your superstitions. Anybody serious knows that Teutates is the best.

[–] fenrrs@lemmy.world 9 points 1 day ago* (last edited 1 day ago) (2 children)
[–] sbv@sh.itjust.works 7 points 1 day ago

Yeah. It's an arms race. Any technological defense will be countered eventually. In the long run, I'm not sure a technical defense like this will be sufficient - we'll need legal defenses that are enforced.

Yeah, that’s the dilemma. The more popular it gets, the more crawlers will be designed to circumvent it.

[–] kbal@fedia.io 7 points 1 day ago

It isn't so great, and it is everywhere.

[–] Ephera@lemmy.ml 3 points 1 day ago

I believe, (far too) much of the commercial world relies on Cloudflare to solve that problem.

And as for Wikipedia, any AI trainer worth their salt should know that they don't need to crawl it, because you can actually just download the whole Wikipedia dataset.