this post was submitted on 23 Aug 2025
-1 points (48.8% liked)

Ask Lemmy

34180 readers
3324 users here now

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have funDoxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spamPlease do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reasonJust remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.
It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.


6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


founded 2 years ago
MODERATORS
 

Disclaimer: I don't have a background in computer science

I recently heard about a lightweight opensource software called Anubis.

Anubis is designed to stop AI crawlers that download a lot of data to train artificial intelligence models

https://anubis.techaro.lol/

Several websites have deployed Anubis:

https://gitlab.gnome.org/GNOME

10 websites have deployed. 10 websites. Out of millions of websites.

My question is extremely simple.

If this software is so damn great, why isn't it everywhere?

Seriously. Why isn't it used on Lemmy? On Wikipedia? On CBC?

top 16 comments
sorted by: hot top controversial new old
[–] Shadow@lemmy.ca 4 points 22 hours ago

It's not great, it's actually pretty ineffective.

https://lock.cmpxchg8b.com/anubis.html

[–] Luffy879@lemmy.ml 1 points 21 hours ago

10 websites have deployed.

Source?

Because many already have solved the Problem via cloudflare, etc.

[–] slazer2au@lemmy.world 25 points 2 days ago

Because it's not a perfect solution and other sites have other solutions in place.

[–] icystar@lemmy.cif.su 0 points 20 hours ago* (last edited 20 hours ago)

It's not that great, and the creator seems to have some weird fixation on little girls.

[–] traches@sh.itjust.works 9 points 1 day ago
  • not every website has the problem it solves
  • not everyone who does likes the solution it offers
  • web development moves fast but not that fast
[–] sbv@sh.itjust.works 11 points 2 days ago (3 children)

Weird that this is getting downvotes. It's a legit question.

[–] icystar@lemmy.cif.su 1 points 20 hours ago

I think we're probably going to see a schism at some point in the fediverse.

The fragile people who can't seem to tolerate anything they don't like will end up in their own bubble.

[–] underline960@sh.itjust.works 10 points 2 days ago

My guess is it's the tone/wording that implies that OP doesn't think it's great.

I agree that it's a legit question, though.

[–] ArgumentativeMonotheist@lemmy.world 6 points 2 days ago (1 children)

People get sensitive when religion is mentioned!

[–] stringere@sh.itjust.works 4 points 1 day ago (1 children)
[–] zloubida@sh.itjust.works 3 points 1 day ago

Pffff stop with your superstitions. Anybody serious knows that Teutates is the best.

[–] fenrrs@lemmy.world 9 points 2 days ago* (last edited 2 days ago) (2 children)
[–] sbv@sh.itjust.works 7 points 2 days ago

Yeah. It's an arms race. Any technological defense will be countered eventually. In the long run, I'm not sure a technical defense like this will be sufficient - we'll need legal defenses that are enforced.

Yeah, that’s the dilemma. The more popular it gets, the more crawlers will be designed to circumvent it.

[–] kbal@fedia.io 7 points 2 days ago

It isn't so great, and it is everywhere.

[–] Ephera@lemmy.ml 3 points 2 days ago

I believe, (far too) much of the commercial world relies on Cloudflare to solve that problem.

And as for Wikipedia, any AI trainer worth their salt should know that they don't need to crawl it, because you can actually just download the whole Wikipedia dataset.