this post was submitted on 12 Feb 2025
163 points (98.8% liked)

Opensource

2294 readers
61 users here now

A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!

CreditsIcon base by Lorc under CC BY 3.0 with modifications to add a gradient



founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] asudox@lemmy.asudox.dev 8 points 1 month ago (1 children)
[–] xtools@programming.dev 2 points 1 month ago

interesting project, thx for sharing! though:

There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS.