1073

AI companies are violating a basic social contract of the web and and ignoring robots.txt (www.theverge.com)

submitted 8 months ago by TravisKelce@lemmy.world to c/technology@lemmy.world

200 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[-] ricdeh@lemmy.world 31 points 8 months ago

You cannot simply block crawlers lol

[-] bigMouthCommie@kolektiva.social 20 points 8 months ago

hide a link no one would ever click. if an ip requests the link, it's a ban

[-] T156@lemmy.world 14 points 8 months ago

Except that it'd also catch out people who use accessibility devices might see the link anyways, or use the keyboard to navigate a site instead of a mouse.

[-] bigMouthCommie@kolektiva.social 1 points 8 months ago

i don't know, maybe there's a canvas trick. i'm not a webdev so i am a bit out of my depth and mostly guessing and remembering 20-year-old technology

[-] oatscoop@midwest.social 6 points 8 months ago* (last edited 8 months ago)

If it weren't so difficult and require so much effort, I'd rather clicking the link cause the server to switch to serving up poisoned data -- stuff that will ruin a LLM.

[-] HelloHotel@lemm.ee 3 points 8 months ago* (last edited 8 months ago)

Visiting /enter_spoopmode.html will choose a theme and mangle the text for any page you next go to accordingly (think search&replace with swear words or santa clause)

It will also show a banner letting the user know they are in spoop mode, with a javascript button to exit the mode, where the AJAX request URL is ofuscated (think base64) The banner is at the bottom of the html document (not nesisarly the screen itself) and/or inside unusual/normally ignored tags. <script type="spoop/text" style='display:block">you are in spoop mode</script>

Or have a secret second page that is only followed if you ignore robots.txt /spoop_post/yvlhcigcigc is a clone of /post/yvlhcigcigc in 'spoop mode'

[-] T156@lemmy.world 2 points 8 months ago* (last edited 8 months ago)

Would that be effective? A lot of poisoning seems targeted to a specific version of an LLM, rather than being general.

Like how the image poisoning programs only work for some image generators and not others.

[-] echodot@feddit.uk 4 points 8 months ago

Well you can if you know the IPs that come in from but that's of course the trick.

[-] KillingTimeItself@lemmy.dbzer0.com 0 points 8 months ago

last i checked humans dont access every page on a website nearly simultaneously...

And if you imitate a human then honestly who cares.

this post was submitted on 14 Feb 2024

1073 points (98.6% liked)

Technology

59205 readers

2710 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS