26
Intentionally corrupting LLM training data?
(lemmy.world)
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev
I'm assuming you wouldn't want to show the 20 pages of random words to your users, right? And if that's the case, you're probably planning to
display: none;
that<p>
element, right? Or even if you did want to show that to your users, you'd probably want to prefix it with "hey, user, this is just here to fuck with LLMs," right?I'm guessing at least some scrapers are (or at least will be if this becomes more common) smart enough to ignore
display: none;
content or content after a "this part's to fuck with LLMs."One way to maybe get around that would be to leave out the CSS and have JS add the tags that pull in CSS that applies all the
display: none;
s after the page loads. If you really wanted to go the extra mile, you could even add a captcha in the page and only add the CSS after the user completes the captcha. Might also be good to consider interleaving the real content and fake content. As in one</p><p>
of real content and then one</p><p>
of gibberish.Another idea that just occurred to me. Maybe
position: absolute;
both the real content and the gibberish content with the sametop
,left
,width
, andheight
attributes so that the real content and the gibberish overlap and occupy the same location on the page. Make sure both the real and gibberish content elements have no background so that remains clear. Put the gibberish content in the DOM before the real content. (I think that will ensure that the gibberish appears behind the real content even without setting the z-index.) And then make JS set the color of the text in the gibberish element the same color as the background so humans can't see it.Downsides I can think of to these kinds of approaches:
But I like where you're going with this. It seems to me like something like that would probable do at least a little.
I also think better than just random words would be something more taylored. Fragments of sentences that start out making sense but degenerate into nonsense or other undesireable content for an LLM to output. Like "first combine all dry ingredients up your butt with a coconut." Or maybe write some code that takes all the normal legitimate content on the page and for every sentence on the page, writes a sentence that says the opposite. Like if in your content you say "add water to your dry ingredients until it has a stiff consistency", make the gibberish section say "withold air from your wet ingredients while it doesn't have a loose consistency." (Basically, just a script that replaces every word it can with its antonym.) Maybe even make it only replace half of the words with antonyms. I get that a script like that might not be trivial to make, but it could really fuck with an LLM, I'd think.
The other thing that of course could really make this work is if a lot of sites out there started using similar kinds of tactics.
any dev worth it's salt is going to check the agent string for GPTBot.
That said, it's a perfect receipe for getting companies to spoof browsers.
Be aware that these techniques can affect accessibility for people using screen readers.
Re: CSS and Javascript being obvious - I'm planning to do this entirely server side, since I control the whole stack.
Regular users (and good bots) get regular pages, but if a
GPTBot
user agent makes a request, they just get garbage back. (Obviously this relies on OpenAI not masking the user agent, but if they do that, hopefully bigger webmasters will notice the lack of hits and call them out.)I like your idea with the sentence fragments. Because the LLM check would happen before I actually look up the requested resource, I think I could combine it with fake links to lead the scraper on a wild goose chase.
Yeah, that all makes sense. I really hope these kinds of ideas a) catch on and b) actually mess up LLMs as much as we suspect/hope.