316
submitted 2 months ago by lemmee_in@lemm.ee to c/fuck_ai@lemmy.world

Meta has quietly unleashed a new web crawler to scour the internet and collect data en masse to feed its AI model.

The crawler, named the Meta External Agent, was launched last month, according to three firms that track web scrapers and bots across the web. The automated bot essentially copies, or “scrapes,” all the data that is publicly displayed on websites, for example the text in news articles or the conversations in online discussion groups.

A representative of Dark Visitors, which offers a tool for website owners to automatically block all known scraper bots, said Meta External Agent is analogous to OpenAI’s GPTBot, which scrapes the web for AI training data. Two other entities involved in tracking web scrapers confirmed the bot’s existence and its use for gathering AI training data.

While close to 25% of the world’s most popular websites now block GPTBot, only 2% are blocking Meta’s new bot, data from Dark Visitors shows.

Earlier this year, Mark Zuckerberg, Meta’s cofounder and longtime CEO, boasted on an earnings call that his company’s social platforms had amassed a data set for AI training that was even “greater than the Common Crawl,” an entity that has scraped roughly 3 billion web pages each month since 2011.

you are viewing a single comment's thread
view the rest of the comments
[-] GarrulousBrevity@lemmy.world -1 points 2 months ago

I guess I don't really see the problem with that though. There are configuration levers you could be pulling, but those sites you're hosting are not. There are lots of shady questions about how these models are getting training data, but crawlers have a well defined opt out mechanism.

The web would not be what we know it as without them, because it's how you find sites. Why shouldn't Alta Vista have one? I don't object to what Alta Vista does with the data.

[-] aniki 6 points 2 months ago

Mate we have absurdly restrictive robots.txt including a custom WordPress plugin that automatically generates the file and the bots don't give a fuck.

[-] GarrulousBrevity@lemmy.world 0 points 2 months ago

But meta's will, and Alta Vista. I'm not angry at them when a script kitty makes a bad crawler

this post was submitted on 25 Aug 2024
316 points (97.9% liked)

Fuck AI

1346 readers
57 users here now

"We did it, Patrick! We made a technological breakthrough!"

A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.

founded 8 months ago
MODERATORS