863
The AI company Perplexity is complaining their bots can't bypass Cloudflare's firewall
(www.searchenginejournal.com)
Skill issue. Cope and seethe
this post was submitted on 19 Aug 2025
863 points (99.3% liked)
Technology
74438 readers
3932 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
I set up a WAF for my company's publicly facing developer portal to block out bot traffic from assholes like these guys. It reduced bot traffic to the site by something like - I kid you not - 99.999%.
Fucking data vultures.
Here comes the ridiculous offer to buy Google chrome with money they don't have: easy delicious scraping directly from the user source
Can someone with more knowledge shine a bit more light on this while situation? Im out of the loop on the technical details
Perplexity (an "AI search engine" company with 500 million in funding) can't bypass cloudflare's anti-bot checks. For each search Perplexity scrapes the top results and summarizes them for the user. Cloudflare intentionally blocks perplexity's scrapers because they ignore robots.txt and mimic real users to get around cloudflare's blocking features. Perplexity argues that their scraping is acceptable because it's user initiated.
Personally I think cloudflare is in the right here. The scraped sites get 0 revenue from Perplexity searches (unless the user decides to go through the sources section and click the links) and Perplexity's scraping is unnecessarily traffic intensive since they don't cache the scraped data.
load more comments
(11 replies)
load more comments
(9 replies)
Cry more, Perplexity.
I don't see a problem here. Maybe Perplexity should consider the reasons WHY Cloudflare have a firewall...?
Words cannot describe how much I hate this person
Gee that's a real removed it ain't it perplexity?
I hate that these bots ruin my read it later app. :(