753
submitted 1 year ago by BrikoX@vlemmy.net to c/worldnews@lemmy.ml
you are viewing a single comment's thread
view the rest of the comments
[-] Veltoss@lemmy.world 74 points 1 year ago

How does Pinterest get around this then? They pollute image searches like crazy, and require you to login to see anything. At least they did, I blocked them from searches so maybe it's different now.

[-] gressen@lemmy.world 14 points 1 year ago

Easy - detect if you're getting accessed by a search crawler or a human. Serve a full page or just a login request.

[-] RGB3x3@lemmy.world 11 points 1 year ago

So how can a user pretend to be a web crawler?

[-] theMightyMoonWorm@lemmy.ml 22 points 1 year ago* (last edited 1 year ago)

This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html

load more comments (4 replies)
load more comments (4 replies)
load more comments (8 replies)
this post was submitted on 05 Jul 2023
753 points (98.3% liked)

World News

32311 readers
744 users here now

News from around the world!

Rules:

founded 5 years ago
MODERATORS