Technology

Is there a more ignored file on the internet than robots.txt?

and I love you

Ooh I have an amazing cheese scone recipe from The Guardian

This is how the internet used to be, for the most part. There's no boatload of 27 JS and 15 CSS files to reference. There's no batch of 110mb splash SVGs to load so I scroll down past 3 words and see 7 stock images before getting 1 sentence of information. It's probably a 200kb site with a few 300kb images to load as well.

This is the work of an enlightened being.

Was just thinking the same

I wonder what is up with http://localghost.com/ it looks like some treasure hunt

https://floor796.com/

Def check out neocities more, also melonking.net and his projects are pretty cool, particularly https://melonland.net/surf-club has a lot of good sites on it

Its so damn snappy too!

Hosted at neocities. Wait GeoCities?? In no. Blah.

The op site is hosted on Neocities. They aim to foster that 2000s vibe. Check them out here

Checkout www.wiby.org 😏

Lol this is such a bizarre comment. Back then, AI wasn't scraping everything humans made for the profit of a few. It was a non-issue, and therefore you have no standing in claiming that "that was the whole point."

This works as well on my phone as it does on my computer, and loads faster than most modern websites making it that much more accessible to MORE humans.

The web designer isn't limiting access, they are expanding on it - for humans. The people actually sentient and able to understand their words rather than just copy and recontextualize them.

When the Web was first designed, some of the concerns we have today were nonexistent.

I believe in freedom of information, and would love for the information I share to be accessed in any way a given user wants.

But I have to stand defensive and support the author here, too. The modern LLM boom aims to essentially replace original resources with AI-generated summaries step by step. This is detrimental to the Internet, and to knowledge as we know and preserve it.

First, there is an event commonly called Google Zero, which is briefly mentioned in the article. If you don't know what it is, it is the not-so-hypothetical-anymore moment when Google (or, really, any other large player) essentially accumulates all information on the Web, feeds it to AI, and since then doesn't serve links anymore, going straight to answers based on training data. Users will jump to this - they already do - because it offers convenience. But for any independent creators it means having no audience, no money, and no means to produce new quality content, trapping users in a self-containing loop that loses nuance, actuality, and truthfulness, and stays under corporate control. This goes beyond cooking recipes and personal notes - it permeates science, political discussion, and much more.

Second, LLMs multiply traffic coming to sites, which becomes an infrastructural problem. Bots access sites at much higher rates than humans do, and when their intent is to scrape your entire website every now and again and there are dozens of them, this becomes huge.

Third, having proprietary models train on the data I provide without any attribution, copyright etc. makes giant corporation profit off my back, while at the same thing making it so that less genuine users will see what I produce. This means careers of authors, journalists etc. are dying, and this also means they are left free to abuse each and every one of us without any consent.

Fourth, and I wonder if you see it by now, LLMs and the way they represent data, along with SEO tools meant to drive information through the search bots, begin to shape how we talk. All I say doesn't have to be a list of points, yet it is. It could be less verbose, more readable, yet it is the way it is. Because when we interact with the products of such developments too much, we begin shaping our own language in a way that is less human-readable and more meant for machines, without us often being aware of it. This is a real issue of communication.

So, as much as I hate it, I'm gonna protect a lot of the data I share.

Open until your server is down because LLM are overloading it

They did have a lot of concerns with abuse though and you can see that in the way the cookies debate went before they were supported in their current form. I think AI crawlers tanking bandwidths for websites and misusing the data they scrape would 100% be something the Mozilla from back then would've had concerns over allowing or encouraging.

Instructions unclear, built whole site with nested tables.

haha tarpit goes brrr