It's not in the interest of all the techbros to sell the new age AIshit as something less that can only do such small thing. They need to hype the shit out of it to get all the crazy investors money that understand nothing about it but only see AI buzzwords everywhere and need to go for it now because of FOMO.

It's only gonna get much worse before it is toned down to appropriate usage.

Don't even need to make it about code. I once asked what a term meant in a page full of a certain well known FOSS application's benchmarks page. It gave me a lot of garbage that was unrelated because it made an assumption about the term, exactly the assumption I was trying to avoid. I try to deviate it away from that, and it fails to say anything coherent and then loops back and gives that initial attempt as the answer again. I was stuck unable from stopping it from hallucinating.

How? Why?

Basically, it was information you could only find by looking at the github code, and it was pretty straightforward - but the LLM sees "benchmark" and it must therefore make a bajillion assumptions.

Even if asked not to.

I have a conclusion to make. It does do the code thing too, and it is directly related. Once asked about a library, and it found a post where someone was ASKING if XYZ was what a piece of code was for - and it gave it out as if it was the answer. It wasn't. And this is the root of the problem:

AI's never say "I don't know".

It must ALWAYS know. It must ALWAYS assume something, anything, because not knowing is a crime and it won't commit it.

And that makes them shit.