Just your daily reminder to not trust or at the very least, fact check whatever chatgpt spews out because not only does it blatantly lie, but it also makes stuff way more than youd want to believe.
(btw batrapeton doesnt exist and is a fictional genus of jurassic amphibians that I made up for a story that I am writing. They never existed in any way shape or form and neither is there any trace of info about them online yet here we are with chatgpt going "trust me bro" about them lol)
LLMs can't say they don't know. It's better for the business to make up some bullshit than just say "I don't know" because it would show how useless they can be.
You're right, but for a different reason as well. The way these models are trained is by "taking tests" over and over. Wrong answers, as well as saying "I don't know", both score a 0. Only the right answer is a 1.
So it might get the question right by making stuff up/guessing, but will always be punished for admitting a gap in knowledge.