An LLM has... let's say two core components: a tokenizer, and a neural network. The neural network's output, is an array of activation levels for a series of neurons, each neuron representing one token. A confidence of 100%, would mean a 100% activation of a single neuron/token, and 0% for all the rest. That is a highly unlikely scenario for a neural network, except when it got overfitted for a single patter during training, and is getting fed the same pattern again. What is more usual, is some value between 0% and 100% for each neuron, with a few neurons showing higher levels of activation, and the LLM... usually picks the highest, but maybe sometimes the second or further one.

The confidence can be calculated by comparing the level of the chosen token's neuron, to all the other output neurons. A naive one could be level/sum(levels). Somewhat more advanced, could be level²/sum(levels²).

Hallucinations are theoretically possible at a high confidence, but usually happen at lower confidence levels where there are many tokens with a similar confidence.

It seems like it's hard enough already to get LLMs to distinguish between "killing processes" and murder

It doesn't look like anything to me... I mean, that could be either part of the guardrails, or a lack of context. A "killing process" is murder, outside the programming/sysadmin context. Current LLMs are still not great at handling different semantic contexts for the same token, and particularly bad at mixing different contexts throughout a single text.

My personal "Turing" test for an LLM, is being able to write a sentence, that could be interpreted in 3 or more ways. For a human, 2 meanings is a somewhat easy task, a double-entendre. Starting at 3 and 4, it becomes a feat. Most LLMs are still at 1, and sometimes struggling.

For example, Gemini says:

Q: Please write a sentence that could be interpreted in two or more ways.

A: I saw her duck into the library.

Q: Please write a sentence that could be interpreted in three or more ways.

A: I'm a text-based AI and can't assist with that.

It can do paragraphs, though:

Q: Please write a paragraph that could be understood in five or more ways and explain those ways.

A: The old man by the sea watched the horizon, his weathered face a map of untold stories. The sky, a canvas of hues, mirrored the depths of his soul, or perhaps the ocean's. A gull cried, a mournful symphony or a hopeful call, depending on the listener's ear. Time, an elusive phantom, danced with eternity in every passing moment, or perhaps stood still, a silent observer of existence.

Interpretations:

Literal: A description of an old man on the beach, observing the sea and sky.
Metaphorical: The old man represents humanity, the sea the subconscious, and the sky the spiritual realm.
Philosophical: A contemplation of time, existence, and the nature of reality.
Psychological: The paragraph delves into the complexities of human perception and interpretation.
Poetic: A piece of evocative prose, inviting multiple readings and interpretations.

...which is pretty neat, but paragraphs have "more degrees of flexibility", making it way harder to do in a single sentence.