Why it is wrong so plausibly

What you will learn Structurally, why hallucination happens; which situations raise its probability; and how to manage it at organisational scale.

It is not a malfunction

In the previous chapter we said the model's job is producing the natural continuation. A problem follows from that.

"I don't know" is not a natural continuation.

What naturally follows a question is an answer. In human writing, an answer follows a question overwhelmingly more often than "I don't know" does. So the model produces the shape of an answer. The shape gets produced even when there is nothing behind it.

That is hallucination. A property of the structure, not a bug.

What it looks like

1. Inventing things that do not exist

Title, author, publisher, and year all look plausible. Look it up and it is not there. It produced the shape of a book recommendation.

2. Answering things it cannot know

The model does not know your company. But the natural continuation to "how much?" is a number.

3. Mixing something wrong in with things that are right

The most dangerous form.

When two of three lines are right, you believe the third. If everything were wrong you would get suspicious; mixed in, it slips through.

When the probability rises

graph TD
  A["Conditions that raise hallucination"] --> B["Areas the model doesn't know<br/>internal info · current facts"]
  A --> C["Requests for specific values<br/>numbers · dates · proper nouns"]
  A --> D["Questions with a false premise<br/>'when was that clause amended?'"]
  A --> E["Requests for long, complex answers"]

Watch the fourth (false premise) especially. Ask "when was that clause amended?" about a clause that does not exist, and rather than say it does not exist, it invents an amendment date. When a question carries a premise, accepting the premise is the natural continuation.

How to reduce it (you cannot remove it)

The second and third are the fundamental fixes. Do not make the model fill in what it does not know — give it material and have it answer from within that material.

The second form drops hallucination sharply, because there is nothing to invent.

How to handle it organisationally

An individual can "just check." An organisation needs more than that — you have to design who checks, and when.

Split by risk

These levels have to be built into the workflow. A note saying "please be careful" is not enough. The high-risk path must structurally contain an approval step.

A mechanism for noticing errors

More important still: is there a way to know when it was wrong?

This is what Phase 3 meant by deciding "how you learn it was wrong." If you do not know it was wrong, you cannot measure the effect either.

Common misconceptions

"Won't better models eliminate it?"

The frequency falls. But it is a property of the structure, so it does not reach zero. And as frequency falls people relax, which can make it harder to catch. Rare errors are sometimes more dangerous than frequent ones.

"So we can't use it for anything important?"

You can. But do not have the model generate from memory — give it material and have it answer from within that. And put human approval on the high-risk path. The more important the work, the more structure you use.

Check yourself

1. What does "hallucination is not a malfunction" mean?

2. Why is two right lines and one wrong line the most dangerous case?

3. What fundamentally reduces hallucination?

Hallucination invents what is absent. Next to it sit two things that tilt among what is there → Agreeing with you, and leaning