What you will learn This part runs to more than ten chapters. Here is the map first: those chapters are each digging into one of four properties. And you will see why memorising a strengths list and a limits list separately is wasted effort.
People learning AI are usually handed two lists like these.
Memorising these two lists buys you nothing. Meet a task that is on neither list and you cannot judge it; change the model and the lists change too.
What is useful sits underneath. The left and the right are not different facts. They are one property seen from two sides.
The reason it translates well and the reason it cannot do arithmetic are the same reason. Both follow from it being a machine that continues text.
| Property | What it makes possible | What it rules out, for the same reason |
|---|---|---|
| 1. Continuation | Writes naturally in any register | Produces the shape of an answer while knowing nothing |
| 2. Learned knowledge | Knows context you never mentioned | Knows nothing after its cutoff, nothing about your company |
| 3. Working memory | Handles material you paste in, right there | Anything past the edge disappears silently |
| 4. Instruction response | You change behaviour with words | What you meant and what landed drift apart |
In each row, the left and the right are the first and second half of one sentence. Remove the right-hand side and the left goes with it.
HyperTeamsgraph TD
A["One property"] --> B["The capability side"]
A --> C["The limit side"]
B --> D["e.g. writes in any register"]
C --> E["e.g. cannot do arithmetic"]
D --> F["Both come from one thing —<br/>continuing the next words"]
E --> FIt is not a search box. It is an extremely sophisticated autocomplete. Given a question it does not pull an answer out of a store; it lays down, piece by piece, the words likely to follow that question.
Everything inside the model came from its training data. That leaves two kinds of hole: a hole in time (anything after training ended) and a hole in scope (your company's material, which was never public in the first place).
Both holes are filled the same way — by giving it the material.
There is a fixed amount it can look at in one go. That means the more material you put in, the better it answers, and it also means when it overflows, the front falls off.
The falling-off happens without warning. The model does not say "I forgot"; it just fills in.
Tell it "no tables" and it stops using tables. Then a table reappears in the third answer. That is because it is not keeping a rule it understood as meaning; it is producing the continuation of a text that contains a rule.
Incidents in the field are seldom explained by a single property. Two of them overlap.
Falling off alone does not cause an incident. It becomes one because continuation fills the gap. So the response is also two-part — manage the window (restate the constraints) and demand grounding (make it quote the clause text).
| Symptom | Properties colliding | Look at first |
|---|---|---|
| Cites a source that does not exist | 2 + 1 | Did you supply material? |
| Rules break down as the chat gets long | 3 + 4 | Did you restate the constraints? |
| Just keeps agreeing with you | 4 + 1 | Did you ask for the counter-case? |
| States a current figure confidently | 2 + 1 | Did you attach a tool? |
Diagnosis does not start with "why was it wrong?" but with "which properties collided?" Name the property and the response follows.
No. They can be reduced.
Every remaining chapter in this part is about what you put in that slot.
1. Why is memorising a strengths list and a limits list separately wasted effort?
Because they are two sides of the same property. The reason it translates well and the reason it cannot do arithmetic both come from it being a continuation machine. Knowing the properties lets you judge tasks that appear on neither list; knowing only the lists leaves you stuck on anything new.
2. What does "falling off alone does not cause an incident" mean?
Material leaving the window is just information going missing. It becomes an incident because continuation fills the empty slot with something plausible. That is why the response has two halves: manage the window, and demand grounding.
3. Do these four properties disappear as models improve?
No. They come from the structure, so only the frequency drops. And as the frequency drops people relax, which can make the remaining errors harder to catch.
Now the first property in detail → What a language model actually does
Good at Bad at───────────── ─────────────Summarising Recent informationTranslating ArithmeticDrafting Checking factsClassifying Remembering long documentsIncident: asked to review a contract, and about thirty minutes in it confidently cited a clause number that does not exist.Property 3 (working memory): the front of the contract fell out of the windowProperty 1 (continuation): the vacated slot got filled with "the shape of a clause number"✗ "A better model will fix it" → The frequency drops; the property does not. Rarer errors are harder to catch, not easier.✓ "Which property does this task run into? What do I put in that slot?" → Supply material (2), manage the window (3), restate constraints (4), demand grounding (1).