What you will learn Three limits of AI, and how to decide what you must verify.
This chapter is not here to scare you. Knowing how far you can trust it actually makes it easier to use. It is like knowing where your car's blind spots are — that is what lets you drive relaxed.
The most important limit.
When AI meets something it does not know, it sometimes does not say "I don't know" and instead produces something plausible. This is usually called hallucination.
The problem is that a wrong answer arrives with exactly as much confidence as a right one. You cannot tell them apart by tone.
AI cannot possibly know your company's revenue. And yet a number appeared. In the course of producing something shaped like an answer, it filled in a plausible figure.
Or this:
Those books may not exist. Title, author, and year all look plausible; look them up and they are not there.
In what AI is we described AI as "a program that got very good at guessing the next word." A plausible answer is a more natural continuation than "I don't know," so sometimes it goes that way.
| For this | Do this |
|---|---|
| Numbers, dates, statistics | always verify |
| Names, book and paper titles, legal citations | always verify |
| Internal company information | it cannot know. You have to supply it |
| Polishing, summarising, translating | generally safe (it works on what you gave it) |
| Suggesting ideas | safe (you are choosing anyway) |
Rule of thumb: work that processes material you provided is safe; work where AI has to fill in information it does not have is risky.
One thing that helps Adding "if you're not sure, say you don't know" reduces how often it invents things. It does not eliminate it.
AI is built from material up to a certain point in time. It does not know what happened afterwards.
That said, AI with tools attached is different. Connect it so it can search or look through internal material directly and it can answer with current information. That is tools, covered later. At this stage, assume "by default it does not know."
You need to know where what you type goes. It differs by service.
Be especially careful with:
1. Check your company's policy first. There may already be rules.
2. Strip what is not needed. If you want a contract reviewed, the company name and the contact's name can go. The clause text alone is enough to review.
3. If it is for work, use a work environment. A personal account and an environment your company has contracted for handle data differently. Some cases need an air-gapped setup.
4. Check whether your conversations are used for training, and turn it off if you can. Depending on the service, what you type may be used to train the model further. Most have a setting for it somewhere.
The default differs by service and by plan. Personal plans tend to have it on; an environment your company has contracted for tends to have it off. So "our company has a contract, so we are fine" and "my personal account must be the same" are two different statements.
Once, right after you sign up. Turning it off usually does not reach back to conversations you already sent, so the order matters — turn it off, then paste in company material.
It comes down to one question.
What happens if this is wrong?
HyperTeamsgraph TD
A["You got an answer"] --> B{"What if it's wrong?"}
B -->|"just redo it"| C["Use it as-is<br/>polishing · ideas · summaries"]
B -->|"money, trust, or law"| D["Always verify<br/>numbers · external docs · regulations"]This same test is used at the organisational level later — it is exactly how you decide where to put human approval.
Separate producing a draft from final judgement. AI takes the first, a person takes the second. Editing a draft is far faster than starting from a blank page, and that time saved is larger than the cost of checking.
Then that task is not one to use it for. You do not have to use it for everything. If verification costs more than writing, write it yourself.
This judgement matters because the same mistake repeats at organisational scale — the third item in how transformations fail is exactly this case.
1. AI answered "our revenue last year was $4.7 million." What is wrong?
AI cannot know your company's revenue. Instead of saying so it produced a plausible number. That is hallucination, and because a wrong answer carries the same confidence as a right one, tone gives you no signal.
2. What separates safe use from risky use?
Processing material you supplied is safe (summarising, translating, polishing); filling in information AI does not have is risky (numbers, sources, factual claims).
3. What single question tells you whether verification is needed?
"What happens if this is wrong?" If you can just redo it, use it. If money, trust, or law is on the line, always verify.
You have finished “New to AI”. You should now be in this state:
That is what you need working alone. Now the map of where these four chapters sit in the whole, and what else is needed once work goes to other people → Delegate · Describe · Discern · Take responsibility
You: What was our company's revenue in 2023?AI: Revenue in 2023 was approximately $4.7 million.You: Give me 3 books worth reading on this topicAI: 1. "Data-Driven Organisations" (Kim, 2021) 2. ...You: What's today's exchange rate?AI: (doesn't know, or gives an old value, or invents one)(as-is) "Hanbit Industries Co., represented by Kim Young-su..."(stripped) "Party A..."In settings, look for a name like: "Help improve the model" "Data controls" "Use for training"