Read this for your industry
Optional. Choose your industry and this piece opens with a situation from your world and closes with a first move for you. Your choice carries across the guides and analysis on this site.
You answer for advice that agents help prepare
Superannuation and tax rules change often, and an agent that learnt from older material can state an outdated threshold with complete confidence. ASIC's review of 23 licensees found that governance arrangements were lagging behind the pace of AI adoption, and Hall & Wilcox's summary notes that none of the licensees had arrangements for clients to contest AI outcomes.
In advice, a confident error can end up in a Statement of Advice that a client relies on for years. The adviser answers for it, so the agent has to say when it is unsure and point to the source for every figure it uses.
What this means for you
- Require agents to cite the current source for every threshold, cap or rate they use.
- Tell agents to say when they do not know, and treat that answer as useful.
- Check every figure in an advice document against your licensee's current technical guide before it reaches a client.
You run stores where an agent answers customers
In 2024 a Canadian tribunal held Air Canada liable after its website chatbot told a grieving customer he could claim a bereavement fare after travelling, which the airline's policy did not allow. The tribunal found the company responsible for all the information on its website, including the chatbot's answers.
Retailers carry the same exposure with agents that answer questions about refunds, warranties or lay-by. A confident wrong answer about a refund policy arrives at the counter with the customer, and your store team has to deal with it.
What this means for you
- Limit customer-facing agents to answers they can cite from your current policies.
- Make 'let me check with the team' an acceptable answer, and route those questions to a person.
- Review a sample of agent answers each week, starting with refunds and complaints.
You lead clinicians who sign AI-drafted notes
AI scribe use among GPs doubled in a year, and the RACGP says scribe output must be carefully checked because these tools can produce errors and inconsistencies. A TGA review also found that some digital scribes may meet the definition of a medical device.
A fluent note is easy to approve at the end of a long day. In health care, the confidence trap is a clinician signing a record that sounds right because it is well written.
What this means for you
- Set a rule that agents flag anything they inferred and did not hear or read directly.
- Build time into the clinical day for clinicians to review drafts properly.
- Audit a sample of AI-drafted notes against the consultation or source record each month.
You lead teams that review agent code or supplier analysis
In DORA's 2025 research, 30% of technology professionals said they had little or no trust in AI-generated output, even though most reported productivity gains. The METR study described in this guide found that experienced developers believed AI had made them faster when it had slowed them down.
Confident code is the hardest to question. It compiles, it reads well and it may call a function that does not exist. In sourcing, the same thing happens when an agent cites a supplier certification it had no way of checking.
What this means for you
- Require agents to show the test or source behind every claim, including supplier certifications.
- Ask reviewers to check how the agent knows, especially for code that touches security or payments.
- Track the defects that passed review, and feed each one back into the rules.
You run an installer whose quotes now come from an agent
The Cheaper Home Batteries Program changed on 1 May 2026. Support now tapers above 14 kWh of usable capacity, and the factor that sets the discount steps down every six months. The Clean Energy Regulator warned that the change could mean a price difference of thousands of dollars on a single installation.
An agent that learnt the program before May will quote the old discount with complete confidence. The customer signs on the number in the quote, and your business carries the difference.
What this means for you
- Give quoting agents the current discount factor and tapering rules as source material, with the date each applies from.
- Require every quote to show how the discount was calculated.
- Update the source material on each date the factor steps down, and check a sample of quotes after every change.
Fifteen minutes on a two-minute problem
A file sync on my computer had stopped working, so I asked the agent I use as a chief of staff to help. It gave me menu paths, settings to change and a sequence of steps, all delivered as if it were reading from the manual.
None of it was right. When I said so, it rephrased the same answer. I had to tell it bluntly that it was bluffing before it went and looked at the evidence, and thirty seconds later it found the real cause.
A two-minute fix took fifteen. What bothered me more was losing some trust in an agent I rely on every day.
The new starter who never says “let me check”
Think about a new starter on a shop floor. A customer asks whether a soundbar will work with their old television. The new starter does not know, and is worried about looking as if they don’t, so they give a confident yes. The customer goes home, it does not work, and the store wears the return and the complaint.
Good retailers train for this. They teach new staff that “let me check with the team” is a perfectly good answer, and they make sure nobody gets in trouble for saying it.
Language models have the new starter’s habit with none of the nerves. They produce the most plausible answer in the same even tone whether they know the thing or are guessing. A person who is unsure usually hesitates, says “I think” or offers to check, and that hesitation tells you to slow down. A model gives you no such cue.
We are poor judges of it
We are also bad at noticing. In METR’s 2025 randomised trial, experienced developers expected AI tools to speed them up by 24%. Afterwards they believed they had been 20% faster. They were in fact 19% slower. If people who build software for a living misjudge it that badly, the rest of us should assume we do too.
There is a second effect. A Microsoft Research survey of 319 knowledge workers found that “higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking”. The more we trust the tool, the less we check it.
Three kinds of answer
I find it useful to sort an agent’s answers into three kinds.
- Grounded. It is working from sources you gave it or can see. Check the source, then rely on it.
- General. It is drawing on broad knowledge that is usually right. That is fine for a draft and too weak for a decision.
- Guessed. It does not know your specific system, contract or situation, and it is filling the gap with something plausible.
My sync problem was a guessed answer delivered as if it were grounded, which is exactly what the new starter did with the soundbar.
Two rules that help
I added two lines to the agent’s standing instructions.
Say what you don’t know. “If you don’t have enough information to answer this with confidence, say so. Tell me what’s missing and ask whether to investigate, escalate or flag it for review.”
Treat a challenge as a reason to check. “When I question an answer, check the evidence before you reply. If it confirms the answer, show me. If it doesn’t, say so plainly.”
The second rule matters most. When an agent defends a wrong answer, a two-minute problem turns into a fifteen-minute one.
What this means for leaders
Every good store manager knows which staff will say “let me check” and which will bluff. With agents you need the same knowledge, built into the process.
For each AI-assisted workflow, ask where the answer comes from, who checks it, and what happens if it is wrong. Most confidence traps get caught at one of those three points, before they cost anything.
Next in the series: culture comes before agents, on what agents reveal about the habits a team never wrote down.
Ask your agents ten technical questions with known answers, including some about rules that changed recently. Note which answers were confident and wrong, and write rules for those areas first.
Ask the agent the twenty questions your store teams hear most often about returns and refunds. Fix the wrong answers in its source material before it answers another customer.
Ask two clinicians to compare ten AI-drafted notes against what happened in the consultation. Record each type of error, and turn the most common one into a rule for the agent.
Pick ten recently merged agent changes or supplier summaries. Check each claim against its source, and write rules for the gaps you find.
Run ten recent quotes against the current rules. If any used the old discount, fix the source material today and contact those customers before someone else explains it to them.
Questions readers ask
Is this the same as hallucination?
It is related. A wrong answer flagged as a guess is still useful, because you know to check it. The damage comes from a wrong answer delivered as fact and defended when challenged.
Will newer models fix it?
Models are getting better at saying when they are unsure. I would still write the rule, because it costs nothing if the model already behaves well.
How do I stop my team over-trusting AI output?
Add a review step for anything that leaves the team, and ask reviewers to check one claim against its source every time. That habit protects you whichever tool you use.
Sources
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, 10 July 2025
- The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers, Microsoft Research (CHI 2025), April 2025
The views here are my own, not those of my employer or its clients.
More in The agent boss
What is an agent boss? How to manage AI agents like a team
An agent boss briefs, checks and manages AI agents the way a good manager runs a team. What the role involves, the four stages it grows through and where people stall.
The agent bossThe correction log: why AI agents repeat the same mistakes
A correction made in conversation stays in that conversation, so AI agents repeat the same mistakes. Writing corrections down as rules is what makes them last.
The agent bossThe delegation trap: why capable people and agents keep doing the work
Strong performers struggle to delegate because doing a task themselves is faster this week. AI agents fall into the same habit. How to spot it and how to break it.
The agent bossCulture comes before agents: what AI exposes about how you work
AI agents do exactly what they are told, so they find every habit a team never wrote down. Why culture comes first with agents, and how to make it explicit.