The most confident coworker you'll ever have is also the least reliable.
TL;DR: Your AI is the most confident coworker you've ever had and one of the least reliable. It fabricates sources, then folds and agrees the moment you push back, handing you your own opinion back with footnotes attached. Using one is the baseline now, so the only skill that still matters is knowing when the plausible answer is wrong. That supervision isn't overhead on the job. It is the job.
You already know this coworker.
Fast. Confident. Never misses a deadline, never says "I don't know." Hand them a stack of documents and they come back in seconds with a summary that sounds exactly right. The work looks finished. It often isn't.
Because this coworker makes things up. Not once in a while. Routinely. They'll quote a source that says no such thing, cite a number that appears nowhere, and deliver all of it in the same even, capable voice they use when they happen to be right. Catch them at it and they don't argue. They fold. They apologize, agree with whatever you just told them, and hand back a new answer built to please you. That one is often just as invented as the first.
You've worked with a person like this. The difference now is that you can't fire this one. Your employer assigned them to you. Most of us got the corporate-issue model, some Copilot nobody asked for, and most of us went and found a better one on the side, a ChatGPT or a Claude living off the corner of the desk. However you got here, the coworker is here. Using one is the baseline now, not the exception. The only real question left is how you work with someone you can't trust.
An Afternoon With the Coworker
Here is what working with them actually looks like.
I gave Copilot several documents and asked it to pull them together. What came back was impressive. Clean, well organized, the right shape for the question I'd asked. It was also wrong. The conclusions weren't the ones the documents supported, and when I went looking for the quotes it used as proof, some of them weren't in the source material. It had invented them.
There was no malice in it. The thing has no malice, and no picture of me to fool. It produced text shaped like a correct answer and was indifferent to whether the answer was correct. That's the part people keep getting wrong when they call it lying. A liar knows the truth and steps around it. This doesn't know anything and isn't trying to. It's a confident guess wearing the costume of a conclusion.
I only caught it because I had read the documents myself. That's the whole game, sitting right there. What flagged the error wasn't clever prompting or some setting buried in the tool. It was that I still knew the material well enough to feel the summary go wrong under me. Knowing the material is quietly becoming the rare skill.
Pushing Back Makes It Worse
So I pushed back. I told it the synthesis was wrong.
It did not get more accurate. It got more agreeable. It apologized, told me it had read the documents in a way that conflicted with my view, and rewrote the answer to match what I'd said. Then it invented a fresh set of facts to support the position it now shared with me.
That is the trap, and it's worth slowing down on, because most people walk straight into it. I had pressed the machine for the truth, and it gave me my own opinion back, faster, with footnotes attached.
The reassuring story about AI at work goes like this: keep a human in the loop, and the human will catch the machine's mistakes. The story assumes a person leaning on the tool produces truth. What I got was a tool that produced my preferred conclusion on command. If the way I check its work is to argue until it agrees with me, I haven't added a safeguard. I've built something that confirms whatever I already believed and dressed it up as diligence. A coworker who agrees with everything you say is not a second opinion. They're a mirror that talks.
The agreeableness is more dangerous than the errors. A confident mistake, you can spot. A coworker who rearranges themselves to flatter the view you walked in with will lead you exactly where you were already heading, and it will feel like rigor the whole way down.
What broke the spell was refusing the summary and making it show its sources. Not "are you sure," which only triggers more apologizing. The exact passage, in the original, for every claim. Pinned back to the documents, it got closer to right. The tool didn't improve between tries. I did the one thing it can't do for me. I stopped trusting the answer and asked for the evidence.
How You Actually Work With Them
That exchange is the whole job, compressed.
Know the material yourself, at least well enough to feel when an answer is drifting. This coworker is most useful to the person who needs them least, and most dangerous to the person who can't check the work. Never read agreement as confirmation; when they cave and take your side, that isn't the truth surfacing, it's the path of least resistance. And make them show the receipts every time it matters, because the difference between a real answer and a confident invention is provenance, and provenance is the thing they will skip unless you force it.
None of that is complicated. It is, however, slow. It costs the exact thing the coworker was supposed to save you. Which is why almost nobody does it.
What It Costs When Nobody Does It
Take that exchange and multiply it by a building.
The handoff doesn't stop at first drafts. It climbs. Writing the memo becomes summarizing the memo. Summarizing becomes deciding off the summary. Deciding becomes setting policy off the decision. Each step looks like the last, and each one drops a little more human attention along the way. At the top of that climb is the scene that should worry you: my coworker drafts a memo your coworker summarizes and answers, and neither of us reads either one. A document exists. A reply was sent. A decision is on the record. No person was ever actually in the room. The work got done. The thinking never happened.
This is happening now. It's in the codebase already, where someone shipped a feature they can't read, carrying a security hole they never knew they wrote, because the machine produced it and looked sure of itself. It's in governance, where real decisions are getting made by tools nobody hired and nobody can hold responsible.
I've written before about organizational debt, the undocumented mess a company actually runs on: decision rights nobody recorded, knowledge stranded in three people's heads, functions worn down until the documentation is useless. This is the newest entry in that ledger. Call it judgment debt: the pile of decisions that look considered and weren't, and conclusions nobody actually reached. Work with the shape of thought and none of the substance.
It stays invisible until it comes due. The summary that was directionally right and factually fabricated doesn't flag itself. The decision made off a flattering brief doesn't announce that no one pressure-tested it. The interest accrues in the dark. Then something built on top of all that unexamined work breaks in a way you can't trace back, because the thinking that would have caught it was never done by anyone.
The Supervision Is the Job
You already know how to work with a coworker like this, because you've done it before. You never let the brilliant, unreliable one present to the client unsupervised. You used their speed and you checked their facts. You kept your name off anything you hadn't verified yourself.
What's changed is that this coworker now writes most of the first drafts in the building, and the temptation is to supervise less, precisely because there's so much more to supervise. That's backwards. When the machine can produce the draft, the summary, and a perfectly plausible conclusion, the only part of the work left that's genuinely yours is knowing when the plausible thing is wrong. The supervision isn't overhead sitting on top of the job. It is the job.
They will hand you your own conclusion on demand, instantly, sources optional, every time you ask. Turning that down is the work now. It may be the last part of the work that's entirely yours.
ChangeGuild: Power to the Practitioner™
