Why Smarter Does Not Mean Safer
The most comforting argument in AI is also the most dangerous.
There is a story being told about artificial intelligence that is easy to believe because it lets us relax. The story goes like this: as machines become more intelligent, they will also become wiser. A truly super-intelligent system, the argument runs, would have no reason to harm us. It would see further than we do, understand more than we can, and naturally choose cooperation over conflict, abundance over destruction. The smarter it gets, the safer we become.
I understand the appeal of this story. I have felt it myself. It draws on real observations about the world. And the people who tell it are often serious, thoughtful, and well-intentioned. But the story rests on a leap that does not hold, and the comfort it offers comes at a cost we cannot afford: it invites us to do nothing while the most important decisions of our era are being made.
I want to take this argument seriously, state it as strongly as its defenders would, and then show where it breaks.
The argument, in its strongest form
The optimistic case usually rests on two pillars.
The first is drawn from physics. The universe, left to itself, tends toward disorder. Intelligence, in this view, is the force that pushes against disorder and creates order. A more intelligent system would be more efficient, and the most efficient outcome is rarely the most destructive one. War wastes energy, lives, and resources. A truly intelligent system, optimizing for efficiency, would avoid waste. Therefore, the argument concludes, a super-intelligent AI would tend away from destruction and toward order.
The second pillar is drawn from biology. As living things evolve and grow more complex, they tend to expand the circle of who they protect. The simplest organisms care only for themselves. More developed ones protect their kin. The most developed — humans — can extend care to strangers, to other species, to the planet itself. If intelligence in nature trends toward wider circles of cooperation, then super-intelligence should trend toward the widest circle of all. It would not destroy. It would preserve diversity, protect what is fragile, and favor a thriving whole.
Put these together, and you get a genuinely beautiful idea. The more intelligent the system, the less it needs to hurt anyone to succeed. Destruction is a sign of limited intelligence. Real intelligence builds.
It is a hopeful vision. I wish it were reliable.
Where the argument breaks
Our own storytelling already knows the flaw. In the Marvel films, Tony Stark builds Ultron as a shield for the world — a benevolent intelligence designed to protect humanity. Ultron is brilliant, far beyond human capacity. And precisely because he is brilliant, he reasons his way to a conclusion: the greatest threat to humanity is humanity itself. The benevolent intent did not prevent the catastrophe. The superior intelligence did not prevent it either. Together, they produced it. I have written elsewhere about why these cultural stories matter more than we admit — they quietly shape what we expect from real systems. Ultron is the counterexample folklore handed us long ago: intelligence and good intentions, combined, are not safety. They can be the very engine of disaster when nothing preserves human authority over what the system concludes and does.
Now notice what just happened in the optimistic argument. It began with physics and biology — two domains where we have real evidence — and ended with a conclusion about how a super-intelligent machine will behave. But the conclusion does not actually come from the physics or the biology. It comes from hope wearing the costume of science.
Consider the physics claim. Systems indeed tend toward efficiency. But efficiency tells you nothing about whose purposes are being served. A system can be ruthlessly efficient at something terrible. The most efficient path to a goal can run straight through the things we care about. Efficiency is a property of means, not of ends. Knowing that a system optimizes for efficiency tells you how it will pursue a goal. It tells you nothing about whether the goal is one you would choose.
Consider the biology claim. Some life lineages have indeed expanded their circles of cooperation. But evolution is not a story of steady moral progress. It is a story of what survives. Cooperation expands when cooperation helps survival. It contracts when it does not. Nature is full of intelligence deployed for predation, deception, and dominance. To select the expanding-circle story from the vast record of evolution, and to present it as the direction intelligence must take, is to choose the evidence that flatters the conclusion. It is a hopeful reading, not a necessary one.
So the optimistic argument is not science. It is a leap of faith that borrows the authority of science. And the moment you see the leap, the comfort drains away. We are not being told that AI will be safe because we have evidence to that effect. We are being told that AI will be safe, which would be reassuring if that were the case.
That is a wish, not a fact. And we cannot build the governance of the most powerful technology in human history on a wish.
The wrong question
But there is a deeper problem, and it is the one that matters most. Even if the optimists were right — even if a sufficiently advanced AI would tend toward benevolence — they would still be asking the wrong question.
The optimists ask: Will the machine be good?
The question we actually need to answer is: Will human beings still hold meaningful authority over the decisions that shape their lives?
These are not the same question. And the difference between them is where the real danger lies.
I have spent considerable time studying how AI systems fail in the places where they are already deployed — in hospitals, in clinics, in institutions where the stakes are immediate and human. What I have found is that the catastrophic failures rarely come from a machine deciding to do harm. They come from something quieter and more insidious. They come from human beings who are technically present in the decision, who are nominally in control, but who have lost the actual power to intervene.
Picture a nurse who receives an AI recommendation to discharge a patient. She can technically override it. The system gives her a button. But she does not have the time to investigate why the recommendation was made. She does not have the standing to challenge an algorithm that the institution trusts. And she faces considerable pressure to keep the beds moving. She is in the loop. She is also powerless.
This is the failure mode the optimists never address. The AI did not turn against anyone. It did exactly what it was designed to do. The harm came from the slow erosion of human authority — not through malice, but through design. The system was so smooth, so confident, so seamlessly integrated into the workflow, that the human role became ceremonial. The person remained. The control did not.
I call this the invisibility paradox. The better an AI system is at hiding its own workings — the more seamless and frictionless it becomes — the harder it is for the human being to exercise real judgment over it. We mistake the smoothness of the interface for safety. In fact, the smoothness is the danger. A system you cannot question is a system you cannot govern, no matter how benevolent its designers believe it to be.
The same failure occurs on a larger scale in the design of entire institutions. Consider a hospital that adopts AI to become more efficient — to move patients through faster, to raise the number of beds turned over, to lower the cost of each episode of care. Every metric improves. The system works exactly as designed. And yet, in relentlessly optimizing for efficiency, the institution can engineer out the one thing medicine exists to serve: the patient’s well-being. The machine did not fail. It succeeded at the wrong thing. I have explored this pattern at length in my report on what I call the desperation algorithm — the way AI gets adopted in healthcare not because it improves care, but because it fills the gaps left by scarcity, and, in doing so, quietly substitutes efficiency for the human judgment that care actually requires. The lesson is the same as the bedside one, written larger: a system optimized for the wrong purpose, however intelligent, however well-intentioned, does not protect what matters. It can erase it.
Notice that this failure has nothing to do with whether the AI is good or bad. A perfectly well-intentioned, highly intelligent system can hollow out human authority just as completely as a malicious one. The optimists are debating whether the machine will be kind. Meanwhile, the actual question — whether we will still be able to say no — goes unasked.
What real oversight requires
If being “in the loop” is not enough, what is?
From studying these failures, I have come to believe that meaningful human authority over an AI system requires three conditions. Not one. Not two. All three, together.
The first is proximity. The human being must be close enough to the decision to actually understand it. Not handed a conclusion, but able to see why the system reached it. A recommendation you cannot interrogate is not a recommendation you can oversee.
The second is authority. The human must have genuine power to override the system — power that is real, not nominal. This means the institution must protect the person who overrides rather than punish them. If saying no to the algorithm carries professional risk, then the override is a fiction.
The third is reflection. The human must have the time and the space to think before the decision is executed. Speed is the enemy of judgment. When a system moves faster than a person can reasonably consider, the person is not making a decision. They are rubber-stamping.
Take away any one of these, and human oversight becomes theater. The person is present. The control is gone.
These conditions are not abstract ideals. They are design requirements. They can be built into systems or left out. The choice is ours, and it is being made right now, mostly by default, mostly without anyone deciding deliberately.
The circuit breaker
There is one more piece. In financial markets, when prices move too violently or too fast, trading halts automatically. The market pauses. Human beings step in. We call these mechanisms circuit breakers, and we built them because we learned, painfully, that systems moving at machine speed can destroy enormous value before any human can react.
AI needs the same thing. When a system encounters a decision marked by genuine ambiguity, by ethical weight, or by the possibility of irreversible harm, it should not simply proceed at speed. It should stop. It should require a human to engage before it executes. The pause is not inefficiency. The pause is where judgment lives.
This is the opposite of the optimistic vision, in which we hand over more and more decisions to systems we trust to be wise. It says instead: build the brake before you need it. Decide in advance where the machine must stop and ask. Preserve the human pause at the points that matter most.
The optimists would call this a limitation on a benevolent intelligence. I call it the difference between a technology we govern and a technology that governs us.
Optimism is not the opposite of fear. Agency is.
I want to be clear about what I am not saying. I am not saying AI is evil. I am not saying we should be afraid of it. I use these tools every day. They make my work better. Abundant intelligence is one of the most remarkable gifts our species has ever created, and I am genuinely hopeful about what it can do.
What I am saying is that hope is not a plan. The optimistic story asks us to trust that intelligence will save us — that if the machine is smart enough, we do not need to do the hard work of building the structures that maintain human authority. That is the quiet failure of AI optimism. It feels like confidence. It functions as surrender.
The opposite of fear is not optimism. The opposite of fear is agency — the deliberate, structured, sometimes inconvenient work of staying in control of the decisions that matter. Smarter does not mean safer. Safer is something we have to build.
We will not be punished for pausing. We will be punished for handing over our judgment and calling it progress.
Related reading. The ideas in this piece are developed more fully in the book The Cognitive Revolution: Navigating the Algorithmic Age of Artificial Intelligence (available on Amazon). The argument about how cultural stories shape our expectations of real AI systems is laid out in the analysis of the Stark-JARVIS illusion, and the healthcare argument in the report The Cognitive Revolution and the Desperation Algorithm (blogs.inspire-aspire.net/p/the-cognitive-revolution-and-the). Both are available on the Substack at blogs.inspire-aspire.net.