The Trust Ceiling: The invisible limit on what AI is allowed to do

William Tunstall-Pedoe Jul 20, 2026 2 1 2 Share Over the past decade, artificial intelligence has advanced into territory long assumed to be permanently human. Machines learned to speak, to hear, to see, to answer our questions; then, most strikingly, they mastered language itself. Systems like ChatGPT, Claude and Gemini now converse fluently, draw on more of humanity’s written knowledge than any person could absorb in a thousand lifetimes, and increasingly reason, using language as a substrate for thought.

For the first time we can plausibly imagine the arrival of what some call Artificial General Intelligence (AGI), defined in OpenAI’s charter as “ highly autonomous systems that outperform humans at most economically valuable work ”. In practice: systems that automate large parts of every business, and can act as ever-present assistants in our daily lives for almost everything we want to do.

But despite this extraordinary progress, something fundamental is broken.

The systems we are building today are powerful, but they are, this essay argues, intrinsically untrustworthy.

A sample of recent mainstream coverage of AI failing in deployment. The four headlines are four different symptoms of the same underlying cause, which this essay describes. Hallucinations, lack of accuracy, inconsistency and the inability of such systems to reliably explain their results are holding back AI adoption behind a ‘ trust ceiling ’ where trillions of dollars of value are stuck. The AGI prize might be imaginable but who is going to allow AI to fully automate their business or to take control of their personal life when at every step the AI system might do huge damage to their brand or reputation, cost them money or tell them something that is outrageously untrue? What regulator is going to allow AI systems to become widespread in an industry when every decision they take is a black box and subject to unpredictable error? The “ ...most economically valuable work ” from the AGI definition will remain in human hands until this issue is addressed.

If we can solve this trust problem while preserving and extending the enormous capabilities we have developed, humanity has a path to reaping these extraordinary benefits.

This essay explains the technology issues behind this challenge and its companion essay, published alongside it (and linked at the end), describes the path that will get us there. It is not the path the industry is on - and without change, the industry’s path will fail.

To understand why, it helps to step back and look at computer software itself. Broadly speaking, there are two fundamental paradigms.

The first is what we might call symbolic software. This is software from the traditional world of programming: algorithms, data structures, and databases. This is the majority of software even in use today. In this world, fully correct is the standard. There is a definition of what the software should do, and the software either meets it or it doesn’t. If a spreadsheet adds two cells together we expect it to produce the right answer every single time. If it does not, we call it a ‘bug’: the publisher finds it, fixes it, and knows it will stay fixed, because the same inputs always produce the same behaviour. When a software system that helps with the company payroll retrieves data from a database we expect it to never get a payment wrong. These systems are typically deterministic, consistent, explainable, and fully controllable, and when they fail, the failure is an exception to be eliminated rather than a permanent property of the system.

The second paradigm is statistical machine learning, and in particular the one machine learning method that has come to dominate: large neural networks (often referred to as ‘deep learning’). These systems are not programmed in the traditional sense. Instead, they learn from data. They are large mathematical formulae and begin with randomly initialised parameters and the parameters are then improved systematically through a training process, exposing them to data, until the formulae stop improving and they generate useful results.

The results however are never guaranteed. These formulae are approximations with performance always below 100%. Best guesses. However, they do very important things that symbolic software cannot.

What makes modern AI so powerful is that these deep learning systems improve as we scale them - a trend that has survived every prediction of its end. More data for training, more compute, larger models - and performance continues to increase.

When neural networks first saw commercial use in the late 1980s, they typically had tens of thousands of parameters and trained on megabytes of data. Today’s frontier models have hundreds of billions to trillions of parameters and train on a meaningful fraction of everything humanity has ever written, on hardware performing computations at rates that were science fiction then.

These forces of astronomical increase in data and compute, plus innovation, have driven a similarly astronomical expansion of AI capability.

This is the engine behind the current wave of progress.

But with this power comes a fundamental problem: neural systems are intrinsically untrustworthy.

Untrustworthy here means something specific. It is not the mere willingness of people to rely on a system: people can come to rely on systems they should not, out of convenience or lack of alternatives, a pattern the research literature on trust in AI warns about directly ( Everett et al., Nature Reviews Psychology, 2026 ). Trustworthy in this essay means the properties that make reliance warranted, and for serious work those properties go beyond just being right: knowing where an answer came from, getting the same answer to the same question every time, being able to reconstruct and contest a decision, and knowing which rules it was made under. Measured against that standard, neural systems fail in four significant ways.

The first issue is accuracy. In AI research, progress is measured in percentage scores on established benchmarks. A better model improves a score, say, from 87.3% accuracy to 89.2%, and this is celebrated as a breakthrough. But (glass half empty), this also means the system is wrong more than one time in ten. The field’s own habits concede this point: benchmarks are declared ‘saturated’ when scores reach the mid-90s, and the community moves on to a harder one. Saturation never means 100%. A residual error rate is not a temporary embarrassment on the way to perfection; it is where every benchmark ends. In some applications of AI a meaningful error rate is acceptable; in many, it is not. When errors affect one’s brand or reputation, financial decisions, legal outcomes, or safety-critical systems, even very small error rates can prevent a system being used. However, all neural systems have an error rate and at scale even a very small error rate equates to a torrent of bad results. When neural systems are wrong, there is no ‘bug’ to fix: error is a natural consequence of what they are.

Deep learning training process. Accuracy is measured as the model is trained, training ceases when further improvements stop and accuracy is always less than 100% Secondly, neural systems can generate answers that are not just wrong, but the worst kind of wrong: convincing and wrong. Large language models (“LLMs”) - the form of neural networks that unlocked language and debuted with ChatGPT - frequently produce answers that are fluent, detailed, completely believable but outrageously false . This phenomenon, usually called hallucination, creates a uniquely dangerous failure mode. A system that sounds authoritative but fabricates facts is dramatically worse than one that simply fails. This failure mode cannot be engineered away because it follows from how the machine works. An LLM is built to produce the most plausible continuation of a text, and when it does not know the answer, it produces the most plausible continuation anyway: fluent, confident, and fabricated. Hallucination is not the system going wrong; it is the system doing exactly what it was built to do, at a moment when that is exactly the wrong thing to do.

Research continues to reduce how often this happens but the leading labs acknowledge the cause lies in how these systems fundamentally work. OpenAI’s own researchers have shown that some level of factual error is a statistical inevitability of next-word prediction, and that hallucination persists because training and evaluation reward a confident guess over an admission of uncertainty. Their proposed remedy is telling: train models to say “I don’t know” more often. Calibrated abstention is real progress, but notice what it is - a statistical guess about whether the system is guessing. It reduces the bluffing; it makes nothing checkable. The rate falls; the kind of machine it is does not change. This is borne out in practice: several years of recognition of hallucination as a major industry problem and billions spent on R&D have reduced it, not removed it. State-of-the-art models from every provider continue to hallucinate, and trust does not average out: so long as any answer might plausibly be fabricated, every answer must be checked. For high-stakes work, trust is governed by tail risk, not by average benchmark performance.

Third, neural systems are not explainable in any meaningful sense. Modern neural networks consist of billions or even trillions of parameters, initialised at random and then shaped by training on vast datasets with astronomical amounts of compute deriving their final values. We can observe their behaviour, but we cannot explain why they arrive at a particular answer. A formula with a hundred billion weights derived this way is arguably one of the least explainable human-produced things we can imagine: the truest ‘black box’ it is possible to be. Even the researchers who have dedicated their careers to these systems do not fully understand their internal workings. An entire scientific field, mechanistic interpretability, now works to open the box, and its findings are genuinely illuminating. But a science of the black box is not the same as an explanation you can rely on: partial accounts assembled from outside, after the fact, are a different thing from a full, faithful and human-understandable record of the reasoning behind every answer. The most this field can promise is a better understanding of the box. Trust requires that there be no box.

Additionally, the reasoning traces produced by the latest LLMs are fundamentally different from what a good explanation should be: an argument we made recently in a peer-reviewed paper presented at ICML’s 2026 Trust Workshop . A real explanation points to the facts that made the difference to the outcome, but neural networks do not represent data as anything like independent facts, so there is nothing to point to. The paper’s conclusion is one this essay reaches by another route: where complete explainability is essential, some part of the system must be something other than a neural network.

Finally, neural systems are not even reliably consistent. The same system can produce different, contradictory answers to the same question. This cannot be dismissed as a settings problem: even with randomness switched off (’temperature zero’), outputs shift with trivial rephrasings and reorderings of the input that change nothing about the meaning. In isolation, this may seem like a minor issue. In practice, it erodes trust rapidly. A system that cannot make the same decision twice under identical conditions cannot be relied upon.

Taken together, these issues amount to a single, central problem.

It may be tempting to reply that humans have an error rate too, and that we handle this with checks and balances: review, audit, accountability, escalation. Surely the same machinery can wrap an imperfect AI? But look at what that machinery runs on. When a tr