The Future of Intelligence: second, policy is a broken promise
Part II of three. Part I argued that as machines make analysis cheap, the value migrates upstream into standards and downstream into accountable judgement. This part is about what is needed to make an organisation's use of AI safe.
When I was younger, I drove a Mazda 3 Turbo Diesel, and for about 8 years I managed to remember every time I filled it up that it needed diesel - not petrol - in its tank. It helpfully had a little label on the fuel cap door that in loud letters said DIESEL ONLY. Fast forward a few years, I was driving a Subaru - a petrol car - and in autopilot mode I filled it up with Diesel. As I was making that expensive mistake, I thought to myself - jeez, I wonder why this petrol pump doesn't fit into the car?! Oh well, I'll just hold it in place as I fill up...
Late last year, I wrote about how to safely leverage LLMs for high stakes work: my core tenet was that "the harness must be stronger than the force it seeks to contain." My argument there was about the non-deterministic nature of these models, but I think that assertion applies more broadly.
As AI models get increasingly capable, whether their behaviour aligns with what is considered appropriate or safe becomes increasingly important: a model that is capable of breaking out of its environment and attacking real systems means that misalignment has real-world consequences. Recent reporting has demonstrated that there are some pretty serious alignment issues that haven't been adequately controlled for in the state-of-the-art models.
For organisations that cannot afford to be wrong, this presents a conundrum - there is a very strong drive to do more with less - and these new technologies offer great promise - but there exist rules or policies that must not be disregarded and without which the technology cannot be used, no matter its promise.
Such rules amount to a promise about behaviour: what will or won't be done, and they exist to mitigate the risk of otherwise unchecked behaviour.
Two kinds of safety
There are two ways to make a system safe.
The first is by adding a control: a rule, a permission, an approval step, a register, a procedure (and then rely on it being effective and being followed). This is policy.
The second is by designing and building the system so that the unsafe state cannot exist at all. This is structure.
The first can be retrofitted, while the second often can't. Both belong in any serious system, but they have different failure domains, and that difference is the point of this essay.
Having a human "in the loop" in an AI system is a policy control and, I would argue, not the kind of control that can apply at scale.
There is a simple test for any system you manage or framework you evaluate: ask what stops it from going wrong. Does the answer describe a state the system cannot enter, or does it merely name someone who is supposed to do something?
"The record cannot be overwritten" versus "the procedure is that users always create a new entry, and makes sure they don't reuse the old one."
One is an invariant of the system - a nozzle that simply does not fit. The other is a "DIESEL ONLY" sticker that describes hope.
An old engineering idea
Structure over policy must be one of the oldest ideas in safety engineering.
In 1856 John Saxby patented mechanical interlocking for railway signalling, and in 1889 - following a railway disaster caused by manual signalling breakdowns - the UK Parliament passed legislation that legally mandated mechanical interlocking for signalling across the entire passenger network.
Aviation followed, with a "weight-on-wheels" switch to stop pilots retracting the landing gear while the plane sat on the runway by making it physically unreachable without a further override.
Modern software systems carry the same lineage. Type systems exist so that whole categories of error are unrepresentable before the program ever runs - make illegal states unrepresentable, in Yaron Minsky's phrase. Systems that leverage immutable data create a shared state that cannot be corrupted, rather than state everyone promises to lock correctly. An append-only record cannot lose its history; there is no discipline to maintain because there is nothing to forget to do. I've written before about building software with these properties.
Every day is a bad day for policy
Policy controls degrade under precisely the conditions they exist for.
The threat model in most people's heads is the adversary - the movie villain probing the perimeter. The realistic threat is the configuration mistake at 4:45pm: the access granted "temporarily" during an incident, the exception that outlived its emergency, the permission copied from the last case because the deadline was today.
Policy asks humans for vigilance, and the bad day is, by definition, the day vigilance runs out - pressure, fatigue, turnover, a deadline. While structural controls do not care what kind of day it is.
Consider a dilemma familiar to anyone who has run compartmented work - casework, investigations, multi-agency anything. Shared reference material - legislation, doctrine, standards, precedent - has to serve every compartment.
There are two traditional ways to do it, and they fail in opposite directions. Copy the material into every compartment, and the copies drift; each case slowly ages against a different version of the truth. Or share it under permissions, and you are one misconfigured grant away from leakage between compartments that must never touch.
Both are policy failures. In this scenario, a structural solution would be a shared layer that is one way, or read-only by construction - not a write permission that nobody should grant, but a write path that does not exist.
The AI version of the policy question
This distinction matters now because AI assurance is actively being written as frameworks, principles, registers, and editable policy templates. Nearly all of it is policy: statements about what organisations should do - sometimes checked by attestation and sometimes enforced by vigilance.
But ask the question: what stops the model change nobody evaluated from reaching production? What stops an experiment from grading its own homework? What stops the automated re-run from quietly doing - and spending - the same work twice?
If the answer is "the AI governance committee reviews changes quarterly," you only have hope.
But structural answers exist, and none of them are exotic. A change that cannot ship without its evaluation evidence attached. Evaluation where the thing being judged is blinded from the thing doing the judging, so nothing marks its own work. Changes that reverse in one step, so the cost of being wrong collapses - and with it the need for the slow, defensive approval queues organisations build when mistakes are expensive.
When undo is expensive, institutions protect themselves by slowing down; a surprising amount of what we call governance is a reversibility problem wearing a governance hat.
This - I think - is how the adoption of AI in high stakes environments will be decided.
An accredited environment cannot absorb what it cannot describe. The blocker will rarely be whether the model is good enough. It will be whether anyone can state - precisely, and after the fact - what changed, when, on what evidence, and how it would be taken back.
Assurance written as policy cannot answer that under pressure. Systems whose unsafe states are unrepresentable answer it by existing.
Count your answers
In Part I, I observed that as analysis gets cheap, value migrates into the standard. This part adds a corollary: a standard is worth what its enforcement costs on the worst day, not the best.
A standard that lives in a document is a policy - a polite sticker on a fuel flap. A standard the system physically holds every piece of work to is an asset.
Next time you're shown an assurance framework... count the answers. How many name a property? How many name a person? That test tells you whether you are investing in structural safety, or just buying more stickers.
Part III turns from standards to conclusions - what happens to a finished assessment when the world keeps moving? The evidence doesn't stop arriving just because the report is signed. Subscribe to get part III delivered to your inbox systematically, (don't leave it to policy).
