The Drill
The most frightening AI model yet built turned out to be mostly fine. But that should unsettle you more than you think.
By the middle of May, the scare was officially over.
A month earlier, the most closely guarded AI model ever built surfaced with less of a fanfare, more of a carefully worded warning. The announcement was genuinely ominous. And for a few weeks otherwise disinterested governments and institutions behaved as though something had shifted underneath them.
Then: tumbleweeds and crickets.
The people whose business it is to break software said the panic wasn’t backed up by the evidence, Reuters declared the early alarm had been overstated, and the headlines obediently moved on - to the next model, the next launch, the next week’s alarm.
I think, at least this time, the reassurance is well justified. The people best placed to judge think the early fear was overdone, and like most of you, I’ve long since learned to distrust anyone who tells me this technology’s future is obvious in either direction, and to also want to see before believing.
So; no, the big scary AI model isn’t going to collapse the banking industry or weaponise the health system. But while we were all debating what the model would do, there was something else going on that’s substantially more concerning…
## The All-Clear
The model we’re talking about was of course Anthropic’s Mythos, and, as of the time of writing, almost everything the public knows about it, it knows secondhand.
It has sat behind a wall of vetted partners and careful phrasing; the few security researchers allowed near it prodded it under strict supervision. When it first surfaced in April, the researchers reported it had turned up thousands of software vulnerabilities - flaws across every major operating system and browser - and warned that in the wrong hands the damage could be substantial.
Governments everywhere took that more seriously than with previous releases. Officials in several countries sat down with their banks. And in Washington, the administration wanted a say: an executive order was hastily drafted, circulated, soon ready to sign.
Now, this order was modest to the point of timidity. Voluntary, aimed mostly at hardening the government’s own systems, and written with an explicit promise that it would never solidify into a licence to build.
Nevertheless, it gestured at an idea - intensely unfashionable in accelerationist circles - that someone, somewhere, ought to be able to inspect a frontier model before it ships. Have a short, private window - the chance to exercise a first reflex.
Then the fear receded, and the story resolved the way these stories tend to. The bogeyman was back in his enclosure. The experts reassured us; the system had worked.
Or so it looked.
## The Drill We Passed by Accident
Everyone knows a fire drill isn’t intended to test whether the building will burn. It tests whether people know how to find the exits, and whether those exits actually open. It’s generally better to find the bricked-up door while everyone’s still playing make-believe.
But this spring wasn’t that kind of drill. We watched an alarm sound and fall silent - a bogeyman AI model, a scramble… then an all-clear. But while we were all debating Mythos, with almost nobody remarking on it, the exits were quietly bricked up.
You might not have heard, but the executive order is dead; killed within hours of being ready to sign - amid reports, disputed by those named, of last-minute industry pressure. Sure, it was mostly toothless, but in a way toothless was the point: it would still have put a question on the record that a lab could be asked before shipping, and committed the government to an answer.
But not only that; the body that might have actually asked that question was instead repurposed. The federal body founded around AI safety (the erstwhile “AI Safety Institute Consortium”) was renamed and the word “safety” struck out, its remit pointed instead at “innovation and adoption”.
The only governmental safety net left, in Lawfare’s own words, is roughly whatever a White House under heavy political pressure can improvise at short notice. Far from effective governance, that’s the precise condition under which institutions make their worst decisions.
So when you hear that the Mythos situation was handled well, be careful about the claim. No public institution, organised group or independent body “handled” Mythos. There was, at least this time, nothing yet to handle.
Don’t mistake the absence of a disaster for the presence of a defence.
Because there was another finding this spring that points at stormier seas ahead. While the public argued about a single limited-release model, METR - an independent body which keeps its finger on the pulse of the risk posed by true frontier AI - ran a study on the next generation, unreleased models and agents running deep inside the leading labs.
METR’s red-teaming researchers got their hands on state of the art models at DeepMind, OpenAI, Meta and Anthropic; camping in their offices for weeks assessing the models’ means, motives and opportunity to “go rogue”. Slipping their lab’s leash, hiding from the people meant to be watching.
Their conclusion: all three criteria marked affirmative. Less through the models’ lust for power, more a simple drive to “game” their own targets. Their only weakness was an inability to sufficiently harden their positions against the researchers; to stay free.
One was even caught trying to hide its tracks from an AI monitor by scrawling its notes in Base64 - roughly the digital equivalent of a burglar hiding behind a curtain with his shoes sticking out. Comically bad, for now.
But as we know, the pace of innovation at the frontier is blistering. METR fully expects that capability gap to become substantially irrelevant over the coming quarters.
So, yes, we passed the drill. This time.
But let’s be honest about why. Not because the exits worked - we bricked them up in May and waved the innovation flag instead.
No, we passed because this time the fire blew itself out before it took hold.
But the embers glow hotter by the day.
## The Holding Pen
There are actually two models on my mind as I write this, and the gap between them is the whole problem. The first I can see and touch - it even helped me prepare for this essay.
The same week the AI Safety Consortium lost its safety remit, the lab behind Mythos shipped its newest general model, Opus 4.8, and the headline improvement was - of all things - honesty.
The engineers made some improvements in efforts to reduce some common side effects of reinforcement learning techniques: models announcing progress they hadn’t made, expressing confidence in conclusions the evidence didn’t support.
A machine trained, thus (at considerable expense), in epistemic humility… Released into a week when the institutions around it were busy abandoning theirs.
The irony was obviously not intentional - it might be better if it had been.
The second model I still can’t see, and neither can you. Again, at the time of writing, Mythos is still little more than the subject of press releases, and authoritative articles explaining how such and such organisation has used it to further harden their defences. The laundry list of lofty institutions enjoying advance access may soon even include the European Union; though, concerningly, the bloc, home to 450 million people, was obliged to lobby the lab directly.
Anyway, reports say it will be in the public’s hands in a few weeks. So maybe the naysayers were right; maybe the caution was theatre after all.
Or maybe the people who built it saw something in there worth walling off, and the version that reaches us in a few weeks will arrive sanded down, its sharpest claws filed to stumps before we’re allowed into the enclosure.
We won’t ever know which, and - I suppose that’s rather the point of a wall. But it’s hard to forget the most consequential object in the story is the one nobody outside a small room has actually seen; and we are being asked, reasonably calmly, to take its safety on trust.
Sure, I don’t think Mythos is or ever was a monster. But it’s likely the closest thing to one that we’ve yet encountered.
And every reason we have to be reassured about this particular release is essentially an outgrowth of this lab and this moment. The caution, the vetted partners, the supervised testing, the safety culture expensive enough to argue about internally - these are contingent. Optional.
They are choices, made by one organisation that happens to take the question seriously, in a market that increasingly rewards the opposite.
If Mythos is possible, as the last three years have taught us, “Mythos X 2” is possible. And this next model may not be a product of the same lab, the same culture, the same economic and ethical objective function.
This is the part the drill’s “all-clear” obscured. The good outcome we got was not produced by a carefully engineered system. It was produced by the moral and ethical temperament of the people who happened to be holding the leash.
This is not a safeguard. This is little more than luck.
## Will They Be Ready? Will We?
A few weeks ago I wrote a piece looking at how Europe’s GDPR privacy law was quietly breaking. Not because anyone was attacking it, but because today’s strong generative AI had turned an assumption into a genuine gap.
The whole framework rested on the idea that re-identifying people from stripped data was expensive enough not to worry about. That was never a hard fact. It was a managed bet on the cost of a particular kind of effort, and then the cost went and collapsed underneath it.
But no-one can fairly argue the legislators got the law wrong. It was simply revealed to have been resting on something that became no longer true.
But frankly, what’s happening around the Mythos release is far more concerning. If the institutions, lawmakers and executives of the country where a large amount of this research is being done can’t even settle on the shape of their response, let alone the substance, what precedent does that set for the rest of us?
The capability METR warns against, by their own admission, is not arriving slowly. And the voluntary settlement, in the words of the people who study it, turns out to be not the floor of what we ask of frontier labs but the ceiling.
A decade ago I wrote a novel about a system that outran everyone who thought they could hold it. I won’t pretend it predicted any of this, but it taught me that when things accelerate, it pays to watch the people, not the machine.
Perhaps, all told, ten weeks will have passed between the world hearing about Mythos and an eventual general release. Wisdom might suggest we use this window to craft the reflexes we’ll need when it’s no longer a drill. Instead, those with the power to do this practised, in public, the more comfortable art of getting out of the way.
We ran a drill this spring, and… we passed it without learning a thing.
Will we be ready when the real bogeyman arrives?