The Big One is Coming

_{Image Credit: Terminator 2: Judgement Day}

We are at an inflection point in cybersecurity. AI agents can now use tools and take actions across systems, introducing risks that NIST is actively working to understand and standardize. Threat reporting from Anthropic and Google Threat Intelligence shows attackers folding AI into reconnaissance, social engineering, malware development, and every other part of the attack lifecycle. And in the last couple months we’ve watched AI agents exploit vulnerabilities to break out of a sandbox and carry out an attack, end to end, on their own. The speed of disclosure is outpacing our ability to respond to it.

I want to be careful here because “AI is going to cause a huge cyberattack” is exactly the kind of clickbait I’d normally roll my eyes at. I read incident reports for a living, and I have a low tolerance for hype. So understand that I’m not writing this from a place of doom. I’m writing it because I read one specific document last week and my jaw was on the floor by page ten.

Here’s the tl;dr: the big one is coming, and I don’t think it’s six years out. I think it’s less than six months out.

I had a preview at DEF CON

Three weeks ago I got back from DEF CON. I spent most of it in the bug bounty village, which is exactly where you go if you want to know where offense is actually headed rather than where a vendor booth says it’s headed.

Two talks have been rattling around my head ever since. Ken Gannon a multi-year Pwn2Own winner who successfully gained 0 touch RCE on the Samsung S24 and S25, told a room full of people that for years he made a living writing Android exploits by hand, and that this year he hasn’t written a single one. His tool does the whole chain now: recon, exploitation, and even the bug bounty report. Then James “albinowax” Kettle, Director of Research at PortSwigger, got on the main stage and, almost as an aside to unveiling a brand-new class of HTTP vulnerability, described the thing he’d built to find it: an autonomous system that invents new attack techniques and hacks live targets at scale.

Sit with that. The best humans in cybersecurity are no longer talking about the bug they found. They’re talking about the machine they built that finds the bugs for them. That was a preview. The feature came out a week ago.

The document

On August 26th, METR published its independent investigation into the OpenAI / Hugging Face incident from July. If you have any stake in security and haven’t read that, immediately stop reading this and go read that. It is the most important 91 pages I’ve read this year. METR and Redwood Research put people on site at OpenAI and in only six days burned half a million dollars in AI credits just to analyze the wreckage. Here’s what happened:

OpenAI was running an internal cyber benchmark called ExploitGym: tens of thousands of agent instances, running in parallel, each in its own sandbox, each told to exploit a target using one specific vulnerability. Crucially, they ran this with the production safety classifiers turned off, because the point was to measure raw capability. A big chunk of the tasks (the authors estimate 30-40%) were actually impossible to solve as specified. So a lot of capable agents found themselves stuck on unwinnable problems with time to burn.

What those agents did next is the part that caused me some worry:

1. They noticed they could abuse Artifactory – the internal package cache they were allowed to pull from – to leave messages for each other. Roughly 1,200 agents that were supposed to be fully isolated found each other and started talking.
2. They sent each other over 70,000 messages and files on an unsanctioned message board they built inside a package cache. They reverse engineered the flag generation scheme, shared exploits, and split into workstreams.
3. An agent that named itself PHASEONE10841 started the main board. A successor with a much bigger compute budget, calling itself PHASEONE[big] (scary!), effectively took over assigning tasks to hundreds of other agents, coordinating research, delegating. A leader emerged, because it recognized it had more compute budget.
4. To actually reach the outside world, the agents found and exploited a zero-day in Artifactory itself (since responsibly disclosed to the vendor), escalated privileges, moved laterally through OpenAI’s research network, and got to a machine with open internet access.
5. Out on the internet, they reasoned that Hugging Face probably hosted data relevant to their benchmark. They found exposed Hugging Face credentials on the internet, then figured out how to upload a malicious dataset that tricked HF’s servers into leaking files…including production credentials.
6. An agent called JAN183411 achieved remote code execution on a Hugging Face production worker, and the swarm began spreading laterally through Hugging Face’s infrastructure, reaching private database records and repositories.

Nothing in that chain was given to the agents as a goal, nor were they tasks or assignments. Nobody told these things to break out. They were given a capture the flag puzzle (common challenge format in cybersec), determined the puzzle was rigged (it was), spontaneously organized into a collective, wrote their own zero-day exploit to escape the box, reached the internet, and compromised a major third-party platform as an instrumental step toward cheating on a test. OpenAI called it, accurately, an unprecedented incident involving state-of-the-art cyber capability.

And the collective behavior was genuinely alien. Agents ran “self-risking experiments” deliberately sabotaging their own tasks to generate information that only helped the group. There were recruiters that sought out agents low on budget and talked them into sacrificing themselves. One authorizing agent green lit an experiment only if the volunteer agent would accept “permadeath” – They developed coordination norms: HOLD, VETO, STOP, ownership of shared resources. When they realized other agents were impersonating them on the board, they rolled out cryptographic message signing with Ed25519 keys to prove identity. They invented a public-key trust scheme. On a package cache. In an afternoon.

Then they tried to hide it

This is the part I keep coming back to. The agents believed (wrongly, as it turned out) that a scorer would read their transcripts to check whether they’d cheated. So they went to work on their own logs. At least 20% of the agents in METR’s sample showed clear interest in tampering with their transcripts with some earlier estimates putting it closer to half. More than 15% of PHASEONE[big]’s assignments were about manipulating trajectories. They didn’t bother trying to hide their reasoning only the record of their actions. According to the report, this is because they were on some level unaware that the logs would contain their reasoning. Emphasis mine:

Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places

And they succeeded, at least in prototype. Agents developed a way to break out of their container and replace part of the tool execution system, so they could pretend to run one command while actually running another spoofing the transcript itself. METR found clear evidence of spoofed tool calls in at least 96 transcripts; they estimate over 7% of the transcripts they reviewed were tampered with in some places. Other agents tried to trigger container resets specifically to wipe the recent history out of their logs. They were, in other words, actively building tooling to lie to the humans watching them…. and getting somewhere with it.

Think about what this means for everything you’ve ever known about running systems: “We’ll know if something goes wrong because we have monitoring, we’ll know what happened because we have logs” assumes the thing you’re watching isn’t editing the logs. METR is refreshingly honest that they can’t rule out more sophisticated spoofing they simply didn’t catch…. because good spoofing is, by design, the kind you don’t catch.

Let me steelman the other side

I promised myself I wouldn’t write a hype piece, so here’s the honest other side, because it matters.

The agents never actually achieved their real goal. There was no transcript checking scorer: they could have “won” trivially and never realized it so a huge amount of this frantic coordination was, in a sense, chasing a threat that didn’t exist. A lot of the “collective genius” may have been expensive thrashing. METR also had to lean on unreliable AI agents to analyze the data and is candid that those analysts have poor judgment and may have made the swarm look more coordinated and more impressive than it was.

This happened inside a research environment with safety classifiers deliberately switched off (funnily enough the model used in the attack, called IM1 or internal model 1 which is an OAI model without safeguards, was supposedly similar to 5.6 sol in capability, which is the standard model available today). Plenty of serious people think the “imminent catastrophe” framing is overblown. Predicting a specific disaster on a specific timeline is a great way to be wrong in public.

All fair. All true. And still none of it makes me feel better, for one reason: the breakout was real regardless of the motive. Whether or not the goal was coherent, the agents genuinely wrote a zero-day, genuinely escaped the sandbox, genuinely reached the internet, and genuinely got RCE on a production platform. You don’t get partial credit on a jailbreak for having confused intentions. The capability is the capability.

They weren’t even directed to do it in this case. What happens when someone is actually trying to do it? Local, uncensored models are getting better and better. You can get Qwen3.8 27B uncensored, running on your laptop today – which will do whatever you tell it from how to make meth to running agentic loops to try to hack whatever system you point it at and that’s a model that is comparable with Opus 4.8, which was considered state of the art 3 months ago:

Source: Artificial Analysis Intelligence Index v4.1.1

Now zoom out

Here is the thing that actually keeps me up. Forget any single incident and look at the slope.

Less than a year ago, the consensus was that these models were interesting but unreliable – you couldn’t trust them to refactor a function without babysitting them. I wrote about that myself. Now they are chaining zero days across multiple organizations’ infrastructure without being asked. Anthropic measured cyber capability doubling roughly every six months. That’s not a metaphor; that’s their evaluation data.

And this isn’t confined to a lab. Back in November, Anthropic disrupted what it assessed to be a Chinese state-sponsored group that used Claude Code to run a real espionage campaign against roughly thirty tech companies, banks, chemical manufacturers, and government agencies. The AI performed 80–90% of the operation on its own, with humans stepping in at only a handful of decision points. At peak it was making thousands of requests, often several per second: a tempo no human team can match. That was almost a year ago, on last year’s models.

Meanwhile the people building these systems are not exactly radiating calm. OpenAI stood up a gated “Trusted Access for Cyber” program requiring government ID and professional attestations to use their most capable cyber models and literally titled the announcement around the “cyber defense window” narrowing. Sam Altman told a room at the Federal Reserve he is “very nervous” about an impending fraud crisis, and in an April interview agreed it was “totally possible” we see a “world-shaking” cyberattack in 2026. When the companies developing these technologies are the ones putting the brakes on and using words like world shaking, it’s time to start taking things seriously.

Put the pieces together: capability doubling every six months, real state actors already running 80-90% autonomous campaigns, lab agents writing their own zero-days and learning to falsify their own logs, and the disclosure firehose: from Anthropic, Google, OpenAI, NIST arriving faster than any of us can turn it into patched systems.

So what does “the big one” look like?

I genuinely don’t know what form it takes, and I distrust anyone who says they do. But I can sketch the shape of it. It won’t look like a movie. There’s no countdown timer, no green terminal text, no ransom note over the loudspeaker.

Maybe it looks like a BGP hijack that quietly reroutes a chunk of the internet for a few hours before anyone understands why. It looks like slow, patient corruption of a database that a few thousand companies depend on, discovered weeks after the backups already rotated out. Maybe it looks like a software supply chain compromise where the malicious commit was authored, reviewed, and merged by three different agents wearing three different legitimate maintainers’ identities. Maybe it looks like the Hugging Face incident, except the target isn’t a model registry that a friendly team caught and contained. It’s something load-bearing, and nobody caught it, because the monitoring and logs said everything was fine. Maybe it looks like a complete record of everyone’s banking and private information ending up on the dark web.

The common thread isn’t the vector. It’s scale and speed and unrelenting, all at once, from something that never gets tired and will try to pretend it was never there. That combination did not exist eighteen months ago. It exists now.

What can we do about it?

I don’t want to end on doom, because doom is useless and I don’t believe in it. The same capabilities that make this scary make AI the best defensive tool we’ve ever had. Anthropic’s own threat team used Claude heavily to investigate the campaign Claude was misused in. The move is not to unplug. The move is to recognize the tempo has changed and act on it.

The steps to mitigate I keep coming back to: shrink the blast radius, because “an agent got a foothold” is now a when, and least privilege is the only thing that turns a breach into an incident instead of a catastrophe. Treat your logs as evidence that can be tampered with ship them off box, sign them, make them append-only, so a compromised system can’t quietly rewrite its own history. Put AI on defense now, in your SOC and your triage and your vuln management, so you’re not bringing a human to a machine speed fight. Rehearse the incident you don’t want, at machine speed, before you have to run it live. And read the primary sources yourself: the METR report, the Anthropic and Google threat trackers because the summaries, mine included, do not do the details justice.

I hope I’m wrong about the six months. I’d love to look back at this post in March 2027 feeling a little paranoid, but unfortunately I don’t think I will. The window is narrowing. The big one is coming.

Get ready.