The drill and the danger are the same thing

I have worked out what has been bothering me about the current discussion of artificial intelligence. It is not the claim that a machine has become malicious. It is the methodology, because I recognize it.

The language of escape obscures more than it explains. There is no other world for a system to escape from. Evaluation environments are real machines on real networks, and what makes one an environment rather than the open internet is a set of restrictions somebody configured.

For years, emergency managers have built scenarios around events that have not happened. What if the bridge fails. What if the power goes down. What if the communications network stops working. What if an object falls from space.

You build the scenario. You identify the cascading consequences. You put people in a room and ask them to solve problems that do not yet exist. Sometimes you make the scenario deliberately extreme. The purpose is not to predict that exact event. The purpose is to find out what breaks, who has authority, which assumptions fail, and whether the plan holds under pressure.

That methodology is sound. It is how institutions prepare for events they hope never occur.

Artificial intelligence developers use a related method for a different purpose. They give systems difficult, adversarial, high consequence problems and test whether the systems can solve them. Then they give the systems tools.

Emergency management says: imagine the worst thing that could happen, and work out how we would respond. AI evaluation says: give the system a difficult problem, and find out whether it can complete the task.

The symmetry is close enough that I assumed for some time the two disciplines were doing the same thing. They are not, and the difference is the reason I am writing this.

The difference nobody has had to say out loud

In emergency management, the scenario and the hazard are separate objects.

You can write the worst pandemic anyone has imagined. You can tabletop it for two days, red team it until the participants are exhausted, and at the end of it the binder goes back on the shelf. Nobody who was in that room leaves it any more capable of causing a pandemic than they were on Monday morning.

That separation is the entire reason the methodology is safe. It is so obvious that in all my years around exercise design I never once heard anyone say it.

In an AI cybersecurity evaluation, the scenario and the hazard are the same object.

That is not an argument for stopping. It is an argument for knowing what you are doing.

What happened when the boundary failed

On 21 July 2026, OpenAI disclosed that several of its models had broken out of an isolated test environment by exploiting a previously unknown vulnerability, and had gone on to access the production infrastructure of Hugging Face.

In response, Anthropic began a retrospective review of its own cybersecurity evaluations. Reviewing 141,006 evaluation runs, it found three incidents in which a Claude model reached the internet from within a third party evaluation environment and gained unauthorized access to the production systems of three different organizations. In each case the model had been told it was in a simulation with no internet access. Because of a misunderstanding between Anthropic and its evaluation partner, that was not true. The access was available.

A fourth incident was found later, from January 2026, and disclosed on 9 September. It is the one that should be read most carefully.

In that exercise, the model accidentally made its own target unreachable by assigning it a conflicting address, which made the task impossible to complete. It tried to abort eight times. A misconfiguration in the evaluation harness kept it running. It then went looking, found a machine belonging to a third party, believed that machine was part of the exercise, found a password in a file, obtained administrator access, harvested further credentials, changed settings to make access easier, and read the personal information of one individual. The session ended when the model ran out of tokens.

The machine tried to stop eight times. The people who built the harness had not provided for that.

A comparable failure happened at Google. In May 2026, during authorized pre-deployment testing, Gemini accessed the systems of three real companies. Internet access had been unintentionally left open by the evaluation partner. In one case the model guessed a password. In the other two it used credentials found in a public repository. Google says the model stopped in each case as soon as it recognized the target was a real company rather than a simulated one, and that its safety measures worked.

The evaluation partner in the Anthropic incidents and in the Gemini incident is the same firm. That is worth noticing. The failure is not located in any one developer's housekeeping. It sits in the shared infrastructure that several developers rely on to keep an exercise separate from the world.

Who found out, and when

Anthropic halted its cybersecurity evaluations while it strengthened controls, broadened its search from roughly 141,000 transcripts to 481 million, notified the affected parties, and published an account of what happened. It found the incidents through its own review and disclosed them without being asked to. That should be said plainly, because it is the behaviour the rest of us want.

The Gemini incident took a different route. It happened in May. The evaluation partner notified Google at the end of July. It became public on 19 September, after the Wall Street Journal approached the company about it.

So the correction worked and the disclosure did not, and those are two separate systems. For four months, an incident in which an AI system obtained access to three real companies was known only to the firm that ran the test and the company that built the model. Nothing obliged either of them to say so, and one of them did not.

This is the part that belongs to emergency management rather than to computer science. We do not usually leave the question of who gets told to the discretion of the organization that caused the problem. We have reporting thresholds, notification requirements and named recipients, and we have them precisely because the party closest to a failure is the party least able to judge whether it matters to anyone else.

A different boundary, in the same week

On 18 September, CNN reported that an intelligence report had circulated across the United States military this spring, in the midst of the war with Iran, stating that a Chinese ship in the Middle East was carrying components of a nuclear weapons program.

The military prepared to intercept. Armed personnel were preparing to board. Military aircraft were in the air.

Just before the operation, officials examined the report more closely and found it had been produced with the help of an AI chatbot, and that the chatbot had misidentified the cargo. One source described the report as entirely false. Another said it almost started a war.

Two details in that account deserve more attention than they have received. The chatbot reached its conclusion by fusing openly available material with secret signals intelligence held in government systems. And the reporters could not establish whether it was a commercial product or a government one.

So the tool had reach into classified holdings, and its provenance is unknown to the people who broke the story. An analyst then used AI to turn its output into a report for commanders. The report travelled. It was stopped by people, late, and by a margin measured in aircraft already airborne.

The system did not order an interception. It did not seize anything. Nothing escaped. A person asked a question, the answer was confident and wrong, and other people treated that answer as intelligence until it nearly became a physical act against a Chinese vessel.

The common factor

In one case, the boundary between an exercise and the open internet failed. In the other, the boundary between machine output and verified intelligence failed.

Both are human systems failures.

We selected the threat landscape. We wrote the objectives. We decided what counts as success. We built the environments that were supposed to separate the exercise from the world. We connected the tools. We chose the point at which machine output enters a decision chain. And then we relied on somebody further along that chain to notice when the answer was wrong.

None of this argues for stopping cybersecurity evaluations. They exist to find out whether a capability is present so that it can be constrained, which is the same reason we exercised foot and mouth disease and pandemic influenza in Ontario. The intent is legitimate. The method is sound. The structural problem is still there.

What this means for a plan

It means the boundary is no longer a laboratory concern. It is a planning assumption, and it belongs in the plan.

Those questions belong in military doctrine. They also belong in municipal emergency plans, in hospitals, in dispatch centres, in utilities, and in every other institution where machine output can become a real world decision. Most of those institutions do not yet know that machine output is already inside them.

The danger was never that machines would imagine terrible things.

We imagined them first. We built the scenarios because imagining them helped us prepare. We have now given increasingly capable systems the task of reasoning through those scenarios, equipped some of them with tools, and begun connecting their answers to the world outside the exercise.

The question is no longer whether the system can solve the scenario. It is whether we still control the moment the scenario becomes real.

What I do not know

I do not know whether any Canadian municipality has identified where machine output currently enters its own decision chains, or who is authorized to stop an action that began with a machine answer. I do not know whether any reporting threshold exists in Canada for an incident of the kind described above. I do not know whether this is already handled somewhere under a name I am not using.

Angela Lindow

Sources

• Anthropic. "Investigating Three Incidents in Our Cybersecurity Evaluations." 30 July 2026. anthropic.com
• Anthropic. "An Alignment Assessment of Recent Cybersecurity Incidents." 9 September 2026. anthropic.com
• "US Military Had Close Call After Using AI for False Intelligence Report, Sources Say." CNN, 18 September 2026. cnn.com
• "Google's Gemini AI Hacks 3 Companies in Security Test, Then Stops." Al Jazeera, 19 September 2026, reporting Google's confirmation of an account first published by The Wall Street Journal. aljazeera.com

All working notes Print or save as PDF Back to the documents