We Have Seen This Before

A silent summary of this note, with its sources on screen. Every card carries its own text.

Most of what is on this site looks forward, at systems still being installed. This note looks back. The failures that artificial intelligence could bring to public services have already happened with older technology, in Canada and elsewhere. In each case a public inquiry or a court went through the wreckage afterward and wrote down what went wrong.

Five of those records follow. Read together, they show the ways responsibility comes apart when a system fails, and AI makes every one of them faster and harder to spot.

1. Walkerton, Ontario

May 2000

After days of heavy rain, bacteria from nearby farmland got into one of the wells that supplied Walkerton, a town of about five thousand people. A private laboratory found the contamination in routine samples. The results went to the local utility. They did not go to the medical officer of health, because since the province had moved water testing to private laboratories in 1996, no regulation required them to. By the time the health unit put the pieces together and told people to boil their water, residents were already sick. Seven of them died, and more than 2,300 became ill.

Justice Dennis O'Connor's inquiry found people who did their jobs badly, starting with the two brothers who ran the utility. It also found a system that looked complete on paper: a utility, a ministry, a laboratory and a health unit, each with its own mandate and its own filing cabinet. What it lacked was anyone whose job was the space between them, no one whose phone rang when a bad result went to the wrong office and stopped there. Anyone who has worked in those rooms will recognize the feeling of going home believing your office did its part.

For an AI system inside a municipal service, the cast changes but the shape does not. The vendor builds it, a ministry may fund or regulate the service, the municipality runs it, and the model underneath may come from a company with no office in Canada. Each will have its own mandate and its own filing cabinet.

2. Pickering, Ontario

12 January 2020

A little after twenty past seven on a Sunday morning, phones went off in kitchens from Windsor to Kenora. The alert said there had been an incident at the Pickering Nuclear Generating Station, that no radioactivity had been released, and that emergency staff were responding. It did not say what the incident was, and it did not tell anyone to do anything.

I was at home when mine sounded. I had spent years around the provincial emergency plans, and I knew what a real alert about Pickering would look like: sirens for the people within three kilometres of the station, instructions for the ten kilometre zone around it, and a message written for them rather than for a grandmother in Sudbury. This one was written for everyone. I knew it was a mistake before I reached the end of it. Most people had no way of knowing that, and nothing to do but wait for someone to tell them.

Ontario Power Generation, which runs the station, said on social media within the hour that the alert had gone out in error. The province sent its own correction just after ten past nine, 108 minutes after the first message. The investigation found that a duty officer running a shift-change test believed they had switched to the training system and had not. It cited:

When the error was discovered, supervisors were unsure how to send a correcting alert. A province with a nuclear emergency plan, planning zones and an operations centre was undone by one person who had not logged out, and nobody who knew how to say so.

An automated system in that duty officer's chair would put the test and the live system even closer together. Its messages would go out faster and further, and somebody would still have to know how to take one back.

3. Nova Scotia

April 2020

Over the night of April 18 and into the next day, a gunman killed 22 people across rural Nova Scotia. At 11:32 p.m. on the first night, police posted on social media about a "firearms complaint." Nothing more was shared publicly until eight the next morning. Alert Ready, the same national system that had woken Ontario three months earlier, was never used.

The Mass Casualty Commission reported on March 30, 2023:

The Commission also found that the widespread belief that alerts cause panic had no evidence behind it. The system worked. What was missing was someone who owned the decision to use it, and who believed it was theirs to make.

Emergency management handles shared authority by letting ownership switch on: a head of council declares an emergency, and command becomes clear. That arrangement assumes someone has already been named to flip the switch. If no one owns the decision to treat an AI failure as an emergency, it will be treated as a service issue until the harm is obvious to everyone.

4. The Post Office Horizon scandal, United Kingdom

1999 onward

In 1999 and 2000, the Post Office installed an accounting system called Horizon in its branches across the United Kingdom. Over the following fifteen years, more than 700 subpostmasters were prosecuted when the system showed money missing from their tills. Many were convicted of theft or false accounting, and some went to prison. Those who questioned the figures were often told they were the only one having problems.

In 2019 the High Court found that the system contained:

Those defects could produce exactly the shortfalls the subpostmasters had been blamed for. The courts of England and Wales presume that a computer was working properly unless someone proves otherwise, and a subpostmaster at a village counter had no way to prove anything about software she could not see inside. In 2024 Parliament passed legislation quashing the remaining convictions as a group. The public inquiry chaired by Sir Wyn Williams has described at least 10,000 people as affected.

Researchers call this automation bias: the habit of believing the machine over the person in front of you. AI systems produce fluent, confident answers by design. A caseworker or a dispatcher looking at one of those answers on a screen is under the same pressure the Post Office's investigators were, and the resident across the counter faces the same disbelief.

5. The childcare benefits scandal, the Netherlands

2013 to 2021

The Dutch tax authority used a risk-classification model to flag childcare benefit claims for possible fraud, and nationality was among the factors it used. Tens of thousands of families were treated as fraudsters, ordered to repay benefits they were entitled to, and pushed into debt. Some lost their homes and their jobs.

In December 2020 a parliamentary inquiry published its report under the title:

On January 15, 2021, the Dutch government resigned.

Nothing in this case looked like an outage. The model flagged, officials followed up, letters went out and repayments were demanded, every step according to procedure. It ran for years because it looked like normal operation. An AI failure inside Ontario's income support would most likely arrive the same way: as a letter, a suspended payment, a flag on a file. It would reach the kitchen table long before it reached a council chamber.

What the record shows

None of these needed artificial intelligence. Each of them can now happen inside an AI-enabled public service, with less warning and at greater scale. Unlike the people in these five cases, we have the record in advance.

Questions every municipality can ask now

1. For each AI-enabled system in an essential service, who owns the decision to declare that it has failed?
2. If a system sends a wrong notice to the public, who sends the correction, through which channel, and how fast?
3. When a resident says the system is wrong, what process lets staff believe the resident over the screen?
4. Which of your systems could be failing right now in a way that looks like normal operation?
5. Who holds the vendor's obligations, and has anyone in your organization read them?

None of these needs new legislation. Each fits within the emergency management program Ontario municipalities are already required to have.

Sources and verification notes

• Report of the Walkerton Inquiry, the Honourable Dennis R. O'Connor, 2002.
• CBC News. "Human error, 'systemic issues' led to false Pickering nuclear alert, province says." February 2020. cbc.ca
• Global News. Reporting on the Pickering Nuclear Generating Station emergency alert, 12 January 2020. globalnews.ca
• City of Pickering. "Nuclear Emergencies," on the three and ten kilometre zones. pickering.ca
• Mass Casualty Commission. Final Report. 30 March 2023. Reporting by Global News. globalnews.ca
• Bates and others v Post Office Ltd, High Court of England and Wales, 2019; Post Office Horizon IT Inquiry, chaired by Sir Wyn Williams. Summary by Computer Weekly. computerweekly.com
• Computer Weekly. "Post Office attacked subpostmasters who questioned Horizon, say victims," on subpostmasters being told they were the only one experiencing problems. computerweekly.com
• House of Representatives of the Netherlands, parliamentary interrogation committee on childcare benefits, "Unprecedented Injustice," December 2020; Dutch Data Protection Authority findings on the Tax and Customs Administration's use of nationality. Summary: eulawenforcement.com
• For the claim that the second emergency has no owner, see Working Note 01, "The second emergency has no owner." For the failure types and the kitchen-table consequences these cases illustrate, see Working Note 03, "When System Failure Reaches the Kitchen Table."

Verification note: figures are taken from the inquiry reports and the reporting listed above. The Dutch figure is given as tens of thousands because published counts of affected families vary between sources.

All working notes Print or save as PDF Back to the documents