An algorithm used for the first time by the city of Gothenburg in 2020 to allocate school places wrongly assigned hundreds of children to schools miles from their homes, the city’s own auditors later confirmed. Designed to optimise distances, preferences and capacity, the system instead calculated routes “as the crow flies” – a fatal oversight in a city split by a major river and fjords. More than 12,000 children were processed, but for roughly 700 of them the algorithm spat out placements that meant hour-long commutes across impassable walking or cycling routes. Those children, now in junior high, would remain in the “wrong” schools for their entire secondary years.
Nearly a year passed before city auditors acknowledged that the algorithm had been given flawed instructions. Procedures were improved for the following school year, but the 700 already affected received no remedy. Charlotta Kronblad, a former lawyer who now researches digital transformation at the University of Gothenburg, was among the parents staring at the decisions in disbelief. Her then-12-year-old son was one of the children displaced. Rather than appeal his individual placement – a route the city publicly suggested was sufficient – she took the city to court, challenging the legality of the entire decision-making system and all its output. She argued that the algorithm’s design violated applicable legislation.
Broader algorithmic injustice
The Gothenburg failure is not an isolated error, but part of a pattern that has played out across Europe. In the United Kingdom, the Post Office Horizon IT system – operated between 1999 and 2015 – falsely accused more than 900 subpostmasters of theft, fraud and false accounting. The Post Office prosecuted 700 of them, leading to bankruptcies, imprisonment and suicides. A group action by 555 subpostmasters resulted in a High Court ruling in 2019 that the system contained “bugs, errors and defects”. A settlement of £58 million left claimants with just £12 million after legal costs. Convictions began to be overturned in December 2020; by February 2024, 100 had been quashed, with total compensation expected to exceed £1 billion. Public outrage surged after a 2024 ITV drama, prompting new legislation to exonerate the wrongly convicted.
In the Netherlands, between 2005 and 2019, the Tax and Customs Administration used a self-learning algorithm to create risk profiles for childcare benefit fraud. Roughly 26,000 parents were wrongly accused of fraudulent claims. The system disproportionately flagged parents with foreign backgrounds and dual nationality – racial profiling that Amnesty International said was embedded in its design. Families were hit with demands to repay tens of thousands of euros, driving them into debt, homelessness and, in some cases, suicide. Over a thousand children were taken into foster care. The scandal forced the resignation of the third Rutte cabinet in January 2021, and a parliamentary inquiry concluded that fundamental principles of the rule of law had been violated. The tax authority was fined millions of euros for data protection breaches, but compensation and debt relief efforts have been slow and obstructed.
Legal challenges and the need for reform
Kronblad’s own legal battle in Gothenburg illustrates the deep structural barriers that individuals face when trying to contest algorithmic decisions. She repeatedly requested disclosure of the algorithm’s code; the city did not respond. Without access to the system itself, she conducted a painstaking analysis of hundreds of placements, reconstructing from addresses and school choices how the system must have operated, and presented that as evidence. The city’s defence was that the algorithm functioned merely as a “support tool” – and that they had done nothing wrong. They provided no technical documentation, no code, no process explanations. The court placed the burden of proof squarely on Kronblad. The judges said she had to demonstrate the system’s unlawfulness, yet her analysis of decisions was deemed insufficient without direct evidence of the code. The case was dismissed. “Prove what is in the black box, or lose,” she concluded.
This problem of proving algorithmic error is compounded by a phenomenon that Kronblad and her colleagues have termed “organisational ignoring”, referring to how public institutions can overlook or ignore algorithmic faults, leading to uncorrected injustices. Their research, published in MIS Quarterly, coined the term “multilayered blackboxing” to describe the layers of technical and social opacity that shield automated systems from scrutiny. In Sweden, the municipality of Trelleborg similarly cited a vendor’s non-disclosure agreement when refusing to disclose the logic of an algorithm used to decide eligibility for subsistence allowance. Even when the technology is relatively simple – as in Gothenburg, where the error was a basic distance-calculation mistake – citizens face a black box they must crack open to contest decisions.
Kronblad, a former lawyer with a decade of experience, holds a PhD in digital transformation from Chalmers University of Technology and has conducted postdoctoral research at the Stockholm School of Economics. Her work has been recognised by the Royal Swedish Academy of Engineering Sciences as one of the most promising projects for societal impact. She argues that the burden of proof must shift from the harmed individual to the party that designed and deployed the algorithm. Without that shift, courts will remain unable to interrogate algorithmic systems, and injustice will persist. The Swedish Social Insurance Agency has faced concerns about embedded bias in its algorithms since at least 2013; a 2018 report indicated its algorithm did not meet equal treatment standards. Amnesty International has called on Swedish authorities to discontinue discriminatory AI systems that unjustly flag marginalised groups for fraud inspections based on gender, foreign background, income and education level.
Sweden itself is re-evaluating its digital-first approach to public services, with education moving towards more analogue methods amid concerns about declining literacy and excessive screen time. But the challenge of algorithmic accountability remains urgent. As Kronblad put it, when courts defer to technology rather than interrogate it, and when procedural frameworks remain stubbornly analogue, algorithmic injustice will not only appear but can go on for years. The need for legal reform – for new procedural rules that allow systematic redress and compel transparency from those who build and deploy these systems – has never been more starkly illustrated.
