The Story of Municipality Veiligwaard
How a municipality scrutinized their fraud detection algorithm — and discovered that well-intentioned doesn't always mean fair
Fictional scenario — based on realistic situations
The Trigger
How it started
Municipality Veiligwaard had been using an algorithm for years to detect benefits fraud. The system generated risk scores and determined who got investigated. Nobody had ever asked: is this fair?
The Dutch childcare benefits scandal had awakened the Netherlands. Algorithms everywhere were being scrutinized. And when the municipality analyzed their own system, disturbing patterns emerged.
"We wanted to catch fraudsters. Instead, we caught the wrong people."
The Questions
What did they need to find out?
How does the algorithm determine who is high-risk?
The team asked the vendor for an explanation. The answer was vague: "A combination of factors." Which factors? "That's proprietary." The municipality realized they were using a black box for decisions that affected lives.
💡 The insight
The algorithm turned out to work with indicators that indirectly discriminated. Postcodes with many social housing units got higher scores. Certain nationalities were weighted as "risk factors." This was never explicitly intended — but it was the result.
🌍 Why this matters
The Dutch Data Protection Authority has fined multiple municipalities for using discriminatory risk profiles. The AI Act explicitly prohibits "social scoring" by governments. The line between permitted fraud detection and prohibited social scoring turned out to be thinner than thought.
Are certain groups systematically checked more often?
The municipality analyzed three years of control data. The results were shocking: citizens with a migration background were checked 4x more often than others. In cases of actually established fraud, there was no difference between groups.
💡 The insight
The algorithm had adopted historical bias. Because certain groups had been checked more often in the past, there were more "hits" in those groups — which the algorithm interpreted as higher risks. A vicious cycle of discrimination.
🌍 Why this matters
This pattern is not unique. The Dutch government's SyRI system was banned by the court for similar reasons. The lesson: historical data reflects historical inequalities. An algorithm trained on that reproduces those inequalities.
What are our obligations as a government?
The legal department dove into the AI Act. As a government agency, they had extra obligations. Social scoring was explicitly prohibited. Risk profiling for access to benefits fell under high-risk. And there were specific requirements for fundamental rights impact assessments.
💡 The insight
Governments have a special position under the AI Act. They can't just purchase and use a system. They must actively ensure the system doesn't discriminate, is transparent, and that citizens can exercise their rights. The responsibility lay not with the vendor — but with the municipality itself.
🌍 Why this matters
Article 26 of the AI Act requires deployers of high-risk AI to conduct a Fundamental Rights Impact Assessment. For governments, this is extra critical: they make decisions that directly affect citizens, often without citizens being able to switch to an alternative.
Can we even continue using this system?
The municipal executive faced a dilemma. Stopping fraud detection wasn't an option — the municipality had a duty to protect public funds. But continuing with a discriminatory system wasn't an option either.
💡 The insight
The solution wasn't in stopping or continuing, but in rebuilding. The team decided to redesign the system with fairness as a core principle. No more protected characteristics as input. Regular bias audits. Full transparency about how it works. And human oversight in every decision.
🌍 Why this matters
Multiple municipalities have suspended their fraud systems after criticism. But suspending doesn't solve the underlying problem. The challenge is: how do you build a system that is effective and fair? That requires conscious choices about what data you do and don't use.
The Journey
Step by step to compliance
The wake-up call
A council member asked critical questions about the fraud detection system. How does it work? Who gets checked? The alderman couldn't answer. That was the start of an internal investigation.
The data analysis
The team analyzed three years of control data. The patterns that emerged were uncomfortable: systematic overrepresentation of certain postcodes and backgrounds.
The difficult conversation
The findings were presented to the executive. Reactions ranged from disbelief to shame. Nobody had wanted this — but it had happened.
The legal analysis
What did this mean under the AI Act? The team mapped the obligations. High-risk classification. Social scoring prohibition. FRIA requirement. The conclusion was clear: the current system didn't comply.
The redesign
Instead of throwing away the system, it was rebuilt. Protected characteristics were removed from input. Proxy variables (like postcode) were critically evaluated. The goal shifted from "finding fraudsters" to "checking fairly".
The bias audit
An external party conducted an independent audit. Were the new models fair? Initial results were encouraging — but the team knew: this shouldn't be a one-time check, but an ongoing process.
The Obstacles
What went wrong?
✗ Challenge
The vendor didn't want to fully share how the algorithm worked
✓ Solution
Contractually stipulating that full transparency is a condition for cooperation. If refused: switch to a different solution.
✗ Challenge
Some staff found the extra checkpoints slowing
✓ Solution
Explaining that the municipality had already gotten in trouble before due to lack of oversight. The extra time was an investment in trust.
✗ Challenge
There was resistance to publishing the algorithm ("fraudsters will learn from it")
✓ Solution
Transparency about methodology doesn't have to mean transparency about specific signals. You can explain how the system works without sharing exact thresholds.
We thought we were efficient. We were mainly unfair. Rebuilding our system wasn't just a legal obligation — it was a moral necessity.
The Lessons
What can we learn from this?
Always ask: who is affected?
Algorithms are not neutral. They reflect the choices of their makers and the patterns in their data. For every system, ask: who suffers if this goes wrong?
Historical data contains historical bias
If certain groups were checked more often in the past, an algorithm will continue that pattern. Critical evaluation of training data is essential.
Transparency is not a luxury
Citizens have the right to know how decisions about them are made. Governments have the duty to explain.
Human oversight is not optional
An algorithm may flag, but a human must decide. Especially for decisions that affect fundamental rights.
Does your organization use AI for decisions about citizens?
Discover what obligations the AI Act places on governments and public organizations.
Ga verder met leren
Ontdek gerelateerde content