When governments buy AI, they are not just procuring software. They are making decisions about whose lives get automated, whose risks get miscalculated, and who bears the cost when systems fail. At the Emma Kallina — PostDoc Researcher at the Compliant & Accountable Systems Group across the University of Cambridge and the Research Center Trustworthy Data Science and Security, and Public Interest Tech Lead at AI & Equality made a compelling case: public procurement is not an administrative function. It is a site of democratic power. And right now, that power is largely going unused.
The problem: public sector AI is entering without strategic checks
Drawing on 17 semi-structured interviews with UK and EU procurement experts, public buyers, technology vendors, and policy advisers, Emma painted a picture of a public sector that has become a largely passive consumer of AI. Private sector systems are entering through three main channels: ad hoc tenders with no AI-specific requirements; procurement via framework contracts with large vendors like Microsoft that receive far less scrutiny; and informal routes — AI embedded in system updates, IoT devices, or pilot projects quietly kept running for years without ever going through formal procurement review.
The people for whom AI systems most often fail are the same people most dependent on public services. When procurement is passive, the harms are not equally distributed.
Six practices for procurement as governance
Emma’s research identifies six concrete practices that can shift procurement from passive consumption to active governance of AI in the public interest:
- Provide clarity, vision, and operationalization. Central guidance on what AI is, where it should and shouldn’t be used, and how those principles translate into actual contract language — while leaving room for local adaptation. Right now, even defining AI is contested terrain.
- Share knowledge across the public sector. Build platforms for municipalities and councils to share procurement experiences — including failures. The wheel is being reinvented in every local authority. Collective demand-pooling could also give smaller councils the bargaining power that currently only large vendors possess.
- Build AI literacy in procurement teams. Procurement officers are often technically under-resourced and time-constrained, yet they are the first line of defense against vendor influence. Interdisciplinary teams — combining technical, social science, and justice expertise — are essential.
- Define the problem, not the solution. Tenders that describe a specific AI product (often based on a vendor’s sales pitch) exclude local innovators and lock in big tech. Outcome-focused procurement opens the market and directs purchasing power toward genuine need rather than what is marketed.
- Manage AI as infrastructure, not isolated applications. AI sits within complex, interdependent IT systems. IP generated from citizen data, legacy system lock-in, and data supply chains must be addressed at the contracting stage. Open source reuse and shared system repositories are promising directions.
- Monitor and evaluate after contract award. Responsible AI criteria currently appear in tenders as box-ticking exercises with no weight in procurement decisions and no enforcement after contracts are signed. Staggered payment structures and ongoing auditing — including with affected communities — are needed to hold providers accountable.
Procurement as democratic infrastructure
What makes this research particularly important is the political framing Emma brings to what is often treated as a technical-administrative question. The EU AI Act and growing discussions around digital sovereignty are generating policy energy but procurement is where principles meet practice. And right now, big tech companies are present at every stage of that process: shaping what public sector buyers are aware of, influencing what gets written into tenders, and holding framework contracts that bypass scrutiny entirely.
The comparison raised during the session to gender-responsive procurement is instructive. Both efforts require a shift in institutional culture: procurement officers need to understand why it matters and not just what box to check. Both require governance structures flexible enough for local context while coherent enough to drive systemic change.
Participatory governance is another thread running through Emma’s work: who should be involved in buying AI, and when? Current citizen consultation models are often ineffective, reaching the same engaged voices while excluding the minority communities most impacted by algorithmic systems. This is a research and practice gap that deserves urgent attention.
Steps ahead
Emma and colleagues have translated their research findings into a public-facing website designed to make the six practices accessible to practitioners — not just academics. Ongoing work includes workshops with VNG (the Dutch municipalities union), a practitioner survey on procurement practices including open source, and advisory input into the Council of Europe’s AI procurement guidance — where AI & Equality by Women at the Table now holds official observer status.
Governments are customers. They have purchasing power, legal leverage, and a democratic mandate to shape the technology entering public services. The question is whether they will use it.
Watch the full recording and explore more on womenatthetable.net.
Cover Image: Ulysse Gerkens & FARI / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/