Who Governs the Algorithm? AI, Healthcare, and Democratic Self-Determination
In August 2023, Babylon Health, a UK startup that had claimed to cover over 20 million people across 17 countries with its AI-driven symptom checker, filed for bankruptcy in the United States, entered administration in Britain, and began winding down services in Rwanda. About 2.8 million people had their healthcare disrupted overnight.
The company’s pitch had been familiar to anyone tracking technological solutionism in the Global South. There are too few doctors, too many patients, too little infrastructure; and now here was a platform ready to step in where institutions are weak. What followed was not a story of technological failure. It was a story about governance, about who holds power over systems that shape people’s lives, and what happens when nobody is accountable when those systems collapse.
That question is one of the defining political questions of our moment. And in healthcare, most of us have no standing in the answer.
AI is not an actor. It’s a product of actors
The first mistake people make when thinking about AI is treating it as a subject rather than a specific product. When we talk about what “the algorithm decided,” we commit a category error with real consequences. Products are built by people with particular interests, whether commercial, institutional, or political, and those interests shape what the product does and who it serves.
When Babylon began investing in Rwanda, the interesting questions were not technical. They were instead who governed the deployment of this system? Would any data collected be used for training other products? What institutional guarantees existed for the patients whose care depended on this suite of products when the company could no longer meet its obligations to shareholders? Were African patients, in effect, enrolled in large-scale real-world experimentation without the safeguards that wealthier jurisdictions would expect? Questions of data ownership, consent, and accountability are central elements to the Babylon story.
A public good, as healthcare access is, had become contingent on the financial viability of a private firm. When that viability collapsed, governance did not disappear so much as it was revealed that it resided not in any democratic institution but in corporate structures and contractual arrangements that were opaque to the people they served.
The gap between patients and power
Democratic self-determination rests on a foundational bet. This is predicated not on the infallibility of majorities, but on the proposition that ordinary people are the most legitimate authors of their collective fate. No superior class of governors stands above the rest of us with sufficient virtue or knowledge to make decisions on our behalf, least of all decisions that serve their own interests in the process.
That bet is being contested now, on new terrain, by forces with unprecedented resources and reach.
AI systems in healthcare are making consequential moral judgements at scale; about who receives care, on what terms, and through what pathways. When an algorithm determines triage priority, flags a patient for early intervention, or allocates scarce clinical resources, it is not producing a neutral technical output. It is encoding values, trade-offs, and priorities, priorities set by whoever built the system, trained it, and deployed it. And those decisions are being made, predominantly, by a small number of private corporations accountable primarily to their shareholders, operating within regulatory environments they have actively shaped to their advantage.
This creates what we might call a global epistemic asymmetry. Private corporations headquartered in the Global North know more about individuals than individuals can know about the systems governing them in the Global South. And due to global hierarchies between countries, there is little recourse. With accountability out of reach, this is not a mere accident, oversight, or lapse in judgement. It is a feature of systems designed to extract value from data, where knowing more about the people from whom value is extracted is a competitive advantage, and those people’s ability to understand or contest the system is a liability to be managed.
Most patients do not know when an algorithmic product has shaped their care. They often have no right to contest it. They have no meaningful recourse when it fails them. That is not a technical problem. It is a democratic one.
The state that cannot govern cedes governance
The question is not whether to regulate AI products in healthcare. It is whose interests regulation serves. And right now, that question is being decided without most of us in the room.
In most jurisdictions, democratic institutions simply do not yet have the capacity to govern AI at the speed and scale at which it is being deployed. That is not a neutral state of affairs. A state that lacks the capacity to govern the corporation does not remain neutral. It cedes governance to the corporation. And corporations, whatever their stated commitments, are not constituted to serve the public interest. They are constituted to serve their shareholders. The gap between those two things is precisely where democratic politics should live.
The collapse of Babylon Health made that gap visible in its starkest form. But the problem it exposed did not go away when the company did. Every AI system currently operating in a healthcare setting without robust public accountability carries the same structural vulnerability. The people whose care depends on it have no standing to govern it, no mechanism to contest its decisions, and no guarantee that it will be there when they need it.
What democratic governance of healthcare AI would require
The ethical case for changing this follows directly from the democratic premise itself. If AI is reshaping the material conditions of health (and it is) then those conditions are precisely what collective self-determination must address. This is not an abstract principle. It has concrete institutional implications.
Democratic governance of healthcare AI would require, at minimum, that patients and communities have meaningful information about when and how algorithmic systems are shaping their care so that they can make decisions about consent. It would require avenues for contesting algorithmic decisions through institutions with the power to act on those challenges. It would require that public health data used to train AI systems remain under public governance, not private ownership. And it would require that the procurement and deployment of AI in public health settings be subject to the same democratic scrutiny as any other consequential public decision making.
None of this is beyond reach. The same technological capacities currently serving concentrated private interests could, under different governing principles, serve genuinely public ends. Babylon’s rise invited us to imagine a future in which healthcare could be delivered at scale through platforms and algorithms. Its fall raises the prior question about what democratic conditions should such systems be permitted to operate at all.
The bet on ordinary people as the authors of their collective fate remains open. In healthcare, as elsewhere, the question is whether we intend to honour it. And whether we are organised enough, institutionally and politically, to make that intention stick.
