The Patient Who Teaches the Algorithm

When a health AI is deployed in a region facing severe physician shortages, it provides something of genuine value. Diagnostic support. Triage assistance. Health information that can improve outcomes where the alternative is no care at all. That value is real, and it would be dishonest to deny it.

But the deployment is also doing something else, something the patient and the clinician are rarely told about and have no mechanism to control.

In Nairobi, an AI clinical decision-support tool was deployed across 15 clinics, serving 39,849 patient visits. Clinicians using the tool regularly corrected its diagnostic suggestions. Their corrections reduced diagnostic errors by 16 percent, treatment errors by 13 percent, and history-taking errors by 32 percent. Those are meaningful improvements. They happened because trained Kenyan clinicians applied their professional judgment to the AI’s recommendations, catching what the model missed, adding context the model could not see, overriding suggestions that did not fit the patient in front of them.

Every correction taught the model. Every acceptance validated the model’s reasoning. The corrections and the acceptances alike carried the weight of professional judgment, Kenyan clinical expertise applied to Kenyan patients in Kenyan conditions. That accumulated intelligence improved the model not just for Nairobi but globally. The Kenyan health system generated the learning. The platform captured it.

The clinicians were not paid for this contribution. They were not informed that their corrections were being used to improve a commercial product. They had no mechanism to see what the platform learned from their judgment or how that learning was being used elsewhere. They experienced themselves as users of a tool. They were also, simultaneously, unpaid teachers of the system they were using.

Now consider the patients. The people who contribute the richest learning data are often those in the most desperate circumstances. A patient facing a 26-day wait for primary care does not choose between a human physician and an AI. The human physician is not available. The rural clinic has closed. The AI is not one option among many. It is the only option.

That patient enters symptoms with a candor born of having no alternative. She describes pain she might minimize with a doctor she trusts. She discloses history she might withhold in a less urgent setting. Her desperation, the absence of any other source of care, produces the richest possible data for the platform to learn from.

This is the dynamic I have described in earlier work as the desperation algorithm: the system learns most from the people with the fewest choices. The patient’s vulnerability is not a side effect of the deployment. It is the condition that makes the deployment most valuable to the platform.

The platform provides a real service. The clinicians provide real corrections. The patients provide real data. All of this improves the model for everyone, everywhere, permanently. But the flow of value runs in one direction. The service flows from the platform to the user. The learning flows from the user to the platform. The service is visible, temporary, and consumed in the moment. The learning is invisible, permanent, and accumulates into the platform’s most valuable asset.

The Kenyan health system (its clinicians, its patients, its disease patterns, its clinical judgment) became a training ground for a global product. The system paid subscription fees for a tool whose own clinical intelligence was making it more capable. It trained the tool that now charges it more.

Some will argue that this is simply how technology works, users improve products through use, and those improvements benefit everyone. That argument has some merit when the user has genuine alternatives, when the contribution is trivial, and when the value generated is modest. None of those conditions hold in the healthcare scenario described above. The users had no alternative. The contribution, professional clinical judgment correcting a medical AI, is not trivial. And the value generated, a continuously improving global diagnostic model, is not modest. It is the platform’s core commercial asset.

The asymmetry becomes clearer when you compare it to an analogous situation we already govern. A pharmaceutical company conducting a clinical trial in Kenya must obtain informed consent from participants, provide oversight by an ethics review board, report adverse events, and, in many cases, compensate participants for their contributions. The trial participants’ biological responses generated the safety and efficacy data that enabled drug approval. Society recognized that those contributions were valuable and that the contributors deserved protection, not because of charity, but because the combination of vulnerable participants, powerful institutions, and high-stakes outcomes demanded governance.

The same combination is present in AI healthcare deployments. Vulnerable patients. Powerful platforms. High-stakes outcomes. The model architecture is the company’s invention. The clinical learning that makes the model valuable is, in part, the population’s contribution. Both are real. Neither erases the other. The governance framework for recognizing the second (for making the contribution visible, accountable, and subject to negotiation) does not yet exist. Building it is one of the most urgent tasks of our time.

The pharmaceutical governance framework took decades to build: consent protocols, ethics review boards, liability frameworks, compensation standards. The AI equivalent is at the beginning of that construction, not the end. The mechanisms proposed in this series (reverse token accounting, contribution thresholds, inference escrow) are the first steps in building the governance architecture that AI healthcare deployments urgently need.

This article is drawn from Digital Sovereignty in the Cognitive Age, available at blogs.inspire-aspire.net.