Article 56 · Diagnostic.X · AGI Future Foundation
The Clinician in the Loop: Where AI Assistance Helps, and Where It Must Not Decide
Here is what is true. Diagnostic.X is Series 3 in the April 2026 internal list of 33 named W3 sub-series, and the AGI Future Foundation — the steward of the W3 plan — describes it on its site as AI diagnostics governed by a safety mandate. The Foundation itself is a filed California Nonprofit Public Benefit Corporation, Entity No. B20260196664, with Articles filed on 2026-04-24. In the wider world, AI assistance in medicine is real and growing: the U.S. Food and Drug Administration's public list of authorized AI-enabled medical devices holds 1,614 entries as of its September 2026 update, and exactly one autonomous AI diagnostic system — IDx-DR, now LumineticsCore — has been authorized to make a screening decision without a clinician interpreting the image. The evidence on how clinicians actually use decision-support tools is also real, and it is sobering: alerts are overridden at very high rates, automation bias is well documented, and independent validation has more than once found widely deployed tools performing far below their vendors' claims.

Illustration: a generated image. The loop is the machine's work — fast, tireless, continuous. The gap is the human decision, which no loop can close on its own. The checkmark is not the machine's verdict. It is the clinician's.
What is true, and what is not
That question matters because "human in the loop" is the most invoked and least examined phrase in medical AI. It appears in product pages, in regulatory submissions, in governance documents — including the Foundation's own plans for its series — as if the mere presence of a human somewhere in the workflow settled the question of safety. It does not. A human who is rushed, overloaded with alerts, shown a confident recommendation with no reasoning, and given no real authority to disagree is in the loop in name only. The loop has to be designed: the human needs time, information, authority, and a workload that allows genuine scrutiny. Otherwise the human is not a safeguard. The human is a rubber stamp with a licence.
Three ways a machine can be involved
It helps to be precise about what "involvement" means, because the phrase covers three very different arrangements, and the safety questions are different for each.
Assistive. The machine helps a clinician do something the clinician remains responsible for: highlighting a region on a scan, drafting a note, sorting a worklist so the urgent cases come first, suggesting a differential diagnosis the clinician may or may not accept. The clinician sees the machine's output, judges it, and owns the decision. Almost every AI-enabled device on the FDA's list works this way. The risk here is influence, not abdication: a confident suggestion can anchor a clinician's thinking, and a missed suggestion can narrow it. But the decision, formally and practically, stays human.
Autonomous with human review. The machine makes a decision or a classification, and a human reviews it — sometimes every case, sometimes only the ones the machine flags as uncertain or high-risk. This is the model of many screening programmes, and it is where the design of the review matters most: which cases get reviewed, how much time the reviewer has, what information they see, and whether they can actually change the outcome. A review step that a human cannot meaningfully perform is not a safeguard; it is a ritual.
Fully autonomous. The machine makes the decision and no human reviews it. In medicine, this exists in one authorized form: IDx-DR (now LumineticsCore), authorized by the FDA in April 2018 through the De Novo pathway as the first device to provide a screening decision without the need for a clinician to interpret the image or results. It analyses retinal photographs of people with diabetes and decides whether more than mild diabetic retinopathy is present, in a defined population with explicit exclusions — adults with diabetes who have not previously been diagnosed with diabetic retinopathy, excluding people with eye symptoms, prior laser treatment or surgery, or pregnancy, among others. In its pivotal trial of 900 patients at ten primary care sites, it correctly identified more than mild diabetic retinopathy 87.4% of the time and correctly identified those without it 89.5% of the time, according to the FDA's announcement. The key point is not the percentages. It is the narrowness: one condition, one defined population, one locked algorithm, one prospective pre-registered trial, and a regulator's authorization that names exactly what the device may do. Fully autonomous diagnosis is not a general capability that has arrived. It is a single, tightly bounded exception, and every attempt to generalize from it should be treated with suspicion.
The Foundation's planned model for Diagnostic.X sits firmly in the first category — assistive — and says so explicitly: agents support, humans decide, and a clinician is accountable for every diagnostic output. That is the right placement, and the rest of this article is about what it takes to make that placement real rather than rhetorical.
Where assistance genuinely helps
It would be a mistake to write about the risks of diagnostic AI without acknowledging where it genuinely helps, because the help is real and the clinicians who use these tools can describe it concretely. Four kinds of help recur in the evidence and in practice.
A second reader that does not tire. The dominant use of authorized diagnostic AI is in radiology — 1,230 of the 1,614 devices on the FDA's list, about 76%, sit under the radiology review panel — and the reason is straightforward: image interpretation is high-volume, repetitive, and fatiguing work, and a system that flags a possible finding for a second look can catch what a tired human misses. The value is not that the machine is smarter. It is that the machine is tireless, and that tirelessness complements a human whose attention is finite. A second reader is the oldest form of quality assurance in radiology, and AI is, at its best, a second reader that is always available.
Sorting the queue. In a busy clinic or emergency department, the order in which cases are seen matters as much as how each case is handled. A tool that sorts a worklist — urgent first, routine later — can move the sickest patients to the front of the queue without making any diagnostic claim at all. Triage is a lower-stakes use of AI than diagnosis, because the decision it supports is about priority, not about what is wrong, and because the clinician still examines every patient. It is also the use where the benefit is easiest to measure: time to treatment for the patients who needed it most.
Drafting and retrieval. A large share of clinical work is documentation and information-gathering: summarising a record, retrieving the relevant guideline, drafting a referral letter, checking a dose. AI assistance here is genuinely useful precisely because the stakes of an error are lower and the clinician's review is easier — a wrong draft is visible in a way a wrong diagnosis is not.
Pattern-spotting across data a human cannot hold in mind. A clinician sees one patient at a time. A system can compare that patient's trajectory — their results over months, their pattern against similar patients — and surface a change that no single visit would reveal. This is the closest to the screening-support scope the Foundation describes for Diagnostic.X: flagging change against a person's own usual pattern and referring it to a clinician. It is a modest claim, and its modesty is its strength: it does not ask the machine to know what is wrong. It asks it to notice that something has changed, and to say so to a human who can find out.
None of these uses requires the machine to decide. All of them make the clinician's decision better informed. That is the test a proposed use should pass: does it give the clinician something they did not have — a second look, a better order, a retrieved fact, a noticed change — while leaving the decision, and the accountability for it, exactly where it was?
Automation bias: the machine is usually right, so we stop checking
The central psychological problem of the human in the loop has a name, and it has been studied since long before medical AI existed. Automation bias is the tendency to defer to the outputs of automated systems without applying independent scrutiny. It was first described formally in aviation research in the 1990s, where pilots were found to follow faulty automated guidance even when the correct information was available to them, and it has since been replicated across medicine, law, finance, and knowledge work.
The mechanism is not laziness. It is rational, in a narrow sense: when a system is usually right, checking it every time is expensive in attention, and attention is the scarcest resource in a clinic. A clinician who overrides a correct suggestion ten times will eventually stop overriding, because the eleventh override costs time and the system has been right ten times before. The problem is that the eleventh case is the one where the system is wrong, and the clinician who has learned not to check is the clinician who will miss it. Automation bias is not a failure of character. It is a predictable consequence of a system that is right often enough to earn trust and wrong rarely enough that the trust is never tested — until it is.
The evidence in clinical settings is direct. Studies of clinical decision support systems have found that clinicians accept erroneous machine advice a significant share of the time, even when the error would have been obvious with a few seconds of independent thought. The effect is strongest when the machine's output is confident and unexplained — a bare recommendation, a red flag, a score with no reasoning attached — because a confident output invites deference and an unexplained output gives the clinician nothing to check against.
There is a mirror-image problem, and it matters just as much. Algorithm aversion is the systematic preference for human over machine judgement even when the machine performs comparably or better — and its clinical cousin, anti-automation bias, is the distrust or disdain for computer advice that leads clinicians to ignore or disable alerts. Both extremes are clinically problematic: over-trust misses the machine's errors, and under-trust discards its help. A 2026 survey of 239 registered nurses using AI-supported clinical decision support found the split in practice: 26.8% tended to over-trust the AI, 41.0% showed balanced trust, and 32.2% resisted it — and alert fatigue was the factor most strongly associated with the intention to override alerts. Trust, in other words, is not a property of the machine. It is a property of the workload, the team climate, and the design of the alerts themselves.
A diagnostic support tool should not aim to maximize the clinician's trust. It should aim to calibrate it: to make the clinician trust the machine exactly as much as the machine deserves to be trusted, in this case, on this evidence. That means showing uncertainty in the same size type as the recommendation, and showing the reasoning rather than only the conclusion. A machine that is honest about its own limits is a machine a clinician can use well. A machine that is always confident is a machine that will be obeyed until the day it is wrong.
Alert fatigue: when the loop has too many voices
If automation bias is the problem of trusting the machine too much, alert fatigue is the problem of the machine talking too much. Clinical decision support systems generate alerts — drug interactions, abnormal results, sepsis warnings, reminders — and clinicians override them at rates that should alarm anyone who believes the human in the loop is a safeguard. Reviews report override rates of 73% to 96% for clinical decision support alerts, and some studies have documented override rates above 90% for particular alert categories. One analysis calculated that a hospital had to generate 2,700 alerts to prevent a single adverse drug event. At that ratio, the alerts are not a safety system. They are background noise, and the human in the loop has been trained — by the system itself — to click "dismiss" without looking.
The mechanism is the same as automation bias, run in reverse. When nearly every alert is irrelevant, the rational response is to stop reading them, and the clinician who stops reading them will eventually dismiss the one that mattered. Intensive care clinicians receive a median of 17.5 alerts per day, and more than half report becoming desensitized. Time pressure is the most commonly cited barrier to engaging with decision support, and pop-up alerts that interrupt a doctor mid-task have been shown to increase procedural errors. The system designed to protect patients ends up training doctors to ignore it, and the training is effective precisely because it is learned from experience: the alerts really are mostly noise.
The Epic Sepsis Model, examined in detail in the companion article on evaluation, is the cautionary case. In its independent external validation at Michigan Medicine (Wong et al., JAMA Internal Medicine, 2021), the model generated alerts for 18% of all hospitalized patients while missing 67% of patients with sepsis — the worst of both worlds, a large burden of alerts and a large number of missed cases. The authors' own words describe the consequence: a large burden of alert fatigue. A tool that cries wolf on nearly one in five patients does not only waste time. It teaches the clinicians who use it that its alerts can be ignored, and that lesson generalizes to the next tool, and the one after that.
The remedy is not to exhort clinicians to read every alert. It is to generate fewer, better alerts. The informatics literature has a framework for this, the "Five Rights" of clinical decision support: the right information, to the right person, in the right format, through the right channel, at the right time in the workflow. Each of those five is a design decision, and each is testable: does this alert reach the person who can act on it, in a form they can use, at the moment they can act, and only when action is needed?
There is also a workload question that no amount of interface design can solve on its own. Genuine scrutiny takes time, and time is finite. If a clinician's shift requires reviewing fifty AI outputs, the fiftieth will not receive the scrutiny the first did — not because the clinician is careless, but because scrutiny is a depletable resource. Any honest account of the human in the loop has to include a number: how many AI-assisted decisions is this clinician being asked to review, in how much time? A governance model that does not ask that question assumes infinite attention, and infinite attention is the one resource a clinic never has.
Bias: the machine learns who it has seen
A third problem sits underneath the other two, and it is the one most likely to be missed by a team that has tested its tool carefully on the data it has. A diagnostic AI system learns from the patients it has seen, and if the patients it has seen are not the patients it will serve, its performance will differ — usually worst for the people least represented in its training data.
The landmark demonstration is Obermeyer et al., published in Science in 2019: an algorithm used by Optum, a subsidiary of UnitedHealth Group, to manage the care of approximately 200 million Americans was systematically under-referring Black patients for additional treatment. The algorithm used healthcare costs as a proxy for health needs, and because Black patients historically received less healthcare spending for the same level of illness, the algorithm concluded they were healthier than they were. The bias was in the choice of proxy and the data behind it — not in anyone's intent — and the algorithm was, by its own measures, accurate, while systematically wrong about who needed help.
The lesson generalizes. When developers filter for "clean" data to train their models, they select for patients who happen to have more complete records — who tend to be older, sicker, and disproportionately from certain demographics. When a model is trained on data from one hospital, one scanner, one population, its performance on another population is an open question until it is tested there. And when a tool is evaluated only on its average performance, the average can hide a subgroup where the tool is dangerously wrong. Averages are where bias hides, and subgroup analysis — performance reported separately by age, sex, ethnicity, site, and input quality — is where it is found.
For a proposed series like Diagnostic.X, this is a design requirement that precedes any model: the evaluation plan must name the subgroups in advance, and the decision to deploy must depend on the worst subgroup's performance, not the average. A tool that works well on average and badly for one group of people is not a tool that works well. It is a tool that works well for some people, and the people it fails are the ones least likely to be in the room when the deployment decision is made.
What meaningful oversight actually requires
Pulling these threads together, "human in the loop" turns out to be a demanding specification, not a reassurance. A clinician in the loop is a safeguard only when five conditions hold, and each of them is a property of the system and the workflow, not of the clinician's good intentions.
Time. The clinician must have enough time to genuinely scrutinize the machine's output. That means the workload — the number of AI-assisted decisions per shift, per hour — must be set at a level where scrutiny is possible, and the time must be protected rather than assumed. A review step added to an already overloaded shift is not a safeguard. It is a queue.
Information. The clinician must be shown what they need to judge the machine's output: the reasoning, the uncertainty, the data the machine used, and the cases like this one where the machine has been wrong. A bare recommendation with no reasoning gives the clinician nothing to check, and a clinician with nothing to check will defer. Explainability is not a luxury feature. It is the precondition for oversight.
Authority. The clinician must be able, practically and culturally, to disagree with the machine — and to have that disagreement stick. That means the override must be easy, not buried three clicks deep; it means disagreement must not be punished, formally or informally; and it means the system must record the disagreement rather than silently re-presenting the same recommendation. A human who cannot say no is not in the loop. They are in the audience.
Accountability. Someone must own the decision, and the ownership must be real. In clinical practice, that means the clinician's licence and the institution's responsibility, not the vendor's disclaimer. Courts, boards, and peer-review committees do not accept "the system did not flag it" or "the system told me to" as a defence. The Foundation's planned model gets this right in principle — agents prepare, humans execute, and agents never decide — but the principle has to survive contact with a real workflow, where the pressure to accept the machine's suggestion is constant and the cost of disagreeing is paid in time.
Audit. The loop must be observable after the fact: which outputs were accepted, which were overridden, by whom, and with what outcome. Without an audit trail, there is no way to know whether the human in the loop is functioning or merely present, and no way to learn from the cases where the machine was wrong and the human was right — or the reverse. An audit trail is also the only way to detect the slow drift that matters most: the gradual rise in acceptance rates as trust calcifies, and the gradual fall in scrutiny as fatigue sets in.
Three modes, and what each demands of the human
| Assistive | Autonomous with human review | Fully autonomous | |
|---|---|---|---|
| Who decides | The clinician, with the machine's help | The machine proposes; a human reviews | The machine; no human reviews |
| Example | A second reader flagging a region on a scan | A screening system whose positive cases are reviewed | IDx-DR / LumineticsCore for diabetic retinopathy |
| Main risk | Anchoring: the suggestion narrows the clinician's thinking | Rubber-stamping: the review becomes a ritual | No human to catch the machine's error |
| What the human needs | The reasoning and the uncertainty, visible | Time, authority to disagree, and review of the right cases | Nothing — which is why the authorization must be so narrow |
| How many exist | The great majority of authorized devices | Common in screening programmes | One authorized system, tightly bounded |
| What would justify it | Evidence the clinician decides better with it than without it | Evidence the review catches the machine's errors | A prospective pre-registered trial, a defined population, explicit exclusions |
What Diagnostic.X would have to build
Diagnostic.X, as proposed, has no clinician in any loop, because it has no loop at all yet. The series is Series 3 in the April 2026 internal list of 33 named W3 sub-series, and the Foundation site describes it as AI diagnostics governed by a safety mandate. That description is site copy, not a filing and not evidence of a working system. The verified record is narrow: the name and number, the published direction, and an anchored metaverse world at 5.343, 101.566 whose fabric definition is published but whose assets are not yet hosted, so it does not render fully.
What would have to exist before Diagnostic.X could put a clinician in a loop that meant something is a concrete list, and it follows directly from the five conditions above. A named accountable clinician for every output — not a role, a person, with a licence and a signature. A defined population and a defined first use, so the clinician knows whom the tool serves and whom it does not. A workflow that gives the clinician the machine's reasoning and uncertainty, not only its conclusion, and the time to read them. A workload limit, set in writing, on how many AI-assisted decisions a clinician reviews in a shift. An override that is easy, recorded, and free of penalty. A pre-registered safety threshold, judged by someone other than the team that built the model — the Foundation's proposed Alignment Evaluation Gate, described on its site as a proposed system with no live feed claimed, points in this direction. A monitoring plan with a named human owner and the authority to pause the tool. And an audit trail that records every acceptance and every override, so the loop can be examined after the fact rather than assumed.
None of those exists in a verified document today. Until they do, the correct description of Diagnostic.X is the one this article has used throughout: a proposed series with a stated direction and a standard to meet, not a tool anyone can use, and not a loop anyone sits in.
The governance the Foundation plans for its series is the frame for that list. In the intended model, each series would be DAO-managed day to day — agents operating routine work within limits set by humans — with a human Steward Council deciding everything that carries legal, financial, or clinical consequence. Agents prepare; humans execute. Agents never sign legal documents, never move funds, never hire or fire, never offer securities, and never decide a diagnosis. That model is itself only a plan: no Steward Council is verified as seated for Diagnostic.X, no charter has been signed, and no proposal has moved through the process. But the principle it encodes is the one this article has argued for throughout: the decision belongs to a named human, the machine's job is to make that human's decision better informed, and the loop is only real when the human has the time, the information, the authority, and the audit trail to actually decide.
The Foundation's fundraising, as filed, is donation and grant only. Nothing in this article offers securities, and no price, terms, or return is promised. Readers who want to support the Foundation's work — including the slow, unglamorous work of designing a human loop that is real before any tool is built — can find the Foundation at agifuturefoundation.org. Donations are not claimed to be tax-deductible: no IRS 501(c)(3) determination letter is on file for the Foundation.
Glossary
Human in the loop — An arrangement in which a human reviews, judges, or decides on a machine's output before it takes effect. Real only when the human has time, information, authority, and accountability.
Automation bias — The tendency to defer to the outputs of automated systems without independent scrutiny, first described in aviation research in the 1990s.
Algorithm aversion / anti-automation bias — The opposite tendency: preferring human judgement over machine judgement, or distrusting computer advice, even when the machine performs comparably or better.
Alert fatigue — Desensitization to alerts caused by their volume and low relevance, leading clinicians to override or ignore them — including, eventually, the ones that matter.
Clinical decision support (CDS) — Systems that analyse patient data and present information, reminders, or recommendations to clinicians at the point of care.
Audit trail — A record of which machine outputs were accepted or overridden, by whom, and with what outcome, allowing the loop to be examined after the fact.
Frequently asked questions
No. A human who is rushed, overloaded with alerts, shown a confident recommendation with no reasoning, and given no real authority to disagree is in the loop in name only. Safety requires time, information, authority, accountability, and audit — and a workload that allows genuine scrutiny.
As a second reader that does not tire, as a way of sorting the queue so urgent cases are seen first, as drafting and retrieval support, and as pattern-spotting across data a human cannot hold in mind. All four leave the decision — and the accountability for it — with the clinician.
The tendency to defer to automated systems without independent scrutiny. It is not laziness: when a system is usually right, checking it every time is expensive in attention, and clinicians learn from experience to stop checking — until the case where the system is wrong.
Because most alerts are irrelevant to the patient in front of them. Reviews report override rates of 73% to 96% for clinical decision support alerts, and one analysis calculated that a hospital had to generate 2,700 alerts to prevent a single adverse drug event. At that ratio, the rational response is to stop reading them — which is exactly the danger.
In the United States, yes, as authorized by the FDA: IDx-DR, now LumineticsCore, authorized in April 2018 to provide a screening decision for more than mild diabetic retinopathy without a clinician interpreting the image. It is one condition, one defined population with explicit exclusions, one locked algorithm, and one prospective pre-registered trial. Fully autonomous diagnosis is a single, tightly bounded exception — not a general capability that has arrived.
The clinician and the institution. Courts, boards, and peer-review committees do not accept "the system told me to" as a defence, and a governance model that implies otherwise is misleading its clinicians. That is why the accountable human must be named before any tool is deployed.
References
- U.S. Food and Drug Administration. List of Artificial Intelligence-Enabled Medical Devices. Updated 4 September 2026. FDA.
- U.S. Food and Drug Administration. FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems. Press announcement, 11 April 2018. FDA.
- Wong A, et al. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine. 2021. doi:10.1001/jamainternmed.2021.2626.
- Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447–453.
- Why Nurses Intend to Override AI Alerts. Healthcare. 2026;14:2063. doi:10.3390/healthcare14142063.
- Why Most AI-Powered Clinical Decision Support Systems Fail and What Works Instead. BairesDev. bairesdev.com.
- AGI Future Foundation. Diagnostic.X — AI Diagnostics Governed by a Safety Mandate. agifuturefoundation.org.
Status note: Diagnostic.X (W3 Series 3) is a proposed series in the April 2026 internal list of 33 named W3 sub-series. It is not formed, funded, or offered. The AGI Future Foundation is a California Nonprofit Public Benefit Corporation, Entity No. B20260196664, Articles filed 2026-04-24. No IRS 501(c)(3) determination letter is on file, and donations are not claimed to be tax-deductible. Nothing in this article is medical advice, a diagnosis, or a recommendation about any test or treatment. If you have a health concern, consult a qualified clinician.