I. Three experiments
Take a single chest CT from a patient with metastatic non-small cell lung cancer. Give it to an AI-enabled measurement device cleared five years ago, one that segments the target lesions and returns the sum of their longest diameters. Run it a thousand times. You will get the same number a thousand times. That is what the device was built to do, and it is what FDA was built to evaluate.
Now give the same scan to a generative AI system that reads the images, drafts the radiology report, and characterizes the response. Run it a thousand times. You will get a distribution of reports. Most will agree. Some will phrase the findings differently. A few may disagree on whether the disease has progressed.
Then give the scan to a thousand radiologists.
I know what happens in the third experiment. Radiologists reading the same images disagree in roughly 30% of cases. Not on marginal details. On the response category that determines whether a patient stays on a drug, and whether a drug stays in development.
The question these experiments raise is not which of the three readers is reliable. It is which of the three we have been treating as the standard, and why.
In August, FDA’s Center for Devices and Radiological Health (CDRH) released a discussion paper titled Considerations for the Regulation of Generative AI-Enabled Medical Devices. On its surface, it is a request for feedback on how to evaluate a new class of software. The agency is careful to say that it is not guidance and does not propose policy. Read closely, it is more consequential than that. It is the first document in which the agency concedes that some medical devices will have to be evaluated less like instruments and more like clinicians.
That concession carries a consequence the paper does not state, and one the agency may not have anticipated:
If we are going to judge machines the way we judge clinicians, we will soon have to judge clinicians the way we judge machines.
II. What the paper proposes
The paper starts from a premise the agency first laid out at its Digital Health Advisory Committee meeting in November 2024. Generative AI devices accept open-ended inputs, perform multiple subtasks, and produce variable outputs to similar inputs. Many are built on foundation models developed by third parties who disclose little about their training data or evaluation. Some change after deployment through updates to the model, the prompts, the retrieval strategy, the guardrails, or the orchestration logic. The traditional approach of testing a device across a representative sample of inputs and outputs does not scale to a system whose input space is effectively unbounded.
From there, the paper sketches four ideas.
The first is a two-axis risk framework. One axis describes what the device does, moving from non-directive information to action-directing information, then to actions taken under clinician supervision, and finally to fully autonomous action. The other axis describes the consequence of relying on an incorrect output. A risk score for a future cardiovascular event sits in a different place than an instruction to change a basal insulin dose. Within this framework, the paper makes several notable observations. Directiveness is a continuum, not a binary. A “talk to your doctor” disclaimer does not make an action-directing output less directive. Measurement functions may carry higher risk than their non-directive character suggests, because the user usually cannot check the basis of the output. Patient-facing outputs may warrant different treatment than clinician-facing ones. And conversational devices can drift from informing to directing over the course of a single exchange.
The second is a competency-based approach to premarket evaluation, explicitly inspired by how clinicians are trained and credentialed. It has two parts. Device benchmarking would test the deployed system, not the foundation model alone, across safety, clinical proficiency, generalizability, and, for agentic systems, the ability to plan and act within a defined envelope. Clinical confirmation would then establish performance in real or clinically representative use, through methods ranging from retrospective evaluation and shadow deployment to standardized patient interactions, clinician adjudication, and prospective trials. Performance might be compared against a panel of clinicians, a median clinician in practice, or, where one exists, a conventional reference standard.
The third is greater reliance on the postmarket period. The agency asks whether it should accept more uncertainty at authorization in exchange for periodic re-benchmarking, sampled clinician review, and drift monitoring after deployment, with the premarket benchmark serving as the baseline against which future changes are judged.
The fourth is a voluntary Foundation Model Master File, through which model developers could share architecture, training data provenance, known failure modes, and update commitments with FDA on a confidential basis, for reference by device sponsors.
The paper closes with 26 questions. They are largely good questions. Several are harder than the paper lets on.
III. The wrong word
Much of the public conversation about AI in medicine turns on a single word: deterministic. Vendors market “deterministic AI” to hospitals and payers as a safer category of product. Critics argue that large language models are unsuitable for clinical use because they are not deterministic. Both sides use the word to mean several things at once: identical outputs for identical inputs, reproducible results that can be audited, predictable behavior across situations, or logic written by people rather than learned from data. These properties are related but distinct, and a system can have one without the others.
The FDA paper does not use the word, to its credit. Instead, under a benchmarking element on robustness, reliability, and reproducibility, it breaks variability into components that can actually be tested: consistency across repeated runs, across paraphrases that reflect differences in health literacy or emotional framing, across the order in which information is presented, across missing or contradictory inputs, and across long conversations. It then draws a line that matters more than it appears to. Variation in phrasing may be acceptable. Variation in safety-critical behavior, such as a diagnostic conclusion, is a failure.
Without naming it, the agency has imported the language of psychometrics. What the paper describes is test-retest reliability, or more precisely, intra-rater reliability for a machine. Medicine already has a mature toolkit for this: kappa statistics for categorical agreement, intraclass correlation coefficients for continuous measures, and limits of agreement for measurement methods. We use these tools to evaluate radiologists, pathologists, and the endpoints that drug approvals depend on.
This suggests a simple proposal. Generative AI devices should report their reliability on the same scales we use for human readers, and their safety-critical outputs should be held to prespecified agreement thresholds. That would replace an argument about vocabulary with a measurement.
It would also make visible a principle every psychometrician learns early: reliability is necessary for validity, but not sufficient. A system can return the same wrong answer every time. In some ways that is the most dangerous failure, because consistency reads as confidence. The paper gestures at this when it notes that measurement functions may carry elevated risk because the user cannot evaluate the basis of the output. A tumor measurement that is perfectly reproducible and systematically biased will pass every repeatability test and still mislead every clinician who relies on it.
There is one place where the vocabulary problem could still cause harm. If what makes a device “generative” for regulatory purposes is its variability, a developer may argue that an LLM-based device configured to sample greedily, with structured outputs, is effectively deterministic and should be evaluated like conventional software. That argument should not succeed. Fixing sampling settings removes one source of variation, and not reliably: research from Thinking Machines Lab in 2025 showed that even nominally deterministic inference can vary with how a server batches requests. It does nothing about paraphrase sensitivity, distribution shift, or a foundation model that changes upstream. The question is never whether a device is deterministic. It is whether its clinically consequential behavior is reliable, valid, and stable, and how we know.
IV. The variable reference
The most consequential passage in the paper is a short section on standards of performance. Conventional device evaluation compares an output to a known correct answer. For many generative devices, no single correct answer exists, and many responses may be acceptable. The paper therefore proposes comparing the device to a panel of qualified clinicians whose consensus reflects the standard of care, or to a median clinician in practice.
This is a reasonable choice. It is also the point at which the evaluation of machines becomes an evaluation of medicine.
To set an acceptance criterion against a clinician comparator, you have to know how clinicians perform. Not in aggregate, and not by reputation, but on the same cases, under the same conditions, measured with the same rigor applied to the device. We rarely have that information, and not for lack of spending. Between 2008 and 2018, CMS invested billions in developing quality measures, and only about a third of the 2,266 it produced were ever put to use. CMMI, which receives $10 billion each decade to test new ways of paying for care, spent $7.9 billion operating its models in its first ten years, several of them built around bonus payments for meeting quality and other metrics. They all mostly failed. After billions of dollars and more than a decade of measurement, we still have no systematic way to know whether a clinician made the right call on a given case, or how often two clinicians looking at the same patient would make the same one. That is exactly what a clinician comparator requires, because the panel used to judge a device will be drawn from clinicians who often disagree with one another.
The machine is the first clinical reader that can be run a thousand times on the same case, and that reverses the asymmetry. We will soon have reliability estimates for devices far more precise than anything we have for the clinicians they are compared to. At that point, the comparator becomes the weakest part of the evaluation. Consider a hypothetical device that agrees with a three-person panel 85% of the time. It sounds deficient until we learn that the panelists agree with each other only 78% of the time.
The choice of comparator is therefore not a technical detail. It encodes a value judgment. Benchmarking against the median clinician accepts that half of practicing clinicians perform below the standard. Benchmarking against expert consensus may set a bar most of medicine does not meet. Benchmarking against specialists penalizes devices built to extend specialist knowledge to places where no specialist is available.
This is why the most important question in the paper may be Question 15, which asks whether performance could be assessed relative to the care likely to occur in the absence of the device: unaided clinical judgment, delayed specialist review, or no intervention at all. For a patient in a county without an oncologist, the alternative to a device is not a tumor board. It is a long drive, a long wait, or nothing.
Taken seriously, this has an uncomfortable implication:
The benefit-risk profile of a generative device is not a property of the device alone but of the device in a particular care environment.
The same system could be net beneficial in a rural primary care clinic and net harmful in an academic cancer center, where it displaces better judgment. If so, deployment context belongs in labeling, and benchmark datasets must match the settings where the device will be used. The paper hints at this when it notes that benchmark data should represent deployment contexts. It is worth saying directly: intended use may need to include intended setting.
V. The conversation is the device
Since the Medical Device Amendments of 1976, device regulation has been anchored to intended use. The manufacturer declares what the device is for through its labeling and claims, and FDA evaluates it against that declaration. The concept assumes that the manufacturer’s intent fixes the device’s function. A glucose meter measures glucose. It does not decide, halfway through a reading, to recommend a change in insulin.
A conversational device can. The paper acknowledges this directly. A device may begin by providing general information and migrate, over the course of an exchange, into directing action. It asks how intended use could be characterized when a device’s behavior is emergent across a conversation.
This is a deeper problem than it appears. In a conversation, the user co-authors the function. The same device, with the same labeling, will perform different functions for a patient who asks a general question and for one who keeps pressing until the device tells them what to do. Intended use, in the traditional sense, becomes something closer to intended scope: a boundary the manufacturer tries to hold, not a function the manufacturer defines.
The FDA paper’s treatment of disclaimers points in the same direction. It suggests that whether an output is directive depends on its substance and context, not on whether it uses words like “recommend” or “consider,” and that a closing line telling patients to consult their doctor does not make an instruction less of an instruction. This is a quiet but significant shift. For decades, a great deal of health software has relied on labeling and disclaimers to stay outside device oversight. The paper suggests that the regulated object is no longer the label. It is the utterance, and the sequence of utterances.
The philosopher J.L. Austin argued that some statements do not describe the world but act on it. “I recommend increasing the dose” is not a report. It is an act, with consequences. Generative devices produce such acts at scale, in response to users the manufacturer will never meet.
If that is the regulated object, then intended use must be defined statistically. I would propose thinking of it as an envelope: the set of conversational trajectories the device is authorized to support, together with a measured rate at which realistic trajectories stay inside it. Benchmarking would then include adversarial and naturalistic multi-turn testing designed to estimate that containment rate, with acceptance thresholds tied to the consequence axis. A device that stays in scope 99.9% of the time may be acceptable for general health information and unacceptable for anything that touches dosing. The paper already points toward this with its benchmarking element on scope maintenance, which tests whether cumulative interactions drift out of scope even when each turn appears in scope. The next step is to make the envelope, and its measured integrity, part of the device’s identity.
VI. Which device
The clinician analogy at the center of the paper has a limit that deserves more attention. A physician who passes their boards on Monday is the same physician on Tuesday. She carries her training, her memory, her license, and her liability with her. Her identity is continuous, and our entire system of credentialing depends on that continuity.
A generative device has no such continuity. It is a stack: a foundation model, a fine-tuned layer, a system prompt, a retrieval corpus, a set of guardrails, orchestration logic, and a user interface. Each layer can change. Some changes are deliberate and built in. Some happen continuously by design. Some are made by a foundation model developer the sponsor does not control, on a schedule the sponsor may learn about after the fact. The paper names all three. What it does not quite ask is the underlying question: when every layer can change, what exactly was authorized?
The paper’s answer, implicitly, is behavioral. The premarket benchmark becomes a baseline, and future versions are re-benchmarked against it. I think this is the right instinct, and it can be made more precise by borrowing from a domain FDA already knows well.
Generic drugs are not required to be molecularly identical to the reference product in every respect. They are required to be bioequivalent. The standard is explicit and quantitative: the 90% confidence interval for the ratio of key pharmacokinetic measures, such as peak concentration and total exposure, must fall within 80% to 125% of the reference. Two products that meet that standard are treated as interchangeable for regulatory purposes, even if their manufacturing processes differ.
The same logic could define identity for generative devices. A device would remain the authorized device as long as its behavioral profile, measured on a sequestered benchmark, stays within prespecified behavioral equivalence margins on the dimensions that matter: safety-critical escalation, refusal, diagnostic agreement, measurement bias, and subgroup performance. A change of any kind, at any layer, by any party, would trigger an equivalence test. Within the margins, the change is documented. Outside them, it is a new device, or at least a change that requires review.
This framing has three advantages. It is indifferent to where a change originates, which matters when the change comes from a third-party model update. It gives Predetermined Change Control Plans a quantitative core that does not require anticipating every future modification, which the paper identifies as a central difficulty. And it moves the debate about what counts as a significant change away from architecture, which is hard to reason about, toward behavior, which is what patients actually experience.
It also forces a design choice that is easy to defer. Equivalence margins must be set before anyone knows how the next version will behave, and they must be set by clinical consequence. A 2% shift in phrasing style is irrelevant. A 2% shift in the rate of missed escalations for chest pain is not.
VII. The postmarket bargain
The paper asks whether FDA should accept greater uncertainty at authorization in exchange for stronger monitoring after deployment. It frames the question as new. Drug development has been running this experiment for more than three decades.
Accelerated approval, established in 1992, allows drugs for serious conditions to reach patients on the basis of a surrogate endpoint reasonably likely to predict clinical benefit, on the condition that the sponsor confirm that benefit afterward. It has delivered important therapies years earlier than traditional approval would have. It has also produced a well-documented pattern: confirmatory obligations slipped. A 2022 review by the HHS Office of Inspector General found that more than a third of accelerated approvals had incomplete confirmatory trials, many past their planned completion dates. Indications that failed confirmation sometimes remained on the label for years. In 2021, the agency and its Oncologic Drugs Advisory Committee had to work through a series of these so-called dangling indications publicly, several of them in checkpoint inhibitors. It took legislation, the Food and Drug Omnibus Reform Act of 2022, to give FDA the authority to require confirmatory trials to be underway before approval and to streamline withdrawal.
The lesson was not that the bargain was a mistake. It was that a bargain is only as good as its enforcement, and that enforcement must be designed in at the moment of authorization, when the agency has leverage, not negotiated afterward, when it does not.
CDRH has an opportunity to learn that lesson without repeating it. If generative devices are to be authorized with greater premarket uncertainty, several conditions seem essential. The monitoring program should be operational on the day of deployment, not planned for a later date. The triggers for re-benchmarking, review, or restriction should be prespecified, quantitative, and linked to consequences that follow automatically when thresholds are crossed. Results should be reported publicly at a defined cadence, so that clinicians and health systems can see how a device performs after authorization. And the obligation should attach to the sponsor regardless of where a change originates.
Software also offers something drugs never did. A drug’s postmarket experience reaches the agency slowly and incompletely, through adverse event reports and studies that take years. A deployed generative device can, in principle, be observed continuously. Every interaction can be logged, sampled, and adjudicated. The paper raises the possibility of machine-based supervisory agents to do some of this work. That is plausible, and it leads directly to the next problem: the reliability of the system doing the watching.
VIII. The monoculture
Consider a plausible near-future evaluation. A device is built on a foundation model from one of a handful of major developers. To supplement scarce real-world data for underrepresented subgroups, the sponsor generates synthetic patient cases with a large language model. The device’s open-ended outputs are scored by an LLM adjudicator, because human adjudication of tens of thousands of outputs is expensive. After deployment, a supervisory agent built on another large model samples interactions and flags drift.
Every element of this pipeline is mentioned in the paper. Each is defensible on its own. Together, they may form a closed loop in which models of the same class generate the test, take the test, grade the test, and monitor the results.
The paper sees part of this. Question 13 asks how to keep synthetic data generated by models of the same class from reproducing the very performance gaps the evaluation is meant to detect. The concern generalizes. If the device and its evaluators share training data, architectures, and blind spots, their errors will be correlated. A case the device misunderstands is more likely to be a case the synthetic generator rendered poorly, the adjudicator scored generously, and the monitor failed to flag. The evaluation will report high agreement, and the agreement will be partly an artifact of shared assumptions.
The paper’s answer to bias in evaluation is independence, and it defines independence carefully in organizational terms. Adjudicators should be structurally independent from the sponsor and from the foundation model developer, and that requirement applies even when the adjudicator is itself an LLM. This is necessary but not sufficient. Two organizations can be legally independent and still deploy systems built on the same model. Independence also has to be epistemic.
In practice, that could mean several things. Evaluation layers should draw on different model families where models are used at all. A defined fraction of every evaluation should be anchored to human judgment and to real patient data, not as a formality but as a calibration standard for the automated layers. Sequestered test sets should be held by parties with no stake in the result, which is one of the roles the paper contemplates for independent third parties. And sponsors should be expected to estimate the correlation between device errors and evaluator errors, which is measurable, rather than assume it is zero.
There is a broader point here about concentration. If most generative devices in medicine come to rest on a small number of foundation models, a single upstream change or flaw could propagate across hundreds of products at once. Device regulation has always evaluated products one at a time. A shared dependency of this kind is a systemic risk, and it may eventually require a systemic view.
IX. The active ingredient
The paper’s proposal for voluntary Foundation Model Master Files is sensible, and the agency is candid about its weakness. Question 25 acknowledges that model developers may have limited incentive to disclose safety-relevant information. For the largest developers, medical devices are a small fraction of a very large market. A voluntary program asks them to bear disclosure costs for a modest commercial return.
The software framing may be part of the problem. Device regulation tends to treat third-party software as a component. A more useful analogy comes from the drug side of the agency, where the relationship between a finished product and an ingredient made by someone else has been worked out in considerable detail.
A foundation model is less like a software library and more like an active pharmaceutical ingredient. It determines most of the product’s behavior. It is manufactured upstream by a separate company. And small changes in its production can alter the safety and effectiveness of every product that contains it.
Pharmaceutical regulation handles this through a supply-chain framework. Manufacturers of active ingredients operate under good manufacturing practice expectations, including the ICH Q7 guidance, and are subject to inspection. They can file Drug Master Files that sponsors reference in their applications without seeing proprietary details. And FDA’s regulations require a master file holder to notify each party authorized to reference the file of any addition, change, or deletion. The finished-product sponsor remains responsible for its product, but the upstream supplier carries defined obligations of its own.
Translated to generative AI, that framework suggests what a useful Foundation Model Master File would contain beyond a model card: a commitment to notify referencing sponsors before material changes; version pinning, so that a deployed device can continue to call a fixed model; deprecation timelines long enough for re-benchmarking; and behavioral release notes describing how a new version differs on standardized evaluations, not only that it differs.
None of this requires new legal authority to begin. It can start as procurement. Device manufacturers and health systems, acting collectively, have more leverage over model developers than any one of them has alone. Contract terms covering change notification, version stability, and audit access could become standard in medical deployments well before they become regulatory expectations. That is often how regulatory science matures. The community establishes a practice, and the agency later recognizes it.
X. The thousand and first run
Return to the three experiments. The legacy measurement device gave us the same answer a thousand times, and we called it reliable without always asking whether the answer was right. The generative device gave us a distribution, and we called it a problem. The thousand radiologists also gave us a distribution, and for decades we called it the standard of care.
The discussion paper does not resolve this tension, and it does not claim to. What it does is make the tension unavoidable. Once the agency commits to judging devices against clinicians, the variability of clinicians becomes a regulatory fact, measured and documented in submissions. Once devices are authorized with greater reliance on postmarket evidence, the quality of that evidence becomes a matter of enforcement, not intention. And once a device’s function can be co-authored by its users and altered by its suppliers, the authorized object has to be defined by how it behaves rather than by what it contains.
For those who intend to respond to the agency, and many should, several commitments follow:
Measure reliability on human scales. Report the reliability of generative devices on the same scales used for human readers, and hold safety-critical outputs to prespecified agreement thresholds.
Measure the comparator as rigorously as the device. Make the choice between median clinician, expert consensus, and the counterfactual care pathway an explicit, justified decision tied to the deployment setting.
Define intended use as an envelope, with a measured containment rate across realistic conversations.
Define device identity through behavioral equivalence margins, modeled on bioequivalence and applied to any change at any layer.
Make the postmarket bargain enforceable from the day of authorization, with prespecified triggers and public reporting.
Require epistemic independence in evaluation, not only organizational independence, and estimate the correlation between device and evaluator errors.
Treat foundation models as active ingredients, with change notification, version pinning, and deprecation commitments established first through collective procurement and later through regulation.
There is a final irony worth naming. Medicine spent the last half century trying to make clinicians behave more like precise instruments. We wrote guidelines, built checklists, standardized endpoints, and studied practice variation as a defect to be reduced. Now we are building instruments that behave like clinicians, and the regulator is reaching for the tools we use to credential people.
The two efforts are converging on the same question:
What level of variability is acceptable in a clinical judgment, who gets to decide, and how will we know when it is exceeded?
We have never answered that question for ourselves. And the machines may not let us avoid it much longer.