Thin evidence behind clearance
Nearly all AI-based medical devices cleared by the U.S. Food and Drug Administration have never been tested to show they improve patient health. A study published August 19 in PLOS Digital Health found that of 1,357 AI-enabled devices cleared for patient care, only three were evaluated on patient-centered outcomes such as mortality, hospital readmissions or quality of life.
Researchers from the University of Toronto and MIT Critical Data conducted a systematic analysis of the FDA device database and the ACR Data Science Institute catalogue, with linked searches of ClinicalTrials.gov and PubMed. They found that only 34 of the 1,357 cleared devices, about 2.5%, were linked to registered clinical trials. Just 12 had results posted, and only three reported patient outcomes.
Radiology dominates the list
Radiology accounts for 1,059 of the cleared devices, and only three of those had a registered trial. Most of the evidence supporting these tools comes from retrospective accuracy studies, which measure how well an algorithm matches a reference standard rather than whether patients actually get better.
The authors also found that 62% of studies used observational designs with small, homogeneous cohorts, limited subgroup analyses and frequent exclusion of vulnerable populations. Negative or inconclusive findings are seldom published, which the researchers said reinforces a literature dominated by algorithmic success stories.
What the researchers want
'We expected the evidence base to be thin, but not this thin,' said Sebastian A. Cajas Ordonez, an AI researcher at MIT Critical Data. 'Out of 1,357 AI devices the FDA has cleared for use in patient care, only three have been tested on whether patients actually live longer or better. Clearance tells you a device resembles something already on the market. It does not tell you it helps anyone.'
The team argues that AI tools must be tested on real patient outcomes before they can be called life-saving. They call for regulatory pathways that require clinical validation, not just technical accuracy, as hundreds of new AI devices enter hospitals each year.