FDA Awards Cognita Imaging Contract to Test LLMs in Radiology AI Evaluation

What Happened

Cognita Imaging has secured a contract from the U.S. Food and Drug Administration to develop and test a method for using large language models to evaluate AI-generated radiology reports. The contract was announced this week as the FDA works to establish a regulatory framework for generative AI-enabled medical devices.

The award places Cognita at the center of a process that will inform how the agency assesses the accuracy and reliability of AI tools used in diagnostic imaging, one of the fastest-growing application areas for artificial intelligence in healthcare.

Background

The FDA has been accelerating its engagement with generative AI in medical contexts as an increasing number of device manufacturers seek clearance for AI-assisted diagnostic tools. Radiology has been a focal point of that effort. AI systems capable of analyzing medical images and generating written reports are already in clinical use in some settings, but the agency has not yet established standardized methods for evaluating the quality or safety of those outputs.

Traditionally, evaluating a radiology report requires a trained radiologist to review it against a clinical standard. That process is time-intensive and difficult to scale as the volume of AI-generated reports grows. The FDA is now exploring whether LLMs, which can parse and assess natural language at scale, could serve as a viable automated evaluation mechanism.

Cognita Imaging is a medical imaging AI company. The firm's work under the contract will focus on testing whether an LLM-based evaluation approach produces assessments that are consistent, accurate, and suitable for regulatory use.

What the Contract Involves

Under the terms of the award, Cognita will test a specific methodology for using large language models to assess AI-generated radiology reports. The goal is to determine whether such an approach can function as a credible evaluation tool that the FDA might incorporate into its review process for generative AI-enabled imaging devices.

The contract reflects a broader FDA effort to develop regulatory infrastructure that keeps pace with the speed at which AI is being deployed in clinical environments. Generative AI-enabled devices occupy a distinct regulatory category because their outputs, such as written reports or clinical summaries, are not static and can vary across uses.

MedTech Dive, which reported the award, described Cognita's method as a potential pathway for the agency to evaluate AI-generated content at a scale that human review alone could not sustain.

Why Radiology AI Regulation Is Complicated

Regulating AI in radiology presents specific challenges. Unlike traditional software used in medical devices, generative AI systems can produce different outputs for similar inputs, making performance harder to characterize through conventional testing protocols. A model trained on one patient population may behave differently when applied to another, and written report outputs can vary in phrasing even when the underlying clinical conclusion is the same.

The FDA has previously issued discussion papers and draft guidance documents on AI and machine learning in medical devices, but formal frameworks specific to generative AI outputs in radiology remain under development. The Cognita contract is one mechanism the agency is using to gather technical data that could inform those frameworks.

The use of LLMs as evaluators, sometimes described in research literature as LLM-as-a-judge approaches, has been studied in general AI development contexts but has limited precedent in regulated medical device evaluation. Cognita's contract will generate data on whether the methodology meets the evidentiary standards the FDA requires.

What Happens Next

Cognita Imaging is expected to conduct its testing and deliver findings to the FDA, which will use the results to inform ongoing regulatory guidance development for generative AI-enabled medical imaging devices.