General-purpose large language models outperform specialized clinical AI
A study comparing specialized clinical AI tools (OpenEvidence and UpToDate Expert AI) with frontier LLMs (GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6) found that the general-purpose models outperformed the specialized tools in all evaluations, including medical knowledge tests, clinical alignment, and real-world clinical queries. The clinical AI tools performed similarly to Google Search AI Overview. The findings underscore the need for independent, real-world evaluation before clinical deployment.
Abstract
Specialized clinical artificial intelligence (AI) tools are entering medical practice despite scarce independent evaluation. We quantitatively evaluate two clinical AI tools, OpenEvidence and UpToDate Expert AI, built on large language models (LLMs) against three frontier LLMs: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Our evaluation has three stages: (1) 500 MedQA questions testing medical knowledge, (2) 500 HealthBench items measuring alignment with clinicians and (3) the real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment. For the RCQ benchmark, 12 US clinicians performed randomized, blinded review of model outputs, producing 1,800 model–question annotations. Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ. These findings highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.