2026-05-20 04:00 UTCOriginal source2 min readUpdated: 2026-06-30 13:03 UTC

Prompting language influences diagnostic reasoning and accuracy of large language models

A study evaluating five LLMs on diagnostic reasoning in English versus French found that all but o3 performed significantly better in English, highlighting language as a critical factor for equitable clinical AI deployment.

SourcearXiv Computational LinguisticsAuthor: Adrien Bazoge, Josselin Corvellec, Sofiane Djillali Sid-Ahmed, Pierre-Antoine Gourraud

Article intelligence

InvestorsAdvanced

Key points

Five LLMs tested on 180 clinical vignettes in English and French. Four models showed better performance in English (mean difference 0.37-0.91, adjusted p<0.05).
o3 was the only model with no overall language effect.
Language affected multiple reasoning dimensions: differential diagnosis, logical structure, and internal validity.
Findings underscore the need for multilingual evaluation to ensure fair LLM-based clinical support globally.

Why it matters

This matters because five LLMs tested on 180 clinical vignettes in English and French. Four models showed better performance in English (mean difference 0.37-0.91, adjusted p<0.05).

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

[2605.19173] Prompting language influences diagnostic reasoning and accuracy of large language models

[Submitted on 18 May 2026]

Title:Prompting language influences diagnostic reasoning and accuracy of large language models

View a PDF of the paper titled Prompting language influences diagnostic reasoning and accuracy of large language models, by Adrien Bazoge and 3 other authors

View PDF HTML (experimental)

Abstract:Large language models (LLMs) are increasingly explored for clinical decision support, yet most evaluations are conducted in English, leaving their reliability in other languages uncertain. Here we evaluate the impact of prompting language on diagnostic reasoning and final diagnosis accuracy by comparing English and French performance across five LLMs (o3, DeepSeek-R1, GPT-4-Turbo, Llama-3.1-405B-Instruct, and BioMistral-7B). A total of 180 clinical vignettes covering 16 medical specialties were assessed by two physicians using an 18-point scale evaluating both diagnosis accuracy and reasoning quality. Four of the five models performed better in English (mean difference 0.37-0.91, adjusted p

new | recent | 2026-05

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)