BMJ health & care informatics · 2026
How much do language and context affect the accuracy of AI health advice?
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Garg Prashant, Fetzer Thiemo
Аннотация
OBJECTIVES: Large language models (LLMs) are increasingly used for health advice, but evidence on how performance varies across languages, topics and sources remains limited. We assess linguistic and contextual variation in Artificial Intelligence (AI) health-claim verification.
METHODS: We evaluated seven widely accessible LLMs using UK and European Union (EU) health-claim registers and 9088 journalist-vetted PUBHEALTH claims. The register-language analysis used authorised entries translated into 21 languages; status-labelled UK/EU extensions added non-authorised entries for diagnostic checks. PUBHEALTH claims spanned COVID-19, abortion, politics and general health across government, media and other source domains, including nature.com. Models classified claims as supported or unsupported using modal responses across repeated runs.
RESULTS: Authorised-claim sensitivity was highest in English and closely related European languages and declined with syntactic distance from English. In status-labelled UK and EU register extensions, models recognised authorised claims but often overclassified non-authorised claims as supported. PUBHEALTH accuracy was lower and varied by topic and source; inaccuracy was highest for general-health claims and claims linked to nature.com.
DISCUSSION: High authorised-claim sensitivity for English register claims can hide lower sensitivity in other languages and weaker performance in other public-health contexts. Training-data exposure, editorial framing and topic-specific tuning are plausible contributors to these gaps.
CONCLUSIONS: LLM performance in health-claim verification varies substantially by language, topic and source. English-language performance does not reliably generalise across contexts, underscoring the need for multilingual, domain-specific evaluation before public-health deployment.
METHODS: We evaluated seven widely accessible LLMs using UK and European Union (EU) health-claim registers and 9088 journalist-vetted PUBHEALTH claims. The register-language analysis used authorised entries translated into 21 languages; status-labelled UK/EU extensions added non-authorised entries for diagnostic checks. PUBHEALTH claims spanned COVID-19, abortion, politics and general health across government, media and other source domains, including nature.com. Models classified claims as supported or unsupported using modal responses across repeated runs.
RESULTS: Authorised-claim sensitivity was highest in English and closely related European languages and declined with syntactic distance from English. In status-labelled UK and EU register extensions, models recognised authorised claims but often overclassified non-authorised claims as supported. PUBHEALTH accuracy was lower and varied by topic and source; inaccuracy was highest for general-health claims and claims linked to nature.com.
DISCUSSION: High authorised-claim sensitivity for English register claims can hide lower sensitivity in other languages and weaker performance in other public-health contexts. Training-data exposure, editorial framing and topic-specific tuning are plausible contributors to these gaps.
CONCLUSIONS: LLM performance in health-claim verification varies substantially by language, topic and source. English-language performance does not reliably generalise across contexts, underscoring the need for multilingual, domain-specific evaluation before public-health deployment.