Gender-Attributed Persona Prompts and the Diagnostic Accuracy of Proprietary and Open-Weight Large Language Models in Chagas Disease and Visceral Leishmaniasis: A Paired Experimental Study.

Background

Large language models (LLMs) are increasingly considered as adjuncts for differential diagnostic reasoning, yet their sensitivity to gender-attributed cues in the persona prompt-particularly for neglected tropical diseases (NTDs) and in non-English clinical settings-remains poorly characterised.

Objective

The objective of this study was to compare the diagnostic accuracy of one proprietary and three open-weight LLMs for Chagas disease (CD) and visceral leishmaniasis (VL) under paired persona-prompt conditions in which the only manipulated variable was the linguistic gender of the simulated medical persona.

Methods

This experimental, paired study evaluated ChatGPT-4o, LLaMA 3 70B, Meditron-70B, and Mixtral 8x7B across 12 cases per disease (n = 24) from real records at a Brazilian teaching hospital. The primary outcome was top-five diagnostic accuracy. A committee of five infectious-disease specialists assessed the biological plausibility of all differentials. Paired comparisons used Wilcoxon signed-rank tests; 95% confidence intervals were calculated using the Wilson-score method.

Results

ChatGPT-4o achieved the highest accuracy (CD: 100% under both prompts; VL: 83.3-91.7%). LLaMA 3 70B and Mixtral 8x7B showed moderate performance (41.7-83.3%); the medically fine-tuned Meditron-70B exhibited paradoxically poor accuracy (16.7-25.0%) and the lowest committee-rated plausibility scores. A consistent small numerical trend favoured the female prompt across most model-disease combinations (differences of 0-16.7 percentage points), but no comparison reached statistical significance (all p > 0.05).

Conclusions

Gender-attributed persona-prompt variation did not produce a systematic effect on LLM diagnostic accuracy for CD or VL. ChatGPT-4o outperformed the three evaluated open-weight alternatives, and medical-domain fine-tuning did not confer the expected advantage. Expert-validated assessment of hypothesis plausibility should complement target-disease accuracy in clinical LLM evaluation studies, particularly for NTDs.