To Evaluate the Efficacy of Zero-Shot Prompting Using Large Language Models in the Extraction of Microbial Keratitis Descriptors.

PURPOSE: To extract microbial keratitis (MK) descriptors from clinician notes in electronic health records using Large Language Models (LLMs) with a zero-shot prompting approach and compare the descriptors with those identified by expert human annotators.

METHODS

Two hundred fifteen patients with culture-proven MK seen between 2019 and 2023 at Aravind Eye Hospital, Salem, India was gathered. Free-text clinical notes from each patient's first encounter corneal examination were obtained. Each of the 3 MK descriptors-centrality, infiltrate depth, and thinning-was annotated by expert consensus and coded as 1 (present), 0 (absent), or 9 (details unavailable). GPT-4o and GPT-4o mini were prompted to extract the 3 MK descriptors. LLM responses were compared with human annotations using agreement measures such as Cohen Kappa scores at 95% confidence interval, in addition to sensitivity and specificity.

RESULTS

GPT-4o demonstrated mean sensitivity of 92%, 86%, and 97%, for centrality, depth, and thinning, respectively, and 96%, 93%, and 99% mean specificity. Cohen Kappa scores calculated for centrality, depth, and thinning were 0.88, 0.73, and 0.88, respectively, indicating good agreement. The overall sensitivity, specificity, and Cohen Kappa of GPT-4o mini were lower compared with GPT-4o, but the difference was not statistically significant.

CONCLUSIONS

Both GPT-4o and GPT-4o mini showed good agreement with human annotations in extracting MK descriptors. Detection of MK descriptors was influenced by limitations in the quality and consistency of electronic health record documentation. The trade-off between efficiency and performance among LLM architectures remains a consideration for implementation of LLM for large-scale MK data analysis.