JAMIA open · 2026
Large language model-based triage to identify antiretroviral therapy adherence barriers and risk levels in patient messages.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Ma Yuanchao, Achiche Sofiane, Lessard David, Engler Kim, Vicente Serge, Tu Gavin, Lemire Benoît, Del Balso Lina, Paisible Nathalie, MARVIN Chatbots Patient Expert Committee, Lebouché Bertrand
Аннотация
OBJECTIVE: This study aimed to develop large language models (LLMs) to automatically identify antiretroviral therapy (ART) adherence barriers and stratify nonadherence risk levels from patient-generated messages.
MATERIALS AND METHODS: With a co-construction committee of people with HIV and providers, 15 480 sentences were annotated for barrier and risk levels. General-domain LLMs (eg, Flan-T5) and clinical foundation models (eg, Clinical-T5) were fine-tuned for multiclass classification and evaluated using Macro-F1. Best-performing models were compared with GPT family and other open-source LLMs. Model fairness, error patterns, and environmental footprints were also assessed.
RESULTS: Flan-T5-xl achieved the best barrier detection (Macro-F1 = 0.83 test/0.71 external), and Flan-T5-large excelled in risk stratification (0.79/0.57). Fine-tuned general-domain LLMs significantly outperformed clinical foundation models (P < .001) on the test split. Error analysis identified both annotation challenges (ambiguous inputs, cross-category overlap, and inconsistencies) and inference limitations (semantic overlap, implicit meanings, and contextual misinterpretation). On external datasets, Flan-T5 models outperformed other LLMs (P < .001), and were more robust to demographic descriptors, while GPT-4.1 and Gemma3 often misclassified inputs as "None." Additionally, GPT-4.1's inference consumed ∼30× more energy than that of Flan-T5-xl.
DISCUSSION: Fine-tuned Flan-T5 models demonstrated strong classification performance, greater robustness to demographic attributes, and lower energy consumption, though challenges remained for subjective and underrepresented categories, reflecting both data imbalance and model limitations in implicit reasoning.
CONCLUSION: LLM-based approaches show promise for real-time ART adherence monitoring, offering a scalable solution to individualized HIV care. Beyond performance, our findings highlight the importance of fairness and environmental sustainability in clinical AI development, with next steps focused on real-world validation through deployment in patient-facing digital tools.
MATERIALS AND METHODS: With a co-construction committee of people with HIV and providers, 15 480 sentences were annotated for barrier and risk levels. General-domain LLMs (eg, Flan-T5) and clinical foundation models (eg, Clinical-T5) were fine-tuned for multiclass classification and evaluated using Macro-F1. Best-performing models were compared with GPT family and other open-source LLMs. Model fairness, error patterns, and environmental footprints were also assessed.
RESULTS: Flan-T5-xl achieved the best barrier detection (Macro-F1 = 0.83 test/0.71 external), and Flan-T5-large excelled in risk stratification (0.79/0.57). Fine-tuned general-domain LLMs significantly outperformed clinical foundation models (P < .001) on the test split. Error analysis identified both annotation challenges (ambiguous inputs, cross-category overlap, and inconsistencies) and inference limitations (semantic overlap, implicit meanings, and contextual misinterpretation). On external datasets, Flan-T5 models outperformed other LLMs (P < .001), and were more robust to demographic descriptors, while GPT-4.1 and Gemma3 often misclassified inputs as "None." Additionally, GPT-4.1's inference consumed ∼30× more energy than that of Flan-T5-xl.
DISCUSSION: Fine-tuned Flan-T5 models demonstrated strong classification performance, greater robustness to demographic attributes, and lower energy consumption, though challenges remained for subjective and underrepresented categories, reflecting both data imbalance and model limitations in implicit reasoning.
CONCLUSION: LLM-based approaches show promise for real-time ART adherence monitoring, offering a scalable solution to individualized HIV care. Beyond performance, our findings highlight the importance of fairness and environmental sustainability in clinical AI development, with next steps focused on real-world validation through deployment in patient-facing digital tools.