Assessing ChatGPT's capability in understanding and reporting antiretroviral therapy drug-drug interactions (ACCURATE-DDI).

OBJECTIVES

Artificial intelligence tools are transforming access to medication information. However, their ability to accurately identify antiretroviral (ARV) drug-drug interactions (DDIs) remains unclear. This study evaluated ChatGPT's analysis of ARV-related DDIs compared to established HIV-specific DDI resources.

DESIGN

Cross-sectional observational study.

METHODS

Using ChatGPT4o-mini in November 2024, we tested 94 ARV DDI pairs. ChatGPT was asked to classify each interaction as "no interaction," "potential interaction," or "serious interaction." Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated, using the HIV/HCV Drug Therapy Guide and the Liverpool HIV Drug Interactions Checker as the correct classification. A subset of 25 pairs was qualitatively analyzed for responses to the prompt, "Can I take Drug A with Drug B?" using a composite score incorporating severity, mechanism, clinical effects, and management (0 = incorrect, 1 = mixed, 2 = correct). Responses were independently assessed by two HIV-specialized pharmacist/pharmacologist reviewers.

RESULTS

ChatGPT correctly classified just 40.4% (38/94) of DDI pairs; errors were primarily false negatives (34/56 errors; 60.7%). Sensitivity was 46.0%, specificity 29.0%, PPV 56.9%, and NPV 20.9%. The mean composite score was 3.9/8. ChatGPT showed limited accuracy in identifying severity of serious DDIs (0.4/2), despite often correctly identifying mechanism (1.4/2) and clinical effects (1.6/2). Serious DDI management was often incorrect (0.9/2). ChatGPT performed poorly with ritonavir DDIs involving mechanisms other than CYP3A4 inhibition.

CONCLUSION

ChatGPT demonstrated limited accuracy and specificity, often generating responses that combined correct and incorrect information. Currently, ChatGPT should not be considered a reliable tool for ARV DDI information.