Evaluation of the applicability of AI chatbots in addressing health management issues for living kidney donors.

BACKGROUND

Living kidney donors (LKDs) encounter significant health issues and increasingly use AI chatbots for medical information. However, the effectiveness of these chatbots in providing relevant and accurate health guidance for LKDs remains underexplored.

METHODS

We assessed four chatbots-ChatGPT-3.5, ChatGPT-4, Copilot, and Bard-on their responses to health-related questions for LKDs. Using Google search queries related to kidney donation from the top ten countries in transplants (2015-2019), we generated a list of "Frequently Asked Questions" via the Keyword Magic Tool and literature review. Questions were submitted to the chatbots between November 10 and 12, 2024. Responses were evaluated by six experts on accuracy, integrity, comprehensibility, and operability using a five-point Likert scale.

RESULTS

The "frequently asked questions" developed in this study consisted of five health-related topics comprising a total of 29 items. Among the four AI chatbots, only Copilot failed to respond to 2 out of the 29 questions (6.9%), exhibiting the lowest performance in both accuracy [3.35 (3.18, 3.52)] and integrity [3.40 (3.09, 3.71)]. In contrast, ChatGPT-4.0 achieved the highest median accuracy [4.05 (3.89, 4.21)] and integrity [4.00 (3.82, 4.18)], closely followed by ChatGPT-3.5[4.01 (3.88, 4.16)], [3.98 (3.85, 4.11)]. Overall, all chatbots demonstrated high levels of language comprehensibility and operability.

CONCLUSION

AI chatbots showed potential as supplementary tools for managing the health of LKDs, with ChatGPT-4.0 being the preferred option. However, notable variations in accuracy and integrity underscore the need for effective pre-training, robust oversight, and repeated validation of the information they provide. Relevance to clinical practice (a) The identification of a diverse and detailed "Frequently Asked Questions" list for LKDs. Healthcare providers and support organizations should use this list to develop targeted educational materials and interventions that address the specific health inquiries of LKDs, especially concerning the management of common abnormal symptoms. (b) Enhancing LKD Support. The findings underscore the potential of AI chatbots, particularly ChatGPT-4.0, as valuable supplementary tools for addressing health inquiries among LKDs. Unlike FAQ-based static resources, chatbots provide dynamic, on-demand support for unpredictable health scenarios. Integrating these chatbots into patient education and support systems could enhance information accessibility and empower LKDs in managing their health.