Evaluating the Agreement Between ChatGPT-4 and Validated Mental Health Scales in Older Adults: A Cross-Sectional Study.

BACKGROUND: The chat generative pretrained transformer (ChatGPT), an artificial intelligence-based web application, has demonstrated substantial potential across various knowledge domains, particularly in medicine. This cross-sectional study assessed the validity and possible usefulness of ChatGPT-4 in assessing mental health by comparing several scales.

METHODS: A cross-sectional study recruited 127 older adults (≥60 years old) from December 2023 to October 2024 in Wuhan. Scenarios contextualized to daily life were adapted using ChatGPT-4 to adapt six scales (PHQ-9, GAD-7, K10, PSS-4, ULS-6, WHO-5). The level of agreement between the ChatGPT-4 adapted scales and the traditional scales was compared using Spearman correlation, Cronbach's alpha coefficient, intraclass correlation coefficient (ICC), and Bland-Altman analysis.

RESULTS: The ChatGPT-adapted questionnaires showed moderate to strong correlations with traditionally validated measures of anxiety, depression, psychological distress, perceived stress, loneliness, and well-being. Significant positive correlations were observed for total scores, including PHQ-9 (rho = 0.61), GAD-7 (rho = 0.66), K10 (rho = 0.75), PSS-4 (rho = 0.71), and ULS-6 (rho = 0.73), with slightly weaker correlation for WHO-5 (rho = 0.41). Reliability analyses yielded Cronbach's α values ranging from 0.66 to 0.91 and ICCs ranging from 0.47 to 0.81, confirming strong internal consistency and test-retest reliability.

CONCLUSIONS: Moderate to high correlations were found between the adapted ChatGPT-4 questionnaire and the traditional scale, indicating that it shows promise as a supplemental mental health assessment tool. However, further research is needed to explore its broader applicability.