Preliminary Evaluation of a Large Language Model-Powered Chatbot for Osteoporosis Self-Management Education: Formative Randomized Controlled Trial.

BACKGROUND

With the increasing burden of chronic diseases, self-management education (SME) is crucial. Traditional SME based on face-to-face delivery by clinicians is resource-intensive, and general digital tools such as web-based platforms often provide limited interactivity for patient learning. Although chatbots based on large language models (LLMs) show promise in interactivity, their real-world effectiveness lacks empirical evidence.

OBJECTIVE

This study aimed to explore the feasibility and preliminary effectiveness of an LLM-based chatbot specifically designed for osteoporosis SME.

METHODS

A formative randomized controlled trial was conducted in a tertiary hospital from February 2024 to March 2025. Adults aged ≥18 years with osteoporosis were recruited and randomly assigned (1:1) to either the intervention (OPBot) group or a control group receiving traditional health education. The chatbot provided interactive educational content and question-and-answer support, while the control group received face-to-face education and written materials. Osteoporosis knowledge was assessed using the Osteoporosis Knowledge Assessment Tool at baseline and discharge. Nurses' time spent on health education was self-recorded during each intervention session and aggregated across sessions. Adherence to disease management was assessed at 1, 3, and 6 months after discharge via telephone using Likert-scale questionnaires. The reliability of OPBot responses was evaluated by 2 clinician assessors using a 5-point Likert scale, with interrater agreement calculated using Cohen κ. Group comparisons were conducted using 2-tailed independent t tests, Mann-Whitney U tests, and chi-square tests, and adherence outcomes were analyzed using mixed-effects models.

RESULTS

A total of 100 participants were randomized; 12% (12/100) were excluded due to loss to follow-up, refusal of the second knowledge assessment, or death, leaving 88% (88/100) participants for analysis (n=45, 51.1% in the OPBot group and n=43, 48.9% in the control group). The OPBot group showed significantly higher postintervention knowledge scores than the control group (median 80.0, IQR 70.0-89.0 vs median 75.0, IQR 65.5-80.0; P=.01). Nurses in the OPBot group spent lesser time on SME than those in the control group (median 5.0, IQR 2.0-17.0 vs median 23.0, IQR 20.0-25.0 minutes; P<.001). For adherence outcomes, a significant group×time interaction was observed for calcium supplement intake (odds ratio 1.49, 95% CI 1.08-2.06; P=.02), indicating differing adherence trajectories over time. The OPBot group also showed higher odds of consuming calcium-rich foods across time points (odds ratio 2.87, 95% CI 1.04-7.89; nominal P=.04), although this association did not remain significant after Holm correction. No significant effects were observed for sun exposure (P=.56), exercise (P=.79), or total adherence scores (P=.33). In the question-and-answer module, most OPBot responses were rated as highly reliable 89.4% (76/85), with high interrater agreement (Cohen κ=0.83).

CONCLUSIONS

LLM-based chatbots specifically designed for osteoporosis SME may improve patient knowledge, supporting adherence behaviors, and reducing healthcare workload. However, further large-scale studies are needed to confirm these findings.