Artificial Intelligence Chatbots' Performance on Dental Trauma Case-based Queries: Examining the Effect of Prompt and Content Engineering.

INTRODUCTION

Traumatic dental injuries (TDIs) require prompt and accurate guidance, yet little is known about how the prompting influences the quality of artificial intelligence (AI) chatbot responses in these situations.

METHODS

Four dental trauma scenarios were developed by expert endodontists. For each scenario, a series of questions was posed to 4 AI chatbots (Claude Sonnet 3.5, Microsoft Copilot, GPT-4, Gemini Pro 2.5) using 2 approaches: unprompted layperson phrasing (n = 10) and endodontist's prompts referencing International Association of Dental Traumatology guidelines (n = 10). Responses were independently evaluated by 2 raters for validity, completeness, and relevance using a 5-point ordinal scale.

RESULTS

Scores (n = 1920) of responses to endodontist questions were associated with significantly higher odds of receiving superior ratings across all 3 domains: validity (odds ratio [OR] = 1.82; 95% confidence interval [CI]: 1.35-2.38; P < .001); completeness (OR = 2.50; 95% CI: 1.85-3.33; P < .001) and relevance (OR = 2.94; 95% CI: 2.04-4.17; P < .001). When a threshold-based acceptability (score ≥4) was applied, responses to endodontist queries were more often acceptable in criterion-based as well as overall analyses (P < .05).

CONCLUSIONS

The quality of AI chatbot guidance in TDIs is significantly associated with how questions are asked. While clinically structured prompts yield more reliable responses, most patients facing dental trauma are unlikely to formulate questions in this way. This gap highlights an important limitation of current AI chatbot applications in TDIs and underscores the need for caution when relying on these tools without professional input.