AI Quality and Clinical Roles Impact Decision Quality in AI-Augmented Medical Decisions More than AI Explanations.

BACKGROUND

Explainable AI has garnered significant attention in recent years, particularly in high-risk domains such as health care. Despite the enthusiasm, empirical evidence on the impact of AI explanations remains mixed. This study explores the effects of AI-generated recommendations and various types of AI explanations on clinical decision quality.

METHODS

We conducted 3 experiments involving clinicians who made drug-dosing decisions using AI recommendations of differing quality and explanations. We evaluated 4 types of explanations: a global explanation about how the AI system worked and 3 case-specific local explanations that respectively included the What (what factors were considered for the case), What+Why (why the factors mattered), and What+Why+How (how the system considered them). We recruited clinicians with varying levels of domain expertise, including pharmacists, physicians, nurses, and midlevel providers.

RESULTS

Our results suggest that compared with AI explanations, AI recommendation quality played a more important role in decision quality. Clinical roles, a proxy for domain knowledge, moderated the effects of AI recommendation quality on decision quality and AI influence. Compared with pharmacists and physicians with greater domain knowledge, midlevel providers and nurses were more susceptible to being influenced by low-quality AI recommendations. Contrary to theoretical predictions, local AI explanations increased trust in the AI system but did not improve decision quality over global explanations. Instead, trust in AI seemed largely unrelated to clinical decision quality.

CONCLUSION

This study underscores the importance of AI recommendation quality in clinical decision making and the need to personalize AI design and AI training to clinicians with different roles and domain knowledge. Our findings also suggest that detailed AI explanations do not necessarily improve AI-augmented medical decisions even though they can increase trust in AI recommendations.