Evaluating the human-artificial intelligence interaction in optical diagnosis of early colorectal carcinoma through endoscopist responses to simulated artificial intelligence: a multinational online study.

BACKGROUND

Suboptimal human-artificial intelligence (AI) interaction is a potential roadblock for implementation of AI in clinical practice. We aimed to evaluate the interaction between AI and endoscopists in the optical diagnosis of colorectal carcinoma (CRC).

METHODS

International endoscopists were invited to diagnose colorectal lesions online. After a pretest of 15 videos, the diagnosis could be adjusted in the subsequent 35 videos after receiving a simulated AI diagnosis. Histology was the gold standard. Endoscopists' diagnoses before and after AI were compared using the signal detection theory. Inappropriate response to AI was defined as decreased diagnostic performance by the endoscopist after AI.

RESULTS

In 2730 diagnoses, the 78 participating endoscopists reached diagnostic accuracies of 71.4 % before and 73.7 % after AI (P < 0.001). Before AI, endoscopists missed only a few true CRCs (mean hit rate 0.86), at the cost of a mean false alarm rate of 0.35. Inappropriate response to AI was observed in 28.2 % of endoscopists. After AI, most endoscopists (41.0 %) shifted toward a more conservative attitude, tending to diagnose a lesion as non-CRC.

CONCLUSIONS

Inappropriate response to AI is an issue in human-AI interaction for the optical diagnosis of CRC. Before using AI, endoscopists should become familiar with their own optical diagnostic performance, and that of the AI system, to possibly improve interaction.