Optimizing generative artificial intelligence for clinical summarization: a blinded comparison study of automated versus human care planning synopses.

BACKGROUND

Data, information, and knowledge in health care has expanded exponentially over the past 50 years, leading to significant challenges with information overload and complex, fragmented care plans. Generative artificial intelligence (AI) has the potential to facilitate summarization and integration of knowledge to enable efficient care planning and reduce cognitive load of clinicians.

OBJECTIVE

To determine the value of AI generated summarization through optimization of short synopsis creation at care transitions.

DESIGN

Blinded, randomized comparison study comparing human- vs AI-generated synopses using the data-information-knowledge-wisdom framework.

PARTICIPANTS

De-identified records of 64 patients with multiple chronic conditions from the Medical Information Mart for Intensive Care III database.

MAIN MEASURES: Accuracy, succinctness, synthesis, and usefulness of synopses using a standardized scale with scores > 80% indicating success.

KEY

RESULTS

Artificial intelligence and clinicians summarized 64 patients with 12% overlap. In blinded trials, AI synopses were rated as useful 75% of the time vs 76% for human synopses. AI had lower succinctness ratings for the data synopsis task (55%-67%). For accuracy and synthesis, AI had near equal or better scores in other domains (AI: 72%-79%, humans: 68%-84%), with best scores from AI in wisdom. Interrater agreement was variable.

CONCLUSIONS

Artificial intelligence-created synopses that were nearly equivalent to human-created ones; they were slightly longer and did not always synthesize individual data elements compared to humans. We showed that expert-guided optimization of an open-source large language model can lead to synopses that match those produced by humans, reducing cognitive load.