Is reflective writing a valid summative assessment in the era of GenAI? A scoping review.

INTRODUCTION: Reflective practice is central to medical education at all levels and is commonly assessed using asynchronous written reflections. The emergence of new generative artificial intelligence tools such as ChatGPT has led to concern that they may be used to generate written reflections and so to bypass the reflective process, potentially undermining the value of written reflections as a summative assessment. This scoping review asked whether written reflections remain a valid summative assessment in the age of GenAI by identifying papers which examined the ability of GenAI to write reflectively, the detection of such writing, and the use of GenAI for reflection by health professionals.

METHODS: A scoping review was conducted in accordance with PRISMA-ScR guidelines using. Medline, CINAHL, Education Research Complete and Google Scholar. Data were charted and the descriptive content thematically analysed.

RESULTS: Eighteen publications were included. We find that free GenAI chatbots such as ChatGPT can write reflections to a high academic standard, and these are difficult to detect either by humans or software. We also find that health professionals and students are using GenAI for this purpose, but also that GenAI might be used to support and enhance the reflective process through scaffolding, acting as a conversational partner and assisting in the redesigning of assessments.

DISCUSSION: The findings suggest that asynchronous written reflection is vulnerable as a summative assessment in the presence of GenAI. Future research should seek to answer the question as to how the validity of reflective writing as a summative assessment can be maintained in a world where GenAI use continues to grow.