Think-Reflect LLM: A Multi-Method Approach to Using Think Aloud Protocol Analysis and Prompted Reflections to Understand LLM Use.

Publicly available, large language models (LLMs) can enhance citizen management of their health and chronic illnesses, yet there are few research methods that can be used to understand the technology's impact upon cognitive processes such as information seeking, decision-making, problem solving and mental model development. Even less research has examined how these cognitive interactions influence subsequent use of the citizens socio-technical system which includes health services (e.g. regional health authorities), community services that support health (e.g. physiotherapy services, pharmacies), visits to health professionals (e.g. physicians, nurse practitioners or physiotherapists) and interactions with informal caregivers (e.g., family, friends). In this paper we outline a methodology we have developed to assess the impacts of LLMs on the cognitive and sociotechnical activities of their users which represents an integration of Think Aloud Protocols with Prompted Reflections called Think-Reflect LLM (TR-LLM). The methodology is described as well as key considerations for data collection and analysis.