Human versus GPT-4 in qualitative analysis: A comparative reanalysis of patient interview data following anterior cruciate ligament injury rehabilitation.

OBJECTIVE

The purpose of this study was to prompt GPT-4 to analyze qualitative data used in a published scientific article where qualitative content analysis was performed by human researchers, and to qualitatively compare results from the published article with the results generated by GPT-4.

METHODS

This study was conducted using the full interview dataset from a published qualitative study that aimed to explore experiences of patients treated with rehabilitation alone after an anterior cruciate ligament (ACL) injury. Interview transcripts were analyzed by GPT-4 through iterative prompting to replicate the original six-step content analysis process. Different attempts were conducted to improve GPT-4's output. GPT-4's final output was qualitatively compared with the human-generated results.

RESULTS

While the human-made analysis produced one overarching theme supported by three main categories and nine sub-categories, GPT-4's analysis resulted in four themes, six main categories, and 15 sub-categories. Both analyses captured uncertainty and the impact of knee-related symptoms. GPT-4's results showed a suspiciously equal distribution of codes across sub-categories, and introduced a theme not grounded in the source data. Multiple prompts were required to produce and organize the material.

CONCLUSION

The analysis performed by humans and GPT-4 had similarities and differences. The use of GPT-4 for qualitative analysis in its present form is challenging and needs to be performed across several steps. Currently, GPT-4 should not be used as the only tool in a qualitative analysis of interview data.