Analyzing Nurse Sentiment in a Dutch Regional Newspaper During the COVID-19 Pandemic: Comparative Content Analysis of GPT-5, Gemini 2.5 Pro, and Human Coding.

BACKGROUND

Large language models (LLMs) are increasingly used in qualitative research, but their reliability compared to human analysis, especially on large, non-English datasets, is unclear. Previous studies on older models (like GPT-4) show limitations in nuance and token capacity.

OBJECTIVE

This study compared the qualitative analysis capabilities of OpenAI's GPT-5 and Google's Gemini 2.5 Pro (Gemini) with a human qualitative analysis. The study uses a large dataset of 317 Dutch newspaper articles from January 1, 2020, to December 31, 2023, investigating the sentiment toward nurses during the COVID-19 pandemic.

METHODS

The study used a 2-phase methodology. First, a thematic comparison was conducted where the human researchers, GPT-5, and Gemini independently generated inductive coding trees from the entire corpus. Second, a comparative test was performed where all 3 coders applied a predefined codebook to a 10% stratified random subsample. The human baseline was validated through double-coding by 2 independent researchers, achieving an acceptable intercoder reliability (α=0.703). The AI analysis was iterative, using model-optimized prompts and an article-by-article approach.

RESULTS

Both AI models successfully identified third-order themes (eg, "Health care heroes") consistent with the data. In deductive application, however, both models systematically overcoded compared to the human consensus (181 and 183 codes vs 138), resulting in low intercoder reliability against the human baseline (α=0.487) for GPT-5 and (α=0.507) for Gemini.

CONCLUSIONS

This study suggests a potential divergence in analytical logic. The observed coding frequencies indicate that LLMs may default to semantic presence (literal frequency), whereas human coders appear to prioritize interpretive significance (contextual weight), leading to systematic overcoding. Consequently, this article argues that LLMs should not be viewed as autonomous researchers but as high-sensitivity filtering instruments requiring human calibration. This study concludes that AI can serve as a valuable assistant for qualitative researchers. Still, it benefits from a rigorous, iterative, and human-in-the-loop approach to manage methodological friction and ensure nuanced, valid analysis.