Differences between human and AI scoring: A meta-analysis of english language assessments.

Despite a burgeoning body of research on using AI scoring systems in English assessments, concerns regarding their reliability persist. To fill this gap, this meta-analysis examined the AI-human scoring differences and the variables moderating these differences by synthesizing the results of 21 empirical studies with a total of 401,698 participants. Results indicate no statistically significant differences between AI and human scoring; the small effect size implies that the average systematic difference between the two was relatively modest. However, extremely high heterogeneity suggests that this overall finding masks considerable variability across study conditions. Moderator analyses reveal that AI-human scoring differences are significantly influenced by factors such as AI system type, number of human raters, agreement index employed, learner proficiency level (CEFR), and publication year. These findings suggest that while AI cannot fully replace human judgment, it can serve as a diagnostic reference tool within broader quality assurance frameworks. When significant discrepancies arise, they warrant investigation of both scoring sources. Based on these findings, this study offers evidence-based recommendations for educators on the effective use of AI scoring systems in language assessments.