Predicting the replicability of social and behavioural science claims in COVID-19 preprints.
СтатьяПоддержка службы здравоохранения СШАПоддержка NIH: внешняя программаНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Marcoci Alexandru, Wilkinson David P, Vercammen Ans, Wintle Bonnie C, Abatayo Anna Lou, Baskin Ernest, Berkman Henk, Buchanan Erin M, Capitán Sara, Capitán Tabaré, Chan Ginny, Cheng Kent Jason G, Coupé Tom, Dryhurst Sarah, Duan Jianhua, Edlund John E, Errington Timothy M, Fedor Anna, Fidler Fiona, Field James G, Fox Nicholas, Fraser Hannah, Freeman Alexandra L J, Hanea Anca, Holzmeister Felix, Hong Sanghyun, Huggins Raquel, Huntington-Klein Nick, Johannesson Magnus, Jones Angela M, Kapoor Hansika, Kerr John, Kline Struhl Melissa, Kołczyńska Marta, Liu Yang, Loomas Zachary, Luis Brianna, Méndez Esteban, Miske Olivia, Mody Fallon, Nast Carolin, Nosek Brian A, Simon Parsons E, Pfeiffer Thomas, Reed W Robert, Roozenbeek Jon, Schlyfestone Alexa R, Schneider Claudia R, Soh Andrew, Song Zhongchen, Tagat Anirudh, Tutor Melba, Tyner Andrew H, Urbanska Karolina, van der Linden Sander
Аннотация
Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.