Reliability of accelerated confirmatory trials during the COVID-19 pandemic: the consequences of low pretrial plausibility.

OBJECTIVES: The reliability of confirmatory clinical trials (ie, the post-trial probability) depends on the pretrial probability. During the COVID-19 pandemic, many early COVID-19 confirmatory trials were launched despite limited phase II evidence and weak mechanistic or preclinical support. Whether this accelerated development strategy reduced the reliability of therapeutic evidence remains uncertain.

STUDY DESIGN AND SETTING: Overall empirical pretrial plausibility was estimated at the trial level as the proportion of early randomized trials evaluating interventions that were ultimately considered effective in the reference network meta-analysis. For each trial, counterfactual statistical power was estimated using the reported sample size and the number of events. Bayes' theorem was used to calculate the post-trial probability of efficacy, based on sensitivity (ie, trial power), specificity (ie, 1 minus 0.025), and empirical pretrial plausibility. We then compared the positive predictive values (PPVs) of COVID-19 and non-COVID-19 development pipelines.

RESULTS: Of the 463 early trials evaluating 150 candidate molecules, 51 assessed interventions ultimately supported by the reference network meta-analysis, corresponding to an empirical trial-level pretrial plausibility of 11%. The median power of COVID-19 trials was 9% (min-max: 6%-100%). Accordingly, the probability that a statistically significant result reflected true efficacy was 31% (20%-83%), compared to 97% for non-COVID-19 trials characterized by the conventional phase II to phase III success benchmark of 50%. With an empirical pretrial plausibility of 11%, a replication framework consisting of two independent, 80% powered trials yielded a PPV of 99.2%. This contrasts with a single 90% to 95% powered trial, which yielded a PPV of 81% to 82%.

CONCLUSION: The early COVID-19 trial ecosystem was frequently underpowered and conducted under conditions of low pretrial plausibility, resulting in lower post-trial probabilities of true efficacy than in conventional development pathways. These findings highlight the need for stronger pretrial evidence, coordinated trial prioritization, and replication-oriented strategies in future pandemic responses.

PLAIN LANGUAGE SUMMARY: During the COVID-19 pandemic, hundreds of clinical trials were launched within a very short period to identify effective treatments. Many of these studies started before strong laboratory data, animal studies, or early clinical trial results were available. At the same time, many trials enrolled only small numbers of patients, making them less able to reliably detect true treatment effects. In this study, we examined how these conditions may have affected the reliability of COVID-19 treatment research. Using data from more than 400 randomized trials, we estimated how often a statistically significant result was likely to reflect a truly effective treatment. We found that when treatments are tested with limited supporting evidence and low-powered studies, positive trial results are more likely to be misleading or difficult to reproduce. We also explored alternative research strategies. Our findings suggest that repeating smaller but well-designed independent trials may provide more reliable evidence than relying on a single large trial, especially when the initial evidence supporting a treatment is weak. These results highlight the importance of carefully selecting candidate treatments before launching large confirmatory trials during future pandemics. Coordinated international research strategies, adaptive platform trials, and simplified large-scale studies such as the RECOVERY trial may help produce more reliable evidence while avoiding unnecessary duplication and waste of resources.