Background
Efficacy and futility interim monitoring are standard practice in randomized clinical trials comparing new therapy to standards of care. Such monitoring allows a trial to be stopped early for either convincingly positive or negative results, enabling implementation of improved treatment practices in a timely manner. If the positive trends that the new therapy is superior to standard of care are sufficiently strong, early stopping with appropriate regulatory procedures ensures patients can be offered the new therapy as soon as possible. Conversely, futility bounds permit early stopping if the data suggests that it is unlikely for the new therapy to be more efficacious than the standard of care, especially if it is associated with excess toxicity. This safeguards patients from being unnecessarily exposed to inefficacious therapy and allows new promising treatments to be tested in a timely manner.
The decision to stop a trial early, either for efficacy or futility, should be guided by rigorous statistical justification and clinical judgment due to the potential implications to patients, both current and future. Formal statistical boundaries for possibly early termination in interim monitoring of randomized clinical trials should be pre-specified and there are various statistical approaches for doing so [ 1 ]. Interim analyses performed during such monitoring introduce repeated hypothesis testing. Consequently, the overall significance level, alpha, must be adjusted in order to maintain control of the type 1 error across the study; that is, the probability of a false positive result. Most approaches for error rate control are based on group sequential monitoring [ 2 ] and one of the most used futility boundary rules was developed by O’Brien and Fleming [ 3 ] with the Lan-DeMets alpha spending function [ 4 ]. Additionally, Anderson and High have proposed alternative futility boundaries [ 5 ]. For a comparison of futility monitoring guidelines illustrated using data from a completed phase III oncology trial, we refer the reader to Zhang et al. [ 6 ].
Stopping a trial early for efficacy when positive results are observed and the efficacy boundary has been crossed is not without controversy, and there are statistically valid reasons to continue a trial after an efficacy boundary has been passed. Notably, continuing the trial allows information to be collected about endpoint(s) other than the primary endpoint(s) utilized for the interim analysis. However, other concerns such as early stopping resulting in excess false positives are not backed by statistical theory. A review of 27 trials stopped early for positive results highlights the benefit of stopping early for positive results in trials with well-designed interim monitoring plans [ 7 ].
A trial may be stopped for futility when the trial is unlikely to achieve its objectives, either by crossing the statistical futility boundary or due to compelling evidence the trial cannot be completed as planned. Appropriately specifying futility bounds is critical to provide convincing results. Typically, in phase III randomized trials, futility boundaries are specified to control type I and type II error. If a trial is stopped for crossing a futility boundary, it may not be the case that the experimental treatment is ineffective—rather, it may not be as effective as the study design hypothesized, or it may be effective in only a subgroup of patients. Depending on the monitoring guidelines utilized, crossing the futility boundary does not necessarily imply that the trial has little probability of achieving a positive result if it were to continue [ 8 ]. As a cautionary example, the LUME-Lung 2 phase 3 randomized trial comparing nintedanib plus pemetrexed versus placebo plus pemetrexed in non-small-cell lung cancer crossed the futility bound at a pre-planned early interim analysis, yet the final analysis had a significant positive result [ 9 ]. These potential issues highlight the importance of well-designed stopping rules, when appropriate, to avoid such situations. Beyond consideration of statistical futility boundaries, if the trial cannot be completed as planned, there may be little chance the trial will provide quality evidence. This may be due to substantial missing data that would compromise analyses, low accrual making timely achievement of enrollment goals impossible, or low event rates preventing meaningful comparisons [ 10 ]. For a full consideration of futility assessment methods, we refer the reader to Snappin et al. [ 11 ].
Interim results that are negative or absent of early signs of promise can be incredibly discouraging, but futility bounds provide critical guardrails against informal or premature conclusions. Stopping a trial early when the pre-specified futility bound is not crossed risks missing a treatment benefit—and this risk is most acute for therapies with delayed benefits. Here, we provide an example case study of the SWOG S0777 trial where two interim analyses provided minimal, and seemingly diminishing, evidence of treatment benefit, but the final results of the study were positive [ 12 ]. In this manuscript, we review the statistical design of the study and assess the interim findings, as presented to the Data and Safety Monitoring Committee (DSMC) during the conduct of the study, in the context of the primary study outcomes. This provides a tangible illustration of pre-planned interim analyses functioning as intended and a cautionary tale for those tempted to close a study early due to lack of early promise. Such cases are not often reported in the literature, as such trials may not have their non-positive interim analysis results published, or have been closed early.
Patients and methods
The SWOG S0777 study (ClinicalTrials.gov number NCT00644228 ) was a multicenter, open-label, randomized phase III trial comparing bortezomib, lenalidomide and dexamethasone (VRd) with Rd (lenalidomide and dexamethasone) in newly diagnosed multiple myeloma patients [ 12 ]. Major eligibility criteria included age 18 or older, Eastern Cooperative Oncology Group (ECOG) performance status 0–3, presence of CRAB criteria (C = calcium elevation; R = renal impairment; A = anemia; B = bone involvement), and measurable disease (by assessment of free light chains). Full inclusion and exclusion criteria have been described previously [ 12 ]. Patients were randomly assigned 1:1 stratifying by International Staging System stage and intent to transplant. During the recruitment period between April 2008 and February 2012, 525 patients were randomized: 264 to VRd and 261 to Rd.
Study design
The primary objective of S0777 aimed to determine whether the addition of bortezomib during induction therapy (VRd versus Rd) improved progression-free survival (PFS) in newly diagnosed myeloma patients. This study was designed to detect a 50% increase in median PFS at an overall one-sided significance level of 0.025 with 87% power. The design assumed a median PFS of 3 years in the control arm and an accrual rate of 110 eligible patients per year over 4 years. The final analysis was scheduled to be performed when approximately 278 events had occurred, after all patients have been followed for approximately 2.5 years. This trial was monitored by the SWOG DSMC, which meets semiannually coinciding with the SWOG Group meeting.
Pre-planned interim analyses
The S0777 trial followed SWOG’s general guidelines for interim stopping, which focus on a study’s primary endpoint. Other endpoints, such as safety or secondary efficacy endpoints may be informally evaluated. However, given that any multiple hypotheses testing not accounted for in a study’s design leads to a false positive rate much greater than initially targeted, it is important that any unplanned analyses are not formally considered. For large phase III trials, SWOG generally aims to limit the number of formal interim looks, e.g., to two or three, and to employ conservative alpha levels, such as a two-sided 0.0025 for an interim test of efficacy. Furthermore, interim analyses typically address both efficacy and futility, formulated as tests of the null and alternative hypotheses, respectively, thus allowing a study to stop early for either positive or negative therapeutic outcomes.
Two interim analyses were pre-planned for S0777 to evaluate both efficacy and futility as measured by the comparison of PFS between the two treatment arms. The timing of the interim analyses followed a maximum-information approach, wherein analyses were scheduled based on the number of events pooled over the control and experimental arms, at approximately 33% and 66% of the total events expected under the alternative hypothesis. Both interim looks had a two-sided significance level of 0.005, based on Haybittle-Peto Alpha Spending rule (HP). Other methods, such as the O’Brien-Fleming Boundary, would have prescribed a smaller alpha for earlier interim analyses and larger alpha for later interim analyses.
Two hypotheses were specified for each interim analysis: (1) to test the alternative hypothesis of a 50% increase in median PFS with the VRd arm relative to Rd which is equivalent to a hazard ratio (HR) of 0.667 (1/1.5); (2) to test whether the experimental arm with VRd had better PFS than Rd. Both of these hypotheses were tested at the one-sided 0.0025 level.
Results
Baseline characteristics and results of the final analysis have been described previously [ 12 ]. For comparison to the interim analysis results, we summarize the results of the final analysis here where relevant. Both interims occurred after all 525 patients had been accrued and randomized, and analysis eligibility and baseline characteristics across timepoints (interim 1, interim 2, and final) were similar (Supplemental Tables S1–S2). The analyses of the primary and secondary endpoints [ 12 ] demonstrated significant improvement in both PFS and overall survival (OS) for patients randomized to VRd. In the primary endpoint analysis of PFS, the one-sided stratified log-rank p -value (0.0018) fell well below the pre-specified one-sided 0.02 significance level and, notably, did so following two interim analyses that suggested little promise of significant survival benefit. The survival data maturity, measured as median follow-up time calculated by the inverted Kaplan–Meier method, and loss to follow-up at each analysis is summarized in Table 1 . Table 1 Table of median follow-up and patients lost to follow-up at interim 1, 2, and final analysis Timepoint Median follow-up in months Patients lost to follow-up VRd Rd VRd N (%) Rd N (%) Total N Interim 1 11 12 2 (0.83) 3 (1.28) 5 Interim 2 23 23 4 (1.65) 3 (1.28) 7 Final analysis 54 53 6 (2.48) 4 (1.75) 10
Table 1
Table of median follow-up and patients lost to follow-up at interim 1, 2, and final analysis
| Timepoint | Median follow-up in months | Patients lost to follow-up | |||
| VRd | Rd | VRd N (%) | Rd N (%) | Total N | |
| Interim 1 | 11 | 12 | 2 (0.83) | 3 (1.28) | 5 |
| Interim 2 | 23 | 23 | 4 (1.65) | 3 (1.28) | 7 |
| Final analysis | 54 | 53 | 6 (2.48) | 4 (1.75) | 10 |
First interim analysis
The first interim analysis occurred in February 2012 (by which 107 progression events had occurred) due to the timing of these semi-annual DSMC meetings. At that time, the two-sided p -value for the test of Hypothesis 1 was p = 0.73, which indicated that there was not sufficient evidence to rule out the possibility that VRd increases the median PFS by at least 50% over Rd. The test of Hypothesis 2, with an HR of 0.71 (99.5% confidence interval (CI) 0.41, 1.23) in favor of VRd and a two-sided p -value = 0.08, indicated that there was not sufficient evidence to reject the null hypothesis of no treatment benefit at the 0.0025 level. Figure 1 a displays PFS by treatment arm presented to the DSMC. Although OS was not formally specified as an interim analysis endpoint, OS was also analyzed and presented to the DSMC. At the time of the first interim analysis, there had been 19 deaths reported on the VRd arm and 29 on the Rd arm. The HR was 0.65 (99.5% CI 0.28, 1.48; two-sided p -value = 0.14). The test of a 50% improvement in OS with VRd had a two-sided p -value of 0.92, indicating that one could not rule out the possibility that the VRd increases median survival by 50% over Rd. Figure 1 b displays OS by treatment arm presented to the DSMC. Given that no statistical boundaries set for the first interim analysis were crossed, the statisticians recommended, and the DSMC concluded that the trial should continue as planned. Fig. 1 Progression-free and overall survival at the time of the first and second interim and final analyses. A : PFS, first interim; B : OS, first interim; C : PFS, second interim; D : OS, second interim; E : PFS, final; F : OS, final
Second interim analysis
The second interim analysis took place after 177 progression events had occurred, again due to the timing of the semi-annual DSMC meetings, this time in March 2013. Postponing the analysis until at least 184 events had occurred would have delayed the analysis time by about 6 months. Again, the analysis lacked sufficient evidence to conclude either futility (two-sided p -value = 0.27) or efficacy. In fact, the PFS efficacy results (Fig. 1 c) appeared less promising than those of the first interim analysis (HR = 0.79; 99.5% CI: 0.52–1.20; two-sided p -value = 0.11). At the time of the second interim analysis, 34 and 41 deaths had been reported to date on the VRd and Rd arms, respectively. The OS findings (Fig. 1 d) were, again, consistent with the PFS results (HR = 0.80; 99.5% CI: 0.42–1.53; two-sided p -value = 0.32). The test of a 50% improvement with VRd had a two-sided p -value of 0.45. Again, the DSMC recommended that the trial continue as planned.
Final analysis
The primary endpoint analysis occurred in November of 2015, after 303 events had occurred, 137/242 in the VRd and 166/229 in the Rd arm. At this time, the one-sided stratified log-rank p -value (0.0018) fell well below the specified 0.02 significance level, providing convincing evidence that VRd compared to Rd during induction offered significant benefit in terms of PFS outcomes (Fig. 1 e). The stratified HR for PFS was 0.712 (96% Wald CI: 0.560, 0.906). Furthermore, the secondary efficacy analysis of OS (Fig. 1 f) supported the primary results, indicating that VRd also improved survival outcomes as well (HR = 0.709; two-sided log-rank p -value = 0.0250). A total of 76/242 and 100/229 deaths had been observed in the VRd and the Rd arms, respectively.
Retrospective analysis using alternative stopping rules
We also explored alternative O’Brien-Fleming (OB) stopping rules to determine whether we would have stopped the trial early based on those boundaries. Compared to Haybittle-Peto (HP), which applies the same boundary at every interim look, OB may be more stringent at early interim analyses but less stringent at later interim analyses. This raises the possibility that the trial could have been stopped early under OB but not HP. We determined the efficacy and futility boundaries for the same event-based timepoints at which we performed the two interim analyses: approximately 1/3 and 2/3 of the total expected events. The first interim look using an OB boundary for early stopping for efficacy at the first timepoint at 1/3 the total expected events (84 events) would have had a p -value boundary < 0.001) and thus, would not have been reached. The futility p -value boundary would have been 0.87, meaning that the study would not have been stopped due to futility, given that the two-sided p -value testing the null hypothesis was 0.08. At 2/3 the total expected events (169), the OB (one-sided p -value) boundary for early stopping due to efficacy was 0.006, meaning that this efficacy boundary would not have been crossed. Similarly, the futility p -value boundary of 0.805 was also not reached, thus the trial would have continued as planned.
We also compared the probabilities of stopping early for either efficacy or futility using the approach we took (HP) to those of OB. Assuming that VRd was superior to Rd, as the final results indicated, the probabilities for stopping early for efficacy and for futility under the OB model would have been 0.03 and 0.008, respectively, at the time of the first interim analysis, and 0.51 and 0.06, respectively, at the time of the second interim analysis. The cumulative alpha spent at the first and second looks would have been 0.00001 and 0.006, compared to 0.005 and 0.01, as for the HP design implemented.
Discussion and conclusion
Given that both interim analyses provided minimal, and seemingly diminishing, evidence of treatment benefit, one may have been tempted to stop S0777 early due to lack of a significant finding had no formal interim analysis boundaries been specified a priori. Both the planned stopping boundaries as well as retrospective exploration of alternative boundaries were not crossed at the interim analyses. Despite the lackluster interim analysis results, the final results of the study were positive and demonstrated the addition of bortezomib to lenalidomide and dexamethasone results in significantly and clinically improved PFS as well as better OS in newly diagnosed multiple myeloma patients without an intent for immediate autologous stem cell transplant. This contributed to establishing this drug combination as the standard of care [ 13 ]. Limitations of the trial have been discussed in the primary report [ 12 ]; notable limitations include early discontinuation of VRd due to neuropathy and difficulty assessing the prognostic contribution of age, which was not a stratification factor. Had the study been closed early, these positive findings would have remained undetected and the benefits never translated to patients.
Often, nonsignificant results are not reported in the literature, limiting visibility of trials where the accumulating data may have not been promising at interim analysis, yet the final outcome was positive. In this case study of the SWOG S0777 trial, the interim analyses functioned as intended, serving as a safety net preventing patients from being treated longer than necessary on an ineffective regimen and allowing the trial to continue as planned in the absence of early signs of promise. This underscores the critical role of pre-specifying interim analyses as part of a study design, as well as remaining committed to the study design and vigilantly refraining from drawing informal, premature conclusions.
Supplementary Information
Supplementary Material 1: Table S1. Patient enrollment and eligibility at each analysis timepoint. Table S2. Baseline characteristics by randomization group at each analysis timepoint
Abbreviations
Confidence interval
Calcium elevation, Renal dysfunction, Anemia and Bone disease
Data and safety monitoring committee
Eastern Cooperative Oncology Group
Haybittle-Peto
Hazard ratio
Progression-free survival
O’Brien-Fleming
Overall survival
Lenalidomide, and dexamethasone
SWOG Cancer Research Network
Bortezomib, lenalidomide, and dexamethasone
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Authors’ contributions
EG, RS, and AH wrote the manuscript. RS and AH analyzed data. All authors contributed to data interpretation, critically reviewed the manuscript, and approved of the final version.
Funding
Supported by NIH/National Cancer Institute grant U10-CA180819.
Data availability
Information on data requests are available at https://www.swog.org/datarequests .
Declarations
Ethics approval and consent to participate
The SWOG S0777 study was conducted in accordance with the Declaration of Helsinki and other relevant guidelines and regulations. The protocol received institutional review board approval from eachguidelines and regulations. The protocol received institutional review board approval from each participating center. All patients studied provided written informed consent for participation in a SWOG clinical trial.
Consent for publication
All authors have seen and approved the final version and agree to its submission for publication.
Competing interests
The authors declare no competing interests.
References
1. Ciolino JD Kaizer AM Bonner LB Guidance on interim analysis methods in clinical trials J Clin Transl Sci 2023 7 1 e124 10.1017/cts.2023.552 37313374 PMC10260346 Ciolino JD, Kaizer AM, Bonner LB. Guidance on interim analysis methods in clinical trials. J Clin Transl Sci. 2023;7(1): e124. 37313374 10.1017/cts.2023.552 PMC10260346
2. Dignam JJ Bryant J Wieand HS Early stopping of cancer clinical trials Handbook of statistics in clinical oncology 2005 Chapman and Hall/CRC 247 266 Dignam JJ, Bryant J, Wieand HS. Early stopping of cancer clinical trials. In: Handbook of statistics in clinical oncology. Chapman and Hall/CRC; 2005. p. 247–66.
3. O'Brien PC Fleming TR A multiple testing procedure for clinical trials Biometrics 1979 10.2307/2530245 497341 O’Brien PC, Fleming TR. A multiple testing procedure for clinical trials. Biometrics. 1979. 10.2307/2530245. 497341
4. Demets DL Lan KKG Interim analysis: the alpha spending function approach Stat Med 1994 13 1341 52 10.1002/sim.4780131308 7973215 Demets DL, Lan KKG. Interim analysis: the alpha spending function approach. Stat Med. 1994;13:1341–52. 7973215 10.1002/sim.4780131308
5. Anderson JR High R Alternatives to the standard Fleming, Harrington, and O’Brien futility boundary Clin Trials 2011 8 3 270 276 10.1177/1740774511401636 21441286 Anderson JR, High R. Alternatives to the standard Fleming, Harrington, and O’Brien futility boundary. Clin Trials. 2011;8(3):270–6. 21441286 10.1177/1740774511401636
6. Zhang Q Freidlin B Korn EL Halabi S Mandrekar S Dignam JJ Comparison of futility monitoring guidelines using completed phase III oncology trials Clin Trials 2017 14 1 48 58 10.1177/1740774516666502 27590208 PMC5300958 Zhang Q, Freidlin B, Korn EL, Halabi S, Mandrekar S, Dignam JJ. Comparison of futility monitoring guidelines using completed phase III oncology trials. Clin Trials. 2017;14(1):48–58. 27590208 10.1177/1740774516666502 PMC5300958
7. Korn EL Freidlin B Mooney M Stopping or reporting early for positive results in randomized clinical trials: the National Cancer Institute cooperative group experience from 1990 to 2005 J Clin Oncol 1990 27 10 1712 1721 10.1200/JCO.2008.19.5339 PMC2668973 19237631 Korn EL, Freidlin B, Mooney M. Stopping or reporting early for positive results in randomized clinical trials: the National Cancer Institute cooperative group experience from 1990 to 2005. J Clin Oncol. 1990;27(10):1712–21. 10.1200/JCO.2008.19.5339 PMC2668973 19237631
8. Freidlin B Korn EL A comment on futility monitoring Control Clin Trials 2002 23 4 355 366 10.1016/S0197-2456(02)00218-0 12161079 Freidlin B, Korn EL. A comment on futility monitoring. Control Clin Trials. 2002;23(4):355–66. 12161079 10.1016/s0197-2456(02)00218-0
9. Lesaffre E Edelman MJ Hanna NH Park K Thatcher N Willemsen S Statistical controversies in clinical research: futility analyses in oncology–lessons on potential pitfalls from a randomized controlled trial Ann Oncol 2017 28 7 1419 26 10.1093/annonc/mdx042 28184431 Lesaffre E, Edelman MJ, Hanna NH, Park K, Thatcher N, Willemsen S, et al. Statistical controversies in clinical research: futility analyses in oncology–lessons on potential pitfalls from a randomized controlled trial. Ann Oncol. 2017;28(7):1419–26. 28184431 10.1093/annonc/mdx042
10. Ellenberg SS Shaw PA Early termination of clinical trials for futility — considerations for a data and safety monitoring board NEJM Evid 2022 10.1056/EVIDctw2100020 38319261 Ellenberg SS, Shaw PA. Early termination of clinical trials for futility — considerations for a data and safety monitoring board. NEJM Evid. 2022. 10.1056/EVIDctw2100020. 38319261 10.1056/EVIDctw2100020
11. Snapinn S Chen MG Jiang Q Koutsoukos T Assessment of futility in clinical trials Pharm Stat 2006 5 4 273 281 10.1002/pst.216 17128426 Snapinn S, Chen MG, Jiang Q, Koutsoukos T. Assessment of futility in clinical trials. Pharm Stat. 2006;5(4):273–81. 17128426 10.1002/pst.216
12. Durie BG Hoering A Abidi MH Rajkumar SV Epstein J Kahanic SP Bortezomib with lenalidomide and dexamethasone versus lenalidomide and dexamethasone alone in patients with newly diagnosed myeloma without intent for immediate autologous stem-cell transplant (SWOG S0777): a randomised, open-label, phase 3 trial Lancet 2017 389 10068 519 527 10.1016/S0140-6736(16)31594-X 28017406 PMC5546834 Durie BG, Hoering A, Abidi MH, Rajkumar SV, Epstein J, Kahanic SP, et al. Bortezomib with lenalidomide and dexamethasone versus lenalidomide and dexamethasone alone in patients with newly diagnosed myeloma without intent for immediate autologous stem-cell transplant (SWOG S0777): a randomised, open-label, phase 3 trial. Lancet. 2017;389(10068):519–27. 28017406 10.1016/S0140-6736(16)31594-X PMC5546834
13. Mikhael J Ismaila N Cheung MC Costello C Dhodapkar MV Kumar S Treatment of multiple myeloma: ASCO and CCO joint clinical practice guideline J Clin Oncol 2019 37 14 1228 63 10.1200/JCO.18.02096 30932732 Mikhael J, Ismaila N, Cheung MC, Costello C, Dhodapkar MV, Kumar S, et al. Treatment of multiple myeloma: ASCO and CCO joint clinical practice guideline. J Clin Oncol. 2019;37(14):1228–63. 30932732 10.1200/JCO.18.02096