“Optimism Bias” in Contemporary National Clinical Trial Network Phase III Trials: Are We Improving?

Abstract

Background

Previous studies have found that overestimating treatment effects (i.e., "optimism bias") leads to underpowered clinical trials. The prevalence of "optimism bias" in contemporary National Clinical Trials Network (NCTN) cancer clinical trials is unknown.

Methods

We conducted a systematic review of NCTN phase III randomized trials published from January 2007 to January 2017. We compared the hypothesized vs. observed treatment effects in each trial, and examined whether trial-related factors were correlated with the study results. We also reviewed the methods of each protocol to assess whether a rationale for the hypothesized effect size was provided.

Results

We identified 161 clinical trials, of which 130 were eligible for analysis. Original protocols could not be located for 8 trials (5.0%). Twenty-eight trials (21.5%) observed a statistically significant difference in the primary endpoint favoring the experimental treatment. The median ratio of observed-to-expected HRs among trials that observed a statistically significant effect on the primary endpoint was 1.07 (range: 0.33-1.28) vs 1.32 (range: 0.86-2.02) for trials that did not, compared with 1.34 and 1.86, respectively, for National Cancer Institute (NCI) trials published between 1955 to 2006. An effect size at least as large as the one projected in the protocol trials was observed in 9.8% of trials, compared to 17% of NCI trials published from 1955 to 2006. Most trials (64.6%) provided no rationale to support the magnitude of the proposed treatment effect in the protocol.

Conclusions

Despite a reduction in "optimism bias" compared to previous eras, most contemporary NCTN phase III trials failed to establish statistically significant benefits of new cancer therapies. Better rationalization of proposed effect sizes in research protocols is needed.