The statistical properties of RCTs and a proposal for shrinkage

We abstract the concept of a randomized controlled trial as a triple (Formula presented.), where (Formula presented.) is the primary efficacy parameter, b the estimate, and s the standard error ((Formula presented.)). If the parameter (Formula presented.) is either a difference of means, a log odds ratio or a log hazard ratio, then it is reasonable to assume that b is unbiased and normally distributed. This then allows us to estimate the joint distribution of the z-value (Formula presented.) and the signal-to-noise ratio (Formula presented.) from a sample of pairs (Formula presented.). We have collected 23 551 such pairs from the Cochrane database. We note that there are many statistical quantities that depend on (Formula presented.) only through the pair (Formula presented.). We start by determining the estimated distribution of the achieved power. In particular, we estimate the median achieved power to be only 13%. We also consider the exaggeration ratio which is the factor by which the magnitude of (Formula presented.) is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so. We show that our shrinkage estimator successfully addresses the exaggeration. As an example, we re-analyze the ANDROMEDA-SHOCK trial.