The multiplicity problem arises when studies report the results of multiple statistical tests raising the probability that at least some of the results will be found to be statistically significant even if there is no underlying effect. Run enough tests and you will eventually find something, even when there’s nothing to be found.
EMA has explained very nicely that if multiplicity is not properly handled, unsubstantiated claims for the efficacy of a drug may be made as a consequence of an inflated rate of false positive conclusions. For example, if statistical tests are performed on five subgroups, independently of each other and each at a significance level of 2.5% (one-sided directional hypotheses), the chance of finding at least one false positive statistically significant test increases to approximately 12%.
EMA has recently drafted a guideline on multiplicity issue in clinical trials. This guideline replaces the ‘Points to consider on multiplicity issues in clinical trials (CPMP/EWP/908/99)’. Public consultation on this draft guidelines has started on 1 Apr 2017 and final comments shall be sent to EMA by 30 June 2017.
There are multiple ways to address all aspects of multiplicity but EMA has focused on issues that have been found to be of importance in European marketing authorization applications. Main questions answered in this draft guidance are:
- Adjustment of multiplicity – when is it necessary and when is it not?
- How to interpret significance with respect to multiple secondary endpoints and when can a regulatory claim be based on one of these?
- When can confirmatory conclusions be drawn from a subgroup analysis?
- How should one interpret the analysis of ’responders’ in conjunction with the analysis of raw variables and how should composite endpoints be handled statistically with respect to regulatory claims?
- How should multiplicity issues be addressed in estimation?