From statistical power to statistical assurance: It's time ...

From statistical power to statistical assurance: It's time for a paradigm change in clinical trial design Presented by: Jomy Jose GlaxoSmithKline, Bangalore

1

Contents – Clinical Development Paradigm – Power not powerful – Power is not knowledge – Assurance vs Power – Assurance Simplified – Assuring Success – Assurance in practice – Case study

– Key summary points – References – Acknowledgements ICSA 2018

From statistical power to statistical assurance: It's time for a paradigm change in clinical trial design

2

Clinical Trial Crisis

Clinical trials are crumbling under modern economic and scientific pressures. Nature looks at ways they might be saved Source: NATURE REV. DRUG DISC ICSA 2018


3

Clinical Development Paradigm

ICSA 2018


4

Power not powerful !!!

A protocol might say something like this ... Assuming a clinically relevant difference of 2 points on the primary endpoint scale, with a standard deviation of 6.2, 200 subjects per arm are required to provide 90% power at the 5% alpha level (two-sided).

We are assuming with 100% certainty that the true effect of the drug is 2 points.

The percentage of phase III programs that were unsuccessful due to lack of efficacy is 54% (Hay, et al., 2014) ICSA 2018


5

Power is not Knowledge

•

Knowledge is expressed by the probability distribution describing our belief in drug effect

– Strongly influenced by relevant data – Also influenced by expert and team belief Beliefs about the true (unknown) effect before a trial starts 100

Belief

80 60

Expert Belief

40 Power Calculation Assumption

20 0 0

1

2 3 4 True effect

This histogram compares the assumption made in the power calculation (orange bar) with a more realistic view of the possible treatment effect (green bars) ICSA 2018


6

Assurance vs Power Assurance

Power

Assurance was first introduced into clinical study design by O’Hagan et al. (2005) ICSA 2018


7

Assurance Simplified

True Power effect size

Expert belief on the effect size

Power x Belief

0

2.5%

20%

0.5%

1

36%

30%

10.8%

2

90%

40%

36%

3

99.8%

10%

9.9%

Power = 90% The probability of success assuming the true (unknown and never known) effect of the drug is 2 points

ICSA 2018

BUT

54%

Assurance = 54% The average of the power calculations, weighted by the belief about how big the true effect size is


8

Assuring Success

•

Useful alternative to power but not for sample size calculation

Steps – – – – –

Assume a treatment effect Calculate the sample size for a chosen power and a treatment effect Define a prior distribution for the treatment effect Define the success (it does not need to be based on p-values) Calculate the probability of success for the chosen sample size and a range of sample sizes to see the impact on the assurance when increasing/decreasing the sample size.

ICSA 2018


9

Assurance in practice Using simulations 1. Set the number of iterations (e.g. the number of clinical trials) 2. Sample values for the true treatment difference from the prior 3. Sample values for the endpoint from the sampling distribution 4. Compute the success, e.g. z statistic and p-value 5. Add a flag for the success, e.g. if p-value assurance

ICSA 2018


10

Case study Study background – Phase 2 superiority trial is to be conducted to assess the effect of a new drug in reducing C-reactive protein (CRP) in patients with rheumatoid arthritis. – CRP: marker for disease severity – Endpoint: CFB to week 4 in CRP => negative value indicated improvement – Hypothesis test: – H0: no difference in CFBs between active and control – Two-sided test of superiority at the 5% significance level

ICSA 2018


11

1.Sample Size calculation Normally distributed endpoints

Assumptions 1. Power = 80% 2. True treatment effect = 0.2 (reduction) 3. Variability of the treatment effect to be assumed to be the same in both treatment groups 2 2 = 𝜎𝜎𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 = 0.0605 (population variance of CRP reduction) 𝜎𝜎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎

4. Randomisation allocation 1:1 (active:control) ⇒ 𝑁𝑁𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 = 𝑁𝑁𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 = 25

ICSA 2018


12

2.Assurance Assumptions 1. 2. 3. 4.

Definition of the success at the end of the study =0.2, e.g. significant difference Sample size = 25, e.g. based on the power Prior distribution on the treatment effect ~ N(0.2, 0.246) e.g. from a prior elicitation Variance of the CFB in CRP= 0.0605, e.g. based on other studies True Treatment Difference

ICSA 2018

Assurance

>0

65 %

>= 0.15

61 %

>= 0.2

53 %


13

Key summary points • •

Assurance is related to power but incorporates uncertainty

•

The probability distribution is the most important factor in the assurance computation and it can be constructed from previous data, external opinions or a combination of the two.

Assurance is a valuable tool for decision making and adds value to the overall drug development process.

ICSA 2018


14

References – Hay, M. et al., 2014. Clinical development success rates for investigational drugs.. Nature biotechnology, 32(1), pp. 40-51. – O'Hagan, A., Stevens, J. W. & Campbell., a. M. J., 2005. Assurance in clinical trial design.. Pharmaceutical Statistics, Volume 4, pp. 187-201.

ICSA 2018


15

Acknowledgements – Léa Fortunato,Principal Statistician,RD PCPS,Qsci,GSK – Statistical Innovation Group (SIG),GSK

ICSA 2018


16

Appendix SAS codes /*** Sample size ***/ PROC POWER; TWOSAMPLEMEANS TEST=DIFF ALPHA=0.05 SIDES=2 MEANDIFF=0.2 STDDEV=0.246 NPERGROUP=. POWER=0.80; RUN;

/*****Assurance Calculation********/ data sample; n_actv=25; *** sample size active; n_ctrl=25; *** sample size control; delta0 = 0.2; *** prior mean of delta; delta_var0 = 0.0605; *** prior variance of delta; delta_var_EOS = delta_var0; *** sampling variance; n_simu = 100; *** # of simulations; /*loop for each clinical trial simulated (# of simulations)*/ do i= 1 to n_simu; *** Sample delta (difference actv - ctrl) from the prior distribution Normal(delta0, sd=sqrt(delta_var0))***; delta = rand("normal", delta0, sqrt(delta_var0)); *** rand('normal', mean, sd) ***; *** Calculate SE of the difference ***; se_diff = sqrt(delta_var_EOS/n_actv+delta_var0/n_ctrl); *** Sample (observed) difference from the sampling distribution ***; diff = rand("normal", delta, se_diff); *** Compute the z statistic as needed for assessing the significance; zstat = diff/se_diff; *** Determine the one sided p-value ***; *** (multiple p_ by 2 to get the two sided p-value) ***; pvalue=(1-probnorm(zstat))*2;

ICSA 2018


17

SAS Codes *** Decision flags ***; *** crit1: p < 0.05 (actvie vs control) ***; *** crit2: p < 0.05 AND observed improvement >= 0.15 with active ***; *** crit3: p < 0.05 AND observed improvement >= 0.10 with active ***; if pvalue 0; prob1 = diff>0.15; prob2 = diff>0.20; run; proc freq data=probas; table prob0 prob1 prob2; label prob0='Prior proba that diff >0' prob1='Prior proba that diff >0.15' prob2='Prior proba that diff >0.2';run


18

Thank You

Questions ?

ICSA 2018


20