Logistic Regression

Logistic Regression

Slide 1

Essential Medical Statistics

Applied Logisitic Regression Analysis

Kirkwood and Sterne

Menard

Logistic Regression – 14 Oct 2009

Linear regression: two quantitative variables

Often we have a binary variable, such as affection status.

Slide 2


Slide 3


Slide 4


Slide 5


This can arise from a nonlinear relationship between the probability of the outcome, and a predictor variable A given change in x leads to a smaller change in y when y is closer to 0 or to 1.

Slide 6


This can arise from a nonlinear relationship between the probability of the outcome, and a predictor variable A given change in x leads to a smaller change in y when y is closer to 0 or to 1. So the probability that (y = 1) plotted against x, is going to be a curve, not a straight line This is continuous, but is still bounded within 0 and 1 Could we use a model like this?

PY =abx

Slide 7


Probability and Odds Probability = # of successes / total # of attempts Odds

= # of successes / # of failures = P(success) / P(failure) = P(success)/ 1 P(success)

Odds Ratio

Odds in' exposed ' Odds Ratio= Oddsin ' baseline '

Odds in ' exposed ' =Odds in ' baseline ' ·Odds Ratio

Slide 8


Odds are convenient because they lie between 0 and infinity

OddsY =1=abx PY =1 =abx 1− PY =1 Taking the natural log allows this to vary between –∞ and +∞ :

ln

Slide 9

〚

〛

P Y =1 =abx 1−P Y =1


〚

〛

P Y =1 ln =abx 1−P Y =1

ln 〚 OddsY  〛 =abx OddsY =e abx e abx PY = abx 1e

Slide 10


OddsY =e

abx

a

Odds Y =e · e

bx

e b=Odds Ratio

So if our model is: then,

Slide 11

ln 〚 OddsY  〛 =abx

b=ln Odds Ratio Logistic Regression – 14 Oct 2009

Taking the natural log allows this to vary between –∞ and +∞ :

logit Y =abx We can't use Least Squares! We instead use an iterative process called Maximum Likelihood. The likelihood function will give the probability of the data, as a function of your parameters. What function will depend on what data you're using... You'll end up with maximum likelihood estimates of a and b

Slide 12


logit Y =−8.6023.026 X

An output for the example data:

Call: glm(formula = Affection ~ Predictor, family = binomial(link = logit)) Deviance Residuals: Min 1Q Median -2.2348 -0.7052 0.3778

3Q 0.7525

Max 1.6651

Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -8.602 3.236 -2.658 0.00785 ** Predictor 3.026 1.096 2.762 0.00575 ** --Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 40.381 Residual deviance: 28.741 AIC: 32.741

on 29 on 28

degrees of freedom degrees of freedom

Number of Fisher Scoring iterations: 4

Slide 13


ln Odds Ratio=3.026 Odds Ratio=e3.026=20.6 This is the Odds Ratio for the influence of X on the dependent variable In other words it is the change in logit(Y), for a 1 unit increase in X, but remember that Y and X, or P(Y) and X, do not have linear relationships

Slide 14


Prediction

logit Y =−8.6023.026 X

oddsY =e−8.6023.026 X e−8.6023.026 X PY =1= 1e−8.6023.026 X

The same guidelines about extrapolation apply Slide 15


An output for the example data: Call: glm(formula = Affection ~ Predictor, family = binomial(link = logit)) Deviance Residuals: Min 1Q Median -2.2348 -0.7052 0.3778

3Q 0.7525

Max 1.6651

Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -8.602 3.236 -2.658 0.00785 ** Predictor 3.026 1.096 2.762 0.00575 ** --Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 40.381 Residual deviance: 28.741 AIC: 32.741

on 29 on 28

degrees of freedom degrees of freedom

Number of Fisher Scoring iterations: 4

Slide 16


Wald test

The coefficient divided by its standard error should be distributed as a standard normal variable, and can be tested with:

b W b= ~Normal 0,1 S.E.b We can also calculate the confidence interval:

b±Z 0.975 S.E.b

Slide 17


Likelihood Ratio test

Compares the model fitted, to a model without the parameter of interest, in a chisquare test. For the parameter b: 2

G b=G M −G 0~1 Where GM is the full model chisquare, and G0 is the model chisquare for the model without the term.

Slide 18


Evaluating the model

In linear regression we had the F ratio test, and R2 In logistic regression, computer programs will give you a log likelihood (LL), or sometimes 2LL The LL for the full model, minus the LL for a model with only an intercept can be used to evaluate the significance of your model:

−2 LL full model − LLintercept−only ~2 With degrees of freedom equal to the difference in the number of parameters (e.g. 1). SPSS refers to this as the Likelihood Ratio chisquare test.

Slide 19


Evaluating the model

You may also see: ● AIC (Akaike Information Criterion) ● BIC (Bayesian Information Criterion) ● Deviance These are not great as measures for goodness of fit, and are probably most useful for comparing models (if they are comparable!)

Slide 20


Assumptions ● ● ●

The model is correctly specified Linear relationship between Independent variable and the Logit Zero cells

If you have more than one Independent variables: ● Additivity – modifying factors, interaction ● Multicollinearity – the independent variables shouldn't correlate

Slide 21


Checking Residuals As in logistic regression, we want to make sure: ● ●

Outlier points aren't influencing the model fit Points aren't poorly predicted by the model

Residuals (always standardized) should be Binomially distributed, so we shouldn't worry if they don't look Normal You may see highly influential cases (leverage, or Cook's distance)

Slide 22


Slide 23


Examples

Slide 24


Slide 25


Slide 26
