Comparison of With-Replacement and Without ...

COMPARISON of WITHREPLACEMENT and WITHOUTREPLACEMENT VARIANCE ESTIMATES for a COMPLEX SURVEY Frank J. Potter (MPR) Stephen Williams (MPR) Nuria Diaz-Tena (MPR) James Reschovsky (HSC) Elizabeth Schaefer (HSC)

APHA November 2003

Overview z

Introduction

z

Study Objectives and Methods

z

Variance estimation considerations

z

Comparisons for different assumptions

z

Summary

Community Tracking Study (CTS) z

Data on changes in healthcare system – Primary focus on community  Site-level

analysis

– National estimates as byproduct z

Data made available to researchers

CTS Sample Structure Multi-stage Multi-sample Design z

Two independent samples

z

Multi-stage design – 60 PSUs (called sites) – 9 Certainty PSUs

z

Supplemental sample – Stratified random national sample

Multi-stage Sample Design z

60 PSUs / Sites – 12 for intensity study – 48 other sites 

improve national coverage and precision

z

Probability proportional to size

z

Stratified by MSA size and region

z

Without-replacement selection

Survey Data Variance Estimation

z

Two general approaches – Taylor series linearization – Replication methods Software available – SUDAAN (version 8) – Stata (version 8) – SAS (version 8) Surveyregs/Surveymeans – WesVar (version 4) Recommend SUDAAN for CTS

z

WWW.FAS.HARVARD.EDU/~STATS/SURVEY-SOFT

z

z

Why Without Replacement? z

Without-replacement selection of PSUs (sites)

z

Probability proportion to size – Certainty PSUs

z

Small PSU frame – Sizeable finite population correction factor (FPC)

z

FPC → Joint inclusion probabilities

z

Only SUDAAN has capability

COMPARISON of ALTERNATIVES z

Study Measure: Reldiff (%): Reldiff = 100*(SEwr – SEwor) / SEwor

z

SUDAAN used for analysis – SEwor using DESIGN = UNEQWOR – SEwr using DESIGN = WR

Comparison of Variances Using WOR and WR Assumption zMethods

compared

–SUDAAN, Stata, and SAS with-replacement –SUDAAN without-replacement zHousehold

survey

–126 Estimates (samples of 6000-60,000) –Domains: All, Hispanic, low income uninsured zPhysician

survey

–35 Estimates (samples of 4,000-12,000) –Domains: All, high MC revenue, solo, group

Ref Difference of Standard Errors ALL HOUSEHOLDS >50 45

2 4 6

RelDiff

35

16

25

28

15

51

5

15

-5 -15

3 frequency

ALL vs LOW-INCOME HH LOW INCOME HOUSEHOLDS

ALL HOUSEHOLDS

45

2

>50

4 6

R e lD iff

35

16

25

28

15

51

5

15

-5 -15

3 frequency

R e lD iff

>50

45

3

35

3 7

25

34

15

51

5 -5

2

-15

3 frequency

ALL HH vs HISPANIC HH HISPANIC HOUSEHOLDS

ALL HOUSEHOLDS 45

>50

2

35

R e lD iff

45

4

35

6

25

16

15

28

5

51

-5 -15

15

R e lD iff

>50

25 15

2 2 6 12 19

5

29 28

-5 -15

3

-25

-25

frequency

14 frequency

HOUSEHOLD SURVEY SUMMARY

ALL HH

NOT LOW HISP INCOME INSURED

AVERAGE RELDIFF(%)

12

-3

8

6

PERCENT WITH RELDIFF < 0

14

63

20

30

PHYSICIANS: ALL ALL PHYSICIANS >50

2

45

7

RelDiff

35 25

8

15

8 3

5

2

-5 -15

1 frequency

HOUSEHOLDS vs. PHYSICIANS ALL PHYSICIANS

ALL HOUSEHOLDS

45

>50

2 6

R e lD iff

35

28

15

51

5

15

-5 -15

7

35

16

25

2

45

4 R e lD iff

>50

25

8

15

8 3

5

2

-5

3

-15

frequency

1 frequency

ALL PHYS. vs SOLO PRACTICE ALL PHYSICIANS 2

45

7

35

8

R e lD iff

25

8

15

3

5

2

-5

35

1

25

1

R e lD iff

45

-15

SOLO and TWO-PHYSICIAN PRACTICES

7

15

5

5

3

-5

1

7

-15 -25

-25

frequency

1 frequency

PHYSICIAN SURVEY SUMMARY ALL PHYS

HIGH M.C. REVENUE

SMALL PRACTICE

GROUP PRACTICE

PCP

SPECIALIST

AVERAGE Rel Diff (%)

22

18

2

28

12

21

PERCENT WITH Rel Diff < 0

10

3

44

4

15

4

Comparison for Descriptive Statistics Relative Differences in RSEs All

Hispanic

Low Income

Mean

11.8

-2.9

8.4

6.2

%