Introduction to Survey Methodology - Naval Postgraduate School

Introduction to Survey Methodology Professor Ron Fricker Naval Postgraduate School Monterey, California

1

Goals for this Lecture • Introduce professor and course • Define what we mean by the term “survey” – Characteristics of typical surveys – Distinction between “polls” and “surveys”

• Lay out the basic steps in conducting a rigorous research survey • Discuss various survey modes – Pros and cons – Trade-offs

• Considerations in making mode selection 2

Contact Information • Professor Ron Fricker – Phone • Commercial: 831-656-3048 • DSN: 756-3048 • Fax: 831-656-2595 – E-mail: [email protected]

3

Professional Experience • Academic credentials – Ph.D. and M.A. in Statistics, Yale University – M.S. in Ops Research, The George Washington University – B.S. in Mathematics from the United States Naval Academy

• Teaching credentials – Started teaching post-graduate courses in mid-80s – Have taught at NPS, RAND Graduate School, and USC

• “Real world” credentials – Former active duty naval officer – Senior Statistician specializing in survey and military research at the RAND Corporation

• Can find out more at http://faculty.nps.edu/rdfricke/ 4

Class is All About Collecting and Analyzing Survey Data • In this class you’ll learn: – How to design good survey questions and craft an effective survey instrument – About the trade-offs between various survey modes – The rudiments of sampling for surveys – Some statistical methods useful for analyzing survey data

5

Course Schedule

6

Course Materials • Lecture notes: Binder with all slides • Textbook: Survey Methodology – Also excerpts from Mail and Internet Surveys and Sampling: Design and Analysis

• Software: JMP version 7.0 ¾ All on-line at http://faculty.nps.edu/rdfricke/MCOTEA_Short_Course.htm

– Readings password protected: MCOTEA 7

What is a Survey? • A survey is a: – systematic method for gathering information – from (a sample of) entities – for the purposes of constructing quantitative descriptors – of the attributes of the larger population of which the entities are members

• In many ways, a survey is just a form of data collection… 8

A Special Type of Data Collection • Typically surveys: – Gather information by asking people questions – Collect information by either (1) interviewers asking questions and recording responses or (2) respondents reading and recording their own answers – Collect information from a subset of the population, a sample, rather than from all members

9

Polls vs. Surveys • No clear distinction between the two terms – “Poll” often used for private sector opinion studies • Use many of the same design features as studies that would be called surveys – “Poll” rarely used to describe government or scientific surveys

• To me, the term poll implies either – a commercial or less-scientific study, or – a quick turn-around survey whose results may be of short-term interest 10

Steps in Conducting a Survey (1) • Clearly state research objective(s) • Decide on survey mode(s) – How will you contact potential respondents? • “Contact mode” – web, e-mail, phone, etc. – In what media will the survey be given? • “Survey or response mode” – web, e-mail, phone, etc. – How will you follow up with non-respondents? • “Follow-up mode” – web, e-mail, phone, etc.

• Determine fielding strategy – How to maximize response rates?

• Design the survey questions and the survey instrument 11

Steps in Conducting a Survey (2) • Determine sample size and sampling strategy • Obtain Institutional Review Board (IRB) or other approval as necessary – Are respondents promised confidentiality? – What is the impact if their survey responses become known?

• Pre-test, pre-test, pre-test – Give the survey to some test subjects and get their feedback • What works and what doesn’t? • Are you getting correct data/information? – Revise and re-pre-test as necessary

• Draw sample and field the survey 12

Steps in Conducting a Survey (3) • Follow up with non-respondents • Assemble the survey data – Clean data as necessary

• Analyze the data and report results – Summarize the data • Weight as necessary and appropriate to infer back to population • Calculate and report margin of error – Evidence of bias? • Unit and item non-response

• Communicate survey results to respondents if promised 13

One View of the Steps

14

Another View of the Steps

Excerpted from “What is a Survey?” by Fritz Scheuren

15

Examples of Large-scale Surveys (from your textbook) • • • • • •

National Crime Victimization Survey National Survey on Drug Use and Health Survey of Consumers National Assessment of Educational Progress Behavioral Risk Factor Surveillance System Current Employment Statistics Program

16

National Crime Victimization Survey (NCVS)

17

National Survey on Drug Use and Health (NSDUH)

18

Survey of Consumers (SOC)

19

National Assessment of Educational Progress (NAEP)

20

Behavioral Risk Factor Surveillance System (BRFSS)

21

Current Employment Statistics (CES) Program

22

Examples of DoD Surveys • Annual Active Duty Personnel Survey • Survey of MWR Services and Family Programs • Military Recruiter Quality of Life Survey • Adult Influencer Poll (AIP) • Youth Attitude Tracking Study (now JAMRS Youth Poll)

23

Which Survey Mode Should I Use? •

You are going to conduct a research survey – Goal is to make the best inference possible from sample to population within budget constraints

•

To important questions: – What is the most appropriate method to choose for a particular research question? – What is the impact of choosing a particular method on survey cost and accuracy?

24

Traditional Survey Modes • Mail – Paper questionnaire sent to respondents – Self-administered and mailed back

• Telephone – Interviewers call respondents on telephone – Interviewer-assisted

• In-person – Interviewers go to respondent’s home or office – Interviewer-assisted 25

Options Have Multiplied in the Age of Computers and Telecom

CATI

TDE

IVR/T-ACASI

CAI Methods

CAPI TextCASI

AudioCASI

VideoCASI

26

Mode Considerations • Modes differ by: – – – – –

Degree of interviewer involvement Degree of interaction with respondent Degree of respondent privacy Channels of communication Technology usage

• Considerations: – Cost – Timeliness – Accuracy 27

Broadly speaking… • Interviewer-assisted surveys more costly than selfadministered – Face-to-face most expensive, then telephone, then mail, then web

• Interviewer-assisted surveys less prone to various errors and item nonresponse – Particularly compared to self-administered paper survey – More or less compared to self-administered computerassisted surveys depending on sophistication of program

• Self-administered less subject to sensitivity bias – Can combine some self-administered questions into a predominantly interviewer-assisted mode 28

However, timeliness is not just about… • …the length of time a survey is in the field – Really, it’s from instrument design through analysis • Much of that is not affected by survey mode – Compared to a mail survey, web-based surveys save on two main things: • Mailing time to send out and return surveys • Time required to do data entry, coding, and cleaning for paper surveys • Yet, it can also take a lot of time to code, test, and de-bug very complicated web (more generally, computer-assisted) survey programs 29

And, cost is not just about… • …manhours – On large mail surveys, printing and postage costs can be significant • 1,500 surveys x 4 mailings at $1.50 each = $9,000! • 1,500 $1 incentives = $1,500 – On telephone surveys must consider fixed costs of phone and CATI equipment + variable costs – On face-to-face interviews, travel costs can be significant (not to mention interviewer time)

• But in terms of manhours, don’t forget: (1) instrument design time; (2) in non-electronic modes, data entry and coding time; (3) non-response follow-up efforts; and (4) analysis time

30

And accuracy is not just about… • … response rates and missing items – Total survey error: what mode or modes will most decrease all types of error?

• Lots of considerations – Frame coverage – Nonresponse rates – Sensitivity bias, etc… 31

32

Cost and Frame Availability Often Drive Mode Choice • Survey budgets never unlimited – Frequently cost will eliminate one or more modes • And likely drive other survey design decisions – I.e., face-to-face interviews too expensive

• On the other hand, surveys using area probability frames usually have to start out using face-to-face interviews • Similarly, if a complete e-mail frame is available, web survey is possible option – But general population surveys eliminate web… 33

Some Modes Logically Group (1) • Telephone and face-to-face surveys – Telephone often considered an alternative to faceto-face – Both interviewer-assisted and can have similar frame coverage issues – Many longitudinal surveys start out with a face-toface interview with telephone follow-ups

• Mail sometimes an alternative to telephone – If both telephone and addresses available

34

Some Modes Logically Group (2) • Web surveys often considered a replacement for mail surveys – Coverage is still an issue for web surveys – Web superior in terms of automation and speed – However, literature consistently finds that nonresponse rates are higher for web surveys

35

Using Multiple Modes • Reasons to use multiple modes: – Maximize response rates • Reduce/eliminate selection & coverage bias • Allow respondents to use most convenient mode – Reduce costs • Cross-sectional surveys: start with least costly mode and move to more expensive modes as necessary to get response • Longitudinal surveys: Use less expensive modes in follow-on survey rounds 36

Considerations When Using Multiple Modes • Focus should be on designing instruments to ensure equivalency across modes – Rather than optimizing for a particular mode – E.g., make the follow-on web survey resemble its counterpart paper survey

• Ensure operation is set up to track cases across the modes – Must avoid duplications and uncoordinated respondent contact

• Must design and field so that mode effects can be disentangled from other sample characteristics – May have to randomize survey mode over a subset 37

Other Data Collection Methods • Surveys are not the only (nor necessarily the best) way to collect data • Other methods include – Administrative records – Focus groups and qualitative investigations – Randomized experiments

• Which is “best” depends on the research question(s) and/or the purpose for which the data will be used 38

What We Have Covered • Defined the term “survey,” including the – Characteristics of types of surveys this class will focus on – Distinction between “polls” and “surveys”

• Discussed the basic steps in conducting a rigorous research survey • Considerations when choosing a survey mode – In various ways, choice likely to affect survey • Timeliness • Accuracy • Cost – And cost may drive mode choice possibilities 39