Designing Questionnaire and Survey Research (updated Jan 2011)

Hangzhou, January 2011Prof. Dr. Hora Tjitra, Zhejiang University

Designing Questionnaire and Survey ResearchQuantitative Research Methodologies (4/6)

Designing Ques*onnaire and Survey Research

2

Designing Questionnaire and Survey Research

Survey research: a specific type of field study that involves the collec3on of data from a sample of elements drawn from a well-‐defined popula3on through the use of a ques3onnaire.

The reasons to conduct survey research…

• Study a representa-ve sample through field research is rela-vely easy and prac-cal.

• Research can confidently generalize the finding from selected sample to en-re popula-on.

• It provides ideal condi-ons for explora-on of process × individual difference interac-on.


3

14 years in Germany

7 years in China

Born and grew up in Indonesia

Prof. Dr. Hora Tjitra - Cross-cultural and Business Psychology

Dipl.-Psych., Technical University of BraunschweigOrganizational Psychology and Human Resource Management

Dr. Phil., University of RegensburgIntercultural Psychology and Strategic Management

Executive Education, INSEADHR Management in Asia


Outline

1 Doing Survey Research 4

2 Designing Questionnaire 13

3 Sampling Design in the Survey Research 19

4

Doing Survey Research


6

Types of Surveys

…the broad area of survey research encompasses any measurement procedures that involve asking ques3ons of respondents.

Ques%onnaires: Ques-onnaires are usually paper-‐and-‐pencil instruments that the respondent completes.

Interviews (constructed): Interviews are completed by the interviewer based on the respondent says.


7

The Issues in Survey Research

Question Issues✓ What types of questions can be asked?

✓ How complex will the questions be?

✓ Will screening questions be needed?

✓ Can question sequence be controlled?

✓ Will lengthy questions be asked?

✓ Will long response scales be used?

Population Issues✓ Can the population be enumerated?

✓ Is the population literate?

✓ Are there language issues?

✓ Will the population cooperate?

✓ What are the geographic restrictions?

Bias Issues✓ Can social desirability be avoided?

✓ Can interviewer distortion and subversion be controlled?

✓ Can false respondents be avoided?

Sampling Issues✓ What data is available?

✓ Can respondents be found?

✓ Who is the respondent?

✓ Can all members of population be sampled?

✓ Are response rates likely to be a problem?

Content Issues✓ Can the respondents be expected to know

about the issue?

✓ Will respondent need to consult records?

Administrative Issues✓ Costs & Facilities

✓ Time & Personnel


8

Three Types of Measurement

Scaling is the assignment of objects to numbers according to a rule.

Likert or "Summative" scaling there are a variety possible response scales (1-to-7, 1-to-9, 0-to-4).

Guttman or “Cumulative” scaling the answering item is “Yes” or “No”

Thurstone or Equal-appearing interval scaling

rate each statement on a 1-to-11 scale in terms, then choose “yes” or “No”


9

The Basic Steps in Developing a Likert or "Summative" Scale.

Defining the Focus: define what it is you are trying to measure

Administering the Scale: reversal items 1= strongly disagree 2= disagree 3= undecided 4= agree 5= strongly agree

Generating the Items: create the set of poten-al scale items. These should be items that can be rated on a 1-‐to-‐5 or 1-‐to-‐7 Disagree-‐Agree response scale.

Rating the Items: have a group of judges rate the items. Usually you would use a 1-‐to-‐5 ra-ng scale where:1= strongly unfavorable to the concept ; 5= strongly favorable to the concept

Selecting the Items. compute the intercorrela-ons between all pairs of items, based on the ra-ngs of the judges.

•Throw out any items that have a low correla<on with the total (summed) score across all items • For each item, get the average ra<ng for the top quarter of judges and the boFom quarter. Then, do a t-‐test of the differences between the mean value for the item for the top and boFom quarter judges.


10

Example: The Employment Self Esteem Scale

INSTRUCTIONS: Please rate how strongly you agree or disagree with each of the following statements by placing a check mark in the appropriate box.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 1. I feel good about my work on the job.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 2. On the whole, I get along well with others at work.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 3. I am proud of my ability to cope with difficulties at work.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 4. When I feel uncomfortable at work, I know how to handle it.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 5. I can tell that other people at work are glad to have me there.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 6. I know I'll be able to cope with work for as long as I want.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 7. I am proud of my relationship with my supervisor at work.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree

8. I am confident that I can handle my job without constant assistance.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 9. I feel like I make a useful contribution at work.

Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 10. I can tell that my coworkers respect me.


11

Survey-Research Design

Cross-Sectional Designthe collection of data at a single point in time from a sample drawn from a specified population

Successive Independent Samples Designa series of cross-sectional survey are conducted over time

Longitudinal Designthe same sample of respondents is surveyed more than once


12

Total survey error

The ultimate goal of survey research is to accurately measure particular constructs within a sample of people who represent the population of interest

• Coverage Error: the bias that can result when the pool of potential survey participants from which a sample is selected does not include some portions of the interest

• Sampling Error: the random differences that invariably exist between any sample and the population from which it was selected

• Nonresponse Error: the bias that can result when data are not collected from all of the members of a sample

• Measurement Error: all distortions in the assessment of the construct of interest, including systematic biases and random variance that can be brought by respondents’ own behavior, interviewer behavior, and the questionnaire

Designing Questionnaire


14

Questionnaire Design:The six steps for preparing the questionnaire

1. Decide what informa0on should be sought

2. Decide what type of ques0onnaire should be used

3. Write a first dra; of the ques0onnaire

4. Reexamine and revise the ques0onnaire

5. Pretest the ques0onnaire

6. Edit the ques0onnaire and specify the procedure


15

Question Content

• Is the Question Necessary/Useful?

♪ Do you need the age of each child or just the number of children under 16?

♪ Do you need to ask income or can you estimate?

• Are Several Questions Needed?

♪ double-barreled question (eg. Teacher & parents) ♪ does not cover all possibilities ♪ does not give you enough context ♪ does not determine the intensity

• Do Respondents Have the Needed Information?

♪ You should ask a filter question first (e.g., Have you ever watched the show ER?) before asking them their opinions about it.

• Does the Question Need to be More Specific?

♪ How well did you like the book? ♪ Did you recommend the book to others?

• Is Question Sufficiently General?

• Is Question Biased or Loaded? ♪e.g. What do you see as the benefits of a tax cut? ♪e.g. What do you see as the disadvantages of eliminating

welfare?

• Will Respondent Answer Truthfully?


16

Question Words

♪ Can the Question be Misunderstood? e.g. What kind of headache remedy do you use?

One of the major difficulty in wri-ng good survey ques-ons is geRng the wording right

♪ Is the time frame specified? Whenever using the words "will", "could", "might", or "may" in a ques<on, you might

suspect that the ques<on asks a <me-‐related ques<on.

♪ What Assumptions Does the Question Make?

♪ Is the wording too direct? E.g. How did you feel about being in the war?

How well did the equipment hold up in the field? How well were new recruits trained?

♪ Does the question contain difficult or unclear terminology?…..


17

Question Placement

Considering following questions: ♪ Is the answer influenced by prior ques<ons?

♪ Does ques<on come too early or too late to arouse interest?

♪ Does the ques<on receive sufficient aFen<on?

A Checklist of Considerations✓ start with easy, nonthreatening ques<ons

✓ put more difficult, threatening ques<ons near end

✓ never start a mail survey with an open-‐ended ques<on

✓ for historical demographics, follow chronological order

✓ ask about one topic at a <me

✓ when switching topics, use a transi<on

✓ reduce response set (the tendency of respondent to just keep checking the same response)

✓ for filter or con<ngency ques<ons, make a flowchart

The Golden RuleDo unto your respondents as you would have them do unto you!


18

Other Important Issues

Question Order✓ Grouping Questions by topic may be useful✓ It is easier for the respondents to think in the same topic

❖ For bipolar scales ( e.g., running from positive to negative with neutral in the middle), reliability and validity are highest for about 7 point.( Matell & Jacoby, 1971)

❖ The reliability and validity of unipolar scales (e.g., running from no importance to very high importance) seem to be optimized for a bit shorter scales, approximately 5 points lon (Wikman & Waarneryd, 1990)

❖ A good number of studies suggest that data quality is better when all scales points are labeled with words than only some are. ( Krosnick & Berent, 1993)

Ra#ng Scale Formats

Sampling Design in the Survey Research


20

Sampling in Survey Research


21

Statistical Sampling Terms

Variable: is a specific measurement value that a sampling unit supplies

Statistic: the responses that we get for our en<re sample

Parameter: the measurement the entire population


22

Different Approaches to Sampling Methods

Probability Sampling Non-Probability Sampling

Any method of sampling that utilizes some form of random selection

➡ Simple Random Sampling

➡ Stratified Random Sampling

➡ Systematic Random Sampling

➡ Cluster (Area) Random Sampling

Methods do not involve random selection

➡ Convenience Sampling

➡ Purposive Sampling


23

Probability Sampling

any method of sampling that utilizes some form of random selection

Some Definitions

✓ N = the number of cases in the sampling frame

✓ n = the number of cases in the sample

✓ NCn = the number of combinations (subsets) of n from N

✓ f = n/N = the sampling fraction


24

Simple Random Sampling

Objective: To select n units out of N such that each NCn has an equal chance of being selected.

Procedure: Use a table of random numbers, a computer random number generator, or a mechanical device to select the sample.


25

Stratified Random Sampling

Objective: Divide the population into non-overlapping groups (i.e., strata) N1, N2, N3, ... Ni, such that N1 + N2 + N3 + ... + Ni = N. Then do a simple random sample of f = n/N in each strata.

Procedure:Divide your population into homogeneous subgroups and then taking a simple random sample in each subgroup.


26

Systematic Random Sampling

• number the units in the population from 1 to N

• decide on the n (sample size) that you want or need

• k = N/n = the interval size

• randomly select an integer between 1 to k

• then take every kth unit


27

Cluster (Area) Random Sampling

• Divide population into clusters (usually along geographic boundaries)

• Randomly sample clusters

• Measure all units within sampled clusters

Beijing

JinanNanjing

ShanghaiHangzhou

Changsha

Shenzhen

Chongqing

Tianjin

ShijiaZhuang

(Hebei)Zhengzhou

(Henan)

Qingdao

Hefei(Anhui)

Suzhou

Chengdu

Shantou Xiamen

Wuhan

Xi’an(Shaanxi)

Zhanjiang

Guiyang

Kunming

Eg. Measuring Chinese Personality


28

Non-Probability Sampling

does not involve random selec7on

• Modal Instance Sampling• Expert Sampling • Quota Sampling • Snowball Sampling

Purposive Sampling: sample with a purpose in mind.

or called Accidental, Haphazard samplingthey are not representative of the populations

Convenience Sampling:tradi0onal "man on the street"


29

Advantage and Disadvantage of Sampling Approach

Probability Sampling

Non-Probability SamplingAdvantage Research can be confident that a

selected is representative

It permits researchers to precisely estimate the amount of variance present in a given data set that is due to sampling error

Disadvantage✓ Calculation

Advantage Convenient

Flexible

Disadvantage✓ We may or may not represent

the population well,

✓ It will often be hard for us to know how well we've done so

Thank You

Contact us via …

Mail: [email protected]: twitter@htjitraWebsite: http://horatjitra.com

Zhejiang University, Hangzhou (China)

mailto:[email protected]?subject=

mailto:[email protected]?subject=

http://horatjitra.com

http://horatjitra.com

Designing Questionnaire and Survey Research (updated Jan 2011)

Education

Transcript of Designing Questionnaire and Survey Research (updated Jan 2011)