Question Design (1)

8/16/2019 Question Design (1)

1/58

WHAT IS A QUESTIONNAIRE?

An instrument (form) tocollect answers to questions

collect factual data

gathers information or measures A series of written questions/items in a fixed,

rational order


2/58

WHY USING A QUESTIONNAIRE?

A well designed questionnaire:

Givesaccurate andrelevant information to your

research question

Minimizes potential sources of biasWill more likely be completed

⇒ As simple and focused as possible


3/58

Statistical Terms

• Population: Discrete group whose members can be defined by a set

of at least one shared characteristic

• Sample: A selection of that target population (can also be a

subsample)

• Random Sample: Sample in which every member of a given

population has an equal chance of being selected

• Margin of Sampling Error (MOSE): Every random sample has a

margin of error (statistical price you pay for not interviewingeveryone)

• Response Rate: ate at which people agreed to participate in

survey! "f those who refuse are different from those who agree#

results may be s$ewed (higher response rate % less chance for s$ew)


4/58

Statistical Terms

• Population: Discrete group whose members can be defined by a set

of at least one shared characteristic

• Sample: A selection of that target population (can also be a

subsample)

• Random Sample: Sample in which every member of a given

population has an equal chance of being selected

• Margin of Sampling Error (MOSE): Every random sample has a

margin of error (statistical price you pay for not interviewingeveryone)

• Response Rate: ate at which people agreed to participate in

survey! "f those who refuse are different from those who agree#

results may be s$ewed (higher response rate % less chance for s$ew)


5/58

ADVANTAGES OF QUESTIONNAIRES

Can reach a large number of people

relatively easily and economically

Provide quantifiable answers

Relatively easy to analyse


6/58

DISADVANTAGES OF QUESTIONNAIRES

Provides only limited insight into

problemLimited response allowed by questions

Maybe not the right questions are asked

Varying response

Misunderstanding/misinterpretationNeed to get it right first timeHard to chase after missing data


7/58

TYPES OF QUESTIONNAIRE

Face to face

Telephone

By mail

E-mail/InternetSelf-administrated

Interviewer

-administrated


8/58

SELF-ADMINISTERED QUESTIONNAIRE

Advantages:Cheap and easy to administer

Preserves confidentiality

Completed at respondent's convenienceNo influence by interviewer


9/58

SELF-ADMINISTERED QUESTIONNAIRE

Disadvantages:Low response rate

Questions can be misunderstood

No control by interviewerTime and resouces loss


10/58

INTERVIEW-ADMINISTERED

QUESTIONNAIRE

Advantages:Participation by illiterate people

Clarification of ambiguity

Quick answers


11/58

INTERVIEW-ADMINISTERED

QUESTIONNAIRE

Disadvantages:Interviewer bias

Needs more resources

Only short questionnaires possibleEspecially on telephone

Difficult for sensitive issues


12/58


13/58

Basic Survey Process

Steps:

1 !eciding on a mec"anism

# !ra$ing a sample

% !esigning t"e &uestionnaire

&! 'ielding a survey and collecting the data

! Analysis

! eporting


14/58

STAGES IN DESIGNING A QUESTIONNAIRE (1)

Planning the study:Decide on goalsIdentify risk factors for getting bitten by dogs

Know the subjectLiterature, experts on dog bites

Formulate a hypothesisPostmen more likely to get bitten by dogs than thenormal population

Define information needed to test hypothesisOccupation, owning dog, outdoor activities, attitudetowards dogs


15/58

STAGES IN DESIGNING A QUESTIONNAIRE

(2)

Determine study population:

Know the respondentsOccupation

Special sensitivitiesEducation

Ethnic

Language

⇒Questionnaire needs to be adapted to yourpopulation, not the opposite!


16/58

STAGES IN DESIGNING A

QUESTIONNAIRE (3)

Design questions:

Content of the questions

Format of the questions

Presentation and layout

Coding schedule (if appropriate)

Pilot and refine questionnaire


17/58

WHAT DO YOU PREFER?


18/58

WHAT DO YOU PREFER?


19/58

WHAT MAKES A WELL DESIGNED

QUESTIONNAIRE?

Good appearance

→ easy on the eye

Short and simple

Relevant and logical

⇒ High response rate

⇒ Easy data summarisation and analysis


20/58

QUESTION ORDER

Decide onorder of items/questions Easy → difficult

General→ particular

Factual → abstract

Where to placesensitive questions?

Be aware of ordering effects!


21/58

QUESTION ORDER

Questions should be ordered so as to seem

logical to the respondent

First questions should be relevant and easy

Questions are effectively ordered from mostsalient to least salient

Demographic questions should not be

covered at the beginning

Potentially objectionable questions areplaced near the end


22/58

QUESTION ORDER

Group questions by topic/ response options

Starting questionsSimple

With closed format

Relevant to main subject

Non-offending

Neither demographic nor personal questionsDon’t put most important item last


23/58

QUESTIONNAIRE INTRODUCTION

Survey/interview introductionWho you are/ what is your ILP/thesis topic

Why you are investigating

Where you obtained the respondent’s nameHow and where you can be contacted

Guarantee of confidentiality

Length of interview (be honest)

⇒

Usefulness of study should be clearto all respondents


24/58

CONTENT OF QUESTIONS

Clearfocus on research question

Avoid sidetracking

Avoid unnecessary information

Demographic informationContact information (if non-anonymised)


25/58

DO:

Use simple wording

Be brief

Be specific


26/58

DO NOT:

Be vague

Be condescending or talk down to

respondent

Use biased wordingUse abbreviations or scientific jargon

Use objectionable questions

Be redundant


27/58

FORMAT OF QUESTIONS

Adjust toresponding audience

Professionals vs. public

Keep sentences simple and short

Define key wordsRemember option “don’t know”


28/58

FORMAT OF QUESTIONS

Ask for one information at a time

Do you own a dog or have frequent contacts with dogs?

Yes

No

Use mutually exclusive and exhaustive answeroptions

Vertical order of answer options


29/58

BE ACCURATE

Do you often touch dogs?

Yes

No

vs.How often did you touch a dog during thepast 3 months?

Once

Twice Three times or more Not at all Don´t now


30/58

BE APPROPRIATE

!re you a drun?

Yes

No

vs.

How often have you consumedalcoholic "everages during the past #months?

Daily $%# times&wee

Once a wee

'ess than once a wee

Don´t now


31/58

BE OBJECTIVE

Did you drin the strange "rownish drin in(rague?

YesNo

Vs.

)hich "everage did you consume?

)ater

*eer )ine +arad, None of them Don´t now


32/58

BE SIMPLE

Did you smoe not less than a meanamountof - cigarettes&$ days from ./// onwards?

Yes

No

vs.

Did you smoe an average of $ pac ofcigarettes&wee for the last 0 years?

Yes No

Don´t now


33/58

BIAS

Bias = systematic differences in the

measurement of a response


34/58

INFORMATION BIAS

Recall biasCases more likely to remember than controls

Observer bias Different interviewer – different interpretations

Different interpretation of similar questions

Reduce by structured questionnaire


35/58

NON-RESPONSE BIAS

Those who respond are different from those who

do notTelephone interviews: more females, elderly

ReduceEnsure high response rate

Random choice of interview partners

Correct during analysis (eg age, sex)


36/58

FORMAT OF QUESTIONS

Two main question formats

Closed format→ forced choice

Yes

!lways

No 1ometimesDon2t now Never

Open format→ free text

)hat is your most distressing symptom?(lease descri"e444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444


37/58

OPEN OR CLOSED?

Closed

Advantages:Simple and quick

Reduces discrimination against less literateEasy to code, record, analyze

Easy to compare

Easy to report results


38/58

CLOSED QUESTIONS

Disadvantages: Restricted number of possible answers

Loss of information

Possible compromise: Insert field „others“


39/58

OPEN QUESTIONS

Advantages:Not directive

Allowsexploration of issues

to generate hypothesisqualitative research, focus groups, trawling questionnaires

Used even if no comprehensive range of alternative choices

Good for exploring knowledge and attitudes

Detailed and unexpected answers possible


40/58

OPEN QUESTIONS

Disadvantages:Interviewer bias

Time-consuming

Coding problemsDifficult to analyze!

Difficult to compare groups


41/58

CLOSED QUESTIONS

1. Straightforward response

)hat is your age in years? 444 years

How long have you owned a dog? 444

years )hat is your se5 6gender7?

8ale

9emale

Did you stay in Hotel : on $3&-&;0? YesNo

Don2t now


42/58

CLOSED QUESTIONS

2. Checklist

)hich of the following outdooractivitiesdid you do last wee?


43/58

CLOSED QUESTIONS

3. Rating scale

Did you do use sunscreen during the followingoutdoor activities during the past si5 months?

!lways 1ometimes 1eldomly Never


44/58

CLOSED QUESTIONS

4. Rating scaleNumerical

How useful would you thin that informationonthe ris of "iting from stray dogs would "e?(please circle) . $ 3 > 0 # -

Not at all useful ery useful

!nalogueHow much is your pain severe 6put the ticon the line7

;.;


45/58

CLOSED QUESTIONS

5. Scales for measuring attitude (Lickert)

1tray dogs carry a higher ris of ra"ies

No@ A strongly disagreeNo@ A disagree quite a lot

No@ A disagree Bust a little

A2m not sure a"out this

Yes@ A agree Bust a little Yes@ A agree quite a lot

Yes@ A strongly agree


46/58

PROBLEMS AND PITFALLS

Avoid questions that asktwo things at once -

you won’t know which ‘bit’ people are answering:

Have you ever had stomach ache anddiarrhoea?

Ambiguity.....

Do you go to the woods a lot?


47/58


Avoid jargon/abbreviations/slang

How often do you get up at night to (C?6pass urine7

1hould ADCs "e treated in the community?

Avoidnot mutually exclusiveoptions )hat age are you?

.#%$;

$;%$0

$0%3;

30%>;


48/58


Avoidleading questions

Do you thin that the food in the hotelmade you sic?

Did the hotel sta seem unhygenic to you?

Do you agree that the hospital sta wereclose to e5haustion?

Avoid making questionnairetoo long

Typographical / spelling errors


49/58

PILOTING AND EVALUATION

Pilot with asimilar group of people to your

intended subjects

Highlights problems before startingEffects of alternative wording

Overall impression on respondents and interviewers

Final polishing after several amendments


50/58

PRESENTATION AND LAYOUT

Clear consistentlayout Adequate space to answer

Large font size Appropriate page breaks

Avoidexperimental layouts

fancy logos

printed on recycled paper/is an equal opportunity employer etc

S O O


51/58

PRESENTATION AND LAYOUT

Usingcolour or printing questionnaire on coloured

paper may help

Usefilter questions, if necessary

Giveclear instructions about how to answer the

questions


52/58

WAYS TO INCREASE PERCEIVED

REWARD

Show positive regard

Say thank you Ask them for “advice”Give social validationGive a tangible reward

Make the questionnaire interestingState an upcoming deadline


53/58

WAYS TO DECREASE PERCEIVED

COST

Avoid subordinating language

Do not embarrass the respondentDo not inconvenience the respondentMake questions appear short and easy Avoid asking personal information (if it must

be asked, use ‘soft’ wording and inform aboutconfidentiality)Keep subsequent requests similar


54/58

CREATE A NAVIGATIONAL PATH

Instructions should be placed right where

they are needed

Matrices are confusing: it is best to order

questions from top to bottom, and left to

right

Uselarger font to attract attentionUse color shading to attract attention or

show groupingsUse spacing and similarities to show

groupings


55/58

WAYS TO ESTABLISHTRUST

Provide a token of appreciation

Provide a sense of legitimate authorityMake completing the questionnaire seem

important

Remind respondent of previous relationship

with sponsor (if applicable)


56/58

SUMMARY

A well designed questionnaire:

Will give appropriate data which allow toanswer

your research question

Will minimise potential sources ofbias, thusincreasing the validity of the questionnaire

Will much more likely becompleted


57/58

FINALLY, KEEP YOURQUESTIONNAIRESHORT

AND THE QUESTIONSSIMPLE,FOCUSED AND

APPROPRIATE


58/58

Question Design (1)

Documents

Transcript of Question Design (1)