1 3. Drafting Each Module. What we typically think of as “Designing a Survey Instrument” Going...

1

3. Drafting Each Module


• What we typically think of as “Designing a Survey Instrument”

• Going question by question to ensure it meets the research objective

• Deciding question format (A)• Specify wording (B)• Detailing (C)• Translation (D)

2


Questions should flow logically from one to the next.

Within each module, questions should flow:• from the more general to the more specific; • from the factual and behavioral to the attitudinal;• from the least sensitive to the most sensitive.

3


A. Deciding Question Formats

1. Open vs. Closed2. Single vs. Multiple Response3. Rating4. Ranking

4


A. Deciding Question Formats1. Open vs. Closed Questions

“Why don’t you save in a formal bank?” gives precise judgmentsBut….if you are concerned, as most surveys are, in summarizing replies to

produce picture of population, better approach is probably a closed ended question:

Do you not save in a formal bank because? (prompt each response)

5

OPTION Yes No Bank is too far 1 2 Does not offer interest 1 2 Cannot get money when I need it 1 2 Don’t have the required papers to open account 1 2 Bank staff were not helpful 1 2 Do not have enough for minimum amount 1 2 You have never tried 1 2 Other (specify)__________________________________________________ 1 2


A. Deciding Question Formats2. Single vs. Multiple ResponseDo you want respondent to give single or multiple answer? Example of Multiple Response:

6

Which of the following means do you use to save? [circle all that apply]

CODE

Formal bank 1 SHG 2 Chit fund 3 Friend 4 Family 5 Employer 6 Etc…. ….. Other__________ Don’t know Refuse to Answer


A. Deciding Question Formats2. Single vs. Multiple Response

Example of single response:

7

Which is your most usual means of saving? [one response only]

CODE

Formal bank 1 SHG 2 Chit fund 3 Friend 4 Family 5 Employer 6 Etc…. ….. Other__________ Don’t know Refuse to Answer


A. Deciding Question Formats3. Likert (“Rating”) Scales: Another Popular Approach

8

How important are the following aspects of a lender/creditor? VI=very important, I=Important, N=Neutral, U-Unimportant, VU=Very unimportant. VI I. N U VU

Distance to lender 1 2 3 4 5

Term of repayment 1 2 3 4 5

Speed of granting loan 1 2 3 4 5

Loan amount 1 2 3 4 5

Flexible repayment 1 2 3 4 5

Collateral requirement 1 2 3 4 5

Interest rate 1 2 3 4 5

Other __________ 1 2 3 4 5


A. Deciding Question Formats3. Likert (“Rating”) Scales: Another Popular Approach• Response categories such as strongly agree, agree, neutral, disagree, and

strongly disagree. • Advantage: good for opinions and attitudes. • Disadvantage: Not good for factual information.

AVOID asking descriptive terms for facts. Ask directly to provide numerical estimate.

Example: • Inaccurate: How often do you deposit savings in your account?

Never, hardly ever, sometimes, constantly• Accurate: In the last 1 month, how many times did you deposit savings?

Number of times_______________.

9


A. Deciding Question Formats4. Ranked ResponsesSometimes useful for respondent to rank a set of options by numbering.

Trouble spots: Assumes respondent feels differently about each item.Can be difficult to compute.Do not add too many categories.

10

What are the three most desirable features of a person or place which gives loans, starting with the most important?

[INDICATE BY NUMBERING FROM 1-3 WHERE 1 IS THE MOST IMPORTANT]

OPTION RANK

Cost of borrowing

Length of repayment

Speed of getting loan

Amount of loan


B. Specifying Wording – Coding

• Designing good response codes requires strong knowledge of the phenomena being studied.

Background research and possible qualitative research are handy here. See similar questionnaires.

Ideally responses to almost all questions should be given code numbers only code numbers, as opposed to words or phrases, are recorded on the questionnaire

11


B. Specifying Wording – Precoding

Precoding requires that codes be clear, simple, and

1. mutually exclusive;

2. that they exhaust all likely answers ,

3. that respondents will not all provide the same response,

4. and that none of the codes apply to only a handful of respondents.

12


B. Specifying Wording – PrecodingUseful tricks:

• To ensure codes are mutually exclusive , add a qualifier where more than one answer could apply:– What is most recent….”– “What is biggest…,”– “What is closest…”– “What was first (or last, or principal) reason for…?”

• To ensure codes encompass all possible answers , add “other (specify______)” code to questions where precoding each possible response is impossible/ inconvenient.

13


B. Specifying Wording – Recall periodsTrade off potential recall (memory) bias from long period reporting against

potential variance from short periods

Example, consumption:• Frequently purchased items (e.g. food), recall period between one week

and one month• When practical: diaries for a week or two• Durable goods: annual recall basis

Reference periods can be different across questions, and sections

Different recall periods can present a problem when combining or comparing data across modules.

14


B. Specifying Wording: be concise, clearNo Jargon! Asking a rural household:

“Do you believe the government should ban higher interest rates?” Will get better response than:

“Do you believe the Microfinance Bill / RBI should take action to…”

Be clear: Asking

“Have you borrowed any money lately?” is more ambiguous than

“Have you borrowed any money from friends, employers, groups, etc in the last 30 days?”

15


B. Specifying Wording: Avoid Double Questions

Sometimes questions hide a dual question, for example:

Instead ask:

16

In the next 12 months, are you planning to borrow less and save more? CODE

Yes 1 No 2

In the next 12 months, are you planning to borrow less? CODE

Yes 1 No 2

In the next 12 months, are you planning to save more? CODE

Yes 1 No 2


B. Specifying Wording: stick to observables

• It’s tempting to take shortcuts

• Stated preference and revealed preference

• Questions about future intentions are unreliable

• Can infer preferences from observed behavior

17


B. Specifying Wording: Be Precise

“What is your date of Birth?”

Is better than:

18

Are you……?what is your age range CODE

Under 14 1 15-30 2 31-50 3 51-65 4

Over 66 6 DK RA


B. Specifying Wording: Why Be Precise?

Asking exact age will also suffice and have the added advantages that:

• Exact ages may very simply be recoded into groups

• Can alter these groups while keeping the same ages. Not only could you inspect different age groups than first envisaged, but you could backdate your age groups. Imagine that a government program makes 20 year old women eligible for a credit and savings program…

• Can obtain continuous descriptive statistics, e.g., mean and standard deviation.

19


B. Specifying Wording: final wordsAvoid leading questions• Example: “do you believe that the clutches of the rural moneylenders is a

good reason for the government to make a ban on high interest rates?” • Avoid “would”, “could”, “should”

Ask short questions with simple answers• Increases size of questionnaire, but decreases time to administer• Make use of skip patterns and filter questions

Answers should be read out exactly as written• Error rate is up to 20% higher when questions are not read verbatim

20


C. Detailing• Definitions

– who is a household member? – what is a business? – who is a migrant?– what is a health event? etc.

• Don’t bias your respondent– What did you use this loan for? instead of where did you invest

this loan? (investment) or what did you buy with this loan? (consumption)

– Avoid questions that are open to interpretation• How often do you deposit savings in your account?

– Never, Hardly ever, Sometimes, Constantly• In the last 1 month, how many times did you deposit savings?

– Number of times

21


C. Detailing• Cross-check important questions

– Sources of income; assets; land ownership; health events and health expenditure; household consumption – at different points in the survey

• Unit of data and components – food consumption in Rs or Kg.– “bhiga” for reporting land in rural areas– loans, asset ownership is usually recorded at household level

• Self-reporting v.s . other means of evaluating data – business income – you could capture inputs and sales– perceived resale value of assets instead of cost price (actual

expenditure)

22


D. Translation• The local language is probably not English, which makes things

tricky as to the wording of certain questions• Translate

– Ensures that every surveyor knows the exact wording of the questions, instead of having to translate on the fly

• Always back-translate!– Outsource to 2 people: to translate from English to local

language and another to back translate to English. Compare• Contextualize – refine during pilots. Find local terms for

quantities (units – acre, bhigas, etc.); • Adapt local terms/language – keep it informal; meaning should

be clear (so avoid using language experts!)

23

1 3. Drafting Each Module. What we typically think of as “Designing a Survey Instrument” Going...

Documents

Transcript of 1 3. Drafting Each Module. What we typically think of as “Designing a Survey Instrument” Going...