Fundamentals of questionnaire design

Professor Steve StrandInstitute of EducationUniversity of Warwick

Advanced Quantitative Methods (ARM3) 2011/12

Aims of the session

• Understand where questionnaires fit in to the research process;

• Know how too design a ‘good’ questionnaire and how it impacts crucially on the quality and validity of the resulting data;

• To consider sampling and response issues, and how these can impact on interpretation of your results.

How to do survey research• Select a research topic or area

• Formulate specific research questions or hypotheses

• Define what information you need

• Decide what your population is

• Decide how to sample from the population

• Design instruments

• Pilot and revise instruments

• Distribute and collect data

• Analyse and report

Questionnaire design

“The world is full of well meaning people who believe that anyone who has a modicum of common sense can produce a good questionnaire”

Oppenheim (1966). Questionnaire design and attitude measurement. pvii.

Task 1: Design a questionnaire

• You want to know about student attitudes to school and how these might impact on educational attainment.

• Design a questionnaire to measure students attitude to school.

• 10 minutes

The Nature of Questions - ABC

• Affect: What people feel about something (Do you enjoy school? What are your views about...)

• Behaviour: what people actually do (have you participated in an after school club in the last term?)

• Cognition: what people know or are aware of about something (Are you aware of the Single Equalities Discrimination legislation?)

• + Attributes: Who people are and what they have (demographic information).

Question type

• Two main types of questions: open and closed questions.

– How do you teach mathematics?– How much time per 60 min lesson do you spend

talking to the whole class?– Less than 10 minutes?– Between 10 and 20 minutes?– More than 20 minutes?

• What are advantages and disadvantages of open vs. closed questions?

Question type (cont.)• Open questions

– allowing richer responses (extended comments);– can bring out perspectives/elements you had not thought of;– can lead to a lack of clarity in analysis, as you can’t probe

what respondents mean,– can be time consuming, and– can be hard to code for analysis.

• Closed questions– are less time consuming to answer;– lead (if well-constructed) to unambiguous analysis and coding;– best option if very clear hypotheses and research design;– But less good if your research is more exploratory;– Can allow for detailed statistical analysis.

Closed question response types• Categories e.g., age 20-29, 30-39 etc. (watch for

overlap)

• List can tick multiple options e.g. subjects studied

• Ranking e.g., place the following in order of importance to you (e.g., in terms of school choice)

• Scaled Most frequent format for closed questions is a scaled response

– Dichotomous (e.g., Yes/No)– Multiple scale points (see examples overleaf)

Rating scales• Likert scaling (“I enjoy being a student in this school”)

Strongly agree Agree Neither agree nor disagree

Disagree Strongly disagree

Strongly agree Agree Disagree Strongly disagree

Excellent Good Satisfactory Unsatisfactory

Strongly agree

10 9 7 6 5 4 3 2 1 Strongly disagree

• How many scale points? (more than seven difficult to interpret)

• Should you use a neutral midpoint (e.g., JRS above)?

• (-) Tendency for people to sit on the fence, no decision

• (+) People can be genuinely neutral, and allows for not sure, don’t know, or don’t understand?

Semantic differential example

English

hard 1 2 3 4 5 easy

mainly for girls 1 2 3 4 5 mainly for boys

I really enjoy it 1 2 3 4 5 I really dislike it

important for job/career 1 2 3 4 5 not needed for job/career

important for adult life 1 2 3 4 5 not needed for adult life

too much of it 1 2 3 4 5 not enough of it

useful for me now 1 2 3 4 5 useless for me now

helped me to develop as a person 1 2 3 4 5 hasn’t helped me to develop as a person

lessons usually build on the last one1 2 3 4 5 lessons don’t usually build on the last one

too little homework 1 2 3 4 5 too much homework

I feel overworked in class 1 2 3 4 5 I feel underworked in class

I’ve made poor progress in this subject1 2 3 4 5 I’ve made good progress in this subject

• School is ….fun 7 6 5 4 3 2 1 boring

Useful 7 6 5 4 3 2 1 useless

easy 7 6 5 4 3 2 1 difficult

Can get very stupid, here

is a silly example!

SCALE A

This school now

Questions SCALE B

Creating a more effective school

Strongly

Disagree

Disagree

Uncertain

Agree

Strongly

Agree

Not at

all important

Not very

important

Fairly import

ant

Important

Crucial

1 2 3 4 5 Teachers regularly discuss ways of improving pupils' learning

1 2 3 4 5

1 2 3 4 5 Teachers encourage pupils to be independent learners 1 2 3 4 5

1 2 3 4 5 Pupils respect teachers 1 2 3 4 5

1 2 3 4 5 Staff ensure pupils receive constructive feedback about their work

1 2 3 4 5

1 2 3 4 5 Senior staff are available to discuss curriculum / teaching matters

1 2 3 4 5

1 2 3 4 5 Teachers respect pupils 1 2 3 4 5

1 2 3 4 5 Staff encourage pupils to try their best 1 2 3 4 5

1 2 3 4 5 School targets/goals are challenging 1 2 3 4 5

1 2 3 4 5 Whole-school meetings are worthwhile 1 2 3 4 5

Multiple dimensions for a single Question

Question wording: The Pitfalls• Ambiguity

– How would you describe your social class (working class, middle class, upper class)?

– Do you complete regular homework?

• Assumptions

– Does the college make adequate provision for counselling?

• the use of confusing negatives

– Would you rather not use group activities when teaching maths?– Without a PTA you cannot talk about the progress of your children?

Infant school Which type of school does your child attend? Primary School Comprehensive School Grammar School Other (please specify)

Question wording (cont.)• Overly complex or theoretical jargon

– “Would you prefer a short, non-award bearing course (3-5 sessions) with part-time day release (e.g. Wednesday afternoons) and one evening per week with financial reimbursement for travel, or a longer award bearing course (6-8 sessions) with full day release?”

– “What is your opinion on the value of a social-constructivist perspective on learning?”

• Double questions– Do you attend maths and chemistry classes?– Do you attend and enjoy after school clubs?

• Leading questions– Is it fair that students have to take unsocial part-time jobs to pay

their university top-up fees?– Do you prefer theory based student led styles or old-fashioned

teacher directed styles?

Task 2

• Attached are OFSTED questionnaires to

– (1) sixth form students about your school or centre – (2) Parent and carers’ about their children’s school

• What do you think of the two OFSTED questionnaires in relation to:

– Degree of openness/closed– Assumptions or leading questions– Inappropriate complexity– Ambiguity, potential for confusion– Response bias

Response bias

• Respondents tend to give positive responses to questions, give researcher what they think they want

• Must counter-balance your questions

• When you score your questionnaire, reverse score these items so high scores are in a consistent direction with regard to the construct under consideration

Item Strongly agree

agree disagree Strongly disagree

I look forward to coming to school 4 3 2 1

I do not enjoy school 1 2 3 4

Sequencing the questions• Begin with a brief introduction that clearly states the

purpose of the study, who you are and your affiliation.

• Questions should flow logically from general to specific, and from the least sensitive to the most sensitive, from factual and behavioural to attitudinal and opinion questions

• But address important questions early rather than late in the questionnaire

• Use branching questions if appropriate (e.g. if you answered yes to Q1 go to Q4), but use sparingly in paper questionnaires

• Keep open-ended questions requiring narrative responses to the end

Layout of the questionnaire• A brief introductory statement is useful, especially

if the introductory letter could go adrift.

• Layout - spacing between items, columns line up, separate instructions section, looks professional

• Is there a “natural” grouping for the questions?

• Number individual questions for reference purposes (to aid data entry and analysis)

• Contact details and return date should be included on the questionnaire (addressed return envelopes).

Ethical issues

• Confidentiality/anonymity – No individual/case will be identifiable in any

reports – Sealed pre-paid envelope

• Informed Consent - stress voluntary nature; respondent has access to results, maybe on website; opportunity for debrief/questions

• Incentives - book tokens, prize draws etc (not permitted by DCSF)

Other practical considerations

• Length – four sides A4 maximum, 20 minutes completion time. Is every question really necessary?

• Covering letter – purpose of research, voluntary nature, access to results

• Medium – face-to-face, postal, web, telephone? We will look at online questionnaires later this morning.

Validity• Internal validity

– Respondents understanding, ability and willingness, good questionnaire design will help this

• External validity

– Usually cannot access the whole population, need to select a sample

– Key issue then is the representativeness of the sample (and therefore the generalisability of your results)

Sampling - An example

• Spanish-American wars in 1898. Casualty rate in the US Navy was 10 in every 1000. During the same period the number of deaths among the inhabitants of New York was 20 in every 1000.

• Can we conclude it is safer to fight in a war than live in New York?

Huff (1991). How to lie with statistics. Penguin.

Population

• Sampling should be underpinned by the question ‘who do I want to generalise to?’

• The group we would like to generalise to is called the population.

• Population has a set of characteristics that define it (not the size per se)

• Populations are generally too large to assess all members, we select a sample from the population.

SamplingThere are broadly two types of sampling:

• Probability sampling uses an element of random selection, to ensure that participants have an equal opportunity to be chosen.

• Non-probability sampling does not involve random selection. It does not mean that the sample will not represent the population, but this cannot be assumed.

Probability sampling

• Comprehensive sampling frame required (e.g. all pupils in a school, all schools in an LA etc.)

• Simple random sampling - random (names from a hat) or systematic (e.g. every 5th pupil on the register)

• Stratified random sampling - divide the population into sub-groups and draw a random sample within each sub-group, can be representative with a much smaller sample.

Non-probability sampling

• Convenience sampling – e.g. much market research & street interviews

• Quota sampling – e.g., match particular quotas related to the population (e.g. achieve 20% minority ethnic pupils), but still non-probability

• Purposive sampling – no wish to generalise e.g. views of experts, policy makers etc.

Sampling - procedure• What is the population you are interested in?

• Do you have a sampling frame?

• How will you draw your sample?

– Probability sample– Opportunity sample

• What was your achieved sample (sample size and response rate)?

– it is not uncommon for response rates to be 20% or less. Certain analyses require a minimum sample size, so if you need 60 responses send out 300 questionnaires

– Always quote the response rate. Check the balance of Responders/Non-responders on key demographics if you have them.

Core reading• Bell, J. (2005). Doing your research project (4th

Ed.). Maidenhead: Open University Press. Chapter 8 ‘designing and administering questionnaires’ is an excellent introduction.

• Gorard, S. (2001). Quantitative methods in educational research: The role of numbers made easy. London: Continuum. Chapter 2 ‘Sampling’ is an excellent introduction.

• Cohen, Manion & Morrison (2007). Research Methods in Education (6th Ed). Abingdon: Routledge. Chapter 15 ‘Questionnaires’ is a good overview; Chapter 4 ‘Sampling’ is a detailed overview.