Evaluating behaviour change programs - WasteMINZ · Evaluating behaviour change programs Liz Ampt...

Evaluating

behaviour change

programsprograms

Liz Ampt

Concepts of Change

What are we measuring?

- Whether people are

doing things differently

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

doing things differently

How can we measure it?

- Measuring changes in levels of reducing/ reusing/

recycling

- Observing/recording behaviours

- Asking about change


- Each is a form of survey or data collection exercise

SampleDesign

PilotSurvey

Conductof

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning


ofSurvey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof Results

Tidying-Up

The survey

process

Richardson, Ampt,

Meyburg 1995

Each is a form of survey Method of measurement Aspects of survey design needed

Measuring changes in levels of

reducing/ reusing/ recycling

- Sample selection

- Pilot

- Survey

- Expansion/weighting

- Analysis

Observing/recording behaviours - Sample selection

- Pilot

- Survey




- Analysis

Asking about change - Sample selection

- Survey design

- Pilot

- Survey


- Analysis

Key elements of survey design

1. Preliminary planning

2. Selection of method

3. Sample design

4. Survey instrument design

5. Pilot


5. Pilot

6. Survey implementation

7. Expansion/weighting

• Analysis – over to you..

1. Preliminary planning

• Define survey objectives

– Very specific: what, by whom, over what period, where

• Review of existing information

– Useful methodologies from elsewhere; use of stated

preference?


preference?

• Define terms

– From your objectives and from respondent’s perspective

• Survey content

– Dot points

7

Exercise

• Think of 2 terms you would want to use in a survey

related to your work

– Write down clear definitions for both

– Ask another person (not from your organisation) to do the

same with your terms


– Compare

2. Selection of a Survey Method for Measurement

• Observation surveys

• Intercept surveys

• Self-administered surveys

• Telephone surveys

• Personal interview surveys


• Personal interview surveys

• Internet/online surveys

9

Observation methods

• Chosen when

– Possible to count accurately

– Possible to count all or select a representative sample

• Can be manual, automatic, video


10

Intercept Surveys

• Intercepting people

– At an activity centre (e.g. workplace, transfer station, shopping centre)

• Possible methods

– distribution - mail-back/on-line

– personal interview


– collect phone no.

• Always needs a total classification count

11

Intercept Surveys

• Advantages

– Able to reach specific populations

– Can combine with observational counts

– Can use multiple survey methods

• Disadvantages

– Generally low response rates (20-30% for self-completion)


– Generally low response rates (20-30% for self-completion)

– Hurried conditions

– Must allow for non-random sampling

– No follow-up possible in most cases

12

Self-administered Surveys

• Possible targets

– households

– activity centres/workplaces/transfer stations

• Method of Distribution

– mail-out vs. hand delivered


• Method of Collection

– mail-back vs. hand collection

13

Self-administered Surveys

• Advantages

– Can get extensive geographical coverage

– No interviewer effects

– Can obtain considered responses

– Hand-collection to good response rate

• Disadvantages


• Disadvantages

– Layout and wording must be clear - hard to design

– No probing possible

– Answers not independent

– Response rates lower than face to face

14

Telephone Interviews

• Advantages

– Wide geographic coverage

– Intermediate costs

– Good supervision - CATI

– Multilingual capabilities


– Computerised

Telephone Interviews

• Disadvantages

– Sample usually weak

• Low phone ownership for some groups

• Answer-phones, mobile phones, screening devices

• Hard to know how it represents the population


• Hard to know how it represents the population

– Credibility of interviewer (confusion with telemarketing)

– Low response rate

– No follow-up for refusals

Personal Interviews

• Can be paper or computer

• Advantages

– Generally higher response rates (60-90%)

– Flexibility of information

– Presence of interviewer

Maintain interest


– Maintain interest

– Spontaneous answers

Personal Interview

• Disadvantages

– High costs

– Interviewer influence

• personal characteristics

• interrupt household/work routine

opinions of interviewers


• opinions of interviewers

• interpretation of vague answers

– Considered response difficult

On-line Surveys

• Advantages

– Low costs

– Can use elaborate visual effects

– Can use adaptive techniques (can give different scenarios

for different responses)

– Good for workplaces if sufficient follow-up


– Good for workplaces if sufficient follow-up

• Disadvantages

– Usually very biased sample

– Low response rate

– Hard to get all people in household if needed

Exercise

• Think of a behaviour you would like to measure

• Discuss with a partner

– Best method of collection

• Strengths

• Weaknesses


• Weaknesses

3. Sample Design in the Survey Process

SampleDesign

PilotSurvey

Conductof

Survey


SurveyInstrument

Design

PreliminaryPlanning


DataCoding Data

Editing




Tidying-Up

Sampling Methods

Preliminary Concepts

• What is a sample?

– a collection of things which is some part of a larger population and which is selected so as to be representative of some or all of that population

• Target Population

– who are we trying to survey?


• Sampling Units

– what are we going to sample?

• Sampling Frame

– where are we going to get a list of these things?

Sampling Methods

Sampling Frame

• a base list to identify the sampling units

• should contain all the sampling units

• examples,

– all households on a street (e.g. Council records)

– telephone directories


– telephone directories

– mailing lists

– maps

– electoral rolls

– blocklists

Sampling Methods

Sampling Frame Problems

• inaccuracy

• incompleteness

• duplication

• inadequacy

• out-of-date


• out-of-date

• Must check the reason for which the list was

originally compiled to understand likely deficiencies.

Sampling Methods

Sampling Error & Sampling Bias

• Sampling Error

– due to the simple fact that we are taking a sample, and not

the population.

– can minimise error by taking larger sample.


• Sampling Bias

– due to systematic omission of some elements from our

final sample.

– cannot minimise error by taking larger sample.

Sampling Methods

Random Sampling

• Each unit is selected independently

and each unit in the population

has an equal probability of being selected.


• Must use random sampling to avoid sampling bias.

Sampling Methods

Random Sampling Methods

• Simple Random Sampling

• Stratified Random Sampling

• Variable Fraction Stratified Random Sampling

• Multi-stage Sampling


Multi-stage Sampling

• Cluster Sampling

• Systematic Sampling

• Note quotas not on list

Sampling Methods

Sample Size

• How much data do we need?

• Too much data >>> too expensive

• Not enough data >>> not able to draw conclusions


• Somewhere in the middle is a sample size which

enables us to draw sufficient conclusions at a

reasonable cost

Sampling Methods

Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.

Exercise

• Hand out random sampling sheet

– Explain

– Questions?


4. Survey design in the survey process

SampleDesign

PilotSurvey

Conductof

Survey


SurveyInstrument

Design

PreliminaryPlanning


DataCoding Data

Editing




Tidying-Up

Instrument design for reliable measurement

• Question content

• Question types

• Physical design - also for observation/counting


Question Content

• Reliability

– repeatable

– easy to answer

• Accuracy

– no question bias


– measures what we want

• Relevance

– must appear relevant to respondent

Question Types

• Factual

– “What did you do?”

• Classification (e.g. socio-demographic)

– for comparing with secondary data

• Opinion and attitude questions


• Opinion and attitude questions

– “What do you think about ……?”

• Stated Response Questions

– “What would you do if ……?”

Physical Design of Forms/Apps

• Observational surveys

– Ergonomic

– Size/format – not too big or small

– Weather-proof

– Need log forms

Test under actual conditions


– Test under actual conditions

Physical Design of Forms

• Self-administered forms

– Layout vital

– Minimal writing should be required

– No coding aids should appear

– Instructions very clear

– Professional appearance


Professional appearance

– Include ID number

35

5. Pilots in the survey process

SampleDesign

PilotSurvey

Conductof

Survey


SurveyInstrument

Design

PreliminaryPlanning


DataCoding Data

Editing




Tidying-Up

Pilot Surveys

• Why not do a pilot survey?

– too expensive

– not enough time

• Why do a pilot survey?

– too expensive to omit it


– too expensive to omit it

– not enough time to omit it

Pilot Surveys

• pilot survey is a test of ALL aspects of design

• scope for experimental design

• saves expensive mistakes


Uses of the Pilot Survey

• "Skirmishing" of wording

• Adequacy of questionnaire– definitions clear?

– too many "don't knows“?

– too long?

– open to closed questions


• Efficiency of interview/surveyer training

• Non-response rate

• Analysis

• Cost and duration

6. Survey implementation

SampleDesign

PilotSurvey

Conductof

Survey


SurveyInstrument

Design

PreliminaryPlanning


DataCoding Data

Editing




Tidying-Up

Conducting the Surveys

• Need high response rate for validity

• Consider

Gross sample . . . . . . . . . . . . . . . . . . = 100 (houses who are eligible to put out bin)

Sample loss (vacant, invalid phone no.) = 5 (vacant houses)

Net sample . . . . . . . . . . . . . . . . . . . . . . = 95

Responses . . . . . . . . . . . . . . . . . . . . . . = 50 (put out green bin)

Response rate . . . . . . . . . . . . . . . . . . . . = response/net sample = (50/95) = 53%


• Consider

– Announcement letter/message >> higher response

– Follow-up regime >> higher response rate

Exercise – dot points for a survey you need

• What method?

• How to get a sampling frame?

• What questions? Need any for weighting?

• Other issues/questions?


– Richardson, Ampt, Meyburg (1995)

– http://www.geog.ucsb.edu/~deutsch/geog111_211a/code_books/Survey_Methods_For_Transport_Planning.pdf

7. Weighting (correction)/Expansion

SampleDesign

PilotSurvey

Conductof

Survey


SurveyInstrument

Design

PreliminaryPlanning


DataCoding Data

Editing




Tidying-Up

Weighting & Expansion of Data

• Getting the sample data to represent the population

from which it was drawn, as nearly as possible

• Why? – systematic errors

– Non-response >> weighting certain type of respondents

higher


higher

– Missing Data >> can make assumptions – or note

– Inaccurate Reporting >> e.g. social desirability bias

Example of weighting

• Your response is 50/95 – what about the 45?

Say 30 males (67%) 15 females (33%)

- Your secondary data (e.g. counts, other data)

males 50% females (50%)

missing responses from females


– missing responses from females

- Responding females are therefore ‘weighted’ with a

slightly higher value (1.5) males (.75)

Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.

Summary

• To measure behaviour change it is essential to understand the data collection and survey process

• In particular need to understand:

– Survey method

– Sampling principles


Sampling principles

– Importance of a pilot

– Implementation options

– Need for weighting

• Vital for future funding as well as for sharing methodologies

• [email protected]


Evaluating behaviour change programs - WasteMINZ · Evaluating behaviour change programs Liz Ampt...

Documents

Transcript of Evaluating behaviour change programs - WasteMINZ · Evaluating behaviour change programs Liz Ampt...