Task 3: Reliability and Constraints

OCR Nationals Level 3 Unit 3

March 2012 M Morison

Objectives

Know the different types of errors that can affect a study

Understand why it is necessary to identify potential errors and constraints

Be able to identify and describe some of the constraints and areas for potential errors in your own study, and identify ways in which you can prevent these from affecting your study

March 2012 M Morison

Deadline for this unit:

 A02 - Design and Carry Out a data collection activity

Monday April 2nd

March 2012 M Morison

Pass/Merit/Distinction

Pass Candidates will list some of the constraints that could affect the reliability of their study.

MeritCandidates will describe most of the constraints that could affect the reliability of their study and identify some areas of potential error in their sampling regime.

DistinctionCandidates will describe the constraints that could affect the reliability of their study and identify areas of potential error in their sampling regime.

4

March 2012 M Morison

Introduction

No matter reliable your sample size is or how good your data collection method is, there is always the potential for errors to occur.

There are several different types of errors: Sampling errors Design errors Human errors Respondent errors

March 2012 M Morison

Sampling error

The sample selected may be too small to get a reliable enough range of data;

It may be too small to be truly representative;

An inappropriate sampling method may have been used;

March 2012 M Morison

Solutions

You may need to expand your sample and collect more data.

You may need to begin your data collection again using a different method.

March 2012 M Morison

Sampling error

Generally decreases as the sample size increases (but not proportionally)

Depends on the size of the population under study

Depends on the variability of the characteristic of interest in the population

Can be accounted for and reduced by an appropriate sampling plan

March 2012 M Morison

Design errors

If there is an error in the questionnaire design, this could cause problems with the respondent's answers

If you ask the wrong questions, you won’t get the data you require

This in turn, can create processing errors

Processing errors can also arise through incorrect calculations /functions/formulae within your spreadsheet

These types of errors often lead to a bias in the final results.

March 2012 M Morison

Solutions Pilot your questionnaire to

make sure it returns the data you are looking for

Give exemplar formats for answers – make it clear that when you ask for gender you are expecting M or F, for example

Test your spreadsheet out with test data first and check your answers against a calculator to make sure your functions and formulae are working correctly

March 2012 M Morison

Human error

People make mistakes! Questions could be

asked incorrectly Answers could be

recorded incorrectly The person collecting the

data might select the wrong people /inappropriate people to interview

March 2012 M Morison

Solutions

Pilot your questionnaire to make sure it returns the data you are looking for

Give exemplar formats for answers – make it clear that when you ask for gender you are expecting M or F, for example

Thorough training for anyone who is collecting data on your behalf

March 2012 M Morison

Response errors

Poor questionnaire design It is essential that sample survey or

census questions are worded carefully in order to avoid introducing bias.

If questions are misleading or confusing, then the responses may end up being distorted.

For more information, refer to this section on Questionnaire design.

March 2012 M Morison

Response errors (cont’d)

Interview bias An interviewer can influence how a respondent

answers the survey questions. This may occur when the interviewer is too

friendly or too aloof, or prompts the respondent. To prevent this, interviewers must be trained to

remain neutral throughout the interview. They must also pay close attention to the way they ask each question. If an interviewer changes the way a question is worded, it may impact the respondent's answer.

March 2012 M Morison

Respondent errors

Respondents can provide incorrect answers.

Faulty recollections, tendencies to exaggerate or underplay events, and inclinations to give answers that appear more 'socially desirable' are several reasons why a respondent may provide a false answer.

March 2012 M Morison

Non-response errors

Non-response errors are the result of not having obtained sufficient answers to survey questions. There are two types of non-response errors: complete and partial.

March 2012 M Morison

Complete non-response errors

Complete non-response errors The results fail to include the

responses of certain units in the selected sample. the respondent is unavailable or

temporarily absent the respondent is unable or refuses to

participate in the survey

If a significant number of people do not respond to a survey, then the results may be biased since the characteristics of the non-respondents may differ from those who have participated.

March 2012 M Morison

Partial non-response errors

This type of error occurs when respondents provide incomplete information.

For certain people, some questions may be difficult to understand.

To reduce this form of bias, care should be taken in designing and testing questionnaires.

March 2012 M Morison

Non-response errors (cont’d)

The most common non-response error is caused by people simply not returning the questionnaire. You may need to increase your

proposed sample size to cover the people who may not respond.

Another common cause is people not understanding the questions. This can be avoided by including a

help sheet with the questionnaire, or comments on the questionnaire itself.

March 2012 M Morison

Ensuring reliability

To ensure the reliability of your data, there are some questions you need to ask yourself: What is the source of the data? Did you collect it

yourself, or was it collected by somebody else on your behalf?

Could the subjects have a reason for misrepresenting the information/ answering incorrectly?

When you are entering your data into your spreadsheet, is it possible that the data might have been altered by anybody in any way?

Do you think the sample size was adequate? What is the level of sampling error?

Were the survey questions easy to understand?

March 2012 M Morison

Extraneous Variables

List potential errors that could affect results

How can you minimise these?