The meaning of The meaning of ReliabilityReliability and and

ValidityValidity in in psychological psychological

researchresearch

White board timeWhite board time

Write down the Write down the meaning of meaning of reliability reliability

Clue: If your car is reliable Clue: If your car is reliable what does this mean?what does this mean?

Now write Now write down the down the meaning of meaning of validityvalidity

The point she made was valid!The point she made was valid!

Imagine you’ve produced a new Imagine you’ve produced a new test to measures people’s reading test to measures people’s reading ability but each time you take the ability but each time you take the test it gives different results.test it gives different results.

Obviously this means your new Obviously this means your new reading test isn’t any good!reading test isn’t any good!

Why isn't this reading test any good?Why isn't this reading test any good?

What if….What if….

You get one score when I mark You get one score when I mark the test but a completely the test but a completely different score when another different score when another person marks it! person marks it!

Clearly these scores are not Clearly these scores are not veryvery

____________________________?____________________________?

So if the results are So if the results are inconsistent they cannot be inconsistent they cannot be relied on because they are not relied on because they are not trustworthy. trustworthy.

Psychologists would say these Psychologists would say these results lack ‘reliabilityresults lack ‘reliability’.’.

AlsoAlso

The test doesn’t The test doesn’t measure what it set measure what it set out to measure so it out to measure so it lacks validity as lacks validity as well!well!

In order to conduct research, In order to conduct research, psychologists must find ways of psychologists must find ways of measuring things. measuring things.

Many instruments of measurement Many instruments of measurement we are already familiar with like we are already familiar with like bathroom scales, rulers, bathroom scales, rulers, speedometers etc. speedometers etc.

But for these things to be useful we need But for these things to be useful we need to be able to trust them! to be able to trust them! The instrument The instrument of measurement has to be reliableof measurement has to be reliable

It must measure what we want it to measure It must measure what we want it to measure and it must be and it must be consistent,consistent, in other words in other words you should measure the same height you should measure the same height wherever and whenever you measure wherever and whenever you measure yourself – Your height will stay the same yourself – Your height will stay the same and should be consistent provided your and should be consistent provided your tape measure is reliable that is!tape measure is reliable that is!

Reliability refers to the Reliability refers to the consistencyconsistency of a measuring of a measuring instrument.instrument.

If a measurement is reliable it will If a measurement is reliable it will be consistent and stable.be consistent and stable.

If a If a measurementmeasurement is reliable we is reliable we can trust the results.can trust the results.

How can we be sure if a How can we be sure if a measurement is reliable?measurement is reliable?

Reliability can be checked using correlational Reliability can be checked using correlational techniquestechniques

For example we could use the For example we could use the Test-Retest Test-Retest methodmethod

Participants take a test twice and if the test is Participants take a test twice and if the test is

reliablereliable the two scores will be highly correlated, so the two scores will be highly correlated, so if they score high in one test they should score high in if they score high in one test they should score high in the other one.the other one.

We say a piece of research is reliable if we can We say a piece of research is reliable if we can replicate and get similar results replicate and get similar results

Which is likely to be more Which is likely to be more reliable?reliable?

((show a high positive correlation between show a high positive correlation between the two scores)the two scores)

A.A. The correlation of the marks from two The correlation of the marks from two examiners marking the same multiple choice examiners marking the same multiple choice test?test?

OROR

B.B. The correlation of marks awarded to the The correlation of marks awarded to the same essay by two different examiners?same essay by two different examiners?

Can you explain why?Can you explain why?

The agreement on the multiple choice The agreement on the multiple choice test should be high, that is there test should be high, that is there should be a high positive correlation should be a high positive correlation between marks obtained by the two between marks obtained by the two sets of scorers.sets of scorers.

On the other hand, if the two scorers On the other hand, if the two scorers were marking essays then there were marking essays then there would probably not be such a high would probably not be such a high correlation. correlation.

Inter–rater reliabilityInter–rater reliability

This type of reliability is concerned This type of reliability is concerned with how closely different people who with how closely different people who are marking a test or observing a are marking a test or observing a behaviour agree with each other. behaviour agree with each other.

It is not therefore concerned with the It is not therefore concerned with the

reliability of the test itself (test-retest) – reliability of the test itself (test-retest) – it is more to do with how much we can it is more to do with how much we can trust the person doing the marking!trust the person doing the marking!

In observation studies it is often necessary for In observation studies it is often necessary for trainingtraining to take place in order to be able to to take place in order to be able to establish the highest level of inter-rater/observer establish the highest level of inter-rater/observer reliability as possible.reliability as possible.

Pilot studies are useful in this instance – Pilot studies are useful in this instance – observers can try out the observation observers can try out the observation schedule and then compare their results schedule and then compare their results with another observer. If the observation with another observer. If the observation schedule is a reliable measurement tool the schedule is a reliable measurement tool the observers will have similar results… and observers will have similar results… and observers can ensure they are looking for observers can ensure they are looking for the exact same things!the exact same things!

ValidityValidity The validity of a psychological The validity of a psychological measure is the extent to which it measure is the extent to which it

measures what it intended to measures what it intended to measure – measure –

Sometimes even experienced Sometimes even experienced researchers have accidently researchers have accidently

measured something else insteadmeasured something else instead

If you can think of If you can think of any other possible any other possible explanationexplanation for your findings then it could be for your findings then it could be that your research isn’t that your research isn’t validvalid. .

For example does an IQ test actually tell us For example does an IQ test actually tell us how intelligent a person is? Or is it actually how intelligent a person is? Or is it actually measuring something else? Like how well a measuring something else? Like how well a person can read English or how good their person can read English or how good their general knowledge is. general knowledge is.

If an IQ test doesn’t actually measure If an IQ test doesn’t actually measure INTELLIGENCE then it is not VALID.INTELLIGENCE then it is not VALID.

A tape measure may be a A tape measure may be a reliablereliable measurement tool – measurement tool – butbut if we were using it to if we were using it to measure the circumference measure the circumference of peoples’ heads as a way of of peoples’ heads as a way of measuring intelligence it measuring intelligence it wouldn’t be very wouldn’t be very valid! valid!

Testing validityTesting validity The easiest way to discover whether a test is valid is The easiest way to discover whether a test is valid is

to examine it and decide whether it to examine it and decide whether it lookslooks as as though it is. though it is.

If a test which was supposed to measure If a test which was supposed to measure intelligence contained a large memory test section intelligence contained a large memory test section then it would not have FACE VALIDITY (it looks then it would not have FACE VALIDITY (it looks wrong)wrong)

Clearly it LOOKS more like a memory test than an Clearly it LOOKS more like a memory test than an intelligence test intelligence test

(assuming of course that intelligence is more than just (assuming of course that intelligence is more than just having a good memory).having a good memory).

You can always ask an You can always ask an expert!expert!

If independent experts in the subject If independent experts in the subject area agree that the test looks as area agree that the test looks as though it does what it is supposed to though it does what it is supposed to do, then the test is said to have do, then the test is said to have CONTENT VALIDITY.CONTENT VALIDITY.

Check your Check your understandingunderstanding

The extent to which a The extent to which a test measures what test measures what

was was intendedintended is is known as what?known as what?

Answer: Answer: ValidityValidity

The extent to which the The extent to which the measurement object of a test measurement object of a test is is self-evidentself-evident is known as is known as what?what?

Face validity Face validity Content Validity Content Validity Construct validityConstruct validity

Answer: Answer:

Face Face ValidityValidity

Reliability means the Reliability means the measurement is? measurement is?

a)a)Valid Valid b)b)Consistent Consistent c)c)Meaningful Meaningful d)d)UsefulUseful

Answer: Answer: ConsistentConsistent