RELIABILITY AND VALIDITY

After we have chosen an appropriate method we must consider how useful the data from the study is likely to be.

Can we draw conclusions.

KEY ISSUES

Is it reliable? Is it valid? Any biases due to the sampling of the

population.

Issues with reliability. this links to one of the key features of the scientific approach, replicability.

If your not able to get reliable results then it is not being replicated.

So something has gone wrong.

We should be able to measure something time and time again and get similar or same results.

Any two measurements of the same thing should be consistent with each other.

HOW DO WE ASSESS RELIABILITY

External reliability: produce same results each time the test is carried out. test-retest method. You can also do a correlation between the two sets of scores. High correlation co-efficient indicates that the test is reliable. A low-correlation coefficient indicates poor reliability.

Internal reliability usually associated with measures such as attitude scales or personality tests. if a test you do has people scoring really high on one part but not the other then it has low internal consistency.

This type of reliability refers to the extent to which a measure is consistent within itself. The internal reliability of self-report measures, such as psychometric tests and questionnaires can be assessed using the split half method.

This involves splitting a test into two and having the same participant doing both halves of the test.  If the two halves of the test provide similar results this would suggest that the test has internal reliability.

IMPROVING RELIABILITY

When possible take more than one measurement from each participant.

Pilot studies. Standardise the way in which researchers

collect data, inter-rater reliability. Check data very carefully.

ISSUES OF VALIDITY

Internal validity is the ability of the study to test the hypothesis that it was designed to test.

Are we measuring the effects of the IV on the DV.

This leads to operationalisation of variables.

521

OPERATIONALISATION

The construction of actual, concrete measurement techniques; the creation of “operations” that will result in the desired measurements.

The development or choice of specific research procedures (operations) that will result in representing the concepts of interest.

Pg 521 nice example

ANOTHER EXAMPLE

To see the effect of happiness on success in exams you would need to operationalise.

Happiness- score on a scale of happiness Success in exams-The score they have on A

Level exams.

THREATS TO INTERNAL VALIDITY

To improve internal validity you need to remove all confounding variables.

Demand characteristics are a threat to internal Validity.

Orne 1962 see page 521 Experimenter bias. Design used, Repeated measures and

boredom, Independent group introduces participant variables.

IMPROVING INTERNAL VALIDITY

Demand Characteristics. Single Blind Technique. the Ps do not know the hypothesis they are in or what condition they are in

Investigator effects. Double Blind Technique. A procedure in an experiment where neither the Ps nor the Experimenter know the hypothesis or the condition. This would involve someone else carrying out the data collection on behalf of the experimenter.

Rosenthal and Fode (1963)

522

ASSESSING VALIDITY

Face validity, whether or not the test is based on subjective interpretation does it appear to measure what it has been designed to measure

Criterion Validity is more objective and looks at whether a test of a particular construct relates to other measures of it.

Two types. concurrent validity: comparing the results of a

new test with that of an old test known to have good validity.

Predictive Validity. this is the ability to predict performance on future tests.

523

EXTERNAL VALIDITY

How well can the results of the study be generalised outside of the study itself.

Population Validity generalised from the sample to other populations.

Ecological Validity Other situations places and conditions

Mundane realism is real life. Why are the above two confused. Just

copying mundane realism does not necessarily improve ecological validity because it can create demand characteristics.

523

Thurston et al 2008 not truly a non clinical sample in mental health research.

Researchers are more interested in experimental realism. The degree to which the results reflect realistic behaviour.

There is a link between Validity and reliability. Unreliable measures do not have internal validity.

Measures can be reliable but not valid. Lovely example in the book.

524

Internal versus External validity

Trade off. The more you control confounding variables the less likely it will have external validity.

More you allow for external validity the more you allow for confounding variables.

This is a great way of thinking about evaluating studies.

It all depends on what your doing. In general we like internal validity. If we want to apply the results we will want

external Validity.

Issues with sampling.

Who is the sample. Is it representative. Banyard and Hunt 2000 Are volunteer samples representative.

524