1

Chapter 5: Developing a Measurement Strategy

• Scales of Measurement

• Reliability and Validity

• Modalities of Measurement

• Locating and Evaluating Measures

2

Reliability & Validity

• Why is reliability important?– Theory can’t do without it– Constructs can’t be valid unless they are reliable– A phenomenon must be reliably demonstrated before it can have

construct status– E.g. ESP, Loch Ness Monster

• What are some examples of phenomena important to:– Counselors?– I/O psychologists?

3

Reliability & Validity

• Manifest variables: – Directly observable

• Hypothetical Constructs:– Not directly observed (inferred)

• What are some examples?– Behavioral?

– Psychological constructs?

• Which types (manifest/hypothetical) are:– Operational definitions?

– Hypothetical constructs?

• How are Operational definitions and Hypothetical constructs related?– Operational definitions represent hypothetical constructs

4

Reliability & Validityand measurement error

• Reliability & Validity– Reliability – consistency– Validity – accuracy– Polygraph

• Reliable and valid?

– Can something be reliable but not valid?– Can something be valid but not reliable?

• Measurement ErrorXo= T+E (Observed score = True score + Error)

Randon v. systematic error

What kind of error is introduced when an applicant fakes on a personality measure? Other examples?

5

Assessing Reliability

• Forms– Test-retest– Alternate forms (equivalent forms)– Interrater reliability– Internal consistency

• Split half

• Cronbach’s alpha (coefficient alpha) – mean r among all items

• Which one to use?– Give some examples

• Standards for reliability– What’s acceptable?

6

Assessing Validity

• Validity– Inferred– Depends on different types of evidence– Varies in how much (degree)– Trichotomized into: high, moderate, low– Specific to what it’s used for (valid for what?)– refers to inferences drawn, not the measure itself– A unitary construct (with three types of evidence)

• Content,

• criterion-related (predictive, concurrent)

• Construct

7

Validity

• Convergent– Related to other constructs it should be related to

• Divergent– Not related to other constructs it should not be related to

which should be convergent and which divergent?– IQ and Depression– Work motivation and conscientiousness– Happiness and wealth– Need for approval and caring about how one looks– Ability to sell and friendliness– Aggression and frustration level

8

Validity:Determining Degree of Validity

• Measure validation process– See figure 5-4

• Theory of trait• Test hypotheses • Confirm/disconfirm• Do it again• Never ending (seemingly)

– Eg. CPI, WGCTA, WPT, NEO, Beck DI

9

Validity: Differential

• Valid for specific population, group?– Moral reasoning?

• Different cultures, Gender (Gilligan, ’82)?

– Math tests?• Gender?

– PONS test?• Gender?

– Personality inventories? • Fakers v. honest?

• Job setting v. home setting?

10

Reliability & Validity: Modalities of Measurement

• Self-Report Measures– Advantages– Limitations

• Behavioral Measures– Advantages– Limitations

• Physiological Measures– Advantages– Limitations

• Choosing a Measurement Modality

11

Modalities:

• Self-Report– Cognitive– Affective– Retrospective v. Hypothetical (behavioral intention)– Kinesthetic

• Advantages– Easy to collect– Easy to administer– Feelings (cannot observe)– Thought process (Policy capturing)– May be more accurate than observation. Why?

• Limitations– Accuracy of recall– Willingness to report– Verbal skills needed

12

Self Report Modalities:

• How to– Ask about actual previous behaviors– Use dichotomous or categorical for or behavior or behavioral

intentions– Use Likert (5, 7 or 9) for attitudes– Include both positively and negatively scaled items– Use graphic labels or benchmarks– Use multiple items to capture a concept– Avoid leading double barreled questions (usually)

13

Modalities:Behavioral Measures

• Uses– Behavior is object of study– Operational definitions– Nonverbal may be clue to feelings/ physio state

• Types of measures– Frequency (bar presses)– Rate– Duration– Intensity– Accuracy– Persistence

• Examples?

14

Behavioral Measures

• Advantages– Surreptitious (sneaky)

• Avoids evaluation apprehension

– Capture automatic behaviors• Often not premeditated

– More accurate than behavioral intention

• Limitations– “what you see is what you get” What limitation is implied here?

• Interpretation is inferred by observer

– Highly situation-specific• That’s why R. Hogan likes to use personality trait as predictor

– Need trained observers

15

Modalities: Physiological Measures

• Purposes– Of interest in and of itself– As op definition of psychological state

• E.g. anxiety, arousal, lying

• Advantages– Most direct (no intervening human observer– Highly precise

• Limitations– Need equipment, trained administrators– Obtrusive– Constrain freedom– Source of unreliability (testing effects)

• E.g. polygraph

16

Modalities:Choosing & Locating Measurement Modality

• Choosing: Self-report, behavioral, physiological– Rank them for level of validity– Which one should be used?– See table 5-2 relative advantages and limitations– Multiple operationism

– Categories of Measures (figure 5-7, p. 147)• Type: manifest v. hypothetical• Hypothetical construct:

– Psychometric (used for individual scores)– research measures (used for mean scores; norms)

» Developed (use when it fits procedure)» Ad hoc (when you have to develop it)

17

Modalities: Locating & Evlauting Measures

• Locating Measures– MMY

• (Buros)

– Tests in Print– Directory of Unpublished Experimental Measures

• (Goldman & Sanders, 1997)

– Measures of Personality and Social Psychological Attitudes• (Robinson, Shaver,& Wrightsman, ’91)

– Organizational Measures• Warr, Cook and Wall??

18

Modalities: Evaluating Them

• Theoretical Background– How much construct standing?

• Quality of Development– Participant samples, norms

• Reliability & Validity– What are good coefficients for each?– Valid for what?

• Freedom from Response Bias– Social desirability– Acquiescence bias

19

Developing a Measurement StrategySummary

• Scales of Measurement

• Reliability and Validity

• Modalities of Measurement

• Locating and Evaluating Measures