The end of construct validity Denny Borsboom University of Amsterdam.

47
The end of construct validity Denny Borsboom University of Amsterdam
  • date post

    15-Jan-2016
  • Category

    Documents

  • view

    215
  • download

    0

Transcript of The end of construct validity Denny Borsboom University of Amsterdam.

Page 1: The end of construct validity Denny Borsboom University of Amsterdam.

The end of construct validity

Denny Borsboom

University of Amsterdam

Page 2: The end of construct validity Denny Borsboom University of Amsterdam.

Two kinds of validity

• The working researcher’s idea: Validity concerns the question of whether a test measures what it should measure

• The construct validity idea: Validity is an evaluative, integrated judgement of the degree to which test score interpretations are justified in the light of empirical evidence and theoretical rationales (and, possibly, social consequences that follow from test use)

Page 3: The end of construct validity Denny Borsboom University of Amsterdam.

What I will argue

• The working researchers’ conception is theoretically and practically superior

• The construct validity position has some sophication but that is mainly windowdressing; in general, it precisely misses the point of what validity is

Page 4: The end of construct validity Denny Borsboom University of Amsterdam.

The pillars of construct validity

Construct validity is – an evaluative judgement – about ‘test score interpretations’– in terms of ‘constructs’– that is a function of evidence– and a matter of degree

• I will argue that this view a) does not align with the working researcher’s view at

allb) has quite unreasonable consequences that one

should not be comfortable with

Page 5: The end of construct validity Denny Borsboom University of Amsterdam.
Page 6: The end of construct validity Denny Borsboom University of Amsterdam.

Why construct validity theory is dysfunctional

Page 7: The end of construct validity Denny Borsboom University of Amsterdam.

The social consequences of construct validity theory

Page 8: The end of construct validity Denny Borsboom University of Amsterdam.

The social consequences of construct validity theory

Page 9: The end of construct validity Denny Borsboom University of Amsterdam.

A black hole that traps all psychometric problems

Page 10: The end of construct validity Denny Borsboom University of Amsterdam.

Why construct validity has nothing to do with tests

(and why this is wrong)

Page 11: The end of construct validity Denny Borsboom University of Amsterdam.

Every interpretation can have construct validity

Page 12: The end of construct validity Denny Borsboom University of Amsterdam.

There as as many ‘construct validities’ as there are judges

Page 13: The end of construct validity Denny Borsboom University of Amsterdam.

Measurement instruments can ‘become valid’

Page 14: The end of construct validity Denny Borsboom University of Amsterdam.

Some measurement instruments ‘were valid’...

Page 15: The end of construct validity Denny Borsboom University of Amsterdam.

...but then ‘ceased to be’ valid...

Page 16: The end of construct validity Denny Borsboom University of Amsterdam.

Reference is unimportant

‘Aether’

‘Black hole’‘Phlogiston’

‘DNA’

Page 17: The end of construct validity Denny Borsboom University of Amsterdam.

Validity depends on the presence of ‘interpreters’

Page 18: The end of construct validity Denny Borsboom University of Amsterdam.

How construct validity is sold

• Construct validity is an evaluative, integrated judgement of the degree to which test score interpretations are justified in the light of empirical evidence and theoretical rationales (and, possibly, social consequences that follow from test use)

Page 19: The end of construct validity Denny Borsboom University of Amsterdam.

What construct validity really is

• Somebody’s evaluative, integrated and fluctuating judgement of the degree to which test score interpretations, that may have nothing to do with measurement, are justified in the light of time-dependent empirical evidence and that person’s theoretical rationales (and, possibly, that person’s guesses about social consequences that follow from test use as well as his or her valuation of these outcomes)

Page 20: The end of construct validity Denny Borsboom University of Amsterdam.

Why all this sophistication misses the point

Page 21: The end of construct validity Denny Borsboom University of Amsterdam.

• Construct validity is an evaluative, integrated judgement of the degree to which test score interpretations are justified in the light of empirical evidence and theoretical rationales (and, possibly, social consequences that follow from test use)

• However, validity is... – a property, not a judgment– a property of instruments, not of inferences– a function of truth, not of evidence– the object of validation research, not its result

Page 22: The end of construct validity Denny Borsboom University of Amsterdam.
Page 23: The end of construct validity Denny Borsboom University of Amsterdam.

A simple alternative:

• A test is valid for measuring an attribute if and only if variation in the attribute causally produces variation in the measurement outcomes

Page 24: The end of construct validity Denny Borsboom University of Amsterdam.
Page 25: The end of construct validity Denny Borsboom University of Amsterdam.
Page 26: The end of construct validity Denny Borsboom University of Amsterdam.

Attributestructure

Page 27: The end of construct validity Denny Borsboom University of Amsterdam.

Attributestructure

Page 28: The end of construct validity Denny Borsboom University of Amsterdam.

Attributestructure

Scorestructure

Page 29: The end of construct validity Denny Borsboom University of Amsterdam.

Attributestructure

Responseprocess

Scorestructure

Page 30: The end of construct validity Denny Borsboom University of Amsterdam.

g

Responseprocess

IQ-scores

7082

99 115 134

Page 31: The end of construct validity Denny Borsboom University of Amsterdam.

7082

99 115 134

g

Responseprocess

IQ-scores X

f(X| )

Page 32: The end of construct validity Denny Borsboom University of Amsterdam.

g

Responseprocess

IQ-score patterns

X

f(X| )

Substantivetheory

Formalmodel

Page 33: The end of construct validity Denny Borsboom University of Amsterdam.

g

Responseprocess

IQ-score patterns

X

f(X| )

Substantivetheory

Formalmodel

Page 34: The end of construct validity Denny Borsboom University of Amsterdam.

Where to look for validity

• Traditionally, evidence for validity is sought in external relations: relations between test scores and other test scores

• In criterion validity the evidence comes from correlations with a criterion (or with the criterion)

• In construct validity, the evidence comes from correlations with lots of other variables (MTMMs)

Page 35: The end of construct validity Denny Borsboom University of Amsterdam.

IQ-scores Job performance.30

Genetic differences

.50

Working memory

.40

Annual income

.15Extraversion

Numerical ability SESPhysique

Sex

Race

Length

Annual income

Masculinity

Attractiveness

.41

.20

.37

.55

.78

.56

.35

.09

Visual memory

But even if we knew all correlations between all conceivable tests, the validity problem would remain

Page 36: The end of construct validity Denny Borsboom University of Amsterdam.

Where to look for validity

• Validity is not a matter of external relations between the test scores and other test scores

• It is a matter of which processes take attribute differences into response differences

• For many tests we have no idea of what happens between item administration and item response

• This is the reason that the validity problem has proven hard to crack

Page 37: The end of construct validity Denny Borsboom University of Amsterdam.

Where to look for validity

• Ingredients for validity:

– A theory on the structure of the attribute– A theory on the processes that take levels of the

attribute into observed score patterns– A formal model to test the theory against data

• The question of validity then becomes: is this theory true?

Page 38: The end of construct validity Denny Borsboom University of Amsterdam.

Example: The balance scale test

What happens when the blocks are removed ?

Weight item

Distance item

Conflict Weight item

Page 39: The end of construct validity Denny Borsboom University of Amsterdam.

Example: The balance scale test

• Theory on the structure of the attribute: – Cognitive development involves an ordered series of discrete

transitions between stages

• Theory on the processes that take levels of the attribute into observed score patterns:

– Children in different developmental stages use different cognitive rules to solve balance scale items, which results in different response patterns

• Statistical model to test the theory against data– Developmental stages are conceptualized as latent classes

with theoretically driven response vectors

Page 40: The end of construct validity Denny Borsboom University of Amsterdam.

001100001100

110011 110011 111100

Developmentalstages

Responseprocess

Balance scaleTest scores

X

Latent classes

P(X=x| )

Rule 1 Rule 2 Rule 3

Page 41: The end of construct validity Denny Borsboom University of Amsterdam.

The question of validity:

Is this theory of response behavior correct?

Page 42: The end of construct validity Denny Borsboom University of Amsterdam.

• The validity concept is usually applied to many questions simultaneously:

1) Does the test measure the intended attribute?2) How well do the test scores predict other attributes?3) Is the use of the test legally defensible?4) Will using the test improve the human condition?

• which are put under one umbrella; I only deal with (1)

• (2-...) are better left to psychometrics, law, politics, etc.

How does this relate to other issues?

Page 43: The end of construct validity Denny Borsboom University of Amsterdam.
Page 44: The end of construct validity Denny Borsboom University of Amsterdam.

Does this mean that other issues are unimportant?

• No. Interpretations, uses, and consequences matter a great deal

• But they are not thereby issues of validity

• Moreover, they usually belong in the public sphere, not in the domain of validity theory

Page 45: The end of construct validity Denny Borsboom University of Amsterdam.

Bottom line

• To find out what you measure, you have to find out how your instrument works - there is no other way

• If you know how your instrument is supposed to work, and you know how it works, you have a definite answer to the validity problem

• However, if don’t know how your instrument is supposed to work, and you don’t know how it works, you are in trouble

Page 46: The end of construct validity Denny Borsboom University of Amsterdam.
Page 47: The end of construct validity Denny Borsboom University of Amsterdam.

Validity is...

...measuring the right thing