Measurement
description
Transcript of Measurement
![Page 1: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/1.jpg)
Measurement
![Page 2: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/2.jpg)
What is measurement?
“the assignment of a value on a variable to a unit of measurement in accordance with an operational definition” (Kleinnijenhuis 1999: 83 in Pennings et. al. 1999)
![Page 3: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/3.jpg)
Components of measurement
– A value: a categorization or number (e.g. whether or not a country is a democracy or not; or level of democratization)
– A variable: a characteristic that can vary in value among observations
– A unit of measurement: an observation/case (e.g. a country in a particular year)
– An operational definition: a definition of a concept that facilitates the assignment of a value to an observation (e.g. what countries would be classified as democracies)
![Page 4: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/4.jpg)
From concept to operational definition
• Reduce abstraction to facilitate measurement• E.g. Political party positions / electoral appeals
– Concept: the salience of themes to political parties
– Operational definition: the percentage of a manifesto devoted to each of a set of defined themes (including “free enterprise”, “welfare state expansion”, “expansion of military capabilities”)
![Page 5: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/5.jpg)
Levels of measurement
• Nominal– Categories with no rankings (e.g. types of democracies / types of
political parties / ethnic groups) - MECE
• Ordinal– Some observations score higher/lower than others, but we do
not know how much more or less (e.g. Left-Right position of parties / intensity of attachment to ethnic group)
• Interval– The differences between the values of the observations are
meaningful
• Ratio– There is a meaningful zero point (e.g. numbers of decisions
taken by referenda/ cross border trade)
![Page 6: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/6.jpg)
Guidelines for constructing measures
• Reveal assumptions implicit in operationalising a concept: often nothing intrinsic in a concept that implies a particular level of measurement
• Maximise information: Wherever sensible, aim for as high a level of measurement as possible
• Address trade-off between comparability and descriptive richness
![Page 7: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/7.jpg)
Lijphart’s (1977) typology of democratic systems
Structure of society
Elite behaviour
Homogeneous Plural
Coalescent Depoliticized:Austria (1966-)
Consociational:Belgium, Netherlands, Switzerland, Austria (1945-66)
Adversarial Centripetal:Finland, Denmark, UK, USA, Norway, Sweden, Germany
Centrifugal:France, Italy, Canada
![Page 8: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/8.jpg)
Evaluating measurement
• Validity and reliability
• Valid: the measure does measure the concept it purports to measure?– Face validity– Correlational validity (internal validity)– Predictive validity (external validity)
![Page 9: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/9.jpg)
An example of a correlational validity test
• Compared key-informants judgements on “what were the controversial issues” in a particular decision situation with documentation– Key informants identified five controversial issues in
decision making on a legislative proposal in the EU (the tobacco directive)
– To what extent and in what way were these related to the controversial issues identified through analysis of documentation?
For details see:
http://www.dhv-speyer.de/tkoenig/DEU_manuscript_Oct2005.htm
![Page 10: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/10.jpg)
E.g. of correlational validity test cont.
Related to experts’ issues?
Issues from documents
Actors with positions on
point of contention.
Average (range)
Number of Council meetings at which point was raised.
Average (range)
Related 8 7.9 (4-17) 3.0 (1-6)
Unrelated 23 4.8 (2-9) 1.7 (1-4)
31 controversial issues on the basis of documentation
5 issues from the informants
How many of the 31 were related to the 5?
What distinguished document-sourced issues that were related to informants’ issues from those that were not?
![Page 11: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/11.jpg)
Reliability
• Reliable: repeated measures on the same cases give the same results– Intra observer reliability– Inter observer reliability
![Page 12: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/12.jpg)
Another example of testing validity and reliability
• The enactment of election pledges in a coalition system of government: The Netherlands
(Thomson, R. 2001. The programme to policy linkage: The fulfilment of election pledges on socio-economic policy in the Netherlands, 1986-1998. European Journal of Political Research 40: 171-197)
![Page 13: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/13.jpg)
![Page 14: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/14.jpg)
Face validity
• Do the election pledges appear to be substantively important?
• Do the patterns of fulfilment match with qualitative descriptions of the main policy events in the period?
![Page 15: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/15.jpg)
Face validiity
![Page 16: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/16.jpg)
Reliability tests
• Inter-coder reliability– Did other readers of the same election
manifestos identify the same statements as pledges?
• Inter-coder reliability– Did subject-area specialists identify the same
pledges as unfilfilled or fulfilled?
![Page 17: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/17.jpg)
An inter-coder reliability testa / (a+b+c)
= 199/226
= .88
Statement identified as pledge by Thomson?
Yes No Total identified by 2nd coder
Statement identified as pledge by 2nd coder?
Yes 199 (a) 13 (b) 212
No 14 (c)
Total identified by Thomson
213
![Page 18: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/18.jpg)
Measurement error
• Aim at unbiased and efficient measures
• Bias: systematic error– We either overestimate or underestimate the
true value
• Inefficiency: Non-systematic error– On average, we estimate the true value
correctly, but with considerable variation
![Page 19: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/19.jpg)
The ideal is unbiased and efficient estimates
True value Measure/ estimate
Unbiased, but inefficient
Unbiased and efficient
Biased
![Page 20: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/20.jpg)
The impact of bias on inference
• Bias / systematic error– Consistent over or underestimation of the true
value– E.g. consistent overestimation of annual
income
• Leads to bias in description
• If it affects all cases equally, does not lead to bias in estimating causal effects
![Page 21: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/21.jpg)
The impact of inefficiency on inference
• Hampers description, although descriptions are correct on average
• Hampers explanation, but the impact differs depending on whether inefficiency pertains to the IV or DV
• Inefficiently measured DV – estimates of causal effects will be difficult, but correct on average (unbiased)
• Inefficiently measured IV – estimates of causal effects will be difficult and incorrect on average (i.e. biased)
![Page 22: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/22.jpg)
Why inefficiently measured DVs do not bias estimates of causal effects
IV: unemployment
DV: violence
![Page 23: Measurement](https://reader035.fdocuments.in/reader035/viewer/2022062722/56813a07550346895da1d389/html5/thumbnails/23.jpg)
Why inefficiently measured IVs bias estimates of causal effects
IV: unemployment
DV: violence