31 March 2009 @ Crawford School 1 Measurement – 1 Semester 1, 2009 POGO8096/8196: Research Methods...

31 March 2009 @ Crawford School 1

Measurement – 1Measurement – 1

Semester 1, 2009Semester 1, 2009

POGO8096/8196: Research POGO8096/8196: Research MethodsMethods

Crawford School of Economics and Government Crawford School of Economics and Government

231 March 2009 @ Crawford School

This weekThis week

Levels of research and unit of analysisLevels of research and unit of analysis Problems of measurement: Problems of measurement: reliabilityreliability

and and validityvalidity The basic problem of measurementThe basic problem of measurement Reliability and validityReliability and validity

Problems of measurement: Problems of measurement: precisionprecision Precision in measurementPrecision in measurement Precision in measuresPrecision in measures Quantitative vs. qualitative measuresQuantitative vs. qualitative measures


Unit of analysis – 1Unit of analysis – 1

In deciding what to measure for a concept of In deciding what to measure for a concept of your interest, it is very important to understand your interest, it is very important to understand what is what is the unit of analysisthe unit of analysis in your research. in your research.

The unit of analysis is the object that the The unit of analysis is the object that the “working hypothesis” describes. It can be a “working hypothesis” describes. It can be a person (in a survey/interview study), a country person (in a survey/interview study), a country (in a cross-national comparison), an (in a cross-national comparison), an organization (in a comparative case study of organization (in a comparative case study of organizational behavior), a year (in a organizational behavior), a year (in a longitudinal analysis of a given country), a longitudinal analysis of a given country), a policy (in a policy study), or something else.policy (in a policy study), or something else.


Unit of analysis – 2Unit of analysis – 2

The unit of analysis often determines how a variable iThe unit of analysis often determines how a variable in a hypothesis is “operationalized” (i.e., measured). n a hypothesis is “operationalized” (i.e., measured).

All variables in a working hypothesis must have the sAll variables in a working hypothesis must have the same unit of analysis.ame unit of analysis.

For a given hypothesis, there are many ways to operaFor a given hypothesis, there are many ways to operationalize depending on how you define the unit of anationalize depending on how you define the unit of analysis. For example, “The higher the income level, the lysis. For example, “The higher the income level, the more likely voters support the Republican party in the more likely voters support the Republican party in the US.” How can you operationalize this hypothesis? US.” How can you operationalize this hypothesis?


Answer 1: Individual-Answer 1: Individual-levellevel

The unit of analysis: IndividualsThe unit of analysis: Individuals A measure of income: Ask a survey A measure of income: Ask a survey

respondent, “What is your respondent, “What is your household’s annual income?” household’s annual income?”

A measure of partisanship: Ask the A measure of partisanship: Ask the survey respondent, “Did you vote for survey respondent, “Did you vote for a Republication candidate in the last a Republication candidate in the last presidential election?”presidential election?”


Answer 2: State-levelAnswer 2: State-level

The unit of analysis: 50 states in the USThe unit of analysis: 50 states in the US A measure of income: The average A measure of income: The average

household income in 2000, according the household income in 2000, according the Statistical Abstract of the United StatesStatistical Abstract of the United States. .

A measure of partisanship: The A measure of partisanship: The percentage of votes for a Republican percentage of votes for a Republican candidate received in the last candidate received in the last presidential election, according to presidential election, according to America VotesAmerica Votes..


What is a concept?What is a concept? What is a concept? (from What is a concept? (from American American

HeritageHeritage))1.1. A general idea derived or inferred from specific A general idea derived or inferred from specific

instances or occurrences.instances or occurrences.2.2. Something formed in the mind; a thought or Something formed in the mind; a thought or

notion.notion. Concept vs. measure in empirical researchConcept vs. measure in empirical research

A concept = an A concept = an un-observableun-observable (dependent or (dependent or independent) variable in a theory or a independent) variable in a theory or a hypothesis.hypothesis.

A measure = an A measure = an observableobservable (dependent or (dependent or independent) variable in a “working hypothesis.”independent) variable in a “working hypothesis.”


Concept vs. measuresConcept vs. measures

concept concept (unobservabl(unobservabl

e)e)

measuresmeasures

(observable)(observable)

economic economic growthgrowth

e.g., a change in the real e.g., a change in the real GDP per capita (purchasing GDP per capita (purchasing power parity) from 1991 to power parity) from 1991 to 2000 (“interval”)2000 (“interval”)

corruptioncorruption e.g., Transparency e.g., Transparency International (TI) corruption International (TI) corruption index (“ordinal”)index (“ordinal”)

government government typetype

e.g., presidential, e.g., presidential, parliamentary or parliamentary or authoritarian systems authoritarian systems (“nominal”)(“nominal”)


These are all concepts!These are all concepts! IndustrializationIndustrialization Income inequalityIncome inequality PovertyPoverty Technological Technological

progressprogress ProfitabilityProfitability

EfficiencyEfficiency AccountabilityAccountability EffectivenessEffectiveness

Government sizeGovernment size GlobalizationGlobalization DemocratizationDemocratization Civil societyCivil society

Clean airClean air Basic water needsBasic water needs SustainabilitySustainability RestorationRestoration


The basic problemThe basic problem

In any empirical research, we measure In any empirical research, we measure things that should correspond to what we things that should correspond to what we want to measure. But any want to measure. But any observableobservable variable we would choose to measure a variable we would choose to measure a certain certain unobservableunobservable concept corresponds concept corresponds only only indirectlyindirectly to that concept. This is the to that concept. This is the “basic problem of measurement” in social “basic problem of measurement” in social sciences.sciences.

The “basic problem” may result in critical The “basic problem” may result in critical error when making causal inference. (See error when making causal inference. (See the next slide for an example.)the next slide for an example.)


This is the source of a This is the source of a problem.problem.

A A consequenceconsequence


What to do?What to do? We must ensure, as much as possible, that the We must ensure, as much as possible, that the

relationships between concepts and measures relationships between concepts and measures are such that the relationship between the are such that the relationship between the measures mirrors the relationship between the measures mirrors the relationship between the concepts. For this purpose, we must try to use concepts. For this purpose, we must try to use “good” measures.“good” measures.

A measure is A measure is reliablereliable to the extent that it gives to the extent that it gives the same result again if the measurement is the same result again if the measurement is repeated. (Note: In reality, the measurement repeated. (Note: In reality, the measurement is almost always is almost always notnot repeated.) repeated.)

A measure is A measure is validvalid if it actually measures what if it actually measures what it purports to measure.it purports to measure.


Random vs. non-random Random vs. non-random errorerror

Reliability is a necessary but not sufficient Reliability is a necessary but not sufficient condition of validity, because a measure is condition of validity, because a measure is valid if it is free of both random and non-valid if it is free of both random and non-random errors, while a measure is reliable random errors, while a measure is reliable if it is free of random error alone. if it is free of random error alone.

If, on average, a given measure of a If, on average, a given measure of a concept tends to be true, we can assume concept tends to be true, we can assume that any error in the measure is that any error in the measure is randomrandom. .

By contrast, By contrast, non-randomnon-random error is error is systematic error that tends to, on average, systematic error that tends to, on average, distort a given measure of a concept. distort a given measure of a concept.


Reliable

Valid


Improving reliabilityImproving reliability Common sources of unreliabilityCommon sources of unreliability

clerical errors in official statisticsclerical errors in official statistics ambiguous questions in surveysambiguous questions in surveys dishonest interviewers and interviewees dishonest interviewers and interviewees

Careful work (e.g., careful check of data Careful work (e.g., careful check of data sources, careful check of survey questions, sources, careful check of survey questions, etc.) is the best way to achieve reasonable etc.) is the best way to achieve reasonable reliability. reliability.

There are several ways (e.g., test-retest There are several ways (e.g., test-retest check, split-half check) to test the reliability check, split-half check) to test the reliability of a measure (even without repeating the of a measure (even without repeating the measurement).measurement).


Improving validityImproving validity

Since all we can observe is always Since all we can observe is always measures, assessing the validity of the measures, assessing the validity of the measures is quite difficult, if not impossible. measures is quite difficult, if not impossible. It is also difficult to identify common sources It is also difficult to identify common sources of invalidity.of invalidity.

To achieve reasonable validity, you should To achieve reasonable validity, you should conduct a “face validity test” – conduct a “face validity test” – Does a Does a measure look right to you?measure look right to you?

There are several ways (e.g., random-There are several ways (e.g., random-sampling, pilot study) to test/improve the sampling, pilot study) to test/improve the validity of a measure (but not always useful).validity of a measure (but not always useful).


DiscussionDiscussion

Suppose that you are interested in Suppose that you are interested in knowing the determinants of the knowing the determinants of the accountability of a government.accountability of a government.

How do you measure the level of How do you measure the level of accountabilityaccountability

What can be your unit of analysis?What can be your unit of analysis? Is your measure valid?Is your measure valid?


PrecisionPrecision

In theory-oriented empirical research, we In theory-oriented empirical research, we must deal with not only the relationship must deal with not only the relationship between a concept and a measure but also the between a concept and a measure but also the quality of the measure itself. quality of the measure itself.

There are two kinds of There are two kinds of precisionprecision with which with which we shall be concerned: precision in we shall be concerned: precision in measurementmeasurement and precision in and precision in measuresmeasures.. In empirical research (both in quantitative and In empirical research (both in quantitative and

qualitative research), we first decide how much qualitative research), we first decide how much precise our precise our actact of measurement itself should be. of measurement itself should be. Then, we decide how much precise our measured Then, we decide how much precise our measured outcomesoutcomes should be. should be.


Precision in Precision in measurementmeasurement

There are three levels of measurement: noThere are three levels of measurement: nominal, ordinal and interval. minal, ordinal and interval. Nominal measurementNominal measurement

Q. Where do you live? Q. Where do you live? A. “Inner North”, “Woden”, “Belconnen”, “Others”A. “Inner North”, “Woden”, “Belconnen”, “Others”

Ordinal measurementOrdinal measurement ““Lower class”, “Middle class”, “Upper class”Lower class”, “Middle class”, “Upper class” ““Large-sized”, “Medium-size”, “Small-sized”Large-sized”, “Medium-size”, “Small-sized”

Interval measurementInterval measurement Personal income, government expenditures, etc.Personal income, government expenditures, etc.


Levels of PrecisionLevels of Precision

Interval Interval mm is more is more precise than ordinal precise than ordinal mm, which in turn is , which in turn is more precise than more precise than nominal nominal mm..

All All continuouscontinuous measures are measures are intervalinterval

DiscreteDiscrete measures measures are either ordinal or are either ordinal or nominal. nominal.


Precision in measuresPrecision in measures

Once you choose the level of measurement Once you choose the level of measurement (i.e., nominal, ordinal or interval), you may (i.e., nominal, ordinal or interval), you may want to keep distinctions of measured want to keep distinctions of measured outcomes (=measures) as fine as possible and outcomes (=measures) as fine as possible and practical.practical.

ExamplesExamples Countries are categorized into {Asia, Europe, Countries are categorized into {Asia, Europe,

Africa, Latin America, …} or {East Asia, Southeast Africa, Latin America, …} or {East Asia, Southeast Asia, South Asia, …}.Asia, South Asia, …}.

The level of corruption is {high, medium, low} or The level of corruption is {high, medium, low} or {very high, high, medium, low, very low}.{very high, high, medium, low, very low}.

Person A’s income is {$52,325} or {$52,000}.Person A’s income is {$52,325} or {$52,000}.


ImportanceImportance

Preserving a high level of “precision in Preserving a high level of “precision in measurement” always deserves a strong measurement” always deserves a strong priority in research.priority in research.

You should decide a level of “precision in You should decide a level of “precision in measures” measures” given the subject you are given the subject you are studying.studying. When we When we analyzeanalyze data, the degree of data, the degree of

“precision in measures” is determined by what “precision in measures” is determined by what we wish to do with data. The time to ensure we wish to do with data. The time to ensure that your measures will be as precise as you that your measures will be as precise as you want is when you want is when you collectcollect data. data.


Quantitative vs. Quantitative vs. qualitativequalitative

The discussions thus far can be easily The discussions thus far can be easily applied in quantitative research. It is applied in quantitative research. It is important, however, to understand that the important, however, to understand that the same conclusions hold for qualitative same conclusions hold for qualitative research. research.

The differences between quantitative and The differences between quantitative and qualitative measurement involve how data qualitative measurement involve how data are represented. Qualitative researchers use are represented. Qualitative researchers use words like “more” or “less,” “larger” or words like “more” or “less,” “larger” or “smaller,” and “strong” and “weak” and “smaller,” and “strong” and “weak” and quantitative researchers use numbers.quantitative researchers use numbers.


NotesNotes A theory can produce more than one testable A theory can produce more than one testable

hypothesis. Always think as many observable hypothesis. Always think as many observable implications of a theory (i.e., hypotheses) as implications of a theory (i.e., hypotheses) as possible.possible.

A hypothesis can produce more than one A hypothesis can produce more than one “working” hypothesis. A variable can be “working” hypothesis. A variable can be measured in a number of different ways with measured in a number of different ways with different units of analysis. different units of analysis.

It is often recommended to use different units of It is often recommended to use different units of analysis and different measures to test a given analysis and different measures to test a given hypothesis (and make a synthetic interpretation). hypothesis (and make a synthetic interpretation).


Next sessionNext session

SurveysSurveys Differences between interviews and Differences between interviews and

surveyssurveys Types and modes of surveysTypes and modes of surveys Designing a surveyDesigning a survey Rules of asking questionsRules of asking questions

Content AnalysisContent Analysis Types of content analysisTypes of content analysis Steps in content analysisSteps in content analysis

31 March 2009 @ Crawford School 1 Measurement – 1 Semester 1, 2009 POGO8096/8196: Research Methods...

Documents

Transcript of 31 March 2009 @ Crawford School 1 Measurement – 1 Semester 1, 2009 POGO8096/8196: Research Methods...