Modeling Political Modeling Political PhenomenaPhenomena

Using Control Variables and Gauging Validity

Face ValidityFace Validity

Face validity means the measurement of a concept is consistent with an agreed definition.

It does not mean however that this is the best measurement to capture the concept.

Construct ValidityConstruct Validity

construct validity: The match between the land of theory and the land of observation

How effective do our operationalized variables represent the mental image of a concept into the public manifestation of that world?

ReliabilityReliability

We may want, or need to, test for reliability, which is to ask if our variables consistently provide the same results.

To be sure we can measure/test repeatedly or even use multiple measures for the same variable. For development we can also use GDP per capita besides energy consumption per capita.

Key question of Internal ValidityKey question of Internal Validity

When we test a hypothesis and either accept it or reject it, how do we know that we made the right decision?

What about alternative explanations that we did not account for?

What should we do to gain confidence?

Internal ValidityInternal Validity

Are there other causes for what I am observing?

If so, a study will lack internal validity if it cannot rule out plausible alternative explanations.

Internal Validity of a StudyInternal Validity of a StudyInternal Validity of a StudyInternal Validity of a Study

What you measured and what you saw?Your program and your observations?

ObservationObservationObservationObservation

ProgramProgram ObservationsObservations

What you What you dodo What you What you seesee

Is the relationship Is the relationship causalcausal between... between...

AlternativeAlternativecausecause AlternativeAlternative

causecause

AlternativeAlternativecausecause

AlternativeAlternativecausecause

In In thisthis study study

Program-outcome RelationshipProgram-outcome Relationship

The Purpose of Control VariablesThe Purpose of Control Variables

We use control variables to account for possible alternative explanations we can think of.

For example, when I examined whether democracies are generally more peaceful than autocracies I included several control variables.

Explaining Pacifistic DemocracyExplaining Pacifistic Democracy

Peace (Y) = Democracy (X1) + State Power (X2) + Development (X3) + # of Bordering States (X4)

In the model above, I have more confidence that Democracy is related to peace considering I control for the other variables that may skew my test.

• We need to take care that our theory is not missing other factors that may undermine the validity of our theory and tests.

• Our inferences will be flawed if we are actually capturing other processes through our variables.

• This means that the validity of our measures would be undermined.

• Several possible problems arise that are related to model misspecification and spurious relationships.

• Thus, we need to control for confounding factors and alternative explanations!!!

Model Misspecification and Model Misspecification and SpuriousnessSpuriousness

Antecedent variable: A variable that indirectly affecting the relationship between two other variables.

For example, Ivy league education increases income.

However, parental wealth and legacy admissions affect Ivy league education. Thus, income of graduates from Ivy League schools may not be random.

Here Ivy League Parents is an antecedent variable

Ivy League Parents Ivy League Kids high income kids

Hence, admission to Ivy schools clearly not random or pure merit-based, and thus the income earned by these people.

Model Misspecification and Spuriousness

Intervening Variable: These may be spuriously related to another relationship.

How can states fight each other if they are not contiguous with each other? Only the strongest, with large navies, bases, etc., could do so.

Hence, geographic contiguity or distance is an intervening variable. States may or may not be more peaceful, but it is hard to avoid conflict when it is on your borders.

Model Misspecification and Spuriousness

Alternative Variables: We also want to control for variables that would bias our results if omitted.

In this case, the X variables in a model would produce biased estimates, undermining their validity and producing error that leads to inaccurate inferences.

Here is a spurious relationship from my research

IGOs + conflicts

+ +

Powerful states

Powerful states both in more IGOs and conflicts, but these two variables not directly related but a function of state power.

Classic Spurious Case

Ice Cream Consumption Crime

Summer Temperatures

???

+ +

+

Hence we see that despite the fact that ice cream consumption is correlated with crime, the real cause is that summer temperatures increase both ice cream consumption and crime.

Assessing your knowledgeAssessing your knowledge

If your scientific study has taken care to make sure that your variables are measured correctly, used the appropriate control variables, and used proper tests, then what is next?

Conclusion ValidityConclusion ValidityConclusion ValidityConclusion Validity

What you did and what you saw?

Your program and your observations?

ObservationObservationObservationObservation

ProgramProgram ObservationsObservations

What you What you dodo What you What you seesee

Is there a Is there a relationshiprelationship between... between...

In In thisthis study study

Program-outcome RelationshipProgram-outcome Relationship

Group Work

Identify the level on which variables are measured.

Identify problems of construct validity, internal validity, and biased samples

External ValidityExternal Validity

Now that you are confident of what you found in your study, how well does my study or sample relate to the general population?

In other words, how strong is my study able to generalize to other cases?

Research Designs and Research Designs and SamplingSampling

In most studies what is examined are some cases, not an entire population. For example, in Presidential Election polls not every voter is asked how they will vote but still polls can be very accurate.

How does that happen?

Population vs. SamplePopulation vs. Sample

Research in the social sciences typically uses sampling methods.

We draw a sample of subjects from a greater population.

We then draw an inference from the sample about the greater population.

In other words, we are generalizing about a population from a subset (the sample).

Validity and BiasValidity and Bias

In order to draw an accurate inference from a sample, the sample needs to be reflective of the population from which it is drawn.

If a sample is not reflective of the population, then it is biased in some manner and the greater study will lack validity.

Types of SamplingTypes of SamplingNonrandom: snowballing, various improper selection techniques or limited data. Measurement error is greater.

Random: pure chance of lottery and should reflect population the larger the sample. Measurement error decreases.

Quota or stratified: Selecting on groups to form sample so as to reflect greater population. Measurement error is lower.

Census: Includes entire population. No measurement error.

Mathematical PrincipleMathematical Principle

The larger the sample size, the more it will reflect the population estimates/values.

Thus, the larger the sample, the less chance of measurement error.

External Validity of a StudyExternal Validity of a Study

TheoryTheoryTheoryTheory

CauseCauseconstructconstruct

EffectEffectconstructconstruct

What you What you thinkthink

Cause-effect ConstructCause-effect Construct

ObservationObservationObservationObservation

ProgramProgram ObservationsObservations

What You What You DoDo What You What You SeeSee

program-outcome relationshipprogram-outcome relationship

ObservationObservationObservationObservation

ProgramProgram ObservationsObservations

What You What You DoDo What you What you seesee

program-outcome relationshipprogram-outcome relationship

ObservationObservationObservationObservation

ProgramProgram ObservationsObservations

What You What You DoDo What You What You SeeSee

program-outcome relationshipprogram-outcome relationship

ObservationObservationObservationObservation

ProgramProgram ObservationsObservations

What You What You DoDo What You What You SeeSee

program-outcome relationshipprogram-outcome relationship

Can we generalize to other persons, places, times?

External Validity of a StudyExternal Validity of a Study

The last graphic is meant to convey the principle that external validity is gained by additional observations/tests in other studies.

This is why pundits will compare several election polls to see how well they compare. If they do not, then somebody is doing something wrong or different.