Research methods 2 operationalization & measurement

31
Research Methods

Transcript of Research methods 2 operationalization & measurement

Research Methods

Second Stage: Operationalization

1. Formulation of Theory2. Operationalization of Theory3. Selection of Appropriate Research

Techniques4. Observation of Behavior (Data Collection)5. Analysis of Data6. Interpretation of Results

Hypotheses GenerationHypothesis:

an explicit statement that indicates how a researcher thinks the phenomena of interest are related.

It represents the proposed explanation for some phenomenon.

Indicates how an independent variable is thought to affect, influence, or alter a dependent variable.

A proposed relationship that may be true or false.

Good Hypotheses

Hypotheses should be empirical statements: proposed explanations for relationships that exist in the real world.

Hypotheses should be general: a hypothesis should explain a general phenomenon rather than a particular occurrence.

Hypotheses should be plausible: some logical reason for thinking that it may be confirmed.

Hypotheses should be specific: it should specify the expected relationship between two variables.

Hypotheses should relate directly to the data collected.

Directional Hypotheses

Hypotheses should be specific. IOW, they should state exactly how the independent variable relates to the dependent variable.

1.Positive Relationship: where the concepts are predicted to increase or decrease in size together.

2.Negative Relationship: where one concepts increases in size or amount while the other decreases in size or amount.

Unit of AnalysisOne of the most important aspects of research

design is determining the unit of analysis.This is where we specify the types or levels of

political actor to which t he hypothesis is thought to apply.

There are numerous kinds of units we can collect data on: Individuals Groups States Agencies Organizations

U of A continuedCross-level analysis: sometimes we collect

data on one unit of analysis to answer questions about another unit of analysis.

The purpose in CLA is to make an ecological inference: the use of aggregate data to study the behavior of individuals. Data on voting districts individual voting behavior

CAVEAT: Avoid the ecological fallacy: where a relationship found at the aggregate level is not operative at the individual level. State voting data used to infer about the relationships b/w

district voting data.

MeasurementMeasurement: systematic observation and

representation by scores or numerals of the variables we have decided to investigate.

Operational definition: deciding what kinds of empirical observations should be made to measure the occurrence of an attribute or behavior.

Measuring Variables

The level of measurement refers to the relationship among the values that are assigned to the attributes for a variable

It is important to distinguish between the values of a variable and the level of measurment

Levels of MeasurementThere are typically four levels of

measurement that are defined: Nominal Ordinal Interval Ratio

Levels of Measurement

Knowing the level of measurement helps you decide how to interpret the data from that variable.

Knowing the level of measurement helps you decide what statistical analysis is appropriate on the values that were assigned.

It's important to recognize that there is a hierarchy implied in the level of measurement idea.

Nominal & OrdinalIn nominal measurement the numerical values just

"name" the attribute uniquely. No ordering of the cases is implied. For example, jersey numbers

in basketball are measures at the nominal level. A player with number 30 is not more of anything than a player with number 15, and is certainly not twice whatever number 15 is.

In ordinal measurement the attributes can be rank-ordered. Here, distances between attributes do not have any meaning. For

example, on a survey you might code Educational Attainment as 0=less than H.S.; 1=some H.S.; 2=H.S. degree; 3=some college; 4=college degree; 5=post college. In this measure, higher numbers mean more education. But is distance from 0 to 1 same as 3 to 4? Of course not. The interval between values is not interpretable in an ordinal measure.

Interval & Ratio

In interval measurement the distance between attributes does have meaning. For example, when we measure temperature (in Fahrenheit), the

distance from 30-40 is same as distance from 70-80. The interval between values is interpretable. Because of this, it makes sense to compute an average of an interval variable, where it doesn't make sense to do so for ordinal scales. But note that in interval measurement ratios don't make any sense - 80 degrees is not twice as hot as 40 degrees

In ratio measurement there is always an absolute zero that is meaningful. This means that you can construct a meaningful fraction (or ratio)

with a ratio variable. Weight is a ratio variable. In applied social research most "count" variables are ratio, for example, the number of clients in past six months. Why? Because you can have zero clients and because it is meaningful to say that "...we had twice as many clients in the past six months as we did in the previous six months."

Nominal, Ordinal, Interval, and Ratio Scales Provide Different Information

Characteristics of Different Levels of Scale Measurement

Type of Scale

Data Characteristics

Numerical Operation

Descriptive Statistics Examples

Nominal Classification but no order, distance, or origin

Counting Frequency in each categoryPercent in each categoryMode

Gender (1=Male, 2=Female)

Ordinal Classification and order but no distance or unique origin

Rank ordering MedianRangePercentile ranking

Academic status (1=Freshman, 2=Sophomore, 3=Junior, 4=Senior)

Interval Classification, order, and distance but no unique origin

Arithmetic operations that preserve order and magnitude

MeanStandard deviationVariance

Temperature in degreesSatisfaction on semantic differential scale

Ratio Classification, order, distance and unique origin

Arithmetic operations on actual quantities

Geometric meanCoefficient of variation

Age in yearsIncome in Saudi riyals

Note: All statistics appropriate for lower-order scales (nominal being lowest) are appropriate for higher-order scales (ratio being the highest)

Levels & Research Design

At lower levels of measurement, assumptions tend to be less restrictive and data analyses tend to be less sensitive. At each level up the hierarchy, the current level includes all of the qualities of the one below it and adds something new

In general, it is desirable to have a higher level of measurement (e.g., interval or ratio) rather than a lower one (nominal or ordinal).

True Score Theory

True Score Theory is a theory about measurement. Like all theories, you need to recognize that it is not proven -- it is postulated as a model of how the world operates. Like many very powerful model, the true score theory is a very simple one.

Essentially, true score theory maintains that every measurement is an additive composite of two components: true ability (or the true level) of the respondent on that measure; and random error.

True Score Theory

We observe the measurement -- the score on the test, the total for a self-esteem instrument, the scale value for a person's weight. We don't observe what's on the right side of the equation (only God knows what those values are!), we assume that there are two components to the right side.

1.The ‘true’ value

2.The error in our measurement of that value

Error The true score theory is a good simple model for

measurement, but it may not always be an accurate reflection of reality.

In particular, it assumes that any observation is composed of the true value plus some random error value. But is that reasonable? What if all error is not random?

Isn't it possible that some errors are systematic, that they hold across most or all of the members of a group?

One way to deal with this notion is to revise the simple true score model by dividing the error component into two subcomponents, random error and systematic error. here, we'll look at the differences between these two types of errors and try to diagnose their effects on our research.

Random ErrorRandom error is caused by any factors that

randomly affect measurement of the variable across the sample. For instance, each person's mood can inflate or deflate

their performance on any occasion. In a particular testing, some children may be feeling in a good mood and others may be depressed.

If mood affects their performance on the measure, it may artificially inflate the observed scores for some children and artificially deflate them for others.

Random Error is often referred to as ‘noise.’Random Error does not effect averages.

Random Error

The important thing about random error is that it does not have any consistent effects across the entire sample. Instead, it pushes observed scores up or down randomly.

This means that if we could see all of the random errors in a distribution they would have to sum to 0 -- there would be as many negative errors as positive ones.

Systematic ErrorSystematic error is caused by any factors that

systematically affect measurement of the variable across the sample. For instance, if there is loud traffic going by just

outside of a classroom where students are taking a test, this noise is liable to affect all of the children's scores -- in this case, systematically lowering them.

Unlike random error, systematic errors tend to be consistently either positive or negative -- because of this, systematic error is sometimes considered to be bias in measurement.

Systematic Error

Systematic error, or bias, is a real threat to your research.

Because it affects the average results, it may cause you to report a relationship that doesn’t exist or miss a relationship that does exist.

Avoiding bias in our research is an important technique for producing good research.

Reducing & Eliminating Errors

So, how can we reduce measurement errors, random or systematic? One thing you can do is to pilot test your instruments,

getting feedback from your respondents regarding how easy or hard the measure was and information about how the testing environment affected their performance.

Second, if you are gathering measures using people to collect the data (as interviewers or observers) you should make sure you train them thoroughly so that they aren't inadvertently introducing error.

R & E Errors Third, when you collect the data for your study you should double-

check the data thoroughly. All data entry for computer analysis should be "double-punched" and verified. This means that you enter the data twice, the second time having your data entry machine check that you are typing the exact same data you did the first time.

Fourth, you can use statistical procedures to adjust for measurement error. These range from rather simple formulas you can apply directly to your data to very complex modeling procedures for modeling the error and its effects.

Finally, one of the best things you can do to deal with measurement errors, especially systematic errors, is to use multiple measures of the same construct. Especially if the different measures don't share the same systematic errors, you will be able to triangulate across the multiple measures and get a more accurate sense of what's going on.

How do we measure Unemployment?

ConceptsDefinitionsHow do we collect data on it?What should that data tell us?Why do we want to know about

unemployment to begin with?

Unemployment: Federal definition

The definition of unemployment used in this report is the standard Federal definition of the percent of individuals in the labor force who were not employed.

The labor force is defined as individuals who were employed, were on lay-off, or had sought work within the preceding four weeks. Although this is the most commonly used measure of unemployment, other measures are used.

Unemployment: How is it measured?

Because unemployment insurance records relate only to persons who have applied for such benefits, and since it is impractical to actually count every unemployed person each month, the Government conducts a monthly sample survey called the Current Population Survey (CPS) to measure the extent of unemployment in the country. The CPS has been conducted in the United States every month since 1940 when it began as a Work Projects Administration project.

Unemployment Defining the Concepts

The basic concepts involved in identifying the employed and unemployed are quite simple: People with jobs are employed.

People who are jobless, looking for jobs, and available for work are unemployed.

People who are neither employed nor unemployed are not in the labor force.

Operational Definition of Unemployment

The survey is designed so that each person age 16 and over who is not in an institution such as a prison or mental hospital or on active duty in the Armed Forces is counted and classified in only one group.

The sum of the employed and the unemployed constitutes the civilian labor force.

Persons not in the labor force combined with those in the civilian labor force constitute the civilian noninstitutional population 16 years of age and over.

Reliability & Validity

In research, the term "reliable" can means dependable in a general sense, but that's not a precise enough definition. What does it mean to have a dependable measure or observation in a research context?

In research, the term reliability means "repeatability" or "consistency".

A measure is considered reliable if it would give us the same result over and over again (assuming that what we are measuring isn't changing!).