1

UAPP 702: Research Design for Urban & Public PolicyClass Notes

Babbie, The Practice of Social Research, Chaps.4&5

Danilo YanichSchool of Public Policy & Administration

Center for Community Research & Service

University of Delaware

2

Ch. 4: Research DesignPurposes of Research

Exploration: typically done for three purposes:

to satisfy the researcher’s curiosity and desire for better understanding

to test the feasibility of undertaking a more extensive study to develop the methods to be employed in a subsequent study

Description: describe situations and events Census is good example of descriptive research

Explanation: the “why?” of events, situations, behavior, attitudes, etc.

3

Logic of Nomothetic Explanation

Nomothetic explanation refers to the accounting of many variations in a given phenomenon

In contrast to…

Idiographic explanation that seeks an in-depth understanding of a single case

4

Criteria for Nomothetic Causality

Correlation: the variables must be correlated

Time order: the cause takes place before the effect

Non-spurious: the variables are non-spurious

Spurious relationship: a coincidental statistical correlation between two variables, shown to be caused by some third variable

5

Correlation

Some relationship---or correlation—between the variables must exist before we can consider causality

Correlation: empirical relationship between two variables such that…

Changes in one are associated with changes in the other

Particular attributes of one variable are associated with particular attributes of the other

6

False Criteria for Nomothetic Causality

Complete causation Causation is incomplete and probabalistic

Exceptional cases Exceptional cases do not disprove general overall pattern of

causation

Majority of cases Causal relationship may be true even if they don’t apply to the

majority of cases

Example: lack of supervision & delinquency… as long as unsupervised juveniles are more likely to be become

delinquent, social science can say there is a causal relationship

7

Necessary and Sufficient Causes

Necessary cause represents a condition that must be present for the effect to follow

Ex: must be female to become pregnant

Ex: must take college courses to get a degree…but… Simply taking courses is not a sufficient cause Must take the right ones

8

Necessary and Sufficient Causes

Sufficient cause represents a condition that, if it is present, guarantees the effect in question

Not saying that sufficient cause is only possible cause for effect

Ex: skipping exam in course would be sufficient cause for failing, but students could fail in other ways, too

So, cause can be sufficient but not necessary

9

Units of Analysis

No limit to what or whom can be studied

Common social science units of analysis:

Individuals Groups Organizations Social artifacts.

Important: what you “call” a given unit of analysis is almost irrelevant—but you must be clear what that unit “is”

Are you studying marriages or marriage partners? Crimes or criminals? Historic buildings or the process for selecting them? Efficiency of the hotel or the satisfaction of customers?

10

Ecological Fallacy Ecological in this context refers to groups or sets or systems, something

larger than individuals.

Fallacy is to assume that something learned about such a unit says something about the individuals comprising that unit.

Babbie uses example of data that shows which precincts supported a female candidate…

Some census data for each precinct that shows that precincts with relatively young voters gave her more support

Could not assume that young voters were most likely to support a female

candidate...

That is…we cannot assume that age affects support

The unit of analysis was the precinct, NOT the individuals in the precinct

11

Reductionism

Tendency to explain everything in terms of a particular, narrow set of concepts

Remember paradigms that predispose researcher to a particular explanation

Definition of order by coercion, shared values, exchange

12

Ch. 5: Conceptualization, Operationalization & Measurement

Conceptualization The refinement and specification of abstract concepts A specific agreed-upon meaning of the concept under

study Ex. “compassion” does not exist in any sense that we can

measure in an objective sense

Operationalization The development of specific research procedures

(operations) that will result in empirical observations representing those concepts in the real world

13

What social scientists measureTable 5.1, p. 129

Examples

Direct observables Physical characteristics of a person being observed/interviewed

Indirect observables Characteristics of a person as indicated by answers given in a self-administered questionnaire

Constructs Level of alienation, as measured by a scale that is created combining several direct and/or indirect observables

14

Indicators and Dimensions

Indicator An observation that we consider as a reflection of the

variable under study Ex: attending church as an indicator or religiosity

Dimension A specific aspect of a concept Ex: action aspects of religiosity (attending church,

giving money) and contemplative aspects (prayer, etc)

15

Operational definition

Specifies precisely how a concept will be measured

Operationalization The development of specific research procedures

(operations) that will result in empirical observations representing those concepts in the real world

16

Progression of measurement steps

Conceptualization

↓ Nominal definition

↓ Operational definition

↓ Measurements in the real world

“conceptual funnel”

17

Progression of measurementTable 5.2, p. 136

Measurement step Example: social class

Conceptualization What are the different meanings and dimensions of the concept “social class”?

Nominal definition For our study, we will define “social class” as representing economic difference: specifically, income

Operational definitionWe will measure economic differences via responses to the survey question: “What was your annual income, before taxes, last year?”

Measurements in the real world

The interviewer will ask: “What was your annual income, before taxes, last year?”

18

Operationalization Choices

Range of variation: Must be clear about the range of variation in any concept that interests you.

Babbie uses as an example studying certain ranges of income, i.e., using $100,000 as the floor for the highest income group rather than a higher amount

Attitudes toward nuclear power...might use a range of “favor it very much” to don’t favor it at all”...

But, that would leave out the people who are opposed to it.

Variations between extremes: Get as much detail in the measurement as possible.

Can always aggregate data (that is, combine precise attributes) into more general categories...

But can never separate out any variations that were lumped together during observation and measurement.

19

Two important qualities of variables: Exhaustive & Mutually Exclusive

Exhaustive: For the variable to have any utility in research, must be able to classify every observation in terms of one of the attributes composing the variable

Babbie uses example of political party affiliation that specifies just Democrat or Republican…

When that would leave out others who do not identify with either

Use “other” or “no affiliation” to make it exhaustive.

Mutually exclusive: Must be able to classify every observation in terms of one and only one attribute.

Babbie uses defining employed and unemployed in such a way that nobody can be both at the same time

Refer to Graber “social type” variable...farmer, n’er-do-well, etc. & Family Court gender variable.

20

Levels of measurement (NOIR)

Nominal: variables whose attribute have only the characteristics of exhaustiveness and mutual exclusivity

Examples: gender, religious affiliation, birthplace, etc

Ordinal: variables with attributes that can logically rank-order; the different attributes represent relatively more or less of a variable.

Examples: social class, conservatism, alienation, prejudice, “coolness”

Interval: variables in which the actual distance separating them can be expressed in meaningful standard variables

Examples: temperature, intelligence tests

Ratio: variables that have all of the characteristics of the previous levels of measurement AND are based on a true zero point

Examples: age, length of residence in a home, duration of news story, etc.

21

Implications of levels of measurement

Requirements of analytical techniques:

Certain analytical techniques require variables that meet certain minimum levels of measurement

Must plan analytical techniques according to the level of measurement at which you will gather your data.

Should anticipate drawing research conclusions appropriate to the levels of measurement used in your variables.

Caution: Seek highest level of measurement possible because...

Although you can reduce a ratio measure to ordinal...

You cannot convert an ordinal measure into a ratio measure...

It is a one-way street

22

Criteria of measurement quality

Precision and accuracy

Precision=fineness of the distinction made between the attributes that compose a variable

Saying that a woman is “43 years old” is more precise than saying that she is “in her forties”

Degree of precision is dictated by your research requirements If your research question does not require her precise age, then additional effort

to gather it precisely is wasted

However, if your needs are unclear, be more precise rather than less

Do not confuse precision with accuracy Saying that someone was born in “Stowe, VT” is more precise than born in “New

England”

But…suppose the person in question was born in Boston

The more general description of “New England” is less precise, but accurate

23

Criteria of measurement quality, p.2

Reliability

Whether a particular technique, applied repeatedly to the same object, yields the same result every time

Example: Measuring weight using two different persons’ estimates versus a scale

Reliability does NOT ensure accuracy Suppose the scale is set five pounds too light

Measurement would be reliable each time, but it would also be wrong each time

Ways to cross-check the reliability of measures Test-retest method

Split-half method

Using established measures (Miller book is useful here)

Reliability of research workers

24

Criteria of measurement quality, p.3

Validity

Refers to the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration

Social research does operate on agreements about the terms we use and the concepts they represent

25

Criteria of measurement quality, p.4

Testing validity

Face validity — empirical measures that jibe with our common understanding of a concept

Ex. Grievances & worker morale

Criterion-based validity — based on external criterion

Ex. College board scores & student success in college

26

Criteria of measurement quality, p.5

Testing validity

Construct validity — based on logical relationships among variables

Ex. Marital fidelity & marital satisfaction

Content validity — refers to how much a measure covers the range of meanings in a concept

Ex: test of math ability can’t be limited to addition alone

27

Criteria of measurement quality, p.6

Tension between reliability & validity

Often a trade-off between the two because resources limit the research

Ex. Measuring morale by spending days on assembly line talking w/ workers seems a more valid measure of morale than counting grievances

If there is no clear agreement on how to measure a concept…measure it several ways

Ex. Recidivism, court success, hotel efficiency, etc.

Concept does not have any meaning other than what we give it. Only justification to give concept a particular meaning is utility

28

Basic Research Outline*

The Social Problem

Present a clear, brief statement of the problem, with concepts defined where necessary

Show that the problem is limited to bounds amenable to treatment or test

Describe the significance of the problem with reference to specific criteria

Source: Miller, Delbert C. 1991. Handbook of Research Design and Social Measurement, 5th Edition. Newbury Park: Sage Publications, pp. 15-16.

29

Basic Research Outline, p.2

The Theoretical Framework

Describe the relationship of the problem to a theoretical framework

Demonstrate the relationship of the problem to previous research

Present alternate hypotheses considered feasible within the framework of the theory.

30

Basic Research Outline, p.3

The Research Question/Hypotheses

Clearly state the research questions or the hypotheses selected for test. (Null and alternate)

Indicate the significance of test hypotheses to the advancement of research and theory.

For policy research state how research might inform policy.

Define concepts or variables (preferably in operational terms).

Describe possible mistakes and their consequences.

Note seriousness of possible mistakes.

31

Basic Research Outline, p.4

Design of the Experiment or Inquiry

Describe ideal design or designs with particular attention to the control of interfering variables

Describe selected operational design

Specify statistical tests including dummy variables

32

Basic Research Outline, p.5

Sampling Procedures

Describe experimental and control samples

Specify method of drawing or selecting sample

33

Basic Research Outline, p.6

Methods of Gathering Data

Describe measures of quantitative variables showing reliability and validity when these are known. Describe means of identifying qualitative variables

Include descriptions of questionnaires or schedules

Describe interview procedure

Describe use made of pilot study, pretest, trial run.

34

Basic Research Outline, p.7

Working Guide

Prepare working guide with time and budget estimates

Estimate total person-hours and cost

35

Basic Research Outline, p.8

Analysis of Results

Specify methods of analysis

36

Basic Research Outline, p.9

Interpretation of Results

Discuss how conclusions will be fed back into theory…OR…

Inform policy/practice.

37

Basic Research Outline, p.10

Publication or Reporting Plans...Communication Plans

Monograph, Executive summary

Testimony to policy makers.

Presentations to institutions, non-governmental agencies, media, public.

Journal publication

38

The Policy Research Process*

Present a clear, brief statement of the problem, with concepts defined where necessary.

Over half of the criminal cases in Delaware exceed the Supreme Court’s standard for the time from arrest to disposition (plea, verdict, etc.).

Show that the problem is limited to bounds amenable to treatment or test.

An analysis of the period from arrest to disposition of criminal cases in Delaware’s Superior Court during a randomly chosen calendar year will provide the required data to examine the issue.

*D. Yanich example using model in: Miller, Delbert C. (1991). Handbook of Research Design and Social Measurement. Fifth Edition. Newbury Park, CA: Sage Publications, pp15-16

39

The Policy Research Process, p.2

Criterion Comment

Timely The Constitution requires that justice is delivered in a timely manner. To the extent that Delaware is not in compliance with its own 120 standard, it jeopardizes that requirement.

Practical problem The costs, the ethics, the legal liability for operating a system in violation of its own standard.

Wide population All citizens bear the cost of a dysfunctional court system, whether in taxes or large policy choices.

Influential/Critical population

Main audience for the research is the Delaware General Assembly and the agents of the court.

Research gap Never has been a comprehensive look at the case processing in Delaware

40

The Policy Research Process, p.3

Criterion Comment

Generalizations Can NOT generalize to populations (court systems) beyond Delaware.

Sharpens concept Offers a more detailed examination of case processing through critical phases

Practical implications Practice and policy will change as a result of the research.

Improve data analysis instrument

The courts never had a data-gathering instrument to understand case processing. The research will provide a base-line.

Data gathering constrained by time

One calendar year is precisely geared to acquire the critical data within a manageable time period.

Fruitful exploration The research extends the analysis of court processing.

41

The Policy Research Process, p.4

Clearly state the research question/hypotheses selected for test.

Null hypothesis: There is no difference between the cases that are disposed within 120-day mandate and those that exceed it. Research hypothesis: Differences exist between the cases that comply and do not comply with the 120-day mandate along case and court’s culture dimensions.

Indicate the significance of test hypotheses to the advancement of research and theory. For policy research state how research might inform policy.

A systematic examination of the case processing activity of Delaware’s Superior Court will give policy-makers a baseline from which to make changes in the court’s policy and practice.

42

The Policy Research Process, p.5

Define concepts or variables (preferably in operational terms).

Contained in coding instructions in which all variables are operationalizedExamples: unit of analysis=caseinstant offense=crime for which case is prosecutedcriminal history=number of previous convictions

Describe possible mistakes and their consequences.

Possible mistakes focus on validity and reliability issues.

Note seriousness of possible mistakes.

Validity or reliability mistakes are fatal to the research process.