Dr. Johnson. www.researchdemystified.org
1
Other Research Approaches
Research Method for Public Administrators
Dr. Gail Johnson
Dr. Johnson. www.researchdemystified.org
2
Research Complexity
Not all research approaches fit into the Xs and Os framework
Some approaches are really analytic techniques (e.g. cost-benefit analysis)
Some approaches are data collection tools (e.g. survey research)
Dr. Johnson. www.researchdemystified.org
3
Key Points about Other Approaches Each approach has unique characteristics
and are useful in specific situations Each has strengths and limitations Each has fundamental requirements Researchers often uses a mix of approaches
Dr. Johnson. www.researchdemystified.org
4
Commonly Used Research Approaches Secondary Data Analysis Evaluation Synthesis Content Analysis Survey Research Case Studies Cost-Benefit Analysis
Dr. Johnson. www.researchdemystified.org
5
Secondary Analysis of Data
In the planning phase, researchers should determine whether there is data they can use
It is more efficient to use data collected by others if it meets the research requirements and is of high quality. For example: That National Longitudinal
Survey of Youth collected data about drug use.
Dr. Johnson. www.researchdemystified.org
6
Secondary Analysis of Data
Large research studies gather data that is specifically related to the research questions and often gather other data that might be useful. The Framingham study focused on factors
related to heart disease, tracking 5,000 people from 1983-2003.
Other researchers used that data to look at why people are happy.Researchers Conclusion: happiness is
contagious!
Dr. Johnson. www.researchdemystified.org
7
Things to Consider
Does the database contain the data needed to answer your research question? If the question is about drug use among high
school seniors, then a study looking at drug use among adults will probably be inappropriate.
Dr. Johnson. www.researchdemystified.org
8
Things to Consider
How did they collect the data and does it meet research standards? How did they measure their key variables? Did they use documented procedures for
selecting people, developing data collection instruments, and pre-testing?
Dr. Johnson. www.researchdemystified.org
9
Things to Consider
What quality control procedures were used to ensure the accuracy of the data in the database?What is the error rate?
This means the difference between the data collected and the data in the database.
Dr. Johnson. www.researchdemystified.org
10
Things to Consider
How hard will it be to gain access to the data? Will the researchers provide all the
documentation, including the data dictionary that specifies every variable and how every value for ever variable is coded? For example, are Men coded 1 and Women coded 2
or is it Women coded 1 and Men coded 2?
Dr. Johnson. www.researchdemystified.org
11
Evaluation Synthesis
Useful when there are many studies that have been done on a specific topic Nutrition programs for the poor to reduce infant
mortality Early learning programs for disadvantaged
children Impact of race on death penalty sentencing
Dr. Johnson. www.researchdemystified.org
12
Evaluation Synthesis
An expanded version of using available data Cheaper than original data collection Creates a larger data base, increasing the
potential for drawing conclusions While individual studies often have limited
ability to draw conclusions, if the findings of many small studies form a pattern, conclusions have greater credibility
Dr. Johnson. www.researchdemystified.org
13
How to Do an Evaluation Synthesis Collect all studies that appear to be relevant Establish criteria to rate the quality of each
of the studies Only those meeting the criteria are included Summarize the key findings and impact
measures Look for similarities, especially among the
highest quality studies
Dr. Johnson. www.researchdemystified.org
14
Content Analysis
Used to measure the amount and/or message of communication It can be used to analyze content of speeches,
newspaper stories, emails, greeting cards, television, movies, blogs, memos, diaries or journals
Subjects covered in MPA textbooks, amount of violence in Saturday morning cartoons, or images of political women portrayed in news journals
Dr. Johnson. www.researchdemystified.org
15
Content Analysis
In the narrowest definition, it converts qualitative data into quantitative data: It can count the number of stories about the Iraq
war on major national network news shows to determine whether it changed over time
It can count the number of times presidential candidates spoke about specific policy issues
Dr. Johnson. www.researchdemystified.org
16
Content Analysis
But it is often used to summarize qualitative data What were the major themes that emerged from
the focus groups with teenaged mothers on welfare?
Is there a discernable political bias in the editorials of the major newspapers (NY times, Washington post, Boston globe, LA times)?
Dr. Johnson. www.researchdemystified.org
17
Content Analysis: Measurement Challenges Conceptual and operational definitions
How would conceptually define “political bias”?
Do you want to look at support for Republican or Democrat candidates or liberal or conservative policy positions?
Dr. Johnson. www.researchdemystified.org
18
Content Analysis: Measurement Challenges How would you operationalize “liberal” and
“conservative”—and are they they only political ideologies you want to include?What about libertarians, for example?What about fiscal conservatives who are
liberal on social policy?
Dr. Johnson. www.researchdemystified.org
19
Content Analysis:Measurement Challenges Conceptual and operational definitions
How would you measure “media images” of women governors?
How would you operationalize “strong” or”weak”, or “masculine” or “feminine”?
Dr. Johnson. www.researchdemystified.org
20
Content Analysis:Reliability Challenges Coding Reliability: Will everyone code the
same images in exactly the same way?Requires clear and very detailed decision
rulesRequires all team members are trained
Dr. Johnson. www.researchdemystified.org
21
Assuring Reliability
Jargon term: Inter-rater reliabilityEach team member will code a small sample
of the same materialTheir coding will be compared and if there
are differences, the team will figure out why there were differences and fix the procedures.
Everyone should be coding the same material in the same way.
Dr. Johnson. www.researchdemystified.org
22
Survey Research
A data collection approach (discussed later under data collection) is used when the researchers want to determine the views of many people
Typically, the exact same questions are asked with limited response options Think of a multiple choice exams (will cover more in
data collection) The structure limits bias of interpretation and is easier
to analyze
Dr. Johnson. www.researchdemystified.org
23
Survey Research
Open-ended questions—like fill-in-the blank exams—are possible but:Can be burdensome to complete if too
many open-ended questions are askedHard to analyze Bias is a threat.
Dr. Johnson. www.researchdemystified.org
24
Examples of Survey Research
Opinion polls to a cross-sectional sample of people in your town
Employee attitude survey of everyone at your place of work
Customer survey on the back of a postcard to all who visit the Post Office during the month of February
Dr. Johnson. www.researchdemystified.org
25
Survey Research
In-personCan be costly to do many in-person
interviewsBut useful when people are not literate
For example: it might be the best way to find out about health care access in rural villages in poor countries
Dr. Johnson. www.researchdemystified.org
26
Survey Research
MailRequires mail address and participants
who can read Cost of printing and postageRequires follow-up to get people to
complete them
Dr. Johnson. www.researchdemystified.org
27
Survey Research
Phone. Useful in locations where most people have phones. Works well when the questions are simple (like who
are you going to vote for in the next election) and can be done in less than 15 minutes.
Random digit dialing is used to get a random sample; Even those with unlisted phones are called.
Challenges of cell phones: pollsters are now including cell phones.
But you have to call many, many numbers before you find someone who is willing to take the survey.
Dr. Johnson. www.researchdemystified.org
28
Survey Research
Email or Web-based Limited to those with access to this technology Requires email addresses Web-based limited to those with access Can work well in work setting or in
associations where everyone has access Web-based applications often create the
database, saving time and money
Dr. Johnson. www.researchdemystified.org
29
Focus Groups
Another data collection approach that is sometimes presented as a research design approach. (discussed in more detail under data collection)
Small groups (6-12 people), trained facilitator moderates the discussion, few broad open-ended questions, food, pay to participate.
Dr. Johnson. www.researchdemystified.org
30
Case Studies
Useful in answering descriptive and normative questions
Used in describing “best practices” Can focus on one or several people, groups,
communities, organizations, processes, cities, etc. When it is not possible to gather data from every
person, community, program or organization—Case Studies provide a way to say, “for example.”
Dr. Johnson. www.researchdemystified.org
31
Case Studies
Case studies are useful when the researchers want to gain an in-depth understanding of a situation, of how a complex process works, or lessons learned about implementing an innovative program
It helps to make a problem or situation concrete: can tell stories of the impact of a program on a few people’s lives
It can capture complexity and variation in a situation
Dr. Johnson. www.researchdemystified.org
32
Choosing Case Studies
How many? No fixed rules It will depend on the situation and available
resources More is better to capture variation but case
studies are always about taking one or a handful of examples from a much larger universe
Dr. Johnson. www.researchdemystified.org
33
Choosing Case Studies
Basis for selection Convenience: easy for researchers to access Particular aspects: best, typical, worst, or mixed
cases Diversity: mix of locations, people, types of
organizations or perspectives Random: removing researcher bias from
decision
Dr. Johnson. www.researchdemystified.org
34
Choosing Case Studies
Researchers have to explain their rationale for whatever basis they use to select cases
Selection should have “Face validity”: the choices should make sense to the average, reasonable person
Dr. Johnson. www.researchdemystified.org
35
How to do a Case Study
Case studies can employ a wide variety of methods to gather data and often use several in combination Triangulation: using different methods to gather
information from various perspectives increases the validity of the results
Can use available data, surveys, interviews, observations, analysis of files and records, content analysis of memos, cost-benefit analyses
Dr. Johnson. www.researchdemystified.org
36
Case Studies: Are They Credible? Case studies are criticized because they
typically have a small scope. This is true. But that is also true of classic experiments. You have to beyond “smallness”—and look at
what they did and how they did it. Researchers should state the limitations of their work and conclusions should stay within the boundaries of their limitations.
Dr. Johnson. www.researchdemystified.org
37
Case Studies: Do They Make Sense? Case studies are criticized because they are
seen as inherently qualitative. Case studies may include quantitative data and
may in fact be entirely quantitative. While case studies may be qualitative, there is
nothing wrong with doing qualitative research as long as conclusions do not go beyond the limitations of the design.
Dr. Johnson. www.researchdemystified.org
38
Case Studies in Public Administration Case studies make sense in public
administration where it is difficult to do national studies because of insufficient resources (time, money, staff) Visiting six IRS offices may be far more doable
than visiting 100
Dr. Johnson. www.researchdemystified.org
39
Case Studies in Public Administration Like any research, quality is built into the
design and procedures used: Relevant research questions Clear and appropriate measures Data collection strategies than minimize error
and bias Honest about limitations Conclusions stay within those limitations
Dr. Johnson. www.researchdemystified.org
40
Cost-Benefit Analysis
Answers Normative Questions: Do the benefits justify the costs? Do the costs exceed the benefits?
Assumes that the costs and the benefits can be meaningfully measured in $$Dollars$$
Dr. Johnson. www.researchdemystified.org
41
Cost-benefit Analysis
Is it cost beneficial to get a college degree?What are the costs?
Tuition, books, room and boardOpportunity cost:
The money forgone from working fulltime in order to go to college
The earnings if that money had been invested
Dr. Johnson. www.researchdemystified.org
42
Cost-benefit Analysis
Is it cost beneficial to get a college degree? What are the benefits?
Difference in income between job with college degree and job without a college degree
Calculated over 20 yearsNeed to take inflation into account
(remember to use constant dollars)
Dr. Johnson. www.researchdemystified.org
43
Cost-benefit Analysis
Is it cost beneficial to get a college degree? What are benefits that cannot be measured in
dollars? Career opportunities: some jobs are only available
for those with college degrees, even if they do not pay that much
Increased self-esteem Transformative experience: exposure to new ideas Sustains a vibrant democracy
Dr. Johnson. www.researchdemystified.org
44
Cost-benefit: Public Sector
Policy analysis: what are 5 different approaches to these issues
1. Reduce recidivism for drug users
2. Prevent unwed teenaged parenthood
3. Provide health care to all citizens
4. Prevent financial meltdowns like we saw in the banks and financial industries in Fall 2008
What are assumptions are you making between cause and effect in each scenario?
Dr. Johnson. www.researchdemystified.org
45
Cost-benefit: Public Sector
Policy analysis: Pick one policy issue and consider how you
might think about calculating the costs and benefits for each one.
Note: one policy choice is always to do nothing Assuming there was that was the clear winner
in the cost-benefits contest, are dollars the only consideration in the political environment?
Dr. Johnson. www.researchdemystified.org
46
Cost-benefit: Public Sector
Values, beliefs, political ideology will all affect the debate
But each side are likely to present their best guess of costs and benefits.
And their estimates are likely to differ
Dr. Johnson. www.researchdemystified.org
47
Other Costs
Social Costs and Externalities Costs paid by the larger society
Take a look at the externalities of the material’s economy:
http://www.storyofstuff.com/Of course—you might want to question where some
of the numbers come from but it does point out that we tend not to take externalities (meaning someone else pays a price) into account.
Dr. Johnson. www.researchdemystified.org
48
Other Costs
Externalities: costs paid by someone other than those receiving the benefit Those producing or using goods and services do not
pay the full costs E.g. Corporations pollute but the taxpayers pay for the
cleanup City A benefits from having manufacturing company
but City B down river pay for the price: either higher health costs or costs for cleanup of toxics
Dr. Johnson. www.researchdemystified.org
49
Other Benefits
Short term benefits are easier to measure than longer-term benefits
Direct benefits are easier to measure than indirect benefits Direct: savings from health promotion program
because less spent on health treatments Indirect: savings from loss of productivity
Dr. Johnson. www.researchdemystified.org
50
Other Benefits
Intangible benefits: avoidance of pain and grief of infant deaths It is harder to attach a dollar value: if it is your
child, the life of your child is priceless
Dr. Johnson. www.researchdemystified.org
51
Cost-Benefit Assumptions
Implied balance between costs and benefitsBut: who gains is not necessarily the
same as who pays! Key Question: Do these estimates and
assumptions affect the results in a particular way?
Dr. Johnson. www.researchdemystified.org
52
Uses of Cost-benefit Analysis
While not as concrete as often portrayed, it is a useful framework for considering policy options.
Can provide a deeper understanding of the many connections between policy options, their potential costs and benefits, externalities, and equity considerations
Dr. Johnson. www.researchdemystified.org
53
Choosing a Research Approach
The choice is guided by the situation If data is available, then using available data
makes sense If you want to gather people’s opinion, then a
survey makes sense If you want an in-depth understanding of a
program or community, then a case study makes sense
Dr. Johnson. www.researchdemystified.org
54
Choosing a Research Approach
The choice is guided by the situation If many studies have already been done, then an
evaluation synthesis makes sense If you want to make sense of communications,
then content analysis makes sense Of you want to determine which approach is
more likely to be cost effective, than cost benefit analysis makes sense
Dr. Johnson. www.researchdemystified.org
55
Choosing a Research Appoach
Research approaches can be combined: A survey might be used to gather data for a
correlation design using statistical controls A case study might include a content analysis
of news clippings, available data and interviews with key community leaders
Dr. Johnson. www.researchdemystified.org
56
Key Takeaway Lessons
If the researchers have posed a cause-effect question, then they should use a design in the Xs and Os framework
The classic experiment is best for impact questions because it is necessary to eliminate all other rival explanations
Be cautious if they have asked a cause-effect question or implied a cause-effect relationship: Is the research design strong enough to
support the conclusions?
Dr. Johnson. www.researchdemystified.org
57
Creative Commons
This powerpoint is meant to be used and shared with attribution
Please provide feedback If you make changes, please share freely
and send me a copy of changes: [email protected]
Visit www.creativecommons.org for more information
Top Related