Mixed Methods Use in Humanitarian Program Impact ... · Mixed Methods Project: An...

Keynote Address

New Directions in Advancing the Methodology of Mixed Methods Research

- John W. Creswell, Ph.D. -

Professor of family medicine and co-director of the Michigan Mixed

Methods Program at the University of Michigan

New Directions in Advancing the

Methodology of Mixed Methods

Research

John W. Creswell, PhD

Senior Research Scientist, Michigan Mixed Methods

Research Program

University of Michigan, Ann Arbor, Michigan

MQIC Conference, Berlin, Germany, February 26 - 28, 2020

Mixed Methods Program

University of Michigan

Institute for Southeast

Asian Qualitative and

Mixed Methods Research,

Osaka and Honolulu

Take home points:

By the end of this presentation,

you will have:

✓ A basic understanding of mixed methods research

✓ 9 procedures to include in your mixed methods study

The reasons for these procedures

Specific examples of these procedures

Creswell’s Books: Research Design, Qualitative

Research, and Mixed Methods Research

2015 lst Edition

2019 6th Edition 2016 lst Edition

2018 3rd Edition 2018 4th Edition

2018 5th Edition

General Research Design

Qualitative

Research

Mixed Methods Research

Honolulu, US Osaka, Japan

China

Germany

Workshops/Speeches in Last Year

A Basic

Understanding

Of Mixed

Methods

Research

1980 1990 2000 2010

Founders of

Mixed Methods

Arrive (Different

Disciplines/Countries-

Books (1985-90)

Mixed Methods

Developed as

Methodology

(1998)

Mixed Methods

Handbook (SAGE)

(2003)

Journal of

Mixed Methods

Research (2007)-

New books on

Mixed Methods

2011 2013 2014

“Best Practices”

Of Mixed Methods

In Health Sciences

(NIH)

Mixed Methods

International Research

Association (MMIRA) (2013)

World-Wide

Expansion of

Mixed Methods

(Regional MMIRA

Conferences)

2015

NIH Mixed

Methods

US training

In health

sciences

A Research Methodology Developed in the Last

30 Years

APA

Standards

on Mixed

Methods

2018

Books Journal US funding International

AssociationTraining

Program

Standards

The Three Key Ideas to Understand Mixed Methods Research

1. We will integrate or mix the data

2. We use mixed methods designs to integrate the data

3. From this integration we obtain insights or

meta-inferences beyond what we learn from

either our quantitative and qualitative data

9 Procedures

To Include in

Your Mixed

Methods Study

When conducting a mixed methods study, map

onto your content the latest mixed methods

procedures or advances

▪ You will build innovative features into your project

▪ You will signal to reviewers that you know the latest

innovations in the field

▪ You will have stronger projects for submission

Why?

1. Identify and label your mixed methods design

2. Include a diagram of your mixed methods design

3. Consider using an implementation matrix

4. Reinforce integration

5. Develop a data table

6. Use rigorous procedures

7. Create a joint display to show integration

8. Include a mixed methods study aim

9. Add a conceptual framework into your design

What are these Advances?

The content of this presentation was developed in collaboration

with the University of Michigan Mixed Methods Program

Executive Committee

Jane Forman

Arden Morris

Sara Warber Program

Coordinator

Satoko Motohara

Research Fellow

Melissa

DeJonckheere

Program Staff

Rae

Sakakibara

Senior Research

Scientist

John W. Creswell

MMP Visiting Scholar

Sergi FàbreguesDirector and Assoc

Director

Michael D. Fetters

Timothy C. Guetterman

Michigan Mixed Methods Program, University of Michigan, 2019

1. Identify and Label Your

Mixed Methods Design

Designs are procedures for

conducting your research. We

have them in quantitative and qualitative

research. Why not have designs in

mixed methods research?

Two Categories of Mixed Methods Designs: Core and

Complex (Creswell & Plano Clark, 2018, 3rdEdition)

Convergent Design

Explanatory

Sequential Design

Exploratory Sequential

Design

Intervention Trial

Multiple Case Study

Program Evaluation

CBPR Participatory Studies

Embedded

Within

Chapter 3: CORE MIXED

METHODS DESIGNS

CHAPTER 4:

COMPLEX

DESIGNS THAT

INCLUDE CORE

DESIGNS

Merge

Quantitative

Data Collection

and Analysis

Qualitative

Data Collection

and Analysis

A Convergent Design

Phase 1

Intent: To compare

results from both

databasesInterpretation

1

Merge

Quantitative

Data Collection

and Analysis

Qualitative

Data Collection

and Analysis

A Convergent Transformative Design

Phase 1

Intent: To develop

a more comprehensive

quantitative assessmentInterpretation

Quantitative Data

Transformation

2

Quantitative

Data Collection

and Analysis

Qualitative

Data Collection

and Analysis

Explained

byInterpretation

Explanatory Sequential Design

Phase 1 Phase 2

Intent: Use the qualitative data to help explain

the quantitative results

3

20

Qualitative

Data Collection

and Analysis

Quantitative

Phase –

Instrument design,

new variables or

intervention activities

Builds into

Interpretation

An Exploratory Sequential Design

Quantitative

Phase –

Instrument design

or intervention design

Phase 1 Phase 2

Phase 3 Intent: To explore first before

building into and testing a quantitative phase

4

How Do You Decide What Core Design to Use?

✓ How do you plan to “mix” the two databases?

✓ Merge or combine them?

✓ Connect them?

✓ If you plan on connecting them, what goes first, quantitative or

qualitative?

✓ Do your research experiences (and field) lean toward quantitative,

toward qualitative or both? (this might influence what you start with)

✓ When mixed methods is used in your field, what are the popular

core designs used?

How Do You Decide Whether You Have a Core

Design or a Complex Design?

✓ Am I simply integrating qualitative

and quantitative data in my project +Qual Quan

+Qual Quan

A Process of StepsOR

✓ Am I integrating qualitative and

quantitative data AND using this

integration within a larger

process of steps of research?

ASK YOURSELF:

Intervention Mixed Methods Design

Qualitative

Interviews

Before Experiment

(Exploratory

Sequential Design)

Experiment with an

Intervention and Pre- and

Post-Test Measures

Qualitative Interviews

During Experiment

(Convergent Design)

Qualitative

Interviews

After Experiment

(Explanatory

Sequential Design)

• Recruit participants

• Develop workable interventions

• Develop good pre-post-test

measures

• Examine participants’

experiences

• Modify treatment

• Explain outcomes

• Modify experiment

Complex Design:

A Mixed Methods Intervention (Experimental) Design Diagram

Integration Integration

Integration

2016

• N=20 Program Part.

• N=4 Experimental/

N=4 Wait List

Pre-Post-test Measures:

GHQ12/PANAS/FIQ/

Social validity

Phase４

Final InterventionTrial

QualitativeFollowup toTrial

• N=6Qual data:. Narrative Data

• TEA/TLMG

Phase3

2014-15

2014-

15

Phase2

PreliminaryIntervention Trial

Single group (quasi-

experimental design)

N=44Quan data:

Pre- Post-test Measures :

GHQ12/PANAS/FIQ/Social Validity

Qual data: Homework

Needs AssessmentN=58Qual data:Text DataKJ methods

Phase1

2013-

15

Exploratory Sequential Design



Integration Integration Integration

A Mixed Methods Intervention Design with Sequential Components to Develop and Test a Behavioral Program Intervention (Hirose

Creswell dissertation, Japan)

Overall Explanatory Sequential Design

= Intent to Develop and Test the Final Program Intervention

Interpretation

Needs

Assessment

Qualitative Stage

• Interviews

• Observations

• Documents

Theory Conceptualization

Specific to Setting

Quantitative stage

• Literature review stage

Instrument and Measures

Development

Quantitative stage

• Measures and

instruments

Program Implementation and

test

Quantitative stage

• Experimental intervention

based on quantitative

measures

Program Follow up and Refinement

Qualitative stage

• Interviews

• Observations

• Documents



A MIXED METHODS EVALUATION COMPLEX DESIGN

Names for the Complex Designs

▪ “Mixed methods intervention design with an exploratory

sequential core component” (or convergent, or explanatory,

or some combination)

▪ “Mixed methods comparative case study design with a

core convergent component”

▪ “Mixed methods evaluation design with both convergent and

sequential core components”

Complex Process

Core

Design

2. Include a diagram of

your mixed

methods design

How can you simplify

the complex procedures

of mixed methods so that

your design can be understood

by a reader?

Aspects of a Diagram to Consider:

❑ A diagram is a visual tool to help understand your project

(much like a theoretical model, or consort sampling figure)

❑ A diagram has many applications

❑ A diagram does not need to be overly complex – keep it simple

❑ A diagram needs to relate to your study aims

❑ A diagram needs to showcase your major procedures

❑ A diagram builds on and simplifies complex

procedures

A diagram is a figure inserted into the “Approach” section or

appended to the project that conveys the major procedures

of your mixed methods design.

• Data Collection

• Data Analysis

QualitativeQuantitative

Interpretation

Merging Data

Aim 1. Explain determinants

of aging through a survey.

Aim 3. Develop a complete

understanding of aging

Aim 2. Exploring personal

experiences with aging

Figure 1. Explaining Aging:

A Convergent Mixed Methods Design

How to Draw a Diagram of Your Convergent Design

Timeline

Start

Oct ‘17

End

Oct. ‘18

2 mos.

4 mos.

6 mos.

8 mos.

10 mos.

12 mos.

• Data Collection

• Data Analysis

• Data Collection

• Data Analysis

Quantitative Qualitative

• Data Collection

• Data Analysis

Connecting Interpreting

How Qual

Explains

the Quan

Aim 1. Explain

determinants of aging

through a survey.

Aim 3. Interpret why

social isolation does not

impact aging through

interviews.

Aim 2. Explain surprising

results in survey (e.g.,

family, social isolation,

etc.

Figure 2. Explaining Determinants of Aging:

An Explanatory Sequential Mixed Methods Design

How to Draw a Diagram of Your Explanatory Sequential Design

Timeline

11 mos.7 mos. 9 mos.5 mos. End

Nov. 2018

Start

Oct. 2017

Source: Ivankova, N. V. (2015). Mixed Methods Applications in Action Research. Los

Angeles, CA: Sage, p. 74

Ivankova (2015) diagram: Glason, J. Change, Chenoweth, et al. (2006) Study of Nursing Care

for Older Patients in an Acute Medical Ward, Journal of Clinical Nursing

Collecting and

analyzing

quantitative

and qualitative

data

Collecting and

analyzing

quantitative

and qualitative

data

If You Have Complex Design: Create Diagram of

Complex Procedures and Embed Your Core

Designs in it

An Example of a Mixed Methods Community-

Based Participatory Research (CBPR) Study

Core

MM

Design

Core

MM

Design


What if you had in your project a figure that related the

major parts of your study to provide an overview?

Study

Aims

Data

Collection

Data

AnalysisOutcomes Integration

Points

A FIGURE OR MATRIX

Aspects of an Implementation Matrix to

Consider:

❑ This table is an alternative to a mixed methods diagram

(entire project versus methods)

❑ Existing implementation matrices often do not contain all of

essential elements of mixed methods research

❑ This table provides a helpful way for a reviewer to easily

see the major elements of a study.

❑ An implementation matrix is popular in the health sciences.

An implementation matrix is a table that summarizes the major

elements in a project in an “Approach” section

Source: Creswell, J. W., Klassen, A. C., Plano Clark, V. L., & Smith, K. C. for the Office of

Behavioral and Social Sciences Research. (2011, August). Best practices for mixed

methods research in the health sciences. Washington, DC: National Institutes of Health.

Retrieved from: https://obssr.od.nih.gov/training/mixed-methods-research/

Source: Baez, J. C. Robert Wood Johnson Fellow

Project on Unidenitified Immigrants. 2018.

Example of an Implementation Matrix

Source: Justine Wu, MD, MPH, Assistant Professor, Family Medicine

and Obstetrics and Gynecology, University of Michigan, presented at

Designing a Mixed Methods Project: An Interactive-Participatory

Workshop, November 9-11, 2016.

Example of an Implementation Matrix

Source: Justine Wu, MD, MPH, Assistant Professor, Family Medicine and

Obstetrics and Gynecology, University of Michigan, presented at Designing a

Mixed Methods Project: An Interactive-Participatory Workshop, November 9-

11, 2016.

An Example of a Filled in Implementation Matrix

4. Reinforce integration throughout your

study

If integration is a centerpiece

for conducting mixed methods

research, why not convey its

importance several times in

your project?

How do we bake bread?

We IntegrateThe Ingredients

Source: Hirose, M.

We can integrate (or connect) the raisins into

the bread batter – the raisins stay intact

✓ Qual and quan stay separate but are

connected

We can integrate (or merge) the

flour into the batter, and the

flour dissolves

✓ Qual and quan merge into each other

We can include all of

the ingredients into the bread

✓ Embed both quan

and qual into the bread

Aspects of Integration to Consider:

❑ Integration is the centerpiece of a good mixed methods

study.

❑ Rigorous mixed methods research requires the identification

of where integration occurs in a study and the form it takes

❑ Consider stating and restating integration in several ways in

your project (e.g., title, study aim, justification, diagram)

❑ Integration should relate to your type of mixed methods

design

❑ Integration ”scripts” can help to convey your integration

Integration is the point in the research process in which the

quantitative data and the qualitative data meet. From this

integration additional insight (meta-inferences) results beyond the quantitative

and qualitative data.

Type of Design Where Integration Occurs

Convergent Design When quantitative and qualitative

data results are matched (compare)

Explanatory Sequential Design When quantitative results lead to the

qualitative data collection follow up

(explain)

Exploratory Sequential Design When the qualitative exploration

leads to the quantitative feature (e.g.,

instrument, etc.) (explore)

Intervention Trial Design When qualitative data is added into

an intervention trial (embedded

before, during, after)

Case Study Design When the quantitative and qualitative

results are compared to yield cases

(multiple convergence)

Social Justice/Participatory Design When the qualitative and quantitative

data are combined in different phases

of the research (multiple

convergence)

Evaluation Design When the qualitative and quantitative

data are combined in different stage

of the evaluation project (multiple

convergence)

Integration by Type of Design

Integration in an Explanatory Sequential Design

Collect/Analyze

Quantitative DataCollect/Analyze

Qualitative Data

• Collect N=250 surveys

• Measure behaviors leading

to lowered cholesterol

• Analyze data descriptively

and multivariately

• Collect N=30

interviews with

patients based

on surprising

quantitative results

• Code data and

identify themes

Explanatory Sequential Mixed Methods

Design

Interpret how

qualitative results help

explain quantitative results

Theory of

Behavioral Change

Integration

Phase 1

Phase 2

Conceptual Framework

or Theory

Scripts for Mixed Methods Integration Statements

▪ “Integration involved merging the results from the quantitative and qualitative

data so that a comparison could be made, and a more complete

understanding emerge than what was provided by the quantitative or the

qualitative results alone.” (Convergent Design)

▪ “Integration involved connecting the results from the initial quantitative phase

to help plan and explain the follow up qualitative data collection phase. This

plan would include what questions need to be further probed and what

individuals can help best explain the quantitative results.” (Explanatory

Sequential Design)

▪ “Integration involved gathering initial qualitative data, analyzing it, and then

using the qualitative results to build a new intervention (or measure or

instrument) that can be cultural specific that will be tested quantitatively with

a large sample.” (Exploratory Sequential Design)


If you are going to integrate

your data, wouldn’t you want

to first determine what

constitutes your quantitative

and qualitative data before

you integrate the data?

Aspects of data to consider:

❑ Clearly separate your quantitative and qualitative data

types (because you will need to be specific about integrating

them)

❑ Qualitative data consists of open-ended information;

quantitative data, close-ended information

❑ In preparation for integration and many parts of mixed

methods, labeling your quantitative and qualitative data is

helpful.

❑ Create a data table to clearly identify your types of data.

The data in a mixed methods study consists of collecting and

analyzing both quantitative (closed-ended) and qualitative (open-ended)

data, and interpreting their integration.

Instrument dataInstruments

Observational checklists

Document data

Qualitative Data (Open –Ended)

Interviews

Observations

Documents

Visual – Social Media Data

Quantitative Data

(Closed-Ended)

Types of Data

Data Source

Table

Based on:

Wittink MN, Barg FK,

Gallo JJ. (2006)

Unwritten rules of talking

to doctors about

depression: integrating

qualitative

andquantitative

methods. Ann Fam Med.

Jul – Aug; 4(4):302-9.

6. Use rigorous qualitative and

quantitative procedures

Just because you are

collecting and analyzing

both quantitative and

qualitative data, you still

need to be rigorous in both

approaches.

US Professional Association Developing Standards:APA Qualitative Article Reporting Standards (QARS)

New Edition:October, 2019

Task Force Developed Standards:

✓Qualitative Research✓Qualitative Meta-

Synthesis Research✓Mixed Methods Research

7. Create a Joint Display

To analyze data in mixed methods

research, you have (1) quantitative

data analysis (2) qualitative data

analysis and (3) mixed methods

integration data analysis. How do

you show integration in your

results?

Qualitative Results

Quantitative

Results

Interpretation

(Meta-Inferences)

A joint display is a table (or

graph) that arrays the

quantitative data with the

qualitative data so that an

interpretation can be formed

across the two databases

Scores

High

Medium

Low

A Joint Display Representing Integration Mixed

Methods Data Analysis in a Convergent Design

Theme 1 Theme 2 Theme 3 Theme 4

Quote Quote Quote Quote



An Example of a Joint Display for a Convergent Design

8. Include a Mixed Methods Study Aim

When you integrate, what question (or aim)

are you attempting to answer with

your integration procedures?

❑ It is a type of study aim that is little known.

❑ Follow the content then method approach

❑ For the quantitative aim, think in terms of variables and

their relationship

❑ For the qualitative aim, think about the central phenomenon

❑ For the mixed methods aim, think about your integration and what you will accomplish with the integration

❑ Resist attempts by others to reduce the number of your aims

❑ Order your aims to fit your design (could be qual first, or quan first, with mixed methods sliced in between)

Aspects of mixed methods aim to

consider:A mixed methods study aim is the aim in the project that captures

the integration of the quantitative and qualitative databases. As an

aim, it needs to be answered through data analysis in the study.

Type of Design Study Aim for the Mixed

Methods Design

Convergent Design The aim conveys a comparison of the

quantitative and qualitative results

Explanatory Sequential Design The aims describes how qualitative

results help to explain the quantitative

results

Exploratory Sequential Design The aim discusses how the qualitative

exploration leads to the quantitative

feature (e.g., instrument, etc.)

Intervention Trial Design The aim describes how the qualitative

data augments the intervention trial

(before, during, after)

Case Study Design The aims describes how the

comparative cases will be developed

based on quantitative and qualitative

results.

Social Justice/Participatory Design The aim indicates how the qualitative

and quantitative data will be integrated

into the social justice/participatory

approach.

Evaluation Design The aim identifies how the qualitative

and quantitative data will be integrated

into the evaluation design.

Mixed Methods Study Aim relates to your Design

Sample Mixed Methods Study Aims

▪ To compare the qualitative interviews about the elderly’s self-esteem with

their perceptions on a survey about their self-esteem. (Convergent Design)

▪ To explain the results from the survey about self-esteem with the elderly with

qualitative focus groups. (Explanatory Sequential Design)

▪ To explore the meaning of self-esteem for the elderly, to develop a survey

based on their qualitative interviews, and then to administer the contextually-

specific survey. (Exploratory Sequential Design)

Content Order for the Study Aims

▪ Template =

Content followed by methods (Best Practices, 2011)

Example (Explanatory Sequential Design Example):

1. Relate personal isolation to depression among older adults based on

survey data. (QUAN aim)

2. Explain how personal isolation affects depression among older adults by

obtaining personal experiences. (MIXED METHODS aim)

3. Explore personal isolation among older adults using semi-structured

interviews. (QUAL aim)

4. Develop an intervention (experiment) to compare different groups of

isolated older adults (FUTURE RESEARCH aim)

9. Add a Conceptual Framework into Your Design

What explains what you hope to

find? (based on the literature)

Aspects of a Conceptual Framework to

Consider:

▪ Preferably select one overarching framework

▪ Consider how it relates to your quan/qual/mixed procedures

▪ Point to where in your diagram (if used), the framework is being used

▪ Keep referring to the framework if possible in different sections of your project

▪ Many conceptual frameworks are social science frameworks – find a popular one

▪ Draw a visual of your framework and insert it into your project

▪ Realize that this could be a framework or a theory

▪ Realize that it could be inductively developed (as in qualitative research) or deductively used (as in quantitative research)

▪ Put an arrow from the label of your conceptual framework into your

diagram where it fits best

A conceptual framework in a mixed methods study provides

a priori explanation as to what you hope to find in your

quantitative, qualitative, and mixed methods analysis.

Place your Conceptual Framework into Your Design Diagram

Collect/analyze

Quantitative DataCollect/analyze

Qualitative Data

• Collect N=250 surveys

• Measure behaviors leading

to lowered cholesterol

• Analyze data descriptively

and multivariately

• Collect N=30

interviews with

patients based

on surprising

quantitative results

• Code data and

identify themes

Explanatory Sequential Mixed Methods Design

Interpret how

qualitative results help

explain quantitative results

Theory of

Behavioral ChangeIntegration

Phase 1

Phase 2

The Conceptual Framework can Help Explain Both the Quantitative

and Qualitative Sections of Your Mixed Methods Study

Source: Betancourt, T. S., Meyers-Ohki, S. E., Stevenson, A., et al. (2011). Using Mixed-Methods

Research to Adapt and Evaluate a Family Strengthening Intervention in Rwanda. African Journal of

Traumatic Stress, 2(1), p. 38.

1. Identify and label your mixed methods design

2. Include a diagram of your mixed methods design


4. Reinforce integration


6. Use rigorous procedures

7. Create a joint display to show integration

8. Include a mixed methods study aim

9. Add a conceptual framework into your design

What are these Advances?

New Directions in Advancing the

Methodology of Mixed Methods

Research

John W. Creswell, PhD

Senior Research Scientist, Michigan Mixed Methods

Research Program

University of Michigan, Ann Arbor, Michigan

MQIC Conference, Berlin, Germany, February 26-28, 2020

Mixed Methods Program

University of Michigan

Institute for Southeast

Asian Qualitative and

Mixed Methods Research,

Osaka and Honolulu

Mixed Methods Use in Humanitarian Program Impact ... · Mixed Methods Project: An...

Documents

Transcript of Mixed Methods Use in Humanitarian Program Impact ... · Mixed Methods Project: An...