DESIGNING AND CONDUCTING MIXED METHODS STUDIES

Beth Angell and Lisa Townsend

Workshop for the 2011 Society for Social Work and Research annual meeting

Overview of Workshop

Definitions and terminology of MM ResearchPhilosophical AssumptionsMixed Methods: Nuts and BoltsBreakPlanning your mixed methods study

SamplingData Collection

Data AnalysisEvaluating Mixed Methods StudiesRepresenting Mixed Methods ResearchExamplesQ and A and Technical Assistance

Resources

Creswell & Plano Clark (2011) Designing and conducting mixed methods research.  Thousand Oaks, CA:  Sage Publications, Inc.

Teddlie & Tashakkori (2009) Foundations of Mixed Methods Research:  Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences.  Los Angeles:  Sage Publications, Inc.

Consensus Definition of MM Research

“Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the purpose of breadth and depth of understanding and corroboration”

Johnson et al. (2007). 

Key terminology

Qualitative (QUAL) component and Quantitative (QUAN) component are often referred to as strands

Philosophical Issues

Quantitative Tradition

Philosophical underpinnings:  positivism/post‐positivism

Deductive logic

Data are represented numerically

Associated terms:  survey research, probability sampling, experimental and quasi‐experimental designs, descriptive and inferential statistics

Qualitative Tradition

Philosophical underpinnings:  constructivism

Inductive logic

Data are represented textually or pictorially

Associated terms:  grounded theory, ethnography, case studies, purposive sampling, categorical vs. contextualizing strategies, trustworthiness, credibility

Mixed Methods Tradition

Philosophical underpinnings:  pragmatism

Both deductive and inductive

Data are represented both numerically and textually/pictorially

Associated terms:  concurrent (parallel) and sequential mixed designs, triangulation, data conversion, inference quality

Philosophical debates about mixed methods

Incompatibility thesis – fundamental differences between QUAN and QUAL approaches are so great that methods cannot be mixed

Pragmatism – what is the best way to answer a research question; both methods offer different ways of answering research questions

Mixed Methods:  Nuts and Bolts

Characteristics of MM Studies

Involves collection and analysis of qualitative and quantitative data in ways that are: 

rigorous framed epistemologically/theoretically

The methods are mixed by ordering them sequentially merging them embedding one strand within the other

Combines the data within the context of a single study or research programEncapsulates the strands within an overall research design that guides the study as a whole

Research Questions that Call for MM

Exploring the meaning of a construct or phenomenon from more than one perspective

Explanation of anomalous findings or getting behind the mechanism of action of an effect

Theory development followed by testing/extension

Measure development using grounded concepts

Augmenting evaluation studies with better understanding of intervention implementation

Ways That Designs Vary

Level of interaction between strands

Relative priority of strands

Timing or pacing of each strand 

Point of interface (at which point in the research process are the strands mixed?):  during interpretation, data analysis, data collection?

Research stance, epistemology

Typology of Mixed Methods Designs

Convergent parallelExplanatory sequentialExploratory sequentialEmbeddedCaveat:  evolving field with evolving language

Adapted from Creswell & Plano Clark (2011)

Convergent Designs

QUAN and QUAL strands are conducted separately yet concurrently and merged at the point of interpretation

Equal priority given to each strand

Used to form a more complete understanding of a topic, or to validate or corroborate quantitative scales

Convergent Parallel Design

QUAL 

Data Collection

Compare or Relate

Interpretation/Meta‐

Inference

QUAN 

Data Collection 

QUAL 

Data Analysis

QUAN

Data  Analysis

Convergent Parallel Design Example: Conceptual Adequacy of the Convergent Parallel Design Example: Conceptual Adequacy of the Drug Attitude Drug Attitude Inventory for YouthInventory for Youth

QUANData

Collection

QUANData

Analysis

QUALData

Collection

QUALData

Analysis

Design Decisions Design Decisions

Choice of instrumentQuestionnaireRating scaleSampling ConvenienceRepresentative

Semi-structured interviewsYouth SEMIParent SEMI

Compare Contrast

Mixed Methods Question: Can prediction of youth attitudes Mixed Methods Question: Can prediction of youth attitudes toward psychotropic treatment be improved by knowledge about thetoward psychotropic treatment be improved by knowledge about thefactor structure of the DAI in youth and their subjective experifactor structure of the DAI in youth and their subjective experiences ences of treatment?of treatment?

DemographicsYouth DAIParent DAIAdherence RatingsClinical Scales (CDRS,YMRS,CBCL)

Choice of methodInterviewEthnographyFocus groupSamplingPurposiveConvenienceSetting

Metainference

Townsend, Floersch, & Findling, 2010

Explanatory Sequential Design

Methods are implemented sequentially, (QUAN →QUAL)

Used when researcher wishes to use qualitative findings to help interpret or contextualize quantitative results

QUAL Data Collection and Analysis

Interpretation/Meta‐

Inference

Follow up with

QUAN Data Collection 

and Analysis

Explanatory Sequential Example:  ACT Social Network Study

ACT Randomized Trial :  No Social Network Effects

QUAN Analysis of Social Network Predictors

QUAL study of RCT Subsample

QUAN Data Collection 

and Analysis

QUAL  and QUAN Data Collection and Analysis

Interpretation/Meta‐

Inference

Interpretation/Meta‐

Inference

Angell & Test, 2002; Angell, 2003

Exploratory Sequential Design

Methods are implemented sequentially, (QUAL →QUAN)

The QUAL strand is considered exploratory, to be followed by further testing and verification during the QUAN phase

Qualitative Data Collection and Analysis

InterpretationBuilds toQuantitative 

Data Collection and Analysis

Exploratory Sequential Design Example:  Measuring Procedural Justice (PJ) in Police Encounters

Review of existing instruments and literature led to research question:  Do existing PJ instruments capture features of  contacts between police and citizens with mental illness?

QUAL strand:•Consumer interviews•Analysis of discrete encounters using grounded dimensional analysis

Interpretation: PJ experiences are a) contextualized by negative expectations and b) sensitive to small gestures of humanity

QUAN strand•Instrument development•Cognitive interviewing•Expert review•Survey of consumers using final instrument (PCES)•EFA and Rasch Analysis

Interpretation: PCES predicted reactions to police encounter (resistance, cooperation)

Watson, Angell, Vidalon, & Davis (2010)

Embedded Design

Researcher conducting either a QUAL or QUAN study embeds a smaller strand of the other method, as an enhancement

Secondary strand can be concurrent or sequential 

Qualitative or Quantitative Design

QUAL or  QUAN Data Collection and Analysis

QUAL or QUAN Data Collection and Analysis (before, during, or after)

Interpretation/Meta‐Inference

Embedded Design Example:  CTI Evaluation and Fidelity Study

RCT of Critical Time Intervention (CTI) for Men Leaving Prison.  QUAN Data Collection, n=220

Fidelity/Process StudyQUAL data collection, n = 24 

Data Collection Decisions:InterviewsFocus GroupsFieldnotesRecord abstractionSampling criteria

MM Research Questions:  In what ways is Critical Time Intervention modified or adapted when used with a population of recently released prisoners?  What processes contribute to the adapted program’s level of effectiveness?

Data Analysis Decisions:Coding (open, selective, axial) of interviews and documents/Narrative analysis?Draine, Angell, Barrenger, & Kriegel (in progress)

Variations on the MM Designs

Multiphase format

Multilevel format

Monostrand Conversion (not truly mixed methods):  conversion of QUAL data to QUAN or QUAN data to QUAL, without additional strands

Transformative stance

Questions?

Break

Planning Your Mixed Methods Study

Sampling:  General Considerations

Strategy chosen should be appropriate to each respective strand

Balance between saturation of phenomenon or theory (qualitative goal) and representativeness (quantitative goal)

Sampling Strategies (Teddlie and Tashakkori, 2010)

Parallel mixed methods sampling (parallel use of probability and purposive strategies, either concurrently or with a time lapse). 

One sample may be a subset of the otherBoth studies may use same total sample

Sequential mixed methods sampling (information from the first sample is used to draw the second)

Multilevel mixed methods sampling:  using probability and purposive sampling techniques at different levels of analysis (e.g., clinicians and clients)

Data Collection(Teddlie & Tashakkori, 2009)

Self‐Report TechniquesInterviewsQuestionnairesAttitude  ScalesPersonality inventoriesProjective instruments

Observational MethodsParticipant observation, non‐participant observation

SociometrySocial network analysis

Secondary Data AnalysisArchival analysisMeta‐analysis

Multiple Modes of Data Collection (Tashakkori & Teddlie, 1998)

Data Analysis

Quantitative Data Analysis

Descriptivesummarizing data, looking for trends and patterns; means, frequencies, measures of variability

Inferentialhypothesis testing, inferences about a population characteristic; significance tests (χ2,t, F), multiple regression, ANOVA, MANOVA, MANCOVA, hierarchical linear modeling, time‐series, event history

Qualitative Data Analysis

Often ongoing during data collection (e.g., purposive sampling, modification of interview questions, etc.)

Grounded theoryThematic analysisNegative case analysis

FRACTURING VS. CONTEXTUALIZINGCategorical strategies:  produce categories that facilitate comparisons; e.g., constant comparative methodContextualizing strategies:  interpret narrative data in the context of the whole text, focusing on interconnections between statements, events, etc.; e.g., phenomenology

SIMILARITY VS. CONTRAST

Mixed Methods Data Analysis(Creswell and Plano Clark, 2011)

QUAN+QUAL = converge results  CONVERGENT DESIGNQUAN → qual = explain results  SEQUENTIAL EXPLANATORY DESIGNQUAL → quan = generalize findings SEQUENTIAL EXPLORATORY DESIGNQUAN (+qual) = enhance experiment  EMBEDDED DESIGNTRANSFORMATIVE DESIGN – uses a transformative theoretical perspective to advocate for social change, address social injustice, or give voice to marginalized/underrepresented group.MULTIPHASE DESIGN – a program of research that involves several studies; can have combinations of sequential and concurrent designs

Mixed Methods Data Analysis(Creswell & Plano Clark, 2011)

Convergent parallel:  merged data analysis for purposes of comparing results

Collect and analyze QUAL and QUAN dataStrands are analyzed independently (could be qualitizing/quantitizing strategies also)How will the two strands be compared?How will they be represented?

Explanatory:  connected data analysis to explain findingsCollect and analyze quantitative data; derive second research questionDesign and conduct qualitative researchAnalyze qualitative data for answers to secondary research questionLink results from both strands – how do qualitative results explain quantitative findings?

Convergent Parallel Design: Data Analysis of the Drug Attitude Inventory

QUANData

Collection

QUANData

Analysis

QUALData

Collection

QUALData

Analysis

Univariate descriptivesBivariate correlationsStructural Equation ModelingFactor analysisParallel Analysis(SPSS, LISREL)

In vivo codesIntermediate codesSuperordinate codesPeer reviewConstant comparative approach(Atlas TI)

Compare Converge

Can prediction of youth attitudes toward psychotropic treatmentCan prediction of youth attitudes toward psychotropic treatment be be improved by knowledge about the factor structure of the DAI in improved by knowledge about the factor structure of the DAI in youth and their subjective experiences of treatment?youth and their subjective experiences of treatment?

Does the factor structure of the DAI in adults fit the youth data?

If not, what is the factor structure of the DAI in youth?

How well do DAI items correlate with one another?

Do they measure a single construct or multiple constructs?

Are there elements ofyouth medication experience that the DAI does not capture?

Mixed Methods Data Analysis(Creswell & Plano Clark, 2011)

Exploratory:  connected data analysis to generalize findings

Collect and analyze qualitative data; use qualitative data to design quantitative componentCollect and analyze quantitative dataLink results from both strands:  how do quantitative results extend qualitative findings?

Embedded design:  merged (concurrent design) or connected (sequential design) analysis

Collect and analyze primary data set; decide how embedded data will be used and where they should be incorporated into the primary analysisAnalyze secondary data set dictated by where it is embedded in the larger designHow do the embedded findings integrate with the primary study findings?

Evaluating Mixed Methods Studies

Mixed Methods Validation Framework (VF)(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)

Five elements:Foundational element

Quality of literature review and theory baseConstruct validation

Validity of QUAN, QUAL, and mixed elementsInferential consistency

Consistency of links between various strands of the study (see table on following slide)

Utilization/historical elementWhether and how the study’s findings went on to be used in future work

Consequential elementSocial acceptability and consequences of study findings

Construct Validation(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)

Representing Mixed Methods Data(Creswell & Plano Clark, 2011)

Side‐by‐side comparison

Joint comparison

Merged category/theme display

Writing the article(Creswell, 2003)

Introduction – explicit integration of both paradigms from the outset

Literature review – integration of inductive/deductive reasoning, why the literature needs this type of study

Posing the research question – what are the questions and why do they call for two paradigms?

Methods – present both methodologies, in their respective languages, integrated under the umbrella of the research question

Results – present results of both modes of data collection

Discussion – role of meta-inference

Findings from DAI Study 

(Townsend, Floersch, & Findling, 2010)

Example I

Study Flowchart

QUANData

Collection

QUANData

Analysis

QUALData

Collection

QUALData

Analysis

DemographicsYouth DAIParent DAIDecision-Making ScalesAdherence RatingsClinical Scales (CDRS,YMRS,CBCL)

Univariate descriptivesBivariate correlationsStructural Equation ModelingFactor analysisParallel Analysis(SPSS, LISREL)

Semi-structured interviewsYouth SEMIParent SEMIBrief Parent SEMI

In vivo codesIntermediate codesSuperordinate codesPeer reviewConstant comparative approach(Atlas TI)

Compare Converge

Can prediction of youth attitudes toward psychotropic treatmentCan prediction of youth attitudes toward psychotropic treatment be be improved by knowledge about the factor structure of the DAI in improved by knowledge about the factor structure of the DAI in youth and their subjective experiences of treatment?youth and their subjective experiences of treatment?

Parallel Data Reduction Strategies

@stigma {0-56}

&Desire_for_Normality {0-0} &Crazy_Identity {0-0} &Educating_Others {0-0}

+want_to_live_normal_life_without_meds {1-0}

+nobody_else_takes_meds {2-0}

+for_crazy_people {2-0}

+Labeled_psycho {2-0}

+did_bipolar_slide_show_at_school {1-0}

+I_like_explaining_meds_to_people {1-0}

Structural Equation Model OneDAI Original Factor Structure

RMSEA .061 [ideal = <.05 (Kaplan, 2000)]

CFI .925 NNFI .913

[ideal = >.95 (Kaplan, 2000)]df 258

X2 420.38

Exploratory Factor AnalysisFactor Selection Criteria:

Maximum likelihood estimationEigenvalue > 1.0Minimum item loading > .30Retained 4+ items>4 items but differentiated well from other factorsQualitative data indicate retention of items/factors is justified

EFA OneTwo factors were not interpreted further because they were each comprised of only one item  (items 8 and 13) and had values > 1.0. (Heywood cases)

EFA Two‐28 items (Supplemented by Parallel Analysis)Two factors retained, accounting for 36.61% of the variance in DAI score.Factor labels:

“Positive Feelings toward Medication”“Negative Feelings toward Medication”

Four items did not load on any component (10R, 11R, 20R, and 30)Cronbach’s alpha = .889Youth DAI correlated positively with youth self‐reported adherence (r = .205, p<.05)DAI showed no significant correlations with clinical outcome measures (CBCL Competence and Symptom Scales).

Qualitative Analysis Methods

Constant comparative approach  – analysis of in vivo codes followed by intermediate level coding and synthesis into higher order superordinate concepts.

@stigma {0-56}

&Desire_for_Normality {0-0} &Crazy_Identity {0-0} &Educating_Others {0-0}

+want_to_live_normal_life_without_meds {1-0}

+nobody_else_takes_meds {2-0}

+for_crazy_people {2-0}

+Labeled_psycho {2-0}

+did_bipolar_slide_show_at_school {1-0}

+I_like_explaining_meds_to_people {1-0}

Summary of Qualitative Themes

DAI‐related concepts

Positive feelings

Negative feelings

Health/Illness ModelHealth/Illness Model

Internal LOC

External LOC

Relapse PreventionRelapse Prevention

Concepts not represented in the DAI

Balanced responsibility

Ambivalence

Change over Time

Adherence if effective

Expectations

Inclusion in treatment decisions

Autonomy

Stigma

Side‐by‐Side Comparison Example:  Conclusions Section

In summarizing the findings of the adolescent qualitative and quantitative analyses, it is apparent that the seven dimensions found in the original DAI employed with adults are present in teens’ conceptualizations of their views of medication.  However, adolescents may think about those dimensions, such as side effects and relapse prevention, differently than adults do.  These findings are supported by both the EFA and qualitative results.  EFA findings indicate that side effects differentiated into specific cognitive effects as well as other harms for youth.  Adolescents did not appear to view relapse prevention as the adults did, instead evaluating their perceived need for medication in relation to their symptom stability in forming their attitudes toward medication.

The qualitative data highlighted the presence of eight additional themes not reflected in the original factor structure of the DAI, including important themes such as personal autonomy, inclusion in medication decision‐making, the experience of stigma, and level of personal responsibility vs. external control over behavior.  These themes highlight that medication experiences and the formation of attitudes toward pharmacological treatment for youth are complex and multiply‐determined, rendering it difficult to represent adolescent attitudes toward medication fully with a single instrument.  These findings point to the potential value of creating individualized, qualitative assessment tools to capture youth experiences with medication rather than relying on a single quantitative measure or set of subscales.

Quantitative Findings Qualitative Findings

Discussion section highlights divergent results

Joint Data Display ExampleHighlights Convergence of Factor Analytic and Thematic Analysis Results

Positive Feelings Factor Loading

2 – good outweighs bad .474

21 – thoughts are clearer .789

26 – happier on meds .487

29 – in better control .598

Qualitative Themes Quotations

Emotional “I can either not take it and be a grouch, or take it and be happy.  So I would much rather take it and be happy.”

Cognitive “Sitting down, paying attention to the teacher, not talking.  Just paying attention and doing what I need to do.”

Physical “I gain energy to want to do my work and listening and focus and try to do my best and give 110%.”

Negative Feelings Factor Loadings

5R – take b/c of pressure from others

.315

14R – medication is slow‐acting poison

.714

19R – I’d rather be sick than taking medications

.658

Qualitative Themes Quotations

Emotional “I don’t like sometimes I feel like out of my body or just like not myself, more anxious sometimes.”

Cognitive “Well I had it told me the Topamax or whatever causes short‐term memory loss, so I assume it’s the Topamax or whatever.”

Physical “Make me sick.  Make me get the bubble guts and stuff.  And I wish that there was a medicine that didn’t have side effects.”

Merged Data Display ExampleCounts of quotations represented graphically to demonstrate convergence 

with factor analyses

“positive” and “negative” factors supported by all forms of QUAN and QUAL evaluation

Distribution of DAI-Related Quotations across Respondents

0

5

10

15

20

25

30

35

40

4 7 10 12 13 14 16 17 18 20 22 23 24 25 27 29 30 32 38 41

Respondent

Num

ber o

f Quo

tatio

ns

PositiveNegativeIllness ModelILOCELOCHarmRelapse Prev.

Number of adolescent quotations linked to original DAI factors

Positive Subjective

Feeling

Negative Subjective

Feeling

Health/Illness Model

External Locus of Control

Internal Locus of Control

Harm/Toxicity

Relapse Prevention

103 31 21 20 10 48 34

Example II

Forensic Assertive Community Treatment (FACT) Evaluation (Angell & Watson, in progress)Key mixed method questions:

How is ACT modified in the context of prison reentry?How does the agency’s recovery‐oriented mission shape the translation of ACT to FACT?How does engagement of consumers occur and to what extent is leverage involved?What are the unique features of client‐provider relationships in FACT?How does engagement relate to rates of success (avoidance of reincarceration)?

Overall Design:  Convergent Parallel

Sample=21 adults with mental illness leaving prison and entering FACT program

QUAN Data Collection

QUAL Data Collection

Panel Design•Client reported outcomes assessed at BL, 1, 3, 6, 9, 18 mos• staff reported outcomes monthly

Ethnographic approach• Participant and passive observation•Interviews with consumers  and staff• Record abstractionsQUAN 

Data AnalysisQUAL 

Data Analysis

Within‐Subject Cross‐Data Comparisons

Interpretation/Meta‐Inference

Analysis Strategies

Sample=21 adults with mental illness leaving prison and entering FACT program

QUAN Data Collection

QUAL Data Collection

QUAL• Descriptive•Pre‐Post Comparisons• Outcome differences by risk level

QUAL• open and focused coding• creation of synthetic participant narratives

QUAN Data Analysis

QUAL Data Analysis

Within‐Subject Cross‐Data Comparisons

Interpretation/Meta‐Inference

Mixed Analysis:Comparing phenomena•Across data type•Across time•Across subjects

Q and A/Technical Assistance