Conducting Rigorous Qualitative Research

VA HSR&D Methods Cyberseminar SeriesJane Forman, ScD MHS

Ann Arbor VA Center for Clinical Management Research

May 11, 2009

2

Objectives of the SeminarBriefly describe key uses and

features of qualitative research

Describe rigorous procedures for: Formulating research questions Research design Sampling Data Collection Data Analysis

3

How do we use qualitative research in the health sciences?• Understanding complex social processes difficult

to measure quantitatively (e.g., organizational culture)

• Communication studies

• E.g., patient-physician interactions• Implementation studies

• Formative evaluation

• Mechanisms underlying outcomes• Instrument development

• Qualitative exploration needed to develop survey items

4

Type of Study:

  Qualitative Quantitative

Goal UnderstandingMeasurement; Determining associations

Research Process

Iterative & Emerging

Sequential & Fixed

Sampling Purposeful Representative

Data Collection

Open-ended Closed-Ended

Data Analysis

Primarily Inductive Deductive

Qualitative vs. Quantitative Methods Qualitative vs. Quantitative Methods

5

Engagement: Constant and Iterative

Research QuestionsSampling

Data CollectionAnalysis

Rigorous Procedures

7

Research Questions: Clearly defined and focused Nature of qualitative research questions

Explore, explain, describe, understand....experiences, processes, beliefs, perceptions...

Don’t think in terms of variables that you’re trying to relate to each other.

Do think in terms of identifying factors and understanding how they work.

Non-directional language

AVOID BEING OVERLY AMBITIOUS

What are the communication problems between nurses and physicians?

How do nurses perceive communication problems with physicians in the ICU?

How do nurses in ICUs at VA hospitals perceive communication with physicians during rounds?Sub-question: How do perceptions differ across types of ICUs?

Focus

Focus

8

What data sources and methods are appropriate?What can I do given resource constraints?

Research Questions

Data sources and methods

Justification Practicalities (eg resources, access, skills)

Ethical Issues

  

       

  

       

  

       

  

       

  

       

(Mason 2002)

Research design: Matching Data Sources and Methods to Research Question

Research design: Matching Data Sources and Methods to Research Question

9

Triangulation

Inclusion of multiple data sources and methods in a study.

Different methods and sources reveal different aspects of empirical reality, e.g., Comparing what participants say to what

you observe. Comparing the perspectives of people who

have different points of view

Test for consistency. Convergence strengthens validity. Understanding inconsistencies can be

illuminative and important.

10

Sampling

Sampling is a key strategic part of the detailed design that you develop before you begin your study.

You make sampling decisions during data collection based on what you find in your data.

Topics: Purposeful vs. representational sampling Types of purposeful sampling Sample size

11

Sampling: Representational vs. Purposeful

Representational in quantitative studies Goal: Enable generalizations from study

samples to populations.

Purposeful in qualitative studies Goal: To understand a phenomenon,

not to represent a population. Select of information-rich cases for

intensive study. ▪ Cases that will provide information needed to

address your research questions.

12

Selected Types of Purposeful Sampling

Typical caseExtreme or Deviant caseStratifiedMaximum variationHomogenousSnowball or ChainRandomConvenience

Source: M. Q. Patton, pp.230-246

13

Types of Purposeful Sampling: Examples Typical Case

Useful in understanding, e.g., how an intervention is experienced by a typical patient.

Extreme Case Useful in understanding unusual cases or

outliers, E.g., outstanding successes (Covey), notable failures

Often useful, though comparison, in understanding typical cases.

Stratified Vary on characteristics relevant to the topic

under study. Creates comparison groups Main purpose is to capture major variations

rather than identify common core themes.

14

Sample Size

Factors that affect sample size: Number of comparison groups (more

comparisons larger sample size). Detail, complexity and depth of data

(more detailed, complex, and in-depth smaller sample size)

Saturation Seeing nothing new in newly sampled units Data is sufficient to create a valid product Depends on judgment based on experience

There’s a tendency to over-sample in the health sciences.

▪ Investigators used to representative sampling ▪ Haven’t yet experienced the volume of qualitative data

generated and the intensity of data analysis

15

Data Collection

• Choose data collection technique(s)• There’s a tendency to design studies around

the data collection technique vs. matching techniques to research questions.

• Systematically translate research questions to data collection instruments (e.g., interview guide, observation protocol)

• Additional procedures to ensure rigorous and valid data collection

16

Choosing data collection techniques: Individual InterviewsWhen individual’s experience and

unique interpretation of it is of interest

Decision: How structured? The more structured the interview, the

less rich your data will be Key decision in study design

17

Structured, open-ended vs. semi-structured interviews Structured, open-ended

Fixed questions asked in a particular order

When less complex data is appropriate Easier to analyze, e.g., by question

Semi-structured Cover topics while following the

interviwee’s train of thought Generates rich, narrative data Data analysis resource intensive and

more highly interpretive

18

Semi-structured interviews: Skills and experience especially important

Challenging to conduct Simultaneous management of

intellectual and social dynamics Establishing rapport Listening skills “think[ing] on your feet…in ways

consistent with your research questions” (Mason, p.67)

▪ Constantly making choices about where to go next

19

Choosing data collection techniques: Focus groups Facilitated group interviews

When breadth of data generated is more important than depth.

When group interaction will help address your research question.

Myth – focus groups are easier and less time-consuming to conduct than individual interviews.

Focus groups and interviews can both be used in a single study. E.g., start with focus groups, then conduct

interviews to gather more in-depth information about selected topics

20

Choosing Data Collection Techniques: Observation

Often used to triangulate with data produced through other techniques.

Objects: e.g., daily routines, interactions, non-verbal behavior, physical space

Challenges Figuring out what you want to observe

and how▪ Can’t just go and “see what’s there”

Skills, training Resource constraints when combined

with other techniques

21

Systematic translation of research questions to data collection instruments

Process of developing instrument serves to sharpen your idea of what you want to get from your data.

Not a restatement of your research questions What is the best way to address them.

Use your understandings of the topic from your previous work, the literature (theoretical and/or empirical), and experience in living. Components of theoretical framework and/or

conceptual model you develop for your study.

22

Example: Semi-structured Interview Guide Lists the topics that are to be explored

in the interview, usually along with questions that suggest lines of inquiry.

Narrower components of each topic listed as probes under each topic, to guide interviewer to elicit more detail.

Most important for early interviews. Interview usually relies less and less on the

interview guide as the study goes on.

23

Additional procedures to ensure rigorous and valid data collection

Involvement of data collectors as early as possible in the study. Best if PI is involved in data collection

▪ Knows the most about what data will best address research questions.

▪ Particularly important when conducting sem-structured interviews.

Data collection training Monitoring quality

“Qualitative analysis transforms data into findings. No formula exists for that transformation. Guidance, yes. But no recipe.”

-- M.Q. Patton

Here’s some guidance.

25

Data Analysis

Goal: Create new knowledge from raw, unordered data. Make sense of your data.

Qualitative analysis requires considerably more than just reading to see what’s there.” (Patton, 2001)

It’s vital to develop a systematic approach to data, emphasizing study aims.

Data analysis is constant and iterative. Be engaged in data analysis the first piece of data is collected.

Focus data analysis early on, but be open to new topics to explore.

26

Data Analysis Procedures Memoing Coding scheme development

Coding agreement Coding of the data Applying codes and rearranging data by

code Summaries by units of analysis, by code Synthesis/Interpretation

Data Display Conclusion Drawing and Verification

27

Memoing

Create memos throughout the study Write anything that will help make sense

of the data. Get engaged in the data by recording early

thoughts and analytic hunches. Identify and sharpen key categories and themes Develop and record data interpretation and

findings. Document analytic procedures and

decisions. Audit trail that makes analysis process

transparent

28

Codes: Why use them?

Provide rigor to the analytic process.Allow rearrangement of the data into

analytically meaningful categories with study aims always in mind

Create a way to exhaustively identify and retrieve data out of a data set

Enable the researcher to see the bigger picture of data that is difficult to see in transcript form.

29

Codes: Where do they come from?

Deductive codes: Questions and categories generated during

the conceptual and design phases of the study▪ Research questions▪ Conceptual model▪ Interview guide▪ Unit(s) of analysis (e.g., clinic, ICU, provider

group) Inductive codes:

▪ The data.

30

Coding Scheme Development

Iterative approach2 or more researchers involved

Multiple perspectives Resource intensiveProduct is a codebook: Code title,

definition, example

31

Early development

Team members engage in high quality conceptualization through an iterative, negotiated process.

Produce an initial list of codes Who?

All team members who might be involved in coding or later stages of the analysis, including PI

How? Read and code transcripts independently Highlight text relevant and interesting test and

comment in the margins (preliminary coding) Test a priori codes Meet to share impressions of the data.

32

Further Development

Refine coding definitions, add or delete codes Continue iterative process using the initial

codes, usually with a subset of the research team. Independent coding, meeting

Produce codebook with “stable” codes Clear definitions; applied to text consistently by

coders Document changes in codes and code

definitions. Transparent and systematic process

33

Codebook Excerpt

CODE DESCRIPTION EXAMPLE

ENVIRONMENTAL CONTEXT [ENV]

Tree Node

General description of the environmental context not included in the other ENV codes.

 

External guidelines [GUIDE]

Child Context related to external entities that impacts the organization being interviewed and/or their ability to adopt IC practices. Reference to the role or influence of guidelines in getting practices adopted or implemented. Examples include JCAHO, CDC guidelines, Keystone initiative, and standards of practice reporting mandates 6/13/06. Also, references to relationships with outside entities.

"We've done a good bit with trying to work through JCAHO stuff and change our preps and things like that.""Keystone has really been instrumental in getting the resources we need."

Health system [HS]

Child Description of umbrella entities that are over the purview of the hospital. Examples include the VISN for the VA, a corporate office or sister entities in the case of privately owned hospitals that may be part of a network.

 

34

Coding agreement: establish coding reliability

Coding agreement: Two or more coders using the same codebook consistently apply the same codes to the same text segments.

35

Measurement statistic

Use quantitative measure of agreement to establish coding reliability

▪ Standard: Kappa statistic >= .7

Underlying philosophy: positivist view that bias introduced by human involvement in research must be minimized to increase the validity of research results.

36

Consensus approach: negotiated agreement processWhen discrepancies occur coders

discuss the rationale they used to apply particular codes to the data.

Explain their perspectives and justifications, how and why it differs from other team members’ perspectives, and reach consensus on how the data ultimately should be coded. Involves reflexivity: confronting your

assumptions, recognizing how they affect what you see

37

Consensus approach: underlying philosophy

Constructivist view that validity is derived from community consensus, through the social process of negotiation. Measuring coding agreement leads to

over-simplification that compromises validity

Reflexivity and reason-giving are more important than achieving a pre-specified level of agreement independently.

38

Which approach?

Factors to Consider Complexity of data and degree of

inference in analysis▪ More complex, higher inference – use

consensus approach Use of the data?

▪ E.g., if transforming qualitative to categorical data, agreement statistic appropriate.

Who is your audience?▪ Some journals expect agreement statistic,

although that’s changing

39

Applying codes and rearranging data

Apply codes to all of the text in the data set. When applying a code to a segment of

text, include text that will provide sufficient context so that its meaning can be discerned out of the context of the transcript.

Text is rearranged into code reports, which list all of the text to which each particular code has been applied.

40

Data Summaries

Descriptive and Interpretive Summaries Includes main points obtained from reading

code report, quotations to provide evidence for those points, an interpretive narrative at the code and case levels.▪ Case summaries grouped together so each can be

examined before making cross-case comparisons Integrate memos

Often done by analysts at the project manager level. Entire team, including those doing the

summaries, must understand which questions the team wants to answer, papers team wants to write

PI and team engagement throughout study

41

TranscriptsTranscriptsTranscripts

Alternative to traditional coding: Team-based Approach

Assign transcripts to pairs of analysts

Develop Memos (+ Quotes)

Large Group Discussion

Modify Memos(+Quotes)

Begin Case Analysis

Pre-existing frameworkor codebook

Initial Memos

Group Memos

Features:• Move more quickly into analysis & interpretation• Multi-disciplinary group process• Multiple perspectives coalesce through consensus• Foster reflexive thinking through deliberation• Efficient for principle investigator• Reduce volume of data going into analysis• Ensure validity

CaseTranscripts

42

Interpretation and synthesis

Data analysis products are further analyzed, interpreted, and synthesized to formulate results.

Identify patterns, formulate preliminary conclusions Tool: data display

▪ Helpful in looking across cases

43

Tool: Data Display

Leadership Provider Commitment

Resistance to Change

Champions

Site 1

Site 2

Site 3

Site 4

44

Ensuring validity of findings

Conclusion-drawing and verification Test conclusions by going back to the data to assess evidence. Look for data that supports alternative explanations Look for negative or deviant cases

▪ e.g., minority group in formative evaluation – why are they in minority? What about them or their situation is different and what does that tell us about the program we’re evaluating?

Ensure that findings are not driven by forgone conclusions or preconceived biases, and that findings are grounded in sound evidence. Reason-giving and examining assumptions in group discussions

Assess degree to which findings contribute to theory or practice, i.e., how useful are they

Review of rigorous procedures for conducting

qualitative research

46

Sustained Engagement

Sustained engagement of the whole team throughout the research process vital to maintaining rigor PI = main interpreter Data collectors Coders/analysts Use multiple coders and analysts Involvement of PI and other team members at

all stages Engage in reflexive discussion

47

Rigorous Procedures: Study Design and Sampling

Match data sources and methods to research questions

Best design given resource constraints

Triangulation of data sources and methods

Purposeful sampling strategy tailored to your study

48

Rigorous Procedures: Data Collection

Systematic translation of research questions into data collection tools

TrainingTranscription rules and

verification

49

Rigorous Procedures: Data Analysis

Systematic approach Constant and iterative Focused Rigorous code development Coding agreement process Document procedures, changes in

codebook, analytic decisions Interpretation and synthesis involving

reflexive discussion to ground findings in sound evidence

Conclusion verification

5050

Michael Q. Patton, Qualitative Research & Evaluation Methods, 3rd ed., Sage, 2002

John Creswell, Qualitative Inquiry and Research Design, Sage

Jennifer Mason, Qualitative Researching, Sage, 2002

Margarete Sandelowski, series of articles on qualitative research in Research in Nursing and Health.

Catherine Marshall and Gretchen B. Rossman, Designing Qualitative Research, 4th Ed., Sage, 2006

Interviewing: Robert S. Weiss, Learning from Strangers, The Free Press, 1994

Focus Groups: David L. Morgan and Richard A. Krueger, The Focus Group Kit, Sage, 1998

Data Analysis: Forman and Damschroder, Coffey and Atkinson

ResourcesResources

51

Questions? Comments?

Jane Formanjaneform@umich.edu