Thoughts on the Role of Thoughts on the Role of Surveys and Qualitative Surveys and Qualitative Methods in Evaluating Methods in Evaluating Health ITHealth IT

National Resource Center for HITNational Resource Center for HIT

2005 AHRQ Annual Conference for 2005 AHRQ Annual Conference for Patient Safety and Health Information Patient Safety and Health Information TechnologyTechnology

Starting with Starting with Surveys…Surveys… Design issuesDesign issues

– comparison groupscomparison groups

Instrument developmentInstrument development

Sampling methodologySampling methodology

– probability of selectionprobability of selection

– size, response rate for significance *and* powersize, response rate for significance *and* power

Survey administrationSurvey administration

Data analysisData analysis

Instrument Instrument Development: Development: Key ConsiderationsKey Considerations Internal validityInternal validity

– are you getting what you think you are are you getting what you think you are getting?getting?

Question orderingQuestion ordering

– easy to difficulteasy to difficult

– logic to flow and transitionslogic to flow and transitions

Keep it shortKeep it short

Pilot test!Pilot test!

Sampling Sampling Methodology:Methodology:Key ConsiderationsKey Considerations Define your population, sampling frameDefine your population, sampling frame

Random selection of participantsRandom selection of participants

– No ‘cherry picking’ (eliminate bias)No ‘cherry picking’ (eliminate bias)

– Everyone has a non-zero chance to be Everyone has a non-zero chance to be selectedselected

Stratification and over samplingStratification and over sampling

Target response rate for a statistically Target response rate for a statistically meaningful resultmeaningful result

– Significance and powerSignificance and power

Survey Administration:Survey Administration:Key ConsiderationsKey Considerations Selecting your mode of administrationSelecting your mode of administration

– Self-administered questionnaire (SAQ)Self-administered questionnaire (SAQ)• Mail, WebMail, Web

– Telephone interviewsTelephone interviews– In-personIn-person

Tips on increasing response rateTips on increasing response rate– Personalize initial outreachPersonalize initial outreach– Use the Dillman approachUse the Dillman approach– Questionnaire formatQuestionnaire format– Optimize follow-up: bang for buckOptimize follow-up: bang for buck

““When you can’t express it in numbers … When you can’t express it in numbers … your information is of an unsatisfactory kind” your information is of an unsatisfactory kind”

-- Lord Kelvin (1824 – 1907)-- Lord Kelvin (1824 – 1907)

So, why use surveys *and* qualitative So, why use surveys *and* qualitative methods?methods?

… … Because evaluating Because evaluating Health IT is a SOCIAL Health IT is a SOCIAL sciencescience

What is qualitative What is qualitative methodology?methodology?

A A non-numericalnon-numerical examination and examination andinterpretation of observations for the purposeinterpretation of observations for the purposeof of discovering underlying meanings anddiscovering underlying meanings andpatternspatterns..

Qualitative methods provide a powerful tool forQualitative methods provide a powerful tool forunderstanding the understanding the contextcontext surrounding surrounding

quantitativequantitativedata and for explaining phenomena data and for explaining phenomena that cannot that cannot

bebemeasured statisticallymeasured statistically due to a lack of validated due to a lack of validatedinstruments, sample size, or passage of time.instruments, sample size, or passage of time.

Using qualitative Using qualitative methods...methods... Qualitative methods offer highly Qualitative methods offer highly

valid results through an in-depth valid results through an in-depth understanding of processes, understanding of processes, attitudes, and effects.attitudes, and effects.

Qualitative methods do not Qualitative methods do not produce statistically significant produce statistically significant results and are generally less results and are generally less reliable than quantitative results.reliable than quantitative results.

Qualitative Methods: Qualitative Methods: ObservationObservation Direct observation by individual(s) Direct observation by individual(s)

of activities, processes, or behaviorsof activities, processes, or behaviors– Is implementation going as planned?Is implementation going as planned?– Validating self-reported survey resultsValidating self-reported survey results– Requires skilled observers ($)Requires skilled observers ($)

Method of observation/data capture Method of observation/data capture variesvaries– E.g., Time and motion studiesE.g., Time and motion studies

Qualitative Methods: Qualitative Methods: Semi-structured Semi-structured InterviewsInterviews Qualitative Interviews Qualitative Interviews

– Use discussion guide or list of themesUse discussion guide or list of themes• Not a static, closed-ended questionnaire Not a static, closed-ended questionnaire

– Key stakeholder perspectivesKey stakeholder perspectives– Detailed information, flexibilityDetailed information, flexibility

Methodology variesMethodology varies– Sampling approach can be scientificSampling approach can be scientific– Content analysis on notes or Content analysis on notes or

transcriptstranscripts

Qualitative Methods: Qualitative Methods: Focus GroupsFocus Groups Moderated discussion with 10-12 Moderated discussion with 10-12

respondentsrespondents– discussion guides w. pre-selected discussion guides w. pre-selected

themesthemes Improved validityImproved validity

– getting all “on the same page”getting all “on the same page” Can explain contradictory Can explain contradictory

statistical resultsstatistical results

Qualitative Methods: Qualitative Methods: Case StudiesCase Studies An in-depth study of a small set of sites.An in-depth study of a small set of sites.

– ObservationObservation– InterviewingInterviewing– Review and/or content analysis of Review and/or content analysis of

background information, documents, background information, documents, communications, etc.communications, etc.

– Informal conversations and encountersInformal conversations and encounters Provides rich descriptions of activities Provides rich descriptions of activities

and the influences of external factors on and the influences of external factors on activities. activities.

Highly valid, but not generalizable.Highly valid, but not generalizable. Case studies are NOT equivalent to Case studies are NOT equivalent to

program site visits.program site visits.

Analyzing Qualitative Analyzing Qualitative DataData Process is variableProcess is variable

– but for any study assumptions should be but for any study assumptions should be clearly describedclearly described

General approachGeneral approach– Coding raw qualitative informationCoding raw qualitative information

• Nudist or ATLAS.tiNudist or ATLAS.ti– Identify themes and trendsIdentify themes and trends– Identify and describe outliersIdentify and describe outliers– Draw conclusions carefully and caveatDraw conclusions carefully and caveat

Analyses should occur both within and Analyses should occur both within and across individual data collections (e.g. across individual data collections (e.g. within a single interview and across all within a single interview and across all interviews). interviews).

Recommended Recommended ReferencesReferences Coffey, A., and Atkinson, P. (1996). Coffey, A., and Atkinson, P. (1996). Making Sense of Making Sense of

Qualitative Data: Complementary Research Strategies. Qualitative Data: Complementary Research Strategies. Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage.

Edmunds, Holly. Edmunds, Holly. The Focus Group Research HandbookThe Focus Group Research Handbook. . McGraw Hill. 2000.McGraw Hill. 2000.

Hyman, Herbert H., with William J. Cobb, Jacob J. Feldman, Hyman, Herbert H., with William J. Cobb, Jacob J. Feldman, Clyde W. Hart, and Charles Herbert Stember. Clyde W. Hart, and Charles Herbert Stember. Interviewing in Interviewing in Social ResearchSocial Research. University of Chicago Press, 1954. . University of Chicago Press, 1954.

Krueger, A. Richard. Krueger, A. Richard. Focus Groups: A Practical guide for Focus Groups: A Practical guide for Applied Research. Applied Research. Sage Publishing. 1994.Sage Publishing. 1994.

Miles, M.B, and Huberman, A.M. (1994). Miles, M.B, and Huberman, A.M. (1994). Qualitative Data Qualitative Data Analysis,Analysis, 2nd Ed., p. 10-12. Newbury Park, CA: Sage. 2nd Ed., p. 10-12. Newbury Park, CA: Sage.

NSF “User Friendly Handbook for Mixed Method NSF “User Friendly Handbook for Mixed Method Evaluations.” 1997.Evaluations.” 1997.

Payne, Stanley L. Payne, Stanley L. The Art of Asking QuestionsThe Art of Asking Questions. Princeton . Princeton University Press, 1951.University Press, 1951.

Turner, Charles F., and Elizabeth M. Martin, eds. Turner, Charles F., and Elizabeth M. Martin, eds. Surveying Surveying Subjective PhenomenaSubjective Phenomena. Russell Sage, 1984. . Russell Sage, 1984.