Research Article Mixed-Methods Research in a Complex ...

Hindawi Publishing CorporationEvidence-Based Complementary and Alternative MedicineVolume 2013, Article ID 701280, 10 pageshttp://dx.doi.org/10.1155/2013/701280

Research ArticleMixed-Methods Research in a Complex Multisite VAHealth Services Study: Variations in the Implementation andCharacteristics of Chiropractic Services in VA

Raheleh Khorsan,1,2 Angela B. Cohen,1 Anthony J. Lisi,3,4 Monica M. Smith,1

Deborah Delevan,1 Courtney Armstrong,5 and Brian S. Mittman1

1 VA Center for Implementation Practice and Research Support, VA Greater Los Angeles Healthcare System,16111 Plummer Street, Sepulveda, Los Angeles, CA 91343, USA

2Military Medical Research Program, Samueli Institute, Corona del Mar, CA, USA3 Chiropractic Program, Patient Care Services, Veterans Health Administration, Washington, DC, USA4Chiropractic Service, VA Connecticut Healthcare System, West Haven, CT, USA5 RAND Corporation, Boston, MA, USA

Correspondence should be addressed to Brian S. Mittman; [email protected]

Received 22 September 2013; Accepted 30 September 2013

Academic Editor: Cheryl Hawk

Copyright © 2013 Raheleh Khorsan et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

Maximizing the quality and benefits of newly established chiropractic services represents an important policy and practice goalfor the US Department of Veterans Affairs’ healthcare system. Understanding the implementation process and characteristicsof new chiropractic clinics and the determinants and consequences of these processes and characteristics is a critical first stepin guiding quality improvement. This paper reports insights and lessons learned regarding the successful application of mixedmethods research approaches—insights derived from a study of chiropractic clinic implementation and characteristics, Variationsin the Implementation and Characteristics of Chiropractic Services in VA (VICCS). Challenges and solutions are presented inareas ranging from selection and recruitment of sites and participants to the collection and analysis of varied data sources. TheVICCS study illustrates the importance of several factors in successful mixed-methods approaches, including (1) the importanceof a formal, fully developed logic model to identify and link data sources, variables, and outcomes of interest to the study’s analysisplan and its data collection instruments and codebook and (2) ensuring that data collection methods, including mixed-methods,match study aims. Overall, successful application of a mixed-methods approach requires careful planning, frequent trade-offs, andcomplex coding and analysis.

1. Introduction

There is growing consumer interest in complementary andalternative medicine (CAM) in the USA and internationally[1–3]. Healthcare systems have responded to this demand byoffering a range of CAM services in outpatient and inpatientsettings [4, 5]. Patients enrolled in the USDepartment of Vet-erans Affairs (VA) healthcare delivery system often use CAMservices outside of VA but have a strong interest in receivingthese services within the VA system [6–11]. In response, VAbegan providing selected in-house CAM services in about

2001 [12]. VA’s most substantial undertaking in delivering anyCAM-related service has been its introduction of chiropracticservices.

Chiropractic care is often described as sitting at thecrossroads of CAM and mainstream medicine [13], andits introduction into the VA healthcare system exemplifiesthat duality. In 1999, Congress directed VA to establish apolicy regarding chiropractic services for musculoskeletalconditions (Public Law 106–117) [14]. Although specificaction was not mandated, in response to this legislation, VAbegan providing limited access to chiropractic care by paying

2 Evidence-Based Complementary and Alternative Medicine

for services delivered outside the VA healthcare system. In2001, Public Law 107–135 made chiropractic services part ofthe standard medical benefits available to all Veterans andrequiredVA to deliver these services on-site byVA chiroprac-tors at a minimum of one VA medical facility in each of VA’s21 geographic regions (Veterans Integrated Service Networksor VISNs) [14, 15]. This required the incorporation of a newprovider type, doctors of chiropractic (DCs), intoVA’s clinicaland administrative policies and procedures.

The establishment of chiropractic clinics within VA waschallenged by the rarity of existingmodels in other healthcaresystems and by the widely varying perception of chiroprac-tic services by medical physicians and other stakeholders[16]. VA convened a Federal Advisory Committee to makerecommendations on the implementation of chiropracticservices and in July 2004 issued Directive 2004–035 whichestablished the overall policy for VA chiropractic services.While chiropractic care is now part of VA’s standard medicalservices, in practice and perception chiropractic care stillretains many of the limiting features of a CAM service withina traditionalmedical setting.The introduction of chiropracticservices in VA faced not only the typical challenges ofintroducing any new clinical service or program into a largehealthcare system but also the unique obstacle of integratinga nontraditional healthcare service into conventionalmedicalsettings [14, 17–19].

By the end of 2005, VA had successfully complied withthe requirement of establishing a minimum of one chiro-practic clinic within each VISN. This initiative was looselycoordinated by VA Central Office (VACO), leaving muchof the details to individual facilities. Over the followingyears, the use of chiropractic services at these initial facilitiesdramatically increased.This growth, along with interest fromVeterans and providers at other VA facilities, stimulated theexpansion of chiropractic clinics into other VA facilities.From fiscal year 2005 to fiscal year 2011, without furtherCongressional mandate, the number of VA chiropracticclinics increased from 24 to 43, and the number of Veteransreceiving care at these clinics increased from just under 4,000to over 81,000. Also during this time, VACO establishedcentral leadership for the chiropractic program in the Officeof Rehabilitation Services which began to monitor and assessthe ongoing uptake and expansion of services [20]. Because ofexpected challenges facing the introduction of a new providertype (issues of privileging, competencies, and facility inte-gration), unique features related to chiropractic care (varyingperception and prior experience of other clinicians), and therelatively decentralized manner in which initial clinics wereestablished, the chiropractic program office sought to gaindeeper knowledge of the program’s continuing developmentand features.

Early studies of chiropractic care in VA described patientcharacteristics and outcomes in individual VA chiropracticclinics [17, 21–23], characteristics of patients and clinics atthe national level [14], and elements of academic trainingprograms [24]. However, a more in-depth understanding ofVA’s implementation of chiropractic services was needed toinform future policy and practice decisions and ultimatelyto ensure the highest quality of care delivered to Veterans.

Program implementation initiatives within VA, as well assimilar efforts outside VA, require careful planning andexecution to achieve success.The chiropractic program officelacked the resources and expertise to conduct a large-scaleprogram evaluation but was positioned to build partnershipswith the VA research community.These circumstances led toa research-policy-practice partnership established to designand obtain funding for a program of research, beginningwith a pilot study entitled “Variations in the Implementationand Characteristics of Chiropractic Services in VA (VICCS).”The VICCS study was guided by prior research examiningthe introduction and integration of nurse practitioners inVA [25] and related research examining the introductionand role of nurse practitioners and physician assistants inother healthcare delivery settings [26], as well as additionalstudies documenting the implementation and integration ofnew clinical services in a range of settings.

The VICCS research-policy-practice partnership soughtto explore the chiropractic services program in parallel withother VA integrated care initiatives. These include programsfor Veterans returning from operations in Afghanistan andIraq (i.e., VA’s Post-Deployment Integrated Care Initiative) aswell as a national palliative care program (ComprehensiveEnd-of-Life Care Initiative) and the ongoing primary caremedical home initiative (Patient Aligned Care Teams) [17].

This paper describes the design and methods of theVICCS study and insights gained from the application ofa mixed-methods approach to address study questions. Theexperiences and insights from the study offer guidance forfuture research-practice partnerships and methods suitablefor assessing the introduction of other new clinical services—traditional or CAM—in VA and other large healthcaresystems. The paper describes the mixed-methods designemployed, as well as specific challenges and issues relatedto data collection instruments and data collection logistics,analyses of diverse data types for distinct study aims, andother issues.

2. VICCS Study Development and Aims

The primary objective of the VICCS study was to identifyvariations in the implementation processes and organiza-tional arrangements of VA chiropractic services and examinethe causes and consequences of those variations. A mixed-methods approach was used to pursue the study’s threespecific aims.

(1) Document and characterize (a) the implementationof chiropractic services into individual VA healthcaredelivery facilities and (b) the characteristics and orga-nizational arrangements through which these servi-ces are delivered, including their integration withexisting clinical services.

(2) Identify (a) key factors leading to different imple-mentation patterns and clinic characteristics acrossdifferent VA facilities and (b) selected impacts andconsequences of different implementation patternsand clinic characteristics.

Evidence-Based Complementary and Alternative Medicine 3

(3) Develop and refine research methods and tools for(a) a larger, more definitive study of chiropractic careprograms in VA and for (b) studies examining theimplementation of other new services and disciplines(including CAM services) in large healthcare deliverysystems.

To address VICCS study aims (1) and (2), the studyemployed a comparative case study approach relying on (a)interviews to gather data fromkey stakeholders, (b) collectionand content analysis of policy and procedure documentsand other archival/documentary material to supplementinterview-provided data, and (c) administrative data on useof VA chiropractic services. The study team’s experienceand identification of several methodological and logisticalchallenges encountered during the study contribute toVICCSstudy aim (3), in which the study team used many of the“lessons learned” to inform and guide planning for futurestudies, whether of chiropractic services in VA or other newservices and disciplines in any large US healthcare deliverysystem.

The mixed-methods approach included qualitative andquantitative analysis methods for inductive (hypothesis gen-eration and exploration) and deductive (hypothesis testing)analyses. VICCS study data collection occurred in 2010 and2011, followed by data analysis and reporting in 2012.

3. Mixed Methodology and HealthServices Research

The VICCS study’s core conceptual framework relied onDonabedian’s structure, process, and outcomes model(Table 1) [27]. Donabedian [28] suggests that the qualityof health care can be conceptualized and evaluated alongthree main dimensions of care delivery: structures ofcare, processes of care, and care outcomes. Structurerefers to the setting in which care is delivered, includingfacilities and equipment, qualification of care providers,administrative structure, and operations of programs.Process encompasses how care is provided and is measuredin terms of appropriateness, acceptability, completeness, orcompetency [29–33]. These measurements are typically lessdefinite than those obtained through assessing outcomes.Lastly, outcomes refer to the end points of care, such asimprovement in function, recovery, or survival. Outcomesare usually concrete and precisely measured [34].

The VICCS data collection framework was designed toinclude several key categories of variables. These includedfeatures of each site and the background and motivation forthe establishment of each chiropractic clinic. For example,information was collected on the initial impetus for eachclinic (i.e., did VISN leadership require establishment of aclinic at a given facility or did facility leadership voluntarilyestablish a clinic?) and other key features of the healthcaresetting prior to chiropractic clinic implementation. Thedata collection framework also distinguished several distinctphases in the clinic planning, implementation, and main-tenance process and several distinct categories of variables

Table 1: Key domains and variables.

Domains VariablesEnvironment andcontext

Societal, VA, VISN (region), and facilitycharacteristics

Clinic planning andimplementation

Planning process, milestones,communication and coordination,resources, and team characteristics

Clinic structure Clinic characteristics and staffcharacteristics and competencies

Care processes Clinical services provided and quality ofcare

Impacts andoutcomes

Quality of implementation, satisfactionwith processes, services, and care (patient,leadership, and staff)

describing the clinic context, chiropractic clinic itself, and keyoutcomes and measures of performance at each clinic.

Table 1 lists the domains and illustrative variables selectedfor the VICCS study. The conceptual model was furtherrefined through an iterative process as the study was under-way, as described in the following.

(i) Environment/context includes local factors, such aslocal stakeholder attitudes toward innovation in gen-eral and chiropractic in particular, as well as VAregional and national factors, and non-VA externalfactors such as Veteran Service Organization influ-ences.

(ii) Planning/implementation includes features of a facil-ity’s planning process and the participation of variousstakeholders with differing levels of subject matterexpertise.

(iii) Clinic structure includes characteristics of the indi-vidual DC clinician(s), organizational alignment, andphysical features of the clinic, the formal relationship,and extent of integration or collaboration with otherfacility programs and stakeholders.

(iv) Care processes include characteristics of healthcareservices provided, features of case management orcare pathways, and quality of services.

(v) Impacts/outcomes include the status of clinic accessand use, patient-based outcomes, system perceptionof value, and external stakeholder opinions.

Semistructured interview guides were developed foreach type of stakeholder. Interview subjects included VAfacility leaders, chiropractors, chiropractor supervisors, chi-ropractic clinic support staff, other clinicians from vari-ous departments, administrative planners, VA patients withmusculoskeletal pain complaints (from both chiropracticand nonchiropractic clinics), and external stakeholders (aca-demic affiliates, Veteran Service Organizations, and formerFederal Advisory Committee participants). Interview guideswere adapted from existing instruments employed in similarstudies, augmented by new questions and content specificto this study. Questions to measure stakeholder satisfactionand views were informed by existing instruments such as


theChiropractic SatisfactionQuestionnaire [35], theMeasureof Clinicians’ Orientation toward Integrative Medicine (IM-30) [36], and input from subject matter experts.

4. Mixed-Methods Approach

Mixed-methods research is increasingly used in the social,behavioral, and health sciences. The VICCS mixed-methodsapproach employed an explicit conceptual framework iden-tifying key variables and data sources relevant to the study’sprimary aims. This approach improves the quality and com-pleteness of data in health care research to study issues asvaried as health disparities, cultural differences, behavioralfactors contributing to disability and health, processes andfactors involved in implementation of health research find-ings, and much more. The increasing use of mixed methodsreflects growing recognition of the value of qualitative andother social science research methods, collaborative inter-disciplinary research teams (also known as team science)[37, 38], and the use of multilevel approaches to investigatecomplicated health issues [39]. Such approaches are oftencombined with clinical trials, surveys of attitudes and beliefs,and epidemiological measures to better understand healthand illness [40, 41].

Creswell et al. [40] document current trends in theapplication ofmixed-methods approaches in a broad range ofhealth-related research, such as in cardiology [42], pharmacy[43], family medicine [44], pediatric oncology nursing [45],mental health services [46], disabilities [47], and publichealth [48].The settings vary from the clinic [49] to the socialcontext of daily activities and relationships [50]. Trends inmixed-methods research are also documented in a study ofNIH-funded investigations that incorporated “mixed meth-ods” or “multimethods” in NIH-funded research abstracts[51]. Qualitative methods are used in mixed-methods studiesto address broad, open-ended, and interconnected questionsthat are often quite different from conventional clinicalhypotheses [52]. Many social scientists view inductive, inter-pretive, and related applications of qualitative methods as animportant advantage over quantitative methods in develop-ing insights into values, beliefs, attitudes, and interpretationsof current or past events and other phenomena, but thesemethods can be used to supplement quantitative methods toexamine many other phenomena as well.

5. Results

5.1. Sampling, Site and Subject Selection, and Recruitment.The VICCS study data included extensive notes from onehundred eighteen interviews. Most interviews were con-ducted in person (84%, 𝑛 = 99) during two-day site visits atseven facilities. The remaining interviews were completed bytelephone (16%, 𝑛 = 19) withVA facility staff unavailable dur-ing the site visit and with external stakeholders. Documentswere collected during site visits and received via fax and e-mail prior to and following site visits.

Sampling procedures and criteria were developed forseven study sites. As the VANational Director of chiropractic

services, the study’s co-PI (AJL) was invaluable given hisexpertise and experience in administrative and subspecialtymatters. However, to avoid potential coercion or bias, AJLdid not participate in the site selection decision-makingprocess, recruitment, or interviews with any chiropractic staff(including chiropractors, their supervisors, or support staff).

Sites were selected to ensure diversity on key dimensions,including:

(i) facility geographic location (regions of USA, urban/suburban/rural);

(ii) facility type (medical center versus outpatient clinic,complexity);

(iii) administrative alignment (facility service line over-seeing the clinic);

(iv) chiropractor characteristics (appointment type, full-time versus part-time, clinical experience, prior prac-tice setting, credentials);

(v) clinic establishment (how long the clinic had been inexistence);

(vi) involvement with academic affiliate(s).

At the time of site selection, forty-one (41) VA facilitiesoffered on-site chiropractic services. Two (2) sites involvedthe study’s co-PI as a staff member and were thus excluded.Two (2) other sites were deemed ineligible because theyfunctioned as independent outpatient clinics and were notdirectly linked administratively to VA. Another site wasineligible because there was no chiropractor on staff at thetime of site selection. Therefore, a total of thirty-six (36) siteswere eligible for recruitment.

We set a sample size of seven sites, including one pilot site.A total of 12 sites were invited to participate until we reachedour sample of seven, for a total site response rate of 58%. Ofthe five (5) sites that declined, 3 declined due to workload orother time conflicts, and 2 declined because facility leadershipdetermined that local IRB review would be required (basedon their perception that their sites would be actively engagedin the research, thereby triggering the need for local IRBreview), thus disqualifying them from meeting the study’stimeline. Facilities invited but declining to participate in thisstudy remain confidential to those outside of the study teamas well as to the study’s co-PI, as per the study’s recruitmentprotocol and a confidentiality feature designed to minimizecoercion.

At each site, we targeted a variety of stakeholders forinterviews:

(i) facility leadership (facility directors, chiefs of staff);(ii) key department heads (e.g., primary care, physical

medicine and rehabilitation, orthopedics, neurol-ogy, pain clinic, rheumatology, radiology, and spineclinic);

(iii) chiropractors;(iv) clinicians (primary care and specialty providers who

both did or did not refer patients to chiropracticservice) (2 per discipline, for a total of 6–8 per site);


(v) chiropractic clinical and administrative support staff;(vi) VA back or neck pain patients (2-3 from each chi-

ropractic clinic and 2-3 from a nonchiropractic backpain or related clinic, each seen at VA three or moretimes for the same neck or back issue);

(vii) external stakeholders (local, such as academic affili-ates, and national, such as national VSO representa-tives and federal advisory associates).

Prior to data collection, all study teammembers attendedinterview training and observed at least two pilot interviews.Two or three members of the study team attended each sitevisit, with one or twomembers conducting each interview.Allsubjects orally consented prior to participation. For respon-dents who agreed to be audio-recorded, these interviewswere transcribed. Patients were not audio-recorded as perour protocol, but copious notes were taken and debriefingsessions occurred immediately afterwards to ensure thatas much verbatim information was retained as possible.Excluding the 18 patients interviewed, 96 of the remaining100 interview subjects agreed to be audio-recorded. Toensure confidentiality of sites and subjects, all identifierswere removed and replacedwith study-generated consecutiveidentification codes prior to data coding and analysis.

5.2. Qualitative and Quantitative Data. Between December2010 and November 2011, 118 semistructured interviews wereconducted at seven sites, including one pilot site. Interviewsubjects included sixty-two non-DC clinicians (53%), eigh-teen patients (15%), eleven leaders (9%), seven chiropractors(6%), six chiropractic support staff (5%), five staff involvedin planning chiropractic clinics (4%), four chiropracticsupervisors (3%), three former federal chiropractic advisorycommittee members (3%), and two academic affiliates (2%).Participation response rates for subjects and sites were 43%and 58%, respectively.

Documents reviewed and analyzed included significantchiropractic care-related policy documents obtained fromthe study sites, including VA regional (VISN) policies,local facility policies, local service agreements, chiropractorclinician privileges, and other public documents such ascongressional bills/resolutions related to VA chiropracticservices.

Content analysis of interviews and documents assesseda priori hypotheses derived from prior literature, as well asnew themes emerging from transcript review. The codebookfor the interviews and documents was developed and refinedthroughout the coding process using top-down (deduc-tive/a priori hypothesis testing) and bottom-up (induc-tive/emerging hypothesis generating) methods.

Data collected from both interviews and documentswere coded using NVivo (QSR International) and Excel(Microsoft) software, respectively. We observed high inter-rater agreement (𝑘 = 0.8) among coding team members.Data were coded in a two-phased process utilizing high levelcodes first (double coded) for general themes and variabledomains and thenmore specific detailed codes for subthemesand individual variables.

Additionally, the study obtained quantitative administra-tive data on clinic use and utilization characteristics such aspatient visit counts, patient demographics, diagnoses seen,and services delivered. These data were obtained from theVA Corporate Data Warehouse via VA Informatics andComputing Infrastructure (VINCI).

The VICCS study was approved by four separate institu-tional review boards (IRBs): VA Greater Los Angeles Health-care System, VA Connecticut Healthcare System, WesternIRB, and US Army Medical Research and Materiel Com-mand.

6. Discussion

6.1. VICCS Study Methodological Issues and “LessonsLearned”. Most of the methodological challenges in theVICCS study fall into 4 main categories: (1) subject selec-tion and recruitment, (2) site selection, (3) data collection in-struments and logistics, and (4) analysis of and interpretationof diverse types of data related to the study’s multiple researchquestions and goals.

6.1.1. Selection and Recruitment of Participants. The site selec-tion process was designed to ensure wide variability of chi-ropractic clinics and their internal structures and processes.However, because participation of sites and stakeholders ateach site was voluntary, some bias may have been introducedthat will limit any generalizations outside of VA.

Recruitment of busy healthcare professionals (both clin-icians and administrators) is a well-recognized challenge toconducting research in healthcare settings [53]. Because someclinicians and staff are unionized, sites also had to have unionnotification and, in some cases, approval (local and national).

VA and related federal regulations on patient sur-vey/interview research allow small numbers of patients tobe sampled in all IRB-approved studies, but, for studiesattempting to recruit larger numbers of patients (>9), addi-tional approval of the Office of Management and Budget isrequired. Because OMB approval is lengthy, we sampled asmaller number of patients directly and attempted to gatheradditional patient perspectives indirectly through providerinterviews.

6.1.2. Site Selection. In partnership research involving pro-gram leaders who serve as research team members, researchsubjects (i.e., staff from participating sites) may be concernedthat they are being evaluated on their performance andactions rather than studied for the purpose of scientificknowledge development. Therefore, issues of possible coer-cion and sensitivities may affect the validity of data collected.

Staff turnover and limited memories also threaten datavalidity (temporal bias) when studying programdevelopmentand evolution in a retrospective manner. Archival records arelimited, and thus we relied heavily on VA staff (where stillavailable) and their memories.

To improve the validity and integrity of data collected,this study performedmost interviews in person during a two-day site visit (with two to three research team members) at


each of the seven sites; 84% of all interviews were conductedin person (𝑛 = 99). To minimize data security and privacyconcerns with IRBs, patients were recruited for interviewsonsite and only first names were used.

Two patient respondent groups participated in this studybased on their personal experience with three or more visitsrelated to back or neck pain issues: those who received chiro-practic services and those who received other nonchiroprac-tic services (e.g., primary care, neurology, and orthopedics).Patients in clinic waiting areas were recruited systematically.No one was missed or avoided. And, because of the study’sIRB-approved minimal risk status, this study was granteda waiver of documentation of informed consent; thus allsubjects orally consented to participate in this study.

Interviews at each site were scheduled to fill target quotasfor each of the various roles (respondent groups) needed.After each site’s chiropractor and facility leaders approvedparticipation in the study, lists of providers by department (orservice line) were compiled. Recruitment involved sendingindividual invitations to VA employees to fill the two-daysite visit schedule. Initial invitations were sent by e-maildescribing the study and announcing the planned site visitdates, and up to five follow-up contacts were attemptedper person. Relatively low subject response rates (45.6%)may have resulted partly from the limited two-day interviewtimeframe scheduled at each site. Of those who did notparticipate, 69.5% (𝑛 = 98)were subjectswhodid not respondto these initial or follow-up e-mails requests for participation.

6.1.3. Data Collection Instruments. Data collection instru-ments and protocols in qualitative research are often infor-mal, flexible, and subject to large variations in application.While flexibility represents a strength in traditional qualita-tive research, it can result in inconsistent and unfocused datacollection and variable data quality when qualitativemethodsare applied in deductive research. For example, interviewguides specifying general topics of interest, using broad,open-ended questions, can be very effective in assessing inter-view subjects’ assessment of important concepts and issuesand their beliefs and values but ineffective in ensuring thatcomplete and comparable measures of identified variablesare collected consistently across a range of subjects (e.g.,assessing organizational participants’ views or their ratingsof concepts or variables deemed important by the researchteam). In part, the distinction here is between data collectionapproaches designed to develop new insights and frameworksfor understanding and describing the phenomena of interest,versus applying a priori frameworks to collect predefined dataand test aspects of these frameworks. Similar problems resultfrom the use of observation guides or protocols lacking ade-quate specificity and a firm foundation in a priori hypothesesand clearly identified variables: such protocols often produceinconsistent data by (1) encouraging the observer to recordevents as they unfold and to record a wide range of attributesof the situation under study (whether or not they are deemedrelevant to the hypotheses of interest), (2) limiting thelikelihood that the observer will note the significance ofevents that do not occur, and (3) limiting the likelihood that

the observer will collect complete, consistent data requiredfor direct comparisons across observation samples.

Considerations of validity, intrusiveness or subject reac-tivity (Hawthorne effects), and triangulation (to minimizebias) are also too often neglected in deductive applicationsof qualitative methods. Distinctions between subjective andobjective data and between formal and informal organiza-tional structures and processes are also frequently neglected,threatening the validity of study conclusions.

Avoiding these problems requires careful design of datacollection plans, based on study goals and hypotheses, involv-ing use of systematic tables or other methods for specifyingkey variables and suitable, multiple measures. Depending onthe importance of each variable and the validity of availablemeasures, two or more data sources are typically needed inqualitative research. Data planning tables listing concepts orvariables, definitions, and data sources are effective in ensur-ing appropriate rigor; data collection instruments (includingdocument coding forms, survey questions, and other dataspecifications) can be developed directly from these tables.

Rigor and validity are also enhanced through develop-ment and use of data collection instrument specificationsand training protocols, including variable and measure def-initions and instructions in instrument use. When used inresearch examining health care delivery organizations, suchprotocols should include plans and instructions for approach-ing sites, making contacts, arranging interviews/visits, iden-tifying and obtaining documents, following up (to obtaindocuments and other postvisit/call information), managinginformed consent and confidentiality, and so forth. Adequatepilot testing helps ensure the appropriateness of data sourcesand measures although data collection protocols must beflexible and allow for changes in data collection plans andstrategies, when pilot testing fails to reveal valuable new datasources or validity problems that eventually emerge duringthe main study period.

Finally, study validity is further enhanced through devel-opment of data analysis protocols and plans together withthe actual instruments, rather than after the completion ofdata collection. Data planning tables created to guide datacollection activities can be used to develop data reportingtemplates and specifications for translating raw data intovariables and preparing for analyses; data fromorganizationalstudies are often reported in a standardized “organizationalprofile” or other comparative formats.These profiles store rawdata and summary variables from all data sources, which arethen converted into tables for analysis. Additional challengesarise in studies pursuing inductive and deductive study aimssimultaneously. For deductive studies with a rigid a prioriframework, the number of variables to be measured shouldbe relatively small and easilymanaged. Inductive, exploratorywork involves open-ended questions and unlimited data andis thus more challenging to plan and conduct.

6.1.4. Analyzing Diverse Types of Data. Thefield of health ser-vices research has benefited from several insightful, compre-hensive discussions of qualitative research methods and theirappropriate use [54]. Proponents have convincingly argued


that qualitative methods contribute to findings and insightsthat cannot be derived from “conventional” or “quantitative”researchmethods and that research in the clinical, social, andpolicy sciences requires careful application of both types ofapproaches to properly study their phenomena of interest.

For qualitative research, patterns of observed individualand organizational practices, behaviors, and outcomes arehighly variable (across time and site) and are subject to widevariations and interviewer/observer bias and interpretation.This heterogeneity challenges those designing the researchto find and describe consistent patterns and topics withinthe findings. The patterns and topics identified are heavilyinfluenced by idiosyncratic factors, such as an individualleader’s personal views or situation or unrelated pressuresor events within a site. Also, each site’s situation (planning,implementation, clinic structure, etc.) is influenced by a verylarge number of factors, whose combined and interactingeffects lead to highly variable outcomes (as described inchaos or complexity theory, e.g.). Data collection relyingon interviews with individuals entails potential bias, limitedvalidity, and inaccuracies due to challenges in recall anddiffering perspectives and views of events and differentialaccess to information.Therefore standard challenges in qual-itative/case study research also apply here.

Another challenge arises from the limitation that datafor many key variables are not always reliably available orare difficult to access, in addition to having questionablevalidity in some instances. This study also examined a longchain of causal links andmultiple determinants and outcomes(independent and dependent variables). Overall, the mixed-methods approach employed here was challenging becausedual inductive and deductive research is inherently demand-ing, for example, deciding how to allocate limited interviewtime to measure variables identified a priori (for deductive)versus open-ended interviewing to maximize the likelihoodof learning something new and interesting (for inductive).Lastly, the lack of available standardized, validated measures,concepts, definitions, and so forth was a significant challengeto this study.These challenges are especially common to pilotstudies.

However, there are steps tominimize these biases, includ-ing adequate training of data collection staff; comprehen-sive plans for data collection, validation, and storage; andfrequent reviews of data quality and interpretation. Whiledata validity and completeness can be enhanced throughthe recording of interviews, other methods should also bepairedwith the traditional recording of interviews such as useof paired interviewers, postinterview debriefing, and othermethods. Quality assurance methods should be consideredand operationalized for each instrument and data sources.Problems such as incomplete, missing, unusable data shouldbe identified and resolved during the data collection phaserather than after its completion.

Finally, analysis plans are relatively quantitative in mostconventional health care studies that often pursue narrow,explicit aims and test explicit, confirmatory hypotheses.However, mixed-methods studies that address both induc-tive and deductive study questions and employ methods

features, such as randomized sampling, that are more typ-ically associated with experiments—combined with open-ended interviews more typically associated with qualitativeresearch—offer what is termed by Morse “alternative formsof evidence” [55, p. 86]. Therefore new opportunities forqualitative inquiries have the ability to emerge [56]. Thesepoints are illustrated well by this passage from Ronald J.Chenail:

Take a pragmatic posture to creating studiesthat marry the most fitting design and method-ology choices with the focus of your researchcuriosity. . .remain true to your interests and thenexplore a variety of research approaches which canhelp in the designing and conducting studies tomeet your needs. The bottom line is to be prag-matic in creating the design, but remain curious soevery reasonable methodological option is consid-ered. However, like taking too many medicationscan lead to adverse effects to your body, using toomany methodologies might produce negative sideeffects which could be unhealthy for your study. Tohelp remedy this potential risk, please rememberthis simple research commandment: Thou shallnot select an additional methodology for a study,until thou is sure the first methodology selectedcannot manage all of the design issues [56].

7. Conclusion

To our knowledge, VA’s introduction of chiropractic servicesrepresents the most extensive introduction of any nontradi-tional medical service into the largest integrated US health-care system. This is likely to present future research areas ofinterest to multiple stakeholders. VA policy makers may seekdata to inform efforts to best assess and improve the deliveryof chiropractic services tomeet Veterans’ needs. Stakeholdersin the chiropractic profession may look to VA’s experienceas an indicator of future opportunities and integration intoother healthcare systems. Other CAM disciplines seekinginclusion into VA and other systems may be interested in thepolicy and practice implications ofVA’s chiropractic program.At a broader level, beyond the unique chiropractic and/orCAM implications, our study may have implications forresearchers to assess the introduction of any new healthcareservice into VA or other large healthcare systems.

For these types of inquiries, research-practice-policypartnerships facilitate research that can be more useful todecision makers (relative to traditional academic research).Decision maker involvement increases the likelihood that(1) useful questions are answered by developing methodsand data analyses relevant to service delivery and (2) theinterpretation and reporting of results will inform futurepolicy.

Analysis of qualitative observational data in studies com-bining deductive and inductive aims should be guided byprespecified, model-based hypotheses and detailed analysisplans developed at the outset of the study. Unfortunately,while quantitative analysis methods are well established and


accepted, methods for analysis of qualitative data are subjectto variability and lack of consensus. Analyses of qualitativedata are too often informal, ad hoc, and emergent, withlow reliability and validity. These threats can be counteredthrough the use of formal table approaches, in which keyvariables relevant to each hypothesis are listed in tables andmanipulated in a blinded fashion, using qualitative patternidentification and nonparametric quantitative techniques.The analysis tables summarizing and synthesizing informa-tion from diverse sources in a standardized format may alsoserve as reporting tools, in papers and reports. Combiningthe use of qualitative methods for hypothesis testing andinterpretive, inductive applications in this manner representsa powerful application of thesemethods, using their strengthsto enhancemanagement studies and other empirical researchin important ways.

In conclusion, qualitative case study research allowsfor the collection of rich data for deep understanding ofthe phenomenon of interest. Yet data collection relying oninterviews with individuals entails potential bias, limitedvalidity, and inaccuracies due to challenges in recall anddiffering perspectives and views of events and differentialaccess to information. Also, data for many key variables maybe unavailable or difficult to access, in addition to havingquestionable validity in some instances. Researchers who arenot trained in qualitative methods (and who are accustomedto conducting empirical research using quantitative methodsalone) are less likely to be interested in applying qualitativemethods in inductive or interpretive research but can—andshould—be interested in applying qualitative methods toenhance the data available for more conventional deductiveforms of empirical research. The use of mixed methodologycan enrich health services research testing a priori studyhypotheses and narrowly defined research questions and canalso help suggest detailed causal explanations and generateimportant new exploratory questions and findings as well.

Conflict of Interests

The authors declare that there is no conflict of interests withany financial organization regarding thematerial discussed inthe paper.

Acknowledgments

The authors would like to acknowledge Joan A. Walter, J.D.,P.A., and Rick Welton, M.D., for their support and contri-butions to the project. They would also like to acknowledgethe members of the project advisory board: Lucille Beck,Ph.D.; Charles Burgar, M.D.; A. Lucile Burgo-Black, M.D.;Ian Coulter, Ph.D.; Paul Shekelle, M.D., Ph.D.; Joan Walter,J.D., P.A. Oral presentations by (1) Khorsan R, Cohen A,Smith M, Lisi A, Mittman B, Coulter I, and Walter J. Inte-grative Chiropractic Services in VA: A Pilot Study (VICCS).American Public Health Association (APHA) 140th AnnualMeeting and Exposition, Session: 3418.0 Policy, regulation,comparative and cost effectiveness research: San Francisco,CA. Monday, October 29, 2012: 5:15 p.m. (Abstract no.

269497); (2) Cohen A, Smith M, Khorsan R, Lisi A, MittmanB, JenkinsD, andArmstrongC.Methodological and logisticalchallenges to conducting multi-site partnered research in theVA healthcare system. American Public Health Association(APHA) 140th Annual Meeting and Exposition, Session:3323.0 Chiropractic Research: Current status and updates:San Francisco, CA. Monday, October 29, 2012: 3:00 PM.(Abstract no. 268597); (3) Smith M, Cohen A, Khorsan R,Jenkins D, Armstrong C, Lisi A, and Mittman B. TrackingVA’s chiropractic program: Implementation timelines andvariations across selected facilities. American Public HealthAssociation (APHA) 140th Annual Meeting and Exposition,Session: 4133.1 Veteran’s Health Care and Health Risks:San Francisco, CA. Tuesday, October 30, 2012: 11:30 PM.(Abstract no. 265497); (4) Lisi A, Mittman B, Khorsan R,Smith S, Cohen A, Armstrong C, MacGregor C, CarucciM, and Jenkins DM. Variations in the Implementation andCharacteristics of Chiropractic Services in VA: A Pilot Study.Association of Chiropractic Colleges (ACC) and ResearchAgenda Conference (RAC): Las Vegas, NV. March 20-21,2012; (5) Lisi A, Mittman B, Khorsan R, Smith S, Cohen A,Armstrong C, MacGregor C, Carucci M, and Jenkins DM.Studying the introduction, integration, and effectiveness ofchiropractic services within the VA healthcare system: Anovel clinical/research collaboration. Association of Chiro-practic Colleges (ACC) and Research Agenda Conference(RAC): Las Vegas, NV. March 19–21, 2011; and (6) Lisi A,Mittman B, Khorsan R, Smith S, Cohen A, Armstrong C,MacGregor C, Carucci M, and Jenkins DM. Variations in theimplementation and characteristics of chiropractic services inVAAssociation of Chiropractic Colleges (ACC) andResearchAgenda Conference (RAC): Las Vegas, NV. March 19–21,2011. This material is based upon work supported in partby the Department of Veterans Affairs, Veterans HealthAdministration, Office of Research andDevelopment, HealthServices Research Development Service.The views expressedin this paper are those of the authors and do not necessarilyreflect the position or policy of the Department of VeteransAffairs or the United States government. In addition, thiswork was supported, in part, by a grant to Brian S. Mittmanby the Samueli Institute. This grant is supported by theUS Army Medical Research and Materiel Command underAward no. W81XWH-06-1-0279.The views, opinions, and/orfindings contained in this report are those of the author(s)and should not be construed as an official position, policy,or decision of Department of the Army unless so designatedby other documentation. Also, the views, opinions, and/orfindings contained in this report are those of the author(s)and should not be construed as the opinion or policy of theSamueli Institute. In the conduct of research where humansare the subjects, the investigator(s) adhered to the policiesregarding the protection of human subjects as prescribed byCode of Federal Regulations (CFR) Title 45, Volume 1, Part46; Title 32, Chapter 1, Part 219; and Title 21, Chapter 1,Part 50 (Protection of Human Subjects). Monica M. Smithwas supported in this work by Grant no. K01-AT002391, agrant from National Institutes of Health National Center forComplementary and Alternative Medicine (NIH-NCCAM).Study findings and conclusions are those of the authors


and do not represent the opinion or position of NIH orNCCAM. Institutional review board (IRB) approvals were byVA Greater Los Angeles Healthcare System (IRB of record),VAConnecticut Healthcare System (IRB for Co-PI),WesternIRB (IRB for Samueli Institute), and US Army MedicalResearch and Materiel Command (IRB for Department ofDefense).

References

[1] H. A. Tindle, R. B. Davis, R. S. Phillips, and D. M. Eisenberg,“Trends in use of complementary and alternative medicineby US adults: 1997–2002,” Alternative Therapies in Health andMedicine, vol. 11, no. 1, pp. 42–49, 2005.

[2] D. M. Eisenberg, R. B. Davis, S. L. Ettner et al., “Trends inalternative medicine use in the United States, 1990–1997: resultsof a follow-up national survey,” Journal of the AmericanMedicalAssociation, vol. 280, no. 18, pp. 1569–1575, 1998.

[3] R. L. Nahin, P. M. Barnes, B. J. Stussman, and B. Bloom,“Costs of Complementary and Alternative Medicine (CAM)and frequency of visits to CAM practitioners: United States,2007,” National Health Statistics Reports, no. 18, pp. 1–14, 2009.

[4] S. Ananth, “Integrative healthcare for the military: a SamueliInstitute/Department of Defense partnership,” Explore, vol. 8,no. 4, pp. 256–257, 2012.

[5] S. Ananth, “Applying integrative healthcare,” Explore, vol. 5, no.2, pp. 119–120, 2009.

[6] L. M. Denneson, K. Corson, and S. K. Dobscha, “Comple-mentary and alternative medicine use among veterans withchronic noncancer pain,” Journal of Rehabilitation Research andDevelopment, vol. 48, no. 9, pp. 1119–1128, 2011.

[7] C. M. Baldwin, K. Long, K. Kroesen, A. J. Brooks, and I. R. Bell,“A profile of military veterans in the southwestern United Stateswho use complementary and alternativemedicine: implicationsfor integrated care,” Archives of Internal Medicine, vol. 162, no.15, pp. 1697–1704, 2002.

[8] D. G. Campbell, A. P. Turner, R. M. Williams et al., “Comple-mentary and alternative medicine use in veterans with multiplesclerosis: prevalence and demographic associations,” Journal ofRehabilitation Research and Development, vol. 43, no. 1, pp. 99–110, 2006.

[9] F. P. McEachrane-Gross, J. M. Liebschutz, and D. Berlowitz,“Use of selected Complementary and Alternative Medicine(CAM) treatments in veterans with cancer or chronic pain: across-sectional survey,” BMC Complementary and AlternativeMedicine, vol. 6, article 34, 2006.

[10] K. Kroesen, C. M. Baldwin, A. J. Brooks, and I. R. Bell,“US military veterans’ perceptions of the conventional medicalcare system and their use of complementary and alternativemedicine,” Family Practice, vol. 19, no. 1, pp. 57–64, 2002.

[11] K. Bent and L. Hemphill, “Use of complementary and alter-native therapies among veterans: a pilot study,” Federal Practi-tioner, vol. 21, no. 10, pp. 43–52, 2004.

[12] Healthcare Analysis & Information Group (HAIG), Com-plementary and Alternative Medicine Survey, Office of Policyand Planning, Veterans Health Administration, Departmentof Veteran Affairs, Washington, DC, USA, 2011, http://shfwire.com/files/pdfs/2011CAM FinalReport.pdf.

[13] W. C. Meeker and S. Haldeman, “Chiropractic: a profession atthe crossroads of mainstream and alternative medicine,”Annalsof Internal Medicine, vol. 136, no. 3, pp. 216–227, 2002.

[14] A. J. Lisi, C. Goertz, D. J. Lawrence, and P. Satyanarayana,“Characteristics of Veterans Health Administration chiroprac-tors and chiropractic clinics,” Journal of Rehabilitation Researchand Development, vol. 46, no. 8, pp. 997–1002, 2009.

[15] Public Law 107-135, Department of Veterans Affairs Health CarePrograms Enhancement Act of 2001, section 204, Program forProvision of Chiropractic Care and Services to Veterans, Wash-ington, DC, USA, 2012, http://veterans.house.gov/legislation/107/hr3447b.html.

[16] J. W. Busse, C. Jacobs, T. Ngo et al., “Attitudes toward chiroprac-tic: a survey of North American orthopedic surgeons,” Spine,vol. 34, no. 25, pp. 2818–2825, 2009.

[17] A. J. Lisi, “Management of Operation Iraqi Freedom andOperation Enduring Freedom veterans in a Veterans HealthAdministration chiropractic clinic: a case series,” Journal ofRehabilitation Research and Development, vol. 47, no. 1, pp. 1–6,2010.

[18] B. N. Green, C. D. Johnson, A. J. Lisi, and J. Tucker, “Chiroprac-tic practice in military and veterans’ health care: the state of theliterature,” Journal of the Canadian Chiropractic Association, vol.53, no. 3, pp. 194–204, 2009.

[19] B. N. Green, C. D. Johnson, and A. J. Lisi, “Chiropractic in U.S.military and veterans’ health care,” Military Medicine, vol. 174,no. 6, pp. 6–7, 2009.

[20] M. W. Evans Jr., G. Page, H. Ndetan et al., “Are patientsreceiving health promotion advice in the chiropractic teachingclinic setting?: an impact assessment of a brief intervention toincrease advising rates and goal setting,” Journal of ChiropracticEducation, vol. 25, no. 2, pp. 132–141, 2011.

[21] A. S. Dunn, B. N. Green, L. R. Formolo, and D. Chicoine,“Retrospective case series of clinical outcomes associated withchiropractic management for veterans with low back pain,”Journal of Rehabilitation Research and Development, vol. 48, no.8, pp. 927–934, 2011.

[22] A. S. Dunn and S. R. Passmore, “Consultation request patterns,patient characteristics, and utilization of services within aVeterans Affairs medical center chiropractic clinic,” MilitaryMedicine, vol. 173, no. 6, pp. 599–603, 2008.

[23] A. S. Dunn, J. J. Towle, P. McBrearty, and S. M. Fleeson,“Chiropractic consultation requests in the Veterans Affairshealth care system: demographic characteristics of the initial100 patients at the western New York Medical Center,” JournalofManipulative and PhysiologicalTherapeutics, vol. 29, no. 6, pp.448–454, 2006.

[24] A. S. Dunn, “A survey of chiropractic academic affiliationswithin the department of Veterans Affairs health care system,”Journal of Chiropractic Education, vol. 21, no. 2, pp. 138–143,2007.

[25] P. Y. Huang, E. M. Yano, M. L. Lee, B. L. Chang, and L. V.Rubenstein, “Variations in nurse practitioner use in VeteransAffairs primary care practices,”Health Services Research, vol. 39,no. 4, part 1, pp. 887–904, 2004.

[26] P. D. Jacobson, L. E. Parker, and I. D. Coulter, “Nurse practi-tioners and physician assistants as primary care providers ininstitutional settings,” Inquiry, vol. 35, no. 4, pp. 432–446, 1998.

[27] A. Donabedian, “The quality of medical care. Methods forassessing and monitoring the quality of care for research andfor quality assurance programs,” Science, vol. 200, no. 4344, pp.856–864, 1978.

[28] A. Donabedian, “Evaluating the quality of medical care,” Mil-bank Quarterly, vol. 83, no. 4, pp. 691–729, 2005.


[29] A. Donabedian, “Evaluating the quality of medical care,” TheMilbank Memorial Fund Quarterly, vol. 44, supplement 3, pp.166–206, 1966.

[30] A. Donabedian, “The evaluation of medical care programs,”Bulletin of the New York Academy of Medicine, vol. 44, no. 2,pp. 117–124, 1968.

[31] A. Donabedian, “Quality of care: problems of measurement. II.Some issues in evaluating the quality of nursing care,”AmericanJournal of Public Health and the Nation’s Health, vol. 59, no. 10,pp. 1833–1836, 1969.

[32] A. Donabedian, “Some basic issues in evaluating the quality ofhealth care,” American Nurses Association Publications, no. 124,pp. 3–28, 1976.

[33] A. Donabedian, “Measuring and evaluating hospital and medi-cal care,” Bulletin of the New York Academy of Medicine, vol. 52,no. 1, pp. 51–59, 1976.

[34] A.Donabedian, J. R. C.Wheeler, and L.Wyszewianski, “Quality,cost, and health: an integrativemodel,”Medical Care, vol. 20, no.10, pp. 975–992, 1982.

[35] I. D. Coulter, R. D. Hays, and C. D. Danielson, “The chi-ropractic satisfaction questionnaire,” Tech. Rep. RP-374, RANDCorporation, Santa Monica, Calif, USA, 1995, http://www.rand.org/pubs/reprints/RP374.

[36] A.-F. Hsiao, R. D. Hays, G.W. Ryan et al., “A self-reportmeasureof clinicians’ orientation toward integrative medicine,” HealthServices Research, vol. 40, no. 5, part 1, pp. 1553–1569, 2005.

[37] N. E. Adler and J. Stewart, “Using team science to address healthdisparities: MacArthur network as case example,” Annals of theNew York Academy of Sciences, vol. 1186, pp. 252–260, 2010.

[38] D. Stokols, S. Misra, R. P. Moser, K. L. Hall, and B. K. Taylor,“The ecology of team science. Understanding contextual influ-ences on transdisciplinary collaboration,” American Journal ofPreventive Medicine, vol. 35, supplement 2, pp. S96–S115, 2008.

[39] K. Borner, N. Contractor, H. J. Falk-Krzesinski et al., “A multi-level systems perspective for the science of team science,”Science TranslationalMedicine, vol. 2, no. 49, Article ID 49cm24,2010.

[40] J.W. Creswell, A. C. Klassen, V. L. P. Clark, and K. C. Smith, BestPractices for Mixed Methods Research in the Health Sciences,Office of Behavioral and Social Sciences Research, Washington,DC, USA, 2011, http://obssr.od.nih.gov/scientific areas/meth-odology/mixed methods research/pdf/Best Practices forMixed Methods Research.pdf

[41] V. L. P. Clark and J. W. Creswell, Understanding Research: AConsumer’s Guide, Prentice Hall-Merrill, Upper Saddle River,NJ, USA, 2010.

[42] L. A. Curry, I. M. Nembhard, and E. H. Bradley, “Qualitativeand mixed methods provide unique contributions to outcomesresearch,” Circulation, vol. 119, no. 10, pp. 1442–1452, 2009.

[43] A. B. Almarsdottir and J. M. Traulsen, “Multimethod researchinto policy changes in the pharmacy sector-the Nordic case,”Research in Social and Administrative Pharmacy, vol. 5, no. 1,pp. 82–90, 2009.

[44] K. C. Stange, B. F. Crabtree, andW. L. Miller, “Publishingmulti-method research,” Annals of Family Medicine, vol. 4, no. 4, pp.292–294, 2006.

[45] K. Wilkins and R. Woodgate, “Designing a mixed methodsstudy in pediatric oncology nursing research,” Journal of Pedi-atric Oncology Nursing, vol. 25, no. 1, pp. 24–33, 2008.

[46] J. W. Creswell and W. Zhang, “The application of mixed meth-ods designs to trauma research,” Journal of Traumatic Stress, vol.22, no. 6, pp. 612–621, 2009.

[47] S. Beake, L. L. Clark, T. Turner, and D. Bick, “A mixed methodsstudy to develop and pilot a competency assessment tool tosupport midwifery care of women with intellectual disabilities,”Nurse Education Today, vol. 33, no. 8, pp. 901–906, 2013.

[48] A. C. Klassen, K. C. Smith, M. M. Black, and L. E. Caulfield,“Mixed method approaches to understanding cancer-relateddietary risk reduction among public housing residents,” Journalof Urban Health, vol. 86, no. 4, pp. 624–640, 2009.

[49] K. McVea, B. F. Crabtree, J. D. Medder et al., “An ounce ofprevention? Evaluation of the “put prevention into practice”program,” Journal of Family Practice, vol. 43, no. 4, pp. 361–369,1996.

[50] R. J. Pasick, N. J. Burke, J. C. Barker et al., “Behavioral theory ina diverse society: like a compass on Mars,” Health Education &Behavior, vol. 36, no. 5, pp. 11S–35S, 2009.

[51] V. L. P. Clark, “The adoption and practice of mixed methods:U.S. trends in federally funded health-related research,” Quali-tative Inquiry, vol. 16, no. 6, pp. 428–440, 2010.

[52] National Institutes of Health, Office of Behavioral andSocial Sciences Research, Qualitative Methods in Health Re-search: Opportunities and Considerations in Application andReview, National Institutes of Health, Office of Behavioraland Social Sciences Research, Washington DC, USA, 2001,obssr.od.nih.gov/pdf/qualitative.pdf.

[53] J. Meiklejohn, J. Connor, and K. Kypri, “The effect of low surveyresponse rates on estimates of alcohol consumption in a generalpopulation survey,” PLoS ONE, vol. 7, no. 4, Article ID e35527,2012.

[54] I. D. Coulter and R. Khorsan, “Is health services researchthe Holy Grail of complementary and alternative medicineresearch?” AlternativeTherapies in Health andMedicine, vol. 14,no. 4, pp. 40–45, 2008.

[55] J. M. Morse, “The politics of evidence,” in Qualitative Inquiryand the Conservative Challenge, N. K. Denzin and M. D.Giardina, Eds., pp. 79–92, Left Coast Press,Walnut Creek, Calif,USA, 2006.

[56] R. J. Chenail, “Ten steps for conceptualizing and conductingqualitative research studies in a pragmatically curious manner,”Qualitative Report, vol. 16, no. 6, pp. 1715–1730, 2011.

Submit your manuscripts athttp://www.hindawi.com

Stem CellsInternational

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014


MEDIATORSINFLAMMATION

of


Behavioural Neurology

EndocrinologyInternational Journal of



Disease Markers


BioMed Research International

OncologyJournal of



Oxidative Medicine and Cellular Longevity


PPAR Research

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

Immunology ResearchHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Journal of

ObesityJournal of



Computational and Mathematical Methods in Medicine

OphthalmologyJournal of


Diabetes ResearchJournal of



Research and TreatmentAIDS


Gastroenterology Research and Practice


Parkinson’s Disease

Evidence-Based Complementary and Alternative Medicine

Volume 2014Hindawi Publishing Corporationhttp://www.hindawi.com

Research Article Mixed-Methods Research in a Complex ...

Documents

Transcript of Research Article Mixed-Methods Research in a Complex ...