Assessing Applicability of Medical Test Studies in Systematic Reviews

Assessing Applicability of Medical Test Studies in

Systematic ReviewsPrepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Medical Test Reviews Methods Guide

www.ahrq.gov

Overview of a Medical Test Review

Prepare Topic

•Develop the Topic and Structure the Review

•Choose the Important Outcomes

Research Sources

Search for and Select Studies for Inclusion

• Search for Studies

Extract Data From Studies

Analyze and Synthesize Studies

•Assess Risk of Bias as a Domain of Quality

•Assess Applicability

•Grade the Body of Evidence

•Meta-analysis of Test Performance Evidence With a “Gold Standard” — or —

•Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard

•Decision Modeling

Report Medical Test

ReviewHartman KE, Matchar DB, Chang SM. Assessing applicability of medical test studies in systematic reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

Explain how applicability extends to both the individual study and bodies of evidence

Describe three main challenges related to assessing applicability and how they are often related to context

Understand the four principles for assessing and reporting applicability

Apply the PICOTS (population, intervention, comparator, outcomes, timing, and setting) typology to assess applicability

Learning Objectives

Hartman KE, Matchar DB, Chang SM. Assessing applicability of medical test studies in systematic reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

Decisionmakers need to understand whether a medical test is meaningful in a specific context in order to make an informed decision. For example: Whether the test has been studied in the patient/disease/care

settings of interest Whether the test has been part of a management strategy of

interest

Evidence-based Practice Centers use four principles to assess and report applicability: Determine the most important factors that affect applicability Systematically abstract key characteristics that may affect

applicability Make judgments about major limitations to the applicability of

individual studies Consider the applicability of the body of evidence

Background


1. Key questions lack clarity.

2. Studies are not specific to the key questions.

3. Tests are rapidly evolving.

Challenges Specific to Assessing Applicability(1 of 4)


Key questions lack clarity How the test is to be used should be stated clearly in the

key questions to avoid confusion among decisionmakers about the management strategy being evaluated. Example: Studies comparing the accuracy of one test

versus another (i.e., as a replacement), but the new test is actually meant to be a triage/add-on test

Hinders the review process (slows inclusion/exclusion decisions)

Leads to inappropriate “lumping” and “splitting” of groups Lumping subgroups that are fundamentally different

can cause a false finding of a difference (type 1 error). Splitting subjects into subgroups without evidence of

different effects may reduce power to detect a difference (type II error).



Studies are not specific to the key questions Studies may not reflect the subgroups or

comparisons identified in the key questions. Reviewers need to decide whether the deviation is

minor or crucial. Crucial deviations are those deemed likely to

meaningfully affect: Test performance Clinical decisionmaking Health outcomes



Tests are rapidly evolving and change often. The literature often references tests that are not

yet widely available or no longer common. Secular trends in use patterns and market forces

may shape applicability in unanticipated ways. Example: An old test that gives dichotomous,

qualitative results (present vs. absent) is replaced by a test that reports a continuous, quantitative measure. Review the old test as a point of reference for

determining the merit of a new test? Only use the more limited, newer data from the new

test?



1. Identify important contextual factors

2. Be prepared to deal with additional factors affecting applicability

3. Justify decisions to restrict the scope of a review

4. Maintain a transparent process

Principles of Assessing Applicability


It is not practical to consider all possible factors related to the impact of test on health outcomes Consider factors that affect steps in a causal chain of

direct relevance to the key question. Example: Slice thickness is a relevant factor for assessing the

applicability/accuracy of cardiac magnetic resonance imaging for detecting atherosclerosis.

Also consider factors that affect later links in the causal chain. Example: What factors impact effectiveness of treatment for

lesions identified by cardiac MRI versus angiography?

Consider contextual issues relevant to medical tests, for example, patient populations, management strategies, time effects, and secular trends in disease prevalence and severity.

Principle 1: Identify Important Contextual Factors (1 of 10)


Spectrum effect The severity or type of disease may affect test

accuracy. Example: Cardiac magnetic resonance imaging is

generally accurate in identifying cardiac anatomy/functionality, but arrhythmias, lesion location, and obesity can affect performance of the test.

Reviewers must identify these disease-related factors ahead of time and justify when to split key questions or to conduct subgroup analyses.



The medical test as part of a management strategy It is important to clarify conditions in which a test is

studied and used. Example: The test performance of cardiac magnetic

resonance imaging (MRI) differs when used as an add-on test after initial screening versus when used for triage or for patients without symptoms. When cardiac MRI is used as add-on, the

comparator for diagnostic accuracy is x-ray coronary angiography (the gold standard).

When cardiac MRI is used for triage, the comparator is stress electrocardiography or other initial screening test.



Methods of the medical test over time Diagnostics evolve rapidly. Awareness of these changes should guide selection of the

date range and eligible test types for included literature, Example 1: The slice thickness of magnetic resonance

imaging scans has decreased steadily over time, allowing resolution of smaller lesions. Excluding studies with older technologies and presenting

results of included studies by slice thickness may be appropriate.

Example 2: Antenatal tests are applied earlier and earlier in gestation. Studies of test performance need to be examined by

varied cutoffs for stage of gestation.



Secular trends in population risk and disease prevalence Direct/indirect changes in secular trends (or cultural

differences) can influence test performance and applicability. Example: Obesity and the average age of pregnancy

have risen dramatically in the past 30 years, both of which are linked to type 2 diabetes. The underlying prevalence of undiagnosed type 2

diabetes in pregnancy is increased. As a result, the predictive values, the cost/benefit

ratios, and the sensitivity and specificity of screening tests for gestational diabetes change over time.



Secular trends in population risk and disease prevalence Secular trends in population characteristics can

influence the ability of clinicians to conduct a medical test. Example: Obesity diminishes image quality in tests,

such as ultrasound for gallbladder disease or magnetic resonance imaging for spinal conditions. Studies of these tests often restrict enrollment to

people with normal body weight. These tests may not perform the same in practice

because of the rising trend of obesity in the actual patient population.



Secular trends in population risk and disease prevalence Prevalence is tied to disease definitions that can change.

Some examples are: Criteria to diagnose AIDS Cystometrically defined detrusor instability/overactivity

versus the symptom complex “overactive bladder” Continuous refinement of classifications of mental health

conditions in the Diagnostic and Statistical Manual of Mental Disorders

The literature may not capture changes in diagnostic criteria. Having expert knowledge with a historical vantage point is

helpful.



Routine preventive care over time Routine use of a medical test as a screening test may

indirectly alter population prevalence. Preventive lipid testing in individuals with cardiovascular

disease has changed the proportion able to be diagnosed and have the course of their disease altered.

The vaccine for the human papilloma virus (HPV) is postulated to change the distribution of viral subtypes in the population.

Preventive practices change the utility of medical tests. Example: The utility of the test to detect HPV infection is

decreasing as increasing numbers of people are being vaccinated for the virus.



Treatment trends Literature about the impact of diagnostic tools on

outcomes needs additional interpretation in light of new therapeutics. Example: The implications of testing for carotid arterial

stenosis are changing as hypertension treatment and the use of lipid-lowering drugs have improved.



Two steps to ensure that population/subgroup data are uniformly collected and useful: Use the PICOTS (population, intervention,

comparator, outcomes, timing, and setting) typology to identify the range of possible factors that might affect applicability and consider the six contextual factors.

Review the list of factors affecting applicability with stakeholders to: Ensure common vantage points Identify hidden factors that may influence applicability Capture crucial applicability features in the evidence

review



Using the PICOTS Framework To Assess and Describe Applicability of Medical Tests (1 of 7)

PICOTS Element

Potential Core Characteristics To Describe and Assess

Challenges When Assessing Studies

Example

Potential Systematic Approaches for Decisions About Including Studies

Population

Justification for lumping or splitting key questions

Method of identification/ selection

Inclusion and exclusion criteria of review

Demographic characteristics of those included in the review

Prevalence of condition in practice and in studies

Spectrum of disease in practice and in studies

Source of population not described

Study population poorly specified

Key characteristics not reported

Unclear whether test performance varies by population

Education/ literacy level not reported in study of pencil-and-paper functional status assessment

Exclude a priori if key element crucial to assessing intended use case is missing

Or include, but:– Flag missing

elements in tables/text

– Organize data within key questions by presence/absence of key elements

– Include presence/ absence as parameter in meta-regression or sensitivity analyses

– Note need for challenge to be addressed in future research



PICOTS Element



Example


Intervention

Version of test used in practice and in studies

How and by whom tests are conducted in practice and in studies

Cutoff/diagnostic thresholds applied in practice and in studies

Skill of assessors when interpretation of test required in studies

Version/ instrumentation not specified

Training/quality control not described

Screening and diagnostic uses mixed

Ultrasound machines and training of sonographers not described in a study of fetal nuchal translucency assessment to detect aneuploidy

Exclude a priori if version critical and not assessed

Or include, but: – Contact authors for

clarification– Flag version of test

or deficits in reporting in tables/text

– Discuss implications

– Model cutoffs and conduct sensitivity analyses



PICOTS Element



Example


Comparator

Use of gold standard versus “alloy” standard in studies

How the test is used as part of a management strategy (e.g., triage, replacement, or add-on) in practice and in studies

Gold standard not applied

Correlational data only

Cardiac computerized tomography compared with a treadmill stress test without use of angiography as a gold standard

Exclude a priori if there is no gold standard

Or include, but:– Restrict to

specified comparators

– Group by comparator in tables/text



PICOTS Element



Example


Outcome(of using the test)

How accuracy outcomes selected for review relate to use in practice:

Accuracy of disease status classification

Sensitivity/ specificity

Predictive values Likelihood ratios Diagnostic odds

ratio Area under curve Discriminant

capacity

Failure to test “normal” patients, or a subset thereof, with the gold standard

Precision of estimates not provided

Tests may be used as part of management strategy in which exact diagnosis is less important than “ruling out” a disease

P-value provided for mean of continuous test results by disease status, but confidence bounds not provided for performance characteristics

Exclude a priori if test results cannot be mapped to disease status (i.e., 2×2 or other test performance data cannot be extracted)

Exclude if a subset of “normal” patients not tested

Or include, but: Flag deficits in

tables/text Discuss implications Assess heterogeneity

in meta-analysis and comment of sources of heterogeneity in estimates



PICOTS Element



Example


Clinical Outcomes(from test results)

How studies addressed clinical outcomes selected for the review:

– Earlier diagnosis– Earlier

intervention– Change in

treatment given– Change in

sequence of other testing

– Change in sequence/intensity of care

– Improved outcomes, quality of life, costs, etc.

Populations and study designs of included studies heterogeneous with varied findings

Data not stratified or adjusted for key predictors

Bone density testing reported in relation to fracture risk reduction without consideration of prior fracture or adjustment for age

Exclude if no disease outcomes and outcomes key to understanding intended use case

Or include, and: Document details of

deficits in tables/text Discuss implications Note need for

challenge to be addressed in future research



PICOTS Element



Example


Timing Timing of availability of results to care team in studies and how this might relate to practice

Placement of test in the sequence of care (e.g., relationship of test to treatment or follow-on management strategies) of studies and how this might relate to practice

Timing of assessment of disease status and outcomes in studies

Sequence of use of other diagnostics unclear

Time from results to treatment not reported

Order of testing varies across subjects and was not randomly assigned

D-dimer studies in which it is unclear when results were available relative to imaging studies of deep vein thrombosis

Exclude if timing/sequence is key to understanding intended use case

Or include, and: Contact authors for

information Flag deficits in

tables/text Discuss implications Note need for challenge

to be addressed in future research



PICOTS Element



Example


Setting How setting of test in studies relates to key questions and current practice:

– Primary care versus specialty care

– Hospital-based– Routine processing

versus specialized lab or facility

– Specialized personnel – Screening versus

diagnostic use

Resources available to providers for diagnosis and treatment of condition vary widely

Provider type/ specialty vary across settings

Comparability of care in international settings unclear

Diagnostic evaluation provided by geriatricians in some studies and unspecified primary care providers in others

Exclude if care setting known to influence test/outcomes or if setting is key to understanding intended use case

Or include, but: Document details of

setting Discuss implications


Some contextual factors are only found after a large volume of literature has been reviewed. Example: A meta-analysis may find that a test is

inaccurate in older patients, but age was not considered in the key questions. May be a false association May be an oversight

When in doubt between a false association or oversight, it is appropriate to comment on an apparent association and clearly state that it is only a hypothesis.

Principle 2: Be Prepared To Deal With Additional Factors Affecting Applicability


May be necessary to restrict the scope of the review to: Specific versions of the test Selected study methods or types Specific populations (e.g., women, patients who are

obese) Specific settings (e.g., primary care practice)

Restriction is appropriate when: All agree that a top priority is applicability to a

particular target group or setting Test performance is different in a specific subgroup,

setting, or from another version of the test Restriction is not appropriate when a fully

comprehensive review is desired

Principle 3: Justify Decisions To Restrict the Scope of a Review


Address applicability when defining review methods and documenting decisions in a protocol. Example: Factors that change with time should

prompt adding timeframes to inclusion criteria or description/ analysis of possible impacts on applicability.

Transparency is essential when a review decision may be controversial. Example: Exclusion criteria leave a reviewer “empty

handed;” the reviewer may decide some excluded evidence is relevant by extension or analogy.

In this event, clearly state how evidence may not be directly applicable to key questions but represents the limited state of the science.

Principle 4: Maintain a Transparent Process


2002 Cervical Cancer Screening Summary of the Evidence for the U.S. Preventive Services Task Force

Challenges: Changes in secular trends and availability/format of

tests Liquid-based cervical cytology (new technique) rapidly

being used in practice Resource shortages for reviewing conventional Pap

smears New computer systems to screen cytology slides to

enhance accuracy Literature starting to include adjunct usage of human

papilloma virus vaccine No randomized controlled trials available

Illustrative Example: Cervical Cancer Screening (1 of 3)

Hartman KE, Matchar DB, Chang SM. Assessing applicability of medical test studies in systematic reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.Hartmann KE, Hall SA, Nanda K, et al. Systematic Evidence Review No. 25. Available at www.ncbi.nlm.nih.gov/books/NBK42831.

Since no data compared new screening tools/strategies to outcomes, the 2002 Cervical Cancer Screening Summary of the Evidence for the U.S. Preventive Services Task Force report focused on medical test characteristics (sensitivity, specificity, predictive value, and likelihood ratios). Three computer technologies Two liquid cytology approaches All methods of human papilloma virus testing

Included all technologies Restricting technologies only to those available in the United

States (most applicable setting) would have reduced the scope substantially.

The purpose was to determine if comparable or superior methods were being overlooked or no longer offered.



There are two specific areas of applicability interest — age and hysterectomy status. The underlying incidence and prevalence data included:

Provided context Informed modeling of testing impact

Reviewers could not estimate the usefulness of human papilloma virus testing among older women from trends in younger women. Extrapolating results from one context to another rarely

enhances applicability.

The review found that older women are disproportionately more likely to have false-positive findings under current screening recommendations and supported the discontinuation of Pap testing after hysterectomy.



To adequately assess applicability of a medical test, reviewers need to familiarize themselves with: Availability of the test Technology of the test Contemporary clinical use of the test Current treatment modalities for the related disease

condition Potential interplay of disease severity and the

performance characteristics of the test Implications of particular study designs and sampling

strategies for bias in the findings about applicability

Summary (1 of 3)


Restricting inclusion of only marginally related or outdated studies can aid applicability.

Extrapolating results from one context to another does not help.

Guidance from clinical, medical testing, and statistical experts is helpful when considering restriction of scope or exclusion of publications with specific types of flaws.

Summary (2 of 3)


The review team must group findings in a meaningful way, whether the review is exhaustive or narrow. For medical tests, this means gathering and

synthesizing data in ways that help to readily understand applicability.

Grouping summaries of findings using familiar structures like the PICOTS (population, intervention, comparator, outcomes, timing, and setting) enhances clarity in framing applicability.

Studies may be presented more than once within a review, in different “applicability axes.”

Summary (3 of 3)


Identify contextual factors that may affect test performance early in the review process.

Carefully consider and document justification for addressing these factors by restricting key questions or creating a broader review. Protocols should document which factors will be

excluded and how subgroups will be assessed. Reviewers should document how they plan to handle

studies that only partly fit the key questions or inclusion/exclusion criteria.

Final reviews should include a description of test use in usual practice and how the included studies fit with this use.

Key Messages


1. When conducting most systematic reviews, it is important to limit the contextual factors considered in applicability assessment rather than considering all possible factors.

a. True

b. False

Practice Question 1 (1 of 2)

Explanation for Question 1:

The statement is true. It is not practical to consider all possible factors that may impact applicability in all cases. This approach would be overwhelming and, in some cases, leave the reviewer with no applicable studies. A reasonable list of factors should be considered and agreed upon by the stakeholders in the systematic review before initiating the literature review.


2. The routine use of the human papilloma virus vaccine to prevent cervical cancer may change the utility of the test to detect the virus.

a. True

b. False



The statement is true. This is an example of how the changes in routine preventive care over time can affect the prevalence of disease. Knowledge of these trends is important when looking at the applicability of a body of literature.


3. One of the general principles in assessing the applicability is to keep the process as confidential as possible without involving the stakeholders and future decisionmakers.

a. True

b. False



The statement is false. Transparency is essential while addressing applicability. Review methods should be documented clearly in a protocol.


4. The PICOTS (population, intervention, comparator, outcomes, timing, and setting) framework is a tool that can help to systematically abstract and report key characteristics that may affect applicability.

a. True

b. False



The statement is true. The PICOTS (population, intervention, comparator, outcomes, timing, and setting) framework is a tool that can be used in a systematic review to meticulously think through all of the factors that may affect applicability.


This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center.

The module is based on chapter 6, Assessing applicability of medical test studies in systematic reviews. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 6.1-13. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm

Authors

Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Review. Rockville, MD: Agency for Healthcare Research and Quality; April 2012. AHRQ Publication No. 10(12)-EHC063-EF. Chapters available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

Hartmann KE, Hall SA, Nanda K, et al. Screening for Cervical Cancer. Systematic Evidence Review No. 25 (Prepared by the Research Triangle Institute/University of North Carolina under Contract No. 290-97-0011, Task Order No. 3). Rockville, MD: Agency for Health Care Research and Quality; January 2002. www.ncbi.nlm.nih.gov/books/NBK42831.

Hartman KE, Matchar DB, Chang SM. Assessing applicability of medical test studies in systematic reviews. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 6.1-13. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

References (1 of 2)

Hartmann KE, Nanda K, Hall S, et al. Technologic advances for evaluation of cervical cytology: is newer better? Obstet Gynecol Surv. 2001 Dec;56(12):765-74. PMID: 11753179.

Hutchinson ML, Zahniser DJ, Sherman ME, et al. Utility of liquid-based cytology for cervical carcinoma screening: results of a population-based study conducted in a region of Costa Rica with a high incidence of cervical carcinoma. Cancer. 1999 Apr;87(2):48-55. PMID: 10227593.

McCrory DC, Matchar DB, Bastian L, et al. Evaluation of Cervical Cytology. Evidence Report/Technology Assessment No. 5 (Prepared by the Duke University Evidence-based Practice Center under Contract No. 290-07-0014. Rockville, MD: Agency for Health Care Policy and Research; January 1999. AHCPR Publication No. 99-E010. www.ncbi.nlm.nih.govbooks/NBK11840.

References (2 of 2)

Assessing Applicability of Medical Test Studies in Systematic Reviews

Documents

Transcript of Assessing Applicability of Medical Test Studies in Systematic Reviews