Generating the right kind of clinical evidence and guidance

Professor Peter LittlejohnsClinical and Public Health Director

Summary of presentation

• What do we mean by evidence and guidance?

• Variation (too much or too little) can mean stopping things – NICE approach to disinvestment. Describe a joint project with the Cochrane Collaboration.

• Observations

What is meant by evidence ?

• There are differing views on what the ‘evidence’ in evidence-based health care should be.

• There are three categories of evidence: medical effectiveness research (context-free scientific evidence); social science-oriented research (context sensitive scientific evidence); or the expertise, views, and realities of stakeholders (colloquial evidence).

• These views of evidence are not incompatible and each has a role to play in producing evidence-based guidance for the health system.

• Under usual circumstances, there are no magic technical processes available to combinethese different forms of evidence to create health system guidance. Thus each form ofevidence must be entered into a deliberative process, with representation from both thescientific and stakeholder communities, if they are to be converted into a final consensusaround appropriate, feasible, and realistic guidance for the health system.

Conceptualizing and Combining Evidence for Health System GuidanceJonathan Lomas, Tony Culyer, Chris McCutcheon, Laura McAuley, and Susan Lawfor the Canadian Health Services Research FoundationMay 2005

Evidence

Evaluating the evidence

Research Evidence

Patient Experience

Clinical Practice

Judgements to be madeJudgements to be made

The Nature of Evidence (i)

‘What is now proved

was once only imagined’

Picture of Sir Isaac Newton by William Blake – English Poet and Artist (1757-1827)

To support innovation NICE needs to be able to handle ‘immature’ evidence

The Nature of Evidence (ii)

‘God forbid that truth should be confined to mathematical demonstration’

Picture of the grand architect William Blake – English Poet and Artist (1757-1827)

NICE makes scientific and social values judgements

The Nature of Evidence (iii)

‘One law for the lion and the ox is oppression.’

Picture of Job after his tribulations by William Blake – English Poet and Artist (1757-1827)

Assess evidence in relationship to its ‘fitness for purpose’ rather than according to pre-ordained hierarchies

NICE approach has 4 steps to assessing evidence

Step 1: Define the clinical or public health question

Step 2: Identify the evidence

Step 3: Synthesise and assess the body of evidence

Step 4: Issue the recommendations

A short history of NICE’s products

QOF

Public health

Interventional Procedures

Clinical guidelines

Quality standards

NICE range of products aimed at reducing variation and saving money

All NICE guidance

Commissioning guides

Costing tools

‘Do not do’ database

Referral database

NHS Evidence QIPP page

Disinvestment

• Reducing variation can mean encouraging clinicians to do something but more often means encouraging clinicians to stop doing something.

• Health technology appraisal has been used as a tool to manage the entry of new interventions into system.

• Interest turning to methods involving identification of existing interventions thought to be clinically or cost ineffective.

• Concept described by the term disinvestment:

‘The practice of stopping or restricting the use of low-value health care practices to enable resources to be shifted to higher-value care.’

BMJ 2011; 343:d4519 doi: 10.1136/bmj.d4519 (Published 27 July 2011) Cite this as: BMJ 2011; 343:d4519

NICE and Cochrane work together

• NICE and Cochrane produce gold standard products of evidence-based medicine.

• For a pilot project the UK Cochrane Centre sent NICE published Cochrane systematic reviews concluding an intervention could not be recommended.

• NICE summarised reviews and published as ‘Cochrane Quality and Productivity’ topics on NHS evidence.

Summary of the process from Cochrane review to QIPP topic

Results

• 65 reviews were appraised over five months• 43% highlighted candidate interventions for

local disinvestment • 57% of reviews were rejected • 36 Cochrane Quality Improvement topics have

now been published on the NHS website suggesting potential candidates for local disinvestment.

• http://www.evidence.nhs.uk/qipp

Results: Reasons for rejecting reviews

0 2 4 6 8 10 12

Intervention not available in UK (15%)

Clinical Qs not relevant to UK (13%)

Change in line with UK practice: no gains (8%)

Additional analysis required e.g patient safety concerns (25%)

Review concludes intervention is beneficial (10%)

Clash NICE guidance (15%)

Complex or controversial topic (13%)

NICE guideline in progress (3%)

Lessons learnt

• This pilot project was designed as a rapid response process to identify potential low-value health care practices that would not be considered priorities for national guidance, based on the conclusions of high-quality systematic reviews.

• The process proved very resource-intensive, with a rejection rate of 57%.

• The process highlighted the need for more detailed data on health care expenditure than is routinely collected in the NHS.

• Few interventions identified as ineffective were in regular use in the NHS.

• Additional work is generally needed to evaluate interventions for which there is inadequate evidence: Stopping an intervention based on insufficient evidence can be potentially harmful and requires more rigorous investigation.

Implications for the health service (i)

• An evidence-based approach to identifying clinical activity for disinvestment proved challenging

• Often there is an absence of evidence rather than conclusive evidence of a lack of benefit

• More resource-intensive analysis is needed to facilitate local NHS implementation of disinvestment

• To achieve real productivity savings, people, structures and values need to be assessed as well as marginal clinical activity

Implications for the health service (ii)

• This suggests alternative disinvestment strategies may be needed to achieve productivity savings requiring infrastructure, resources and robust national processes.

• Encouraging guideline developers to identify inappropriate practices as opportunities for disinvestment is likely to be more productive than relying on systematic reviews alone.