Study into the impact of the Belgian Health Care Knowledge ...

Study into the impact of the

Belgian Health Care Knowledge Centre

Assigned by the Belgian Health Care Knowledge Centre

Poortvliet. E.P Vijfvinkel, D. Vennekens, A. van Hoesel, P. Daue, F.

April 2010

1

Scientific Summary Table of contents

LIST OF ABBREVIATIONS ........................................................................ 4

1 INTRODUCTION............................................................................... 5

1.1 ORGANIZATIONAL OUTLINE OF KCE .................................................... 5 1.1.1 Mission, objectives and tasks ...................................................... 5 1.1.2 Main research fields ................................................................... 5 1.1.3 Organizational structure ............................................................. 6

1.2 REQUEST FOR IMPACT ASSESSMENT ................................................... 6

1.3 SCOPE AND LIMITATION OF THIS IMPACT STUDY .................................. 7

1.4 STRUCTURE OF THE REPORT .............................................................. 7

2 LITERATURE REVIEW ON MEASURING IMPACT ............................... 8

2.1 INTRODUCTION ................................................................................ 8

2.2 LITERATURE SEARCH......................................................................... 9 2.2.1 Searching relevant articles.......................................................... 9 2.2.2 Selection of articles using inclusion and exclusion criteria ...............11

2.3 REVIEWING THE ARTICLES................................................................11 2.3.1 Developing fact sheets ..............................................................11 2.3.2 Selecting possible methodologies for KCE impact assessment .........11

2.4 STRENGHTS AND WEAKNESSES OF SELECTED METHODOLOGIES ...........13 2.4.1 Methodology of SDO .................................................................13 2.4.2 Methodology by Oortwijn ...........................................................14 2.4.3 Methodology by Hanney ............................................................16 2.4.4 Methodology by Meagher ...........................................................17 2.4.5 Methodology by Alberta Heritage Foundation for Medical Research...18 2.4.6 Conclusion...............................................................................19

2.5 TOWARDS THE CONSTRUCTION OF A FRAMEWORK...............................20

3 FRAMEWORK FOR KCE IMPACT STUDY .......................................... 21

3.1 STARTING POINTFOR DEVELOPING THE FRAMEWORK ...........................21 3.1.1 Definition of impact...................................................................21 3.1.2 Model of the rationale underlying KCE’s activities ..........................21 3.1.3 Scope of the study....................................................................23 3.1.4 Research questions...................................................................24

3.2 SET OF ‘HYPOTHESES’ GUIDING THE IMPACT ASSESSMENT ...................25

3.3 RESEARCH DESIGN ..........................................................................26 3.3.1 Introduction.............................................................................26 3.3.2 Analysis framework...................................................................27 3.3.3 Research phases ......................................................................28 3.3.4 Phase 1: Research institute........................................................30 3.3.5 Phase 2: Research projects ........................................................33 3.3.6 Phase 3: International comparison ..............................................36 3.3.7 Phase 4:Overall analysis............................................................36

4 RESULTS OF THE IMPACT STUDY................................................... 38 4.1 INTRODUCTION ...............................................................................38

4.2 ORIGIN OF THE RESEARCH PROJECTS.................................................39 4.2.1 Pre proposal phase ...................................................................39 4.2.2 Proposals submitted..................................................................39 4.2.3 Proposal selection and research prioritization................................40

4.3 RESEARCH PROCESS ........................................................................45 4.3.1 Quality of the research process...................................................45 4.3.2 Intended impact .......................................................................53

2

4.4 DISSEMINATION ..............................................................................55 4.4.1 Outputs produced as a result of research projects .........................55 4.4.2 Dissemination activities performed..............................................59 4.4.3 Stakeholders addressed by outputs .............................................61 4.4.4 Engagement of stakeholders in outputs and dissemination..............63 4.4.5 Actual take up of outputs...........................................................65

4.5 CONTRIBUTION TO THE DECISION MAKING PROCESS...........................66 4.5.1 Relevance of projects for policy-making or practice .......................66 4.5.2 Feasibility and usefulness of the recommendations ........................70 4.5.3 Timely issuing of the report........................................................71 4.5.4 Degree and type of contribution to health care policy and practice ...72 4.5.5 Reactions on finished projects from stakeholders ..........................75

4.6 EXTERNAL IMAGE OF KCE..................................................................76 4.6.1 General reputation of KCE..........................................................77 4.6.2 Judgment on responsiveness......................................................78 4.6.3 Judgment on research quality.....................................................79 4.6.4 KCE’s added value compared to other organizations ......................80

4.7 CASE STUDIES ................................................................................81 4.7.1 Introduction.............................................................................81 4.7.2 Transparent and careful proposal selection procedure ....................82 4.7.3 Usability and advance formulation of specific goal .........................83 4.7.4 Explicit goals for user engagement and dissemination....................85 4.7.5 High quality research process.....................................................86 4.7.6 Active stakeholder involvement during research............................88 4.7.7 Clear relationship research findings and decision-making process ....91 4.7.8 Active dissemination based on dissemination strategy....................92 4.7.9 Identifying specific target users and tailoring products to needs ......94 4.7.10 Feedback and incorporating lessons in research process ..........95 4.7.11 Image of KCE amongst stakeholders .....................................96 4.7.12 Indication of impact............................................................98 4.7.13 Other success and failure factors ........................................ 100 4.7.14 Cross case analysis........................................................... 101

4.8 INTERNATIONAL COMPARISON ........................................................ 102 4.8.1 Introduction........................................................................... 102 4.8.2 Institutional design ................................................................. 102 4.8.3 Budget, staffing and funding .................................................... 102 4.8.4 Proposal selection................................................................... 104 4.8.5 Research process.................................................................... 104 4.8.6 Dissemination ........................................................................ 104 4.8.7 Contribution to the decision-making process............................... 105 4.8.8 Challenges and conclusions ...................................................... 106

5 DISCUSSION AND CONCLUSION...................................................107

5.1 DISCUSSION ................................................................................. 107 5.1.1 Scope ................................................................................... 107 5.1.2 Strengths and limitations......................................................... 107

5.2 CONCLUSIONS............................................................................... 108 5.2.1 Conclusions on hypotheses ...................................................... 109 5.2.2 General conclusions ................................................................ 111

6 REFERENCES.................................................................................113

7 APPENDICES ................................................................................117

APPENDIX 1: SEARCH STRATEGY .........................................................117

APPENDIX 2: FACT SHEETS ..................................................................122

APPENDIX 3: SPECIFICATION THEMES.................................................156

3

APPENDIX 4: SOURCES ........................................................................175

APPENDIX 5: DOCUMENT ANALYSIS.....................................................176

APPENDIX 6: QUESTIONNAIRE GROUP SESSION .................................179

APPENDIX 7: QUESTIONNAIRE STAKEHOLDERS...................................183

APPENDIX 8: QUESTIONNAIRE SELF REPORTS.....................................190

APPENDIX 9: TABLE PRE-SELECTION CASES ........................................198

APPENDIX 10: CASE STUDY PROTOCOL................................................199 CASE STUDY FRAMEWORK ...................................................................... 199

CASE STUDY PROTOCOL ......................................................................... 201

APPENDIX 11: CASE STUDY QUESTIONNAIRES ....................................204

APPENDIX 12: CASE STUDY RESULTS...................................................209

APPENDIX 13: COMPARISON FRAMEWORK INTERNATIONAL AGENCIES.....................................................................................................211

APPENDIX 14: SELECTION OF AGENCIES .............................................213

APPENDIX 15: REPORTS ON FOREIGN AGENCIES.................................218 THE HAUTE AUTORITÉ DE SANTÉ (HAS) .............................................. 218

THE SWEDISH COUNCIL ON TECHNOLOGY ASSESSMENT IN HEALTH CARE (SBU) .......................................................................................... 227

NATIONAL INSTITUTE FOR HEALTH RESEARCH (NIHR) ....................... 239

4

LIST OF ABBREVIATIONS Abbreviation

English

Other languages

AHFMR Alberta Heritage Foundation for

Medical Research

BoD Board of Directors Raad van Bestuur Conseil d’Administration

FPS Federal Public Service Federale Overheidsdienst (FOD) Service Public Fédéral (SPF)

IMA-AIM Common Sickness Funds Agency Intermutualistisch Agentschap

Agence Intermutualiste KCE Belgian Health Care Knowledge

Centre Federaal Kenniscentrum voor de Gezondheidzorg Centre fédéral d’expertise des soins de santé

FPS Health Federal Public Service Health, Food

Chain Safety and Environment Federale Overheidsdienst Volksgezondheid, Veiligheid van de Voedselketen en Leefmilieu Service Public Fédéral Santé publique, Sécurité de la Chaîne alimentaire et Environnement

FPS Social Security

Federal Public Service Social Security Federale Overheidsdienst Sociale Zekerheid Service Public Fédéral Sécurité sociale

GCP Good Clinical Practice HERG HSR HTA

Health Economics Research Group Health Services Research Health Technology Assessment

NCCSDO National Co-ordinating Centre for the

SDO Programme

NIHDI National Institute for Health and Disability Insurance

Rijksinstituut voor Ziekte- en Invaliditeitsverzekering (RIZIV) Institut National d’Assurance Maladie-Invalidité (INAMI)

PPF pre-project form

SDO Service Delivery and Organisation

TPF topic proposal form

ZonMw the Netherlands Organization for Health Research and Development

Nederlandse organisatie voor gezondheidsonderzoek en zorginnovatie

5

1 INTRODUCTION

1.1 ORGANIZATIONAL OUTLINE OF KCE

1.1.1 Mission, objectives and tasks

The Belgian Health Care Knowledge Centre (KCE) was constituted by law in 2002, and came into being in 2003. It is set up as an independent institution responsible for producing analyses and studies to facilitate decision making and policy development in three health care research fields.

The mission, activities and financing of KCE are defined by law. KCE’s objectives as put down in this law are “the collection and dissemination of objective elements resulting from processing registered and validated data, health economic analyses and all other information sources in order to qualitatively support de realization of the best possible health care and to allow for optimal transparency in the allocation and use of the available resources from health care insurances by authorized organizations, taking into account the accessibility of health care for patients and the objectives of health care policy and health care insurances”.i

Official tasks of KCE are:

(Facilitate) making quantitative and qualitative analyses and developing a coherent data model for doing so, using already collected information and information made available to KCE with the aim of supporting health care policy.

Providing this (anonymous) information and data to other parties.

Collecting and disseminating data and information of scientific origin concerning the evaluation of medical practice and the evaluation of health care technologies.

Collecting and analyzing information concerning choices about the allocation of available resources for health care.

Building a knowledge network containing experts from universities, hospitals, scientific associations of caregivers and the Common Sickness Funds Agency (IMA/AIM).

Developing and expanding expertise and know-how on the various terrains belonging to the mandate of KCE (HSR, HTA, GCP).

(Facilitate) making health-economic analyses.ii

KCE is legally obliged to perform studies for at least the following governmental institutions: NIHDI, federal public services and Ministersiii.

1.1.2 Main research fields

The three main research fields for KCE are: iv

Good Clinical Practice (GCP)

Health Technology Assessment (HTA)

Health Services Research (HSR)

Good Clinical Practice is aimed at providing a framework or guidelines for clinical practices in line with (international) quality standards, at acceptable financial costs.

Health Technology Assessment critically evaluates the added value of (new) technologies, to ensure that innovation will result in high quality products that are cost-effective and acceptable in daily medical practice.

Health Services Research involves analysing the organisation and financing of health services in order to ensure that health care will remain affordable and

i art. 2621 ii art. 2631 iii art. 2641 iv Until July 2008, the KCE issued reports in two additional fields: Equity and Patient Behaviour (EPB) and

Objective Elements - Communication (OEC). KCE report 192, 503 and 804 belonged to the EPB field, while KCE reports 305 and 416 were issued under the label OEC. In 2008, the KCE decided to limit the number of research fields to HSR, GCP and HTA by issuing future EPB and OEC reports under the HSR label. The same decision was taken for Phase 2 of this study: the three EPB and the two OEC reports were categorized as HSR report.

6

accessible. The recently added field of Equity and Patient Behaviour focuses specifically on questions of accessibility to high quality care.

1.1.3 Organizational structure

KCE has about 40 FTE personnel. There are 50 employees in total, of whom 36 (or 72%) are experts. Besides the chairman and three directors, KCE has 11 staff members and administrators7. All KCE-experts have a university qualification and most have obtained a doctorate degree. KCE experts may combine their function at KCE with activities at a university, hospital or clinical practice, provided that any potential conflict of interest is precluded. The combination of functions enables the experts to remain in touch with the practice of their field of expertise. KCE encourages continuous education and specialization and regularly organizes courses and seminars for her employees. KCE experts may dedicate 15 % of their time to courses, self-study and writing scientific articles. The experts also share their expertise externally, by lecturing and tutoring at universities (K.U. Leuven, UCL, ULB, VUB), university colleges (HUB, Instituut Cardijn) and scientific institutions and societies.8

1.2 REQUEST FOR IMPACT ASSESSMENT Five years after KCE started its first activities as centre of expertise, it requested an independent study into the amount of impact their activities have had until now, and into the factors that may have influenced the degree of impact achieved. Also, KCE wanted to know what lessons could be learnt from their efforts so far.

Therefore, KCE formulated a special research project within its regular program to have its impact measured. The aim of the study as formulated by KCE: “to study the impact that KCE reports might have (had) on policy making and on decision making in Belgian health care. This study aims at a qualitative and a quantitative analysis of this impact”.v

The project was to address three general questions, dealing with:

a) methodological issues

to analyze the available methodologies to measure the impact of health research programmes. This wasn’t a systematic review, but an inventory of published literature on impact measurement (published since 2006).

to consider what models are available to measure impact – and their strengths and weaknesses

b) measuring the impact of the KCE studies

to assess the impact of the KCE health research reports, published since its inception in 2004

to what extent, and in what ways, are KCE health research reports used in health policy decision-making / clinical decision making in Belgium

to identify factors that are associated with the utilizations (or non-utilization) of the research findings in the particular Belgian health care policy context

to compare the impact of KCE health research to the experiences of other agencies in Europe

c) possible improvements for the future

to explore with the different stakeholders how the impact might be increased, in particular in relation to the three research domains of the KCE

to make recommendations for improvements in the use of KCE reports by decision makers in Belgium

January 2009, Research voor Beleid was assigned by KCE to execute the impact assessment in the course of 2009, and present its findings ultimo 2009. The report in hand presents the final results of this study.

v KCE, Pre-Project Form Onderzoeksprogramma 2009, 2009-55, p. 2.

7

1.3 SCOPE AND LIMITATION OF THIS IMPACT STUDY For the understanding of the results of this impact study, it is important to consider its scope and limitations:

First of all, it is important to note that this impact study is not an assessment of KCE as an institute. The focus is mainly on KCE research projects. It is a process oriented approach in the sense that it studies the process of KCE’s research agenda setting, the research process, dissemination and a general impression of the use of research findings.

Second: the scope of the study is limited to those aspects directly influenced by KCE. We specifically focused on the ways in which KCE created the conditions to enhance the impact of its research findings. Also, albeit to a lesser extent, we gained insight into the perception and image of KCE and its role in Belgian health care.

Third: the study is aimed at identifying possible improvements at KCE. This implies that we elaborate in more detail on opportunities to learn and less on what KCE already achieved. This might come across as a fault finding and critical approach.

Fourth: this impact study is, with exception of a self report questionnaire among project coordinators, qualitative in nature. Core in the data collection in this impact study are interviews, in which different types of respondents, from different angles, give their own vision on KCE and its impact. In this report we reflect those views. Perhaps needless to say is that these respondents often represent interest groups, which are involved in Belgians policy arena. All opinions should be read against this background.

1.4 STRUCTURE OF THE REPORT First step in conducting the KCE impact study was a literature review on existing approaches to study the impact of health research programs. The findings from this phase are presented in Chapter 2.

From the review we constructed a framework for the KCE impact study. The framework, the choices made in coming to this framework and the different research phases are presented in Chapter 3.

The results of the impact study are presented in Chapter 4. This chapter consists of five paragraphs:

4.1. Origin of the research projects

4.2. Research process

4.3. Dissemination

4.4. Contribution to the decision-making process

4.5. External image of KCE

Each paragraph contains different subparagraphs, addressing various aspects of the central subject. Each subparagraph is concluded by key findings, summarizing the most important facts. The research team has aimed to make the structure of each subparagraph as uniform as possible, using the following arrangement:

1. Procedures on the relevant subtopic (addressing any existing internal procedures)

2. Internal view on the relevant subtopic (addressing KCE experts’ and management opinions)

3. External view on the relevant subtopic (addressing stakeholders’ opinions)

4. Reported findings on the relevant subtopic (addressing findings from the project coordinators’ self reports).

However, sometimes not all data sources provide information on the specific subtopic causing some parts to be left out. This especially applies to paragraph 4.4 because it addresses results rather than processes. Paragraph 4.5, in its nature, consists entirely of external views (opinions from stakeholders) and therefore does not address procedures, internal views or reported findings.

Finally, in Chapter 5 we arrive at the conclusions from the impact study. Recommendations to KCE are made separately from this report.

8

2 LITERATURE REVIEW ON MEASURING IMPACT

2.1 INTRODUCTION The first phase of the impact study consisted of a literature review. The aim of this review has been to identify the most appropriate methodology for conducting the KCE impact assessment.

Research questions used for this literature review were:

What are the existing methodologies for measuring the impact of health research programmes?

What are the strengths and weaknesses?

What is the most appropriate methodology for this study?

In consultation with KCE management, several criteria were formulated to guide the literature review. These criteria were primarily based on the type of findings KCE found most usefulvi:

1. How much impact did the projects have and what factors in KCE’s sphere of influence have affected the amount of impact their projects have had?

2. To what degree and in what way did KCE support policy makers in making decisions (by providing them with valid and essential information)?

3. What (different) type(s) of impact did KCE have?

4. How do different stakeholders perceive KCE and value their achievements and services?

Therefore, in the literature review we took the following core characteristics as indicators for the usefulness of the methodology:

- Encompassing the entire process from programme objectives to the utilization of research knowledge produced by the research programme

- Defining factors associated with impact in each stage of the research process

- Identifying different stages or types of impact

- Focus on impact as influence on policy processes and decision-making

- Identifying different target groups (with different needs)

During the review process, the research team concluded that the methodologies found proved to be too specific (concerning e.g. scope, local context, available budget, subject) and at the same time insufficiently worked-out to be easily adopted as a blueprint for the way in which KCE’s impact was to be assessed. The research team thus considered it undesirable to choose one ‘complete’ methodology.

Therefore, the research team somewhat changed the aim of the literature review and decided to look for and distil from the literature found those insights, theories, hypotheses, variables and/or instruments useful for the development of a research framework tailor-made for the specific context and demands of KCE.

This chapter describes the whole itinerary of the literature review:

Set-up

Search and selection procedures

Yield

Appraisal

vi This list can be seen as a simplified version of the research questions presented in paragraph

9

2.2 LITERATURE SEARCH This paragraph describes the way in which the research team searched for relevant articles, selected possible relevant articles for further reading and thereupon conducted a review of the articles leading to a final assessment of five possible methodologies to use for the impact assessmentvii.

Due to limited time and resources the process of selecting and reviewing the articles was done by one researcher and has not been validated internally or externally.

2.2.1 Searching relevant articles

The main starting point for our literature review was the literature review conducted by Stephen Hanney and others9. The Hanney study yielded a list of possible methodologies suitable for impact assessment. These methodologies are:

Payback approach

Monetary value approach

Non-academic impact of socio-economic research

Societal impact of applied health research

Impact of health research on policy (target-audiences)

Organizational theory approach

HTA policy impacts and costs savings

Knowledge utilization model

Levels of impact model

The monetary value approach was discarded as too narrow and not complying with the core characteristics (see paragraph 2.1.).

‘Quick’ search

For each remaining approach a reference list with ‘key-publications’ was made according to Hanney’s source citation. This resulted in 11 articles selected for further reading.

We then used the Science Citation Index in order to identify which published articles had cited these key-publications. Titles and/or abstracts from the articles thus identified were scanned to select those possibly relevant for the purpose of our literature review. This resulted in 15 articles selected for further reading.

Extensive database search

To find as many articles as possible that might contain impact assessment methodologies meeting the set criteria, the research team searched multiple bibliographic databases. After consultation with KCE, the following databases have been searched: EMBASE, Medline, Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, Sociological Abstracts, PsychINFO, EconLit.

For each database a different search strategy was constructed. In doing this, we took into account the following factors:

Database-specific instructions on using Boolean operators, wildcards and truncations, field terms and exact phrasing.

Availability of database-specific descriptors or key-words that were suitable for our search objectives

The terminology used in a specific scientific field (e.g.: when searching sociological databases we chose those words that were frequently used in the sociological field to denote research impact)

Aiming at less than 200 search results in order to get as narrow results as possible

Narrowing down our search to journal articles only (in EMBASE this was ‘translated’ into limiting the results to the availability of abstracts).

A time-span from 1995-2009, to allow recent as well as older (but not too old) publications to come into the final selection.

vii In searching for relevant literature, we only took into account literature written in English.

10

The process of constructing fitting search strategies for each database was an iterative process, in which we tried multiple combinations of words in different positions, and tried several field tags. Eventually this resulted in the search strategies elaborated in the template (see appendix 1). Below, each search strategy is being clarified.

EMBASE – we searched for important keywords that are regularly used in articles related to impact assessment, and also for multiple EMTREE terms. These are used as major headings or descriptors, because they are considered as focal subjects in our search. We limited the search to the years 2006-2009 because this search functioned as a miniature ‘update’ of the Hanney study, in addition to phase B.

Medline - the strategy for Medline is similar to the one used for EMBASE, only EMTREE terms were ‘translated’ into MeSH-terms.

Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index – we searched for relevant words in either title and/or abstract and/or author keywords and/or ‘keywords plus’. Diffusion and dissemination were added because these are widely used in the sociological field as synonyms for words like impact or utilization. The concepts decision-making and policy making are used as substitution for descriptors like ‘health care policy’, used in EMBASE.

Sociological Abstracts – we searched for relevant words in either title, abstract, descriptor, and/or identifier fields. We added descriptors to the strategy that covered the principal focus of our search for a methodology, in order to narrow the results. The ‘WITHIN 3’ and ‘NEAR’ operators are used to search for articles in which the respective search terms are likely to be interrelated, but do not necessarily have to be positioned right next to each other. We considered this necessary because the scope of the literature covered by this database is reasonably large en differentiated, and narrowing down too much was not a good option.

PsychINFO - The strategies used for SSCI and Sociological Abstracts were adapted for use on PsychINFO, altering a few keywords to get better results, and choosing descriptors most closely related to the ones used in Sociological Abstracts.

EconLit - We tried finding similar descriptors as used in Sociological Abstracts and PsychINFO, but unfortunately EconLit defines no such descriptors. Therefore the search strategy was somewhat altered and extended with synonyms. Furthermore, EconLit does not allow to simultaneously search in title, abstract and descriptor terms. The closest option was ‘text search’, which we consequently used.

The database search amounted to 31 additional, unique articles selected for further reading.

Expert consultation

In addition to our search activities, we asked an external expert (Dr. John Ovretveit)viii for input. He received the research proposal for the KCE impact study and we asked him to send us documents or articles regarding impact assessment and/or containing methodologies other than the ones identified in the Hanney study. This resulted in an additional 5 unique documents about impact assessment, suitable for further reading.

Result

In total, the search for relevant articles rendered 62 articles which were selected for further reading. The research team judged the quality and quantity of those articles as satisfactory and sufficient, considering the purpose of the literature search and the time and resources available.

The results were discussed with an internal expert panel. Members of this panel are Prof. Dr. van Hoesel, drs. Tom Weijnen, drs. Ir. Félicie van Vree and François Daue. The internal panel reflected on the research approach and the intermediate findings (fact sheets) and discussed ways to come to a main framework to asses the impact of the KCE studies.

viii Director of Research, Karolinska Medical Management Centre Sweden and Professor of Health Management, The Nordic School of Public Health, Sweden, and Faculty of Medicine, Bergen University.

11

2.2.2 Selection of articles using inclusion and exclusion criteria

From the 62 articles thus identified, 32 actually described a methodology or a conceptual framework for assessing the impact or utilization of research findings that fit one or more of the formulated core characteristics of a suitable methodology (see paragraph 2.1.).ix

These articles could be arranged into four categories:

1. Articles describing an application of the payback approach sometimes modified (7 articles)

2. Articles describing a methodology, other than the payback approach (14 articles)

3. Articles describing a conceptual framework not further developed into a directly applicable methodology (8 articles)

4. Articles containing ‘loose’ elements which focus on specific issues that might be fitted into another conceptual framework or methodology (3 articles)

2.3 REVIEWING THE ARTICLES

2.3.1 Developing fact sheets

Fact sheets were developed for all articles in category 1 and 2 (21 in total) and are available in Appendix 2. This was done to summarize the articles, keeping only those aspects that might be relevant to use for the KCE impact assessment framework. Possible relevant elements of the articles in category 3 and 4 were put into separate documents so as to be easily retraceable for possible use in a later stadium. Detailed information on this is available in Appendix 1.

2.3.2 Selecting possible methodologies for KCE impact assessment

A summary of the information contained in the fact sheets is given in table 2.1. The columns show all articles reviewed and the rows display all variables by which the methodologies have been judged. The upper half (research phases and focus in data collection) of the variables is normative: the more check marks, the more suitable the methodology. The lower half of the variables is more descriptive: the check marks indicate how data has been collected and for what purposes the methodology has been used. For the further reviewing of strengths and weaknesses, we selected all methodologies that have:

a check mark at the ‘outcome/impact’ field

a check mark at the ‘research(ers)/institute’ and ‘end-users’ field

a minimum amount of six check marks in the fields under ‘research phases’ and ‘focus in data collection’

a minimum amount of three different data collection methods used

The methodologies thus selected are described and used by the following authors:

1. the NHS Service Delivery and Organisation Research and Development Programme (SDO) 10

2. Oortwijn et al., 2008 11

3. Hanney et al., 2003 12

4. Meagher et al., 2008 13

An exception is made for the methodology of the Alberta Heritage Foundation for Medical Research (AHFMR) : as one of only two methodologies it covers all the upper half variables, but uses interviews as data collection method only. Although this feature renders it unsuitable for an overall impact assessment, it might provide a useful basis for case studies (in which interviews are usually the main or sole data collection method used).14

ix The selection process was conducted without involvement from or consultation with KCE, because KCE had

already been involved in formulating the core characteristics of a suitable methodology - which formed the main basis for the selection.

12

Table 2.1: Summary of fact sheets

SD

O,

year

unkn

ow

n 1

0

AH

FMR,

2003 1

4

Han

ney

et

al.,

2007 9

Raf

tery

et

al.,

2009 1

5

Oort

wijn e

t al

., 2

008 1

1

Kw

an e

t al

., 2

007 1

6

Woodin

g e

t al

., 2

005 1

7

Han

ney

et

al.,

2003 1

2

Buxt

on e

t al

., 2

000 1

8

Mea

gher

et

al.,

2008 1

3

Tet

roe

et a

l.,

2008 1

9

Ott

oso

n e

t al

., 2

009 2

0

Landry

et

al.,

2001

21

Car

den

, 2004 2

2

Hiv

on e

t al

., 2

005 2

3

Gag

non e

t al

., 2

008 2

4

Gag

non e

t al

., 2

006 2

5

Bodea

u-L

ivin

ec e

t al

.,

2006 2

6

Str

yer

et a

l.,

2000 2

7

Mola

s-G

alla

rt e

t al

., 2

000

28

Jaco

b &

McG

regor

, 1997

29

Research phases: Input ? Activities ? Output Diffusion/dissemination ? Outcome/impact Focus in data-collection: Research(ers)/institute Dissemination activities ? End-users of research ? Data-collection methods: Questionnaire (web/postal) Interviews (telephonic/f-t-f) ‘Social analysis’ Document analysis Media search (also bibliometric) Observation Focus groups/workshops Scoring on a scale Case-studies used? Research programme as subject Application level: overall Application level: case-studies

13

2.4 STRENGHTS AND WEAKNESSES OF SELECTED METHODOLOGIES After having selected five methodologies, the research team conducted a more in-depth analysis of the methodologies’ strengths and weaknesses. We also considered whether a model’s weaknesses could be compensated, for example by using aspects of other methodologies. This way the research team planned to construct a suitable methodological framework to measure the impact of the KCE studies.

Although the analysis of strengths and weaknesses played important part in the process of coming towards a framework for the KCE impact assessment, none of the analysed methodologies formed the basis for this framework. The methodologies selected (with or without ‘repairs’) proved to be too specific (concerning e.g. scope, local context, available budget, subject) and at the same time insufficiently worked-out to be easily adopted as a blueprint for the way in which KCE’s impact was to be assessed.

Nonetheless, the research team and KCE chose to retain this paragraph in the final report because it constituted and important step and provides insight in the process leading up to the making of the framework. In paragraph 2.5 the process of coming towards a framework is shortly described, leading to chapter 3 in which the final framework is being presented.

2.4.1 Methodology of SDO

This study aimed to provide a descriptive account and summary assessment of the impact of the research projects commissioned between 2001 and 2006 by the NHS Service Delivery and Organisation Research and Development Programme.10

For this purpose the SDO devised an ‘Impact Analysis Framework’ (see figure 2.1). The payback model was considered to provide a useful framework and informed the organization of the data collected.

Figure 2.1: Impact Analysis Framework (SDO)10

14

From the 55 completed research projects with an approved final report, a sample of 23 projects was selected to demonstrate the range of outputs and impacts. The selection was made by the National Co-ordinating Centre for the SDO Programme (NCCSDO) academic team and consisted of projects spread across the SDO commissioning themes of which some evidence of impact was known to be available. Data collection was done in three phases:

1. Documentary review, literature review, bibliometric analysis and interviews for identifying primary outputs (outputs produced by the NCCSDO or the funded researchers). Data used: NCCSDO documents, NCCSDO monitoring data, NCCSDO staff, number of website downloads, location of externally published data using online databases and search engines, key journals, etc.

2. Bibliometric analysis for identifying secondary outputs (citations/mentioning of primary outputs in policy documents, practice guidance, newsletters, websites and the media). Data used: searches using surname and initial of PI and members of research team plus keywords describing the research. Verification of outputs as being related to the SDO (criteria mentioned).

3. Selection of 11 case-studies to supplement the findings and to provide insight in how knowledge generated by SDO-funded research is used. Data collection included semi-structured telephone interviews with the PIs of each study covering topics mentioned in the analysis framework. Also interviews with users of the research (identified through interviews with PIs, citations, NCCSDO database and publications, interviews with the SDO staff) by telephone or via e-mail.

Strengths Covers all phases in research and implementation cycle

Focuses on producers, dissemination activities and users

Size and duration of the evaluated research programme is similar to the KCE programmes

Discerns between primary and secondary outputs

Uses case-studies

Weaknesses Does not include an overall assessment of all research projects

commissioned, but selects projects according to expected impact

Involves producers only at case study level

Involves users only at case study level

It is possible to overcome the weaknesses by:

Selecting KCE studies by other criteria than in the SDO methodology.

Using for example a payback-questionnaire for all principal investigators in overall phase of the impact assessment.x.

Adding interviews in the overall phase of the study, based on the interview methods used in the AHFMR approach (see 2.4.5.).

2.4.2 Methodology by Oortwijn

The main HTA programme in the Netherlands is the Health Care Efficiency Research Programme, coordinated by ZonMw (the Netherlands Organization for Health Research and Development). The aim of this study was to assess the impact of all 43 research projects granted between 2001 and 2003. Most projects took the form of prospective clinical trials.11

The evaluation framework has been based on the HERG (Health Economics Research Group) payback framework and consists of two components:

1. A logic model of the research process, used to interpret the relationships between input, process and output.

2. Evaluation criteria for output and outcome of funded research. Payback criteria used: a) Knowledge production; b) Research benefits; c) Informing policy; d) Changing health practice; e) Broader impact on health.

x For the KCE impact study we proposed a phase where we would examine all KCE studies (overall) and

another phase were would study specific cases in depth.

15

Data collection methods used are:

Dossier review of all projects: reading of key documents including original research grant proposals, referees reports, final reports, information about the program (annual reports, minutes of committee meetings, etc.)

Postal survey based on HERG questionnaire (Hanney et al., 2007).9 Postal survey, including data from the dossier review, was sent to all PIs to collect the most recent data on their projects. Results were presented as standard one page summaries using a common structure.

Case study analysis in which a purposive sample of five HTA projects (that mirrored the diversity of ZonMw funding) were investigated in more detail to illustrate useful levels of payback. Three semi-structured interviews per case study were conducted (either face-to-face or by telephone) with the PI, at least one other researcher and users. In each case study initial and longer-term output and outcome of the research were explored in more detail, including the translation of the research, e.g. into practice. The interviews aimed to verify conclusions from written materials and to identify any payback that might have been missed.

Scoring of payback by HTA experts (in three rounds) from 26 HTA projects that had already led to a final report and for which survey results were available. Scoring on a scale of 1-10 (0,2,4, etc.) on sub-criteria of the five payback categories. Once the score (per payback category_ per sty was calculated, a comparison was made between the different subsidy rounds of the HTA program. Therefore average scores were calculated per category per round. An example of a resulting plot is shown in figure 2.2.

Figure 2.2: Example of scoring plot 11

Strengths

Covers nearly all phases in the research and implementation cycle


Uses case studies

Scoring method provides possibilities for benchmarking

Weaknesses In data collection no attention is paid to ‘input’ factors


Limited feasibility: number of projects commissioned/evaluated is smaller than the number of KCE projects to be evaluated

Scoring method might prove too elaborate considering time and resources available for the KCE impact study


Using data collection methods from other approaches. For example Carden, Lafortune, Bensing or from a part of the documentary analysis and semi-structured interviews by Hanney.12, 30, 31

Adding interviews in the overall phase of the study, based on the interview methods used in the AHFMR approach (see 2..4.5.).14

16

The third and fourth weaknesses are hard to overcome. It is possible to leave the scoring method out, but then we lose one of the strengths of the methodology.

2.4.3 Methodology by Hanney

Aim of this study was to evaluate the NHS R&D Implementation Methods Programme in terms of: a) quality of outputs; b) lessons to be learned about the communication strategy and the commissioning process; c) the benefits or payback from the projects. The Commissioning Group funded 36 projects in two commissioning rounds (1995 and 1997). In 2000, the programme was evaluated by a small-scale assessment of the impact of the IMP on policy and practice in the health service.12

A multi-method approach was chosen, using both quantitative and qualitative methods.

Documentary analysis: apart from a review of the published documents from the IMP. IMP’s files were also screened, looking for information about the scores given to proposals, potential users of the project identified from application forms and scores given to the final project reports.

Semi-structured interviews (mostly face-to-face, some by phone) were conducted with Chairs of the Advisory and Commissioning Groups, the Deputy Chair of the Commissioning Group and other members of both groups. The checklist was based on the documentary analysis and some early interviews. Prime focus was on the commissioning process.

Questionnaires were sent to all lead researchers of the 36 projects fundedxi. Its structure is based on the HERG payback framework and contains the following topics: knowledge production, training and further research, possible impact on health policy and practice, features of dissemination that might have contributed to their impact, role of the programme as a whole. Two reminders and a covering letter were used to heighten the response.

Questionnaires (electronic) were also sent to potential users of the research findings, who were identified via existing networks and the message board. The questionnaire focused on specific projects, asking potential users about whether they had heard of the projects, read the articles described in the abstracts attached, had already been influenced by the findings, etc. To heighten the response a postal version was sent to practitioners and researchers in the maternity care field.

Desk analysis in which searches in various databases were performed to make assessments of the quality of the research outputs. This included a citation analysis of journal articles in several databases and recording the journal impact scores. Journal impact score was recorded. The HERG Research Outputs Database was searched for IMP publications as well.

Strengths Covers nearly all phases in the research and implementation cycle

Focuses on producer and users

Involves potential users, not related to specific projects/cases

Uses four different data collection methods

Overall assessment is fairly extensive (no case studies to supplement findings)

Weaknesses No assessment of final (societal) outcomes was attempted


The fact that that no attempt was made to assess final societal impact might be a weakness. The focus of this impact study is more on intermediate outcomes.

xi A full copy of the questionnaire is available

17

2.4.4 Methodology by Meagher

This study aimed to assess the impacts of a group of responsive-mode projects and fellowships awarded in the field of psychology, funded by the UK’s Economic and Social Research Council (ESRC). The projects ended in the years 1998, 2001 and 2004. The study focused on measuring impact on public policy and public service users, and also focused on the most immediate channels through which research knowledge flows into policy and practice worlds.13

A framework of core questions was developed which aimed to assess user-engagement, actual impacts and impact-generating processes (see figure 2.3). These formed the basis for the interview topic guide and survey questions, and facilitated the integration of findings across methods in the final analysis.

A multi-method approach was chosen, using five data collection methods.

Content analysis of all issued funded research reports along with ‘rapporteur’ evaluations, focused on indications of connectivity or orientation toward users and on any indications of real or potential impacts. Particular attention was given to research objectives (oriented towards end-users?) and achievements (implementation of connection with end-users?). Reported activities, outputs, impacts, dissemination and audience descriptions were assessed for relevance to or connection with users. Each project was ranked for user connectivity on a five-point scale: 1) Exclusively academic approach; 2) recognition of possible relevance; 3) planned dissemination; 4) dissemination; 5) genuine engagement of users during the project.

Figure 2.3: Framework of core questions 13

18

Survey of 109 award-holders using a 4-page questionnaire (mix of Lickert-scale, pre-coded and free text response), about non-academic research impacts arising from the project, and activities undertaken possibly leading to impact.

Short survey of department heads in order to identify their contacts with user communities and the types of non-academic impact arising from their research, with the aim of placing the results of the other survey in the broader context of research practice.

Focus groups (2) consisting of 10 of (psychology) departments heads to provide further insights into the contexts within which connections with users might be made and how impacts might be generated. The focus groups also helped in identifying key users.

Semi-structured telephone interviews with four users, a senior researcher active in assessment across the field, a funder, and a knowledge intermediary in a key representative organization.

Media-related searches in which press releases from ESRC and the British Psychological Society were reviewed for stories that highlighted work by award-holders (not always related to the specific research project conducted by award-holders).

Case studies of four particular award-funded projects in four different research areas, specifically chosen to identify likely success-stories (selected on the basis of information from end-of-award reports and award-holder surveys). The case studies consisted of interviews with the principal researcher and one or two users. In 3 case studies also bibliometric methods were used to assess dissemination in grey literature and applied articles (search strategy linked to the name of the researcher).

The findings were synthesized into three main categories: 1) level of engagement with users and orientation toward knowledge transfer; 2) impacts; 3) processes, activities and roles leading to impacts.

Strengths Covers nearly all phases in the research and implementation cycle


Measures the researchers’ orientation towards end-users

Uses five different data collection methods

Uses case studies

Weaknesses In data collection no attention is paid to ‘input’ factors


Likely success stories only are selected as case studies


Adding aspect of other input-focused approaches focused. There are several options: Carden, Lafortune, Bensing or the documentary analysis and semi-structured interviews from Hanney.12, 22, 30, 31

Adding interviews in the overall phase of the study, based on the interview methods used in the AHFMR approach.14

In the KCE impact study we can select both successful and unsuccessful cases and adjust questionnaires.

2.4.5 Methodology by Alberta Heritage Foundation for Medical Research

The aim of this study was to assess the impact of the products produced by the HTA Unit of the Alberta Heritage Foundation for Medical Research (AHFMR) from 2001 to 2002. All projects (12) that had been publicly released and had a clearly identifiable requester, were included.14

The emphasis in the impact analysis is on utilization. Although it has been acknowledged that this is a limited definition of impact, it has also been stressed that utilization of research knowledge is a (necessary) precursor to many other impacts. The six main research questions were:

1. To what extent do HTA products meet the Unit’s goals and objectives?

19

2. To what extent do the Unit’s products provide advice that informs policy and decision-making in the health care system?

3. How do the Unit’s products provide advice that informs policy and decision-making in the health care system?

4. How satisfied are clients with these products?

5. What influences client utility and satisfaction?

6. What are the opportunities for improving the HTA Unit’s activities?

The research approach consists of interviews only, both with requesters/users and with researchers.

Interviews with requesters/users - 9 interviews were conducted with a total of 11 interviewees (where possible face-to-face, otherwise by telephone) with requesters, requesters/users and users of HTA products (reports) issued by the programme. Guides for interviews were developed from the existing HTA Evaluation Form, an internal questionnaire used earlier in similar research, the payback framework and the HTA Unit’s continuing review of the literature on HTA impact and research utilization. Separate guides were developed for requesters, requesters/users and users. Initial requesters were identified by the steering committee. The evaluator contacted each potential interviewee and determined whether they were a requester, a requester and user or just a user of the HTA product. Topics dealt with are:

Product need

Service provided

Utility of the report

Rating of the report

Utilization and function

Processes for utilization

Considerations for service improvements

Supplementary interviews with 3 staff members of the HTA Unit to elicit perspectives on internal processes. Topics dealt with are:

Role in relation to the report

Helping the requesters define need

Relation/contact with target audience (users)

Client follow-up

Perceived barriers to utilization

Personal learning

Strengths Covers all phases in the research and implementation cycle


Users: focus on if they use it, how they use it and how they rate the products

Producers: focus on relation with target audience

Weaknesses Uses interviews as sole data collection method


No assessment of final (societal) outcomes was attempted

The first weakness makes this approach unsuitable as core methodology, but it could be useful for case studies. It could also be useful to add this approach to other methodologies to question users.

2.4.6 Conclusion

All selected methodologies meet the basic qualifications, but all have certain weaknesses or limitations. Furthermore, the literature reviewed gives little or no explanation on how the data-collection and -analysis have been carried out. Therefore, not one is totally and directly suitable to be used as a framework for the KCE impact study. However, it might be possible to choose one core methodology

20

as basis for the methodological framework, and to complement this with aspects of other approaches. In most cases the weaknesses detected can be overcome by altering the research design and/or using aspects of other methodologies. The studies done by SDO, Oortwijn, Meagher and Hanney may provide suitable methodologies for this purpose.10-13

2.5 TOWARDS THE CONSTRUCTION OF A FRAMEWORK Originally, the research team considered best option was to choose one of these methodologies as a starting point for constructing a framework optimally equipped to answer the research questions at hand. The weaknesses were to be repaired by adding other elements from other methodologies and frameworks, some of which were mentioned in the first deliverable.

However, as the ideation process continued the research team decided to take a somewhat different approach. This decision was nurtured by the realization that choosing a ‘prefab’ methodology with some added elements would entail the risk of not leaving enough room for addressing all specific research questions at hand and incorporating into the research the specific context in and rationale in which KCE operates. This was a risk the research team did not want to take.

Therefore, the research team chose a different point of departure in constructing the research strategy. This point of departure consists of three dimensions:

1. KCE’s definition of impact

2. The rationale underlying the activities developed by KCE

3. The research questions formulated in our research proposal

4. From these three dimensions and the (broad) findings from the literature review, a set of hypotheses was formulated. Together with the research questions, the hypotheses served as a basis for building the research design. Each core question was then elaborated into different topics using the findings from the literature review. Subsequently, all available data sources and possible data collection methods were identified. Finally each topic was linked to at least one source of information. This was done on a ‘common sense’ basis: which information sources might be able to give us information on this specific topic? The scheme in appendix 4 gives an overview of the way each topic is linked to at least one source of information.

Ultimately, the findings from the literature review thus mainly served to construct the elements on which the data collection and analysis should focus instead of providing a ready-made methodological scheme.

21

3 FRAMEWORK FOR KCE IMPACT STUDY

3.1 STARTING POINTFOR DEVELOPING THE FRAMEWORK The point of departure in developing the framework to be used for the KCE impact assessment consists of three dimensions:

1. KCE’s definition of impact

2. The rationale underlying the activities developed by KCE

3. The research questions formulated in our research proposal

Below, each dimension is being elucidated.

3.1.1 Definition of impact

The mission of KCE is to support policy makers in Belgium in decision making processes concerning the health care and health insurance field by conducting scientific studies and providing reliable information.

KCE therefore defines the impact it is aiming at as the extent to which it will succeed in:

a well-funded and transparent way of research programming, fitting KCE’s official tasks;

conducting high quality research;

effectively transferring (actively and passively) the knowledge and research findings it has generated to the policy makers for whom it is meant.

The ultimate goal of transferring knowledge is the actual usage of this knowledge and research findings in policy developing processes and the eventual gains in terms of cost savings and improved health. In consultation with the KCE management it was nonetheless decided not to incorporate the degree of actual usage in measuring the impact of KCE. Reasons for this choice are that the management does not perceive this as KCE’s responsibility and that the degree in which knowledge is actually being used by actors in the field of health care is influenced by numerous factors outside KCE’s influence.

3.1.2 Model of the rationale underlying KCE’s activities

Figure 3.1 presents a model that reflects the rationale underlying the activities developed by KCE. It represents is a highly simplified, linear model of the process leading towards impact and the various parts influencing this process. The figure shows multiple elements of different sorts:

1. Actors (rectangles)

a. KCE

b. groups of stakeholders

2. Actions (arrows)

a. proposal selection

b. project preparation

c. passive dissemination

d. active dissemination

e. decision making (based on research)

f. bilateral contacts

g. submitting research proposals

3. ‘Tangible’ results (ellipses)

a. research proposals

b. research projects

c. policy decisions (by actively disseminating its reports, the KCE can possibly influence policy making)

22

Figure 3.1 Model of the rationale underlying kce’s activities (next page).

KCE

RESEARCH PROJECTS

STAKEHOLDERS (direct and indirect)

HEALTH CARE POLICIES

pro

ject p r e pa r at io n

dissemina

tion

de cis ion

m

akin g

RESEARCH PROPOSALS

proposal se

lection

? ?

Minister Soc.Zaken en Volksgezondheid

Pharmacies

FOD Volksgezondheid, Veiligheid van de Voedselketen en

Leefmilieu+ commissiesRIZIV + Committees

Mutualiteiten Intermutualistisch

Agentschap

Care providers

Patient organisations

Universities

Industry

Focus of the studyB

ilate

ral c

ont

acts

(e.

g. i

nvita

tion

s to

writ

e ce

rta

in p

ropo

sals

)S

ubm

itting re

search

proposa

ls

23

ASPECTS OF THE RATIONALE UNDERLYING KCE’S ACTIVITIES

The rationale underlying KCE’s activities is made up of three basic ‘aspects’: a basic notion, a way of thinking and a fact. Each is being clarified in the text below.

The first aspect is a basic notion underlying this and other impact assessments (and the literature review as well). The process-oriented outlook of among others the payback-approach starts from the notion that there is a causal relationship between the degree of successfulness of different phases in the process leading to impact. This means that if one phase in the process is successful, this enhances the probability of the following phase to be successful, and so on. So the proper functioning of one element is considered to be an important factor for the proper functioning of following elements in achieving impact on policy. This doesn’t mean that a project will never have any impact if elements are missing or that a project will always have impact if all elements are successful, but the probability will be lower or higher.

The second and third aspects are both closely interrelated with the KCE management definition of impact and KCE’s influence on the actual usage of findings. The rationale underlying KCE activities supposes a ‘weakening’ of the influence from KCE occurs down the line (visualized in the figure by the arrows getting smaller and paler): KCE’s influence on achieving impact is biggest during the proposal selection and gradually declines and gets more diffuse during project preparation, dissemination and actual decision making. This is due to the growing influence of other factors on the actions, actors and results. This is partially caused by dynamics within individual projects (i.e. related to the professional autonomy of researchers and the engagement of external parties) but also by external influences on the dissemination process and the way in which stakeholders deal with KCE findings.

The third aspect of the rationale is the fact that health care policies are not exclusively informed by decisions that are made by stakeholders based on research. Other factors that have a part in this are decisions made by stakeholders based on other grounds, such as the political climate or ideological views. Also, policies are being influenced by other actors and their actions.

3.1.3 Scope of the study

The consequences of the formulated definition of impact and the rationale underlying KCE’s activities are that the scope of the study should be limited to those aspects directly influenced by KCE. These aspects are indicated in Figure 3.1 by the boundary called ‘focus of the study’. The aspects are:

1. research proposals (result)

2. proposal selection (action)

3. KCE (actor)

4. project preparation (action)

5. research projects (result)

6. dissemination (action)

7. external stakeholders (actor)

External stakeholders in this study are defined as a person or an organization that has direct or indirect stake in KCE because it can affect or be affected by KCE’s actions, objectives and policies (see figure 3.1). We consider two groups of external stakeholders:

1. Key stakeholders: NIHDI, Federal Public Services and Ministersxii. These are key stakeholders because KCE is legally obliged to perform studies for at least these governmental institutions.

2. Other stakeholders are for example: care providers (hospitals, pharmacies, etc.), health care professionals, health care industry, insurers and universities. For the most part we focused on organisations presenting these groups (e.g. the association of general practitioners, of specialists and of pharmacists).

xii art. 2641

24

3.1.4 Research questions

With the findings from the study the research team will answer the following research questions, which are based on the Pre-Project Form, the proposal from Research voor Beleid, insights from the literature studied and consultation with KCE management.

OBJECTIVES AND RATIONALE

1. What is the rationale behind the studies performed within the KCE programmes? (topic selection, project selection, formulating research questions)

2. What was the intended impact of the KCE studies / reports?

3. To what extent do research questions anticipate the use of outcomes in policy and practice?

4. How do the studies relate to KCE objectives and to broader policy developments in the Belgium health care sector (relevancy)?

DISSEMINATION OF RESEARCH RESULTS

1. What types of research results is KCE aiming to disseminate? (research outcomes in terms of products, insights, recommendation, policy ideas, guidelines, process experiences, etc.)?

2. What are the intended target groups for these research results?

3. What is KCE’s (perceived) responsibility in disseminating research results? What other parties are involved?

4. What strategy is formulated for transferring the outcomes of research to relevant target groups?

5. What resources (financial, human and organizational) are dedicated to dissemination activities?

6. Have the dissemination strategies been implemented properly? If not, why?

7. Did the dissemination activities lead to the expected output? Which target groups are reached and which are not?

8. What are the success and fail factors for disseminating the outcomes of the KCE studies?

OUTCOME/IMPACT

1. In what way did the dissemination activities actually influence target groups’:

a. Knowledge?

b. Attitude?

c. Behaviour?

2. Did the KCE studies actually influence policy makers and decision makers in the Belgian health care sector? If so, how and why? If not, why?

3. What factors are associated with the utilization of research knowledge?

4. How do other agencies in Europe aim for impact?

25

3.2 SET OF ‘HYPOTHESES’ GUIDING THE IMPACT ASSESSMENT On the basis of the broad findings from the literature review and the specific scope of this research (point of departure), the research team has formulated a set of basic ‘hypotheses’. The hypotheses serve as a basic guide for the impact assessment: they mainly describe impact-enhancing factors, still within the field of influence of KCE itself. Because they are enhancing factors, they also form a ‘predictor’ for impact at the end of the line (e.g. impact on policy making, impact on health care). So in order to find out if KCE has done all in its power to enhance impact, we have to look precisely at those variables. For that reason the hypotheses provide a solid basis for the conceptual framework, research design and even data-collection methods.

Hypotheses are:

1. Carrying out a transparent, careful proposal selection procedure that is primarily aimed at assisting decision making in health care, and that will increase the chances of research findings being accepted and usable.12

2. Formulating in advance a specific and (stakeholder) relevant goal for a research project will improve the chances of the final product being usable for stakeholders.32

3. Formulating explicit goals at the start of a project for the involvement of users during the project and for dissemination of the findings will increase the chances of the final product being effectively disseminated among stakeholders.12, 32

4. Performing high quality research processes (project management and methodology) will increase the chances of achieving the formulated project goals for a specific research project.32

5. Actively engaging relevant stakeholders during a research project will improve usability and acceptance of findings and, subsequently, the chance of implementation.10, 32

6. Making clear the relationship between research findings and the decision-making processes will improve the chances of the final product being usable for stakeholders.32

7. Actively performing dissemination activities (based on a dissemination strategy) after a project has finished will increase the chances of the findings actually being used by stakeholders.12, 33

8. Identifying stakeholder(s) as possible user(s) of a specific product and tailoring this product to the characteristics and needs of this/these stakeholder(s) will increase the chances of the product actually being used33.

9. Structural feedback from (possible) end-users on usability of finished products and incorporating lessons from this in the research process will increase the chances of improved usability. 10

10. A positive image of KCE (e.g. on independence, responsiveness and research quality) amongst stakeholders, will increase the chances of the findings from individual research projects being recognized, acknowledged and used by these stakeholders.33

26

3.3 RESEARCH DESIGN

3.3.1 Introduction

This paragraph presents the analysis framework constructed for this study, consisting of four main themes which have been derived from the research questions. Figure 3.2 gives an overview of the four main themes.

Figure 3.2 Impact assessment

I

Research process

II

Dissemination

III Contribution to the decision making

process

IV

External image of KCE

- Origin of the research

- Quality

- Involvement of stakeholders during research process

- Impact intended

- Types and number of outputs produced

- Dissemination activities performed

- Stakeholders addressed by outputs

- Engagement of stakeholders in outputs

- Actual take-up of outputs

- Familiarity with research projects

- Research utilization

- Relevance of projects for policy making / practice

- Reactions on finished projects from stakeholders

- General reputation of KCE

- Judgment on responsiveness

- Judgment on research quality

I – Research process consists of themes referring to KCE’s efforts during the research projects (proposal selection, quality of the research, involvement of stakeholders during the project and the impact intended).

II – Dissemination involves themes referring to KCE’s dissemination activities after finishing the research project and the reach of these activities.

III – Contribution to the decision making process consists of themes referring to the way in which and the degree to which findings from research projects are being received and used by stakeholders.

IV – External image of KCE consists of themes referring to the way in which KCE is being perceived by stakeholders.

Also, this paragraph presents all research phases and the data sources, data collection methods and research instruments to be used in each phase. The study consists of three phases:

Phase 1: The research institute

Phase 2: Research projects

Phase 3: International comparison with other agencies

The data sources and data collection methods used for this impact were chosen by asking ourselves the question ‘which of the available information sources might be able to give us information on this specific topic?’ For each topic, all possible data sources have been used. The research instruments used have been specially made by the research team for this project, using input from the literature studied (sometimes instruments such as questionnaires had also been publicized) and our own methodological knowledge and expertise.

27

3.3.2 Analysis framework

The analysis framework contains all topics that are to be examined in this impact study. All topics have been elaborated into different criteria (sometimes linked to specific sub-topics). The analysis framework is linked closely to the research questions and the hypotheses formulated.

This framework serves as the main guideline for devising the research instruments, collecting all relevant data and analyzing the results from the data collection phase. Below, all topics and criteria for each main theme are shown. Appendix 3 provides more detail on these topics (data sources and collection techniques).

I – The research process: from proposal selection to research conclusions

Origin of the researchxiii Proposals submitted

- Typology of stakeholders - Typology of subjects

Proposal selection and research prioritization

- Transparent? - Careful? - Policy- or practice-oriented?

Qualityxiv Project management - Focused on usefulness of outputs/results? Methodology - Scientifically based?

- Adequately supervised? Involvement of stakeholders during research processxv

- Before, during and at the end of the project? - Different types of involvement? - High degree of involvement?

Impact intendedxvi - Explicitly formulated in advance? - Primary target audience(s) aimed for - Type of impact aimed for (e.g. instrumental/ conceptual/ symbolicxvii; pre-policy/ developing policy/ policy enactment/ post-policy enactmentxviii)

II - Dissemination Types and number of outputs producedxix

- Reports - Recommendations - Journal articles - Citations - Presentations - Media coverage

Dissemination activities performedxx - Tailored design of the products? - Developed ‘synthesis’ products? - Expanded contacts and working relationships with stakeholders? - Communicated findings to intermediaries? - Developed products easily obtainable? - Communicated on the right time (monitoring policy developments)?

Stakeholders addressed by outputsxxi - Type of stakeholders - Corresponding with research proposal and intended impact?

Engagement of stakeholders in outputsxxii

- Stakeholders involved in developing outputs? - Stakeholders involved in disseminating outputs?

Actual take-up of outputsxxiii - Amount of downloads per project - Number of requests for project information

xiii Derived from SDO10; Hanney et al. 200312 xiv Derived from SDO10 xv Derived from Meagher et al. 200813; Drummond et al. 200832 xvi Partly derived from Meagher et al. 200813 xvii Derived from Hivon et al. 200523 xviii Derived from Ottoson et al. 200920 xix Derived from SDO10; Hanney et al. 200312 xx Derived from McBride et al. 200833 xxi Derived from SDO10 xxii Derived from SDO10 xxiii This is not derived from the literature; however it is a valuable addition to the other indicators/variables (and

is relatively simple to look up for KCE).

28

III - Contribution to the decision-making process Familiarity with research projectsxxiv - Knowledge of relevant research projects among specific

stakeholders - Reading of project reports/articles

Relevance of projects for policy-making / practicexxv

- Timely delivery? - Answering to the information need? - Usable conclusions/recommendations (tailoring)?

Reactions to finished projects from stakeholdersxxvi

- Feedback to KCE: positive of negative? - Comments in media: positive or negative?

Research utilizationxxvii - Influenced by the findings? - Actually use of findings? - Type of use - Reasons for not using the project findings

IV - External image of KCE General reputation of KCExxviii

- Trustworthiness? - Authority - Independence? - Helpfulness?

Judgment on responsivenessxxix - Clear communication? - Timely responding? - Tailoring products to ‘customers’? - Active role of KCE in presenting recommendations? - Fulfilment of information needs?

Judgment on research qualityxxx - Competent researchers? - Good choice of methodology? - Good analysis of data? - Valid conclusions?

3.3.3 Research phases

Table 3.1 shows the different phases. Each phase focuses on different information sources, using specific data collection method(s) and analysis techniques. Phase 4 consist of an overall analysis

The research team used multiple sources to gather the information on the topics and issues presented in the previous paragraph. Table 3.2 gives an overview of the way each topic is linked to at least one source of information.

Table 3.1: Research phases Desk analysis KCE documents Group session KCE experts Group session KCE management team Interviews Board of Directorsxxxi

Phase 1: Research institute

Interviews Stakeholders Questionnaire Principal researchers Media searches Publications Internal monitoring Download counts

Phase 2: Research projects

Case studies Research projects Desk analysis Web, documents agency Media searches Publications on agency

Phase 3.International comparison

Input from interviews Agency staff Phase 4: Overall analysis Analysis by research team Data from the previous

phases

xxiv Derived from SDO10; Hanney et al. 200312 xxv Derived from Landry et al. 1999 xxvi Derived from SDO10 xxvii Derived from Hanney et al. 200312 xxviii This is not directly derived from the literature. However, KCE and the research team consider it a valuable

addition to the research design. xxix Derived from Landry et al. 1999; McBride et al. 200833 xxx Derived from Landry et al. 1999 xxxi Board of directors are interviewed as stakeholders. Additional questions on, for example, proposal selection

and alteration of recommendations will be asked. See appendix 7.

29

Table 3.2 Topics and data sources Desk

analysis of KCE docs

Group session KCE experts

Interviews Board of Directors

Interviews Stake-holders

Question-naire Self-reports

Citations Downloads counts

Case studies of research projects

1. Origin of research 2. Quality 3. Involvement of stakeholders

4. Impact intended 5. Outputs 6. Dissemination activities 7. Stakeholders addressed 8. Engagement of stakeholders

9. Take-up of outputs 10. Familiarity with projects 11. Research utilization

12. Relevance of projects 13. Reactions on projects 14. General reputation KCE 15. Judgment on responsiveness

16.Judgment on research quality

30

3.3.4 Phase 1: Research institute

Activity 1.1 Study of specific organizational issues

OBJECTIVE

The first part of the impact study consisted of mapping specific issues in the organizational structure and internal processes at KCE on institutional and programme level. This phase provided an overall insight into how KCE is organized and the way it operates, focusing on features or processes that might facilitate or hamper the eventual impact on policy.

The objective was to provide answers to the following questions:

1. What kind of research proposals did KCE receive during its existence? Which persons/organizations have submitted these research proposals?

2. What kind of topics remain underexposed in research proposals?

3. How does KCE value the research proposals received and what are the reasons for having accepted or refused these proposals?

4. How is the research process managed in terms of outsourcing, project coordination and goal attainment?

5. How does the project finalization process (formulating conclusions, recommendations and subsequent assessment and appraisal) take place?

6. Which activities does KCE employ to disseminate findings and recommendations (passive and active, formal and informal)?

DATA COLLECTION

Document analysis - The research team first performed an analysis of relevant internal documents. The final selection of documents was done in consultation with KCE. Appendix 5 shows a detailed list of documents used and questions that had to be answered by studying the documents.

Group sessions – The document analysis was followed by two group sessions. First with a selection of KCE experts based on their experience and representing all three fields. Second with KCE management team. The results of both group sessions were used to complement and modify the information gathered during the desk research. The set up of the meeting and a detailed list of the issues discussed can be found in appendix 6. The objective of the meetings was to discuss the results of the document analysis.

Both the document analysis and the group sessions focused on the following topics:

Origin of research

Quality

Involvement of stakeholders

Intended impact

Outputs

Dissemination activities

Engagement of stakeholders

Take-up of outputs

Reaction to projects

METHODOLOGY

For the document analysis we made a list of types of documents needed to collect data related to the topics mentioned (see appendix 5). KCE screened the list and added other useful documents. The documents needed were:

Found on the website of KCE

Sent the researchers by KCE on cd-rom

In the first group session 11 KCE experts participated. The session was chaired by one of the researchers; two others made notes during the meeting. The meeting took 2 hours. Questions were presented one by one on PowerPoint. The group of participants didn’t know the questions beforehand. They were asked to answer the question.

31

During the session the researchers checked for consensus or opposing opinions on the answers given. All participated actively and all questions for this group session were addressed and answered. (See appendix 6).

In the second group session three members of KCE management team participated. This session took place towards the end of the impact study, six weeks after the session with KCE experts. KCE management did know the questions beforehand. Two researchers were present. Both made notes and the interview was audio taped.

ANALYSIS

Document analysis

The research team conducted a close reading of all documents found and received. During reading, the list of topics and questions was held at hand. Every time the contents of a document referred to one or more of the topics and questions, all information useful for answering these questions was extracted. This extracting was done by listing the relevant information in a document analysis report. The result of this analysis was a report containing factual information on for example formal procedures on proposal selection, research process and research outputs. The procedure of extracting useful information was done by one researcher.

Analysis of group sessions

Both group sessions were audio taped. In addition two researchers made notes during the session. Afterwards one of the researchers made a summary of the audio tape, focussing on the opinions about topics mentioned under ‘data collection’. In the summary report, an indication has been provided regarding whether participants agreed or reached consensus on an issue, or whether a response concerned an individual remark. A second researcher has screened the summary, using the notes made during the meeting to see if there were differences with the interpretation of the first researcher. Since the accounts were concordant, no corrections have been made.

Activity 1.2: Study of KCE’s external image

OBJECTIVE

Apart from mapping the KCE as an agency supporting policy decision-making from the inside, it is also important to know how KCE’s target audiences perceive KCE and its contribution to the policy making process in general.

DATA COLLECTION

The research team conducted semi-structured face to face interviews with a purposive selection of 20 stakeholders of:

Government (Cabinet Social affairs & Health of Minister Onkelinx)

Federal Public Service Health

Federal Public Service Social Security

Sickness Funds (Nationaal Verbond van Socialistische Mutualiteiten & Alliance Nationale des Mutualités Chrétiennes)

Pharmacy (L'Association Pharmaceutique Belge/Algemene Pharmaceutische Bond)

Industry (Pharma.be)

Primary Care (Domus medica & SSMG)

Secondary Care – hospitals (Zorgnet Vlaanderen & Cliniques universitaires St Luc)

Secondary Care – specialists (ABSYM/BVAS & a specialist from UZ Leuven)

Private insurers (DKV Belgium)

National Institute of Health and Disability Insurance (RIZIV/INAMI)

Universities (Ghent University, Katholieke Universiteit Leuven)

Politician (member of Belgian federal parliament).

In consultation with KCE it was decided that patient’s associations are not represented in the stakeholders list. The reason for this is the high level of fragmentation in Belgian patient’s associations: all cover a specific group and not one is seen as being representative of patients in general.

32

As stakeholders, the sickness funds were considered to be the closest representative of patient’s interests.

The selection of stakeholders was done in consultation with KCE. A list of 40 possible respondents was compiled, with two representatives per stakeholder organization. One representative was pointed as preferred respondent and the other as possible replacement.

In total 20 persons were interviewed, of which 19 were primarily selected and one was a replacement

METHODOLOGY

The topics for the interviews corresponded with topics listed in paragraph 3.5.2. Appendix 7 provides a list of interview questions. After agreement on the questionnaire with KCE, the questionnaire was translated into Dutch and French.

The semi-structured and responsive character of the interviews allowed for other topics to be discussed during the interviews if judged relevant by the respondent or researcher. Also, there was room for commenting on specific projects if known to the stakeholder. The interviews have been carried out by one researcher in a standardized way using the list of predefined questions (with an extra set of questions for the Board of Directors. The interviews were not incremental in the sense that answers from previous interviews were not used to come up with new questions.

The researcher made notes during the interview using a predefined form. The interviews have not been put on tape. This information later served as a basis to draw up a confidential account of the interview. All interview accounts were sent to the persons interviewed, asking them for approval and corrections on any factual inaccuracies discovered.

ANALYSIS

All answers to closed questions as well as open questions (see appendix 7) were entered into SPSS.

For analyzing the answers to all closed questions the research team made use of descriptive statistics (means and frequencies). Second step was to roughly divide stakeholders in four clusters. We made these clusters based on the impression from the interviews (and single interview accounts) that there were clear differences between them on certain topics. To verify these differences we made cross tables for these clusters on:

External image (question 25 to 31: trustworthiness, degree of independence, serve ability, scientific standard, knowledge concerning health care matters, ability to make findings usable for policy and practice). A five point scale was used on each item (very high, high, average, low, and very low).

Proposal selection (question 6: is proposal selection clear and impartial: yes/no)

Question on frequency of contact with KCE (question 3 and 4. Frequency once a week, once a month, once in three months, once in 6 months, once a year, less).

Satisfaction with the way KCE communicates (question 16. Highly satisfied, satisfied, neutral, not satisfied, highly dissatisfied).

If new findings from KCE are monitored (question 17. Yes, no)

Clusters that were:

1. Government : Cabinet Social affairs & Health of Minister Onkelinx, Federal Public Service Health, Federal Public Service Social Security, National Institute of Health and Disability Insurance (RIZIV/INAMI), Politicians.

2. Insurers: Sickness Funds, Private insurers.

3. Associations of care providers and professionals: Pharmacy, Primary Care, Secondary Care – hospitals, Secondary Care – specialists, Industry

4. Other: Universities

We acknowledge that this is a rough division, with important differences between organizations within a cluster. Besides, the total number of respondents (20) is too low to do quantitative analyses. This is why we were very careful in using these

33

clusters for analysis and mainly used them in a descriptive way to indicate possible differences.

Finally the differences between clusters were discussed within the research team and verified whether or not they were confirmed in relation to other data sources (document analysis and case studies) and reported in the results (see overall analysis, paragraph 3.3.7).

For analyzing the open answers the research team used a qualitative method. Firstly, all open answers were collected from the interview accounts and listed. We used the summarized topics to:

a. Clarify closed answers, 85% of the open answers were used to clarify a closed question. The clarifications were used to illustrate closed questions.

b. Derive new topics from answers to questions that gave the respondent the possibility to come up with new topics (15% of the questions).

The clarification for the closed questions and the new topics were used in the overall analysis (see 3.3.7).

3.3.5 Phase 2: Research projects

Overall objective phase 2

The aim of phase 2 was to get an overview of the KCE studies in three fields (GCP, HSR, and HTA) and to get a clear impression of how findings and recommendations are disseminated and utilized. First, a general overview on all KCE projects was obtained by a web based self report (activity 2.1). Nine case studies were conducted to get a deeper insight into KCE research project (activity 2.2).

Activity 2.1 General overview of projects

OBJECTIVE ACTIVITY 2.1

KCE’s principal task and thus also its principal means to achieve impact is conducting research projects and publicizing the findings from these projects. Therefore this impact measurement also focuses on the research projects themselves, in order to provide insight into the way in which KCE has translated its mission into practice on a project level, and into the results of this strategy. The questions answered in this phase are:

What is the degree of involvement of target audiences during the research process?

What are the main characteristics of the issued KCE reports?

To what extent and in what ways does KCE translate the research findings (reports) into specific products and/or actions that support of policy development?

To which target audiences (or intermediaries) are these products and services transferred?

What methods does KCE use to transfer these reports and/or specific products to its target audiences?

During data collection and analysis we applied a subdivision following the principal field categorization used by KCE:

Health Services Research (HSR)

Good Clinical Practice (GCP)

Health Technology Assessment (HTA).

DATA COLLECTION

Self reports - The most important source for answering these questions are the project coordinators. They know best what activities were performed during and after the project, and what products were issued as a result of the project.

We asked internal and external project managers from all projects to make a self report about their project(s) by filling in an internet-based survey on:

All kinds target audience involvement before, during and after the project;

34

All tangible results of the research project (products);

All dissemination activities performed;

Any evidence of present or future impact on decision making processes;

Factors (if known) that have supposedly influenced or will influence this contribution positively or negatively.

The questionnaire consisted of a clear-cut and structured format to facilitate the handling afterwards. If projects were totally or partially outsourced, a questionnaire was also sent to the main researcher of the subcontracted organization. This provided insight into any possible differences in perspectives. A complete list of questions is in appendix 8.

The questionnaire was send to 66 external and 101 KCE project managers. In case of an outsourced project, an external and a KCE project manager were questioned. The response rate was 27,7% (33 projects) within the group of external managers and 72,3% (86 projects) within the group of KCE managers.

ANALYSIS

Data from the self reports were entered in SPSS. Open questions have been listed, post coded and categorized into answering categories.

Descriptive tables were generated (frequencies, percentages and average scores). Per question (for questions see appendix 8) we analyzed:

1. Total frequencies for all structured questions

2. Differences between internal and external project managers

3. Differences between domains (GCP, HTA and HSR)

4. Differences between years (2006, 2007 and 2008)

In the report, we have not commented separately on responses for projects that were approved in the years 2004 and 2005, because the number of responses was too low (max. 8 and 15 respondents respectively). When a question or category had 15 answers or less, it has not been separately reported on.

A report on the results of this phase was written and used for the overall analysis (see paragraph 3.3.7)

Activity 2.2 Case studies

OBJECTIVES ACTIVITY 2.2

This activity consisted of a multiple-case study. The case studies are an important part in the assessment of the impact. They provide an informative analysis. This provided insight into the different roles played by KCE, the researchers and the stakeholders, and the factors facilitating or hampering impact.

METHODOLOGY

Selection of case studies

In consultation with KCE it was decided to conduct 9 case studies, each taking four days of research (the maximum achievable within available time and resources). The case studies were done to detail and contextualize the main findings and provide more in-depth knowledge on variables. We did a five step stratified sample procedure for the case studies: (also see appendix 9 and 10).

1. Time frame: all reports published between July 2005 and July 2008.

2. Research domain: each KCE research field (GCP, HTA and HSR) had to be covered by 3 case studies.

3. Impact: high/average/low: KCE staff evaluated the impact of all reports published between July 2005 and July 2008. For this they received an online list (see appendix 9) with the projects, in which they gave their personal impression (what projects they considered to have high impact, average impact or low impact). For this purpose, the level of impact was defined as: ‘to what extent recommendations are translated into policy decisions, regulations, professional regulations, guidelines, etc’.

4. Nine cases were selected at random: per KCE research field 3 cases were selected, each with a different estimated level of impact (see 3).

35

5. Correction based on response: Some project managers left KCE. If former KCE employees couldn’t or wouldn’t cooperate in the case study, the case was excluded.

DATA COLLECTION

Designing the data collection protocol - The research team designed a data collection protocol (see appendix 10). The protocol contained the following elements:

1. A framework serving as a starting point for the collection of data with regard to the cases. In each case study we looked at a project from its inception to its finalization, subsequent dissemination and use (or disuse) of the findings (also see topics in paragraph 3.6.2). Compared to the other research activities (for example self report and interviews with stakeholders), the case studies explored the selected cases in depth and were more focused on finding contextual and explanatory information.

2. The data collection method to be followed, contained the following steps:

a. Check the filled in self report questionnaire (see paragraph 3.6.5) on the selected case to identify information gaps and additional data required.

b. Desk research on project

c. First contact with internal project manager

d. Interviews with other key respondents

The desk analysis was focused on a study of, at least, the following documents:

1. Final report

2. Dissemination materials

3. Publications on project

4. Proposal submitted

5. Feedback (if any)

6. Outputs in the form of directives, recommendations, etc (if any)

A stakeholder can also indicate interesting documents that are related to the case.

Finally, on the basis of the desk analysis a questionnaire has been constructed and several interviews took place (maximum four interviews per case study). The project manager has been approached in each case. Other (potential) respondents were:

1. Internal or external researchers participating in the project

2. External project coordinator

3. Stakeholder or representative of a target group in this specific project.

4. External validators

For the different types of respondents listed above, an interview protocol has been made (see appendix 11). The protocol was created on the basis of desk research and adapted as much as possible to each particular case study. A single case report was made for each case.

ANALYSIS

First step in analysing the case studies was to use the data within each case from the desk research and the interviews to summarize data on each topic in the research frame work. Although we gave an equal weight to each data source, we looked specifically for learning opportunities for KCE.

Second step was to report on each specific case.

Third step was to confront the case findings with the hypothesis (see paragraph 3.4.1). Based on this confrontation, we drew conclusions on the impact of each case.

In the fourth step we combined the findings of the nine case studies and undertook a general comparative analysis, based on the hypothesis.

36

With step 5 we looked at the impact of the different cases and what type of approach appeared to have generated higher levels of impact. The different chances and challenges concerned in each approach were evaluated.

Finally, the results of the single case studies have been used for the overall analysis (see paragraph 3.3.7).

3.3.6 Phase 3: International comparison OBJECTIVE

The third phase of the research consisted of an international comparison with other (national) research agencies in the field of health care policy. The objective of this phase is to give an extra perspective to the findings of this study by providing a rough benchmark with the results and lessons to be learned from other agencies. The focus is on successful tools, used by other agencies that can be translated into the KCE context. This international comparison does not provide a description of the context in which these agencies work. Main reason is that this would be a study in itself.

Designing framework for comparison - The data-collection started with devising a framework for comparison, containing a set of items on which KCE could be compared with the other agencies. This was laid down in a comparison framework (see appendix 13). The set of items was drawn up after phase 1 (Research institute) and phase 2 (Research projects).

Selecting other agencies – Three agencies were selected, taking into account at least the following criteria (also see appendix 14):

the agency is preferably established in a European country

the mission of the agency is comparable to that of KCE

the ‘age’ of the agency is somewhat comparable to that of KCE

the agency focuses largely on the same fields as KCE

an evaluative report about the agency’s impact is available, preferably providing information about (at least part of) the items in the framework.

DATA COLLECTION

Reviewing and comparing the agencies - The sources for data collection used were:

the internet (organizations’ websites)

requested documents from the agencies

literature (reports or articles concerning an organization’s impact)

telephone interview with a member of the agency’s management team

ANALYSIS

For the three selected agencies a semi structured questionnaire, with recurring questions, was used to carry out the data collection and to write the three reports on the case studies. As indicated above different data sources were used to answer the questions from the questionnaire. On the basis of these three reports a comparison was made between the three case studies along the lines of the analysis of the impact of KCE in this report. The topics in this comparison were: institutional design, objectives and rationale, proposal selection, research process, dissemination and contributions to decision making. For each of these topics a brief comparison is included between the main characteristics of the agencies with the relevant features of KCE as they were identified in the concluding sections of chapter 4 of this report.

3.3.7 Phase 4:Overall analysis

The analysis of the data has been done in three steps see Figure 3.3.

In the first step the focus was on synthesizing the data collected from the different sources (stakeholders, documents, KCE experts, external project managers, etc.). This step resulted in single reports and summaries per data collection method.

The core was derived from the framework described in paragraph 3.6.2. This means the summaries and intermediate reports had data on the origin of the research, research process, dissemination, contribution to decision making and

37

external image. Paragraphs 3.6.4 to 3.6.6 describe the way the analysis has been done.

In the second step the data from the summaries and intermediate reports of step 1 were listed for each topic of the framework. For example: on the topic ‘origin of the research’ we listed data from the:

stakeholder interviews summary,

document study report,

KCE group session summary

KCE management session summary

In a session with the research team we screened the data for communalities, corroborations, inconsistencies and contradictions. For each topic we drew intermediate conclusions by a first confrontation with the hypothesis and research questions.

Step 2 was done separately for the case studies, because they were more focused on finding contextual and explanatory information than the other research activities.

Finally, in the third step we made an overall analysis on the outcomes per frame work topic. With this we connected to the process-oriented notion that there is a causal relationship between the degrees of successfulness of different phases in the process leading to impact (also see paragraph 3.4). We confronted the overall outcomes with the hypothesis and looked for patterns in the overall process from research agenda setting to the use of outcomes in decision making. We looked for plausible explanations for utilizations (or non-utilization) of the research findings of KCE.

Figure 3.3 Analysis in three steps

Data collection

Step 1: Synthesis of collected data Results in: document analysis report group session summary reports single stakeholder interviews summary of stakeholder interviews Self report summary Single case study reports Cross case summary Reports on foreign agencies

Step 2: Summarizing and screening Results in merged data on: Origin Process Dissemination Decision making External image Case studies Foreign agencies Intermediate conclusions

Step 3: Confronting with framework Results in: Overall conclusions Answers to research questions.

Case studies Single case study reports

Case studies Cross case summary Case Level: Process Case level Dissemination Decision making External image Intermediate conclusions

38

4 RESULTS OF THE IMPACT STUDY

4.1 INTRODUCTION In this chapter we present the outcomes of the data collection and analysis for each of the elements in figure 4.1. It is divided into the following paragraphs:

4.2. Origin of the research projects

4.3. Research process

4.4. Dissemination

4.5. Contribution to the decision-making process

4.6. External image of KCE

Each paragraph contains different subparagraphs, addressing various aspects of the central subject. Each subparagraph concludes with key findings that summarize the most important facts. The research team has aimed to make the structure of each subparagraph as uniform as possible, using the following arrangement:

1. Procedures on the relevant subtopic (addressing any existing internal procedures)

2. Internal view on the relevant subtopic (addressing KCE experts’ and management opinions)

3. External view on the relevant subtopic (addressing stakeholders’ opinions)

4. Reported findings on the relevant subtopic (addressing findings from the project coordinators’ self reports).

However, sometimes not all data sources provide information on the specific subtopic causing some parts to be left out. This especially applies to paragraph 4.4 because it addresses results rather than processes. Paragraph 4.5, in its nature, consists entirely of external views (opinions from stakeholders) and therefore does not address procedures, internal views or reported findings.

In addition we separately present the outcomes of the case studies and the international comparison (paragraphs 4.6 and 4.7).

This chapter mainly summarizes the outcomes, without interpretation. It gives a basis for the final conclusions (chapter 5) on the impact of KCE. Because the research team aimed to identify improvements to enhance impact, it has chosen to focus on learning opportunities for KCE.

Figure 4.1 Impact assessment

I

Research process

II

Dissemination

III Contribution to the decision making

process

IV

External image of KCE

- Origin of the research

- Quality

- Involvement of stakeholders during research process

- Impact intended

- Types and number of outputs produced

- Dissemination activities performed

- Stakeholders addressed by outputs

- Engagement of stakeholders in outputs

- Actual take-up of outputs

- Familiarity with research projects

- Research utilization

- Relevance of projects for policy making / practice

- Reactions on finished projects from stakeholders

- General reputation of KCE

- Judgment on responsiveness

- Judgment on research quality

39

4.2 ORIGIN OF THE RESEARCH PROJECTS This paragraph discusses the origin of research proposals, the selection procedures that have been applied in 2005 and 2008 and the subsequent development of a research program for 2006 and 2009. These two years have been chosen because:

2005 was the second year KCE was operational. At that time procedures and processes were well in place.

2008 was, at the time of this study, the most recent year in which proposals were selected.

The following topics are being addressed:

1. pre-proposal phase

2. proposals submitted

3. proposal selection and research prioritization

4.2.1 Pre proposal phase

From the stakeholder interviews it becomes clear that there is a pre proposal phase in which there is interaction between KCE and organizations that want to submit a proposal. There are regular contacts not related to specific proposals, for example between KCE and stakeholders that are represented in the Board and stakeholders that have regular contact with for example the director of KCE. Respondents from ministers and government have reported that they (as a person) have contacts at least once in three monthsxxxii, one of them even once a week. Also the respondents of insurers state they have contact with KCE at least once in three months. Respondents from associations of care providers or professionals in general have less frequent contacts: one respondent has contact once in three months, the other respondents between once in 6 months and less than once a year.

Furthermore there are contacts to explore if specific issues are likely to be approved when submitted as proposal. If it is already clear a certain subject isn’t likely to be approved, it saves the effort of writing a proposal.

Pre proposal phase: key findings

There are regular contacts between KCE and main stakeholders

KCE and stakeholders sometimes deliberate about the likeliness of specific issues being approved by KCE when submitted as official proposal.

4.2.2 Proposals submitted

Comparing both years, the number of proposals submitted increased from 87 in 2005 to 143 in 2008. Most proposals originate from universities, private institutions, hospitals, FPS Health and NIHDI (see table 4.1).

The most favoured domain is Health Services Research (HSR), making up close to half of the (acceptable) submitted proposals in 2005, and more than half of the submitted proposals in 2008. About one third of the (acceptable) submitted proposals consist of topics in the Good Clinical Practice (GCP) domain, and about a fifth concerns topics for HTA.

xxxii The Board of Directors meets once every three months.

40

Table 4.1: Submitted proposals

Taken

into account Not taken

into account

2005 - University, scientific institution 22 2 - Hospital 16 2 - Private institution or persons 18 4 - NIHDI 5 - - FPS Health 6 4 - Cabinet Social affairs & Health 8 -

Total 2005 75 12

2008 - University, scientific institution 27 10

- Hospital 14 7

- Private institution or persons 47 6

- NIHDI 10 1

- FPS Health 16 1 - International organization 2 -

- Government organisation – non-federal 2 -

Total 2008 118 25 [*When proposals are not taken into account, this is due to the topic concerning activities outside KCE-domain, incomplete proposal forms or doubling with another proposal or running project]

Proposals submitted: key findings

The number of yearly submitted proposals has increased

About 50% of the proposals is within the domain of HSR

The majority of proposals is submitted by ‘universities & scientific institutes’ and ‘private organizations & persons’

4.2.3 Proposal selection and research prioritization

4.2.3.1 Procedures on proposal selection and research prioritization

Proposal selection at KCE consists of three main stepsxxxiii:

1. Priority setting by the Board of Directors of the submitted research topics (TPF or topic proposal form) leads to a year programme (process 1).

2. Research topics with a high priority have to be elaborated (scientifically and operationally) in a pre-project form (PPF) (process 2).

3. The information from the pre project form is used to make a year programme. This programme is presented to the Board of Directors (process 14).

TPF phase (step 1)

The criteria chosen for the approval procedure in 2005 and 2008 arexxxiv:

1. Methodological feasibility score

2. Contribution to improvement of health score

3. Contribution to cost effectiveness score

4. Score concerning the importance in terms of health care policy support

The further detailing of the criteria is limited. The criteria are both practice and policy-oriented (feasibility and contribution to health, cost effectiveness and policy support).

xxxiii See 02 ProcesDoc Specificatie van het onderzoeksonderwerp (Opmaak Pre-Projectform) xxxiv Source: KCE, RvB van 24/6/2008 Punt 11, Document voor de Raad van Bestuur

41

The Board is responsible for the prioritization, while KCE supports the Board in this regard, for example in the preparation (validation of topics and proposing judgment of the topics) xxxv.

Scoring procedure in the TPF phase

A standard proposal form, the scoring procedure and the criteria are published in advance on the KCE website. The scoring procedure remains completely internal. Every proposal is reviewed by at least six members of the KCE expert team and directionxxxvi. They are selected from the different fields (HTA, GCP and HSR). The extent to which experts are related to stakeholders in the medical field is being clarified in a separate process notexxxvii.

The KCE experts each give scores xxxviii to every project on the four criteria formulated. The average of each score is calculated. The scores of the experts are compared. If the difference between scores is higher than 1.3, experts compare their scores to come to a consensus.

Once the scoring is done, the scores per criteria for each proposal are to be published in the minutes of the Board meetingxxxix. In addition, the KCE process documents provide the possibility to form a Scientific Advise Board (Wetenschappelijke Advies Raad, WAR) which gives advice to KCE about their prioritization of the proposed research topicsxl. Yet, this WAR has in fact never been established. Proposals are first prioritized based on the sum of the four scores. The higher the score, the more priority is given to a proposal. In addition, proposals receive a score for the degree to which the subject is related to a subject in the previous year’s programmexli.

PPF phase (step 2)

Not all proposals included in the study programme on the basis of the prioritization procedures constitute the final work programme. For each topic in the study programme, KCE develops a pre-project form (PPF) to refine the proposal and to enable a further ratification of whether the topic should or should not be included in the work programme.

Elaboration of the work programme (step 3)

Finally, the work programme depends on the available means and competencies, which may result in adjustments by the KCE directors and the Board of Directors in the proposed prioritization. Any adjustments should be carefully motivatedxlii. The submitters of all proposals that are not included receive a letter by KCE. Since 2009 the letter clearly states the scores for each item. If the information is not clear, the submitter can call KCE for explanation.

Results of proposal selection and prioritization in 2005 and 2008

The steps taken and final results of the selection procedures in 2005 and 2008 give insight in how they work in practice and how work plans are made.

In 2005, the 25 highest ranked proposals – on a total of 75 proposals that were scored and prioritized – were included as priority projects for the work programme 2006. The number of projects is determined by KCE’s budget and manpower for the coming year. In 2005, this number had to be limited to 25.

In 2008, the 39 highest ranked proposals – on a total of 118 new proposals that were scored and prioritized - were selected. However, not all 39 topics have been included in the work programme of 2009, as the available means and capacity were insufficient. The KCE-directors therefore performed a second selection procedure, dividing the projects into three groups according to their feasibility, urgency and the possibility to integrate parts of this study into other projects. The first group of projects (18 in total) were judged as highly feasible and urgent to be

xxxv Source: ProcessDoc1: Opmaak jaarprogramma met prioritisering onderwerpen 34 xxxvi Source: KCE, RvB van 23/06/2005 Punt 10, RvB 24/06/2008, Punt 11 xxxvii Source: KCE Proces note 0.2: Belangenconflict en Gedragscode voor entiteiten betrokken bij KCE onderzoek

(v20031117) 34 xxxviii Five scores are possible : 5 = excellent, 4= good, 3 =Sufficient, 2 = Poor, 1 = Insufficient. xxxix Source: KCE, RvB van 23/06/2005, Punt 10 xl Source: KCE, Procesnota 00.1: Overzicht van de entiteiten betrokken in het onderzoek34 xli Source: KCE, RvB van 23/06/2005 Punt 10 xlii Source: KCE, RvB van 23/06/2005, Punt 10

42

realized in 2009 (four of these projects were originally high priorities for 2008). The second group of projects (6 in total) was judged as more difficult to realize and not as urgent. The third group of projects (14 in total) was judged not to be carried out as they appeared not to be feasible or possible to integrate into other studies.

The final work programme for 2006 and 2009 also contained (parts of) remaining projects from 2005 and 2008. These are not part of the prioritization process of new proposals for 2006 or 2009.

Table 4.2 shows that in 2005 in particular, some stakeholders are more likely to have their submitted proposals included in the work programme than other stakeholders. The difference is most significant between Universities & scientific organizations (2 out of 22 of their submitted proposals included) and Cabinet Social affairs & Health, with 8 out of 8 of their submitted proposals included. In both years, a relatively large proportion of RIZIV-INAMI topics have been accepted in the work programme.

Table 4.2: Selected proposals in 2005 and 2008

Proposals submitted by

Submitted and taken into account

(a)

Accepted for work

programme of next year (b)

% (b/a *100)

2005 - Universities, scientific associations 22 2 9 % - Hospitals 16 5 31 % - Private institutions or persons 18 4 22 % - NIHDI 5 4 80 % - FPS Health 6 1 17 % - Cabinet Social affairs & Health 8 8 100 % Total 2005 75 24 32 %

2008 - University, scientific institution 27 1 4 % - Hospital (or institute) 14 0 0 - Private institution or persons 47 8 17 % - NIHDI 10 4 40 % - FPS Health 16 1 6 % - International organization 2 ? ? - Government organisation – non-federal

2 0 -

- From 2008 prioritisation 4 n/a

Total 2008 118 18 15%

4.2.3.2 Internal view on proposal selection and prioritization

In the previous paragraph (3.2.3) we described the formal criteria and selection procedure. This paragraph shows how KCE experts and management view the proposal selection procedures applied.

The KCE expertsxliii perceive a change in the degree of objectivity present in the selection procedure when comparing the last selection round to previous rounds.

Observations mentioned to underpin this perception are:

A change in the way policy relevance is being judged - experts are no longer to judge the policy relevance of the topic. Previously, there used to be a scoring followed by an extensive discussion that could continue for up to three days. Now attention is particularly given to the origin of the proposal. Proposals from the FPS, the Minister, NIHDI and Parliament automatically get a high score on policy relevance.

Although the selection procedure itself has not changed so much, experts feel the process has become less transparent because their

xliii From the group interview with KCE staff members

43

own scores are not necessarily followed. Experts say that even when a project was not high on the priority list, the KCE management team can still put the topic back on the list. Some TPFs are hardly completed, but because they are very relevant for policy-making, they are nevertheless included. Because it is not known to KCE experts who submitted the topics (in the TPF phase), they do not always understand the reasoning leading to the inclusion or exclusion of certain topics.

KCE management team states the reason for keeping the selection procedure completely internal is for it to be as impartial as possible. The KCE is an independent organization, not connected to any interest group. The only possible partiality is that KCE experts might have a preference for certain studies.

According to KCE management the scoring item “importance in terms of health care policy support” is influenced by which organisation submits the proposal. NIHDI and the Cabinet Social affairs & Health, for example, are highly relevant organisations for policy making and the most important clients of KCE. It is important to have close contact and a good sense of what is going on in policy to judge the relevance of proposals. Experts have a more scientific point of view and are not implicated in the strategic choices.

It sometimes happens that these submitting organisations are not clear about the priority among their proposals (when several are submitted). The KCE director and the board take this into account during the elaboration of the work programme.

For experts these final steps in priority setting might seem non-transparent. One of the members of the management team says it might be a good idea if experts are still allowed to score policy relevance and are involved in the discussion on policy relevance. This way internally it would be clearer why certain decisions on proposal selection are made.

4.2.3.3 External view on proposal selection and prioritization

The external view described in this paragraph is based on the interviews with 20 representatives from KCE stakeholders. Of the stakeholders interviewed, 17 have submitted at least one proposal during KCE’s existence.

Transparency

From the 20 stakeholders that have been interviewed, 11 answer “yes” to the question if the proposal selection process is transparent, seven answer “no” to this question. Two cannot give a clear answer to this question, they say it is neither yes nor no (see table 4.3).

Table 4.3: Selection process transparent?

Rows Frequency

Yes 11 No 7 No clear answer 2

Total 20

After clustering these questions to different types of stakeholders (clustering see paragraph 3.6.4, page 47), there is a clear difference. Respondents from associations of professionals and care providers generally judge the selection process as not transparent (only two out of six stakeholders says the proposal selection process is transparent). Transparency is judged high by respondents from government and (six out of seven say it is transparent). From the insurers two out of three say it is transparent.

Objectivity and impartiality

To the question whether the proposal selection process is objective and impartial, seven respondents answer “yes” (40%) and eight “no” (40%). Five can not give a clear answer, they say it is neither yes nor no (see table 4.4).

44

Table 4.4: Selection process impartial and objective

Rows Frequency

Yes 7 No 8 No clear answer 5

20

From the respondents of associations of care providers and professionals four out of eight answer “no” to the question whether the proposal selection is objective and impartial. In the other stakeholder clusters, the majority of respondents judge the proposal selection as objective and impartial (see table 4.5).

Table 4.5: Impartial and objective by cluster

Stakeholder cluster Yes No Not clear Total

Associations of professionals and care providers 1 4 3 8 Government 4 2 1 7 Insurers 1 1 1 3 Other 1 1 - 2

Total 7 8 5 20

Respondents had the possibility to clarify the answers to the questions on transparency and objectivity. The following remarks were made (all remarks have been listed):

“Yes, the process is transparent, but I can’t say whether or not the selection is impartial and objective. KCE cannot easily refuse a proposal from the government. This could be a threat if the government sees KCE primarily from the perspective of HTA as an instrument to cut health care expenses”. (Respondent from cluster “government”).

“Yes the process is transparent, the outcomes are published.” (Respondent from cluster “government”).

“Yes, it’s possible to submit the proposals we think are important. The selection of proposals is objective and impartial. The only influence is by current challenges in health care.” (Respondent from cluster “government”)

“No, the selection isn’t transparent and impartial. It is clear that KCE aims at a transparent and impartial proposal selection, but it is endangered by the proposal selection system setup. The main problem lays in the selection of subjects. Decisions are made by KCE experts. The choice of subjects is subjective. For example, a proposal on palliative medicine received a low score on relevance for society” (respondent from cluster “other”).

“No, the choice of proposals is politically oriented”. (Respondent from cluster “associations").

“No, choices are not objective. If two French-speaking partners submit a proposal the chance it will be approved is nil”. (Respondent from cluster “associations”)

“No, we don’t know what the criteria are, so we get the impression it is influenced by interest groups” (Respondent from cluster “associations”).

Communication on proposals that are not selected

Stakeholders answered the question whether KCE had ever refused one of their proposals. And if so, whether the reasons for refusal have been made clear. In total 12 out of 20 stakeholders perceive the feedback of KCE as not being sufficiently explanative about the reasons for exclusion.

45

Proposal selection and research prioritization: key findings

KCE uses various procedures and criteria to guide the process of proposal selection and research prioritization. These are published on the KCE website and therefore are publicly accessible.

The total procedure contains the following steps: 1) Scoring on four criteria; and prioritization on the basis of total scores; 2) Development of pre-project forms (refining and ratification); 3) Final selection on the basis of feasibility and urgency (limited by available means and research capacity).

The criteria formulated are both practice and policy oriented, but they are formulated in quite general terms and the scoring therefore depends on the experts’ (subjective) interpretation of these criteria.

Some stakeholders (such as INAMI/RIZIV and the Cabinet of the Minister of Social affairs & Public Health) are more likely to have their proposals admitted to the work programme because their proposals always receive a high score on the criterion “policy relevance”.

KCE involves no external parties during the proposal selection phase in order to retain a high degree of impartiality.

About one third of the interviewed stakeholders perceived the proposal selection procedure as not being transparent and impartial. This especially counts for the associations for care providers and professionals (2/3) because they see a focus on proposals from the Federal Public Services and RIZIV/INAMI.

To more than half of the stakeholders interviewed the reasons for refusing a proposal are not clear.

4.3 RESEARCH PROCESS In this paragraph attention is given to the way in which KCE has organized its research process in terms of quality (methodology and project management), involvement of stakeholders during the process and the formulation of intended stakeholders and type of impact at the start of the project.

The paragraph consists of the following subparagraphs:

1. Quality of the research process

2. Involvement of stakeholders during the research process

3. Intended impact

4.3.1 Quality of the research process

In this paragraph we focus on the quality of the research process of KCE. We examine to what extent the research process is focused on usefulness of outputs and results. Also, we address the question whether the methodology used is scientifically based and the research process is being adequately supervised in terms of quality. Sources for this paragraph are the formal procedures devised by KCE (document study) and the way in which these are used and valued by KCE experts (group session) and KCE management (interview).

4.3.1.1 Formal procedures on quality of the research process

KCE has formal procedures on project management and methodology to assure the quality of the research process. The roles and responsibilities of each actor regarding quality control in the research process have been clearly describedxliv for different phases of the research process. This is done for both research that is being carried out by KCE and for research projects that are being outsourced. These procedures and process documents could be used by the research team. Other instruments to assure research quality are internal and external validation and the use of an external expert group.

Process documents

Each of the following process documents is geared towards quality control concerning methodology and project management:

xliv KCE Process Note 00.3: Bouwstenen van de wetenschappelijke kwaliteitscontrole binnen het KCE

(v20031117) 34

46

Process 3: “Making projects operational” – this happens after projects have been included in the KCE work programme. From a pre-project form (explained in paragraph 4.2.3 on proposal selection) a final project form is made. This project form contains:

o Results of a pre assessment on the research topic

o Further detailing of the research questions

o Scientific design

o Operational design of the research project

Process 4: “Search for evidence” is aimed at using efficient methodologies to find existing evidence on the research questions.

Process 5: “Data research” defines how data should be collected, processed, transformed and validated in a systematic and valid way.

Process 15: “Personnel and organisation” is aimed at managing resources (financials and administration).

Safeguarding a focus on the usefulness of the outcomes/results throughout the whole research process is being addressed in the following management processes:

Process 2: Specification of the research topic. Target groups are to be included in the communication plan and the implementation thereof.

Process 6: Critical processing and analysis: attention must be given to:

o Policy relevance

o Potential use of results in applications

o Use in subsequent evaluation techniques

Process 7: Synthesis, requires very explicit attention and competence (specialists in the medical field under study) to assure the usefulness of the research and (policy) recommendations. It also explicitly prescribes:

o The adaptation of the report and format to the selected communication media.

o Recommendations for implementation of the proposed policy recommendations;

o Proposed modalities for feedback for the target groups of the study.

o An executive summary with decisions, recommendations and practical settlements and their dissemination and monitoring.

Process 8: Validation. The external validation template contains “readability and clarity of language to policymakers” as one of the aspects to be judged.

Process 9: Acceptance and communication is directed at ensuring usefulness by prescribing:

o Formulation of implement able policy recommendations

o The Board, before final acceptance, also evaluates the extent to which the policy recommendations are implementable

Process 10: Impact monitoring is specifically geared towards periodical evaluation regarding whether policy recommendations have resulted in policy decisions, as well as finding out (content-wise) to what extent the formulated KCE recommendations have been implemented.

Process 11: Knowledge management focuses on the sharing, transfer and use of the generated knowledge through knowledge networks.

External expert group

During the research, the research team may be supported by external experts. This group of experts is established to improve the qualityxlv . Their role is to critically reflect on content and methodology of the research and to advice on how to improve the quality of the research report. Possible members are: scientists

xlv KCE Procesnota: Overzicht van de entiteiten betrokken in het onderzoeksproces

47

from universities (national or international), researchers of KCE, experts from RIZIV/INAMI, associations of professionals and other interested parties.

Validation process

Overall the KCE Management Plan states that 35; The reports of the KCE Knowledge Centre must be as much as possible completely free of faults. Any faults could lead to misguided decisions and could possibly result in grave consequences. The law therefore explicitly provides in a quality control. Quality procedure documents show that after the first draft of the report, there are three standard moments of validation of each research project:

Internal validation

External validation

Final endorsement by KCE Board of Directors (before publication)35

Since 2008, the internal review is done by KCE senior expertsxlvi and two experts. In the final stage of the research project they read the executive summary and the recommendations and comment on them.

The external validation is done by an uneven number of validators (at least three). Validators have to declare potential conflicts of interest.

For the external validation of reports, a template validation report is used, which focuses on:

Scientific state of the art

Presentation of the scientific facts

The interpretations are supported by the facts and no omissions are found

The language used is clear to policymakers and the methods used transparent

Formulation of the scope and research questions

Description and completeness of the sources of information

Results and interpretations

Discussion and conclusions:

o Ethical and societal implications

o Robustness of conclusions

o Limitations

General conclusion on the validity of the report

At the end of their validation report, each validator evaluates the report using one of the following standard notices: 1) Valid report with minor remarks, 2) Valid report with major remarks, 3) Report not valid.

Each validator sends his validation report to the validation chairman (the member of KCE direction who will be chairman of the validation meeting). Subsequently the validation team formulates a common opinion on the validity during the validation meeting. They strive to reach a consensus. In the absence of a consensus, the team proceeds to a voting procedure. There are three possibilities:

1. Valid report with minor remarks: the research is submitted to KCE board of directors for finalization.

2. Valid report with major remarks: if possible, the research will be adapted in accordance with the remarks of the validation team and send to the Board of Directors for finalization.

3. Report not valid: an invalid report can not be submitted to the Board of Directors. KCE will have to examine which arrangements can be made to correct or re-do the research.

Finalization process

After the validation, a week before the Board meeting, the summary and recommendation are discussed in a meeting with KCE management team and the chair and vice chairs of the Board (also called the “Bureau”). They test whether the

xlvi Currently, KCE has three senior experts.

48

recommendations are realistic and feasiblexlvii. Finally the draft research report is being presented by the research team to the entire Board of Directors. Several studies (six or seven) are discussed during one Board of Director meeting. The Board of Directors then ask critical questions and provide feedback concerning the quality of the research processxlviii, the content and target audience of the report and the usefulness and acceptability of the recommendations. They occasionally give advice to the authors about how to further improve the report, after which the KCE report and recommendations are, in most cases, accepted by the Board of Directorsxlix. After the acceptance of the recommendations, the report is published.

4.3.1.2 Internal view on quality of the research process

Having formal management procedures is an important step in achieving and maintaining a high research quality level. Application of these procedures is another step. In this paragraph we present the views on internal procedures, the scientific underpinnings of the research findings, validation procedures and outcomes and research finalization. The internal view we describe is based on the outcomes of the self report questionnaire (project coordinators l ), the group sessions with KCE experts and with KCE management team.

The external view on the research process is reported in paragraph 4.6.3.

Internal view on process documents

The formal procedures were discussed during the group session with KCE management team and a session with KCE experts. In both sessions the procedures for quality control were claimed to be helpful in conducting research of high quality. There is for example a very clear protocol for the search strategy to be followed and which search terms to use, in order to miss as few studies as possible. Because of the standard protocol, the credibility of KCE research is high. It always enables the research team to explain the results and to verify any questions or comments regarding possible omissions.

At the same time KCE experts state not all procedures are equally well known and used. There is no instruction for new experts on how to use these procedures and no internal control to see whether all procedures are followed.

Internal view on external expert teams

Although no regulations have been made about the kind of experts that should be invited, it is common practice to (nearly) always invite representatives from the medical field. KCE management team says that since the beginning of KCE members from the medical field have (almost) always been incorporated in external expert groups. The involvement of stakeholders with a policy interest however has increased substantially. In this context, the management team considers it to be a task of KCE to be vigilant on the fact that the contribution of stakeholders might be aimed at influencing the outcomes to better suit their interests. In fact, a proven downside in involving stakeholders in expert groups is that it sometimes complicates the research. A conflict of interest may cause external experts to hinder or even to obstruct the research.

According to the KCE experts, scientists are always involved in every project. Patients’ associations have never been represented in expert group meetings, because, according to the experts, these groups are not able to give comments of a sufficiently scientific level and quality. Patients’ associations are only being involved as respondents or research subjects, for example in focus groups.

Internal view on validation process

During the group sessions with KCE experts and KCE management team the validation procedure was discussed. The experts view the validation procedure as clear and very important to maintain a high level of research quality. A remark made is that the external validation meeting occurs at a late stage in the project. This is mostly too late for example to adjust the methodology. Often there is only 1 or 2 weeks left before the report needs to be finalized. This time is too short when

xlvii Information from the group session with KCE management team xlviii For example, the extent of involvement of external experts such as professional organizations or RIZIV in the

research xlix KCE, minutes of Board meetings in 2005 en 2008 l Project coordinators are either internal (KCE experts or managers) or external.

49

they have major comments. The quality of the reports could be improved when there would be more time to incorporate the validators’ comments.

KCE management team states both the internal and external validations are very important to ensure high quality research. For the internal validation two things could be improved:

1. Internal validations do not always take place due to busy work schedules of experts; in the future internal validation should have a higher priority. Internal reviews, however, are always performed.

2. The internal validation is at the end of the project. This is often too late to make major adjustments. It would be good to also do an internal validation at the beginning of a project.

One of the members of the management team states these improvements are only possible when more time is appointed, for example by doing less research projects.

Internal view on finalization process

After the validation KCE management team and the chair and vice chairs of the Board (i.e. KCE Bureau) test whether the recommendations are realistic and feasible KCE management team states that only occasionally experts are asked to change the summary or recommendations. KCE management understands this might seem like a compromise to the ‘purity’ of the scientific outcomes to KCE experts. However, it is meant to guarantee that used vocabulary will be adapted for the final user needs and that can increase the usability of the recommendations. According to KCE management team the scientific foundations are never changed by the KCE Bureau.

KCE experts confirm that their recommendations are often followed by the Board. Some experts feel that changed recommendations are not always in line with the scientific conclusions of the research; they consider it a process of push and pull within the Board of Directors, related to the interests of each of the directors.

Safeguarding usefulness of the results

One of the questions the research team addressed during the group sessions is how KCE guarantees the usability of outputs/results (e.g. producing implementable recommendations).

From the group session it clearly appears the KCE experts’ that interference with the usability of outputs stops right after delivering the report. The experts concur that they are not involved in the implementation at all, and furthermore are not allowed by KCE management to do so. Implementation is explicitly not considered to be their task. The experts state that they are not always being informed by KCE management team and the Board of Directors about the further process. Furthermore, they deplore that it is not common practice to invite and involve KCE authors beyond the stage of final approval. Also no time is planned or reserved in the planning of experts to concern themselves with implementation.

On the other hand, the KCE experts themselves do not consider impact maximization as the main goal of KCE. Political willingness to implement recommendations is a factor that the KCE experts feel they can not influence. Some experts argue the degree of usability of the findings is not only dependent on the quality of the report, but also on the way the findings are being disseminated by the stakeholders represented in the Board of Directors. If none of the Board members are interested, it sometimes occurs that a report is not being disseminated (except from being published on the KCE website and a press release). Therefore, much depends on the willingness, motivation and interest of the Board members. The experts feel they cannot influence the Board in this regard.

4.3.1.3 Reported research quality

Methodology

In the self report questionnaires the project coordinators were asked how they judge the scientific underpinnings of the research findingsli . First we show the general response to this question and next we will look at different domains (HTA, GCP and HSR) and the years the research reports were published.

li Response of self report questionnaire (see Appendix 8)

50

On average, project coordinators leaned towards a “high” rating on this aspect (3,9 points on a scale of 5), and none of the respondents judged the scientific underpinnings as very low. (See table 4.6). This applied more or less equally to both the external and internal KCE project coordinators.

Table 4.6: Scientific underpinnings of the findingslii

Count %

Very high 24 19.0 % High 62 49,2 % Average 34 25,4 % Low 4 3,2 % Unknown 4 3,2%

Total 126 100%

The average score for the quality of scientific underpinnings within HTA projects was more than high (4,2 points) and GCP also scored high (3,9 points) on this aspect. The scientific quality of HSR projects was on average rated lower, between average and high (3,5 points) (see table 4.7).

Table 4.7: Scientific underpinnings per field HTA GCP HSR

Count % Count % Count % Very High 12 29,3 % 9 23,1 % 3 8,3 % High 25 61,0 % 19 48,7 % 14 38,9 % Average 3 7,3 % 11 28,2 % 16 44,4 % Low 1 2,4 % 3 8,3 %

Total 41 100 % 39 100 % 36 100 %

When we look at differences between the year of project acceptance, we see that for projects that were accepted in 2006, the quality of scientific underpinnings was on average rated above “high” (4,1 points). In 2007, this declined to 3,7 points and in 2008 quality of scientific underpinnings was rated 3,8 points (above “average”, towards “high”) (see 4.8).

Table 4.8: Scientific underpinnings per year

2006 2007 2008

Count % Count % Count %

Very High 8 25,0 % 3 10,0% 8 21,1 % High 19 59,4 % 17 56,7 % 17 44,7 % Average 5 15,6 % 9 30,0 % 11 28,9 % Low 0 0 1 3,3 % 2 5,3 %

Total 32 100 % 30 100 % 38 100 %

Project management

External project coordinators were asked whether the KCE project coordinator adequately sees to keeping the focus of the study in terms of scope and outcome. The majority (72,7 %) of external project coordinators say it was adequate (see table 4.9).

Table 4.9: Adequately keeping focus by KCE project coordinatorliii

Count % Yes very 14 42,4 % Yes 10 30,3 % Not adequate/not inadequate 3 9,1 % No 4 12,1 % Not at all 1 3,0 % Unknown 1 3,0 %

Total 33 100%

lii Question was: How do you judge the quality of the scientific underpinning of the findings? liii Question was: Did the KCE project coordinator adequately see to keeping the focus of the study in terms of

scope and outcome?

51

Quality of the research process: key findings

Quality procedures and processes are clearly described for:

a. making projects operational

b. conducting the research

c. validation

d. finalization

Usefulness of outputs and results are explicitly addressed in several procedures.

The procedures and processes to secure research quality are highly appreciated by internal (KCE experts) and external project coordinators.

Not all KCE experts are equally well familiar with all the processes and procedures. There is not enough instruction for new experts on how to use these procedures and not always an internal control to see whether the procedures are followed.

Internal and external project coordinators are highly positive about KCE’s research quality in terms of methodology (scientific underpinnings of the findings).

Some KCE experts observe that in some reports the recommendations and summary are not in line with the scientific conclusions of the research.

According to KCE experts and KCE management, more and timely attention should be given to validation in order to enhance the quality of the validation and to give researchers more time to adequately incorporate comments into their research and report.

Involvement of stakeholders during research process

This paragraph shows to what extent KCE involves stakeholders during the research process.

4.3.1.4 Procedures on involvement of stakeholders during research process

No formal procedures have been designed by KCE on the involvement of stakeholders during research projects. However, KCE is currently working on a pre-project form to develop a procedure for the involvement of stakeholders specifically for HTA and HSR projects. Chances are this project will be part of next year’s programme. A separate procedure will be developed for GCPliv.

4.3.1.5 Internal view on involvement of stakeholders during research process

Although there are no formal procedures on stakeholder involvement, within KCE it is the general expectation that in each research process there will be contact with the medical fieldlv. The main way to organise this contact is by the use of an external expert group for the research project. To select external experts, it is possible to use a data base of 1500 external experts with data on their field of expertise.

KCE management team says that since the beginning of KCE the involvement in the external expert group of members from the medical field has not been changed. However, the involvement of stakeholders with a policy interest highly increased. It is the task of KCE to be vigilant for influences of external experts on the results. A downside of involving stakeholders in an expert group is that they sometimes complicate the research. A conflict of interest may cause external experts to hinder or even to obstruct the research.

KCE experts are aware of the importance of involving stakeholders during research projects. One of the KCE experts states that inviting policy makers to take part in external expert groups is a way to positively influence the impact achieved by the project. This is because it enables policy makers to anticipate on the findings and on the implementation of recommendations. Another expert confirms that choosing and involving the right experts is beneficial to the degree of implementation of the recommendations into policy or practice.

According to the KCE experts, the lack of standardized approaches, supervision and guidance is hampering a structural and uniform involvement of stakeholders.

liv From KCE group session lv From KCE expert group session and KCE management session

52

Every project team decides for themselves who they want to involve, at what point and for with what aim. Thus, the moment of involvement and the issues on which to involve stakeholders are left to the interpretation of KCE experts. If stakeholders or target groups are being invited to the external expert group, this is mostly to benefit from their expertise and contribution to the project, and not because it may enhance the possibility to place the topic or the results of the study on the policy agenda.

Another expert argues that sometimes when an important stakeholder is being omitted, this only becomes clear after the project has finished when the stakeholder complains about not being involved.

One of the experts brings forward that there is often also a bias in who to invite: often only stakeholders with whom they have already established a positive contact are being invited, so there is no systematic way to involve new stakeholders.

4.3.1.6 Reported involvement

Extent of involvement in projects

From the self report questionnaire the majority of coordinators (85%) state they involved stakeholders during the research process (before dissemination). A minority of 12,5% of the coordinators said they did not involve stakeholders during the research process. Internal coordinators answered this question positively slightly more often than the external coordinators (see table 4.10).

Table 4.10: Stakeholders involved in research

Yes No Total

Internal 88,5 % 11,5 % 100 % External 83,3 % 16,7 % 100 %

For projects approved in 2007 (93,1% yes) and 2008 (87,2% yes), respondents reported a higher incidence of stakeholder involvement when compared to projects approved in 2006 (80,0% yes). Stakeholder involvement is reported to have occurred most often in HTA projects (90,5% yes), followed by HSR projects (86,1% yes) and GCP projects (80,6% yes).

In most projects, inviting or informing stakeholders identified as target users was done by KCE or by the internal project team (67,6%). However, from 2006 to 2008 an increase in the number of projects in which target users invited themselves can be seen: from 4,2 percent in 2006, to 11,1 percent in 2007 and 14,7 percent of the projects in 2008.

Type and moment of involvement in projects

Coordinators answered the multiple response question about the activities in which stakeholders were involved. For most projects, coordinators reported stakeholder involvement in critical reflection during the research process (80,3% of the projects). Furthermore, stakeholder involvement is reported to have occurred often in determining the scope of the research (42,2% of the projects), drawing conclusions (42,2% of the projects), developing the research questions (40,2% of the projects), selecting data sources (33,4% of the projects) and making data sources accessible (31,3% of the projects). In about a quarter of the research projects, stakeholders were involved in selecting data collection methods (25,5%) and the formulation of recommendations (24,5%). (See table 4.11).

53

Table 4.11: Type of stakeholder involvement

Type of involvement %

Critical reflection during research process 80,3 % Drawing conclusions 42,2 % Determining the scope of the research 42,2 % Framing of the research questions 40,2 % Selecting data source(s) 33,4 % Making data sources easily accessible 31,3 % Selecting data collection method(s) 25,5 % Formulating recommendations 24,5 % Designing research instrument(s) 17,6 % Other 13,7 % Unknown 1,9 % Presented by number of projects (it was possible to select multiple types of involvement per project)

Involvement of stakeholders during research process: key findings

There are no systematically applied procedures regarding the involvement of stakeholders

Forming expert groups is generally the only way in which KCE involves stakeholders during the research process

KCE experts argue that stakeholders are invited into the expert groups because KCE wants to benefit from their knowledge and expertise. This is the main reason why experts of the medical field are nearly always invited, and patients’ organizations are never invited in external expert groups.

KCE experts acknowledge the importance of different forms of stakeholder involvement during the research process for increasing the degree of impact studies are having on policy and practice.

The lack of standardized approaches, supervision and guidance is hampering a structural and uniform involvement of stakeholders by KCE experts.

KCE management team sees an increase of involvement of representatives of interest groups in external expert groups.

According to project coordinators stakeholders are involved in 85% of the research projects. The type of involvement that occurred most often in projects is critical reflection during the research process (80.3%). Involvement in designing research instrument(s) and formulating recommendations is reported to have occurred less often (<25%).

4.3.2 Intended impact

The topics addressed in this paragraph are whether the intended impact is explicitly formulated in advance, which primary target audiences are aimed for, and what type of impact is aimed for (instrumental, conceptual, symbolic).

4.3.2.1 Procedures on intended impact

The law regarding the constitution of KCE defines in general the primary target audiences to be addressed by KCE studies. The following organizations specified by law as primary target audience for KCE studies: RIZIV/INAMI, FPS Health, FPS Social Security and the Cabinet of the Minister of Social Affairs and Public Health.lvi

Apart from this general indication, KCE has thus far not been using standardized protocols or documented procedures to be applied regarding the explicit identification of target audiences in specific projects and the type(s) of impact aimed for.

4.3.2.2 Internal view on intended impact

The KCE experts in the group discussion concur that in KCE studies no attention is being paid to intended impact of the study. Also, no target audiences are being

lvi Law concerning the establishment of the Federal Centre of Expertise in Health Care. Coordinated version

(December 2006), art. 264.

54

mentioned. They do not consider achieving or maximizing impact to be their task or responsibility. This causes them not to ponder about or aim for specific target audiences or types of impact - and, as a consequence, not to make special efforts to make impact happen. The project team may have a notion concerning the probable target audiences, but both the intended impact and target group(s) for a study are never explicitly defined or written.

4.3.2.3 Reported primary target audiences

The research team asked project coordinators to report whether they had explicitly identified certain stakeholders as target groups at the start of the project. According to the coordinators, target groups were nearly always explicitly identified (96.9%). This might be considered somewhat surprising when compared to the internal view on intended impact.

Table 4.12 shows that RIZIV/INAMI and the Commissions have most frequently been identified as target group for specific research projects (58,5%), closely followed by medical professionals.

Table 4.12: Explicitly mentioned target groups

Selected 18. Actors explicitly defined as target group at the start of the project (Multiple Response)

%

RIZIV / INAMI and Commissions 58,5 Medical professionals 56,9 Cabinet Social Affairs and Public Health 35,8

FPS Health 25,2 Sick funds and hospitalization insurers 24,4 Universities and scientific institutes 22,7

Directors/management/ Board of Directors of hospitals / health care institutes 13,0

Medical equipment and materials industry 12,2 Other health care providers / medical professionals 11,4 Pharmaceutical industry 9,8 Patient organizations 9,8 Politicians of administrations of communities or

regions 7,3 Nursing personnel 5,7

Members of parliament or the parliamentary commission on health care 4,8

FPS Social Security 4,8 Physiotherapists 4,8 Pharmacists 3,2 International organizations 1,6 Bio-technology & life sciences industry 0,9

Other 9,8 None 1,6 I do not know 2,5

Table 4.13 shows for each of the most frequently selected target groups, the order of importance in the different projects. RIZIV / INAMI and Commissions are generally considered the most important stakeholder (ranked first in priority) in projects, followed by the Cabinet of the Minister of Social Affairs and Public Health and the federal public services. Although medical professionals are frequently identified as a target group, they often do not come first in order of priority.

55

Table 4.13: Most frequently selected target groups

Most frequently selected Target groups, in order of frequency with priority indication (%) Count 1 2 3 4 5 6 7 8 9 10 >10

1. RIZIV / INAMI and Commissions 63 47,6 36,5 7,9 3,2 0 1,6 3,2 0 0 0 0 2. Medical professionals 63 25,4 28,6 25,4 15,9 4,8 0 0 0 0 0 0 3. Cabinet Social Affairs and Public Health 42 42,9 21,4 14,3 9,5 4,8 0 2,4 4,8 0 0 0

4. FPS Health 29 34,5 20,7 17,2 17,2 6,9 0 3,4 0 0 0 0

5. Sick funds and hospitalization insurers 29 3,4 24,1 27,6 24,1 10,3 6,9 3,4 0 0 0 0

6. Universities and scientific institutes 27 11,1 18,5 18,5 22,2 11,1 11,1 0 3,7 3,7 0 0

7. Directors / BoD / management of hospitals / health care institutes 16 18,8 6,3 25 12,5 25 6,3 0 0 0 0 6,3

8. Medical equipment and materials industry 15 26,7 13,3 13,3 26,7 6,7 13,3 0 0 0 0 0 9. Other 11 27,3 54,5 9,1 9,1 0 0 0 0 0 0 0 10. Patient organisations 12 16,7 16,7 25,0 0 8,3 25,0 0 0 0 0 8,3

11. Other health care providers / medical professionals 10 30,0 40,0 10,0 10,0 0,0 10,0 0 0 0 0 0

Impact intended: key findings

KCE does not have standardized protocols or documented procedures regarding the explicit identification of target audiences and the type of impact aimed for.

KCE experts do not aim for specific target groups or types of impact, mainly because they do not consider it to be their responsibility to achieve or maximise impact from their studies.

However, almost all project coordinators report having explicitly identified target groups at the start of their projects. RIZIV/INAMI and medical professionals are most frequently mentioned as being (primary) target group.

4.4 DISSEMINATION In this paragraph we discuss KCE’s way of producing outputs, disseminating research findings and addressing stakeholders with their outputs. Also, we gain insight into the type and number of outputs produced, dissemination activities performed and the actual take-up of outputs. Internal and external views on these processes and results are also being addressed.

This paragraph contains the following subparagraphs:

1. Outputs produced as a result of research projects

2. Dissemination activities performed

3. Stakeholders addressed by outputs

4. Engagement of stakeholders in outputs and dissemination

5. Actual take-up of outputs

4.4.1 Outputs produced as a result of research projects

4.4.1.1 Formal procedures on outputs produced

The process documentlvii concerning adoption and communication reveals that the main output of each KCE project is a communicated study (report) and

lvii Proces Document 9, Proces Adoptie Communicatie. Bijlage bij Ramaekers, D. and Closon, J.P. (2003). KCE

Management Plan 16 oktober 2003. 35

56

recommendations for follow-up. This report is to be simultaneously published to the minister and the organization responsible for submitting the research topic and ‘systematically’ disseminated via the web-based channels. In addition, the process document explicitly mentions scientific articles and specific communication to target groups (as part of a project-bound communication plan) as potential outputs. The project-bound communication plan for target groups may use various channels and formats for outputs. The project-bound communication plan is to be validated by the KCE Board of Directors before implementation.

4.4.1.2 Internal views on outputs produced

Reports

From the group discussion, one can derive that KCE experts perceive the production of a report with recommendations as not being the most effective means to generate impact. Although every report has a summary with key findings, the KCE experts concur that the KCE reports are often too complicated and not easily understandable. Even in the executive summary, the terminology used may be too difficult. One of the experts states that even specialists in his field hardly read the KCE reports.

Press releases

The KCE experts think that within KCE the press releases receive much attention because they are perceived as the most important dissemination activity. They generally conceive that press releases are carefully prepared and edited, and the process is well supported. However, the KCE experts are not always positive about the use of, and reactions to, press releases. Particularly with very technical topics, they feel that there are generally very few reactions.

Scientific articles

Some experts mention that they used to produce articles in the project-time that was reserved for this purpose (i-project). Nowadays, however, time for publication is included in the 15% of their working hours reserved for self-study and education (besides the projects). Several experts consider this time insufficient; publications can only be written in the weekends or evenings. The result is that less peer reviewed articles are being produced, in spite of the fact that articles are an important instrument for achieving impact in the scientific field and for presenting (the quality of) the KCE research projects to the external world.

Other outputs

Some KCE experts state that they are not being encouraged by their management to produce outputs other than research reports. This is despite the fact that KCE experts judge other types of outputs such as information leaflets (two page brochures aimed at important target groups and representative organizations), presentations (particularly for GCP projects) and the production of scientific articles to be effective for dissemination purposes. These are, however, not the types of outputs that experts are most encouraged to produce.

The project-bound communication plan, mentioned in the formal procedures, actually is never applied in practice. The consequence of this omission is that no structural thought is given to the possibility of addressing specific target groups using media specially geared to these groups, and no structural efforts are being made to do so.

KCE management states in this regard that output mainly consists of the reports that KCE has published on its website. KCE management believes that it is not KCE’s role to enhance the impact in other ways or through other types of outputs. It is important that decision-makers read and hear about the reports and results, but not by other means than the reports themselves. Other types of outputs enhance the chances of inducing “scandalizing discussions” in the press around KCE reports.

4.4.1.3 External views on outputs produced

The majority of the stakeholders interviewed became acquainted with KCE studies through reading the report, followed by the reading of recommendations and active involvement during the project (see table 4.14). Half of the stakeholders interviewed became aware of a KCE study by visiting the KCE website or through personal contact with KCE staff. The low scores for congress / course / poster may

57

be related to the fact that KCE staff does not do many presentations, though this has not been confirmed.

Table 4.14: How did your organisation become acquainted with KCE studies?

Selected

Written research proposal 7 Active involvement during project 12 Reading report 15 Reading recommendations 13 Reading an (journal) article 5 Citations in media/media coverage 9 Internet search 3 Visit to KCE website 10 Congress/course/poster 2 E mailing from KCE 7 Personal contact with staff 10 Heard from friend/colleague 2 [Multiple response]

With regard to the outputs produced, some external stakeholders strongly feel that the research report and the conclusions can be improved, for example by:

Having external organizations or experts formulate the conclusions;

Keeping the conclusions in a separate document from the study;

Ensuring that the conclusions are in line with the content of the report;

Reports (or results) should be published in scientific journals to provide room for discussion, similar to other scientific publications.

4.4.1.4 Reported outputs produced

In the period between 2004 and July 2009 (the time of this impact study), KCE produced and published 114 research reports and an equal number of press releases. All executive summaries of these reports and press releases have been published in both the French and Dutch language. From 2006 onwards, 41 reports have been published in English. In addition, five different seminar presentations and a conference document (plus proceedings) were published and centrally registeredlviii.

Project coordinators mentioned a total of 123 outputs which have been produced in the frame of KCE projects. The ones that were most frequently mentioned are presented in the table below. Besides the reports (mentioned 22 times), which are in fact common for every KCE project, the most frequently produced outputs are scientific articles (22 times), oral presentations at congresses or symposia (18 times), and articles or other publications (16) (see table 4.15)

lviii KCE, Export doclog 20090722_per doc.

58

Table 4.15: Outputs produced as a result of research projects

Outputs count

Scientific article 32 (Final) KCE report/Guidelines 22 Presentation (PowerPoint) at congress/symposium 18 Article/publication 16 Poster presentation 6 Feedback 5 Summary/abstract 4 Information brochure/handouts 4 Chapter in (year)book 3 Doctoral thesis 3 Other 10

From a report generated by KCE, it shows there has been an increase of published articles in 2009lix. This seems to contradict the view of some KCE experts, who consider the time to write articles as insufficient and state less peer reviewed articles are being produced. It is not clear what the cause of this contradiction is. A possible explanation might be that the articles published in 2009, are produced earlier and that KCE experts refer to a more recent time constraint. In addition, the number of experts increased progressively which implicates that the number of publications per expert might have stayed equal.

Figure 4.2 Journal articles produced by KCE

PubMed 2009‐11‐18

0

2

4

6

8

10

12

14

2004 2005 2006 2007 2008 2009

Outputs produced: key findings

The general communication strategy mentions the following possible outputs: reports (including summaries), Web-based dissemination of the report and results; and specific outputs as part of a project-bound communication plan.

Reports are always being produced. Internal experts judge report summaries as not always easily understandable.

The issuing of press releases is well guided and the releases are of good quality according to KCE experts. However, press releases sometimes receive little attention from the public.

KCE experts argue they write few scientific articles because there is no time reserved for this task. An overview of published articles shows an increase of published articles in 2009.

Project-bound communication plans have never been developed, resulting in very limited development of specific outputs for specific target groups. This is

lix Source: Pubmed November 18th 2009

59

despite the fact that KCE experts expect other outputs to be highly effective in reaching specific target groups.

KCE management is reluctant to produce outputs other than reports and press releases because they want to avoid one-sided discussions in the media about KCE reports.

In the period between 2004 and July 2009 (the time of this impact study), KCE produced and published 114 research reports and an equal number of press releases.

Besides the reports the most frequently produced outputs are scientific articles, oral presentations at congresses or symposia and other publications.

Outputs are generally not specifically tailored towards the specific situation and needs of target users (with the exception of guidelines developed as an output of Good Clinical Practice projects).

4.4.2 Dissemination activities performed

4.4.2.1 Procedures regarding dissemination and communication

Due to its public character, KCE is obliged to publish all results and policy recommendations from its studies, provided that the latter have been approved by the Board of Directors.

In general, KCE aims to be an organization that applies maximum transparency and is particularly focused on communication to proposal submitting organizations and the scientific worldlx. The communication of reports is to be organized in a transparent and professional way through collaboration with existing information channels at RIZIV/INAMI, the Federal Government Services and the Belgian Center on Farmacotherapeutical Information etc.35 Communication to the public at large, although not being a core task of KCE, is mainly organized through interactive web

publishing. The KCE Communication strategy lxi provides information regarding the communication and dissemination procedures within individual projects. It states how to communicate what messages, to which target groups these are to be communicated, what is to be the division of responsibilities and how the communication process should be organised.

The KCE Board of Directors and /or management team are accountable for and involved in the communication to nearly all target groups. The only exception is reporting findings and scientific messages to the scientific world, for which the management team is also accountable but the actual communication is done by the scientific staff (researchers and project leaders). The scientific staff is not automatically involved in the communication to clusters of target users: ‘the political world, actors (hospital and professional organisations, etc.), and patients/citizens’. With regard to the cluster of institutional originators, the scientific staff is involved in the preparation of reporting, scientific and policy messages, but not in the actual communication activities.

If a project-based communication plan is being made, the basics of this communication plan (target groups, message and communication channel) should in principle be included in the project-form (process 3).

Between 2004 and 2008, KCE has had 1 FTE on average dedicated to Knowledge management and 1 FTE to HR and Communication advice. The expenses on communication vary substantially per year; it has not been possible to give clear figures within this impact study.

4.4.2.2 Internal views on dissemination activities

Responsibility

According to the KCE experts, they themselves do not have any responsibility for the dissemination of the results. This is considered to be the sole responsibility of the management team. However, the KCE experts do not judge their limited role in the dissemination process positively.

lx Proces Document 9, Proces Adoptie Communicatie. Bijlage bij Ramaeckers, D. and Closon, J.P. (2003). KCE

Management Plan 16 oktober 2003. lxi KCE, process nota PN18.134; KCE communicatie strategie(v031024). Bijlage bij Ramaekers, D. and Closon,

J.P. (2003). KCE Management Plan 16 oktober 2003 35

60

Dissemination strategy

The experts feel that a proactive strategy for dissemination and feedback is lacking. They concur that sometimes they do not even know what kind of or how much impact their reports have and how recommendations or guidelines are picked up at practice level. They only get some incidental feedback, for example when people call to consult the expert. The experts therefore propose that a more proactive and systematic dissemination strategy should be developed, based on reflection, through which the impact of KCE reports could be improved.

First of all, they wish to be kept informed of what happens to their reports and to be more aware of the extent of the impact their reports have. For example, what time and investment is put into the dissemination, and who is on the mailing lists.

They would also enjoy putting more energy into dissemination themselves, as it would give them more honour and satisfaction from their work to be aware of and influence the way in which their report is being received and used. Possibly, making a difference in the dissemination strategy between guidelines and policy-oriented information would be helpful.

Giving presentations and lectures

KCE experts adhere to the view that attending and giving presentations or lectures at conferences is particularly contributing to an effective dissemination of research results to decision makers and the wider public.

Occasionally, KCE experts give presentations on scientific congresses. There is no consistency in this regard, and the KCE experts feel there are no criteria concerning the congresses KCE should visit to strengthen their impact, and the congresses not necessary to visit. Furthermore, in the past possibilities to give presentations were more often discussed and the experts were much more involved. The KCE experts agree that nowadays, presentations are mostly done on invitation by policy makers. Therefore they feel that KCE should perform a structural identification and strategic selection of relevant conferences in order to proactively apply for a lecture.

KCE experts relate this change in strategy to the change of directors; it is felt that the new directors are more reluctant to allow individual KCE experts to speak in public. In fact, as one expert notes, there is increasingly a policy of discouraging the KCE experts to talk in public.

The KCE management state in this regard that when an expert receives a demand for a conference or public presentation, the expert asks for authorization from the Direction. This is normally granted when the expert is asked to present a report or to give clarifications. It is not granted when the expert is asked to give an opinion on the topic.

KCE experts consider this kind of proactive approach, where KCE proposes to present themselves and their work to the target group, as a useful opportunity to explain the research findings and the reasoning behind the results and recommendations.

Another possibility is to host presentations at KCE, as has been done previously in a few GCP projects (presentations have been held by KCE in the auditorium where all stakeholders and professionals were invited). These presentations are considered to have more impact than for example press articles, because they allow a direct discussion, explanation and nuancing of the findings. The KCE experts feel that such presentations may be effective both for projects that attract a broader interest, as well as for smaller target groups, such as specialists.

4.4.2.3 External views on dissemination activities

Table 4.16 shows (multiple) responses from the interviewed 20 stakeholders to the question on how their organization is informed about KCE activities, in order of frequency. The majority is informed about KCE activities through the website and / or mailing list, followed by information in the general press.

One stakeholder comments in this regard: “KCE relies too much on the website. They seem insufficiently aware of the richness of their studies. KCE should increase the awareness and publicity of the studies, which are of general importance to the health care providers. The effect is weak if one has to search for the studies”.

61

Table 4.16: How stakeholders are informed on KCE projects

count

Website 12 Mailing list 11 General press/News media 7 Through KCE Board of Directors 3 Professional/ medical journals 2 Direct contact on initiative of stakeholder 1 Informal contacts 1 KCE news 1 [Multiple response]

4.4.2.4 Reported dissemination activities

Besides the production of the report, press release, and the other outputs as previously mentioned, project coordinators mention that other dissemination activities have been undertaken by the project team, such as:

Giving a presentation (50,8%);

Face to face contact with the target group (31,7%);

Appearing in the media (27,5%);

Giving a lecture (25,0%);

Placing messages on other than KCE websites (20,0%), such as INAHTA and the Cancer Register.

Dissemination activities: key findings

Most communication tasks are being carried out by the KCE management and directors, as well as the communication staff. The responsibility of KCE’s experts is limited to disseminating the research results into the scientific field.

The experts feel that KCE lacks a proactive strategy for dissemination and feedback on the actual take up and use of the reports. They wish to be kept informed about what is being done with their reports and they would enjoy putting more energy into dissemination.

The experts perceive their limited role in dissemination activities as having a negative impact on their awareness of and their influence on the ways their reports are being used.

KCE’s policy regarding presentations and lectures is illustrative for the way in which KCE designs it’s disseminative role: although it is seen as a particularly effective way of communicating findings and organizing discussions on topics, no policy exists on when en where experts should try to participate. Presentations are mostly done on invitation and they are not allowed to participate in discussions or opinion forming.

Despite the limited role of experts in dissemination, half of the project coordinators state a presentation about the research has been given as part of the project’s dissemination activities.

The majority of stakeholders interviewed are being informed about KCE activities through the website and / or mailing list, followed by information in news media.

4.4.3 Stakeholders addressed by outputs

Procedures on stakeholders addressed by outputs

KCE Management states that in communicating objective elements and analyses (research reports), KCE should direct itself tolxii:

1. The government;

2. The health care providers;

3. The health care institutions; lxii Ramaeckers, D. and Closon, J.P. (2003). KCE Management Plan 16 oktober 2003. p. 9-1035

62

4. The patients;

5. The insurance institutions and companies;

6. The pharmaceutical industry and producers of medical materials;

7. The citizens;

8. International organizations.

The general KCE communication strategy determines that the communication from KCE is specifically directed at:

1. The proposal submitting organizations;

2. The scientific world; and

3. Other target groups as identified beforehandlxiii.

4.4.3.1 Internal view on stakeholders addressed by outputs

Some KCE experts explain that, although it may not be stated explicitly in every report, the law regarding the constitution of KCE defines all policy makers as the target audience for “independent policy support”. They also acknowledge, however, that there may sometimes be a distinction between the target audience and impact of KCE in general and of individual projects. In particular, the target users of a project output may be more specific and more narrowly defined than the target groups of KCE as an organisation, than currently the case.

Although citizens - according to the procedures - are one of the groups at which KCE focuses itself in communicating objective results, the KCE management in fact does not consider this the main target group for KCE. This is despite the fact that Belgium currently has no independent institution that provides the public with general and specific information about the health care system, available health care services and developments possibly relevant to them.

4.4.3.2 External views on stakeholders addressed by outputs

Some stakeholders strongly feel that dissemination and communication by KCE could be improved with regard to the stakeholder groups that are actually being reached. For example:

Physicians and general practitioners are insufficiently reached and convinced by KCE studies; guidelines are generally not taken seriously and not being used in the “Local Quality Groups”lxiv;

At the moment, KCE is hardly known, not even with members of the parliament;

It is not wise to rely too much on the press, because they do not always spread the message right;

At present, insufficient information is fed back to the hospitals Suggestions by stakeholders to improve KCE’s reach of target groups involve:

More communication about KCE and project findings to be directed towards the general public;

Increase the awareness of KCE amongst politicians and all actors in the health care sector. For example by more proactively sending a summary to all actors in the health care sector, such as professional organizations, who can copy and distribute it amongst their members;

Undertake more visits to the field to improve the communication;

After finalization of every project, provide for an information session for medical professionals and (if relevant) other affected stakeholders;

After every report, a brief overview of useful elements for practical use should be developed (between a summary and the full report).

lxiii KCE Process note 18-134; KCE Communicatiestrategie (v031024). Bijlage bij Ramaeckers, D. and Closon,

J.P. (2003). KCE Management Plan 16 oktober 200335. lxiv Lokale kwaliteitsgroepen

63

Stakeholders addressed by outputs: key findings

In general, KCE procedures name all organizations and groups at which the communication about project results should be focused on. However, there is no procedure to address specific organizations and groups as target audiences for specific outputs within projects.

In spite of the general procedures, KCE management does not consider it their main responsibility to inform the general public about research results and possible relevance for citizens.

Some stakeholders argue that significant improvements can be made concerning the stakeholder groups that are actually being reached through the outputs produced within studies. Suggestions for improvement boil down to the advice to make different outputs designed to tailor the specific needs of stakeholders and a more proactive delivery of these outputs.

4.4.4 Engagement of stakeholders in outputs and dissemination

4.4.4.1 Procedures on engagement of stakeholders in outputs and dissemination

The KCE Communication strategylxv only speaks of stakeholder involvement in the development of outputs and dissemination activities in those cases where a project-bound communication plan exists. The strategy states that after the study has been approved, scientific and communication staffs are to consult with the proposal submitting organization and the target groups about the further development of the communication plan. Subsequently the communication plan is being further elaborated by the communication staff. The implementation of the project-bound communication plan is the joint responsibility of communication staff, scientific staff, management and IT. The target groups should also be invited in the implementation. After the concrete organisation of the joint communication initiatives together with the organisations that submitted the proposal, all forms of further communication are expected to be arranged by the target groupslxvi.

4.4.4.2 Internal view on engagement of stakeholders in outputs and dissemination

During the group interview KCE experts argued that within KCE dissemination is considered to be the responsibility of the communication staff and KCE Directors. Therefore, they themselves do not participate in the dissemination process and are not able to influence the extent to which and the way in which stakeholders are being involved in the dissemination.

KCE management states that the type of involvement of stakeholders in dissemination depends on what the report is aimed at. For reports that produce guidelines, stakeholders are generally involved in dissemination; for others not necessarily. There has been a trial to produce a guideline in interactive format on the website. However, it was perceived that this would be hard to use, as doctors need to invest a lot of time in filling in data electronically. This was therefore not a successful experience.

4.4.4.3 Reported stakeholder engagement in outputs and dissemination

Engagement in the production of outputs

According to the project coordinators, target groups were involved in the production of 15 out of the 123 outputs (just over 10 percent). In about half of these outputs, medical professionals were involved as target group. In addition, RIZIV/INAMI was mentioned three times (once in combination with medical professionals). Other target groups that were mentioned once are: the minister of Health, a Red Cross director, the medical industry, the Belgian Hospital Hygiene Platform, Pharmaceutical and scientific representatives.

lxv KCE, process nota PN18.134; KCE communicatie strategie(v031024). Bijlage bij Ramaekers, D. and Closon,

J.P. (2003). KCE Management Plan 16 oktober 200335 lxvi KCE, 09 Process doc adoptie communicatie(20031024). Bijlage bij Ramaekers, D. and Closon, J.P. (2003).

KCE Management Plan 16 oktober 200335.

64

For nine outputs, the target group had been invited by KCE, the internal project team or the external project team. Two times, the target group had been involved on their own initiative. Four times, the involvement originated otherwise.

Engagement in dissemination activities

The project coordinators state for 40 percent of the projects that target groups or stakeholders have been involved in dissemination activities. For 34.2 percent, no involvement of target groups in the dissemination is reported. Over a quarter of the coordinators (25.8 percent) do not know whether stakeholders were involved in the dissemination activities. The activities in which stakeholders were involved most frequently are (multiple-response):

Giving (or receiving)lxvii a presentation;

Face to face contact with target groups or stakeholders

Giving a lecture;

Placing messages on other than KCE websites;

Appearing in the media; and

Organizing a conference.

The types of stakeholders involved in these dissemination activities are predominantly (multiple response):

Medical professionals (15 times);

Professional associations (7 times);

Scientific associations and Medical Colleges (7 times);

NIHDI (5 times);

Political actors (5 times).

According to the project coordinators, the stakeholders who were involved in dissemination were involved on the initiative of the stakeholders themselves in 41.7 percent of the projects. In another 22.9 percent of the projects, stakeholders were invited by KCE or the internal project team. In 12.5 percent of the projects the stakeholders were invited by the external project team, and in 22.9 percent of the projects, stakeholders became involved in some other way.

Engagement of stakeholders in outputs and dissemination: key findings

General procedures for the engagement of stakeholders in outputs and dissemination do not exist within KCE. In the development of project-bound communication plans, stakeholder involvement plays an important role. However, these are never developed.

KCE experts do not exert any influence on the extent to which and the way in which stakeholders are being involved in producing outputs and disseminating findings. This is the responsibility of the communication staff and Directors.

According to the KCE management, stakeholders are generally involved in dissemination activities concerning reports that produce guidelines; for other kinds of projects involvement is not common.

Involvement of stakeholders in the production of outputs does not occur often (about 10 percent of the projects). Stakeholder involvement in dissemination activities occurs more often (40 percent of the projects).

Over a quarter of the project coordinators do not know whether stakeholders had been involved in dissemination activities. This fits in with the observation that experts do not have any formal responsibility concerning dissemination.

Dissemination activities in which stakeholders are being involved are generally ‘live’ activities such as giving/organizing a lecture, conference or presentation. Medical professionals are the stakeholder group most often involved in producing outputs and dissemination activities.

For the production of outputs, stakeholders generally are being invited by internal or external project teams while for dissemination activities stakeholders appear to participate more often on their own initiative.

lxvii

From the explanatory comments, it can be concluded that in several cases, presentations were given to the stakeholder, instead of the stakeholder giving the presentation.

65

4.4.5 Actual take up of outputs

Document download counts

The total download count for all of the 114 KCE research reports published between 2004 and 22 July 2009, is 283 987 (excl. supplements, which are counted separately). On average, reports (excl. supplement) have been downloaded 2 691 times per report over the period between 22 July 2004 and 22 July 2009. The 24 published supplements are on average downloaded 758 times per supplement.

Table 4.18 shows that reports published in 2009 have a much lower average download rate. This is due to the fact that these reports were published recently and some had been published for less than a month at the time of the download count. This is evidenced by the comparatively high average download count per month for reports published in 2009. One may also assume that after a year or two, the interest in a report diminishes over time if the attention is not specifically drawn towards it. This partly explains the diminishing average download counts per month for reports published in 2004 and 2005.

Of the research fields, particularly OEC and GCP reports appear most popular. lxviii The average downloads are lowest for EPB and HSR, while HTA reports are in the middle. On average, the average download count for reports that are also published in an English version is somewhat higher than for reports with no English version of the report.

Table 4.18: Average download counts per report

Nr. of reports

Av. Downloads

Av. Downloads per

Month

Overall 114 2.77.6 130 2004 7 3.786 67 2005 17 2.507 52 2006 26 4.125 121 2007 19 2.783 114 2008 27 1.618 134 2009 18 614 252 HTA 44 2.514 GCP 29 3.274 HSR 35 1.811 EPB 3 1.276 OEC 3 3.742 English version also published 42 2.735 No English version published 63 2.349

The most popular report is the GCP Report 48 on chronic low back pain (2006, GCP), which was downloaded 13 433 times,36 followed by the GCP report 6 “National guideline for prenatal care” with 12 910 downloads.37 Other popular reports with over 9 000 downloads are: KCE report Vol. 10 The costs of dental prosthesis (2004, GCP);38 Vol. 36 Pharmacological and surgical treatment of obesity. Residential care for severely obese children in Belgium (2006, HTA);39 and Vol. 51 Chronic care requirements for persons with congenital brain injury between 18 and 65 years (2007, HSR).40

Less popular reports, with less than 700 downloads (excluding 2009 publications) and below 60 counts on average per month are:

Vol. 16 – Study of the possible costs of a potential change in legal regulations regarding medical liability. Phase II: development of an actuarial model and first estimates (2005, HSR);41

lxviii Until July 2008, the KCE issued reports in four fields: HTA, GCP, HSR, EPB (Equity and Patient Behaviour)

and OEC (Objective Elements - Communication). KCE report 192, 503 and 804 belonged to the EPB field, while KCE reports 305 and 416 were issued under the label OEC. In 2008, the KCE decided to limit the number of research fields to HSR, GCP and HTA by issuing future EPB and OEC reports under the HSR label.

66

Vol. 68 – Damage charges as a result of health care -Phase IV Financial division between the Fund and the insurers (2007, HSR);42

Vol. 79 – Reimbursement of radio-isotopes in Belgium (2008, HSR);43

Vol. 81 – Quality of rectal cancer care phase 2: development and test of a set of quality indicators (2008, GCP).44

Actual take-up of outputs: key findings

On average, reports have been downloaded 2691 times per report (over the period between 22 July 2004 and 22 July 2009). That is an average of 130 downloads per month.

Of the research fields, particularly OEC and GCP reports appear to be most popular.

Issuing an English version of the report appears to increase the average download count.

The most popular report is the GCP Report 48 on chronic low back pain (2006, GCP), which was downloaded 13 433 times 36.

4.5 CONTRIBUTION TO THE DECISION MAKING PROCESS In this paragraph, our focus is on the extent to which and the way in which KCE has been able to make their research projects contribute to decision making processes in policy and practice, as well as on their impact to the scientific field.


1. Relevance of projects for policy making or practice

2. Feasibility and usefulness of the recommendations

3. Timely issuing of the reports

4. Degree and type of contribution to health care policy and practice

5. Reactions on finished projects from stakeholders

Because this paragraph discusses results rather than processes and opinions, the findings presented in this paragraph are mainly based on the outcome of self report questionnaires for internal and external project coordinators and (to a lesser extent) on the stakeholder interviews.

More in depth information on contribution to the decision making process can be found in paragraph 4.7 on case studies.

4.5.1 Relevance of projects for policy-making or practice

The question whether KCE projects are considered relevant for policy making or practice is answered by project coordinators (internal and external) and stakeholders. In this paragraph we present both perspectives. Stakeholders were asked whether the research agenda meets their information needs. This is a broad question with a variety of open answers, not always related to specific research projects, and often based on a general impression. Project coordinators answered the question “how do you judge the relevance of the research findings”, subdivided in a) policy making, b) health care practice and c) scientific field. The answers of the project coordinators are related to specific research projects.

4.5.1.1 External views on relevance of research projects

“Do the topics KCE has put on its research agenda so far answer the information need of your organization?” When asked this question, 14 out of the 20 stakeholders interviewed answered “yes”. Respondents who say yes refer to the relevance of topics for their specific field of interest (for example the clinical field) and the fact that most of their submitted proposals have been granted.

Five stakeholders answered no. These stakeholders give the following clarifications for their negative reply:

- The specific field of interest of the stakeholder’s organisation is insufficiently presented in the KCE research agenda. Examples of topics from this specific field of interest mentioned by stakeholders are: “added value in pharma-economical sense of the pharmacist” and “topics on the scientific domain of the general practitioner”.

67

- Some research projects are too academic or too theoretical and have little relevance for the efficiency of Belgian health care as a whole.

4.5.1.2 Internally reported relevance for policy making or practice

Relevance of the research findings for policy development

From the self report questionnaire it appears that project coordinators are predominantly positive regarding the relevance of the results of their research for policy development. A majority (61.9%) states that the relevance of the research results for policy development is (very) high. About 27 percent of the respondents judge the relevance of the research results as average; about 10 percent of the respondents judge this to be (very) low.

Table 4.19: Relevance of research results for policy making

Rating Count % Very high 26 20,6%

High 52 41,3%

Average 34 27,0%

Low 11 8,7%

Very low 1 ,8%

Do not know 2 1,6%

Total 126 100,0% [Question was: “How would you rate the relevance of research results for policy making?”]

Respondents judging the relevance for policy development of their project as low or very low, give the following explanations for this answer:

Problems with the research design or methodology (could not find a methodology that could answer the initial research question, or missing the appropriate data for answering all research questions)

Limited applicability of findings (only relevant for certain professionals or administrative levels)

No direct implementation possible (further regulations must first be developed on the basis of the findings).

No added value (already a strong consensus on the topic, the KCE research findings did not change this)

Priority setting in policy arena (not enough means and manpower to put the recommendations into practice).

Interesting to note (see table 4.20) is that external coordinators are generally more positive in their judgement regarding the relevance of the project for policy making (68.4% high or very high) than the internal KCE coordinators (59.1% high or very high). No substantial differences were observed between projects from different research fields (HTA, HSR and GCP) or different years of project acceptance.

Table 4.20: Relevance of research results (internal and external

Internal External

Count % Count % Very high 16 18,2% 10 26,3%

High 36 40,9% 16 42,1%

Average 26 29,5% 8 21,1%

Low 8 9,1% 3 7,9%

Very low 1 1,1% 0 -

Do not know 1 1,1% 1 2,6%

Total 88 100,0% 38 100,0% [Question was “How would you rate the relevance of the research results of this project for policy making”]

68

Relevance of the research findings for health care practice

Just over half of the respondents judge the relevance of their project for health practice (very) high. About a quarter considers the relevance for health practice as average, while 15.8 percent regards the relevance for health care practice to be (very) low (see table 4.21).

Table 4.21: Relevance of research results for health care practice


High 46 36,5%

Average 32 25,4%

Low 13 10,3%

Very low 7 5,6%

Do not know 5 4,0%

Total 126 100,0% [Question was: “How would you rate the relevance of research results for health care practice?”]

Seven out of the twenty respondents who judged their projects’ relevance for health care practice as low or very low, say the findings are not applicable to practice. Other clarifications given are:

It is not in the (financial) interest of the health care field to change their behaviour, so the research findings are of low relevance.

The data needed were not available, so just a part of the research questions could be answered.

As can be seen in table 4.22, the external are more positive in their judgement (57.9% high or very high) than internal project coordinators (53.4% high or very high). However, here the difference is less substantial than in their judgement on relevance for policy making.

Table 4.22: Relevance of research results for health care practice (internal and external)

Internal External

Count % Count % Very high 13 14,8% 10 26,3%

High 34 38,6% 12 31,6%

Average 21 23,9% 11 28,9%

Low 11 12,5% 2 5,3%

Very low 6 6,8% 1 2,6%

Do not know 3 3,4% 2 5,3%

Total 88 100,0% 38 100,0% [Question was “How would you rate the relevance of the research results of this project for health care practice”]

When the responses are disaggregated to research field one sees that, in line with what might be expected in this regard, the relevance of Good Clinical Practice projects for health care practice is rated higher (77.5% high or very high) than the relevance of Health Technology Assessment projects (57.2% high or very high). Health Services Research (38.3% high or very high) projects score lowest on relevance for health care practice (see table 4.23).

69

Table 4.23: Relevance for health care practice

HTA GCP HSR

Count % Count % Count % Very high 11 26,2% 10 25,0% 2 5,9%

High 13 31,0% 21 52,5% 11 32,4%

Average 8 19,0% 7 17,5% 16 47,1%

Low 6 14,3% 2 5,0% 3 8,8%

Very low 4 9,5% 2 5,9%

Total 42 100,0% 40 100,0% 34 100,0% [Question was: “how would you rate the relevance of research results of this project for health care practice?”]

Relevance of the research findings for the scientific field

In general, project coordinators rated the relevance of the research findings for the scientific field lower than the relevance for health care policy-making or practice. Just 5.6 percent judged the relevance for the scientific field very high and 37.3 percent considered the relevance high. 21.2 percent of the coordinators considered the relevance for the scientific field to be (very) low (see table 4.24).

Table 4.24: Relevance of research results for scientific field


High 47 37,3%

Average 45 35,7%

Low 20 15,9%

Very low 7 5,6%

Total 126 100,0% [Question was: “How would you rate the relevance of research results of this project for the scientific field?”]

Eleven out of the 27 respondents stating that the relevance of their project for the scientific field is low or very low clarify that the scientific quality of their research is not high enough compared to international quality standards. Other clarifications given refer to the specific nature or goal of the study (summarizing only existing knowledge or showing gaps in current scientific knowledge) which causes the study not to come up with ‘new’ findings on the subject.

As was seen in the previous judgements, external project coordinators are slightly more positive (50% high or very high) about the relevance of the research results for the scientific field than internal KCE coordinators (39.7% high or very high) (see table 4.25).

Table 4.25: Relevance of research results for scientific field (internal and external)

Internal External

Count % Count % Very high 4 4,5% 3 7,9% High 31 35,2% 16 42,1% Average 30 34,1% 15 39,5% Low 16 18,2% 4 10,5% Very low 7 8,0% 0 0% Total 88 100,0% 38 100,0%

[Question was “How would you rate the relevance of the research results of this project for the scientific field?”]

70

Relevance of research projects: key findings

14 out of the 20 stakeholders interviewed feel that the topics KCE has put on its research agenda so far answer the information needs of their organization.

A majority of the project coordinators state that the relevance for policy development of the research results from their project is (very) high.

The relevance of the projects might be influenced negatively by problems within the research project, limited applicability of the findings, no possibility for direct implementation, any added value or priority setting in the policy arena.

Just over half of the project coordinators judge the relevance of their project for health care practice as (very) high. The other project coordinators mostly feel their findings are not (directly) applicable to practice.

In general, project coordinators rated the relevance of the research findings for the scientific field lower than the relevance for health care policy-making or practice. Most respondents rating the relevance as (very) low, state that the scientific quality of their research does not meet international quality standards.

4.5.2 Feasibility and usefulness of the recommendations

4.5.2.1 Reported feasibility and usefulness of the recommendations

The feasibility and usefulness of the recommendations of their project was predominantly rated (very) high (63.5%) by project coordinators. Just 6.4 percent rated the feasibility and usefulness of their recommendations as (very) low.

Table 4.26: Feasibility of recommendations


High 61 48,4%

Average 31 24,6%

Low 6 4,8%

Very low 2 1,6%

Do not know 7 5,6%

Total 126 100,0% [Question was: “How would you rate the feasibility/practical usefulness of the recommendations of the project?”]

The difference between internal KCE project coordinators and external coordinators in their rating was very small. The same accounts for differences between the years of project acceptance (2006-2008).

Table 4.27 shows the scores for the rating of usefulness and feasibility of projects per research domain (HTA, GCP, and HSR). There are some differences to be seen between the three main research domains: HSR projects are scoring somewhat less positive (57.6% high or very high) on feasibility and usefulness of the recommendations than GCP (71.1% high or very high) and HTA (73.8% high or very high).

71

Table 4.27: Feasibility of recommendations

HTA GCP HSR

Count % Count % Count % Very high 7 16,7% 8 21,1% 3 9,1%

High 24 57,1% 19 50,0% 16 48,5%

Average 9 21,4% 8 21,1% 11 33,3%

Low 2 4,8% 2 5,3% 2 6,1%

Very low 0 0% 1 2,6% 1 3,0%

Total 42 100,0% 38 100,0% 33 100,0% [Question was: “how would you rate feasibility/practical usefulness of the recommendations of the project?”]

Feasibility and usefulness of recommendations: key findings

In general, a majority of the project coordinators judges the feasibility and usefulness of their recommendations to be (very) high.

HSR projects are judged somewhat less positively regarding feasibility and usefulness of the recommendations than GCP and HTA projects.

4.5.3 Timely issuing of the report

4.5.3.1 Internal view on timely issuing of the report

From the group session it became apparent that KCE experts have varying experiences concerning the timeliness of their reports.

In principle the planned moment of for decision making is an important factor in their research. However, sometimes they cannot avoid the project drawing out which causes the report to be finalized too late for decision makers to be useful in their decision process. This is partly because policy makers do not always have the opportunity to wait for the delayed conclusions, as for instance the ministers are often under pressure of industry and pressed for action. This forces them to go through with decision making without the ability to draw upon the relevant facts. On the other hand, it also happens that decision-makers wait for the research findings (for example when substantial financial costs were involved with the decision).

4.5.3.2 Reported timeliness of issued reports

Findings from the self report questionnaire point out that the vast majority of the project coordinators (81,7%) confirmed that the final report was issued timely enough for the target user(s) to be able to implement the recommendations (see table 4.28).

Noteworthy is the fact that over a tenth (11.9%) of the coordinators does not know whether the publication of the final report was timely enough to implement the recommendations.

Table 4.28: Was the report published timely

Rating Count % Yes 103 81,7% No 8 6,3% Do not know 15 11,9%

Total 126 100,0% [Question was: “Was the publication of the final report in your view timely enough to implement the conclusions or recommendations?”]

The difference between internal and external coordinators was marginal, with internal KCE coordinators being slightly more positive about timely publication of the final report. Furthermore, external project coordinators were somewhat more unknowing about the timeliness of their report (15.8%) than internal project coordinators (10.2%).

72

When looking at the different research domains, it turns out that the percentage of positive responses on timeliness for GCP projects is 100 percent, compared to around 90 percent for HTA and HSR projects (excluding “do not know” responses).

Only minimal differences were found in responses between different years of project acceptance (excluding respondents who answered “do not know”).

Timely issuing of the report: key findings

KCE experts acknowledge the importance of timely reports for policy makers. However, this is not always possible.

Findings from the self report questionnaire point out that the vast majority of the project coordinators confirmed that the final report was issued timely enough for the target user(s) to be able to implement the recommendations.

4.5.4 Degree and type of contribution to health care policy and practice

4.5.4.1 External view on contribution to health care policy and practice

Stakeholders were asked to what extent and in what way their organization has been influenced by the findings from KCE studies. Four stakeholders (from respondent-clusters government, insurers and ‘other’) state that until now KCE did not influence decision-making in their organisation at all. Answers of the other respondents amount up to the following remarks:

The degree of actual use or implementation of findings is not only depending on the quality, usefulness and timeliness of the KCE report but also on other factors such as other factual sources or the interests of stakeholders.

Impact is a slow process: stakeholders in the field of health care practice show a gradually growing interest in KCE reports and findings. Personal experience with KCE (e.g. cooperation during research projects) positively influences the degree of interest in KCE studies.

Findings from well-conducted studies are being used more often, especially if the recommendations are clearly formulated and ready to implement.

The degree of actual use or implementation depends on the specific research subjects. Specific studies have been dispersed in high numbers by professional associations, while others weren’t.

The degree of actual use may vary even within a research project (e.g. the findings relating to financial aspects might have a strong influence in whether or not something is reimbursed, while other findings are less influential).

4.5.4.2 Reported contribution to health care policy and practice

In data collection and analysis on the reported contribution of KCE studies to policy development we have made a distinction between:

A. Contribution to health care policy development

B. Contribution to changes in medical practice

C. Impact on the scientific community

A. Contribution to health care policy development

A small majority of the coordinators is convinced that the project has, in a way, contributed to health care policy development. About one fifth of the respondents state they do not know whether the project contributed to health care policy development (see table 4.29).

Table 4.29: Contribution to policy making

Count %

Yes 73 `57,9% No 26 20,6% Unknown 27 21,4%

Total 126 100%

73

Question was “Has this project, according to you, in any way contributed to policy making in the field of health care?”

External coordinators are substantially more positive than the internal coordinators about the contribution of the project to health care policy development (see table 4.30).

Table 4.30: Internal and external project managers on contribution to policy making

Internal External

Rows Count Percent Count Percent

Yes 47 53,4 26 68,4 % No 20 22,7 6 15,8 % Unknown 21 23,9 6 15,8

Total 88 100% 38 100%

Coordinators of HTA projects are more positive about the contribution of their project to health care policy development than coordinators of GCP projects. Coordinators of HSR projects are most critical concerning the contribution of their project to health care policy making. The table below shows the division of responses (excluding “don’t know” responses) for the three domains (see table 4.31).

Table 4.31: contribution to health care policy making (per field)

HTA GCP HSR

Rows Count % Count % Count %

Yes 26 83,9 % 22 71,0 % 20 64,5 % No 5 16,1 % 9 29,0 % 11 35,5 %

Total 31 100 % 31 100 % 31 100 %

Type of contribution

To the 72 coordinators that said that their project did indeed contribute to policy making, we asked to choose the way(s) in which the results were used (multiple- response question). In table 4.32, the results are summarized. The options have been ranked in order of frequency.

A small majority of coordinators state that the results have been used in policy negotiations and/or to underpin existing policy intentions or policy preparations. Over a third of the projects (also) resulted in the implementation of evidence based policy, have been cited in policy documents, and/or have resulted in the involvement of health care users and/or the general public. In one out of every five cases, the results were (also) used in parliamentary debates.

Table 4.32: How did the project contribute to policy making

Findings have been: Percent Used in policy negotiations (RIZIV/INAMI, Federal Public Services, Cabinet Social affairs & Health) 58,3 Used to support pending policy decisions 52,7 Used For Research informed policy (adoption of recommendations) 38,9 Cited in policy documents 37,5 Led to involvement of health service users and/or the public 33,3 Used in parliamentary debates and/or for parliamentary questions 20,9 Other 23,6

B. Contribution to changes in medical practice

Only 29.6 percent of the coordinators answered positively to the question whether the project had contributed to changes in health care practice. About one third of the coordinators said they did not know whether their project had contributed to changes in health care practice (36,8%) or that their project had not contributed to any changes in health care practice (33,6%).

74

In particular, the percentage that did not know about the contribution of the project to health care practice was higher amongst internal KCE coordinators (40.2%) than amongst external project coordinators (28.9%). Overall, the external project coordinators are marginally more positive than internal KCE project coordinators.

When KCE research domains are considered, coordinators are least positive about the contribution of HSR projects to changes in medical practice (26.3% yes). Responses on HTA projects and GCP projects are significantly more positive (53.3% and 55.6% yes).

Also, differences can be observed between projects from difference years of acceptance. For projects in 2008, only 27.3 percent of the coordinators found the project had already contributed to health care practice. For projects approved in 2006, a small majority (55.0%) of the coordinators stated the project had already contributed to health care practice. This is even higher for and for projects approved in 2007 (61.9%). It is possible that the impact on health care practice of projects approved in 2008 is still to improve.

TYPE OF CONTRIBUTION

Here also, we have asked the respondents who saw a contribution to health care practice (37 in total) to choose the way(s) in which the results of the project have led to changes in health care practice. The answers are summarized in table 4.33 in order of frequency. A change in the behaviour of medical professionals is the most frequently signalled contribution.

Table 4.33: How did the project contribute changes in health care practice?

Changes in Percent

Behaviour of medical professionals (e.g. method of working) 65,0 Management and/or organization of health care services 32,5 Health care facilities (e.g. equipment, new technologies) 21,7 Health care industry (e.g. price setting, supply of products) 18,9 Other 18,9

C. Impact on the scientific community

Overall, the coordinators were divided about whether the project had an impact on the scientific field. Just above a third (33,9%) did not know whether the project has achieved an impact on the scientific community. An equal proportion (33,9%) states the project had no impact on the scientific community. Just below a third (32,2%) states that the project did have an impact on the scientific community.

External project coordinators are more positive (40.5% yes) about the impact of the project on the scientific community, when compared to internal KCE project coordinators (28.7%). Amongst the internal KCE project coordinators, the percentage that does not know the impact of the project on the scientific community (39.1%), is higher than amongst external coordinators (21.6%).

The coordinators of HSR projects are least positive (29.2% yes) about the impact of the project on the scientific community. In contrast, coordinators of HTA and GCP projects are much more positive in this regard (58.1% and 59.1% yes) (excluding “don’t know” responses).

In line with the previous questions on impact, the coordinators of projects that were approved in 2007 (50.0% yes) and in 2006 (57.1% yes) are somewhat more positive than those of projects approved in 2008 (42.3%).

75

Degree and type of contribution to health care policy and practice: key findings

80 percent of the stakeholders agree that KCE reports have had some influence on their decision making processes

About 60 percent of the project coordinators state their project has made some contribution to policy making. Conversely, the reported contribution to health practice is significantly lower (about 30%). About one third of the coordinators do not know whether their project has contributed to changes in health care practice.

Reported impact on the scientific community is rather low compared to the other fields (approx. 30 percent).

Overall, external project coordinators more often report some contribution/impact on health care policy, practice and scientific field than internal project coordinators.

Overall, project coordinators from HSR projects are notably less positive about the contribution of their projects to health care policy, practice and the scientific field than coordinators from HTA and GCP projects.

The amount of impact appears to develop over the years, as the reported impact on the different fields is greater for projects approved in 2006/2007 than for projects approved in 2008.

A small majority of coordinators who signal some contribution to health care policy, state that the results have been used in policy negotiations and/or to underpin existing policy intentions or policy preparations.

A majority of coordinators who signal some contribution to health care practice, state that this has been in the form of a change in the behaviour of medical professionals.

4.5.5 Reactions on finished projects from stakeholders

4.5.5.1 Procedures on asking and receiving feedback

KCE has no procedures put in place to structurally and systematically gather feedback from stakeholders and target users of research findings. Also, there is no structural monitoring of feedback received.

4.5.5.2 Internal view on feedback received

In the group session, KCE experts argue that the feedback they receive on their projects is not always positive. The reason for this (as they perceive it) is that stakeholders often use incomplete information (e.g. press releases) to found their feedback. Because press releases sometimes do not give nuanced information, the reactions are not so nuanced either.

Stakeholders might for example argue that “because of your recommendations, no more money will be made available for further research” when this actually is not a recommendation in the report. Another example of negative feedback based on incomplete information about the project, is an accusation made about a certain study not being conducted ethically. In this case (as in most cases) the stakeholder sent its feedback directly to the press, without contacting KCE. Because KCE experts and management do not want to take part in the press debates, the only thing they can do is check their research and give an official reaction in which they further clarify the research.

Sometimes KCE also gets reactions from abroad, for instance from New Zealand where their guidelines had been implemented, but also from neighbouring countries such as the Netherlands.

4.5.5.3 Reported feed back from the health care field

Feedback from policy makers

The majority of coordinators received feedback from stakeholders in the health policy field about the quality or usefulness of their study (58.9%). The division of responses is approximately the same for both internal and external coordinators.

76

Concerning the different research domains, coordinators of HSR projects tend to more often receive feedback from stakeholder in the policy field (69.4% yes) than coordinators of HTA and particularly GCP projects (60,5% and 48.7% yes).

Concerning the different years of project acceptance, coordinators of projects approved in 2008 receive more feedback (71.8% yes) than those of projects approved in 2007 (55.2% yes) and 2006 (53.3% yes).

Feedback from the field of health care practice

The amount of feedback coming from the field of health care practice is about the same as the feedback coming from policy makers. Overall, 57.3 percent of the coordinators received feedback from health professionals and providers. The division is more or less similar for internal and external coordinators.

Concerning the different research domains, HTA and HSR projects show much resemblance with the amount of feedback from policy makers. However, a significant difference can be seen in GCP projects: as much as 64.1 percent of the coordinators have received feedback from health care professionals and providers.

Concerning the different years of project acceptance, coordinators of projects approved in 2007 received slightly more feedback from health professionals and providers (62.1%) than those of projects approved in 2006 and 2008 (54,5% and 53,8%).

Feedback from the scientific field

Feedback from the scientific field is clearly less common than from policy makers or from the field of health care practice: the majority of coordinators (61.3%) did not receive any feedback from the scientific field on their project. This is particularly the case for internal KCE project coordinators (64.4% compared to 54.1% from external project coordinators).

When observing the different research domains, one sees that the lack of feedback is greatest amongst project coordinators of HSR projects (75% no). A small majority (59,0%) of GCP coordinators did not receive feedback from scientists, while slightly more than half of the HTA coordinators actually did receive feedback from the scientific field 51,2% yes).

Reactions on finished projects from stakeholders: key findings

KCE has no procedures regarding gathering feedback or monitoring received feedback.

KCE experts argue that the feedback they receive is often based on incomplete information. This sometimes causes unfounded criticism and press debates.

The majority of coordinators received feedback from policy makers about the quality or usefulness of their study (58.9%). The amount of feedback coming from the field of health care practice is about the same.

Coordinators of HSR projects tend to more often receive feedback from policy makers and the field of health care practice than coordinators of HTA projects. GCP projects receive significantly more feedback from the field of health care practice than from policy makers.

Feedback from the scientific field is clearly less common than from policy makers or from the field of health care practice: the majority of coordinators did not receive any feedback from the scientific field on their project. The lack of feedback from this field is greatest amongst HSR projects.

4.6 EXTERNAL IMAGE OF KCE This paragraph shows the external image of KCE, by focusing on the way in which stakeholders view KCE in terms of general reputation and their judgment on certain quality criteria.


1. General reputation of KCE

2. Judgment on responsiveness

3. Judgment on research quality

77

KCE’s added value compared to other organizations.

4.6.1 General reputation of KCE

The analysis framework divides the general reputation of KCE into to four subtopics:

Trustworthiness

Authority (knowledge and expertise on health care issues)

Degree of independence

Helpfulness

All four aspects have been addressed in the stakeholder interviews. We asked stakeholders how highly they rate KCE on these items, using a nominative 5-point scale ranging from very low tot very high.

Trustworthiness

Of the 19 stakeholders who answered this question, 16 percent consider the trustworthiness to be low or very low. The others consider the trustworthiness to be average (37%) or high (42%). Only one respondent (5%) judges it to be very high.

Multiple respondents state that although the studies in themselves are of good quality, the conclusions are not primarily drawn on the basis of the findings but on the basis of political and strategic motives (anticipating on reactions from clients, politicians or the health care field). This undermines the trustworthiness of KCE. Furthermore, some respondents argue that a decline in trustworthiness has taken place from KCE’s inception until now.

Authority (knowledge and expertise on health care issues)

A total of 18 stakeholders answered this question. Not one of them judges KCE’s knowledge and expertise as low or very low. In addition, 72 percent considers this to be high or very high. Some critical connotations on this generally positive judgment have been made by respondents, focusing on their observation that the work of KCE is being characterized by an (overly) ‘theoretical’, ‘academic’ or ‘classical/conservative’ approach. According to the stakeholders this results in too strong a focus on high quality research (mainly based on scientific literature) instead of research oriented on the reality of health care practice (with a higher degree of practical relevance).

Degree of independence

Most stakeholders consider KCE’s degree of independence to be average (37%) or high (42%). 16 percent judges it to be low or very low.

Multiple respondents argue that KCE’s independence is being undercut by the large amount of influence different stakeholders exercise on the final contents of the KCE reports and recommendations. The health insurance funds are explicitly named, but also politicians and/or the minister and the Board of Directors as a whole. Another respondent states that KCE is increasingly being perceived as an instrument used by politicians and subjecting itself to political institutions.

Helpfulness

Almost 70 percent of the stakeholders judge KCE’s helpfulness or serviceability to be high or very high. The other answers are somewhat equally divided between average and low. Most of these answers have been clarified, yielding the following remarks:

KCE does not take all questions into consideration because they feel it is not their responsibility

KCE’s projects are of limited use for professionals and health care providers

KCE’s helpfulness is being limited by their highly limited independence and proven political interference

KCE is exclusively focused on the minister

KCE is highly dependant on external experts, whose interests do not always match with the central goals of KCE

78

General reputation of KCE: key findings

Less than half of the stakeholders interviewed judge KCE’s trustworthiness to be high or very high. Multiple respondents state that the conclusions are not primarily drawn on the basis of the findings but on the basis of political and strategic motives

Almost ¾ of the stakeholders interviewed consider KCE’s knowledge and expertise on health care issues to be (very) high. Critical connotations on this generally positive judgment focus on KCE’s (overly) ‘theoretical’, ‘academic’ or ‘classical/conservative’ approach.

Less than half of the stakeholders interviewed judge KCE’s degree of independence as high. Respondents argue that KCE’s independence is being undercut by the large amount of influence different stakeholders exercise on the final contents of the KCE reports and recommendations.

Almost 70 percent of the stakeholders judge KCE’s helpfulness/ serviceability to be high or very high.

4.6.2 Judgment on responsiveness

Communication

Most respondents currently have and/or used to have personal contact with KCE. The intensity of this contact differs between respondents, as can be seen in the table 4.34.

Table 4.34: Frequency of personal contact with KCE

Frequency Count Once a week 4 Once a month 1 Once per 3 months 8 Once per 4.5 months 2 Once per 6 months 2 Once a year 2 Less than once a year 1

Total 20

The research team also verified whether there was a discrepancy between the contact frequencies of stakeholder clusters. Although the research population is too small do draw certain conclusions, the frequency of contact with KCE seems to be higher for insurers, universities and the government/minister than for politicians and associations of professionals and health care providers.

Less than half of the stakeholders interviewed are (highly) satisfied with the way in which KCE communicates with their organization. 30 percent of the stakeholders are not satisfied; 25 percent are neutral. Several suggestions for improving the communication are given, such as: issuing the reports on paper, more pro-active communication about research findings (e.g. publication in scientific journals, workshops/presentations), more interaction (e.g. discussions), more transparency, issuing a summary with the most important conclusions on paper, sending newsletters customized to stakeholders’ specific fields of interest, sending the reports to the parliamentary commission to allow for a public debate about the findings, and setting up structural (non-project related) contact with stakeholders.

Tailoring products to ‘customers’

Seven stakeholders state that one or more of the studies executed by KCE originated out of a proposal submitted by their organization. These stakeholders differ in their opinion about the degree in which the final report was tailored to their information needs in terms of results, the way of presenting these and the timeliness of the report. 4 respondents answer positively. The other respondents are critical about the usability of the studies, for instance because of the chosen methodology and data (which made the findings less usable), diversion from the original research goals, and narrowing down the scope of the study too much. One stakeholder judges the timeliness as being insufficient.

79

The stakeholders were also asked to judge the ability of KCE to make their findings usable for influencing policy or practice in health care. About one third of the respondents judge this ability as high or very high. 40 percent judges it to be average and 25 percent considers KCE’s ability in making usable findings as low.

Clarifications given for the low and average answers can be divided into three main categories:

Comments on the way in which the findings are presented: not in print, difficult to understand and interpret, not succeeding in spreading findings ‘to practice’, no translation from scientific language into ‘marketing language’.

Comments on the degree to and way in which findings are being used: politicians do not systematically use the findings from KCE, but only when it suites their political agenda; the use of studies is purely opportunistic. Also, the support for KCE findings coming from the founding minister was higher than from the last two ministers.

Comments on KCE’s degree of independence: the influence from the Board of Directors on the final reports is too strong (there should be a scientific advisory council guiding the BoD). Now, KCE is not presenting its own conclusions and thereby providing opponents with ammunition for stating KCE is not independent.

Fulfilling information needs

Only five stakeholders (25%) report about contacting KCE with inquiries for information or advice. The majority of these stakeholders states that most of the time KCE responds timely to and helps them adequately with their information requests. One stakeholder states it was unknown to him that it was possible to directly contact KCE with inquiries for information or advice.

Stakeholders’ judgment on KCE’s responsiveness: key findings

Less than half of the stakeholders interviewed are (highly) satisfied with the way in which KCE communicates with their organization. According to stakeholders, KCE can improve this by putting more effort into better adapting its communication to specific target groups and communicating more pro-actively.

¼ of the stakeholders interviewed characterize KCE’s ability to make findings usable for influencing policy or practice as low. Factors causing this are KCE’s nondescript way of presenting findings, the absence of structural use of the findings and KCE’s limited degree of independence.

Only five stakeholders report about contacting KCE with inquiries for information or advice. Most of them feel KCE responded timely to and helped them adequately with their information requests.

4.6.3 Judgment on research quality

First of all, we asked stakeholders to give a general judgment on the ability of KCE to carry out research that is in keep with scientific standards. Almost 8 out of 10 stakeholders judge this ability to be (very) high. The other respondents judge it to be average. Explanations given for average scores are:

Sometimes the choice for a certain methodology is a subjective one

KCE lacks the expertise necessary for adequately interpreting the findings from the studies, especially concerning comparisons with foreign countries. KCE does not bother asking field parties for a clarification of the findings.

KCE houses a lot of experts. However, the contribution of these experts is often limited to carrying out of literature research (using the help of search databases, and hereby implying the risk that they miss certain references).

Secondly, the research team asked stakeholders to give their assessment on the quality of KCE concerning the choices regarding research questions and research methodology, their way of reporting research results, their way of managing projects, and the competency of researchers working on the projects. Table 4.35 shows the results for these questions.

80

Table 4.35: Stakeholders judgment regarding KCE quality on five aspects

Research questions

Research Methodology

Reporting About results

Project management

Competency of researchers

Very bad 0% 0% 0% 0% 0%

Bad 6% 0% 21% 0% 0%

Average 28% 21% 32% 33% 27%

Good 56% 53% 32% 50% 47%

Very Good 11% 26% 16% 17% 27%

Overall, the judgment on KCE’s quality is fairly positive. The quality of the research methodologies chosen and the competency of the researchers relatively often are being judged by the stakeholders as good or very good. The way of reporting about the results is relatively often being judged as bad or average.

Several clarifications given by stakeholders concern the competency of the researchers:

“Asked for a further explanation of the research results, the experts are not able to come up with an answer. They have not always mastered the topics concerned and they do not always have enough understanding of what the questions posed are about”.

“Some experts are excellent, but others are clearly less good at what they do. Some clearly adopt an overly arrogant attitude”.

“As regards the researchers one wonders about why such extensive use in being made of subcontractors”.

Stakeholders’ judgment on research quality: key findings

KCE’s ability to carry out research that is in keeps with scientific standards. Is judged to be (very) high by almost 8 out of 10 stakeholders.

Stakeholders relatively often judge the quality of the research methodologies chosen and the competency of the researchers as good or very good. The way of reporting about the results is relatively often being judged as bad or average.

4.6.4 KCE’s added value compared to other organizations

Other relevant research organizations

At the end of the interview, we asked stakeholders if there are other parties in the domain of health care research (national and international) that produce research relevant for their organization. The following organizations have been mentioned:

National Council for Quality Promotion (Nationale Raad voor Kwaliteitspromotie, Conseil national pour la promotion de la qualité), part of RIZIV/INAMIlxix, Federal Agency for Medicines (Federaal Agentschap voor Geneesmiddelen en Gezondheidsproducten, Agence Fédérale des Médicaments et des Produits de Santé, AFMPS), Commission Reimbursement of Drugs (Commissie Tegemoetkoming Geneesmiddelen, Commission de Remboursement des Médicaments, IMA/AIM (Intermutualistisch Agentschap, Agence Intermutualiste), NICE (United Kingdom), MAP (United States), MEDICINE (United States), Itinera Institute lxx , Europharm forum lxxi , International Federation of Health Funds, Assuralia lxxii , the health insurance funds, pharmaceutical companies, King Boudewijn Foundationlxxiii, Observatory for Health (Henegouwen), Observatory for Health and Welfare (Brussels), Organization for European Cooperation and Development (OECD), Dutch Board of Health Insurances (CVZ), French Agency for

lxix The NRKP is legally responsible for managing the evaluative system “peer review” (critical research by

doctors into the quality of their care provision and an acceptable and adequate execution of their practice), developing recommendations for the use of the ‘globaal medisch dossier’ and developing recommendations for the improvement of teamwork and several cooperations.

lxx The Itinera Institute is an independent thinktank identifying routes towards sustainable economic growth and social protection for Belgium and its regions.

lxxi The members of the Forum are professional associations of pharmacists in countries in the WHO European Region. There are currently 35 Member Associations from 26 European countries. Members of the Forum meet at least once a year in a General Assembly to assess the work of the past year, to plan future activities and to decide upon issues of policy.

lxxii Assuralia is the Belgian branch organization for insurance companies lxxiii The foundation supports projects and civilians that are dedicated to a better society, a.o. in the field of

health care

81

Hygienic Safety of Health care products (AFFSAPS), Research Departments of the FPS and RIZIV/INAMI, Academic Research Centres, Institut Pasteurlxxiv, Institute for Hygiene and Epidemiology, and Woncalxxv.

Some respondents argue that KCE should tune its activities to Belgian organizations also active in the field of health care research, because this allows the different organizations to build op their own specific expertise and activities will be better coordinated.

Added value of KCE

Four respondents doubt the added value of KCE. Two stakeholders feel KCE is too small compared to international agencies to have a real significance in the field of health care research; one respondent therefore poses the suggestion to use studies from abroad and translate the findings to the specific Belgian (budgetary) context. Another respondent feels that studies done by universities are more influential than KCE studies. The fourth stakeholder argues KCE will not have any added value unless it stops obeying “his master’s voice”.

All other respondents indicate KCE has at least some added value for them compared to other institutions. Some mention KCE’s scientific character/approach/authority, others are mainly positive about the great volume of available resources. Objectivity and independence have also been mentioned, as well as a direct personal involvement from KCE personnel in the studies. HTA and HSR studies have been specifically mentioned as having an added value.

Five respondents perceive the added value mainly as the specific Belgian scope of the studies which, according to a few respondents, brings with it the advance of also being able to gain information about budgetary consequences of certain policy decisions.

KCE’s added value compared to other organizations: key findings

All but four of the stakeholders interviewed indicate KCE has at least some added value for them compared to other institutions. This added value ranges from scientific quality to the amount of resources available to KCE’s unique focus on the specific Belgian context.

4.7 CASE STUDIES

4.7.1 Introduction

In this paragraph, we provide the findings of the case study analysis. The table below shows the nine research projects that have been selected for case study analysis. The cases have been selected in accordance with the procedure and criteria as specified in paragraph 3.4.5 (Phase 2: Research projects, Activity 2.2. case studies) of this report.

lxxiv The institute promotes the prevention and treatment of infectious diseases through research, teaching and

public health initiatives. lxxv Wonca is an acronym for the World Organization of National Colleges, Academies and Academic Associations

of General Practitioners/Family Physicians.

82

Table 4.36: Selected case studies

Estimated impact Domain Low Average High HTA Case 61

Negative Pressure Wound Therapy: a Rapid Assessment45

Case 20 Molecular diagnostics in Belgium46

Case 33 Effects and costs of pneumococcal conjugate vaccination of Belgian children47

GCP Case 52 Rapid Assessment: Cardiovascular Primary Prevention in the Belgian General Practice48

Case 60 Pulmonary Function Tests in Adults49

Case 27 Quality and organization of the care for diabetes 250

HSR Case 70 Comparative study of hospital accreditation programs in Europe51

Case 83 International comparison of reimbursement principles and legal aspects of plastic surgery52

Case 68 lxxvi Damage charges as a result of health care -Phase IV Financial division between the Fund and the insurers42

Due to the limited number and non-random selection of the case study projects, the case studies do not allow a quantitative analysis which is representative for all KCE projects. The purpose of these case studies is to provide a deeper insight into the dynamics surrounding KCE research projects in practice, and how different project-related and contextual factors may influence the impact of research projects. In addition, the case studies provide rich and detailed qualitative information regarding critical areas in the research process from which KCE may be able to draw lessons for the future.

The structure of this paragraph is centred on the topics within each of the main themes related to the impact of research projects. These are: origin of the research (proposal selection); advance specification of intended impact (goal); goals for user engagement and dissemination; quality of the research process; stakeholder involvement; active dissemination (strategy); contribution to decision-making; identification of specific target users and tailoring products to their needs; image of KCE amongst stakeholders. In addition, the difficulties surrounding the assessment of impact are discussed, as well as a number of other factors (than the above-mentioned) that have contributed to the success or failure of the case study projects. The paragraph concludes with a brief cross-case analysis and summary of the findings.

The reader has to be aware, however, that this chapter is opinion-based and that certain opinions could not be checked, confirmed nor denied.

4.7.2 Transparent and careful proposal selection procedure

The table below shows that the proposal selection for each of the case studies is not characterized by a very transparent proposal selection procedure. RIZIV/INAMI has been the originator of the majority (five) of the projects included in the nine case studies. Three out of the nine case studies went through the formal KCE proposal prioritisation procedure, which is explained in section 4.2 of this report. Five out of the nine projects were selected after an urgent request by the minister or RIZIV/INAMI, without going through the proposal prioritization and selection procedure. For projects in 2004 and 2005, this was related to the fact that the KCE prioritization and selection procedure was not yet implemented. For projects from 2006 onwards, the KCE programming procedures allow for urgent requests to be added to the work programme.

This may reinforce the notion amongst some stakeholders and KCE staff that the selection of KCE research projects is not always sufficiently transparent and strongly politically oriented (see previous report sections 4.2.5 and 4.2.6).

lxxvi Report 68 is part of a larger project consisting of five phases; most information applies to all stages of the

report.41, 42, 53-55

83

Table 4.37: Case study rating on proposal selection Case number 52 70 61 60 83 20 27 68 33 estimated impact

Low Low Low Avg. Avg. Avg. High High High

Originator Domus Medica

Scientific institution

RIZIV/INAMI Cabinet Social Affairs and Health

How was proposal submitted?

TPF TPF urgent question

TPF urgent question

urgent question

? urgent question

urgent question

Rating on hypothesis 1 “Transparent, careful proposal selection procedure”

average average low average low low average low low

The notion of a strong political bias may be further reinforced by the fact that, besides the originator of a project, other stakeholders (such as associations of hospitals, care providers, and professionals, sub-federal political decision-makers and scientific institutes) are not always invited before the start of the project, for example in the research conceptualization, preparations and formulation of the research questions. The following provides an example of a (high impact) case study, where a negative political bias is suspected in the project proposal selection and conceptualisation:

The project coordinator states that before the KCE study, there was no agreement yet about the dose and cost-effectiveness of the prevention scheme. According to a sub-federal political stakeholder, however, the study was based on a hidden agenda and was actually no longer necessary: “by the time the study commenced, there was already sufficient evidence from foreign countries that the prevention scheme to be studied was effective. The study was used by one of the political communities to have more time to gather sufficient funds for implementation”. The added value from the research does, in the stakeholder’s opinion, not weigh up against the loss due to morbidity as a result of postponed implementation. The stakeholder believes that KCE should have refused to undertake the study. In addition, the study was conceptualised and implemented without consulting all relevant stakeholders (e.g. policy-makers at non-federal level, medical professionals, and relevant Council members). The fact that the project could have been carried out earlier was subscribed by another respondent, who stated that the research topic had been submitted to KCE two years earlier, but only started in 2006. The respondent referred in this regard to the political composition of the KCE BoD, which results in decision-making and prioritisation of projects being based on political grounds: “This is not always in line with long-term benefit. The prioritisation process is also not sufficiently transparent. It would be possible for KCE to anticipate developments in the field of health care, as they and external experts in the field already know that a certain topic will become important”. Moreover, the project coordinator also referred to an earlier recommendation by the Superior Health Council (Hoge Gezondheidsraad / Conseil Supérieur de la Santé) to implement the prevention scheme under study.

The delayed proposal selection has negatively influenced the stakeholder’s perception of the impact of the report. He questions whether the same advice could have been provided with fewer resources and in shorter time, for example based on a quick scan.

Case study proposal selection: Key findings

The case study projects are not characterized by a very transparent proposal selection procedure.

Political bias perceived by some stakeholders and KCE staff in proposal selection is reinforced when stakeholder involvement before start of projects is limited to the originator.

4.7.3 Usability and advance formulation of specific goal

The ratings in the table below illustrates that all case study projects have formulated at least some specific goal, for example one or more research questions to be answered. The case studies differed, however, regarding the extent to which

84

the formulated goal was stakeholder relevant. In case study project nr. 52, the project goals were changed during the course of the project, hence the second rating in the column.

Generally, it can be concluded from the case studies that the formulation of goals and research questions for KCE studies largely depends on the organisation that submits the proposal (e.g. case 33, 60, 61). As can be read in subsequent sections, some stakeholders feel that all the different stakeholders of a research project should be involved in the conceptualisation phase, where goals and research questions are formulated. This might enhance the relevance of the project goals and the research itself.

Table 4.38: Case study rating on advance goal formulation Case number 52 70 61 60 83 20 27 68 33 estimated impact


Rating on hypothesis 2 “Specific and (stakeholder-) relevant goal formulated in advance”

Initially high, later low)

average high average average very high

average high high

The advance formulation of a specific (stakeholder) relevant goal can be one of the success factors to improve the usability of the findings. Although as such it does not guarantee acceptability of all of the results, it does contribute to the perception of the study (and KCE) as being practice-oriented with useful recommendations. This is illustrated by the case study 20 which scored very high on advance goal formulation:

All the respondents for this case study confirm that from the initial proposal onwards, the project has been strongly policy and practice-oriented. For example, the research questions concern, amongst others, the optimal implementation of the tests under study, the organisational aspects of testing techniques and methods, financial and budgetary implications, as well as adjustments to the legal prescriptions for quality control. One of the stakeholders (field of medical practice / centres) of this project had been involved in the submission of the proposal. KCE estimated the impact of this project to be medium, while three out of four case study respondents estimated the impact to be high with a majority of the recommendations being implemented. The stakeholder from the field of medical practice argued that the practical feasibility of some of the recommendations was problematic, yet conferred that this was due to the recommendations going against vested interests and established practice, and not attributable to KCE.

In one of the projects with a low estimated impact, a specific stakeholder relevant goal was originally formulated:

After a while, the externally outsourced research team appeared not to work well due to coordination and management problems. KCE decided to restart the research with an internal team and reformulated the proposal. The research proposal was focused on the usefulness of the guidelines and on the implementation of guidelines in daily practice. The output was focused on the difference between several guidelines, the evidence based treatments in the guidelines and the cost effectiveness of the treatments. The advice expected by the originator on how to disseminate and implement those guidelines was no longer part of the output. This may partly explain the low impact rating for the project.

Case study goal formulation: Key findings

All case study projects have formulated at least some specific goal.

Formulation of goals and research questions for KCE studies largely depends on originator.

The projects differ on the extent to which the formulated goal is relevant to stakeholders, other than the originator.

85

4.7.4 Explicit goals for user engagement and dissemination

As indicated by the ratings in the table below, in only three out of the nine case studies, explicit goals for user engagement and dissemination were formulated. In one of these cases the initial goals were not subverted, while in another case the originally ambitious dissemination goals could only partly be realised due to insufficient budget.

Table 4.39: Case study rating on goals for user engagement and dissemination

Case number 52 70 61 60 83 20 27 68 33 estimated impact Low Low Low Avg. Avg. Avg. High High High Rating on hypothesis 3 “Explicit goals for user engagement and dissemination at start of project”

Initially high, later low

Initially high; no

budget

low low low average low low low

The lack of explicit goals for user engagement and dissemination may be due to the notion in KCE procedures, and held by KCE management and by some project coordinators, that a research report which is in principle aimed at the originators, is sufficient. This is sometimes subscribed by stakeholders as well:

“The conclusions and recommendations are very technical. One has to be involved in the matter, for example as a politician or stakeholder to understand it. This is not a problem because alternatively, there might be oversimplification which does not do justice to the complexity of the matter. Most important is that the people involved know what the study is about, which was the case in this project. To me, the reports are sufficiently readable” (Stakeholder from medical insurance field, in expert group of high impact case study).

In one of the cases with low estimated impact, the originating stakeholder was a representative association for practitioners. Dissemination and implementation were two of the stakeholder’s goals, and objects to be searched for from the very beginning of the research. Yet, when the research was restarted and reformulated by KCE, the dissemination and implementation goals were no longer addressed (see preceding section). This may partly explain the low impact of the project.

In another case with low estimated impact, quality of the dissemination seemed to have a heavy emphasis in the project proposal, but there was no budget to conduct a large campaign as required. Dissemination therefore took place in a limited way via workshops. According to the external researcher the reports of KCE could have a far greater impact on policy makers when KCE opens up its information and knowledge; the availability of the reports on the website is not sufficient.

According to the KCE coordinator, however, all stakeholders were sufficiently informed during the process of the research: “because all stakeholders participated in interviews or discussion groups, dissemination of the outcomes was realised before the report was finished and made available on the website”. The latter is, however, not necessarily true since participation as a respondent or otherwise in (part of) a research project as such is not the same as receiving the integrated and analysed results and recommendations from a study.

Case study goals for user engagement: Key findings

In hardly any of the case studies, explicit goals for user engagement and dissemination were originally formulated (and implemented accordingly).

This may not necessarily be problematic when stakeholders in the political and medical insurance fields are the only target groups aimed at, but more so if stakeholders in the field of medical practice are addressed or aimed at.

86

4.7.5 High quality research process

The table below shows that case studies with a high to very high research quality score are also the ones with a comparatively high estimated impact.

Table 4.40: Case study rating on quality of the research process Case number 52 70 61 60 83 20 27 68 33 estimated impact Low Low Low Avg. Avg. Avg. High High High Rating on hypothesis 4 “High quality research processes (project management and methodology)”

low average average low low very high

very high

very high

high

The cases that received low or medium scores on quality were plagued by coordination problems such as discontinuity in coordination or lack of direction (case 52, 70) and / or methodological disagreements between the external experts or researchers, and KCE. At the end of each project, it is mostly agreed that the quality of the research is nevertheless at least reasonable or acceptable.48, 51 Nevertheless, quality flaws in the methodology may at times be detrimental for the impact. Case 52, 60 and 61 are exemplary:

In one case with a low estimated impact, the research quality (process) was viewed as poor or defective by all respondents. The research project took too long and the results only partly answered the questions of the originator. In this case the outcomes of the research would have been better recognized when besides general practitioners; also specialists would have played a more important role during the research process. In the end, the originator repeated part of the research in a setting, where general practitioners and specialists both could be heard.

In another case with an estimated medium impact, an external expert (from the scientific field) maintained that a major limitation in the methodology was the lack of recent documentation about the topic. The external experts as a group proposed that the practical experience of medical specialists should be used to supplement the findings from the literature. However, the KCE researchers insisted on holding on to the original methodology. In addition, the research only considered activities that were already included for reimbursement. It did not consider newly developed more cost effective alternatives that are presently not yet reimbursed. According to the external expert, insufficient time was taken to reflect on the research methodology. After finalisation, the external experts and other medical specialists in the field did not recognise the report and recommendations as these go against the current practice and their interests of the field. The respondents fear that the medical field may refer to methodological issues to devalue the report.

According to representing stakeholders (from pharmaceutical and medical appliances industries), the research process is not adjusted to the specific characteristics of testing medical instruments. Although the choice of methodology and data analyses were conform internal KCE guidelines, some external experts and the industry did not agree to the conclusions, because their experience was otherwise. The conclusions were based on the evidence found in the literature. The evaluation of the studies was considered rather weak, because intervening variables were not sufficiently taken into account.

Generally, KCE methodological quality procedures are applied in all the cases. In the above-mentioned cases (and case 70), KCE’s methodological procedures are perceived to be limiting or unaccommodating to involving all the necessary data to answer the research questions. Apart from one report (case 61), all the reports in the case studies were also validated by external experts, further enhancing the quality of the reports.

In case 61, a project rated as having a low impact, the report was only validated internally because of time constraints and because the study was considered as a prototype for quick searches.45 Note from the KCE project leader concerned: like all

87

KCE studies, case study 61 was validated by three external and independent validators.

Comments from the external expert groups during the course of the project are also supposed to contribute to the quality of the research. The number of meetings and the extent, to which external expert comments are incorporated in the research, vary substantially between the case studies. For example, for a report with a high estimated impact, the case study finds:

There were a sufficient number of meetings with the external expert group; at least nine regular meetings in total during the project. Expert (scientific and from medical practice) and stakeholder (representing medical centres and committee), comments about the quality of the research and the use of data were fair, and were fairly taken into account.

Whilst for reports with a low impact, the case study reveals that external expert meetings were scarce or validators’ comments were not fully addressed:

During the course of the research one external expert meeting took place. External experts were doctors, who experienced the treatment under study in their own practice. The experts did not agree on the outcomes of the KCE research thus far. The experience of the external experts is mentioned in the report, but not incorporated in the outcomes of the study.

The external experts commented on the execution of the project, and twice provided corrections on the draft report and recommendations. Yet, about half of the major comments by the experts were not integrated into the research. The (scientific and practicing) expert therefore feels that the experts in the expert group were compromised in order to quickly push through the report “with the collaboration and input of experts”. The expert feels the external experts have been used to ratify a report that they do not even approve of, “so KCE and the client should not be surprised and blame us when the guidelines are not being followed”.

When the goal is changing a system or changing the daily practice of care, it is not necessarily plausible that following high quality research process rules shall lead to practical changes. In some cases, stakeholders attempt to preserve their interests by indicating flaws in the research quality or methodology. Stakeholders may also point at possible political influences in the admission of certain data to KCE reports. In such cases, it may be difficult to objectively judge the quality of the research. For example:

In one case with medium estimated impact, external validators argued that the operational definition of the main concepts proved problematic. The reason why some choices were made was not clear and the list of medical interventions was considered ambiguous. Two external validators remarked that too few sources of information were used, as no information from private clinics was involved. In the report it is stated that the administrations of private clinics was not taken into account because there is no confidence on the correctness of the registration. Two external validators mention that it is not credible that the invoices of all private clinics are considered inaccurate, while all invoices of public clinics are considered accurate. During a round table conference organized to discuss the draft outcomes of the study, it was also noticed that the study did not cover the total field under study. There was not enough research time left to repair on this lack on information. The remarks led to some textual changes in the report. The report states that the definitions were not based on scientific research, but on expert opinion, which is open to criticism.

88

Case study research quality: Key findings

Case studies with a high estimated impact are characterised by a high to very high research quality score.

In reverse, case studies with low to medium estimated impact are characterised by low to medium quality scores, due to coordination problems and / or methodological disagreements between the external experts or researchers, and KCE.

KCE methodological quality procedures are applied in all the case study projects.

In some cases with low estimated impact, KCE’s methodological procedures were perceived to be limiting or unaccommodating to use all the necessary data to answer the research questions.

Projects with a high estimated impact were characterised by sufficient external expert meetings where experts’ comments were fairly taken into account.

For cases with a low estimated impact, external expert meetings were scarce or experts’ comments were not fully addressed.

All case study projects were externally validated, enhancing research quality.

Research quality seems a critical factor for impact, as stakeholders may refer to any (real or perceived) methodological flaws to undermine the validity of the research findings, particularly when the latter conflict with their interests.

4.7.6 Active stakeholder involvement during research

The table below shows that cases with a high estimated impact tend to score medium (at least) to very positive on stakeholder involvement. Amongst the cases with a low to medium estimated impact, three scored negatively on stakeholder involvement, whilst the other three score medium or positive.

Table 4.41: Case study rating on stakeholder involvement during research:

Case number 52 70 61 60 83 20 27 68 33 estimated impact Low Low Low Avg. Avg. Avg. High High High Rating on hypothesis 5 “Actively engaging relevant stakeholders during the research project”

low high average low low high very high

average average

Stakeholder involvement in KCE projects predominantly happens through external expert teams. Although there has been a tendency to increase the involvement of stakeholders on the expert teams, some of these external experts feel that their involvement is limited to purely scientific contributions, and that there is insufficient space to contribute from their stakeholder perspective or interest. Other stakeholders maintain that there has been a selection with regard to which stakeholders are involved in which parts of the project, which leads to imbalances in stakeholder representation during the project (case 33, 60, 68, 83, 61). This occurs in projects that are considered low impact as well as those considered high impact.

In one case with a medium estimated impact external experts were, according to the project coordinator, involved in various stages of the research process, such as:

o Formulating the purpose and the scope of the study

o Formulating research questions

o Selecting data

o Access to data sources

o Critical reflection during the research process

o Drawing conclusions and formulating recommendations

o Acting out to the media

89

However, not all external experts appeared to be equally involved in the abovementioned activities. Only the originator was involved during the total period of the project (defining and preparation, executing en discussing the draft report, conclusions and recommendations). Other stakeholders (medical professionals) were involved during the execution of the research and the ending phase of the report. They participated as external experts and as validators. Some experts were only invited to respond on the draft report (i.e. the expert on private clinics). Nose, throat and ear doctors were not represented among the external experts, which is the reason why they eventually protested against results in the report. They did so by means of an article in the “Artsenkrant”. The article bears the headline 'report knowledge centre full of errors'.

The following examples illustrate opinions regarding the consequences of timing and selectivity of stakeholder involvement in the research process:

According to an external expert from insurance field in one of the cases with can estimated high impact: “The involvement of all stakeholders, in a transparent and open way, before the start of the research project could have made a large difference in the relevance and impact of the project. In that case, all stakeholders would have had equal chances in defining the preconditions of the research and delimiting the research, as well as in developing or sharpening / refining the hypothesis. These aspects are very important in determining the direction of the research and the findings. Now, certain stakeholders had lobbied and ensured that their preconditions were established and taken into account in the research design. However, not all stakeholders had this chance so not all the stakeholders views and preconditions were taken into account. In case all stakeholders would have been involved before the start of the project, it would have been clear from the start that, within the preconditions as determined, the political option under consideration was not an implementable option (…)”. In other words, a waste of research inputs could have been prevented.

According to an external expert (from the scientific community) in one of the cases with low estimated impact, a major weakness in the project was that no external experts were involved in conceptualising and defining the project: “The project was developed by people with a GP-background without knowledge of the specific disease to be researched”. The external expert explains that, together with the other external experts, this was somewhat adjusted, but the project remained characterised by major limitations, for example: “Besides the expert group, there was no involvement of stakeholder representatives who represent the interest of a professional group. Also, no one representing the scientific society was involved. Perhaps one of the experts was a member of this society, but this does not imply that the scientific society is factually represented. Also, a commission that was specifically set up to specialise in the topic was never invited to be involved in the project”. The researcher also suggests that general practitioners and patients could have been included in the research as additional stakeholders. These were, however, not determined as key stakeholders by the originator who requested the study and were therefore not included.

According to a sub-federal political stakeholder of a case with high estimated impact, some stakeholders were not involved in the research process, whilst others were. The sub-federal administrators were only presented with the results after the report had been finalised, so they had no possibility to provide suggestions regarding the research, the report or the recommendations. Moreover, those stakeholders who are already represented on the KCE BoD, are automatically more involved in the report and recommendations than other stakeholders. If this is not counteracted by involving all important stakeholders (besides federal policymakers) during the research process, there is insufficient balance. For example, other target users who are supposed to implement the recommendations - such as hospitals, health centres and health professionals - were also not involved. In some cases, this could have also led to better cost indications, based on the practical situation and a more relevant time horizon within the research.

90

One stakeholder from the pharmaceutical / medical appliances industry in a case with low estimated impact feels that KCE is not clear about stakeholder involvement: “Sometimes the industry is allowed to participate as an informant and sometimes KCE closes the doors to the industry as an informant. This seems to be dependent on the individual characteristics of a researcher”.

Positive examples from reports that appeared to have a high impact and a positive score on the involvement of stakeholders are:

According to an institutional stakeholder (originator), the stakeholder was well involved in the beginning of the project: “the research itself was born after discussions with us and they contributed into the precision of the topic”. The topic of the project furthermore made it easy to involve other stakeholders (24 external experts) and the members of the research group were representing stakeholders themselves. For example, the external project coordinator was a chairman of the patient association. By this means, stakeholders were involved from the beginning of the project: from its inception phase, during execution of the project and in the discussions of the draft report and recommendations. Government and syndicates were involved in the project as well, but in later stages: during the discussion of the study results. Nevertheless, one stakeholder from the medical insurance field comments: “The general problem of research in healthcare is that the real stakeholders are difficult to approach. Nurses, doctors, service providers do not have time to participate in research. This is why, if finally high professionals are involved in the KCE projects as stakeholders, the results are nevertheless not reflecting exactly the opinion of practice and are less accepted in the field. There are always the same people who give their opinions on the same topics for KCE (expert group members, stakeholders, etc)”.

In another project, a broad variety of target groups or organisations were, also at the start of the project, explicitly acknowledged as stakeholders of the project. The stakeholders were both from the field of health care policy (RIZIV/INAMI, community politicians, fed. department) as well as practice (centres for molecular diagnostics, medical professionals, hospital directors, pharmaceutical / medical industry), insurers as well as universities and scientific institutes. In the initial project design, all the stakeholder groups were assigned an important role in the project. The proposed methodology practically started with contacting stakeholders, such as clinical biologists, CMDs, CMEs, (molecular centres) the industry and test developers (universities) as well as representatives of patient groups, if feasible. The goal of the interaction was to arrive at an inventory and classification of existing tests. Also in practice, several stakeholders (including medical professionals, scientists and representative organisations) have been involved during the course of the project both as participants as well as for conceptualisation of the study, critical reflection, conclusions and recommendations. According to all respondents that were interviewed, the number and range of stakeholders involved in the project were sufficient.

Nevertheless, not all recommendations were implemented. The reasons given for certain recommendations not being implemented were related to the recommendations going against the established way of working in the centres; the stakeholders feeling their interests being harmed by the report; and, as a result, lobbying by vested industry interests. This was, however, considered not to be within KCE’s sphere of influence. One stakeholder mentioned that the only way stakeholder involvement could have been improved, would have been to allow stakeholders to co-author the report. This example shows that stakeholder involvement is not necessarily sufficient to ensure that all recommendations are acceptable, since conflict of interest is always a possibility. Yet stakeholder involvement may enhance the understanding and appreciation of the validity of the findings, even when conflicting interests are at stake.

91

In some case studies, the researchers, proposal submitting organisations as well as some other stakeholders agree that care must be taken that stakeholder involvement during the research process does not affect the objectivity of the research findings. Fear to loose scientific objectivity and independence is often given as the main reason not to involve stakeholders, for example:

According to a researcher of a case with high estimated impact, the project coordinator was not anticipating involvement of the stakeholders in the project. The researchers consulted the literature in the press, but not directly the stakeholders: “It is important that stakeholders give their opinion, but only after the publication of the project. The objective was to produce an independent study”. This was also confirmed by a researcher and stakeholder of another project (albeit with low estimated impact).

Case study actual stakeholder involvement: Key findings

Cases with a high estimated impact score medium (at least) to very positive on stakeholder involvement.

Half of the cases with a low to medium estimated impact scored low on stakeholder involvement, the other half scored average or high.

Stakeholder involvement in the case studies occurred through external expert teams.

Some external experts in the case studies feel their involvement is limited to purely scientific contributions, with insufficient space to contribute from their stakeholder perspective or interest.

Some stakeholders (not the originators) perceive a selection with regard to which stakeholders are involved in which parts of the project, leading to imbalances in stakeholder representation.

Although stakeholder involvement is not necessarily sufficient to ensure that all recommendations are acceptable to all stakeholders, positive case study examples show that stakeholder involvement is likely to enhance the understanding and appreciation of the validity of at least a majority of the findings, even when conflicting interests are at stake.

4.7.7 Clear relationship research findings and decision-making process

The ratings in the table below illustrate that for the case studies where the intended impact was instrumental (to support decisions regarding changes in medical practice, policy or coverage, etc.); the relationship between research findings in the report and the decision-making process seems to be quite clear. In the two cases with a conceptual intention (to inform debates and for orientation) the relationship between the research findings and the decision-making process is somewhat less clear. This makes sense, as conceptual studies are meant to provide a framework for debate, positioning or orientation and are not directly aimed at decisions to be taken. However, in most cases (particularly those with low estimated impact) the relationship between research findings and the decision-making process was not further clarified through additional efforts to present and clarify this relationship.

Table 4.42: Case study rating on clear relation between findings and decision-making

Case number 52 70 61 60 83 20 27 68 33 estimated impact


Type of impact sought

Instru-mental

Concep-tual

Instru-mental

Instru-mental

Concep-tual

Instru-mental

Instru-mental

Instru-mental

Instr-mental

Rating on hypothesis 6 “Relationship between findings and the decision-making processes clear/clarified”

average low high average average high high high high

92

In many cases the timing was “automatically” in line with the policy cycle, because the originator of the study would wait for the results before taking a decision. The policy cycle of stakeholders was actually adjusted to the dissemination (release) of the report, rather than the other way round. For example:

All the phases of the project were directly requested by Government, so Government was expecting the results before making a new step in legislation.

The timing of the project and dissemination activities was in line with the policy cycle, which was partly due to the fact that the originator (in this case RIZIV/INAMI) who had submitted the topic was obviously keen on using the results for policy adjustment.

In one case with high estimated impact there was some controversy around the timing. Whereas the project coordinator stated that the research was timely as the minister was waiting for the results, a sub-federal political stakeholder argued that the report was actually too late as it had already been requested in 2004 and concerned an attempt (by the minister) to slow down implementation for financing reasons. The coordinator also admits that there were questions from parliament about when the report would be out and comments from stakeholders that the report was too late, alleging that it concerned a deliberate slow-down of implementation.

An example of a case with low estimated impact where timing proved to be an issue is the following:

The recommendations on the legal framework of plastic surgery are aimed at the legislative power. KCE was not aware of the fact that new legislation was being prepared during the execution of the research. The draft law on plastic surgery was discussed in parliament at the same time that the report was finished. Because the two (draft law and KCE report) were available at the same time, some questions were asked in parliament as a result of the KCE report. An important reason for not using the outcomes of this report as a basis for the design of new legislation is the timing of the research. The results were available too late.

Case study timing of research results: Key findings

In most cases the timing was “automatically” in line with the policy cycle, because the originator of the study waited for the results before taking a decision.

In only one case study (with low estimated impact), the impact was hampered by lack of knowledge about the timing of related policy developments.

4.7.8 Active dissemination based on dissemination strategy

As illustrated by the ratings in the table below, for none of the case study projects, a dissemination strategy seems to have been developed. Mostly, the active dissemination entailed the publication of the report on the website, a press release, and a presentation to the originators. For some projects, scientific articles have been written as well. The originators, mostly RIZIV/INAMI and government, generally do not ask KCE to undertake any further dissemination activities (e.g. case 83, 68). Some stakeholders agree that stakeholders should not be involved in dissemination of the research results and recommendations: “this is not the role of stakeholders”

93

Table 4.43: Case study rating on active dissemination (based on strategy) Case number 52 70 61 60 83 20 27 68 33 estimated impact


Rating on hypothesis 7 “Actively performing dissemination activities (based on dissemination strategy)”

low average low low low Partly too late

high low low average

In two case studies with high estimated impact, the dissemination was comparatively active:

According to the project coordinator of one case, the results were presented to the government and insurance companies (mutualities) at their own request. In addition, one of the stakeholders argues that the results and recommendations have been discussed in the “Vaccinatiekoepel”, where general practitioners and pharmacists are represented, as well as in “Kind en Gezin” and “Centrum voor Leerlingbegeleiding”.

In another case with estimated high impact, the results have been presented at several presentations at RIZIV/INAMI to members of the working groups on molecular diagnostics centres. There have also been presentations at FOCUS conferences, directed at clinical biologists, as well as for specialists in flowcytometrics/moleculair tests, with about a 100 participants for each presentation. KCE was invited to do these presentations on the congresses. In addition, there has been expanded contact with the RIZIV/INAMI representatives and the Molecular Biology Association regarding the dissemination and press releases on other than KCE websites, such as www.medinet.be (KCE project coordinator). The expert and the stakeholder both maintain that the report was sufficiently disseminated and that the report is easily accessible.

In another case study with high estimated impact, the dissemination by KCE was not specifically active, but this was partly overcome because the researchers and experts were stakeholders themselves, and they were involved in dissemination activities via presentations, interviews, publishing articles. To further discuss the report, Government and other stakeholders (excluding patient associations) organised meetings. Nevertheless, health care personnel apparently have quite limited knowledge about this report. This is because besides the press release, the fact that KCE is doing the study or working on the report is only known via informal communications between specialists, but not via broad information canals.

In most medium to low impact studies, however, the intended impact seems to have been hampered by the lack of active dissemination. For example:

According to the press release of one case, KCE wanted to start a social debate on the issue. Yet, KCE has not undertaken any activities to initiate such a discussion.

According to the originator of another case study project, neither KCE nor the originator had specified a dissemination strategy, or even thought about this, before the start of the research or even when the report was published. Only after there appeared to be no impact when the originator undertook an evaluation, dissemination started to receive attention from the originator. According to the originator, the expectation of both KCE and the originator was that publication of the report on the internet, with a press release, would be sufficient to ensure implementation. KCE regularly publishes guidelines on their website, so the effectiveness was not questioned. No additional synthesis products were therefore developed. The dissemination of results was limited to the press release and a presentation to the originator on their request, as according to the originator is usual for KCE reports. No other dissemination activities were undertaken by KCE, nor were the findings directly communicated to intermediaries.

94

According to an external expert, the press release was furthermore unstrategically published amongst a number of possibly more critical reports: “This may have pushed the report somewhat into the background so that it received limited attention”. Nevertheless, the external expert thought that most of the targeted medical professionals should have heard about the report. A while after publication, however, the client contacted some professionals and organisations, and found out that most of the professionals did not know KCE, nor their website or the fact that it contains guidelines that may be of interest to the professionals, nor had they heard about the report.

Case study dissemination: Key findings

For none of the case study projects, a dissemination strategy seems to have been developed.

In most cases the active dissemination entailed the publication of the report on the website, a press release, a presentation to the originators and sometimes the development of scientific articles.

The originators, mostly RIZIV/INAMI and government, generally do not ask KCE to undertake any further dissemination activities.

In the case studies with high estimated impact, either the dissemination by KCE was comparatively active, or the researchers and external experts were also stakeholders and actively involved in dissemination activities and organising meetings.

In most medium to low impact studies the intended impact seems to have been hampered by the lack of active dissemination.

4.7.9 Identifying specific target users and tailoring products to needs

In the case studies, KCE has generally identified primarily (or even only) the originator of a project as the main target user, but no other stakeholders are explicitly identified as (potential) users. Neither are outputs generally tailored to specific needs of these others stakeholders. This is reflected by the medium scores in the table below.

Table 4.44: Case study rating on identifying stakeholders as users and tailoring products to their needs

Case number

52 70 61 60 83 20 27 68 33

estimated impact


Rating on hypothesis 8 “Identifying stakeholders as possible user(s) and tailoring product to stakeholder needs”

low average average average average average average high average

In most cases (e.g. 20, 61, 68, 27), the originators and external experts are satisfied with the report as the main output as well as with the content of the research report even if it is considered technical:

According to the originator of one case with medium estimated impact the language used is quite technical, but readable for the intended audience, such as policymakers and professionals in the medical field. The report corresponds to the information need of the stakeholder and expert

Even in high impact projects, the target users of the research outputs may be assumed, rather than explicit:

According to an external researcher: “There was no explicit mentioning that such and so are the intended users of the project. More generally, it is known that KCE is supposed to support

95

policymakers. However, the project was not very clearly and narrowly defined and the report was not written for a specific target audience. It has been left open who can use the report, as the health practitioners and public might also be interested in the report. The researchers see this as actually a good thing. However, he also mentions that it may negatively influence the impact of the report if it does not address a specific target audience, because it may become too general.

In the rare case other stakeholders and interest groups are identified, the outputs are not necessarily tailored to the specific characteristics and needs of these stakeholders. For example, in one case study with low estimated impact, the originator of the study tailored the product of the study after finalization (guidelines for medical specialists and practitioners) to the needs of the target users.

There seems to be some ambiguity regarding the identification of target users and tailoring research outputs to their characteristics and needs. On the one hand, the research goal may be centred on a behavioural change amongst practitioners. On the other hand, KCE states that it primarily aims to address the originator and the report may be written in terminology that addresses the political or institutional decision-makers. The recommendations, again, may be directed at both policy / decision makers as well as medical practitioners or centres (for example, case 61, 83). What seems to be lacking, is a transparent and explicit distinction between stakeholders with different interests and/or roles in the project, and with different output needs. It would, for example, be more transparent to make an explicit distinction between the role of the originator (for example RIZIV/INAMI or the minister), the role of experts (independent scientists) and the role of interested stakeholder parties (e.g. medical practitioners and patients).

Case study identifying target groups and tailoring outputs: Key findings

In the case studies, mostly only the originator of a project was identified as the main target user, but no other stakeholders were explicitly identified as (potential) users.

The target users of the research outputs from a specific project are generally assumed, rather than made explicit.

Outputs are generally not tailored to specific needs of other stakeholders.

In most cases the originators and external experts were satisfied with the report as the main output and with the content of the research report even if it was considered technical.

What seems to be lacking is a transparent and explicit distinction between originators and other stakeholders, each with different interests and/or roles in the project, and with different output needs.

4.7.10 Feedback and incorporating lessons in research process

The structural feedback during the research has in the case studies always been organised in the form of the involvement of external expert groups. As discussed earlier, not all comments are always incorporated in the research process.

Besides the expert groups and quality validation, KCE has in the case study projects not sought other structural feedback on the usefulness of the final research products (e.g. case 27, 60, 61, 52, 70). The feedback received is generally incidental, received from the originator and from stakeholder comments in the media. No follow-up is organised internally and what is done with incidental feedback seems to depend on the project coordinators (e.g. case 60, 52, 68). This is reflected in the low to medium case study ratings in the table below.

96

Table 4.45: Case study ratings on incorporating lessons from structural feedback by end-users

Case number

52 70 61 60 83 20 27 68 33

estimated impact


Rating on hypothesis 9 “Incorporating lessons from structural feedback by (potential) end-users in research process”

average low average low average average high average average

Some examples from two case studies with medium estimated impact are:

As the media paid attention to the study one might conclude that the report catalysed a discussion about this topic. 16 articles appeared in the media. The nature of the feedback was rather neutral, citing the facts and continuing the discussion on reimbursement or regulations on safety. One article was negative on the choices made on the operation of the definitions. It is not clear how many requests for information were made by stakeholders.

The exact number of occasions where stakeholders gave feedback to KCE on the project is not known. It is not know whether there were comments in media, since KCE only started to assemble press articles from 2006 onwards.

Case study feedback: Key findings

Besides the expert groups and quality validation, KCE has not sought other structural feedback on the usefulness of the final research products.

The feedback received is generally incidental, received from the originator and from stakeholder comments in the media.

Within the case studies, no structural follow-up was organised internally and what has been done with the incidental feedback depends on the project coordinators.

4.7.11 Image of KCE amongst stakeholders

In nearly all case studies, the originator of the research was satisfied about the research quality and their collaboration with KCE. Generally, the institutional originators laude KCE’s professional communication, responsiveness, objectivity and fulfilment of information needs. However, one originator referred to the image of KCE as an ivory tower, which is insufficiently in contact with the practicing medical professionals. This affects the nuancing of findings and the practical usability of the research recommendations. Most stakeholders were not entirely negative about KCE, but they had some comments about the KCE’s research methodology, the coordination or the communication by KCE. This is reflected in the predominantly medium case study ratings in the table below.

97

Table 4.46: Case study ratings on positive image of KCE Case number 52 70 61 60 83 20 27 68 33 estimated impact


Rating on hypothesis 10 “A positive image of KCE (e.g. on independence, responsiveness and research quality) amongst stakeholders”

low average average low average high average average average

The following examples show how impeccable quality of the research process may enhance stakeholders’ opinion on KCE, despite potentially adverse findings (conflict of interest):

In one case study with estimated high impact, the stakeholder was in the end satisfied about the collaboration with KCE on the project, despite the rigid and difficult start due to the lob-sided conceptualisation and formulation of preconditions for the research: “The research work was decent and also the hired research firm was decent. The scientific underpinnings were solid”.

In another case study with estimated medium (to high) impact, a stakeholder argues that some recommendations went against the established way of working in the centres and the stakeholders felt their interests were harmed by the report. This was not due to the quality of the report, which was good. The stakeholder regards the conflict of interests during implementation as being outside the KCE’s sphere of influence. In the end most, but not all recommendations in the report were implemented and the feedback received about the report was mainly positive.

In some cases, the image of KCE amongst stakeholders has been adversely affected due to methodological disagreement combined with conflict of interest (case 83, case 60, case 61), which in these cases also negatively affected the impact of the case studies. For example, a stakeholder argued that practitioners and specialists in the medical field were insufficiently consulted. As a result, the recommendations are considered insufficiently acceptable, impractical and not useful.

In one case with low estimated impact, the most important representative stakeholder of the project who also participated in the research was not content with the methodology, nor was the organisation directly informed by KCE about the results of the study.

Case study image of KCE: Key findings

In most case studies, the originator of the research was satisfied about the research quality and their collaboration with KCE.

Generally, the institutional originators lauded KCE’s professional communication, responsiveness, objectivity and fulfilment of information needs.

In most case studies, other stakeholders were not entirely negative about KCE but had some comments about KCE’s research methodology, coordination or communication.

Impeccable quality of the research process may enhance stakeholders’ opinion on KCE, despite potentially adverse experiences or findings.

98

4.7.12 Indication of impact

For only few case study projects, a relatively high positive impact was made plausible and even in these occasions, the impact was contested by one of the stakeholders. This is summarized in the table below.

Table 4.47: Case study ratings on the perceived project impact Case number

52 70 61 60 83 20 27 68 33

estimated impact


Impact (derived from case study)

Policy impact: average Practice:

low

Policy impact:

low Practice:

low


low

Present impact:

low Future: average


low

very high one

stake-holder:

low

Policy: very high

Practice: low

Present impact: average Future social impact very high

high one

stake-holder:

low

The following examples concern case studies with a comparatively high perceived impact:

In one project with an estimated high impact, the expert (simultaneously a stakeholder), the coordinator and the external author all rated the results and recommendations as very relevant, very highly feasible and practical and highly useable. The report has resulted in the implementation of evidence-based policy. The influence on health policy and practice has been high according to all respondents, except for one sub-federal political stakeholder. This stakeholder argues that the policy decision could have been made without the research findings (based on evidence from other countries) and the resources could have had a higher impact elsewhere. Nevertheless, the recommendation in the KCE report has been fully implemented. Thus far, there have been no major problems with the implementation.

The System for diabetes care (Trajet de soin) discussed in one of the cases with high estimated impact was installed in Belgium since 1 September 2009. But according to the project coordinator it would have been implemented without a report from KCE as well, independently. The external researcher believes that policymakers are busy with diabetes, which is regarded an important health issue, so they should be using the report. There are of course also other studies, so the question remains: “If a policy change is made, to what extent is this a result of the KCE report? Probably, policymakers float and go through everything and will pick a bit here and a bit there. The report has also been referenced in other studies and the researchers themselves have been referring to their parts of the report in presentations etc., which complicates the assessment of the impact of the report”.

In another project, with a medium estimated impact, the contribution to the decision making process as described by the internal and external project coordinators, and the external expert, has been high and positive. According to them, the results have been used in policy negotiations and have been cited in policy documents: “The findings have directly resulted in the application of evidence-based policy, as the majority (over half) of the recommendations have been implemented. Many molecular biology professionals also use the report as a guideline”. On the contrary, the stakeholder (scientific and stakeholder field of medical centres) argues that the practical feasibility of the recommendations was problematic and as a result, the project has not contributed to health policymaking or practice.

99

In many cases, the factual impact of the KCE study remains unclear:

In one case study with a high estimated impact, it was too early to say if the project had an impact on the practice of the health care; this depends on whether the Law will be accepted or not (originating stakeholder). If the law is implemented, respondents concur that the project will have a very high social impact.

In a case study with estimated medium impact, the originator used the outcomes of the report as a starting point for an internal discussion on reimbursement policy. It is, however, not clear whether the rules were changed because of the report.

The client, project coordinator and the researcher of another case with estimated medium impact expect further impact in the near future, after the client has sent a letter to all specialists to study the KCE report and adjust their routine practice in accordance with the guidelines. This will be followed-up by a (second) impact evaluation and possibly combined with sanctions for practitioners who do not comply. This is expected to prompt specialists into practice adjustments. However, the initial intention of the research project was to prompt a voluntary, preventative behavioural change amongst professionals, not based on corrective sanctions. According to the client, what happens now is not the ideal way of implementation. More positively, after informal requests to the project coordinator by target users (medical professionals) through private channels, the report will in due time also be presented in local meetings of medical professionals. However, both the client and expert warn that much resistance is to be expected from the interest groups and the professionals. They will probably try to undermine the validity of the conclusions and recommendations in order to justify their resistance to change.

According to the originator and stakeholder (representative of medical practitioners) of a case study project with low estimated impact, the report played a role in the construction of final guidelines in general practice. The KCE coordinator did not know whether the report did contribute in any way to policy making.

In two low cases with low estimated impact, the impact was perceived as low, for the following reasons:

In one case with low estimated impact, the KCE study concludes that “more studies are needed to assess the usefulness of the treatment”. A stakeholder (representative of medical appliance industry) notices in this regard that studies of KCE on medical instruments generally lead to the same conclusions: there is not enough evidence. An important reason of this lack of evidence is the fact that most existing studies on medical instruments are not double blinded. According to the stakeholder, it is impossible the double blind an experiment with a medical instrument, because everybody can observe the use or not-use of the instrument. The stakeholder stated that the HTA-rules on the search for evidence should be altered when medical instruments are at stake. In the general discussion of the report KCE has acknowledged this problem, but stated that in those case the process of double blinding should be carried out by a third party.

In another case with low estimated impact, the study was not used by politicians, the insurers, health care organizations or scientists. Reasons why the study is not used, while it could be an excellent starting point for policy making on quality control, are:

o The fragmented organization of Belgian health care , which causes the creation of a joint system for accreditation to be a very time consuming process;

o Not enough activities for dissemination were undertaken;

o Other political issues on health care were more important at that time.

100

Case study perceived impact: Key findings

For only three case study projects, a relatively high positive impact could be made plausible within the framework of the case studies.

Even in these cases, the positive impact was contested by one of the stakeholders.

In four cases, the factual impact of the KCE study remains unclear because further impact is expected in the future or the respondents are not aware of the impact.

In two cases, the impact was perceived to be low, because either more evidence was required or a combination of lack of dissemination, other important political issues at the time, and fragmented health care organisation and.

4.7.13 Other success and failure factors

Some of the projects that were rated as having low impact by KCE staff, scored on average similar or higher compared to projects that were rated as having average impact. One of the projects that KCE staff rated as having average impact, actually also scored above “average/medium”. This may be related to unavoidable subjective elements in the rating and scoring procedures, but may also be indicative of other (external) factors which influence the impact of research projects. These are discussed below.

Combination of factors

In both high and low impact case studies, there is generally a combination of factors that promotes or inhibits a high impact. A typical example was a combination of factors worked out positively is the following:

Strong pressure on a decision to be taken from marketing / industry, media and public;

Policy makers requested and were awaiting the results of the study;

Fair to high level of acceptability of the research results to most stakeholders;

High quality of the study.

The following is a typical example of a case with medium estimated impact where a combination of factors limited the impact.

In this case, the originator of the study felt that the impact of the project has largely been hampered by the lack of dissemination, resulting in a lack of awareness. However, the external expert argues that the low acceptability of the recommendations for the medical professionals was much more problematic than the dissemination as such. The external expert maintains that because the unacceptability of the recommendations (due to methodological issues and conflict of interests), the professional and scientific societies did not back the report and did not even want to bother with it: “Everyone in the field has tried to ignore the report”.

Actual take up of recommendations by the decision-makers, and effects of lobbying

In most case studies, the interviewed persons indicate that KCE is not expected to directly influence the take up of recommendations by decision-makers, after the release of the report.

A stakeholder maintains that “KCE is responsible for the dissemination, but thereafter, the politicians should continue by starting to work with the conclusions. KCE does not need to influence this process or exert pressure to speed up this decision-making. The role of KCE is, and should be, limited in this regard. KCE should be doing their task as a research centre, namely to feed policy with scientific data and figures. They should not be involved with political decision-making”.

101

In many of the cases, a conflict of interest was indicated as far as the findings or recommendations were concerned.

In one case with medium estimated impact, the majority of the recommendations have been implemented. However, a recommendation that has not been implemented exactly as intended by KCE is the calculated cost-price for tests. The stakeholder mentions in this regard that the recommendations went against the established way of working in the centres and the stakeholders felt their interests were harmed by the report. The reimbursement price is still above the level indicated by KCE, thanks to successful lobbying by the industry. This, according to the stakeholder, was not due to the quality of the report, which was good.

Some stakeholders (e.g. medical centres) also mention that Belgian medical professionals and centres are traditionally not inclined to follow recommendations: “KCE will probably not be able to change that, particularly if the recommendations are not in the professional’s interests. The only room for improvement is in allowing stakeholders to co-author the report”.

Case study success and fail factors: Key findings

In both high and low impact case studies, there is generally a combination of factors that promotes or inhibits a high impact.

In most case studies, KCE was not expected to directly influence the take up of recommendations by decision-makers, after release of the report.

In many of the case studies, a conflict of interest was indicated as far as the findings or recommendations were concerned.

Some stakeholders (e.g. medical centres) mentioned that Belgian medical professionals and centres are traditionally not inclined to follow recommendations.

4.7.14 Cross case analysis

The table in appendix 12 shows a summary overview of the nine case studies that have been conducted. The rating gives an overall indication of how each aspect influencing the impact has been addressed in the project.

The results show that the case study projects which KCE staff estimated to have a high impact typically score positively or very positively on the quality of the research process. In addition, they score positively or very positively on at least three to five other important research aspects (out of ten). These aspects included: active engagement of stakeholders during the research process; a clear relationship between research findings and decision-making processes; active performance of dissemination activities; identifying stakeholders as possible users and tailoring research products to their needs; incorporating lessons from feedback by (potential) end-users in the research process; and a positive image of KCE amongst stakeholders.

The case study projects that KCE staff estimated to have low impact, scored medium or low on research quality. These projects furthermore scored positive on a maximum of just two out of the ten research aspects related to impact.

The case study projects that KCE staff estimated to have medium impact showed a more diverging picture, with two projects scoring similar to low impact projects, and one project scoring similar to high impact projects

All research projects that were included in the case studies scored low or medium on the transparency and carefulness of the proposal selection procedure (none positive). Similarly, all case study projects scored low to medium on the formulation of explicit goals for user engagement and dissemination at start of project.

Most case study projects score medium on the identification of stakeholders as users of the product and tailoring the product to their needs. The main reason for this is that often only the originator is identified as user and research products are mostly not specifically tailored to the needs of other (potential) users. With the exception of one project, all case studies scored medium to positive on the clarity between findings and the decision-making processes.

102

Cross case comparison: Key findings

The case study projects which KCE staff estimated to have a high impact typically score positively or very positively on the quality of the research process.

In addition these projects score positively or very positively on at least three to five other important research aspects (out of ten).

Case study projects that KCE staff estimated to have low impact, scored medium or low on research quality.

These projects furthermore scored positive on a maximum of just two out of the ten research aspects related to impact.

4.8 INTERNATIONAL COMPARISON

4.8.1 Introduction

As explained earlier in section 3.6.6, three agencies have been selected for the international comparison: the French Haute Autorité de Santé (HAS), the Swedish Council on Technology Assessment in Health Care (SBU) and the English National Institute for Health Research (NIHR). The three case study reports are included in appendix 15 to this report. The case studies were centred on identifying the main elements and global characteristics of the agencies. The following section is a summary of the findings of the three case studies of the agencies. In this summary, we have focused the comparison around the hypotheses for the impact study.

4.8.2 Institutional design

HAS and SBU are organizations that are independent from government.

SBU is an independent public body that already exists since 1987. It is centrally organised and managed by a Board representing key authorities and organisations within the Swedish health care system.

HAS is a fairly new organization that was started in 2004. It is a public body with financial autonomy. The independence and financial autonomy of HAS and SBU does not imply that these organisations do not have relations with the central government. For example, HAS works under the patronage of the ministry of Health and reports to government and parliament. In addition, HAS liaises closely with government health agencies, national health insurance funds, research organisations, unions of health care professionals and patient representatives.

NIHR is in fact part of the Ministry of Health and as such closely related to the structure of the NHS. The director of the NIHR, prof. Dame Sally C. Davies, is Director General of Research and Development of the Ministry of Health. NIHR can be characterised as a network organisation that is governed by a specific centre. The network characteristic is reflected by the four main areas in witch the NIHR works: Infrastructure, Faculty, Research programmes and Systems.

4.8.3 Budget, staffing and funding

Table 4.38 presents an overview of the budget and staffing of the three institutions. This overview illustrates the particular situation of the NIHR. It has by far the largest budget for which it fully relies on the government budget, and the centre of the network organisation has a small staff, which is primarily concerned with the governance of the network function and outsourced work. With a budget of between 7 and 14 million euro, the SBU’s budget is comparable to the KCE’s budget. In 2008, the KCE received € 9,3 million in subsidies, and had a staff of approximately 40 full time equivalents.

103

Table 4.48 Overview of main characteristics of the institutional design of the three agencies

Agency / country Budget Staff Funding HAS / France € 67 million

(2008) 400 permanent researchers. 3000 expert collaborating with HAS

Government Health Insurance Fund 10% of tax on Pharmaceutical company spending on advertising Other

NIHR / England € 900 million (₤ 819 million in 2008/09)

45 in the ministry of Health; 22 for outsourced services. In the network: unknown

Budget of the ministry of Health

SBU / Sweden € 7-14 million 40 FTE Industry associations

Objectives and rationale

All three agencies have mission statements which form the fundament for the definition of more specific goals. In the mission statements of the HAS and the NIHR, it is explicitly described that the work of the agencies focuses on improving health care for the patients and the general public. The defined goals of the organisations are summarised in the table below.

Table 4.49: Summary of goals of the three agencies HAS NIHR SBU Establish the HNS as an

internationally recognized centre of research excellence

Attract, develop and retain the best research professionals to conduct people-based research

Commission research focused on improving health and social care

Assess health care technology from medical, economic, ethical and a social standpoint

Develop the concentration and collaboration with the main players of the health care system in France and abroad (Research)

Strengthen and streamline systems for research management and governance

Assist public authorities in their decision to reimburse medical products and services. (Opinions)

Act as a sound custodian of public money for public goods

Promote good practices and good usage of treatment for health professionals and users (Recommendations and guidelines)

Improve the quality of health care in health care organisations (HCOs) and in general medical practice (Certifications and evaluations of practices)

Inform health professionals and public and improve the quality of the medical information (Dissemination)

Compile scientifically based assessment reports to support decision making in health care

Note to table: A blank cell in the table implies that the agency does not have a specified goal that is comparable to the goals of the other agencies.

104

The HAS and the NIHR have the most elaborated set of goals. An attempt has been made to compare the different goals of the three agencies. This comparison gives insight in the differences between the agencies.

There are only a few goals that show some similarities between the agencies. More important are the differences, which refer to the specific tasks of HAS in comparison to the other two agencies: namely to assist public authorities in reimbursement decisions; produce guidelines; and provide quality certificates. It is striking that NIHR is the only agency that does not have an explicit goal about the dissemination of knowledge, although this is an important part of the working of NIHR. The SBU does not rely on their own primary research but use existing scientific research for their reports.

4.8.4 Proposal selection

All three agencies accept study proposals from very different sources, including health care users or individuals. However, there are differences in the way the research is prioritised. Within HAS, priority is given to proposals of the Ministry of Health and the research agenda is in accordance with the policy agenda for the health care sector. SBU selects topics that are of major importance to public health and quality of life. The NIHR uses a variety of selection mechanisms, adjusted to the needs of individual research programmes.

The types of proposals of all three agencies are policy and practice oriented. For the NIHR there are even proposals that refer to a strategic level of decision making.

One of the conclusions about the proposal selection by KCE in this report is that the selection procedure remains completely internal. The same applies to HAS, where the selection procedure is internally determined in correspondence with the policy agenda in the health care sector. Within HAS, priority is currently given to the Ministry as a client. The SBU selection process is based on an exploratory study of possible topics; the actual selection is carried out by the SBU Board and the Scientific Advisory Committee. NIHR uses a set of general principles to arrive at their selection, which is peer reviewed by committees in which also other stakeholders participate, such as NHS managers and patient representatives.

4.8.5 Research process

The three agencies have organised the involvement of stakeholders in all stages of the research process, although each agency employs a different rationale to bring this about. For the HAS, the general principle is to address stakeholders in each project. In Sweden the basic idea is that leading thematic experts are essential for achieving maximal impact. The NIHR has a much broader function than the other two agencies, and also involves stakeholders for the different functions of the NIHR, from research planning to implementation.

The SBU does not explicitly formulate the intended impacts of their projects. In general it is not until the end phase of the project that the desired impact is defined. One of the requirements for NIHR to fund a project is that the research results must be translated into deliverable outcomes for NHS patients. HAS asks the client in advance to fill in a special form about the intended impact and measures of this impact.

In accordance with the formulation of the indented impact, the target audience is also formulated beforehand in the HAS and NIHR proposals. The SBU identifies specific target groups at the beginning of a project as well.

The three organisations have in common with KCE the identification of a fairly broad target audience, which includes policy makers and related institutions, scientific community, medical professionals and patients and the public at large. Of the three organisations, NIHR most extensively focuses on programmes to reach medical practitioners, such as nurses, doctors, specialists and hospital managers.

4.8.6 Dissemination

As can be seen in table 4.40, the NIHR uses the widest variety of instruments to disseminate products of all kinds of research to different target groups. Also SBU and HAS use an extended set of instruments for their dissemination purposes. To maximise the impact of their reports, SBU tailors the combination of dissemination activities in line with the topic. For HAS, the improvement of dissemination activities is one of the priority actions in the next three years.

105

Table 4.50 Most important instruments to disseminate knowledge of HAS, SBU and NIHR

HAS SBU NIHR Monthly letter Newsletter Atonement list Articles in specialised press Results need to be published Reports on website HAS Reports on website SBU Press communications Mailings to medical journals Centre for Review and

Dissemination (CRD) English version of website

English publications/Cochrane library

UK PubMed Central in which research findings are published and freely available

Yearly conference Participation in conferences Conferences (within research process)

Courses Schools for Primary Care and Social Care Research

Training activities Service Delivery and Organisation programme (SDO)

Dissemination through networks CLARHCs Buying space in press Tailor made projects

The approaches towards dissemination of HAS and SBU are fairly similar, although SBU uses a more elaborate combination of instruments. Examples are the participation in conferences, the organisation of courses and trainings, and the dissemination through networks. The overview indicates the extraordinary position of the NIHR, with specific organisations and facilities having been set up to disseminate knowledge about the research and the practical application thereof. One of the most interesting facilities of the NIHR are the regionally operating CLAHRCs (Collaborations for Leadership in Applied Health Research and Care) in which professionals from universities and surrounding NHS organisations work together. SBU dissemination activities include also participation in conferences. NIHR regards participation in conferences as a normal part of the research process carried out by researchers (certainly when research activities are expected to result in a scientific publication) and funds this dissemination activity accordingly.

In their dissemination, KCE focuses in particular on communicating the results of their studies to the political and institutional originators of the research and to the scientific community, and less specifically to the actors of the medical field, the patients and the public at large. SBU and NIHR explicitly intend to address a wider range of stakeholders. For example, NIHR uses a wide variety of facilities and instruments to disseminate their research, whilst the SBU initially used a network of local SBU ambassadors to initiate and promote local efforts to assure that the SBU reports are used by decision makers and clinical practitioners. HAS is presently in the process of improving their dissemination strategy.

4.8.7 Contribution to the decision-making process

One of the goals of HAS is to assist public authorities in their decisions to reimburse medical products and services. By executing this task HAS may be expected to contribute to the decision making process. However, this goal is specific for HAS as none of the two other agencies mentions this specific task explicitly.

SBU has monitored the knowledge of their projects among specific stakeholders through surveys with questions on the extent to which they are familiar with SBU and their reports. The surveys have now been discontinued, as the results of these surveys were mostly positive. This may be related to the already long standing position of SBU in Sweden. There are no specific measures for monitoring whether policy makers are influenced by SBU’s research findings. According to a SBU director, the work of SBU is clearly reflected in the political debate.

The maturity of SBU is also indicated by the decision to stop the programme for recruiting local ambassadors. This programme was initially deemed necessary to build up relationships and create awareness of the work of SBU at local level. Nowadays these structures are already in place and, apparently, the ambassador programme is no longer needed for SBU.

The NIHR organises perception audits amongst practitioners, academics and members of industry and other organisations. So-called Policy review programmes monitor whether policy makers are influenced by the research findings.

106

4.8.8 Challenges and conclusions

One of the main lessons for the NIHR in achieving high levels of impact is that the bottleneck in improving the health care system is the dissemination of already established knowledge, rather than the amount of research and the production of new knowledge as such.

One of the main challenges for the SBU is that (similar to KCE) the SBU does not have legal power to influence the utilisation of research findings into adaptations of practices and procedures. Therefore SBU started in 2006 developing a new strategy and long-term plan aiming at strengthening the impact of its work. The new strategy, adopted in 2007, covers three main aspects:

cooperation between authorities and organisations on a national, regional and local level;

strengthening medical evaluation in Sweden;

improving the internal efficiency.

In general, SBU is very satisfied working together with external experts instead of doing work in-house. Through the external experts, the research is much more integrated in health care practice, from the beginning of the process until the end.

Similarly, the NIHR is a network organisation that builds on involving experts and practitioners from outside the central organisation, in al parts of its activities. NIHR incorporates cooperation with academics, NHS practitioners and managers, people from charities and industries, and last but certainly not least, patients and members of the general public.

The general conclusion about the international comparison is that the three agencies may differ with respect to their mission, goals, institutional design, budget and staffing, yet insights can be gained from the international comparison that can be of value for KCE. When compared to KCE, the NIHR appears to be the most extraordinary of the three agencies and the SBU most similar. This applies not only to the organisations’ budget, staffing and scale, but also to the actual activities of the agencies.

The general picture is that all three agencies seem to have a significant task with respect to disseminating knowledge and contributing to decision making, and addressing medical practitioners. For SBU and in particular for the NIHR these are determining factors in the proposal selection and the research process. The entire set-up (network) of NIHR is geared towards wider dissemination and influencing decision-making, while within the SBU specific efforts are made to influence health care management and practitioners.

It may be worthwhile for KCE to study specific details of these examples, paying particular attention to the working of the NIHR with respect to effective knowledge dissemination and impact monitoring.

107

5 DISCUSSION AND CONCLUSION

5.1 DISCUSSION For the understanding of the results of this impact study, it is crucial to consider its scope, strengths and limitations.

5.1.1 Scope

The scope of this impact study is limited to those aspects directly influenced by KCE. In consultation with KCE management it was decided not to incorporate cost savings and impact in terms of improved health. We specifically focused on ways in which KCE created the conditions to enhance the impact of its research findings. Also, albeit to a lesser extent, we gained insight into the perception and image of KCE and its role in Belgian health care.

This impact study is not an assessment of KCE as an institute. The focus is mainly on KCE research projects. The impact study has a process oriented approach in the sense that it studies the process of KCE’s research agenda setting, the research process itself and the process of dissemination leading up to the (possible) contribution to decision-making processes.

Within this process the study focuses on KCE’s intended and actual interactions with stakeholders, to assess whether these are aimed at creating high-quality, well-grounded, feasible and usable conclusions and recommendations that are being received and accepted by stakeholders and target groups.

The question on whether the findings are actually being used and whether impact has been achieved is answered by capturing:

a. The general impressions of stakeholders on the impact of KCE and random examples of research projects. These impressions are captured by face to face interviews.

b. A more detailed picture at project level generated from the perception of project coordinators. They were asked for evidence of impact on decision making processes. The answers were collected through a web based questionnaire.

c. Examples of the use of findings and the impact of projects as found in the nine case studies we conducted. In the case studies, internal and external parties who had been involved in the projects have been interviewed.

5.1.2 Strengths and limitations

Strengths Choosing a process-oriented approach - The strength of process-

oriented approach for this impact study is that we not only looked at the impact of KCE studies, but also analysed the process leading towards the achievement of impact. Analyzing this process in particular provides building blocks for improving the impact of KCE.

Building on state of the art knowledge about impact assessments - Findings from the literature review preceding the impact assessment were used by the research team in constructing the research framework. Hypotheses formulated on the basis of impact assessment literature served as a basis for building the research design. Furthermore, each core question has been elaborated into different topics using the findings from the literature review.

Using multiple sources of information – this makes it possible to triangulate the findings from different sources and go beyond the perceptions of specific groups of respondents, each having their own subjective outlook towards KCE.

Combining qualitative and quantitative data collection and analysis methods - This avoids having to deal with the limitations inherent to selecting just one methodological approach. For example, an exclusively qualitative approach provides either only rich

108

qualitative information which may not be possible to generalize to the KCE projects in general. Alternatively, an exclusively quantitative approach provides quantifiable data but may fail to grasp the underlying reasons or explanatory factors. Instead, this study combines the strengths of multiple research methodologies and draws on both internal and external sources to arrive at sound overall conclusions.

Evaluation carried out by impartial policy research organization - This impact study is carried out by a policy research company and research team that are not in any way connected to, or have ties with the government responsible for establishing and subsidizing KCE, nor with parties that have or could have interests in the outcomes of this study. As a consequence, the member of the research team has a fresh, impartial and relatively objective view on the matters that have been studied.

Limitations The model used to study the impact is somewhat linear. It is

important to bear in mind that this inevitably abstracts from various feedback loops and secondary effects. The process by which science translates into policy decisions and practice is not as linear as depicted in the model we used. Nevertheless, the advantage of this logical model is that it provides a workable and logical framework within which to evaluate the outputs and outcomes of research and to find ways in heighten the probability of impact.

Conflicts of interest may be suspected as the impact assessment has been assigned by the KCE itself. Also much of the information about the impact comes from the researchers themselves or from the originators of the research. However, the respondents consulted in this impact study seemed to be open, (self) critical and seeking for learning opportunities. Moreover, both the views of positive and critical stakeholders are presented in this research report.

All data collection phases have been carried out simultaneously instead of in sequence. The advantage was that the total process of the data collection could be finalized within a three month period (which was the requested timeframe). As a consequence, the analysis of outcomes from one source (for example the stakeholder interviews) could not be used as input for other data collection phases (for example in case studies). Carrying out the data collection in sequence could have helped to devise better data collection instruments and get more clarification on (not) achieving impact.

In this study we interviewed 20 stakeholders representing 20 stakeholder organisations. The selection of stakeholders may imply a potential bias. First of all the selected respondents, one per organisation, may not necessarily represent the general view of their organisations. On the other hand the respondents selected in general had leading, representing and influential positions within their organisation. Secondly, the selection of stakeholders was done by KCE which could for example lead to the selection of less critical respondents. However we have indications that KCE carefully selected the stakeholders, aiming to cover most of the important organizations in the field of health care.

Given the limited time and resources we had to restrict to nine case studies. Also the time to study each case was limited (three days). The aim of the case studies was therefore to give examples of KCE research projects and to give indications on possible success and fail factors. It is important not to draw overall conclusions on KCE’s impact based only on the case study results.

5.2 CONCLUSIONS On the basis of the literature reviewed and the rationale underlying KCE’s activities, the research team has formulated a set of basic ‘hypotheses’ serving as a basic guide for the impact assessment. The hypotheses refer to impact-enhancing factors derived from literature about impact, still within the field of influence of KCE itself. The central theory guiding this conclusion is that the more

109

impact-enhancing factors are present in KCE’s activities during the process from proposal selection to dissemination of the results, the greater the probability will be that KCE did in fact achieve a certain impact on policy making and/or on health care practice.

First in this paragraph all hypotheses formulated will be addressed. We answer the question to what extent KCE has incorporated these impact-enhancing factors in its activities. Next, in a general conclusion, we give an overall view on the findings and answer the question whether KCE actually achieved impact.

5.2.1 Conclusions on hypotheses

Hypothesis 1: Carrying out a transparent, careful proposal selection procedure that is primarily aimed at assisting decision making in health care will increase the chances of research findings being accepted and usable.12

Transparency: Since 2005, KCE uses a proposal selection procedure. The procedure and criteria are publicly available on the website. The process around prioritization and selection within the Board meetings is recorded in the minutes. These minutes are available on request. The results of the scoring procedure are communicated to the originators. In addition, the procedures provide room to add urgent requests by certain institutional stakeholders to the programme during the year without going through prioritization and selection.

Whether the proposal selection is transparent, is perceived differently between groups of stakeholders. For the majority of the respondents of RIZIV/INAMI and government it is transparent. For other stakeholders, it is in general less clear on what grounds the proposals are admitted or refused to the work programme. Furthermore, they judge the selection to be biased towards proposals from the Federal Public Services and RIZIV/INAMI. To internal experts it is also not always clear why certain final selection choices are being made by the management team and the board.

Carefulness: Formal procedures and criteria have been developed for the prioritisation of proposals. The criteria have been summarily defined and therefore the scoring depends largely on the KCE experts interpretation. The final stages in the selection process depend on the specific political and budgetary context. This stage is carried out in a careful way by KCE management and board, but this isn’t always clear for external parties.

Aiming at assisting decision making in health care: One of the scoring criteria is “importance in terms of health care policy support”. In this criterion, RIZIV/INAMI and the Federal Public Services are explicitly mentioned. Proposals from RIZIV/INAMI, Federal Public Services and Ministers automatically receive a higher score.1 Which, in itself makes sense, because by law KCE is legally obliged to perform studies for at least these organisations.

Hypothesis 2: Formulating in advance a specific and (stakeholder) relevant goal for a research project will improve the chances of the final product being usable for stakeholders.32

For the majority of projects a specific goal was formulated in advance. No internal procedures have been developed concerning the formulation of specific and (stakeholder) relevant goals for research projects. In many cases, the originator of the study has been involved as a stakeholder in the formulation of research goals and questions.

Hypothesis 3: Formulating explicit goals at the start of a project for the involvement of users during the project and for dissemination of the findings will increase the chances of the final product being effectively disseminated among stakeholders.12, 13

Formulating goals for the involvement of stakeholders: Stakeholder involvement in projects is arranged predominantly through meetings with the external expert team. Initially, the external experts were specifically invited on the basis of their scientific merit. Increasingly, experts from stakeholder groups and organizations are also invited on the expert team. The main goal for the involvement of external experts is enhancing the quality of the research, through critical reflections on the content and methodology of the research. For most projects, no specific or additional goals for stakeholder involvement have been formulated.

110

Involvement of stakeholders as a way to increase the chances of the final product being effectively disseminated is not common practice within project teams.

Formulating goals for dissemination of the findings: General (process-oriented) dissemination goals have been formulated in the process documents. The research team found no proof of project-specific (outcome-oriented) goals for dissemination of the findings having been formulated.

Hypothesis 4: Performing high quality research processes (project management and methodology) will increase the chances of achieving the formulated project goals for a specific research project.32

High quality methodology: Several internal procedures have been developed to safeguard the scientific quality of the research process and to guide the validation and finalization process. Overall, internal as well as external project coordinators appreciate these procedures and their role in the research process. Also, they are generally positive about the scientific underpinning of the findings. Most stakeholders judge the quality of KCE’s research process as high. Some stakeholders and external researchers feel that in specific cases the methodological procedures are too rigid and are hampering research questions to be adequately answered.

High quality project management: Several internal procedures have been developed to safeguard the quality of the project management. Almost three out of four external project coordinators say that the KCE project manager has adequately taken care of preserving the focus of the study and of the results to be achieved.

Hypothesis 5: Actively engaging relevant stakeholders during a research project will improve usability and acceptance of findings and, subsequently, the chance of implementation.10, 32

Apart from the formation of external expert teams, there are no internal procedures concerning the involvement of stakeholders during the research process. Although stakeholders are increasingly being invited to join external expert groups, this is done with the aim of enhancing the (scientific) research quality and not so much to enhance the usability and acceptance of the findings. This seems to be informed by a fear to loose objectivity and independence. Currently, each project team decides per project which stakeholders to involve, at what stage and for what reason. The absence of procedures and other forms of guidance in this regard from KCE management hampers a more structural, uniform and balanced involvement of all relevant stakeholders.

Hypothesis 6: Making clear the relationship between research findings and the decision-making processes will improve the chances of the final product being usable for stakeholders.32

In most cases, there is either an implicit or explicit link with decision-making in the report and recommendations. For most projects with an instrumental impact, this link seems clear to the originator of the study (through the report, recommendations and a presentation). Apart from the clear link to the originator, KCE does not structurally engage itself in more explicitly linking research findings to decision-making processes.

The main ways in which KCE disseminates its results are through mailings, press releases and publishing the reports and recommendations on its website. Opportunities for presentations and participating in discussion sessions are taken up by invitation (passive), but are not actively promoted by KCE. KCE management as well as some originators and other stakeholders do not view it as KCE’s role to push for their recommendations to be followed-up.

Hypothesis 7: Actively performing dissemination activities (based on a dissemination strategy) after a project has finished will increase the chances of the findings actually being used by stakeholders.12, 33

There is a general dissemination (communication) strategy. KCE has chosen a dissemination strategy that can be described as fairly passive and limited. The main ways in which KCE disseminates its research results is through mailings, press releases, and publishing the reports and recommendations on the website. Most communication tasks are being carried out by the KCE management, Board of Directors and the communication staff. KCE experts are supposed to just

111

communicate the research results to the scientific world. Limited time is reserved for developing articles and project-bound communication plans are not developed. Interactive activities (by KCE experts) are kept to a minimum. External stakeholders consider KCE’s dissemination and communication activities towards stakeholders as insufficiently proactive, strategic, and structural. KCE management and some of the Board members state it is a deliberate policy to be restrained, in order to stay as independent as possible.

Hypothesis 8: Identifying stakeholder(s) as possible user(s) of a specific product and tailoring this product to the characteristics and needs of this/these stakeholder(s) will increase the chances of the product actually being used.33

The originator of the study is generally identified as the main user of the research outputs. The institutional originators seem mostly satisfied with the format, content and readability of the outputs (report, recommendations, and presentation). Outputs are generally not specifically tailored towards the specific situation and needs of any other target users. This may not be problematic as far as political users are concerned, but more so with regard to actors in the field of health care practice (format, terminology, formulation of findings and recommendations). Exemplary are the guidelines developed as an output of Good Clinical Practice projects. These are aimed at actors in health care practice but are not always sufficiently read and understood by the practitioners. However it can be argued that it is not the role of KCE to translate its findings to practice.

Hypothesis 9: Structural feedback from (possible) end-users on usability of finished products and incorporating lessons from this in the research process will increase the chances of improved usability.10

The KCE procedures and instructions for external validators request, amongst others, that they judge the readability of the final report. In many cases, the research results are presented to the originator of the study, who may provide feedback but is not obliged to. Apart from this, KCE does not ask for feedback from (possible) end-users on usability of finished products on a structural basis. External feedback received by KCE’s communication staff has not been used by KCE experts to improve usability in the future.

Hypothesis 10: A positive image of KCE (e.g. on independence, responsiveness and research quality) amongst stakeholders, will increase the chances of the findings from individual research projects being recognized, acknowledged and used by these stakeholders.33

The external image of KCE is evidently positive regarding the scientific quality of its studies, its knowledge and expertise on health care issues, the competency of its researchers and its serviceability. However, KCE’s ability to make findings usable for policy decision making and the way in which KCE communicates with stakeholders is judged average. This may imply that KCE can improve on translating high-quality research findings into relevant, easy-to-use input for decision making processes.

The external image of KCE concerning its independence and trustworthiness is somewhat worrisome. Although the majority rates this at average or high, frequent criticism (especially by association of professionals or care providers) is being expressed on the large amount of (uneven) influence external parties are perceived to exert on the contents of the final conclusions and recommendations.

5.2.2 General conclusions

The question whether KCE research findings have had an impact on policy making processes can be answered using two lines of reasoning:

the probability of achieving impact on the basis of KCE’s internal processes (whether conditions are fulfilled); and

the actual use of research findings reported by respondents in this impact study.

5.2.2.1 Conditions

It’s clear that KCE has established important conditions to achieve impact on policy making in health care:

Research questions come from primary stakeholders There is involvement from these stakeholders in agenda and priority setting,

112

The quality of KCE research itself is high and in general beyond discussion.

The relevance of KCE research findings is generally judged as high.

However, some important conditions for achieving impact are not fully realized yet. These conditions are:

1. Transparent proposal selection. It is clear KCE aims at making its proposal selection as transparent and careful as possible. It also is viewed like that by most stakeholders. However, it is perceived as unclear by part of the stakeholders (especially associations of care providers or professionals). Most important factor causing the lack of clarity is the final step in proposal selection: the priority setting on policy relevance and the final outcome (KCE year programme). An important part of this research agenda is aimed at assisting decision making for RIZIV/INAMI, Federal Public Services and Ministers. This adds to the impression some of the stakeholders have about KCE not being independent or impartial. Another way of programming the research could help to improve the external image on independency.

2. The usability of research findings. Most of the research findings are judged as usable and the ability of KCE to make research findings usable as average. It could be improved by:

another way of involving stakeholders during the research process. In stead of focusing on research quality, there could be a stronger focus on the use of research findings. This could be possible with a clear division of roles, where the focus of KCE is on the scientific quality and integrity of research and the focus of stakeholders on the usability (from the beginning of the project).

Dissemination is mainly passive. It is clear that the role of KCE is not to implement or push research findings into policy or practice, however a more active approach could be adopted to improve the degree in which research findings are implementable (by increasing the usability). This could maybe be done by a stronger cooperation with representing or intermediary organizations, like associations for professionals or health providers. These institutions can more easily reach the field of health care practice. Examples of different ways to actively disseminate and engage stakeholders can be found in foreign research agencies.

5.2.2.2 Actual use of findings (impact)

From this impact study it is clear KCE research findings are actually being used in decision making in health care. In total 16 out of 20 stakeholders say research findings influence decision making, varying from being the main source for a decision to one of the factors influencing decision making. From the project coordinators approximately 60% of the project coordinators report research findings contributed to decision making. The majority of these coordinators (58%) state that the results have been used in policy negotiations and/or to underpin existing policy intentions or policy preparations (52,7%). According to project coordinators over a third of the projects has (also) resulted in the implementation of evidence based policy (38,9%), been cited in policy documents (37,5%) and resulted in involvement of health care users of the public (33,3%). Least frequently (20,9%), the results were (also) used in parliamentary debates.

5.2.2.3 Overall conclusion

KCE is a relatively young research institute and already established important conditions to achieve a high level of impact. It shows in the fact research findings are actually used in Belgian health care policy. Clearly there is room for further improvements. By assigning an impact study that presents opinions of all important stakeholders and focuses on critical elements, KCE shows it is open to feedback and opportunities to learn, which in itself is a condition for excellence.

113

6 REFERENCES 1. Loi programme: Chapitre 2 - Création du Centre fédéral d'expertise des soins de santé,

Moniteur Belge 24 dec. 2002. Available from: http://www.ejustice.just.fgov.be/

2. Gourbin C, du Boullay D, Philips H, Remmen R, Buylaert W, De Paepe P, et al. Evaluatie van forfaitaire persoonlijk bijdrage op het gebruik van spoedgevallendienst. Report. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2005. KCE reports 19A (D/2005/10.273/21)

3. De Graeve D, Lecluyse A, Schokkaert E, Van Ourti T, Van de Voorde C. Eigen betalingen in de Belgische gezondheidszorg. De impact van supplementen. Equity and Patient Behaviour (EPB). Brussel: Federaal Kenniscentrum voor de gezondheiszorg (KCE); 2006. KCE reports 50A (D/2006/10.273/68)

4. Schokkaert E, Guillaume J, Lecluyse A, Avalosse H, Cornelis K, De Graeve D, et al. Evaluatie van de effecten van de maximumfactuur op de consumptie en financiële toegankelijkheid van gezondheidszorg. Equity and Patient Behaviour (EPB). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2008 2008-07-01. KCE reports 80A (D/2008/10.273/35)

5. Van De Sande S, De Wachter D, Swartenbroekx N, Peers J, Debruyne H, Moldenaers I, et al. Inventaris van databanken gezondheidszorg. Objective Elements - Communication (OEC). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2006. KCE reports 30A (D2006/10.273/14)

6. Vlayen J, Van De Water G, Camberlin C, Paulus D, Leys M, Ramaekers D, et al. Klinische kwaliteitsindicatoren. Objective Elements - Communication (OEC). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2006. KCE reports 41A (D/2006/10.273/43)

7. KCE. Jaarlijks verslag 2008 van het Federaal Kenniscentrum voor de Gezondheidszorg. Annual report. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2009. Jaarlijks verslag (1783-4112)

8. KCE. Jaarlijks verslag 2007 van het Federaal Kenniscentrum voor de Gezondheidszorg. Annual report. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2008. Jaarlijks verslag (1783-4112)

9. Hanney S, Buxton M, Green C, Coulson D, Raftery J. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess. 2007;11(53):iii-iv, ix-xi, 1-180.

10. The impact of the NHS Service Delivery and Organisation Research and Development Programme 2001–2006. London: National Coordinating Centre for the Service Delivery and Organisation (NCCSDO). Available from: http://www.sdo.nihr.ac.uk/files/adhoc/impact-study-full.pdf

11. Oortwijn WJ, Hanney SR, Ligtvoet A, Hoorens S, Wooding S, Grant J, et al. Assessing the impact of health technology assessment in The Netherlands. Int J Technol Assess Health Care. 2008;24(3):259-69.

12. Hanney S, Soper B, Buxton M. Evaluation of the NHS R&D Methods Programme. Uxbridge: Brunel University; 2003. HERG Report 29

13. Meagher L, Lyall C, Nutley S. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Research Evaluation. 2008;17(3):163-73.

14. Alberta Heritage Foundation for Medical Research. A Study of the Impact of 2001-2002 Health Technology Assessment Products - Final Report. Edmonton: 2003. IP-15 Information Paper Available from: www.ahfmr.ab.ca/download.php/5e65b25cc286e5f0d3cd74a3008fdd53

15. Raftery J, Hanney S, Green C, Buxton M. Assessing the impact of England's National Health Service R&D Health Technology Assessment program using the "payback" approach. Int J Technol Assess Health Care. 2009;25(1):1-5.

16. Kwan P, Johnston J, Fung AY, Chong DS, Collins RA, Lo SV. A systematic evaluation of payback of publicly funded health and health services research in Hong Kong. BMC Health Serv Res. 2007;7:121.

17. Wooding S, Hanney S, Buxton M, Grant J. Payback arising from research funding: evaluation of the Arthritis Research Campaign. Rheumatology (Oxford). 2005;44(9):1145-56.

114

18. Buxton M, Hanney S, Packwood T, Roberts S, Youll P. Assessing Benefits from Department of Health and National Health Service Research & Development. Public Money & Management. 2000;20(4):29-34.

19. Tetroe JM, Graham ID, Foy R, Robinson N, Eccles MP, Wensing M, et al. Health research funding agencies' support and promotion of knowledge translation: an international study. Milbank Q. 2008;86(1):125-55.

20. Ottoson JM, Green LW, Beery WL, Senter SK, Cahill CL, Pearson DC, et al. Policy-contribution assessment and field-building analysis of the Robert Wood Johnson Foundation's Active Living Research Program. Am J Prev Med. 2009;36(2 Suppl):S34-43.

21. Landry R, Amara N, Lamari M. Utilization of social science research knowledge in Canada. Research policy 2001;30(2):333-49.

22. Carden F. Issues in assessing the policy influence of research. International Social Science Journal. 2004;26(179):3-4.

23. Hivon M, Lehoux P, Denis J-L, Tailliez S. Use of health technology assessment in decision making: coresponsibility of users and producers? Int J Technol Assess Health Care. 2005;21(2):268-75.

24. Gagnon MP, Legare F, Fortin JP, Lamothe L, Labrecque M, Duplantie J. An integrated strategy of knowledge application for optimal e-health implementation: a multi-method study protocol. BMC Med Inform Decis Mak. 2008;8:17.

25. Gagnon M-P, Sánchez E, Pons JMV. Integration of health technology assessment recommendations into organizational and clinical practice: A case study in Catalonia. International Journal of Technology Assessment in Health Care. 2006;22(2):169-76.

26. Bodeau-Livinec F, Simon E, Montagnier-Petrissans C, Joel ME, Fery-Lemonnier E. Impact of CEDIT recommendations: An example of health technology assessment in a hospital network. Int J Technol Assess Health Care. 2006;22(2):161-8.

27. Stryer D, Tunis S, Hubbard H, Clancy C. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res. 2000;35(5 Pt 1):977-93.

28. Molas-Gallart J, Tang P, Morrow S. Assessing the non-academic impact of grant-funded socio-economic research: results from a pilot study. Research Evaluation. 2000;9(3):171-82.

29. Jacob R, McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997;13(1):68-80.

30. Bensing JM, Caris-Verhallen WM, Dekker J, Delnoij DM, Groenewegen PP. Doing the right thing and doing it right: toward a framework for assessing the policy relevance of health services research. Int J Technol Assess Health Care. 2003;19(4):604-12.

31. Lafortune L, Farand L, Mondou I, Sicotte C, Battista R. Assessing the performance of health technology assessment organizations: a framework. Int J Technol Assess Health Care. 2008;24(1):76-86.

32. Drummond MF, Schwartz JS, Jonsson B, Luce BR, Neumann PJ, Siebert U, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24(3):244-58; discussion 362-8.

33. McBride T, Coburn A, Mackinney C, Mueller K, Slifkin R, Wakefield M. Bridging health research and policy: effective dissemination strategies. J Public Health Manag Pract. 2008;14(2):150-4.

34. Deloitte, KCE. Masterplan KCE. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2003.

35. Ramaekers D, Closon J-P. Plan de Management. Bruxelles: Centre fédéral d'expertise des soins de santé (KCE); 2003. Available from: http://kce.fgov.be/Download.aspx?ID=81

36. Nielens H, Van Zundert J, Mairiaux P, Gailly J, Van Den Hecke N, Mazina D, et al. Chronic low back pain. Good Clinical practice (GCP). Brussels: Belgian Health Care Knwoledge Centre (KCE); 2006 19/02/2007. KCE reports 48C (D/2006/10.273/71)

37. Lodewyckx K, Peeters G, Spitz B, Blot S, Temmerman M, Zhang W, et al. Nationale richtlijn prenatale zorg. Een basis voor een klinisch pad voor de opvolging van zwangerschappen. ReportRapport. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2004 24 december. KCE reports 6A (D/2004/10.273/13) Available from: http://kce.fgov.be/Download.aspx?ID=291

38. Van Den Steen D, Vanobbergen J, De Visschere L, Vermeersch M, Lebrun J-M, Crott R, et al. De kost van tandprothesen. Health Services Research (HSR). Brussel: Federaal

115

Kenniscentrum voor de Gezondheidszorg (KCE); 2005 April. KCE reports 10A (D/2005/10.273/03) Available from: http://kce.fgov.be/Download.aspx?ID=260

39. Lambert M-L, Kohn L, Vinck I, Cleemput I, Vlayen J, Leys M, et al. Pharmacological and surgical treatment of obesity. Residential care for severely obese children in Belgium. Health Technology Assessment (HTA). Bruxelles: Belgian Health Care Knowledge Centre (KCE); 2006. KCE reports 36C (D/2006/10.273/30)

40. Lannoo E, Larmuseau D, Van Hoorde W, Ackaert K, Lona M, Leys M, et al. Chronische zorgbehoeften bij personen met een niet- aangeboren hersenletsel (NAH) tussen 18 en 65 jaar. Health Services Research (HSR). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2007. KCE reports 51A (D/2007/10.273/01)

41. Denuit M, Maréchal X, Closon J-P. Studie naar de mogelijke kosten van een eventuele wijziging van de rechtsregels inzake medische aansprakelijkheid. Fase II : ontwikkeling van een actuarieel model en eerste schattingen. Report. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2005. KCE reports 16A (D/2005/10.273/15) Available from: http://kce.fgov.be/Download.aspx?ID=269

42. Maréchal X, Denuit M, Vinck I, Closon J-P. Vergoeding van schade als gevolg van gezondheidszorg – Fase IV : Verdeelsleutel tussen het Fonds en de verzekeraars. Health Services Research (HSR). Brussel: Federaal Keniscentrum voor de Gezondheidszorg (KCE); 2007. KCE reports 68A (D/2007/10.273/52)

43. Huybrechts M, Cleemput I, Devriese S, Hulstaert F. Terugbetaling van Radioisotopen in België. Health Services Research (HSR). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2008. KCE Reports 79A (D/2008/10.273/25)

44. Vlayen J, Verstreken M, Mertens C, Van Eycken E, Penninckx F. Kwaliteit van rectale kankerzorg – Fase 2: ontwikkeling en test van een set van kwaliteitsindicatoren. Good Clinical Practice (GCP). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2008. KCE reports 81A (D/2008/10.273/38)

45. Vlayen J, Camberlin C, Ramaekers D. Vacuümgeassisteerde Wondbehandeling: een Rapid Assessment. Health Technology Assessment (HTA). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2007. KCE reports 61A (D2007/10.273/30)

46. Hulstaert F, Huybrechts M, Van Den Bruel A, Cleemput I, Bonneux L, Vernelen K, et al. HTA Moleculaire Diagnostiek in België. Health Technology Assessment (HTA). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2005 Oktober. KCE reports 20A (D2005/10.273/23)

47. Beutels P, Van Damme P, Oosterhuis-Kafeja F. Effecten en kosten van de vaccinatie van Belgische kinderen met geconjugeerd pneumococcenvaccin. Health Technology Assessment (HTA). Brussel: Federaal Kenniscentrum voor de gezondheidszorg (KCE); 2006. KCE reports 33A (D/2006/10.273/21)

48. De Laet C, Neyt M, Van Brabandt H, Ramaekers D. Rapid Assessment: Cardiovasculaire Primaire Preventie in de Belgische Huisartspraktijk. Good Clinical Practice (GCP). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2007. KCE reports 52A (D/2007/10.273/03)

49. Van den Bruel A, Gailly J, Devriese S, Vrijens F, Ramaekers D. Longfunctie testen bij volwassenen. Good Clinical Practice (GCP). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2007. KCE reports 60A (D2007/10.273/27)

50. Mathieu C, Nobels F, Peeters G, Van Royen P, Dirven K, Wens J, et al. De kwaliteit en de organisatie van type 2 diabeteszorg. Good Clinical Practice (GCP). Brussels: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2006. KCE reports 27A (D2006/10.273/07)

51. de Walcque C, Seuntjens B, Vermeyen K, Peeters G, Vinck I. Vergelijkende studie van ziekenhuisaccrediteringsprogramma’s in Europa Health Services Research (HSR). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2008. KCE Reports 70A (D/2008/10.273/01)

52. De Gauquier K, Senn A, Kohn L, Vinck I. Internationale vergelijking van terugbetalingsregels en juridische aspecten van plastische heelkunde. Health Services Research (HSR). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2008. KCE reports 83A (D/2008/10.273/43)

53. MARSH : Amsoms K, IMA : Zegers P, Van Ingh D, Avalosse H, Caeyers-De Hondt A-M, Corremans B, et al. Studie naar de mogelijke kosten van een eventuele wijziging van de rechtsregels inzake medische aansprakelijkheid (fase 1). Report. Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2004 juli. KCE reports 2A (D/2004/10.273/3) Available from: http://kce.fgov.be/Download.aspx?ID=281

54. Maréchal X, Denuit M, Vinck I, Closon J-P. Studie naar de mogelijke kosten van een eventuele wijziging van de rechtsregels inzake medische aansprakelijkheid (fase III)-

116

precisering van de kostenraming. Health Services Research (HSR). Brussel: Federaal Kenniscentrum voor de gezondheidszorg (KCE); 2006. KCE reports 35A (D/2006/10.273/26)

55. Yerna B-L, Maréchal X, Denuit M, Closon J-P, Vinck I. Vergoeding van schade ten gevolge van gezondheidszorg – Fase V: Budgettaire impact van de omzetting van het Franse systeem in België. Health Services Research (HSR). Brussel: Federaal Kenniscentrum voor de Gezondheidszorg (KCE); 2009. KCE reports 107A (D/2009/10.273/16)

117

7 APPENDICES

APPENDIX 1: SEARCH STRATEGY Keywords Decision making, policy making, research

utilization, research impact, benefits from research, payback, diffusion, dissemination, methodology, framework

Date 18-03-2009 Database

EconLit, accessed via EBSCOhost

Search Strategy

( TX (methodology OR method OR assessment OR framework OR approach) and TX (impact or relevance or payback or influence or utili?ation or benefit$ or diffusion or dissemination) and TX ("decision mak*" OR "policy mak*" OR policymak* OR decisionmak*) ) and ( TX ("research program*" OR "research findings" OR "research outcomes" OR "research results" OR "research recommendations" OR "research conclusions" OR "research organi?ation$" OR "research institute$") ) Search limited to : Articles published from 1995-2009; Journal articles

Note 41 results, 0 unique articles selected for further reading


Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index (SSCI), Arts & Humanities Citation Index (A&HCI) Accessed via ISI Web of Knowledge [v 4.5] (Web of Science)

Search Strategy

#1: ts=(research-program*) OR ts=(research-findings) OR (ts=research-outcomes) OR ts=(research-results) OR ts=(research-recommendations) OR ts=(research-conclusions) OR ts=(research-organi?ation$) OR ts=(research-institute$) #2 : ts=(methodology OR method OR assessment OR framework OR approach) AND ts=(impact or relevance or payback or influence or utili?ation or benefit$ or diffusion or dissemination) AND ts=(decision-mak* OR policy-mak* OR policymak* OR decisionmak*) AND Language=(English) AND Document Type=(Article) #1 AND #2 Results limited to: Articles published from 1995-2009

Note #1 : 24,415 results. #2 : 2,532 results. #1 AND #2 : 55 results 9 unique articles selected for further reading

118

Articles selected:

1. Policy-Contribution Assessment and Field-Building Analysis of the Robert Wood Johnson Foundation's Active Living Research Program Ottoson, JM; Green, LW; Beery, WL; Senter, SK; Cahill, CL; Pearson, DC; Greenwald, HP; Hamre, R ; Leviton, L AMERICAN JOURNAL OF PREVENTIVE MEDICINE, 36 (2): S34-S43 Suppl. S FEB 2009

2. Health research funding agencies' support and promotion of knowledge translation: An international study Tetroe, JM ; Graham, ID; Foy, R; Robinson, N; Eccles, MP; Wensing, M; Durieux, P; Legare, F; Nielson, CP; Adily, A; Ward, JE; Porter, C; Shea, B; Grimshaw, JM MILBANK QUARTERLY, 86 (1): 125-155 MAR 2008

3. Research-based-decision-making in Canadian health organizations: A behavioural approach. Jbilou, J; Amara, N; Landry, R. JOURNAL OF MEDICAL SYSTEMS, 31 (3): 185-196 JUN 2007

4. Technology assessment and knowledge brokering: The case of assisted reproduction in The Netherlands. van Kammen, J; Jansen, CW; Bonsel, GJ; Kremer, JAM; Evers, JLH; Wladimiroff, JW INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE, 22 (3): 302-306 SUM 2006

5. Policy utilisation of research results in Cameroon. Mbock, CG; Ngo-Mpeck, ML; Kom, D; Belinga, JMZ . INTERNATIONAL SOCIAL SCIENCE JOURNAL, 56 (1): 37-+ MAR 2004

6. The impact of social research on education policy and reform in the Philippines. Miralao, VA. INTERNATIONAL SOCIAL SCIENCE JOURNAL, 56 (1): 75-+ MAR 2004

7. Doing the right thing and doing it right: Toward a framework for assessing the policy relevance of health services research. Bensing, JM; Caris-Verhallen, WMCM; Dekker, J; Delnoij, DMJ; Groenewegen, PP. INTERNATIONAL JOURNAL OF TECHNOLOGY ASSESSMENT IN HEALTH CARE, 19 (4): 604-612 FAL 2003

8. How can research organizations more effectively transfer research knowledge to decision makers? Lavis, JN; Robertson, D; Woodside, JM; McLeod, CB; Abelson, J (Knowledge Transfer Study Group). MILBANK QUARTERLY, 81 (2): 221-+ 2003

9. Utilization of social science research knowledge in Canada. Landry, R; Amara, N; Lamari, M. RESEARCH POLICY, 30 (2): 333-349 FEB 2001

Keywords Health care quality, health care policy, health

technology assessment, health services research, good clinical practice, research utilization, research impact, end-user relevance, payback, methodology, framework


EMBASE & MedLine, access via www.embase.com

Search Strategy

(methodology OR assess* OR framework OR evaluat*) AND (societ* OR policy OR 'non academic' OR 'end user') AND (impact OR relevance OR payback OR influence OR utili?ation OR benefit*) AND ('biomedical technology assessment'/mj OR 'health services research'/mj OR 'good clinical practice'/mj) AND ('health care policy'/de OR 'health care quality'/de) AND [abstracts]/lim AND [2006-2009]/py


119

Articles selected:

1. The use of economic evaluations in NHS decision-making: a review and empirical investigation. Williams I., McIver S., Moore D., Bryan S. Health Technol Assess (2008) 12:7 (iii, ix-x, 1-175)

2. Health care technology adoption and diffusion in a social context. Coyte P.C., Holmes D. Policy Polit Nurs Pract (2007) 8:1 (47-54)

3. Measuring the tail of the dog that doesn't bark in the night: the case of the national evaluation of Choose Life (the national strategy and action plan to prevent suicide in Scotland). Mackenzie M., Blamey A., Halliday E., Maxwell M., McCollam A., McDaid D., MacLean J., Woodhouse A., Platt S. BMC Public Health (2007) 7:- (146)

4. Priority setting for health technology assessments: A systematic review of current practical approaches. Noorani H.Z., Husereau D.R., Boudreau R., Skidmore B. International Journal of Technology Assessment in Health Care (2007) 23:3 (310-315)

5. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Drummond M.F., Schwartz J.S., Jonsson B., Luce B.R., Neumann P.J., Siebert U., Sullivan S.D. International Journal of Technology Assessment in Health Care (2008) 24:3 (244-258)

Keywords Health care quality, health care policy, health

technology assessment, health services research, good clinical practice, research utilization, research impact, end-user relevance, payback, methodology, framework


MedLine via PubMed MeSH Database

Search Strategy

("technology assessment, biomedical"[MeSH Terms] OR "health services research"[MeSH Terms] OR "practice guidelines as topic"[MeSH Terms]) AND ("methodology"[Title/Abstract] OR "assess*"[Title/Abstract] OR "framework"[Title/Abstract] OR "policy"[Title/Abstract] OR "non academic"[Title/Abstract] OR "end user"[Title/Abstract]) AND ("impact"[Title/Abstract] OR "relevance"[Title/Abstract] OR "payback"[Title/Abstract] OR "influence"[Title/Abstract] OR "benefit*"[Title/Abstract] OR "utilization"[Title/Abstract]) AND "health policy"[MeSH Terms] AND 2006[PDAT] : 2009[PDAT]


Articles selected:

1. Bridging health research and policy: effective dissemination strategies. McBride T, Coburn A, Mackinney C, Mueller K, Slifkin R, Wakefield M. J Public Health Manag Pract. 2008 Mar-Apr;14(2):150-4.

2. Use of health impact assessment in the U.S.: 27 case studies, 1999-2007. Dannenberg AL, Bhatia R, Cole BL, Heaton SK, Feldman JD, Rutt CD. Am J Prev Med. 2008 Mar;34(3):241-56.

3. Managing partnerships and impact on decision-making: the example of health technology assessment in genetics. Blancquaert I. Community Genet. 2006;9(1):27-33. Review.

120

Keywords Decision making, policy making, research



PsychINFO, accessed via OvidSP

Search Strategy

(methodology OR assessment OR assessing OR framework OR evaluation).mp. AND (impact or relevance or payback or influence or utilization or benefit$).mp. AND (research or findings or recommendations or conclusions).mp. AND (management decision making OR government policy making).de. Limited to all journals AND abstracts AND yr= “1995 – 2009”


Articles selected:

1. Psychological research, practice, and social policy: Potential pathways of influence. Maton, Kenneth I; Bishop-Josef, Sandra J. Professional Psychology: Research and Practice. Vol 37(2) Apr 2006, 140-145.

2. Using research to support management decision making within the field of education. Hemsley-Brown, Jane. Management Decision. Vol 43(5) 2005, 691-705.

3. Economic evaluation of a nursing-led inpatient unit: the impact of findings on management decisions of service utility and sustainability. Harris, Ruth; Richardson, Gerry; Grifffiths, Peter; Hallett, Nancy; Wilson-Barnett, Jenifer. Journal of Nursing Management. Vol 13(5) Sep 2005, 419-427.

4. Getting Evidence into Practice: Perspectives on Rationality. Sanderson, Ian. Evaluation: The International Journal of Theory, Research and Practice. Vol 10(3) Jul 2004, 366-379.

5. Theory-based Evaluation and Learning: Possibilities and Challenges. Van Der Knaap, Peter. Evaluation: The International Journal of Theory, Research and Practice. Vol 10(1) Jan 2004, 16-34.

6. The Utilization of Training Program Evaluation in Corporate Universities. Bober, Christopher E; Bartlett, Kenneth R. Human Resource Development Quarterly. Vol 15(4) Win 2004, 363-383.

7. Can research influence mental health policy? Whiteford, Harvey. Australian and New Zealand Journal of Psychiatry. Vol 35(4) Aug 2001, 428-434.

121

Keywords Decision making, policy making, research



CSA Sociological Abstracts, accessed via CSA Illumina (ProQuest)

Search Strategy

KW=((research WITHIN 3 methodology) OR assess* OR framework OR approach OR (research WITHIN 3 design)) AND KW=((impact or relevance or payback or influence or benefit* OR utili?ation or diffusion or dissemination) NEAR (research or knowledge or findings or recommendations or outcomes)) AND DE=(knowledge utilization OR decision making OR policy making) Results limited to: Articles published from 1995-2009 Journal articles


Articles selected:

1. Storylines of Research in Diffusion of Innovation: A Meta-Narrative Approach to Systematic Review. Greenhalgh, Trisha; Robert, Glenn; Macfarlane, Fraser; Bate, Paul; Kyriakidou, Olympia; Peacock, Richard. Social Science & Medicine, vol. 61, no. 2, pp. 417-430, July 2005

2. "Shielding" the Knowledge Transfer Process in Human Service Research. Cano, Gretchen L; Crowley, Jocelyn Elise; Guston, David. Journal of Public Administration Research and Theory, vol. 17, no. 1, pp. 39-60, Jan 2007

3. Issues in Assessing the Policy Influence of Research. Carden, Fred. International Social Science Journal, vol. 56, no. 1, pp. 135-151, March 2004

4. Integrating Models of Diffusion of Innovations: A Conceptual Framework. Wejnert, Barbara. Annual Review of Sociology, vol. 28, pp. 297-326, 2002

5. "Knowledge Utilisation" in Dutch Social Policy: The Case of the Disability Insurance Act (WAO). De Gier, Erik; Henke, Roger; Vijgen, Jacqueline. International Social Science Journal, vol. 56, no. 1, pp. 17-35, March 2004

6. A Multimethod Analysis of Knowledge Use in Social Policy: Research Use in Decisions Affecting the Welfare of Children. Hutchinson, Janet R. Science Communication, vol. 17, no. 1, pp. 90-106, Sept 1995

7. When Theories Become Tools: Toward a Framework for Pragmatic Validity. Worren, Nicolay; Moore, Karl; Elliott, Richard. Human Relations, vol. 55, no. 10, pp. 1227-1250, Oct 2002

122

APPENDIX 2: FACT SHEETS Content:

Fact sheets on applied methodologies

d. Overall methodologies (other than Pay Back)

e. Overall methodologies based on Pay Back

f. Methodologies focused on certain aspects

Fact sheets on Theoretical frameworks

Useful notions from an article not focused on methodology

1 Fact sheets on applied methodologies 1 a. Overall methodologies (other than Pay Back)

Requester’s utilization of HTA products

FACT SHEET

Research objectives

1. Determine the degree to which the questions asked by requesters of HTA products were answered

2. Discover how each product was used and whether the use extended to other organizations outside the requester’s

organizations

3. Learn if there were unexpected outcomes that came about as a direct result of the involvement of the HTA unit with a

requester’s organization

Phases within research- and implementation-cycle the methodology focuses on:

Input (resources dedicated to or consumed by research, choices made in the process)

Activities (the actual research process)

Output (tangible and direct outputs of research activities)

Diffusion/reception (the way in which the outputs are disseminated or received)

Outcome/impact (influence on policy and/or practice, ranging from short- to long-term)

Methodology defines impact as:

Utility and utilization of HTA reports

Impact of HTA reports on policy and decision-making

Methods of data collection and sources used:

All projects were included that had been publicly released and had a clearly identifiable requester.

Interviews – 9 interviews were conducted with a total of 11 interviewees (where possible face-to-face, otherwise by telephone) with

requesters, requesters/users and users of HTA products/reports developed by the programme. Guides for interviews were developed

from the existing HTA Evaluation Form, an internal questionnaire used earlier in similar research, the HTA payback model and the HTA

Unit’s continuing review of the literature on HTA impact and research utilization. Separate guides were developed for requesters,

requesters/users and users. Requesters/users were identified by the steering committee. Topics:

Product need

Service provided

Utility of the report

Rating of the report

Utilization and function

Processes for utilization

Considerations for service improvements

Supplementary interviews with the available AHFMR HTA authors to elicit perspectives on internal processes. Topics:

Role in relation to the report

Helping the requesters define need

Relation/contact with target audience (users)

Client follow-up

Perceived barriers to utilization

Personal learning

Attention in impact assessment is focused on:

Research, researchers or research institution

123

The process of active dissemination

The (end-)users of the research outcomes / recommendations

Aggregation level of impact assessment:

Multiple studies in time-span of one year

Time considered necessary before impact can be assessed:

At most 1 year (publication was in 2003, products analyzed from 2001-2002)

Possible application level(s) of this methodology:

Program-level (overall assessment)

Case-level (in-depth analysis) Note: questionnaire added as appendix Source(s): AHFMR 2003

Method for evaluating the impact of applied health research, influenced by the payback approach

FACT SHEET

Research objectives

Identifying the impact of the research programme on policy and practice, as well as contribute to the stock of knowledge








Primary outputs: produced by the NCCSDO or the researchers themselves (e.g. research summaries, journal articles, conference

presentations).

Secondary outputs: Citation /mentioning of primary outputs in policy documents, practice guidance, newsletters, websites and the media.


Selection of a sample of 23 research projects. Data collection in two phases:

focused on primary outputs. Data sources used: NCCSDO documents, NCCSDO monitoring data, NCCSDO staff, number of

website downloads, location of externally published data using online databases and search engines, key journals, etc.

focused on secondary outputs. Data sources used: searches using surname and initial of PI and members of research team +

keywords describing the research. Verification of outputs as being related to the SDO (criteria mentioned).

Further selection of 11 of the 23 projects as case studies (criteria mentioned). Data collection:

Semi-structured telephone interviews based on the analysis framework (mentioned in appendix) with the PI’s of each study,

covering the following topics:

How were policy makers, NHS managers and practitioners (local, regional and national) involved?

Outputs produced from the research (local, regional and national)?

Reaction to the research (local, regional and national)?

How the research had been used (local, regional, national: policy, practice, research?)

Interviews with other members of the research team by telephone or asked for feedback via e-mail

Interviews with users of the research (identified through interviews with PI’s, citations, NCCSDO database and publications,

interviews with the SDO staff) by telephone or asked for feedback via e-mail

Notes were made during the telephone conversations and any further outputs identified were added to the database






Research program


The programme evaluated covered 6 years. Selection criteria for study-sample are related to degree of impact expected, including

minimal time span elapsed. However, time span is not explicitly stated.

124



Case-level (in-depth analysis) Note: Impact analysis framework given in appendix. Contains research questions in the following categories: 1) Researchers; 2) Origin of research; 3) Engagement with policy makers and practitioners; 4)

Research design and methods; 5) Dissemination; 6) Impact of research; 7) Demonstration of impact (utilization of research); 8) The NCCSDO.

Source(s): SDO, year unknown

1 B. Overall methodologies based on Pay Back Payback approach in case studies

FACT SHEET

Research objectives

4. Systematically reviewing and documenting the outputs and outcomes of research funded by the Arthritis Research Campaign

5. To illustrate the strengths and weaknesses of different modes of research funding








Knowledge production

Research targeting and capacity building

Informing policy and product development

Health and health sector benefits

Broader economic benefits


The payback framework was applied to 16 case studies of research grants funded. The selection consisted of project grants, programme

grants, fellowships, institute grants, basic grants, clinical grants and allied health professional grants (high-impact PIs and mid-impact

PIs) by using selection matrix it was aimed to produce a set of case studies that mirrored the diversity of ARC funding in key

dimensions. Case studies consisted of:

Document and literature reviews

Bibliometric analysis

Semi-structured key informant interviews

Each of the case studies was written up and published as a narrative organized according to the structure provided by the payback

framework. Common structure facilitated comparative analysis (of example to identify the factors associated with the successful

translation of research.

Two approaches to compare across cases:

Qualitative assessment of the case studies

Novel method of scoring the case studies on a scale of 1-9 on the five payback categories






Other: research campaign


Grants funded between 1990 and 1994, impact assessment conducted from 2002 onwards. Minimal time elapsed: 8 years


Case-level (in-depth analysis) Note: methodology is further elaborated in other publications Source(s): Wooding et al., 2005

125

Payback approach

FACT SHEET

Research objectives

Retrospective evaluation of the benefits from the R&D funded by a regional office of the NHS Executive


? Input (resources dedicated to or consumed by research, choices made in the process)

? Activities (the actual research process)

? Output (tangible and direct outputs of research activities)

? Diffusion/reception (the way in which the outputs are disseminated or received)



A. Knowledge

B. Benefits to future research and research use

Better targeting of future research

Development of research skills, personnel and overall research capacity

Critical capability to utilise appropriately existing research including that from overseas

Staff development/educational benefits

C. Political and administrative benefits

Improved information bases on which to take political and executive decisions

Other political benefits from undertaking research

D. Health sector benefits

Cost reduction in the delivery of existing services

Qualitative improvements in the process of service delivery

Increased effectiveness of services

Equity, for example improved allocation of resources at an area level, better targeting and accessibility

Revenues gained from intellectual property rights

Organizational development

E. Broader economic benefits

Wider economic benefits from commercial exploitation of innovations arising from R&D

Economic benefits from a healthy workforce and reduction in working days lost


Questionnaire sent to the lead applicants for all projects (164) funded in the 1990s and which were completed by 1997. Design of the

questionnaire was informed by the HERG payback categorization/model.

A sample of the projects was selected for further assessment as case studies.



? The process of active dissemination

? The (end-)users of the research outcomes / recommendations


Geographic region


< 12 months



Case-level (in-depth analysis) Note: No further elaboration is given on the used methodology, although the article states that a full account is available in HERG research report 27 (not available online). Source(s): Buxton et al., 2000

126

‘Small-scale’ payback approach

FACT SHEET

Research objectives

1. Evaluating the (time-limited) Implementation Methods Programme, specifically including the quality of the outputs and the

commissioning process.

2. Drawing conclusions about what lessons could be learnt for future commissioning and communication strategies of the SDO

programme.








Impact of the IMP on policy and practice in the health service


Multi-method approach using both quantitative and qualitative methods:

Documentary analysis – published documents from the IMP, review of IMP’s files. Information recorded about the scores given

to proposals, potential users of the project identified in application forms, scores given to end of project reports

Semi-structured interviews (based on the documentary analysis and some early interviews) – 25 interviews (mostly face-to-

face, some by phone): Chairs of the Advisory and Commissioning Groups, Deputy Chair of the Commissioning Group, other

members of both groups. Prime focus on commissioning process. Interview schedule used as coding frame for analysis.

Questionnaires to lead researchers of all 36 projects funded - Questionnaire structure based on HERG-payback framework

(knowledge production, training and further research, possible impact on health policy and practice, features of dissemination

that might have contributed to their impact, role of the programme as a whole). Two rappels and covering letter used to

heighten the response.

Questionnaires to potential users – Possible users were identified via existing network and message board. Electronic

questionnaire with specific projects as subject. Series of questions asking about whether the recipient had heard of the

projects, read the articles described in the attached abstracts, already been influenced by the findings, etc. Also postal version

was sent to practitioners and researchers in maternity care field to heighten the response.

Desk analysis – Searches in various databases to make assessments of the quality of the research outputs. Citation analysis

for journal articles in several databases. Journal impact score was recorded.






Research program


Programme ran approx. 6 years. No minimal time-span noted.


Program-level (overall assessment) Note: questionnaires used are added as appendices to the text Source(s): Hanney, Soper, Buxton 2003

127

Modified ‘payback’ approach (Hong Kong)

FACT SHEET

Research objectives

1. Quantify the outcomes of completed research projects supported by health research fund using the ‘payback’ framework, in

comparison with the more established health research funds abroad.

2. To explore factors associated with the impact of research outcomes on health policy and provider behaviour.








- Knowledge production

- Career advancement, higher qualifications

- Research capacity building (subsequent research)

- Informing policy

- Changes in behaviour or clinical practice in health service managers, providers and the general public


Revised payback evaluation questionnaire (developed by HERG), with additional items drawn form literature review and consultation with

local experts. Questionnaire was sent to principal investigators of all 205 completed projects. Acquired data included:

- total number of publications, number of citations per publication, impact factor and rank within subject category

- number of academic qualifications or career promotions for research team members

- policies that were influenced

- behaviours that were changed or influenced

- estimated contribution or expected impact of the research project + evidence to support impact assessment

- evidence of participation in health-related policy/advisory committees

- liaisons with potential users of the research findings prior to or during the project

Document analysis:

type of project administering institution, department of work place affiliation of the PI’s

funding award and project duration

Statistical analysis (regression) of information gathered to identify relations between topics.






Research institute


Not stated


Program-level (overall assessment) Source(s): Kwan et al, 2007

128

Modified payback approach (The Netherlands)

FACT SHEET

Research objectives

6. Assessing whether the results of the main HTA program in the Netherlands have led to improvements in health and health care

7. Analyze how best to assess the impact from health research








The logic model constructed provides a way to organize the assessment of this multidimensional categorization of benefits from research.

Conceptual categorization of the benefits arising from health services research:

Knowledge production

Research benefits

Informing policy

Changing health practice

Broader impact on health


Dossier review of all 43 projects granted between 2001 and 2003. Reading of key documents including original research grant proposals,

referees reports, final reports, information about the program (annual reports, minutes of committee meetings, etc.)

Postal survey based on HERG questionnaire (Hanney et al 2007). Postal survey, including data from the dossier review, was sent to all

PIs to collect the most recent data on their projects. Results were presented as standard one page summaries using a common structure.

Case study analysis in which a purposive sample of five HTA projects (that mirrored the diversity of ZonMw funding) were investigated in

more detail to illustrate useful levels of payback. Three semi-structured interviews per case study were conducted (either face-to-face or

by telephone) with the PI, et least one other researcher and users.

In each case study initial and longer-term output and outcome of the research were explored in more detail, including the translation of

the research, e.g. into practice. The interviews aimed to verify conclusions from written materials and to identify any payback that might

have been missed.

Scoring of payback by HTA experts in three rounds from 26 HTA projects that had already led to a final report and for which survey

results were available. Scoring on a scale of 1-10 (0,2,4, etc.) on the five impact categories.






Multiple studies

Research programme


Program started in 2000, projects involved awarded between 2001 and 2003. Year of assessment unknown.



Case-level (in-depth analysis) Source(s): Oortwijn et al., 2008

129

Payback approach as applied in the Hanney-study

FACT SHEET

Research objectives

1. To assess the impact of the first 10 years of the NHS HTA programme

2. Identifying the factors associated with health technology assessment research making an impact.





/Diffusion/reception (the way in which the outputs are disseminated or received)



The ‘payback’ from research, consisting of five categories:

1. Knowledge (e.g. published outputs from research or other methods of dissemination)

2. Benefits to future research and research use (e.g. better targeting of future research, staff development)

3. Political and administrative benefits (e.g. improved information bases on which to take decisions)

4. Health sector benefits (e.g. cost reduction, increased effectiveness of services)

5. Broader economic benefits (e.g. commercial exploitation of innovations)


‘Mixed methods’ approach, consisting of three key elements:

1. Document analysis of records contained at the research institute (description of the programme)

2. Questionnaire sent to all relevant principal investigators

3. A series of 16 more detailed case studies involving desk analysis and at least one semi-structured interview with the principal

investigator or an alternative leading researcher from each project. The case-studies were selected at random.






Research programme


All projects evaluated were completed before the end of June 2003. The impact assessment was conducted in the summer of 2005, so at

least two years were considered necessary for any impact to be measurable.



Case-level (in-depth analysis) Notes: Attention was paid to permitting some comparison between the 16 case studies and the survey (comparing results from a scoring system). Questionnaire used is attached to the article. Although output is measured in different categories and the total process from needs assessment to impact is being described, not much attention is given to factors that cause the impact measured. Source(s): Hanney et al., 2007

130

“Payback” approach

FACT SHEET

Research objectives

Assessing the impact of the English NHS HTA program using the “payback” framework








A multi-dimensional categorization of benefits is used:

Contribution to knowledge

Influence on policy

Influence on practice

Health gain


Survey of 204 principal investigators who had completed projects funded by the HTA program in its first 10 years (and were suitable to

include in the evaluation). The survey was organized around the payback framework which required data on: publications, presentations,

further linked research, impact on policy and behavior. Questionnaire developed by Hanney was slightly adapted and used.

Detailed case studies of 16 research projects: nine primary studies, four secondary, and three NICE TAR’s (stratified random selection).

Case studies consisted of interviews with principal investigators, analysis of documents referred to in the interviews, analysis of key

citations to the main papers, and review studies of the impact of NICE.






Research program


At least 2 years and at most 12 years had passed since submission of the final draft reports by funded researchers



Case-level (in-depth analysis) Note: no detailed information is given about the contents of surveys and case study interviews. Impact on policy or behaviour is measured using self-report from researchers (‘claiming impact’) Source(s): Raftery et al., 2009

131

1 c Methodologies focused on certain aspects

The user’s perspective on research utilization

FACT SHEET

Research objectives

1. Examining how health-care providers, administrator and patient associations across Canada use HTA reports

2. Examining the limitations they encounter when accessing and using scientific knowledge








The use of HTA in decision-making which can be instrumental, conceptual or symbolic and depends on the type of user.


Semi-directed face-to-face or telephonic interviews with formal organizations using HTA’s, representing administrators, health-care

providers and patients. Organizations were selected using inclusion and exclusion criteria.

The interviews (recorded en transcribed) were conducted with individuals occupying the highest administrative level and whose

responsibilities were directly relevant to HTA.






Research institute

Geographical region


Not stated



Case-level (in-depth analysis) Source(s): Hivon et al., 2005

132

Assessment of the impact of recommendations on hospital practice

FACT SHEET

Research objectives

Examining the extent to which CEDIT’s recommendations match user’s expectations and to assess their influence on decision-making








Impact of recommendations on the introduction of technological innovations


Two-method approach:

Semi-directive interviews conducted among the various players affected by recommendations on the dissemination of a technological

innovation. Fourteen interviewees chosen to include both those who were and those who were not involved with CEDIT’s work. Topics

dealt with during interviews were mainly the system for disseminating the recommendations, knowledge of how CEDIT operates, interest

in the recommendations, grounds for requesting an assessment, and the impact of the recommendations.

Case studies focusing on dissemination of the technologies in the wake of those recommendations. Selection: recommendations

published between 1995 and 1998 so as to have a long enough time lag between publication and implementation. Data collection: survey

in the hospitals focusing on equipment inventory, purchase division contracts, opinions of healthcare providers).






Research institute

Other: recommendations


At least 4 years



Case-level (in-depth analysis) Source(s): Bodeau-Livinec et al., 2006

133

Assessing the policy influence of research

FACT SHEET

Research objectives

1. Assessing the policy influence of International Development Research Centre supported research projects

2. Analyzing the way in which the research influences policy

3. Identifying factors that affect the extent of policy influence resulting from its projects








Influencing the policy process: The level at which and the way in which a project or programme contributes to attaining impact and the

characteristics of those contributions.

The (sole) attribution of research findings to the impact measured is impossible: society is complex and a range of factors and conditions

influence change. Furthermore, attaining final impacts are the responsibility of the local partners/end-users. Another problem is that of

timing: impact assessment is like a snap shot: after the shot is taken, the impact may continue to develop and alter, in a good way or in a

bad way.


Review of ‘project completion reports’: providing insights into the results and management of the projects, and contain information related

to policy influences as identified by programme officers.

Review of the programming documents, including programme objectives and a survey of project objectives. This provides a clear picture

of the priority given to policy influence and identifies the nature of policy-related objectives in each case

Review of the 80 project evaluations submitted from July 1999 to March 2001 that were conducted throughout the Centre, to identify

those that had a mandate to address policy influence.

Document review and interviews with senior staff members in order to explore and document the evolution in thinking about policy

influence and acknowledging the relevance of policy influence

A purposive selection of projects as case studies. Projects were selected using diverse criteria, important was a claim of influence

articulated by staff members. Case studies consisted of regional workshops with evaluators, staff of the IDRC and selected researchers

form the region. In a addition to the reflections of staff a cross-case analysis was done to identify key issues relating to policy influence.






Research institute


Not stated



Case-level (in-depth analysis) Source(s): Carden, 2004

134

Multidimensional framework for assessing uptake of recommendations in organizational and clinical practice

FACT SHEET

Research objectives

Understanding factors affecting the uptake of HTA recommendations to support decision-making with respect to the introduction of three

health technologies


Input (resources dedicated to o

r consumed by research, choices made in the process)



Diffusion/dissemination (the way in which the outputs are disseminated)



Utilization (=implementation) of recommendations into practice. Degree and way of utilization are influenced at end-users’:

- institutional level

- organizational level

- professional level


3 case studies were conducted (consisting of administrative health technologies and clinical health technologies) using a

multidimensional framework, based upon a combination of theoretical models.

Methods used were:

Review of the recommendations

In-depth semi-structured interviews with 15 hospitals selected according to criteria; a total of 28 medical specialists and other

key-informants were interviewed. 19 questions were asked:

Hospital: size, age, location, competition, inter-organizational relationships

Internal structure: type of management, level of care, specialization, differentiation, size of units, planning and control systems,

internal communications, decentralization of power

Open-ended questions assessing the respondent’s knowledge of the recommendations and experience regarding the

implementation and use of the technology.

Additional consultation of researchers involved in the studies to provide information concerning the process of the study and

dissemination activities






Research institute

Other: recommendations


At least 1 year


Case-level (in-depth analysis) Source(s): Gagnon et al., 2006

135

Assessing knowledge application in political ,organizational and professional decision-making

FACT SHEET

Research objectives

1. To explore the (non-)utilization by decision-makers of various sources of knowledge (including scientific evidence) to support

implementation of technologies

2. To analyze factors determining the application of scientific evidence

3. To analyze interactions between the three decisional levels that influence the application of scientific evidence in decision-

making








Application of knowledge (scientific evidence) in making decisions about integrating technologies into the health care system


Multiple case study focused on two specific technologies in e-health, and consisting of the following methods/strategies:

Political level:

critical incidents analysis to identify key decisions that have influenced implementation in provincial level document analysis

and/or contact networks (validated by key-stakeholders)

semi-structured interviews with approx. 40 key informants (policy-makers, technology project managers, evaluators and

representatives of lobby-organizations) involved in decisions to identify factors that influenced knowledge utilization

Analysis of characteristics and position of stakeholders (using Monnier’s classification)

Analysis of the creation of coalitions and alliances (using frameworks of Gamson and Lemieux

Organizational level:

Selection of two cases (technology projects)

Semi-structured interviews (10 per project) with key decision makers (regional and local, project managers and health care

professionals)

Document analysis

Clinical level:

Questionnaire amongst approx. 1400 clinicians (aimed at response rate of 40%) to assess psychological determinants of

health care professionals’ intention to use technology in their practice (using theory of Interpersonal Behaviours (TIB)) and

assessing the role of scientific evidence to support professionals’ decision-making.

Synthesis: Triangulation and integration of the results.






Other: the implementation of new technologies (rather than studies)


Not stated


Case-level (in-depth analysis) Note: interesting approach (tracing backwards from technology implementation to the role of scientific evidence/research findings in this process. Might take too much time to develop a good questionnaire (article describes elaborate designing and validation procedures). Source(s): Gagnon et al., 2008

136

Critical incidents analysis to assess the impact of HTA’s on policy decisions and practice

FACT SHEET

Research objectives

To determine whether the selected HTA’s had an impact on policy decisions and practice.

No attention is paid to the possible factors influencing the measured impact








Influencing the diffusion and use of health technology in such a way as to increase the efficiency of the health care system (by increasing

its effectiveness or reducing its costs). If an HTA fails to influence the working of the health care system, it is without impact.


Three steps in determining whether an HTA has had an impact:

1. What impact was intended? What was the hoped-for-effect or ‘message’?

2. To whom was the message in the report directed? Who is the principal decider and is it reached?

3. What is the extent to which the HTA has in fact influenced the decisions of the identified health care deciders, in terms of a)

changes in stated policy; b) actual distribution and utilization of the technology in question.

A case study approach was used to evaluate the impact of HTA’s on policy decisions, in order to reconstruct (for each subject addressed)

the dynamic of the interactions that took place between the HTA in question and the positions taken by deciders. Methods used:

Analysis of available documentation (administrative files, public announcements, etc.) looking for ‘critical incidents’: explicit

referral to specific HTA used for supporting actions consistent with its content; classification into levels of importance and

awarding weight to this.

Analysis of changes in actual practice, e.g. in hospitals, allocation of resources, planning guidelines, etc. by using utilization

data and questionnaires (not further elaborated in the article).

Article states that confirmation of alleged impact by key actors is important (triangulation).






Multiple studies


Not stated


Case-level (in-depth analysis) Source(s): Jacob & McGregor, 1997

137

Determinants of knowledge utilization in decision-making and practice

FACT SHEET

Research objectives

Addressing three questions, based on a survey of 1229 Canadian social science scolars:

1. What is the extent of the use of social science research in Canada?

2. Are there differences between the social science disciplines regarding extent of use?

3. What are the determinants of utilization of social sciences research knowledge in Canada?








The utilization of research knowledge by knowledge-users in decision-making and practice. Six stages of knowledge utilization are being

defined:

1. Transmission of research results to the practitioners and professionals concerned

2. Cognition (reading and understanding) of the research results by the practitioners and professionals concerned

3. Reference to the research results in reports, studies and strategies of action elaborated by practitioners and professionals

concerned

4. Efforts made by practitioners and professionals to adopt the research results

5. Influencing the choice and decision of practitioners and professionals concerned

6. Applications and extension to which the research results have given rise (by practitioners and professionals)


Mail survey aimed at 3252 faculty members in different departments of 55 Canadian universities using a questionnaire of 17 questions

(response rate approx. 40%). Respondents were identified using web sites. Content of the questionnaire:

Dependent variable (knowledge utilization): Respondents were asked to indicate on a 0-5 scale how accurately each stage

(from 1 to 6) described the utilization of their research for the last 5 years.

Independent variables: respondents are asked to comment on the following independent variables:

- Methods used in quantitative studies

- Methods used in qualitative studies

- Publication assets

- Focus on advancement of knowledge

- Focus on users’ needs

- Importance of university internal funding for projects

- Importance of external funding for projects

- Perceptions of users’ context

- Adaptation of products to users

- Dissemination efforts of researchers

- Intensity of linkages with users

Results were statistically analyzed, using regression models and other techniques.






Geographical region (Canada)


Not stated


Program-level (overall assessment) Source(s): Landry et al 1999

138

Multi-method approach for assessing policy and practice impacts from social science research

FACT SHEET

Research objectives

1. To identify ways in which results from funded research projects and fellowships have been utilized and applied by policy-

makers and practitioners

2. To identify how the research had influences policy formation and development

3. To identify how the funded research had influenced changes in professional practice within the public and private sectors

4. To identify examples of impact achieved








Instrumental use or impact – the direct and one-on-one impact of research on policy and practice decisions

Conceptual use or impact – impact on the knowledge, understanding and attitudes of policy-makers and practitioners, often in

complex and indirect ways (e.g. alter ways of thinking, alerting issues, raising consciousness)


Content analysis of all issued funded research reports along with rapporteurs’ evaluations, focused on indications of connectivity or

orientation toward users and on any indications of real or potential impacts. Particular attention was given to research objectives

(oriented towards end-users?) and achievements (implementation of connection with end-users?). Content analysis leads to ranking on a

qualitative 5-point scale.

Survey of 109 award-holders using a 4-page questionnaire (mix of Lickert-scale, pre-coded and free text response), about non-academic

research impacts arising from their award, and activities undertaken possibly leading to impact

Survey of heads of departments in order to identify their contacts with user communities and types of non-academic impact arising from

their research placing results from the other survey in a broader context

Focus groups (2) consisting of 10 heads of (psychology) departments to provide further insights into the contexts within which

connections with users might be made and how impacts might be generated. Also for identifying key users.

Media-related searches in which press releases were reviewed for stories that highlighted work by award-holders (not the specific

research project)

Case studies of four particular award-funded projects in four different research areas, specifically chosen to identify likely success-

stories, interviewing the principal researcher and one or two users. In 3 case studies also bibliometric methods were used to assess

dissemination in grey literature and applied articles (linked to the name of the researcher)

Findings were synthesized into three main categories: 1) level of engagement with users and orientation toward knowledge transfer; 2)

impacts; 3) processes, activities and roles leading to impacts.






Multiple studies (grant-funded, with no theme or subject specified by the funding agency)


All grants had a duration of three years (end in 1998, 2001 and 2004). Impact assessment conducted in 2006(?), so projects were at

least two years old.



Case-level (in-depth analysis) Note: gives the framework of core questions used in the impact assessment Source(s): Meagher et al., 2008

139

Conceptual framework for assessing the non-academic impact of grant-funded research

FACT SHEET

Research objectives

Assessing the impact on non-academic audiences of research program funded by research funding organization








Impact is defined as the utilization (in various forms) of the research outputs by non-academic actors.

Major elements underpinning impact processes are:

The types of outputs that can be expected from research

the channels through which their diffusion to non-academic actors take place

the forms through which these actors can make use of the research outputs

Permutations between outputs, diffusion channels and forms of impact result in different ways (impact mechanisms) in which economic

and social research can have an impact beyond the academic world.


Identifying networks of researchers and relevant non-academic beneficiaries (using a snowball approach) in order to determine

the types and forms of impact these actors are able to identify (mapping the ‘flow’ of research results across interested non-

academic communities). Data was gathered using interviews addressing the origination of these networks (before or as result

of research program) and the impact the program has had.

Tracing the post-research activity of researchers in order to address specific diffusion channels provided through their

subsequent engagement in non-academic work.

Questionnaire was built up using all items in the conceptual framework as a checklist .






Research program (the AIDS programme)


The program evaluated consisted of 15 projects, with 14 final reports. All projects with final reports were used, meaning sometimes as

much as 10 years had elapsed before the evaluation.



Case-level (in-depth analysis)

The whole research programme was assessed, but only on project level. Source(s): Molas-Gallart et al., 2000

140

Assessing policy contribution of research programme results

FACT SHEET

Research objectives

8. Assessing the potential use of research knowledge resulting from the Active Living Research programme

9. Commenting on the trajectory of influence of past and future ALR research findings

10. Commenting on the possible users of completed and disseminated ALR products








The research funding foundation chose knowledge use as a legitimate intermediate objective for ALR after 5 years of operation. Hence,

the evaluation looked at ways that ALR research could enlighten the policymakers and practitioners who link research with policy

discussions.

Stages of enlightenment, knowledge use and policy contribution are discerned.

Indicators of enlightenment: 1) awareness of ALR products; 2) acknowledgement of the relevance of these products to their

responsibilities.

Findings of policy contribution are organized around a conceptual framework that defines contributions broadly and at multiple stages in

the policy process:

Pre-policy: problem described, causes and consequences studied, trend or spread traced.

Developing policy: Develop awareness, mobilize, reframe the issue

Policy enactment: advocacy, frame policy, pass policy/legislation.

Post-policy enactment: implementation, access & equity, sustain, change or abandon


Telephone interviews with key informants (15-60 min) with both open-ended and closed-ended questions, using a standardized protocol

reflecting the evaluation questions and major constructs of the logic model. Variations for each type of key-informant.

Core questions inquired about awareness of ALR, examples of indicators of ALS’s policy-relevant contributions, larger contextual

influences on ALR methods and impacts.

Professionals and policy shapers: questions about the general influence of research on practice and policy and about ALR specifically.

Researchers: questions about field-building facilitators, barriers, and trajectories.

Funders: questions about field building and about the sustainability of the field.






Research program


0-5 years


Program-level (overall assessment) Note: impact is defined as knowledge use / policy contribution because this is the intermediate objective which can be expected to be reached after 5 years of operation. This corresponds with the knowledge utilization theory which states that knowledge use is a sine qua non for eventual impact to take place. Source(s): Ottoson et al., 2009

141

Self-report of ‘salient findings’

FACT SHEET

Research objectives

1. Assessing the outcomes of the Agency for Healthcare Research and Quality’s (AHRQ) first decade of focus on effectiveness

and outcomes research (OER)

2. Identifying needs and opportunities for the study of OER in the coming years








The article presents a ‘levels of impact’ model:

Level 1: Research findings

Level 2: Impact on policies

Level 3: Impact on clinical practice

Level 4: Impact on health care outcomes


Survey of all 95 research grant PIs for Patient Outcomes Research Teams and Medical Treatment Effectiveness Program studies funded

by AHRQ between 1989 and 1997. PIs were asked to report ‘the most salient findings’ from each study and to provide sufficient material

in ‘succinct, bullet-point format’ (max. 3 slides). No further instructions or criteria were given.

A taxonomy for classifying the findings was developed to group results by purpose or methodology. Eleven non-mutually exclusive,

descriptive categories were identified. All ‘salient findings’ were assigned to one or more categories.






Research program


Not stated


Program-level (overall assessment) Note: the model presented is not the basis for the methodology used Source(s): Stryer et al., 2000

142

Agencies’ support and promotion of knowledge translation

FACT SHEET

Research objectives

Asking thirty-three agencies from different countries about their role in promoting the results of the research they fund








Effective translation of (research) knowledge to the public/end-users (the stage before impact)


First a ‘judgment sample’ of health research funding agencies (purposefully selected cases based on particular criteria). Then qualitative

study using:

Semi-structured interviews with a sample of key informants (identified by the authors) from the selected applied health research funding

agencies. The interview framework was constructed to explore issues regarding knowledge translation at several levels of organization:

the country level, the funding agency level and the researcher level. Interviews were conducted face-to-face or by telephone.

Website searches of each of the funding agencies for information relation to any of the questions in the interview framework. Information

was pasted into the framework before the interviews.

Downloading of documents, publications and grant application forms when available, to supplement the data gathered from the

respondents.

(Analysis of the data described in detail)






Research institute

Geographical region


Not stated


Program-level (overall assessment) Source(s): Tetroe et al., 2008

143

2. Fact sheets on Theoretical frameworks

A. Lavis et al., 2003 - Impact measures related to types of target audiences

145

B. Hailey 2003 – Elements of effectiveness

147

C. Bensing et al, 2003 – Policy relevance

Health care policy might benefit from health services research in developing evidence-based policy. However, research will always be only one of the many inputs that determine policy. Weiss (17) uses the metaphor of an arena, where researchers have to deal with the four i’s: interests (of civil servants, politicians, or other stakeholders), ideologies (political agendas, prevalent norms and values), information (research is but one source of information that has to compete with many other sources), and institutional forum (the idiosyncrasy of governmental organizations). Moreover, research is not always used in a unidimensional way.

Utilization can be:

instrumental (i.e., the direct application of results)

conceptual (i.e., generating newpolicy ideas)

strategic (i.e., having a role in the political process)

legitimization or to warn of pending problems.

“enlightenment function”: challenging old ideas, providing new perspectives, and helping to re-order the policy agenda.

Ex Ante Indicators

148

Ex Post Indicators

Different methods for evaluating five key performance areas of policy research programmes

FACT SHEET

Research objectives

The paper aims to advance understanding on how to monitor and evaluate policy research (both policy decisions and processes,

including implementation).








The paper identifies the following broad working definitions of outputs, uptake and outcomes/impacts:

Outputs – the tangible goods and services that a research programme produces

Uptake – direct responses to the research programme (media coverage, mentioning in policy papers, etc.)

Outcomes and impacts – changes in behaviour, knowledge, policies, capacities and/or practices that the research has

contributed to, directly or indirectly


The article provides a range of possible monitoring and evaluation approaches taken from the current experience of policy research

projects.

Evaluating strategy and direction: Logframes, social network analysis, impact pathways (only on project level), modular

matrices

Evaluating management: Fit For Purpose reviews, Lighter Touch quality audits, horizontal evaluation, appreciative inquiry

Evaluating outputs: evaluating academic articles and research reports, evaluating policy and briefing papers, evaluating

websites, evaluating networks (only on project level), after action reviews (only on project level).

Evaluating uptake: impact logs, new areas for citation analysis, user surveys

Evaluating outcomes and impacts: outcome mapping, RAPID outcome assessment (only on project level), most significant

change, innovation histories, episode studies.






149

Single study

Multiple studies

Research program

Research institute


Not stated.



Case-level (in-depth analysis) Source(s): Hovland, 2007

Assessing the societal impact of applied health research

FACT SHEET

Research objectives

Presenting a general outline for an evaluation methodology of societal impact of applied health research, including potentially suitable

indicators, as a basis for further specification and development.








Societal impact of health care research, which can be divided into:

1 relevance for health care providers and the process of health care delivery

2 relevance for policy makers and the process of designing, implementing and monitoring policy decisions


In any impact assessment, it is important to use indicators for:

3 the relevance and urgency of the research problem addressed (like burden of disease, potential benefits

involved)

4 the likelihood that the research outcomes will have impact on decision-making and priority setting in health care.

Elements which are relevant when assessing societal impact are:

a) the research team’s/the organization’s mission;

b) its performance in relation to that mission

Possible criteria and indicators of societal impact of research output are:

150

The criteria and indicators used for an evaluation will always be dependent on the mission of a specific research group or institution.






Research institute


Not stated



Case-level (in-depth analysis) Source(s): KNAW 2002 Note: the different methods are elaborated with examples, frameworks, outlines and so on.

Research Impact Framework

FACT SHEET

Research objectives

Developing a conceptual framework to help health researchers think through and describe the possible outcomes of their work. Definition

of framework: a way of setting out a range of possible variables related to the issue of interest, but does not necessarily identify the

relationships between them or provide an evaluative judgement of the variables.







151


The article describes all 4 areas and sub-areas in detail by identifying descriptive categories to support and help standardise the

development of research impact narratives. These could be used in devising research instruments.


No methods for data collection proposed


Not stated


Not stated


Not stated



Case-level (in-depth analysis)

Source(s): Kuruvilla et al., 2006

Conceptual model linking organizational functions to performance

FACT SHEET

Research objectives

11. Identifying specific dimensions for every organization function and show their relation to performance

12. Presenting the model as a flexible and theoretically grounded tool to assess the performance of HTAO’s.








Impact is defined as goal attainment or effectiveness.

Appraisal of the goal attainment function should capture the ability of the HTAO to produce and disseminate information

(outputs) that in turn affects the way decisions are made (impacts): decisionmaking impact, organization and service impact,

societal impact, research impact, efficiency.

Other organizational functions:

Production function – the technical core of the organization, assessed according to four interlinked dimensions of performance:

quantity, productivity, coordination mechanisms, quality, accessibility, comprehensiveness, technical quality.

Adaptation function – deals with the relationship between the HTAO and its environment: capacity to acquire and maintain

resources, ability to gain external support, responsiveness to needs, innovation and learning.

152

Culture and values maintenance function – captures the shared principles and beliefs that produce meaning and cohesion

among members of an organization: independence, transparency, accountability.

Performance alignments:

Strategic alignment (interchange between the adaptation and goal attainment functions)

Allocation alignment (interchange between production and adaptation mechanisms)

Contextual alignment (interchange between culture and values and adaptation)

Tactical alignment (interchange between production and goal attainment)

Operational alignment (interchange between culture and values and production)

Legitimization alignment (interchange between goal attainment and culture and values)


No data collection methods given, only the conceptual framework






Research institute


Not stated.


Program-level (overall assessment) Note: framework focuses on organizational aspects of HTAO’s in relation to their goal attainment (= impact). Might provide useful insights for mapping KCE. Source(s): Lafortune et al., 2008

Conceptual framework for outcome-oriented measurement of impact

FACT SHEET

Research objectives

13. Advocating the use of outcome-based approach to report the success of medical research

14. Presenting the concept of outcomes-assessment and an according framework








The outcomes of research as opposed to outputs (direct products or publications):

5 Initial outcome: key clinical decision makers become aware of the research evidence

6 Intermediate outcome: Change in clinical practice is implemented

7 Long-term outcome: Implementation of change leads to patient benefit


Initial outcome: Impact factor alone does not suffice. Other suggestions are:

8 a broader approach to scientometrics in which nonjournal media are also counted

9 tracking readership via web tracking data

10 surveys to assess the actual penetrance of information in target-audiences

Intermediate outcome: Examining the relation between knowing and doing. 10 factors associated with implementation of findings:

11 The channel of communication (influence of sources/information)

12 The coherence of findings with prevailing clinical beliefs (do the findings require ‘out of the box’ thinking from

professionals?)

13 The quality and the quantity of the evidence

14 The nature of the change (is the change easily implementable well fitting, easy to learn/do, inexpensive)

153

15 The impact of the change on key clinical branch points (do the findings meet clinical needs of the

professionals?)

Long-term outcome: measuring the patient benefit. Usage of general measures of functioning or well-being is advocated:

16 DALY

17 QUALY

18 SF-36


/ Research, researchers or research institution

/ The process of active dissemination

/ The (end-)users of the research outcomes / recommendations


Single study

Multiple studies

Research program


Not stated



Case-level (in-depth analysis) Source(s): Weiss, 2007

3. Useful notions from article not focused on methodology

E. Drummond et al., 2008 – Improved conduct of HTA’s

The article proposes e set of fifteen principles that can be used in assessing existing or establishing new HTA activities. The principles are organized in four sections.

Structure of HTA programs

Formulating an explicit and relevant goal en scope of the HTA

Conducting HTA’s unbiased and transparently

Focusing HTA’s on all relevant technologies (not only on drugs but also other medical goods and services, and both new and existing technologies)

Using a clear system for setting priorities for HTA (in situations were only some technologies are assessed, selection priorities need to be set, for instance on the following criteria (NICE):

Burden of disease

Resource impact

Clinical and policy importance

Presence of inappropriate variation in practice

Potential factors affecting the timeliness of guidance

Likelihood of the guidance having an impact

8. Incorporating appropriate methods for assessing costs and benefits (rigorous, analytical methods and clarity engender public trust in the process and its findings)

9. Considering a wide range of evidence and outcomes (‘realistic evaluation’: clinical benefits, risks, and costs must be defined broadly to include all relevant outcomes of interest)

10. Considering a full societal perspective when undertaking HTA’s (adopting a broad perspective makes the outcomes relevant to multiple decision-makers and provides an overview of costs and benefits in terms of the broader economy and society)

154

11. Explicitly characterizing uncertainty surrounding estimates (determining robustness of HTA findings and acknowledging the limitations of the analysis)

12. Considering and addressing issues of generalizability and transferability (differences in patients, genetics, disease stage, economic factors, populations, countries, etc. influence generalizability and transferability. Also varying decision perspectives influence this).

Processes for conducting HTA

1. Actively engaging all key stakeholder groups in HTA programs (improves quality, acceptance and the chance of implementation)

2. Actively seeking all available data (allowing confidential data to be used)

3. Monitoring the implementation of the HTA findings (ensuring value of the HTA and ensuring that findings are implemented in a fair and even-handed manner)

Use of HTA in decision-making

1. Timeliness (conducting research when needed/relevant and revising assessment when new data becomes available)

2. Appropriately communicating HTA findings to different decision-makers (multiple communication strategies to meet the disparate needs of different users)

3. Making clear the relationship between HTA findings and the decision-making processes (same findings may lead to different decisions (according to situation or actor), to some extent decisions can be predicted)

F. McBride et al., 2008 – Effective dissemination strategies

1. Engaging end users when framing research agendas to improve research relevance to users

Establishing a ‘research to policy network’ to link between users’ information needs and researchers’ knowledge base and practice

Developing additional ‘synthesis’ products that summarize in an accessible readable format what is known on a topic to identify needed additional research

2. Monitoring policy developments so that previously released research findings are communicated when they are relevant to current policy debates

3. Tailoring the design of the products to meet the needs of the diversity of end users interested in health research

4. Make research products easily findable and obtainable for end users (e.g. e-mail announcements with web links, easy access to relevant research findings)

5. Expanding contact and working relationships with end users (knowledge utilization depends in trust and credibility, which is enhanced by interpersonal contact)

6. Communicating research findings (also) to intermediaries who can connect the researcher’s work to policy activity

7. Earning the trust of policy makers by presenting accurate, evidence-based information, acknowledging data limitations, providing an objective viewpoint and responding timely to requests from users.

8. Investing in developing greater capacity for effective dissemination (it requires dedicated resources to support specialized knowledge and skills in addition to the resources needed for the research activity)

G. Wejnert 2002 – Diffusion of innovations

Conceptual framework is derived by grouping diffusion variables into three major components:

155

Characteristics of innovations

Public benefits versus private consequences (the impact of an innovation’s adoption on entities other than the actor versus that on the actor itself)

Benefits versus costs (cost variables relate to monetary and nonmonetary direct and indirect costs, or risks associated with the adoption of an innovation)

Characteristics of innovators

Societal entity of innovators (adoption processes are different for an individual actor compared to small or large collective actors)

Familiarity with the innovation (the rate of adoption increases as its novelty decreases)

Status characteristics (the prominence of an actor’s social entity and the homogeneity of an actor’s networks)

Socioeconomic characteristics (economic and socio-demographic characteristic of the actor influence the rate of diffusion)

Position in social networks (timing of adoption typically depends on the interaction of social units in a process of communication)

Personal characteristics (the amount of “psychological strength” of individual actors stimulates or impedes the adoption of innovations)

Environmental context

Geographical settings (influences the applicability of innovations)

Societal culture (e.g. belief systems, socialization of individual actors, cultural homo- or heterogeneity influence the adoption of innovations)

Political conditions (political situation, the degree of stability and the legal systems in place might influence the diffusion and adoption)

Global uniformity (institutionalization (rule-like behavioural models), global technology and modern communication systems)

156

APPENDIX 3: SPECIFICATION THEMES

In this document the 16 themes of the document ‘Research Design and Rationale’ are specified. For every theme and topic we show what information has been collected and the data sources and data collection techniques has been used.

Topic 1: Origin of the research

1.1. Proposals submitted

1.1.1 Typology of stakeholders

1.1.2 Typology of subjects

1.2 Proposal selection

1.2.1 Transparent

1.2.2 Careful

1.2.3 Policy or practice oriented

Specific

1.1.1 Typology of stakeholders

Numbers of proposal submitted per stakeholder group.

Stakeholders will be devided in the next groups.

a. Minister van Volksgezondheid en Sociale Zaken

b. RIZIV

c. FOD Volksgezondheid

d. FOD Sociale Zekerheid

e. Commissie Volksgezondheid van de Kamer

f. Universteit of wetenschappelijke instelling

g. Instituten of ziekenhuizen

h. Privéorganisaties of Privépersonen (subdivision will be made see data source)

i. Internationale organisaties

Categorisation comes from year report KCE 2008. A subdivision will be made for group ‘h’

1.1.2 typology of subjects

a. GCP

b. HTA

c. HSR

d. Other (Like Equity and Patient Behaviour (EPB) and Objective Elements –

Communication (OEC).

1.2.1 Transparent (Yes/No)

a. To what extent submitting procedures are clear

b. To what extent proposal selection criteria are clear

c. To what extent proposal selection procedure is clear

d. To what extent proposal selection outcome is clear

1.2.2 Careful (yes/no)

a. To what extent criteria for proposal selection are applied well (objective and impartial).

1.2.3 Policy or practice oriented

a. To what extent the policy and/or practice perspective of stakeholders is taken into account

157

before and during the proposal submitting phase (type and number of contacts in this

phase between KCE and specific stakeholders, to what extent the research agenda fits the

needs of the stakeholder)

b. To what extent the practice perspective of stakeholders is taken into account during the

proposal selection procedure.

c. To what extent proposal selection is Policy or Practice oriented.

Data source

1.1.1. Typology of stakeholders : Reports (documents of KCE) Overview of submitted proposals

2004 t/m 2008, number of proposals submitted per stakeholder group.

1.1.2. Typology of subjects: Reports (documents of KCE) over of submitted proposals 2004 t/m

2008 per field.

1.2.1. Transparency: 20 stakeholders, KCE document on selection and research priority setting

procedure

1.2.2. Carefulness: stakeholders, board of directors, KCE document on selection and research

priority setting procedure.

1.2.3. Policy or practice oriented: stakeholders, board of directors, internal project

coordinators, board of directors, KCE document on selection and research priority setting

procedure.

Data collection techniques

1.1..1 Typology of stakeholders: Desk study Documents: depending on the availability of online documents (website) or by send by e-

mail (by KCE), or by a visit of the researchers to KCE office. Group h: “Privéorganisaties of

Privépersonen” will be reviewed and if possible a sub division will be made.

1.1.2. Typology of subjects: document analysis: depending on the availability of online

Deskstudy: documents (website) or send by e-mail (by KCE), or by a visit of the Researchers to

KCE office (list of needed documents is in appendix 5).

1.2.1 Transparency: a. Desk study (formal selection procedures, criteria) b. Semi-structured interviews stakeholders

1.2.2. Carefulness: a. Semi-structured face to face interviews with stakeholders b. Desk study (see 1.2.1)

1.2.3. Policy or practice oriented: a. Semi-structured face to face interviews with stakeholders b. Group interview with senior KCE staff

Topic 2: Quality

2.1 Project management

2.1.1 Focused on usefulness of output/results?

2.2 Methodology

2.2.1 Scientifically based?

2.2.2 Adequately supervised?

Specific

158

2.1.1 Focused on usefulness of output/results?

Attention paid to usefulness of output/results during the project (proposal selection, during

project execution, in writing the reports

2.2.1 Scientifically based?

Formal procedures to ensure existing scientific knowledge is used (yes/no), external validation

by experts during the project (yes/no). Standards for data collection, synthesis (yes/no). Are

procedures used by KCE staff (Yes/no).

2.2.2 Adequately supervised?

Supervision with clear responsibilities during the course of the project?

Data source

2.1.1 Focused on usefulness of output/results? a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases) c. Board of Directors (see list of 20 stakeholders) d. KCE staff (project level – overall level – case level)

Project level (coordinators of projects between 2004 t/m 2008) Overall level (senior KCE staff in GCP, HTA, HSR), selection to be made by KCE Case level (project coordinator)

2.2.1 Scientifically based? a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases) c. Board of Directors (see list of 20 stakeholders) d. KCE documents (overall procedures valid for KCE as an organisation, summary reports).

2.2.2 Adequately supervised? a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases) e. KCE staff (project level – overall level – case level)


Data collection techniques

2.1.1 Focused on usefulness of output/results? a. Semi-structured face to face interviews b. Desk study: Formal procedures to focused on usefulness of output/results (if existent). c. Desk study on case level (documented proof of attention paid on usefulness of output), d. Semi-structured interviews on case level e. Self reports, semistructed webbased questionnaire f. Group interview with senior KCE staff

2.2.1 Scientifically based? a. Semi-structured face to face interviews b. Desk study (KCE documents with procedures) c. Desk study on case level d. Semi-structured interviews on case level e. Self reports, semistructed webbased questionnaire f. Group interview with senior KCE staff

2.2.2 Adequately supervised? a. Semi-structured face to face interviews b. Desk study (KCE documents with procedures)

c. Semi-structured interviews on case level

d. Group interview with senior KCE staff

159

Topic 3: Involvement of stakeholders during research process 3.1 Before, during and at the end of the project? 3.2 Different types of involvement?

3.3 High degree of involvement?

Specific 3.1 Before, during and at the end of the project?

If stakeholders are involved before, during and at the end of the project. Yes/No.

Before: defining and preparing the project.

During: execution of the project

End: Discussing draft report, draft recommendations, results.

3.2 Different types of involvement?

Types of involvement: For example as external validator, by participating in the research. Types of involvement will be

categorized based on the collected data.


Frequency and amount of time (approximately), per phase (Before, during, end).

Data source 3.1 Before, during and at the end of the project?

a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases) c. Board of Directors (see list of 20 stakeholders) d. KCE staff (project level – overall level – case level)


e. KCE documents KCE level (overall procedures valid for KCE as an organisation). 3.2 Different types of involvement?

See 3.1.


See 3.1

Data collection techniques 3.1 Before, during and at the end of the project?

c. Semi-structured face to face interviews stakeholders d. Desk study on case level e. Semi-structured interviews on case level f. Self reports, semi structured web based questionnaire g. Bibliometric search h. Group interview with senior KCE staff

3.2 Different types of involvement?

See 3.1.


See 3.1..

160

Topic 4: Impact intended 4.1 Explicitly formulated in advance?

4.2 Primary target audience(s) aimed for

4.3 Type of impact aimed for

Specific 4.1 Explicitly formulated in advance?

a. Yes/No

b. SMART

4.2 Primary target audience(s) aimed for (factual)

Possible primary audiences.

Government (check bij Francois)

Ministry of Social and Health Services

Federal Government Department of Health, Food etc.

Federal Government Department of Social Security

RIZIV and committees (p.e. NRKP)

University and scientific institutions

Healthcare industry

Patient organizations

Mutualities, Intermutual Agency

Care providers

o Primary Care

o Secondary Care

o Medical professionals and nursing personnel

o Pharmacies

International Organisations

Politicians (commission santé Chambre)

Other


a. Instrumental (see table)

b. Conceptual (see table)

c. Symbolic (see table)

161

Hivon et al, 2005

Type of impact continued:

a. Pre-policy: problem described, causes and consequences studied, trend or spread traced.

b. Developing policy: Develop awareness, mobilize, reframe the issue

c. Policy enactment: advocacy, frame policy, pass policy/legislation.

d. Post-policy enactment: implementation, access & equity, sustain, change or abandon

Ottoson et al., 2009:

Data source 4.1 Explicitly formulated in advance?

a. KCE staff (project level – overall level – case level) a. Project level (coordinators of projects between 2004 t/m 2008) b. Overall level (senior KCE staff in GCP, HTA, HSR), selection to be made by KCE c. Case level (project coordinator)

b. KCE documents a. Case level (documents on specific projects) b. KCE level (overall formal procedures, aimed at formulation of intended impact).

4.2 Primary target audience(s) aimed for

See 4.1.

4.3 Type of impact aimed for (e.g. instrumental/ conceptual/ symbolic ; pre-policy/ de-veloping policy/ policy

enactment/ post-policy enactment )

See 4.1.

Data collection techniques 4.1 Explicitly formulated in advance?

a. Desk study (present in KCE procedure Yes/No) b. Desk study on case level (present in project documents Yes/No) c. Semi-structured interviews on case level d. Self reports, semi structured web based questionnaire e. Group interview with senior KCE staff

4.2 Primary target audience(s) aimed for (factual) a. Desk study b. Desk study on case level c. Semi-structured interviews on case level d. Self reports, semi structured web based questionnaire e. Group interview with senior KCE staff


See 4.1

162

II Dissemination Topic 5: Types and number of outputs produced 5.1 Reports?

5.2 Recommendations?

5.3 Journal articles?

5.4 Citations

5.5 Presentations given?

5.6 Media coverage?

Specific 5.1 Reports?

Number of reports 2004 t/m 2009

- Fields (GCP, HTA, HSR)

- Topics

- Requesters (proposal written by?)

5.2 Recommendations?

Type of recommendations

- recommendations addressed to specific stakeholder yes/no

- recommendations for policy making

- recommendations for practice / behaviour

- recommendations for short term

- recommendations for long term

5.3 Journal articles?

Number of articles, type of journals

- peer-reviewed journals

- non peer-reviewed journals

5.4 Citations

Number of citations (to be specified in bibliometric search strategy)

- scientific literature

- grey literature

5.5 Presentations given?

Number of presentations given. Type of audience.

- primarily academic audiences

- primarily policy making audiences

- primarily practitioner/service user audiences

Posible stakholders in the audience:

Government (Cabinet)






Healthcare industry



Care providers

o Primary Care

163

o Secondary Care


o Pharmacies


Politicians (commission santé Chambre)

Other (other categories may arise)

5.6 Media coverage?

Media coverage on KCE reports. Type of media. Type of coverage

Data source 5.1 Reports? Data sources:

Overview of all reports published by KCE (2004-2009)

5.2 Recommendations? Data sources:

KCE staff (project level) 2004 t/m 2008.

5.3 Journal articles? Data sources:

a. KCE documents (overview of published articles 2004 t/m 2008) b. KCE staff (project level) 2004 t/m 2008)

5.4 Citations

Data sources: a. KCE documents (overview of citation 2004 t/m 2008) b. KCE staff (project level) 2004 t/m 2008)

5.5 Presentations given? KCE staff (project level) 2004 t/m 2008)

5.6 Media coverage? KCE documents (overview of media coverage 2004 t/m 2009) KCE staff (project level) 2004 t/m 2008)

Data collection techniques 5.1 Reports?

Desk study

5.2 Recommendations? a. Desk study b. Self reports, semi structured web based questionnaire

A differentiation of types of recommendations will be made on basis of the published reports.

5.3 Journal articles? a. Desk study b. Self reports, semi structured web based questionnaire

5.4 Citations a. Desk study b. Self reports, semi structured web based questionnaire

5.5 Presentations given? Self reports, semi structured web based questionnaire

5.6 Media coverage? a. Desk study b. Self reports, semi structured web based questionnaire

164

Topic 6: Dissemination activities performed 6.1 Tailored design of the products?

6.2 Developed ‘synthesis’ products?

6.3 Expanded contacts and working relationships with stakeholders?

6.4 Communicated findings to intermediaries?

6.5 Made products easily obtainable?

6.6 Communicated on the right time (monitoring policy developments)?

Specific 6 6.1 Tailored design of the products?

Tailored in the sense of ‘ready to use’ for stakeholders. Yes/No.

6.2 Developed ‘synthesis’ products?

(Additional) products that summarize the findings on a specific topic, meant to diffuse knowledge and

identify gaps in knowledge and stakeholders need for additional research.


Specifically in dissemination activities.


Intermediaries are people/organizations that can connect the researchers’ work to policy activity.


Easy to access, fast delivery, online available.


Is timing of dissemination activities adjusted to the needs of stakeholders. For example the policy

cycle of stakeholders.

Data source 6.1 Tailored design of the products?



e. KCE documents Case level (documents on specific projects) KCE level (overall procedures valid for KCE as an organisation, summary reports).

6.2 Developed ‘synthesis’ products? a. KCE staff (project level – overall level – case level)


b. KCE documents c. Case level (documents on specific projects)


See 6.1.


165

See 6.2


See 6.1


See 6.1

Data collection techniques 6.1 Tailored design of the products?

a. Semi-structured face to face interviews with stakeholders b. Desk study c. Desk study on case level d. Semi-structured interviews on case level e. Self reports, semi structured web based questionnaire f. Group interview with senior KCE staff

6.2 Developed ‘synthesis’ products? a. Desk study b. Desk study on case level c. Semi-structured interviews on case level d. Self reports, semi structured web based questionnaire e. Group interview with senior KCE staff


See 6.1


See 6.2


See 6.1


See 6.2

Topic 7: Stakeholders addressed by outputs 7.1 Type of stakeholders

7.2 Corresponding with research proposal and intended impact?

Specific 7.1 Type of stakeholders

Government (Cabinet)






Healthcare industry



Care providers

o Primary Care

o Secondary Care


o Pharmacies

166


Politicians (commission santé Chambre) Other


To what extent output is conform intended output (see also 4.1, 4.2 and 4.3)

Data source 7.1 Type of stakeholders

a. KCE staff (project level – overall level – case level) Project level (coordinators of projects between 2004 t/m 2008) Overall level (senior KCE staff in GCP, HTA, HSR), selection to be made by KCE

b. KCE documents Case level (documents on specific projects) KCE level (overall procedures valid for KCE as an organisation, summary reports).


See 4.1; 4.2; 4.3

a. KCE staff (project level – overall level – case level) Project level (coordinators of projects between 2004 t/m 2008) Case level (project coordinator)

Data collection techniques 7.1 Type of stakeholders

a. Desk study b. Desk study on case level c. Semi-structured interviews on case level d. Self reports, semi structured web based questionnaire e. Group interview with senior KCE staff

7.2 Corresponding with research proposal and intended impact? a. Semi-structured interviews on case level b. Self reports, semi structured web based questionnaire c. Group interview with senior KCE staff

Topic 8: Engagement of stakeholders in outputs 8.1 Stakeholders involved in developing outputs?

8.2 Stakeholders involved in disseminating outputs?

Specific 8.1 Stakeholders involved in developing outputs?

Yes/No

If Yes: type of involvement


Yes/No

If Yes: type of involvement

Data source 8.1 Stakeholders involved in developing outputs?



167

e. KCE documents Case level (documents on specific project) KCE level (overall procedures valid for KCE as an organisation on involving

stakeholders on in developing outputs).


See 8.1.

Data collection techniques 8.1 Stakeholders involved in developing outputs?

a. Desk study b. Desk study on case level c. Semi-structured interviews on case level d. Self reports, semi structured web based questionnaire e. Group interview with senior KCE staff


See 8.1

Topic 9: Actual take-up of outputs 9.1 Amount of downloads per project

9.2 Number of requests for information from projects

Specific 9.1 Amount of downloads per project

From first online publication t/m 2008, if available.


From first publication t/m 2008.

Data source 9.1 Amount of downloads per project

KCE, overview.


KCE overview if available

Data collection techniques 9.1 Amount of downloads per project

a. Desk study

9.2 Number of requests for information from projects a. Desk study

168

III Contribution to the decision-making process Topic 10: Familiarity with research projects 10.1 Knowledge of relevant research projects among specific stakeholders

10.2 Reading of reports/articles from projects

Specific 10.1 Knowledge of relevant research projects among specific stakeholders

Research projects spontaneously mentioned by stakeholders.


Channels stakeholders use to obtain information from KCE projects

Data source 10.1 Knowledge of relevant research projects among specific stakeholders

a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases)


See 10.1

Data collection techniques 10.1 Knowledge of relevant research projects among specific stakeholders

a. Semi-structured face to face interviews with stakeholders b. Semi-structured face to face interviews with stakeholders on case level


See 10.1

Topic 11: Research utilization 11.1 Influenced by the findings?

11.2 Actually used the findings?

11.3 Type of use

11.4 Reasons for not using the findings from a project

Specific 11.1 Influenced by the findings?

To what extent policy and/or practice is influenced by findings.

- both direct (use) and indirect (e.g. changes in legislation)


Yes/No.

11.3 Type of use

A differentiation of different types of use, derived from the data collected.


A differentiation of types of reasons for not using the findings, derived from the collected data.

Influential factors for use of findings: The channel of communication (influence of sources/information) The coherence of findings with prevailing clinical beliefs (do the findings require ‘out of

the box’ thinking from professionals?) The quality and the quantity of the evidence The nature of the change (is the change easily implementable well fitting, easy to

learn/do, inexpensive)

169

The impact of the change on key clinical branch points (do the findings meet clinical needs of the professionals?)

Weiss 2007

The following factors enhance the use of HTA in decision-making:

Timeliness (conducting research when needed/relevant and revising assessment when new data becomes available)

Appropriately communicating HTA findings to different decision-makers (multiple communication strategies to meet the disparate needs of different users)

Making clear the relationship between HTA findings and the decision-making processes (same findings may lead to different decisions (according to situation or actor), to some extent decisions can be predicted)

Drummond et al. 2008

Data source 11.1 Influenced by the findings?



e. KCE documents case level (documents on specific projects)


See 11.1.

11.3 Type of use

See 11.1

11.4 Reasons for not using the findings from a project a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases) c. Board of Directors (see list of 20 stakeholders) d. KCE staff (project level – overall level – case level)


Data collection techniques 11.1 Influenced by the findings?

a. Semi-structured face to face interviews with stakeholders b. Desk study on case level c. Semi-structured interviews on case level d. Self reports, semi structured web based questionnaire e. Group interview with senior KCE staff


See 11.1

11.3 Type of use

See 11.1


See 11.1

170

Topic 12: Relevance of projects for policy-making / practice 12.1 Timely delivery?

12.2 Answering to information need?

12.3 Usable conclusions/recommendations (tailoring)?

Specific 12.1 Timely delivery?

Yes/No

Reasons why it was/wasn’t timely according to the stakeholder.


In what measure the project answers to the information need (very high, high, average, low very low).

Reasons why information needs are / aren’t met.


To what extent conclusions/recommendations are usable (very high, high, average, low very low).

Data source 12.1 Timely delivery?

a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases) c. Board of Directors (see list of 20 stakeholders)


See 12.1


See 12.1

Data collection techniques 12.1 Timely delivery?

a. Semi-structured face to face interviews with stakeholders (including B.O.D.) b. Semi-structured interviews with stakeholders on case level


See 12.1


See 12.1

Topic 13: Reactions on finished projects from stakeholders 13.1 Feedback to KCE: positive of negative?

13.2 Comments in media: positive or negative?

Specific 13.1 Feedback to KCE: positive of negative?

Number of occasions where stakeholders gave feedback to KCE (interactivity). Nature of the feedback (positive /

negative). Content of the feedback.

13.2 Comments in media: positive or negative?

Number of occasions where feedback was given on KCE projects in the media. Nature of the feedback (positive /

negative). Content of the feedback.

171

Data source 13.1 Feedback to KCE: positive of negative?


Project level (coordinators of projects between 2004 t/m 2008) Case level (project coordinator)

e. KCE documents Case level (received written feedback) KCE level (received written feedback).

13.2 Comments in media: positive or negative? a. KCE documents, Case level (published feedback on project in case) b. Project level (coordinators of projects between 2004 t/m 2008)

Data collection techniques 13.1 Feedback to KCE: positive of negative?

a. Semi-structured face to face interviews with stakeholders (including B.o.D) b. Desk study c. Desk study on case level d. Semi-structured interviews on case level e. Self reports, semi structured web based questionnaire

13.2 Comments in media: positive or negative? a. Desk study b. Desk study on case level

IV External image of KCE Topic 14: General reputation of KCE 14.1 Trustworthiness?

14.2 Authority

14.3 Independence?

14.4 Helpfulness?

14.5 Position in Belgian health care field?

Specific 14.1 Trustworthiness?

Level of trustworthiness: (very high, high, average, low very low).

Reasons mentioned for a specific score (open)

14.2 Authority

Level of Authority: (very high, high, average, low very low).


14.3 Independence?

Level of independence: (very high, high, average, low very low).


14.4 Helpfulness?

Level of Helpfulness: (very high, high, average, low very low).



KCE’s percieved level of knowledge and expertise concerning health care matters: (very high, high, average, low very

low).Reasons mentioned for a specific score (open)

172

Data source 14.1 Trustworthiness?

a. Stakeholders (see list of 20 stakeholders) b. Stakeholders on case level (to be specified after selection of cases)

14.2 Authority

See 14.1

14.3 Independence?

See 14.1

14.4 Helpfulness?

See 14.1


See 14.1

Data collection techniques 14.1 Trustworthiness?

a. Semi-structured face to face interviews with stakeholders (including B.o.D)

b. Semi-structured face to face interviews with stakeholders on case level

14.2 Authority

See 14.1

14.3 Independence?

See 14.1

14.4 Helpfulness?

See 14.1


See 14.1

Topic 15: Judgment on responsiveness 15.1 Clear communication?

15.2 Timely responding?

15.3 Tailoring products to ‘customers’?

15.4 Fulfilment of information needs?

Specific 15.1 Clear communication?

Yes/No, Reasons


Yes/No. Reasons


Yes No. Reasons


Yes/No. Reasons

173

Data source 15.1 Clear communication?



See 15.1


See 15.1


See 15.1

Data collection techniques 15.1 Clear communication?




See 15.1

15.3 Tailoring products to ‘customers’?, See 15.1

15.4 Fulfilment of information needs?, See 15.1

Topic 16: Judgment on research quality 16.1 Competent researchers?

16.2 Good choice of methodology?

16.3 Good analysis of data?

16.4 Valid conclusions?

Specific 16.1 Competent researchers?

very good, good, average, bad, very bad







Data source 16.1 Competent researchers?



See 16.1

174


See 16.1


See 16.1

Data collection techniques 16.1 Competent researchers?




See 16.1


See 16.1


See 16.1

175

APPENDIX 4: SOURCES Desk

analysis of KCE docs

Group session KCE staff

Interviews Board of Directors

Interviews Stake-holders

Question-naire Self-reports

Citations Downloads counts

Case studies of research projects

1. Origin of research 2. Quality 3. Involvement of stakeholders

I4. Impact intended 5. Outputs 6. Dissemination activities 7. Stakeholders addressed 8. Engagement of stakeholders

9. Take-up of outputs 10. Familiarity with projects 11. Research utilization

12. Relevance of projects 13. Reactions on projects 14. General reputation KCE 15. Judgment on responsiveness

16.Judgment on research quality

176

APPENDIX 5: DOCUMENT ANALYSIS

Document analysis

Documents needed and questions to be answered

Origin of the research

List of submitted research proposals for 2005 and 2008

Q: Which stakeholders have submitted proposals?

Q: Which subjects do they want to address?

Scoring criteria formulated in 2005 and 2008

Q: Who is responsible for formulating the criteria used?

Q: Have the criteria used for the selection of proposals changed during the existence of KCE? If so, how and why?

Scoring procedure formulated in 2005 and 2008

Q: What is the role and position of the Board of Directors in the scoring and selection process?

Q: Does the scoring procedure guarantee enough transparency?

Q: Does the scoring procedure guarantee objectivity and impartiality?

Q: How and by whom is decided upon the score a proposal gets on each criterium?

Q: Are the scores on each criterium given even weight, or are some criteria more important than others? If so:

Is the weight the same for every proposal or does this vary? Why?

Is the weight the same every round or does this vary? Why?

Q: Are the criteria policy- or practice oriented?

Scoring reports for research proposals submitted in 2005 and 2008

Q: Are the criteria being applied according to the procedure?

Q: Is the scoring procedure applied in a transparent way?

Final list of selected research projects in 2005 and 2008

Q: What kind of projects are selected? (Topic, timespan, interested stakeholder, health care field, etc.)

Research planning and allocated resources for these projects (budget + manpower)

Q: What prioritization does KCE apply?

Quality

External validators’ reports for 2005 and 2008

Q: How is the (scientific) quality of the final reports judged?

Existing protocols on project management procedure(s)

Internal projects

Partly outsourced projects

Fully outsourced projects

Q: Is project management focused on usefulness of outputs/results?

Q: Is the research project adequately supervised?

Existing protocols on procedure for:

project launch

project finalization

177

Q: Is project management focused on usefulness of outputs/results?

(or, if not available, descriptions of applied procedure(s))

Involvement of stakeholders during research process

Existing protocols for stakeholder involvement

During project launch

During research process

During project finalization

Q: Does KCE structurally attempt to involve stakeholders during different stages of a project? If so, in what way?

Q: What possible types of involvement is KCE aiming for?

Q: What degree involvement is KCE aiming for?


Impact intended

Existing protocols about identifying (at the beginning of a project):

Target-audiences aimed for

Type of impact aimed for

Evidence of projects in which KCE has, at the beginning of a project, explicitly formulated:

Possible target-audiences for research findings

Type(s) of impact aimed for

Q: Does KCE structurally attempt to explicitly formulate possible target-audiences? If so, in what way?

Q: Does KCE structurally attempt to explicitly formulate the type of impact aimed for? If so, in what way?

Dissemination activities performed

Existing protocols for tailoring to the information need of stakeholders:

the research goals

the type of findings

the products

Existing protocols for dissemination/communication about findings

Q: Does KCE structurally attempt to tailor its products to the information need of stakeholders? If so, in what way?

Q: Does KCE use protocols for dissemination/communication? If so, what is the procedure?


Engagement of stakeholders in outputs

Existing protocols for stakeholder involvement in:

Developing outputs

Disseminating outputs

Q: Does KCE structurally attempt to involve stakeholders in developing outputs? If so, in what way?

Q: Does KCE structurally attempt to involve stakeholders in disseminating outputs? If so, in what way?


Reactions on finished projects from stakeholders

Written feedback on research projects expressed by stakeholders to KCE

Overview of requests for information (if available).

178

Comments on KCE projects expressed by stakeholders in the media

Q: Does KCE receive any feedback on their research projects? If so, what is the content of this feedback?

Q: Does KCE receive any feedback via statements in the media? If so, what is the content of this feedback?

179

APPENDIX 6: QUESTIONNAIRE GROUP SESSION

Chair: François Daue

Reporter: two researchers from Research for Beleid

Participants (11 KCE staf members, divided over the three fields: HSR, GCP and HTA).

Agenda:

Introduction:

g. Information on impact measurement

h. Aim of the meeting (and reporting of outcomes in relation to other data sourcesyyy)

i. Recording of the data (status of tape, to support the research, anonymity guaranteed)

j. Timing and organisation of the meeting

k. Agreement on meeting procedures

Findings from the document analysis

Discussion on questions

Summary of the meeting (by moderator with input from participants)

Aims of group session: To explore the (different) experiences of the KCE experts with regard

to quality control, the involvement of stakeholders, intended impact, outputs, and dissemination.

To focus on facilitating and impeding factors as experienced by experts.

To gain insight in perceptions of experts on what “conditions” or contextual factors influence the impact of a report

Particular attention will be paid to the “learning experience” of individual experts and the organization over time

Questions discussed during the meeting

To ensure that the most critical questions for this group session are answered, the discussion topics have been prioritized.

1. Quality

What KCE quality control procedureszzz are most effective in guaranteeing the scientific quality of (ongoing) research projects? Which procedures are not so effective? Examples.

What are the reasons for this?

Have there been major changes in the KCE quality control procedures over time? Which? In your opinion, how has this impacted on the scientific quality of the research projects?

How could the scientific quality of research projects be further improved?

How do you guarantee the usability of outputs/results (e.g. for recommendations to be accepted and implemented)? Examples: what works, what does not work.

What limiting factors or circumstances play a role in the usability of outputs/results?

What is your experience with the involvement of policy experts (examples)? Did their input improve the usability of outputs and

yyy Document study, self-reports by internal and external project leaders, stakeholder interviews zzz Such as: internal validation, expert meetings, external validation, final evaluation by the Board,

Wetenschappelijke Adviesraad (WAR)

180

results? How / why not? Are they involved in all projects? What does this depend on?

How do your informal contacts with the field ensure usability of the outputs/results? Examples.

l. Which project management procedures aaaa do you perceive as contributing to the usabilitybbbb of results, and which (if any) are limiting? Examples and reasons.

2. Involvement of stakeholderscccc

What are your experiences with the involvement of stakeholders in the research process? Examples of positive and negative experiences.

Which contextual factors play a facilitating or limiting role in the involvement of (certain) stakeholders? How does research domain affect the potential for stakeholder involvement? Does this also depend on the type of stakeholder?

Is a difference made between the types of stakeholders? How is decided which stakeholders are involved?

How has stakeholder involvement (or a lack thereof) in the examples affected the impact of the research?

Have your inclination and approach to involve stakeholders changed over time? How?

Within KCE generally, do you feel that the extent of involvement of stakeholders in projects has changed over time? How?

How could the involvement of stakeholders in the research process be further improved?

From your experience, what role do the project teams play - in practice - in stimulating the involvement of stakeholders in a project? Are they involved in most projects?

3. Impact intended

How are the target-audiences dddd identified and specified? Examples of processes followed.

In practice, do you do any follow-up regarding the impact a project has had? If so, in what ways? Examples.

What do you do with this information?

Is this done for every KCE project? If not, why not? What does this depend on?

To what extent do you perceive the domains in which you are operating as influencing the “impact” of the research?

What are the reasons for this?

How could the impact be improved in the domain(s) where KCE is less effective?

How do informal ways of maintaining contacts with the field influence the impact of your projects?

m. Do you always explicitly formulate the type of impact aimed for with each project?

If so, in what way? Examples.

How does this differ per domain, or per project? What factors play a role in this regard?

aaaa For example: procedure for topic prioritisation and annual work programme, operationalisation procedure,

kick-off meeting process, outsourcing process., expert meetings procedures., internal and external validation procedures, end-phase procedures (final acceptance and publication)

bbbb The usability of research findings and recommendations is influenced by timeliness, relevance, clarity, practicality / feasibility.

cccc Project stakeholders are all groups, institutions or persons that can affect, are affected by, the research project. Stakeholders may be classified in the following groups: politicians, institutional/governmental (FOD, RIZIV), scientists, Actors: hospitals and organized physicians, patients, and the general public.

dddd The target audience may be identified and specified within one or more of the following groups: politicians, institutional/governmental (FOD, RIZIV), scientists, Actors: hospitals and organized physicians, patients, and the general public.

181

4. Outputs produced

What kinds of outputseeee do you consider as having the most impact? Why?

How does this differ per domain, or per project? Which contextual factors play a role?

Are any outputs produced within KCE that are labour or cost intensive but that do not have a major impact? Which?

Who decides which ‘outputs’ are being produced? What reasons are given?

How are the potential users of the outputs identified? Give examples of process followed.

Do you mostly involve stakeholders in developing outputs?

If so, in what way? Examples of successful experiences and facilitating factors.

In what cases is this not possible? What are limiting factors or conditions?

Is it general practice in KCE projects to involve stakeholders in developing outputs?

Are quality checks performed on all the outputs?

If so: who is responsible for this and how is this done?

If not why not?

5. Dissemination activitiesffff

What activities for communication and dissemination of the findings and outputs do you consider most effective? Examples.

Which are not so effective? Why?

Do you mostly involve stakeholders in disseminating outputs? If not, why not?

If so, in what ways? How do you select the stakeholders to be involved?

To what extent does this work? What are facilitating and limiting factors?

What attempts do you make to get outputs from research across to potential usersgggg? Exampleshhhh.

Which contextual factors play a role in the effectiveness of the dissemination activities?

Can you give examples of potential user groups that are currently not easily reached?

What improvements should be made in this regard?

Do you tailor the outputs to the information need of different stakeholders?

If so, in what ways? Examples.

In what cases or circumstances not?

What more or different should be done in this regard?

7. Reactions on projects

Did you frequently receive feedback on your research project?

How have you used the feedback that you have received on your research projects? Examples?

Is a difference made between feedback from different stakeholders? If so, how?

How do you address any negative feedback?

eeee For example: research report, commissioned report, journal article, book, chapter, conference contributions,

discussion papers, oral presentations, film / video, software, data base, media interviews) ffff Sharing information, findings or outputs with others gggg For example: policy makers, scientists, hospitals, medical professionals, patients, media hhhh For example: putting information/reports on website, publishing newsletters, spreading brochures, delivering

presentations, etc.)

182

8. Origin of the research

Could you explain in practice the selection process and how the choices are balanced between HSR-GCP and HTA?

To what extent do you experience the scoring and selection process of topics for the work programme impartial and objective in practice? If not, why not? Examples?

Is there room for conflict of interest? Examples?

Has the objectivity of the selection process changed over time? How?

--- End ---

183

APPENDIX 7: QUESTIONNAIRE STAKEHOLDERS

Research phase 1

Interviews with stakeholders

General questions (approx. 5 minutes)

1. What is your position in <name organization>? (Only ask if unclear)

2. Do you have personal contact with KCE (other than in your position as member of the B.o.D.)? If so, who are you in contact with (name + function)?

3. How often do you as a person have contact with KCE?

Once a week

Once a month

Once per 3 months

Once per 6 months

Once per 9 months

Once a year

Less than once a year

4. How often does your organization have contact with KCE?

Once a week

Once a month

Once per 3 months

Once per 6 months

Once per 9 months

Once a year

Less than once a year

5. Which area of research that KCE is active in, has your organization’s main interest?

HSR

GCP

HTA

Origin of research (approx. 10 minutes) 1. What is you opinion on the proposal selection process at KCE?

Is it transparent (clear)? Yes/no

Is it objective and impartial? Yes/no

Explanation:

2. Did KCE ever refuse a proposal from your organization? If so: were the reasons for refusal made clear to your organization?

3. Are the topics KCE has put on its research agenda so far answer to the information need of your organization?

If so: why?

184

If not: what are you missing?

Involvement of stakeholders (approx. 10-15 minutes)

4. What kind(s) of (active) contact does your organization have with KCE?

Submitting proposal(s)

Involvement in research project begin, during, end?

Involvement in dissemination activities which activities?

Other project related: ……

Asking information or advice

Giving information or advice

Member of B.o.d.

Other not project related: ……

For each type of contact named: when was the last time you had contact?

5. If ‘involvement in research project’ is mentioned:

In what research project(s) was your organization involved by KCE?

How were you involved?

If ‘asking information or advice’ is mentioned:

6. Does KCE respond timely to information requests from your organization?

Always

Most of the time

Sometimes timely, sometimes untimely

Rarely

Never

Explanation:

7. Does KCE adequately help you with your information request?

Always

Most of the time

Sometimes timely, sometimes untimely

Rarely

Never

Explanation:

Dissemination (approx. 20-30 minutes)

1. How is your organization being informed about KCE activities?

(E.g. mailing list, media, internal monitoring, website KCE, etc.)

185

2. Are you satisfied with the way KCE communicates with your organization?

Highly satisfied

Satisfied

Neutral

Not satisfied

Highly dissatisfied

Explanation:

3. Are new findings from KCE being monitored on a regular basis within your organization? Yes/No

If so, how?

Please name all KCE studies that you or your organization is acquainted with:

4. To what extent and in what way was your organization influenced by the findings from these studies - both direct (use) and indirect (e.g. changes in legislation)? Answer per study if possible

5. If findings were used by organization: Why and how did your organisation use these findings?

6. What have been reasons for your organization NOT to use findings reported by KCE?

7. How did you or your organization become acquainted with these studies?

Written the research proposal

Active involvement during the project

Reading report

Reading recommendations

Reading an (journal) article about it

Citation in media / media coverage

Internet search

Visit to KCE website

Congress / course / (poster) presentation

(E-)mailing from KCE

Personal contact with KCE staff

Heard from a friend or colleague (external)

Other: …

If ‘writing of research proposal’ was mentioned:

8. Did the final report meet your information need in terms of findings, way of presenting these and the timeliness? Why/why not?

186

9. Did you give any feedback (positive or negative) to KCE about the quality or usability of the studies/findings? Why/why not?

External image of KCE (approx. 15-20 minutes)

10. How do you judge the trustworthiness of KCE?

Very high

High

Average

Low

Very low

Explanation:

11. How do you judge KCE’s degree of independence?

Very high

High

Average

Low

Very low

Explanation:

12. How do you judge KCE’s helpfulness or serviceability?

Very high

High

Average

Low

Very low

Explanation:

13. How do you judge KCE’s ability to conduct research that meets scientific standards?

Very high

High

Average

Low

Very low

Explanation:

14. How do you judge KCE’s knowledge and expertise concerning health care matters?

Very high

High

Average

Low

Very low

187

Explanation:

15. How do you judge KCE’s ability to make findings usable for influencing policy or practice? on health care issues?

Very high

High

Average

Low

Very low

Explanation:

16. How do you judge the quality of KCE (in general) on:

Choice of research questions (very good, good, average, bad, very bad)

Choice of research methodology (very good, good, average, bad, very bad)

Way of reporting on findings (very good, good, average, bad, very bad)

Way of project management (very good, good, average, bad, very bad)

Researchers working on projects (very good, good, average, bad, very bad)

Explanation:

Conclusive questions (approx. 10-15 minutes)

1. Are there other actors in the field of health care research (national and international) whose research is relevant for your organization?

2. What is KCE’s added value compared to these other actors?

3. Do you have any suggestions for improvements KCE might work on?

Origin of research

n. Involvement of stakeholders

o. Dissemination

p. External image

4. Choose one improvement that should, in your view, be top priority for KCE. Please give a short explanation.

188

Additional questions for Board of Directors:

KCE impact assessment

Interview members of Board of Directors

List of interview questions

1. What is the role and position of the Board of Directors in the scoring and selection process?

2. How would you define your personal task in the scoring and selection process?

3. Who is responsible for formulating the criteria used?

4. Have the criteria used for the selection of proposals changed during the existence of KCE? If so, how and why?

5. How much discussion is there (generally) between Board members about whether:

a. a proposal meets the admission requirement that it has to belong to one of the three research domains?

b. the proposed study concerns an important health problem for the population and/or if the study concerns an important social-ethical issue in health care?

c. the answers from a study might lead to a more efficient use of resources in health care?

d. there is enough information for assessing the feasibility of the project?

e. a proposed study can be of assistance to decision-making and/or whether the topic is considered government priority or is mentioned in the FOD’s agreement concerning management?

f. a proposed study makes it possible to build upon the findings of one the studies carried out earlier by KCE?

6. How and by whom is decided upon the score a proposal gets on each criterium?

7. Are the scores on each criterium given even weight, or are some criteria more important than others? If so:

Is the weight the same for every proposal or does this vary? Why?

Is the weight the same every round or does this vary? Why?

8. What is your opinion on the quality and applicability of the current criteria?

9. Do you consider the research proposal selection process as being transparent for third parties?

10. Do you consider the research proposal selection process as being objective and impartial?

11. How do you deal with situations in which a proposal from your own organization needs to be evaluated?

12. Did you ever have a suspicion about a B.o.D. member being biased in the selection process? If so, why?

13. Does KCE give account of the decisions they made in the selection process and the reasons behind the final list of projects, especially to the applicators?

14. Possible influence on the formulation of the final recommendations of the report. It happens quite often that the BoD changes the recommendations that are proposed. These changes can go in different direction: from

189

addition or reinforcement of a recommendation to, on the other hand, deletion or attenuation of a formulation".

190

APPENDIX 8: QUESTIONNAIRE SELF REPORTS KCE impact assessment

Questionnaire for lead researchers

KCE have asked Research voor Beleid to evaluate KCE in terms of the ‘impact’ it has had on policy and practice in Belgian health care. The goal of this study is to take stock after almost six years of activity, and to gain insight into the ways in which impact may arise. Finally, the study will provide KCE with several recommendations on their internal processes and research procedures, focused on heightening their impact.

One part of this study consists of a questionnaire directed at the project coordinators of all projects (both internal and subcontracted) conducted under supervision of KCE. The questionnaire deals with the research process, outputs generated, dissemination activities performed and utilization of the findings.

We kindly ask you, as lead researcher of at least one KCE study, to participate in this study by filling in this questionnaire. If you are prepared to participate, please read the following text carefully before you start with the questionnaire.

If you have acted as project coordinator on several KCE studies, you will probably receive more than one questionnaire. Please be so kind as to fill in each questionnaire about the specific project mentioned.

Filling in the questionnaire will take approximately one hour per project.

Please answer all questions as completely as possible. If you cannot answer a question, simply fill in ‘unknown’ and move on to the next.

Whenever ‘the project team’ is mentioned, we mean your own project team, so the internal project team OR the external project team. Not both!

If you have any questions concerning this questionnaire or the study in general, please contact Alexandra Vennekens ([email protected] or +31(0)79 32 22 607)

Thank you in advance for your cooperation!

With kind regards,

@@@

Part 1: Usability of research findings

Introduction: This part contains questions about various aspects influencing the usability of research findings.

1. How do you judge the relevance of the research findings from this project for policy making?

Very high

High

Average

Low

Very low

Unknown

Please add any comments to your score if you consider this is relevant:

2. How do you judge the relevance of the research findings from this project for health care practice?

Very high

High

Average

Low

191

Very low

Unknown


3. How do you judge the relevance of the research findings from this project for the scientific field?

Very high

High

Average

Low

Very low

Unknown


4. According to you, was the final report issued timely to allow for conclusions or recommendations to be adopted and/or implemented?

Yes

No

Unknown

Please clarify your answer: why? Why not?

5. How do you judge the feasibility/practicability of the recommendations from the project?

Very high

High

Average

Low

Very low

Unknown


6. How do you judge the acceptability of the recommendations from the project?

Very high

High

Average

Low

Very low

Unknown


7. How do you judge the quality of the scientific underpinning of the findings?

Very high

192

High

Average

Low

Very low

Unknown


Part 2: Research utilization

Introduction: This part contains questions about the ways in which the findings from the project have already been or might be used.

1. Has this project, according to you, in any way contributed to policy making in the field of health care?

Yes

No go to question 10

Unknown go to question 10

Please select the way(s) in which the project has contributed:

The findings from the project have been used in policy negotiations (NIHDI, Ministries, cabinets, etc.)

The findings from the project have been used in parliamentary debates and/or for parliamentary questions

The findings from the project have been cited in policy documents

The findings from the project have led to involvement of health service users and/or the public (e.g. commenting, lobbying, protesting, campaigning)

The findings from the project have been used to support pending policy decisions

The findings from the project have directly led to the application of a research-informed policy (e.g. the adoption of recommendations)

Other: …

Please add any comments to your response if you consider this is relevant:

2. Has this project, according to you, in any way contributed to changes in health care practice?

Yes



Please select the way(s) in which the project has contributed:

The findings from the project have lead to changes in management and/or organization of health care services

The findings from the project have led to changes in health care facilities (e.g. equipment, implementation of new technologies, medication, treatments, etc.)

The findings from the project have led to changes in the behavior of medical professionals (e.g. other method of working)

The findings from the project have led to changes in health care industry (price setting, supply of products, etc.)

Other: …

Please add any comments to your response if you consider this is relevant:

193

3. Do you expect the findings to contribute in any way to policy and/or practice in the future?

Yes, to policy

Yes, to practice

Yes, to both policy and practice

No

Unknown

Please clarify: why/ why not?

4. Has this project, according to you, had an impact on the scientific community in any way?

Yes

No

Unknown

Please clarify: why/ why not?

Part 3: Feedback on quality and usability

Introduction: This part contains questions about any feedback you might have received about the quality or usability of the study and/or findings.

1. Did you receive any feedback (positive or negative) about the quality or usability of the study/findings from…

Actors involved in health care policy making?

Yes

No

If yes: What feedback did you receive?

Professionals in health practice or health care providers?

Yes

No


Scientists?

Yes

No


Part 4: Research process and dissemination activities

Introduction: This part contains questions about the involvement of stakeholders during the research process, outputs that have been produced and dissemination activities that were undertaken as a result of the study.

194

(NOOT: vraag 15 en 16 wellicht combineren bij programmeren)

1. According to you, which actor(s) in the health care field were at the start of this project explicitly identified as target audience(s) for the research findings?

None go to question 17

Do not know go to question 17 Politicians sub:

Ministers and cabinets at federal level

Members of the parliament or the parliamentary committee health care

Politicians at regional level

Federal Government Department of Health, Food Chain Safety and environment.


RIZIV and committees

Healthcare industry sub:

Pharmaceutical industry

Health care equipment & services

Biotechnology & life sciences Health care services sub:

Administration/board of directors

Medical professionals

Pharmacists

Nurses

Physiotherapists

Other: … Sickness funds Patient organization(s) International organizations: … Universities and scientific institutions Other:…

2. Please list the selected target audiences in approximate order of importance: the most important is 1, the second most important is 2, etc.

<alle aangekruiste uit vraag 15 laten terugkomen met open, numeriek, invoerveld ervoor>

3. Were any target audiences involved at any moment during the research process? This does not include dissemination activities.

Yes



4. Which target audience(s) was/were involved? Please name the (e.g. politician or medical professionals).

5. On who’s initiative was/were this/these target audience(s) involved?

On the initiative of the target audience(s)

Target audience(s) was/were invited or contacted by KCE (or internal project team)

Target audience(s) was/were invited or contacted by the external project team

Other: …

Unknown

6. In what activities was/were this/these target audience(s) actually involved?

Determining the scope of the research

195

Framing of the research questions

Selecting data collection method(s)

Selecting data source(s)

Designing research instrument(s)

Making data sources easily accessible

Critical reflection during research process

Drawing conclusions

Formulating recommendations

Other: …

Unknown

Outputs

7. Please list any written outputs which, according to you, were produced directly as a result of this specific project. Examples of possible outputs are: reports, recommendations, journal articles (peer-reviewed or non-peer reviewed), published abstracts, books, chapters, posters or leaflets.

In the first column, please specify the type of output (e.g. ‘journal article’)

In the second column, please mention the name or title of the output.

In the third column, if relevant, please specify the place of publication (e.g. the name of the journal)

In the fourth column, please specify the month and year of publication

In the fifth column, please specify whether the output was produced by KCE or by member(s) of the project team.

In the sixth column, please specify whether one or more target audience(s) was involved in producing the output

Type of output

Name/title Place of publication

Date of publication:

Produced by:

Target audience involved in producing output?

(eventueel drop-down menu opnemen met categorie ‘other’)

MONTH - YEAR

(eventueel drop-down menu opnemen)

(Yes/No)

If target audience(s) is/are involved one or more outputs:

8. Which target audience(s) was/were involved? Please name the type of actor (e.g. politician or medical professionals).





Other: …

Unknown

10. What (other) activities, that you know of, were undertaken by the project team and/or KCE to disseminate the findings/outputs of the study?

Sending out press communiqué

Putting message on KCE website

Putting message on other website (name website:…)

196

Telephone call to target audience(s) or stakeholder(s)

Face-to-face contact with target audience(s) or stakeholder(s)

Sending personal e-mail to target audience(s) or stakeholder(s)

Sending out mailing to wider group of target audience(s), stakeholders and/or possibly interested people/organizations

Organizing conference

Giving a presentation

Giving a lecture

Participating in think tank, brainstorming session or consultative body

Appearance in media (newspaper, television, radio)

Monitoring policy developments (in order to communicate the findings when they are most relevant for policy makers and/or for practice)

Other: …

Unknown

11. According to you, have any target audiences been involved in these dissemination activities?

Yes

No go to question 29 or 31

Unknown go to question 29 or 31

12. Please select the activity/activities in which target audience(s) were involved:

<lijst weergeven van in vraag 26 aangekruiste activiteiten>

13. Which target audience(s) was/were involved? Please name the type of actor (e.g. politician or medical professionals).





Other: …

Unknown

Questions for external project coordinators:

15. Did the KCE project coordinator adequately see to keeping the focus of the study in terms of scope and outcome?

Yes, very

Yes

Not adequate/not inadequate

No

No, not at all

Unknown

Please clarify your answer: why/why not?

16. Do you consider the collaboration with KCE in producing outputs and disseminating findings as satisfactory?

Yes, highly

Yes

Average

197

No

No, not at all

Unknown

Please clarify your answer: why/why not?

Final questions

17. According to you, were there any factors other than the ones already discussed in this questionnaire, that might have affected the degree of impact of this study on policy and practice? If so, please describe this factor(s) and specify whether it affected the impact in a positive or negative way.

18. Do you have any further comments about the project, about KCE or about this questionnaire?

You are now at the end of the questionnaire. Thank you for you participation!

198

APPENDIX 9: TABLE PRE-SELECTION CASES

All KCE studies published between July 1st 2005 and July 1st 2008. High impact Manager (internal

and external) average impact Manager low impact ((internal and

external) Manager

@ project title @ name @ project title @ name @ project title @ name

Good Clinical Practice

Health Technology Assessment

Health Services Research

199

APPENDIX 10: CASE STUDY PROTOCOL

CASE STUDY FRAMEWORK

The case study framework gives an indication of the topics to be studied. It’s a general guideline to conduct case study for this research. First of all we introduce a research framework that will serve as a starting point for the collection of data with regard to the cases. The primary goal of the framework is to make explicit what information is searched.

Research framework

In order to study more in-deep KCE research process and design, case study method was selected as a part of methodological approach. By choosing multiply-case sampling we can strengthen the precision, the validity, and the stability of the findings (Miles and Huberman, 1994). In each case study we look at a project from its inception to its finalization, subsequent dissemination and use (or disuse) of the findings. Compared to the other research activities (for example self report and interviews with stakeholders), the case studies explore the selected cases in depth and are more focused on finding explanations. This provides insight into the different roles played by KCE, the researchers and the stakeholders, and the factors that are associated with the utilization (or non utilization) of KCE studies,

In order to learn from current practices, it is relevant to know what exactly caused the (un)successful impact of KCE studies.

We will combine the findings of the nine case studies and do a comparative analysis. We will look at the impact of the different cases and what type of approach appears to generate higher levels of impact. We will evaluate the different chances and challenges concerned in each approach. What problems may arise when trying to adopt the approach? In what situations might the approach have a good chance of success?

Case selection

In order to detail main findings and provide more in-depth knowledge on variables, 9 case studies were conducted. By choosing multiply-case sampling we strengthened the precision, validity, and stability of the findings (Miles and Huberman, 1994). The selection procedure was as follows: (also see appendix 9 and 10).

1. We provided KCE with a list of pre-selection criteria.

2. KCE staff evaluated the impact of all reports published between July 2005 and July 2008. For this they received an online list (see appendix 9) with the projects in which they gave their personal impression (what projects they considered to have high impact, average impact or low impact). We explained the level of impact as : ‘to what extent recommendations are translated into policy decisions, regulations, professional regulations, guidelines, etc.

3. We selected 9 cases at random (3 per KCE research field), with different levels of impact (see 2).

Pre-selection criteria:

1. Each KCE research field (GCP, HTA, HSR) had to be covered by 3 case studies.

2. Projects with recommendations that have been publicized between July 2005 and July 2008. Projects dating from KCE’s first year of existence were excluded, because we considered these are not to be representative and differ too much from later projects. Reports issued after July 2008 were excluded because time span to reach impact could be too short.

200

Selection criteria:

1. Per field three projects with different impact levels have been selected. One high impact, one average, one low.

2. Some project managers left KCE. If former KCE employees couldn’t or wouldn’t cooperate in the case study, the case was excluded.

Phases in case study

Each case study will be conducted in four stages.

1. Fill in self report by internal project manager

2. First contact with by internal project manager

3. Desk research on project

4. Interviews with project manager and other key respondents

5. Synthesis on each case study

First the internal project managers of the selected projects will be informed. We will check whether the web based self reports of the selected projects are already filled in (conducted under activity 1, also see appendix 8). If a project manager didn’t fill in the questionnaire, he/she is invited to do so. We will study the self report and contact the project manager by phone. The project manager is asked to guide us through the case. The goal of this contact is to receive a brief presentation of a case and discuss “must” documents to be studied and the key stakeholders in order to answer research questions.

Desk analysis will be focused on a study of, at least, the following documents:

final report

dissemination materials

general literature on project

proposal submitted

feedback if any

outputs in the form of directives, recommendations, etc (if any)

Also a stakeholder can point interesting documents that are related to the case. The goal of this phase is to provide a contribution into the overall project result and prepare the interviews as well as possible.

Finally, on the basis of the desk analysis questionnaire will be constructed and several interviews will take place (maximum four interviews). The project manager will be interviewed in each case. Then, as related to specific topic the researcher wants to deep-in per case, a specific respondent will be chosen. Potential respondents are:

researchers participating in process

stakeholders in this specific project.

external validator

An interview protocol will be used. The protocol is created on the basis of desk research and will be adapted as much as possible to each particular case study.

Finally, information received need to be organised, read and analysed. The synthesis on each case study will be produced at the end. Together with a cross case analysis.

201

CASE STUDY PROTOCOL

Topics

The following topics have to be addressed during the case study.

Quality of research

Focused on usefulness of output/results?

Attention paid to usefulness of output/results during the project (proposal selection, during project execution, in writing the reports).

Formal procedures to ensure existing scientific knowledge is used (yes/no), external validation by experts during the project (yes/no). Standards for data collection, synthesis used (yes/no).

Supervision with clear responsibilities during the course of the project? Continuity in supervision (same coordinator during the project).

6. Involvement of stakeholders

Stakeholders involved before, during and at the end of the project.

Before: defining and preparing the project.

During: execution of the project

End: Discussing draft report, draft recommendations, results.

Types of involvement: For example as external validator, by participating in the research.

Frequency and amount of time (approximately), per phase (Before, during, end).

Description of interaction with stakeholders (for example during presentations, etc.)

7. Impact intended

Explicitly formulated in advance? Yes/No. In what way.

Primary target audience(s) aimed for (factual)

Type of impact aimed for

o Instrumental (see table)

o Conceptual (see table)

o Symbolic (see table)

Hivon et al, 2005

202

Type of impact continued:

Pre-policy: problem described, causes and consequences studied, trend or spread traced.

Developing policy: Develop awareness, mobilize, reframe the issue

Policy enactment: advocacy, frame policy, pass policy/legislation.

Post-policy enactment: implementation, access & equity, sustain, change or abandon

Ottoson et al., 2009:

Outputs

Derived from Self report (see appendix 8). Checked with project coordinator for completeness.

8. Dissemination activities

Tailored design of the products? Tailored in the sense of ‘ready to use’ for stakeholders.

Developed ‘synthesis’ products? (Additional) products that summarize the findings on a specific topic, meant to diffuse knowledge and identify gaps in knowledge and stakeholders need for additional research.

Expanded contacts and working relationships with stakeholders? Specifically in dissemination activities.

Communicated findings to intermediaries? Intermediaries are people/organizations that can connect the researchers’ work to policy activity.

Made products easily obtainable? Easy to access, fast delivery, online available.

Communicated on the right time (monitoring policy developments)?

Is timing of dissemination activities adjusted to the needs of stakeholders. For example the policy cycle of stakeholders.

9. Stakeholders addressed by outputs

Type of stakeholders

Corresponding with research proposal and intended impact? To what extent output is conform intended output

10. Actual take up by stakeholders

Amount of downloads compared to other projects

Number of requests for information on this project by stakeholders

11. Contribution to the decision making process

Contribution to the decision making process described by project coordinator and stakeholders.

Use of reports/articles from project, channels stakeholders use to obtain information from KCE project

12. Research utilisation

Influenced by the findings? To what extent policy and/or practice is influenced by findings. - both direct (use) and indirect (e.g. changes in legislation).

Reasons for not using the findings from a project

A differentiation of types of reasons for not using the findings, derived from the collected data (Box see factors that influence the use of findings).

Box: Influential factors for use of findings: The channel of communication (influence of sources/information) The coherence of findings with prevailing clinical beliefs (do the findings require

‘out of the box’ thinking from professionals?) The quality and the quantity of the evidence The nature of the change (is the change easily implementable well fitting, easy

to learn/do, inexpensive) The impact of the change on key clinical branch points (do the findings meet

203

clinical needs of the professionals?) Weiss 2007

The following factors enhance the use of HTA in decision-making:

Timeliness (conducting research when needed/relevant and revising assessment when new data becomes available)

Appropriately communicating HTA findings to different decision-makers (multiple communication strategies to meet the disparate needs of different users)

Making clear the relationship between HTA findings and the decision-making processes (same findings may lead to different decisions (according to situation or actor), to some extent decisions can be predicted)

Drummond et al. 2008

13. Relevance for policy making / practice

Timely delivery? Yes/No

Reasons why it was/wasn’t timely according to the stakeholder.

Answering to information need? In what measure the project answers to the information need (very high, high, average, low very low). Reasons why information needs are / aren’t met.

Usable conclusions/recommendations (tailoring)? To what extent conclusions/recommendations are usable (very high, high, average, low very low). Why?

14. Reactions on finished projects from stakeholders

Feedback to KCE: positive of negative? Number of occasions where stakeholders gave feedback to KCE on this project. Nature of the feedback (positive / negative). Content of the feedback.

Comments in media: positive or negative? Number of occasions where feedback was given on KCE projects in the media. Nature of the feedback (positive / negative). Content of the feedback.

15. Judgment on responsiveness

Clear communication? Yes/No, Reasons

Timely responding? Yes/No. Reasons

Tailoring products to ‘customers’? Yes No. Reasons

Fulfilment of information needs? Yes/No. Reasons

16. Judgment on research quality

Competent researchers? very good, good, average, bad, very bad. Judged by stakeholders on case level. Explanation of judgment.

Good choice of methodology? very good, good, average, bad, very bad. Judged by stakeholders on case level. Explanation of judgment.

Good analysis of data? very good, good, average, bad, very bad. Judged by stakeholders on case level. Explanation of judgment.

Valid conclusions? very good, good, average, bad, very bad. Judged by stakeholders on case level. Explanation of judgment.

Management of case

One researcher is responsible for the case. He takes care that contributions of others meet the requirements and quality standards. This researcher is responsible for the desk research, interviews, synopsis and case analysis. Since more than one researcher will work on the case studies, the overall coordination of the case studies will be done by a coordinator.

Management of the data

Each case will be summarised in an Excel file (so 9 files). The first sheet of each file gives an introduction on the case (research subject, brief description of the content, budget, etc.). Sheet 2 to 14 summarise the data collected on each variable/topic, divided over the different data sources. Finally sheet 15 shows the overall results and conclusions per case. Every case will have its own folder with the Excel file, with for example background information (like published articles, regulations) and summaries of interviews.

204

APPENDIX 11: CASE STUDY QUESTIONNAIRES Content:

Checklist stakeholders on case level

Checklist principal researcher

Checklist Stakeholders

Interviewer to complete: How was the respondent involved as stakeholder during the research process? As:

Policy-maker

Target group or user (organized hospitals / clinical practitioners, insurer, patients, etc.)

Other interest group ((organized hospitals / clinical practitioners, insurer, patients, etc.))

Participant in executing the research (e.g. scientist)

Introduction: KCE impact study

Research voor Beleid: Independent research company

Case studies for more in-depth study of the KCE research process

Interview with important stakeholder, involved in report title …

The interview concerns the roles played by KCE, the researchers, and stakeholders, as well as the factors associated with utilization and impact of the results.

Research selection and inception

1. Were you or your organization involved in the submission of this research topic? How?

2. Were you as a stakeholder involved before the start of the research project? How: defining the project, preparing the project? If not, do you know why not?

3. How frequently were you involved before the start of the project?

4. Was sufficient attention paid to the quality and the utilization of the results at the start of the research project? How?

5. What would you say was the intended impact of the research project? Was this clear from the start?

6. Was the intended target audience (users) clear from the start? Was the intended target group involved in this decision? How?

Research process

1. In what ways was involvement of stakeholders organized? (e.g. e-mail, meetings, presentations)

2. How frequent was the interaction? How much time?

3. On what aspects (for example quality, usefulness) did you as a stakeholder contribute or comment, and how was your input taken into account?

4. Was the research process focused on the usefulness of the results and outputs of the study? How?

5. Was the role and responsibility of all the different actors (KCE project coordinator, researchers, stakeholders, validators, other experts etc.) in the process clear from the start?

6. Are there any other stakeholders that should also have been involved? Who? Why?

Finalization

1. Were you or other stakeholders involved in the writing of the report? How and to what extent? If not, do you think you should have been involved? Why?

205

2. Were you involved in discussing the:

Draft report? How (e.g. meeting, e-mail etc.)?

Draft recommendations? How (e.g. meeting, e-mail etc.)?

The results or findings? How (e.g. meeting, e-mail, presentation etc.)?

3. How frequent was the interaction? Time?

4. What feedback or comments did you provide to KCE about the report or recommendations? How were these taken into account?

5. Did you also give feedback or comments about the project in the media? On how many occasions? What was the nature (positive or negative) and content of the feedback?

Dissemination

1. Were the research outputs tailored to be “ready for utilisation” by you and/or other stakeholders? How?

2. To what extent were the conclusions or recommendations useable for your organization and (other) target users? (v.high, high, average, low, v. low)

3. Were other “synthesis” products developed or activities organized to suit stakeholders’ information needs? Which?

4. Were your organization and/or other stakeholders involved in the design of the outputs? How?

5. Were your organization and/or other stakeholders involved in the dissemination of the outputs or findings? How?

6. Were you involved on invitation by KCE, or did you propose your involvement yourself?

7. As far as you are aware, what attempts did the project team or KCE make to get the outputs across to potential users?

Seeking face-to-face contact with potential user(s)

Seeking telephone contact with potential user(s)

Sending personal e-mail to potential user(s)

Sending out mailing to interested people




Monitoring policy developments: communicating findings when they are relevant to current policy debates


Giving lecture/presentation

Appearance of researcher in media (newspaper, television, radio)

Other: ….

8. Are the outputs easily accessed? If not, how could this be improved?

Utilization of findings / recommendations

1. Was the final report issued timely to allow for conclusions or recommendations to be adopted and/or implemented? Reasons why (not)?

2. To what extent do the results answer to your information need? (Very high, high, average, low, very low). Reasons Why (not)?

3. Have you, your organization or other stakeholders utilized the research findings? How? OR: Why Not?

4. How do you rate the relevance of the research results of the projects for your organization (Very high, high, average, low, very low)? And for other stakeholders (scientific, medical practice, policy)?

5. How do you rate the feasibility/practicability of the recommendations from the project (Very high, high, average, low, very low)?

206

6. How do you rate the acceptability of the recommendations from the project (Very high, high, average, low, very low)?

Quality, please rate the following on scale: very good, good, average, bad, very bad)

1. Competence of the researchers? Please explain.

2. Choice of the methodology? Please explain.

3. Analysis of the data? Please explain.

4. Quality of the scientific underpinning of the findings? Please explain.

5. Validity of the conclusions? Please explain.

Impact

1. How have the research results or outputs impacted on you or your organization?

2. Has this project, according to you, in any way contributed to policy making in the field of health care or health care practice? How? (see self report q-naire) OR: Why Not? (factors)

3. Do you expect the findings to contribute in any way to health care policy and/or practice in the future?

4. Has this project, according to you, had an impact on the scientific community in any way? How? OR: Why Not? (factors)

5. Are there any problems that have (or may) arise(n) when trying to adopt the results or the recommendations?

6. What factors may have influenced the impact of the study? Positively and / or negatively?

Communication and coordination by KCE

1. Was the communication by KCE clear? Yes/No reasons?

2. Did KCE timely respond or communicate? Yes/No reasons?

3. Did the KCE project coordinator adequately see to keeping the focus of the study in terms of scope and outcome?

4. Do you consider the collaboration with KCE on this project as satisfactory?

Potential for improvement

1. How could the involvement of stakeholders in the KCE research process be improved, in each of the following stages:

Before the start of the project

During the process

Finalization

Dissemination

2. How could the impact of KCE research projects be further enhanced?

3. Do you have any further comments about the project or about KCE?

Checklist for Principal Researcher

Interviewer to complete: How was the respondent involved as stakeholder during the research process? As:

Policy-maker

Target group or user (organized hospitals / clinical practitioners, insurer, patients, etc.)

Other interest group ((organized hospitals / clinical practitioners, insurer, patients, etc.))

Participant in executing the research (e.g. scientist)

Introduction: KCE impact study

207

Research voor Beleid: Independent research company

Case studies for more in-depth study of the KCE research process

Interview with important stakeholder, involved in report title …

The interview concerns the roles played by KCE, the researchers, and stakeholders, as well as the factors associated with utilization and impact of the results.

Research selection and inception 1. In which part of the project have you participated: proposal

selection/project execution (methodology selection, formulating research question, data and analysis methods selection, project management)/interaction with potential users/reporting on findings/dissemination of results?

2. Was sufficient attention paid to the quality and the utilization of the results at the start of the research project? How?

3. What would you say was the intended impact of the research project? Was this clear from the start?

4. Was the intended target audience (users) clear from the start? Was the intended target group involved in this decision? How?

Research process 1. Has KCE project coordinator, on your opinion, clearly followed the main

objective of the study and intended results all over the study?

2. Was the frequency of interaction/communication with project coordinator sufficient?

3. Supervision of the project was done by the same project coordinator all along the project?

4. In your opinion, was the work on the project organized in line with KCE existing procedures for project management/for quality of research?

5. Is there any particular information/problems/achievements you would like to mention as regards to the organization of the project?

Involvement of potential users/stakeholders 1. Did the project coordinator stimulate the active involvement of potential

users at the start of the project/during the project/at the end of the project?

2. In what ways was involvement of stakeholders organized? (e.g. e-mail, meetings, presentations)

3. Have you had a direct contact with stakeholders? How frequent was the interaction? How much time?

4. Was the role and responsibility of all the different actors (KCE project coordinator, researchers, stakeholders, validators, other experts etc.) in the process clear from the start?

5. Are there any other stakeholders that should also have been involved? Who? Why?

Finalisation 1. Have you received any feedback on the research results from project

coordinator/stakeholders about the quality or usability of the study/findings?

2. What was the nature (positive or negative) and content of the feedback?

3. Have you integrated them in the final report?

4. Did you also give feedback or comments about the project?

Dissemination 1. Were the research outputs tailored to be “ready for utilization” by you

and/or other researcher?

2. On your opinion, to what extent were the conclusions or recommendations useable for different target users? (v.high, high, average, low, v. low)

3. Were other “synthesis” products developed or activities organized to suit stakeholders’ information needs? Which?

4. Were your involved in the dissemination of the outputs or findings? How?

208

5. What attempts did you/your research team did make to get the outputs across to potential user?

Seeking face-to-face contact with potential user(s)

Seeking telephone contact with potential user(s)

Sending personal e-mail to potential user(s)

Sending out mailing to interested people




Monitoring policy developments: communicating findings when they are relevant to current policy debates


Giving lecture/presentation

Appearance of researcher in media (newspaper, television, radio)

Other: ….

6. Did the project team actively involve any potential users in dissemination of outputs? Which and in what way?

7. Are the outputs easily accessed?

Quality

1. What is your judgment on the scientific quality of the study? (veryhigh, high, average, low, very low)

Utilization of findings / recommendations 1. Was the final report issued timely to allow for conclusions or

recommendations to be adopted and/or implemented? Reasons why (not)?

2. Have stakeholders utilized the research findings? How? Or Why Not?

3. How do you rate the feasibility/practicability of the recommendations from the project (Very high, high, average, low, very low)?

4. How do you rate the acceptability of the recommendations from the project (Very high, high, average, low, very low)?

Impact 1. Was the research process focused on the usefulness of the results and

outputs of the study? How?

2. How have the research results or outputs impacted different stakeholders?

3. Has this project, according to you, in any way contributed to policy making in the field of health care or health care practice? How? (see self report q-naire) OR: Why Not? (factors)

4. Has the findings of this project contributed in any way to health care policy and/or practice?

5. Has this project, according to you, had an impact on the scientific community in any way? How? OR: Why Not? (factors)

6. What factors may have influenced the impact of the study? Positively and / or negatively?

Potential for improvement

1. Do you consider necessary to improve the involvement of stakeholders in the KCE research process and if yes, how?

2. How could the impact of KCE research projects be further enhanced?

3. Do you have any further comments about the project or about KCE?

209

APPENDIX 12: CASE STUDY RESULTS Case nr: 52

{De Laet, 2007 #35}

70 {de Walcque, 2008 #38}

61 {Vlayen, 2007 #32}

60 {Van den Bruel, 2007 #36}

83 {De Gauquier, 2008 #39}

20 {Hulstaert, 2005 #33}

27 {Mathieu, 2006 #37}

68* {Maréchal, 2007 #40}

33 {Beutels, 2006 #34}

Year of publication

2006 2007 2007 2007 2008 2005 2006 2007 2006

Research field:

GCP HSR HTA GCP HSR HTA GCP HSR HTA

Type of impact sought

Instrumental Conceptual Instrumental Instrumental Conceptual Instrumental Instrumental Instrumental Instrumental

KCE estimated impact

Low Low Low Av. Av. Av. High High High

Download count

2249 1822 1738 2944 1654 3629 2769 751 3089

(Partly) Outsourced

Initially, later not.

Yes No No Initially, later not.

Yes Yes Yes Yes

Hypotheses: Case 52 Case 70 Case 61 Case 60 Case 83 Case 20 Case 27 Case 68 Case 33 1. Transparent, careful proposal selection procedure… +/-

(TPF, Domus Medica)

+/- (TPF, sc.

Inst.)

- (QRIZIV)

+/- (TPF, RIZIV)

- (QRIZIV)

- (QRIZIV)

+/- (RIZIV,

FOD)

- QM**

- QM**

2. Specific and (stakeholder-) relevant goal formulated in advance…

+/- Initially +, later -

+/- + +/- +/- ++ +/- + +

3. Explicit goals for user engagement and dissemination at start of project…

+ /-initially +, later -

+/- Initially +;

no budget

- - - +/- - - -

4. High quality research processes (project management and methodology)…

- +/- +/- - - ++ ++ ++ +

210

Hypotheses: Case 52 Case 70 Case 61 Case 60 Case 83 Case 20 Case 27 Case 68 Case 33 5. Actively engaging relevant stakeholders during the research project…

- + +/- - - + ++ +/- +/-

6. Relationship between findings and the decision-making processes clear/clarified…

+/- - + +/- +/- + + + +

7. Actively performing dissemination activities (based on dissemination strategy)…

- +/- - -

- Partly too late

+ - - +/-

8. Identifying stakeholders as possible user(s) and tailoring product to stakeholder needs…

- +/- +/- +/- +/- +/- +/- + +/-

9. Incorporating lessons from structural feedback by (potential) end-users in research process…

+/- - +/- - +/- +/- + +/- +/-

10. A positive image of KCE (e.g. on independence, responsiveness and research quality) amongst stakeholders

- +/- +/- - +/- + +/-

+/- +/-

Score 25 29 29 24 27 37 38 32 29

Impact Case 52 Case 70 Case 61 Case 60 Case 83 Case 20 Case 27 Case 68 Case 33 Impact on Policy +/-

- +/-

-

+/-

Impact (compared to other KCE projects)

Impact on practice - - - - -

++ one stake-holder: -

Policy: + + Practice: -

Present impact: +/- Future social impact ++

+ one stake-holder: -

Report 68 is part of a larger project consisting of five phases; most information applies to all stages of the report.{Maréchal, 2007 #40; Denuit, 2005 #57; MARSH : Amsoms, 2004 #60; Maréchal, 2006 #61; Yerna, 2009 #62} ** QM indicates an “urgent question by the minister”, which does not go through KCE’s formal proposal prioritization and selection procedure; receives score 2 if no other indication is provided. 1 is very low (- -), 2 low (-), 3 average (+/-), 4 high (+), 5 very high (++)

211

APPENDIX 13: COMPARISON FRAMEWORK INTERNATIONAL AGENCIES

For every agency a brief descriptive introduction will be written on:

Mission and goals

Target audience (including stakeholders)

Funding of the agency

Staffing

In general respondents of the agencies will be asked what challenges they experience in achieving impact, what they do to increase impact and what they think is successful. If certain strategies seem successful, is there any evidence?

More specific the comparison framework is derived from the theoretical framework used for the KCE impact study (Research process, dissemination and contribution to decision making process).

Elements we use for comparison are:

I The research process: Origin of the researchiiii

Proposal selection and research prioritization

o Transparent?

o Careful?

o Policy- or practice orientated?

Involvement of stakeholders during research processjjjj

Before, during and at the end of the project?

Different types of involvement?

High degree of involvement?

Impact intendedkkkk

Explicitly formulated in advance?

Primary target audience(s) aimed for

II Dissemination

Dissemination activities performedllll

Tailored design of the products?

Developed ‘synthesis’ products?

Expanded contacts and working relationships with stakeholders?

Communicated findings to intermediaries?

Made products easily obtainable?

Communicated on the right time (monitoring policy developments)?

Stakeholders addressed by outputsmmmm

Type of stakeholders

Corresponding with research proposal and intended impact?

iiii Derived from SDO{, #16}; Hanney et al. 2003{Hanney, 2003 #10} jjjj Derived from Meagher et al. 2008{Meagher, 2008 #11}; Drummond et al. 2008{Drummond, 2008 #7} kkkk Partly derived from Meagher et al. 2008{Meagher, 2008 #11} llll Derived from McBride et al. 2008{McBride, 2008 #5} mmmm Derived from SDO{, #16}

212

Engagement of stakeholders in outputsnnnn

Stakeholders involved in developing outputs?

Stakeholders involved in disseminating outputs?

Actual take-up of outputsoooo

Is it monitored? How? And how is it used.

III Contribution to the decision-making process

Familiarity with research projectspppp

Does the agency monitor if specific stakeholders have knowledge of projects?

Research utilizationqqqq

Does the agency monitor:

o If policy makers are influenced by the findings?

o The actual use of findings?

o Type of use

o Reasons for not using the findings from a project

o If so: what are outcomes of the monitoring?

Not all elements of the theoretical framework are used for the international comparison:

Quality, we are not going to compare the quality of different institutes.

Dissemination numbers of output (publications, reports, presentations),

Number of proposals submitted

Actual take up of findings (not so much in numbers, but we will ask if the

Take up of findings is monitored, how its monitored and used).

General reputation

nnnn Derived from SDO{, #16} oooo This is not derived from the literature; however it it as a valuable addition to the other indicators/variables

(and is relatively simple to look up for KCE) pppp Derived from SDO{, #16}; Hanney et al. 2003{Hanney, 2003 #10} qqqq Derived from Hanney et al. 2003{Hanney, 2003 #10}

213

APPENDIX 14: SELECTION OF AGENCIES Agency/rating European Mission Age Domain

s

Evaluation Conclusion

1. Haute autorité de Sante

(HAS)

France Independent public body, high

degree of scientific rigour. Activities:

Assessments of drugs, medical

devices and procedures as well as

publication of guidelines and

accreditation.

August

2004

HTA,

GCP,

HSR

Yes, with IQWiG

&

NICE{Chalkidou,

2009 #13}

Suitable

2. Institut für Qualität and

Wirtschaftlichkeit im

Gesundheitswesen (IQWiG)

Germany Independent scientific institute that

evaluates quality and efficiency of

health care and services (evaluation

of pharmaceuticals, services,

procedures and clinical practice

guidelines).

June 2004 HTA,

GCP,

HSR

Yes, see 1. (HAS) Suitable

3. The National Institute for

Health and Clinical

Excellence (NICE)

Great-Britain Independent organization providing

national guidance on promotion of

good health and prevention and

treatment of ill health (technology

appraisals, procedures guidance,

clinical guidelines)

1999 /

2005rrrr

GCP Yes, see 1. (HAS) Suitable for GCP

4. The Netherlands

Organisation for Health

Research and Development

(ZonMw)

Netherlands ZonMw, an intermediary

organisation contributes to the

improvement of prevention, care and

health by stimulating and financing

fundamental and practice-oriented

research, development and

implementation.

2001 HSR,

HTA,

GCP

(interme

diary)

Yes{Kleisen en

Partners, 2004

#14}

Mission is different,

ZonMw is mainly an

intermediary agency

5. NIVEL Netherlands NIVEL, the Netherlands Institute for

Health Services Research,

contributes to the body of scientific

knowledge about the provision and

use of health care services.

1965 HSR ? Not clear if

evaluation is available

and age of

organisation is much

older.

6. Danish Centre for Health

Technology Assessment

(entity of National board of

Health) - DACEHTA

Denmark Research contributing to quality,

efficiency and quality of care by

means of solid science-based

information used to improve

planning and prioritising in the health

care system.

2001 HTA Yesssss Suitable, but other

agencies have a

stronger reputation.

7. NHS national Institute for

Health research (NIHR)

Great-Britain Leading edge research about the

effectiveness, costs and broader

impact of health care treatments en

tests for planners, providers and

receivers of care

April

2006

(compara

-tively

new): but

previousl

y NHS

R&D

HSR,

HTA

Yes of the NHS

R & D

programmes up

to 2006{Hanney,

2003

#10},{Raftery,

2009 #15},{,

#16}

suitable

8. National Health

Board

Ireland Key funder of health research,

from basic biomedical and

1988

(more

HSR,

HTA

Case study

evaluations of

Suitable, but

evaluation is only

rrrr Set up in 1999 and merged with the Health Development Agency in 2005. ssss National Board of Health (2003). External Evaluation of DACEHTA

214

clinical research through to

Health Services Research

(HSR), public health and

primary care research, in

order to improve health and

advance the knowledge

economy.

establis

hed)

specific

projects{Naso

n, 2008 #23}

on case level

9- Healthcare

technology Evaluation

Agency (part of

Instituto de Salud

Carlos III)

Spain Meet the consulting needs of

the National Health Care

Systems by objective

assessment of health, social,

ethical, organizational and

economic impact of medical

healthcare techniques en

procedures.

1994 HTA ? Suitable but not

clear if an

evaluation is

available.

10- Canadian Health

Service Research

Foundation / Fondation

Canadienne de la

recherce sur les

service de santé

Canada

(not

European)

The Foundation brings

researchers and decision

makers together to create and

apply knowledge to improve

health services for Canadians.

The Foundation is an

independent, not-for-profit

corporation, established with

endowed funds from the

federal government and its

agencies.

1997 HSR Yes various

internal audits

Suitable, but not

European.

Innovative on

interaction with

stakeholders and

user groups.

11 SBU Sweden SBU has the mandate of the

Swedish Government to

comprehensively assess

healthcare technology from

medical, economic, ethical,

and social standpoints

1992 HTA Yes on project

level

? on

institutional

level

Suitable, strong

reputation.

12. Catalan Agency for

HTA

Spain The CAHTA provides information

based on scientific evidence and

the analysis of the health

scenario, its endpoint being to

ensure that the introduction,

adoption and utilisation of

medical technologies is carried

out according to criteria of

efficacy, safety, effectiveness

and efficiency which have been

scientifically demonstrated, as

well as offering support to

decision-making at different

levels of the health system.

1994 HTA ? Suitable, but no

evaluation.

215

Long list of Agencies

I Danish Centre for Health Technology Assessment (entity of National Board of Health)

The key aims of the Danish Centre for Health Technology Assessment (DACEHTA) include carrying out health technology assessments with the aim of improving quality, standards and value for money. It is also an objective to integrate HTA-principles into the running and planning of the public health service at all levels.

DACEHTA primarily targets health professionals and decision-makers at all levels as well as related research communities.

DACEHTA is charged with employing those activities and methods that will contribute most effectively to quality, efficiency and standards of care in the Danish health service.

The centre is situated as a entity within the National Board of Health.

II NHS National Institute for Health Research (NIHR) GB

The National Institute for Health Research is established as a part of the Government's strategy, ‘Best Research for Best Health'. Aim (together with partners) is to establishing the NHS as an internationally recognised centre of research excellence.

The goal of the NIHR is to create a health research system in which the NHS supports outstanding individuals, working in world class facilities, conducting leading edge research focused on the needs of patients and the public.

The HTA programme is part of the NIHR. It produces independent research information about the effectiveness, costs and broader impact of healthcare treatments and tests for those who plan, provide or receive care in the NHS.

The programme has extensive consultation as part of research prioritisation, requires researchers to propose a dissemination plan within grant application, promotes widespread dissemination of multiple products including Health Technology Assessment monographs freely available in UK and downloadable from the web, and has links to National Institute for Clinical Excellence (NICE). Other parts of NHS R&D programme have commissioned implementation research.

III The Netherlands Organisation for Health Research and Development (ZonMw)

ZonMw is a national organisation that promotes quality and innovation in the field of health research and health care, initiating and fostering new developments. ZonMw also actively promotes knowledge transfer and implementation, ensuring knowledge is exchanged between all relevant stakeholders (health researchers, health professionals, patients/consumers and the general public). This in turn facilitates the structured implementation of newly developed knowledge in the health care system and guarantees emerging health care issues a place on the research agenda. In short, one of ZonMw’s key challenges is to facilitate innovation and inspiration among the various stakeholders along a single ‘knowledge continuum’: from basic, strategic and applied research to daily health care services. ZonMw acts as an intermediary between policy, research and practice. Evidence-based medicine, relevance-based science, information-based policy - it is ZonMw’s firm belief that only the active exchange of ideas, knowledge and experience can allow the innovation cycle to flourish and help create a cutting-edge society.

The majority of ZonMw’s commissions come from the Ministry of Health, Welfare and Sport (VWS) and the Netherlands Organisation for Scientific Research (NWO). The Ministry’s main concern is to contribute to public health, including prevention and health care services. NWO is a non-governmental organisation concerned with fundamental and strategic research. As the national research council, it plays a key role in the development of science, technology and culture in the Netherlands.

In terms of health research and development - and thus in its relations with ZonMw - its main interest lies in contributing to cure, care and prevention by gaining a better understanding of disease and its underlying processes, and supporting research related to medical or health technology assessment.

216

IV Haute autorité de Santé (HAS) France

The Haute Autorité de santé (HAS) - or French National Authority for Health - was set up by the French government in August 2004 in order to bring together under a single roof a number of activities designed to improve the quality of patient care and to guarantee equity within the healthcare system. HAS activities are diverse. They range from assessment of drugs, medical devices, and procedures to publication of guidelines to accreditation of healthcare organisations and certification of doctors. All are based on rigorously acquired scientific expertise. Training in quality issues and information provision are also key components of its work programme. HAS is not a government body. It is an independent public body with financial autonomy. It is mandated by law to carry out specific missions on which it reports to Government and Parliament. It liaises closely with government health agencies, national health insurance funds, research organisms, unions of healthcare professionals, and patients' representatives. HAS has been built on 3 founding principles: a very broad field of action, which means that it can compare a range of healthcare initiatives; a high degree of scientific rigour; and independence

VI Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG) Germany

The Institute for Quality and Efficiency in Health Care is an independent scientific institute that evaluates the quality and efficiency of health care. The Institute investigates what therapeutic and diagnostic services are feasible and meaningful, and communicates its findings to the health care professions, patients and the general public.

On 1 June 2004, the Institute was established in the course of the Health Care Reform as an institution of the Foundation for Quality and Efficiency in Health Care, to undertake commissions from the Federal Joint Committee and the Federal Ministry of Health.

The Institute is financed by a levy on inpatient and outpatient healthcare services, which are mainly reimbursed by the statutory health insurance (SHI) funds.

The Institute's tasks include the evaluation of pharmaceuticals, surgical procedures, diagnostic tests, clinical practice guidelines and aspects of disease management programmes, following the principles of evidence-based medicine. It also publishes health information for patients and the general public. The Institute's primary goal is to contribute to improvements in health care in Germany.

VII The National Institute for Health and Clinical Excellence (NICE, GB)

NICE is the independent organisation responsible for providing national guidance on the promotion of good health and the prevention and treatment of ill health. NICE guidance is developed by a number of independent advisory groups made up of health professionals, those working in the NHS, patients, their carers and the public.

The Board and senior management team set the Institute’s strategic direction and oversee delivery, provide financial stewardship and ensure corporate governance. NICE is structured into centres and directorates. Centres are responsible for delivering guidance. Directorates provide cross-Institute support for the centres, alongside other corporate responsibilities.

NICE produces guidance on public health, health technologies and clinical practice. Each guidance area is the responsibility of one of our three centres of excellence:

Centre for Public Health Excellence. This develops public health guidance on the promotion of good health and the prevention of ill health.

Centre for Health Technology Evaluation develops technology appraisals and interventional procedures guidance.

Centre for Clinical PracticeThe Centre for Clinical Practice develops clinical guidelines. These are recommendations, based on the best available evidence, on the appropriate treatment and care of people with specific diseases and conditions.

217

VIII Healthcare Technology Evaluation Agency (part of Instituto de Salud Carlos III, Spain)

The Healthcare Technology Evaluation Agency was created in 1994 (RD 1415/1994) as part of the Carlos III Healthcare Institute to meet the consulting needs of the National Healthcare System for defining its Healthcare Policies in the major line of socially advanced Healthcare Systems.

To achieve this, it must offer objective assessment of the healthcare, social, ethical, organizational, and economic impact of medical-healthcare techniques and procedures to bolster the decisions taken by authorities and other healthcare agents, either through

the systemised introduction of new technologies in clinical practice,

the definition of the appropriate use of already established technologies,

or the organization of health services.

This general work supporting the decision-making processes on healthcare services was consolidated and specified by Law 16/2003, enacted on 28 May, Cohesion and Quality of the National Healthcare System which expressly binds AETS to the procedure for updating the Services Portfolio.

218

APPENDIX 15: REPORTS ON FOREIGN AGENCIES

THE HAUTE AUTORITÉ DE SANTÉ (HAS)

Draft

Zoetermeer, September 29, 2009

1. Introduction The Haute Autorité de santé (HAS) - or French National Authority for Health - was set up by the French government in August 2004 in order to bring together under a single roof a number of activities designed to improve the quality of patient care and to guarantee equity within the healthcare system. HAS activities are diverse. They range from assessment of drugs, medical devices, and procedures to publication of guidelines to accreditation of healthcare organizations and certification of doctors. All are based on rigorously acquired scientific expertise. Training in quality issues and information provision are also key components of its work programme.

HAS is not a government body. It is an independent public body with financial autonomy. It is mandated by law to carry out specific missions on which it reports to Government and Parliament. It liaises closely with government health agencies, national health insurance funds, research organisms, unions of healthcare professionals, and patients' representatives.

HAS has been built on 3 founding principles: a very broad field of action, which means that it can compare a range of healthcare initiatives; a high degree of scientific rigor; and independence.

1.1. Mission and goals General HAS mission is to increase the quality of French healthcare by accompanying health care practitioners in the continuous process of the clinical health care improvement for the patients and by contributing with its opinion in the public decision-making process in order to optimize the management of the reimbursable goods and services.

HAS mission is to increase health care for the patients and potential users on individual level, by taking care of each patient, but also on the collective level, by contributing into the consistency of the fair health care system.

The goals of HAS are:

To provide health authorities with the information needed to make decisions on the reimbursement of medical products and services, by

o Giving opinion on the medical utility of medicaments, medical dispositions and professional acts taken in charge by insurance companies

o Giving opinion on chronic (long-term) conditions

o Giving opinion on targeted practice agreements

o Conducting medico – economic evaluations and evaluations in public health care.

To encourage good practices and the proper use of health services by professionals and users,by

o Producing clinical practice guidelines and standards

o Producing "how to" guides

To improve quality of care in health care organisations (HCOs) and in general medical practice, by

o Accreditation of health care organisations (HCOs)

o Continuing professional development (CPD)

o Certification of doctors and medical teams

219

To provide information for the public and generally improve the quality of medical information, by

o Methods and tools for quality improvement

o Information on nosocomial infections

o Improving the quality of medical information

Develop the concentration and collaboration with the main players of the health care system in France and abroad, by

o Elaborating join research program

o Development of international relations

o Development of cooperation with collegiums of professionals and scientific societies

o Development of cooperation with patient associations and health care users

1.2. Target audience (including stakeholders) Target audience of the HAS include health care professionals, health care users and Governmental institutions.

1.3. Funding The annual budget is 60 million euros. HAS' revenue comes from taxes on promotional spending by drug companies (33.2%), National Health Insurance (31%), state funding (14%), HCO accreditation fees (14.6%), fees for assessing applications for inclusion on reimbursement lists (7%), miscellaneous (0.6%)

In 2008, HAS budget was 67 Meuro. Up to 40% was financed by subventions (9% from Government, 31% insurance companies), para fiscal taxes – 35% and 25% of the budget was financed by the inventory funds.

1.4. Staffing Located in Paris, HAS is staffed by 400 permanent researches who have, in majority, scientific background (engineers, doctors, dentists, a lot of different profiles).

Over 3000 external experts are collaborating with HAS on different topics during the research. The network is thoroughly chosen and includes the leading experts in all HAS research fields and associated areas.

400 permanent staff including health professionals (GPs, specialists, pharmacists, dentists, nurses, physiotherapists…) and health economists,

a pool of 3000 practising professionals (hospital and non-hospital practice) who collaborate with HAS. This includes regional ambassadors who diffuse HAS reports and know-how, members of working groups and, in particular, two networks,

a network of 35 regional facilitators who help doctors implement the compulsory continuing professional development (CPD) initiatives in continuous quality development (CQI),

a network of 775 surveyors who perform accreditation visits to public and private healthcare organisations.

1.5. Organisation model The HAS Board provides governance. Each Board member heads a specialist Committee and is responsible for a specific mission or specific aspects of a mission. The organisation is divided into several divisions which implement the strategic work-plan conceived by the Board.

The organisation comprises:

5 divisions divided into departments

two external networks of health professionals and experts:

220

o a network of regional facilitators who help implement continuing professional development

o a network of surveyors who perform accreditation visits to HCOs

the members of the Specialist Committees.

Scientific Board (Сollège in French) execute quality control. It is composed of 8 experts (including the President), representing key authorities in the HAS specialization domains.

Each Board member, except the Chair, heads a Specialist Committee:

Transparency Committee

4. Assessment of medical devices and technologies

5. Assessment of diagnostic and therapeutic procedures

6. Health care cover for chronic conditions

7. Assessment of healthcare strategies

8. Accreditation of healthcare organisations

9. Medical information quality and dissemination

Responsibility is shared among Board members for several activities: continuing professional development (CPD) focussing on continuous quality improvement; good practice agreements; targeted practice agreements; and any other activity that does not fall within the scope of a Committee.

Members of College are assigned by the French President for 6 years. They are under the patronage of the Health Ministry. They are not politicians, but professionals. In the head of the College is a Doctor, other members are, for example, doctors, economists directors of hospitals.

Validation committee exists: in the beginning of the work process it is contributing in the development of the framework for the studies; in the final stages it’s task is to evaluate reports and recommendations.

2. Research process 2.1. Origin of the research Types of organisations that submit proposals

Everybody can submit the proposal for the project for the project. The main groups of HAS clients are:

Ministry

Health professionals

Health care users Brief description of the way research is prioritised

The number of proposals submitted to HAS is higher that they can realize, that is why, for the projects proposed by the health care professionals and users, the

221

formal selection and prioritization procedures exist. Projects that are coming from the Ministry or Government structures follow another procedure. Ministry make an official pre-submission of the project and afterwards, together with HAS, define the final research question and final study demand.

For the projects submitted by the health care professionals and users, three criteria are important:

pertinence of the topic – if the topic is really important and needed;

feasibility of the project - if they have available resources internally to execute the project;

if the project corresponds HAS and public authority priorities.

The projects are afterwards evaluated if they are corresponding to these criteria and a score is calculated. That is by prioritization of the score that the projects are chosen.

The notion of feasibility is particularly important within the project selection procedure. When the projects are submitted, the demand is thoroughly analyzed in order to determine if the project is feasible for HAS or not : which studies already exist and if there is enough of the internal capacity to satisfy this demand.

Brief description of the way (procedures) the research projects are selected

Mission of HAS is to improve the quality of Belgian healthcare assisting health care professionals and decision – makers in their process. So, first of all, the priority is given to the projects coming from the Ministry of Health and Sports. Thereby the working priorities of HAS are determined in correspondence with the policy agenda of the health-care decision makers. This part of the projects arrives following “one-window principle”. Thereafter, during 80 days HAS need to give opinion on stated question. These projects are not subject of selection procedure.

Secondly, all the submitted proposals from health care professionals and users arrive centrally to one contact person within HAS. Than all the proposals are send to the concerned departments/directions. These departments make the first selection, they check the feasibility of the projects. College (see organigram) is making final decision if the project will be executed or not. The overall process is transversal.

Thirdly, HAS participate in the evaluation of the medicaments: 10% of medicaments are requested to be obligatory evaluated by HAS before being reimbursed.

Finally, the part of the projects are determined internally within HAS, following their own priorities. There is no described formal procedures for the selection process. This in elaboration now. And even though the new procedures were introduced in 2008, no document describing all the procedures was produced.

2.2. Involvement of stakeholders during research process Involvement of the stakeholders depends on the project: sometimes that is a simple research procedures, sometimes projects involve working groups and reading groups (after reaction from HAS). Normally HAS does literature review, than working group gives their opinion and final opinion is elaborated by HAS.

The stakeholders are involved in all the stages of the project, the level of participation depends a lot on the concrete project. The general principle is to involve the stakeholders in each project.

Stakeholders are involved through expert groups. In the pool of experts are included the representatives of the professionals from the health care sector. Idea is also to involve the special College which will provide new experts each time. Currently they are using the services of the Good practice college, which regroup different experts from different fields. For each expert HAS is each time checking if conflict of interest can occur. Another possibility to involve experts is through the scientific societies.

Health care professionals are also involved through the working groups. The choice of experts is done according to the following criteria: multidisciplinary, geographical variation, experience, political views, etc. They search to have a representative opinion.

222

The involvement of patients depends on a subject. Inside HAS there is a special Representative Commission to maintain the good relations with the patients and their good implication into the process.

2.3. Impact intended The impact is explicitly formulated in advance. During the submission procedure the client need to fill-in a special form. Intended impact and measures of this impact are the questions, which considered as a very important selection criteria. The submission form includes as well a question about target audience of the project. In reality the definition of the targeted audience takes place more spontaneously.

3. Dissemination 3.1. Dissemination activities performed Products available

Dissemination of the results is very important aspect of the HAS activity. The improvement of dissemination activities is seen as one of the priority actions in the upcoming three years.

Concrete activities are:

Press releases

Monthly letter for the health care professionals (actuality topics and professional items)

Atonement list (professionals are subscribed and they receive notifications about the reports published)

Electronic information letters

Annual reports available on line

They buy place in press to publish information

Each year they conduct a conference during 2 days, covering different topics and which normally get together 1000 health care professionals

HAS elaborated special guideline, available on line, which assist clients in elaboration of professional recommendations for CGP.

In order to promote international cooperation HAS actively take part in international conferences. For example, during 2007 HAS took part in more than 82 international event, presenting the results of HAS reports in 47 cases.

Within International dimension, HAS participate in the following international networks and societies:

Guidelines International network

International Network of Agencies for Health Technology Assessment

International society for quality in health care Involvement of stakeholders in dissemination

Stakeholders are not systematically and directly involved in the dissemination of products but do take their own initiatives to spread information.

3.2. Actual take-up of outputs Monitoring of the actual take-up of outputs

There is no official monitoring of the reports follow-up. It really depends on cases. The level of usage depends a lot on the appropriation of the recommendations by the client and public. Generally its very hard to follow the usability of the recommendations.

The first effort is made to make recommendations readable and written in the language adapted to the target audience. The objective is that the results of the studies are later on used by other professionals, that is why they need to be updated to different readers.

223

Very often HAS receive the request of additional information about the projects. As soon as diffusion channels for the results are very different, they do not monitor them.

4. Contribution to the decision-making process 4.1. Familiarity with research projects

Independent consultancy conducted a study for HAS - the knowledge about HAS in the health care sector. The objective was to know if professional have a knowledge and memory about HAS work.

4.2. Research utilization HAS is not monitoring how many times reports are cited, even thought that is possible to do.

When the project is finished they just start new project. They are currently thinking how to improve the impact. Generally the level of the usage of the HAS results by the public decision-makers is very high. They do not have precise statistics. Projects that involve public sector are difficult to judged for now, because the implementation period is very long.

5. Conclusions Main challenges in achieving a high level of impact.

All the major challenges and actions to overcome them are described in the document “Project 2009 – 2011”. This is a document presenting major strategic development objectives of HAS for upcoming 3 years.

Project 2009-2011 determines three main strategic priorities of HAS for the 2009 -2011. The main items of the project are presented below, some of which, in more details.

5.1. Put the quality in the center of the individual practice and behavior

Health Care professional – high quality actor

Together with the health care professionals improve the actions increasing the quality of the health care

Enlarge the actions aimed to improve the practical experiences

Develop the techniques promoting coherence in the care

Reinforce the approach of partnership and delegation with professional organizations

Support the development of Good Practice Colleges for each medical specialization. HAS support introduction and development of the network of federal structures of specialists for the professional practice, or “Colleges des bonnes pratiques”, for which it wants to delegate some projects and the development of the approaches which aim to improve the quality of the professional practice. These structures can also be in charge of the communication and the dissemination of the recommendations to the professionals and scientific communities. They can do consultations on the technical questions or definition of the priority topics for work with the targeted specialists, as well as some questionnaires design and implementation and impact assessments. They also can be regrouped in the thematic networks.

Support and acknowledge the work performed on the initiative of the professionals.

Improve the information products for health care professionals

The portfolio of the information products which HAS provides to the health care professionals is extensive, but its coherence and conformity with the daily practice need to progress.

224

HAS inform health care professionals about good usage (via avis of the Transparence commission and fiches “Good usage of the medicaments”), good health care practices, about therapeutic strategies (via recommendations and fiches). This is important to improve the quality of the health care: synergy of the information disseminated is an important factor of its appropriation and the further implementation into the practice. That is why HAS would like to develop more it information products and in more coherent way. All the information products must be more available and adapted to the needs of the professionals.

Contribute into the coherence of the HAS products and of the public information on the medicaments.

Develop information about the good usage of the medicaments and health technologies

Provide professionals with more operational information on health care practices. Information on the good practice need to be more adapted to the expectations of the professionals in order to increase thee impact on their daily practices. That is why HAS would like to redefine techniques which it provide to the professionals, in order to make them more accessible, more easy to memorize and integrate in daily practices.

Introduction of the specific competence of medical editing (editing documents on medical topics) is seen as an advantage to improve the documents and make them more adapted to the user needs.

Support the good practices by improving the quality of the information technologies. Certification of the logiciels; Follow up of the information technologies available. In order to improve the accessibility in internet and other information technologies, HAS will privilege the conversion in electronic form of its own products (reports, recommendations). Two main requests:

o to develop specific knowledge of electronic production of the recommendations, opinions, practical guidelines to facilitate their utilization

o favorise the dissemination of HAS products, in the form of “reminders” and documents easily accessible via personal assistants.

Promote the best practices via improvement of the quality of information technologies

Patient - the high quality actor

Involve the patient associations in the policy of quality improvement

Provide the patient access to the information about the quality of the healthcare and include him in the management of his desease

Develop the information for the patient

Make, from the patient who has a chronic disease, an actor of his health, especially with the help of therapeutical education

Promote the good usage of internet

Develop information technologies about the quality of the healthcare offer

5.2. Put the quality in the center of the collective choices Act for the active management of the reimbursable products and services

Promote more interdisciplinary evaluation Develop medico-economical evaluation

Develop global evaluation of the different dimensions of services provided to the society

Promote more operational evaluation, which will serve the public decision process

Detect, evaluate, follow up the innovative technologies

Develop post-inscription studies

225

Develop the evaluation methods for the relative efficiency

Improve the evaluation of the medical dispositives Act for the active management of the reimbursable products and services

Develop external evaluation of the quality of the health care offer Develop certification

Extend the external evaluation of the quality

Develop quality indicators

Promote the security of the health care

Support prospective thoughts about the evolution of the health care offer

Reinforce the role of HAS in the process of evaluation of the actions and programs of the public healthcare

Determine the pertinent tendencies in the organisation of the healthcare system in order to evaluate them

Better take in consideration regional and territorial dimension

5.3. Reinforce the performance, the transparency and the expertise of the institution

Reinforce the quality of the expertise

Confirm the scientific quality of the products Start the methodological reference system in the field of health

evaluation and improvement of practices

Develop the research and put forward scientific productions of HAS

Reinforce international cooperation

Satisfy the deontology challenges

Better valorize the experts Reinforce the transparency

Reinforce the concentration Reinforce the dialogue of HAS with its environment

Reinforce reporting and budgetary dialogues

Guarantee concentrated and transparent programming

Develop public consultations

Make a progress in the measurement and readability of the impact of the actions

Develop the procedure of the external evaluation. Each year HAS present publicly an activity report, in order to assure transparency of the public resources. By its own initiative HAS also decided to go through external evaluation procedure, in order to answer the same requests that it asks from the structures that it is evaluating.

Reinforce impact measurement. HAS would like to evaluate the performance of its actions. Some indicators of impact assessment are already used by HAS. This does not concern all HAS activities but only some.

Reinforce the pertinence of the projects for the potential users

Reduce the delays of the production of the reports for the public decision-makers

Follow up the dynamics in the reduction of delays for the “rendu des avis”

Develop the procedure of the short response

Develop actualization of the work process

226

Improve the accessibility of the HAS products for the health care professionals and patients

Organize the dissemination of publications

Promote the accessibility of the information on the website

Improve the notoriety of the institution in order to increase the appropriation of its messages

Improve the performance

Improve the performance of governance

Conduct the policy of externalization and delegation of the work in order improve the quality

Promote sustainable development

Valorize the professional development of agents Reinforce the human resource management

Better to know each other and better to share the information

Develop the management control and optimize budget procedures Develop management control for the internal needs

Optimize budget procedures

Reinforce the offer of the services for the better efficiency Develop information systems in order to improve the quality

Optimize the management and the conservations of the key documents

Reinforce efficiency of the buying procedures on the formal markets

Guarantee the follow up of the Project 2009 – 2011 objectives

If certain strategies are successful, is there any evidence?

Currently its only the first year of the Project implementation. The main results will be seen later on, after the main measures are implemented.

227

THE SWEDISH COUNCIL ON TECHNOLOGY ASSESSMENT IN HEALTH CARE (SBU)

Projectnumber: b3561

The responsibility for the contents of this report lies with Research voor Beleid. Quoting of numbers and/or text as an explanation or support in papers, essays and books is permitted only when the source is clearly mentioned. No part of this publication may be copied and/or published in any form or by any means, or stored in a retrieval system, without the prior written permission of Research voor Beleid. Research voor Beleid does not accept responsibility for printing errors and/or other imperfections.

Draft


1. Introduction The Swedish Council on Technology Assessment in Health Care is known internationally by its Swedish acronym, SBU. SBU was established in 1987, initially as an agency under the Swedish Government Offices, but in 1992, SBU was commissioned as an independent public authority for the critical evaluation of methods used to prevent, diagnose, and treat health problems. As a forerunner in the field, SBU has is on the leading edge of health technology assessment (HTA) for more than 15 years and has been instrumental in promoting scientific assessment in Sweden and abroad.

1.1. Mission and goals SBU has the mandate of the Swedish Government to comprehensively assess healthcare technology from medical, economic, ethical, and social standpoints and to compile scientifically based assessment reports to support decision making in health care.

Reports prepared by SBU present scientific evidence on the benefits, risks, and costs associated with different interventions. SBU identifies methods that offer the greatest benefits and the least risk, focusing on the most efficient ways to allocate healthcare resources. However, SBU also identifies methods currently in use that provide no benefits, have not been assessed, or are not cost effective.

SBU organizes assessments on a project basis. A multidisciplinary team, called a Project Group, is recruited for each project. Project groups consist of leading experts from Sweden and abroad, representing various fields of clinical practice and scientific research. These experts carry out comprehensive and thorough assessments by systematically searching, selecting, reviewing, and evaluating research findings from around the world. SBU does not conduct original research, but collects, reviews, and evaluates available research findings. If an assessment addresses a broad subject area – for example, back pain, substance abuse, obesity, or caries – the process can take several years. However, projects that deal with single interventions may be completed in much less time. Many project groups decide to survey the methods currently used by health services and compare clinical practice with the scientific evidence – do they coincide, or can current practices be improved? Project groups also analyze economic and ethical aspects.

SBU currently publishes three series of assessment reports:

Yellow Reports – assessments carried out by SBU project groups. These comprehensive reports are based on systematic reviews of the existing body of scientific literature in a subject area. External experts and the SBU Scientific Advisory Committee review the manuscripts. The Yellow Reports include an executive summary and conclusions that have been formally approved by the SBU Board of Directors and the SBU Scientific Advisory Committee.

Alert Reports – early assessments of single, new methods that are being developed and disseminated in health care. These assessments also include a systematic literature review. However, in contrast to

228

the Yellow Reports, each Alert Report addresses a single intervention only. The SBU Board of Directors and the Alert Advisory Board formally approve the findings.

White Reports – information on the "state of the art" exploring topics in health care that may need to be assessed. These documents address important issues and may be the starting point for future systematic literature reviews. White reports are reviewed by the project groups and external experts only.

SBU Comments- In this new serie, current literature reviews fro other countries are presented. One the one hand, reports are summarised, one the other hand, it is completened with comments concerning the application of the information in a Swedish context. The report is electrnically published.

Information service for the health care. This activity, including quick answers on clinical questions, started in 2008 and only intends to publish the answers electronically.

Evaluation on the behalf of other authorities. These evaluations, including those carried out on behalf of the government, are presented as regular SBU reports.

1.2. Target audience (including stakeholders) Reports produced by SBU are intended for those who make important choices regarding healthcare options. Target groups include professional caregivers, healthcare administrators, planners, and health policy makers. The findings also concern many patients and their families.

1.3. Funding SBU is funded by the industry and has an annual budget of approximately 7-14 Million Euro.

1.4. Staffing SBU secretariat, located in Stockholm, is staffed by around 30 individuals who organize, coordinate, and disseminate findings from the assessments. However, SBU projects engage an active network of ca 400 clinical and scientific experts in Sweden and abroad. The network includes leading experts in fields such as medicine, nursing, dentistry, health economics, ethics, and epidemiology.

1.5. Organisation model SBU is centrally organized. It is managed by a Board of 11 members, representing key authorities and organizations within the Swedish health care, including dental care. Two scientific boards - SBU's Scientific Advisory Committee and the Alert Advisory Board - with 15 and 12 members respectively, are assisting the board.


SBU receives proposals for assessment projects from many sources, e.g., individuals, organizations, government agencies, and other decision-making bodies, in Sweden and abroad. The SBU Scientific Advisory Committee, which represents a broad range of professions in health care, recommends topics for new projects.

Brief description of the way research is prioritised

Generally, the topics selected are of major importance to public health and quality of life. These issues are of great concern, involving common health problems and

229

technologies with major economic consequences. Some projects focus on conditions for which treatment and medical outcomes vary throughout the country. Ethically controversial issues and interventions that require major changes in organization or staffing also command high priority.

Brief description of the way (procedures) the research projects are selected

The work process looks slightly different for yellow and white reports, on the one hand, and alert reports, on the other hand. In general, before initiating an assessment project, SBU examines the scope of scientific literature available on the topic. An exploratory study shows whether or not it is feasible to draw conclusions based on scientific evidence, or whether major gaps in knowledge need to be filled. The Board of Directors and the Scientific Advisory Committee then determine which of the proposed subjects should receive further assessment and be published by SBU as a Yellow Report, Alert Report, etc. Work Process for SBU Yellow and White reports

A project group, often including more than ten members, is formed for each major SBU project. Initially, the members are trained in systematic literature searching and critical analysis of the literature identified. After that, the work generally includes the following phases:

Defining the topic

First, the purpose of the assessment is defined. As the project begins, the group decides on the issues to be included and excluded.

Setting the standards for quality

Inclusion criteria are established for the quality and relevance of the studies. Only the results from research that is sufficiently rigorous will be used. SBU's list of common pitfalls provides examples of unacceptable results. In some cases, project groups may decide to include studies of lower quality or relevance if the material presents unique information.

Collecting relevant research findings

All available research findings addressing the important issues are systematically searched in computerized databases and by manually scanning reference lists in professional journals and scientific reports. Searches may identify several thousand articles if the assessment concerns a broad area, for instance, common methods to treat obesity or substance abuse.

Selecting studies of acceptable quality

Since the quality of research may range from high to low, project groups must separate the 'wheat from the chaff'. Each research report is carefully reviewed and evaluated. The project groups evaluate every study for quality and relevance. At times, the groups use standardized checklists in this process. Research reports that do not meet the pre-determined criteria for quality and relevance are eliminated in this step of the review process.

Weighting the results

Results from the selected studies are summarized in tables, scrutinized, and used to form the body of evidence. As with searching and selecting the literature, weighing the evidence must also be systematic and rigorous. All conclusions drawn must have scientific support. It is important to address not only the medical effects of different methods, but also the prevalence of the problem, current practices in Sweden, and the economic, social, and ethical aspects.

Summarizing the evidence and draw conclusions

Before SBU publishes its findings, the manuscript is evaluated by external experts and by experts from the SBU Scientific Advisory Committee. Manuscripts are always carefully edited, and the language is revised prior to publication. The SBU Board of Directors and the Scientific Advisory Committee approve the conclusions drawn from the evidence and consider the findings in a broader context. As a rule,

230

the strength of the scientific evidence is noted for each conclusion by using the SBU evidence grading scale.

Work process for SBU Alert Reports

Identifying new methods for evaluation

Suggestions of new methods needing assessment come from many sources. This include: employees within the health care system; organizations such as the National Board of Health and Welfare, the Medical Products Agency; regional authorities; the Alert Advisory Board; the staff of SBU; international counterparts of SBU Alert; and the joint network EuroScan

Prioritizing methods

The Alert Advisory Board decides which of the suggested methods that should be assessed by Alert. The methods that Alert assess must have been tested on patients and the results must have been published in a scientific journal or presented at a conference. The method should not be well-studied or widely used in health care, although, established method that are going to be used for a new purpose, may be assessed. Furthermore, the method should be expected to have a major impact on health care services, for example through: representing a medical breakthrough; affecting a large group of patients; influencing the structure of health care delivery; resulting in controversy or have ethical and/or social implications; having a substantial economic impact

Reviewing

One or more experts are appointed for reviewing the methods, sometimes at the recommendation of the Swedish Society of Medicine. A draft, based on a standardized format, is prepared by the appointed expert in collaboration with SBU. The draft is reviewed by at least one additional expert, the Alert Advisory Board and the SBU working committee. Where appropriate, manufacturers are asked to comment on the draft. If new, relevant, scientific documentation becomes available, the report is updated.

Assessing the scientific evidence

An SBU Alert report begins with summary and conclusions, which includes an assessment of scientific evidence concerning patient benefits and cost-effectiveness.

2.2. Involvement of stakeholders during research process As mentioned above, the project groups consist of leading experts from Sweden and abroad, with strong roots in various fields of clinical practice and scientific research. The basic idea is that leading thematic experts are essential for achieving maximal impact. Besides providing their thematic expertise, they also function as “lawyers” supporting the research results. This is possible since the experts are involved in the whole research process and thus have a great trust in the research outcomes.

The project groups also consist of expert reviewers as well as members of the SBU Scientific Advisory Committee, Alert Advisory Board and the SBU Board. The scientific advisory boards include representatives from the Swedish health and medical care.

The Board includes representatives from the National Board of Health and Welfare (Socialstyrelsen), the National Medical Products Agency (Läkemedelsverket), the National Medical Council as well as local and regional authorities.

These representatives are involved in the whole project cycle, from deciding on which projects to start (in order to ensure that the projects are well incorporated in the Swedish health care) to reviewing the results and supporting the general decision-making (in order to ensure that the results and decision-making enjoys a broad support).

SBU also organizes a network of local "ambassadors" throughout Sweden. The role of the local ambassadors is to support an effective dissemination and implementation of the findings. Over 40 people are engaged in activities aimed at various target groups at the municipal and county levels.

231

SBU also has an extensive network of collaborative partners in Sweden and internationally. In Sweden, SBU collaborates with the the National Board of Health and Welfare (Socialstyrelsen), the National Medical Products Agency (Läkemedelsverket), the National Medical Council and with local and regional authorities.

Within the international arena, SBU serves as the secretariat for the International Network of Agencies for Health Technology Assessment (INAHTA), which promotes collaboration and the exchange of information on planned, ongoing, and completed assessment projects.

SBU is also a member the Euro Scan network. The network has the aim to share information in the field of new medical techniques and evaluation. SBU is actively involved in ongoing discussions and have access to the common database. Through active international collaboration, ongoing evaluations in the same area can be identified, duplication of work can be prevented, project work can accelerate, and exchange of information can be promoted. For the same purpose, SBU has also, over shorter periods, accommodated trainees from various countries, with an aim to build up evaluation competencies within the health care. SBU also supports the WHO Health Evidence Network (HEN), an information service for decision makers in the health field. The cooperation aims to produce and implement evidence-based knowledge as a basis for important policy decisions.

2.3. Impact intended Intended impact

SBU does not explicitly formulate the intended impacts of their projects. In general, it is not until the end phase of the project that the desired impact is defined, although ideas regarding desired impacts may be exchanged already at the outset of the project. In these discussions, the board is involved. In the end phase of the project, communication may be taken up with operational managers and policy-makers as to identify important target groups for which the research results have an important impact.

Target groups

In general, SBU reports are intended for both those working directly with patients and those who organize and plan the health care. The specific target groups for each project are identified in the beginning of the project cycle but may be revised at a later stage.

The target groups for SBU reports vary depending on the topic, but generally include:

healthcare managers and staff;

patients;

decision makers at the administrative;

county, and municipal levels;

purchasers of health services;

quality improvement teams; and

county drug review committees.

3. Dissemination 3.1. Dissemination activities performed

The mission statement states that SBU should distribute their literature reviews in such a way as to maximise the impact in health care practice. SBU should briefly declare the various measures taken in order to reinforce the impact of the research activities as well as specifying the target groups of the information. SBU should also declare the actions taken to spread the knowledge of evidence-based health care to the various categories of staff in the primary and continuous education.

Dissemination procedures and activities

The yellow and white reports and the research conclusions are distributed within the health care to physicians, nurses, healthcare administrators, healthcare policy makers, dentists and other stakeholders. SBU Alert reports are distributed digitally through a free subscription service to SBU's web site.

232

To maximize the impact of their reports, SBU tailor the dissemination practice depending on the topic, conclusions and affected target groups. The choice of dissemination channels therefore vary between the projects. In general, findings are widely communicated. The major dissemination activities include:

Communication of findings via SBU’s newsletter Science & Practice (over 100 000 copies per issue), the Journal of the Swedish Medical Association, and other medical journals.

Publication of articles and comment on new reports in specialised press.

Publication of new reports on the SBU website. Alert Reports and other news on the SBU website are also disseminated free of charge via an Internet subscription service.

Re-publication of earlier published reports.

Newsletter. SBU is regularly sending newsletters to 6000 subscribers containing information and direct links to new reports, press releases, current conferences and other news.

Mailings to medical journalists, news agencies, broadcast media, daily press, specialized press and popular magazines of six press releases on new SBU reports of particular public interest. Efforts to publicize the SBU's findings through the mass media has also meant many personal contacts with journalists and editors.

Internet-based information. Besides the public website, SBU has an intranet as well as a number of extranets for the board, the scientific advisory committee, the Alert Advisory Council and different project groups.

English publications. Several reports have been published as books in English or as supplement to scientific journals. In some cases, In some cases, the content of the reports are published in a series of scientific articles where the major conclusions are presented.

Active participation in conferences, workshops, international evaluation projects in order to distribute newly published results, to increase the knowledge of SBU’s work, and to initiate regional discussion on evidence-based evaluation for policy makers within the health care. They are used in formulating national, regional, and local guidelines in health care, and are often followed by recommendations from other agencies.

Participation in courses. This includes for instance courses and lectures in evidence-based care for medical and dentist students, trainings for nurses and acupuncturists, and courses for administrations at the regional and other authorities. SBU has also participated in a research school.

Organisation of training activities in evidence-based care for teachers from medical vocational schools.

Dissemination of knowledge through networks. SBU is engaged in national networks with representatives from a large number of Swedish regions. Internationally, SBU is also working with a wide range of actors, including: sister organizations in other countries; the international scientific association HTAi; WHO; OECD; and the international networks Cochrane Collaboration, network, INAHTA, EUnetHTA network, and Euro Scan..

Besides these regular activities, additional information and education activities may be carried out concerning individual projects. This include for instance the development of special folders for projects of special current public value.

Products

Besides the regular reports (Yellow Reports; White reports; Alert reports; SBU Comments), SBU produces a number of additional products. The major ones are:

Summaries: In order to make the reports, which are often very extensive, more easy accessible, SBU publishes summaries of the reports.

Special prints: SBU produces special prints that reproduce the introductory chapter of the reports with a summary and conclusions.

233

Such special prints may be published and disseminated, for instance, at video exhibitions, fairs and regional meetings.

Study letters: In order to enable a clinical application of SBU: results, tailor-made interpretation and adjustment is needed in different clinical contexts. In cooperation with the Swedish association of public medicine (SFAM), so called study letters, based on the systematic literature reviews, have been formulated for public doctors. The study letters are disseminated through SFAM to different training groups throughout the country.

Patient information: In order to provide easy accessible but scientifically proof information for patients and relatives, so that they can request evidence-based treatment, SBU produces evidence bases popular versions of their reports, both via web and in printed version. The printed versions are spread via pharmacies and clinics, including dental clinics.

‘Science and Practice’ (Vetenskap & practice): The Journal ‘Science and Practice’ is one of SBU’s most important instruments to reach a broad audience. The Journal refers to and disseminated the results from reports, describes current projects, promote published reports, informs about evaluation projects carried out by sister organisations, and promotes evidence-based and critical reviews of clinical practice. The Journal is published four times a year.

Study materials. SBU reports are frequently used as a basis for study materials, seminars, courses, and lectures.

Tailored dissemination products

There is no systematic strategy for tailor-made dissemination of products but it does take place on an ad-hoc basis, in relation to the publication of specific reports. In these cases, it usually takes the form of a seminar with invited stakeholders. Depending on the report, such seminars can be organized before, in relation to, and after the publication (for instance, if new target groups effected by the results have been identified). For instance, as a result of the arising of ethical discussions and outrages, after the publication of a report on fetal-widening during birth, an extra seminar was organised for affected stakeholders and target groups.

As for the dissemination of the Journal ‘Science and practice’, in 2008, efforts were taken to tailor-make the co-distribution of the Journal of Science & practice with the Journal ‘Nursing magazine’ to selected groups of nurses which resulted in a greater publication as compared to 2007. In 2008, specific e-newsletters with links to the Journal’s contents, were also sent out together with each edition of the journal. The newsletter has been sent by email to all subscribes of the SBUinformation, in total 6 200 copies.

Accessibility of products Website. On the SUB website, information is available on current

reports, conferences, regional activities, on-going projects, the Journal ‘Science and Practice’, as well as a section about evidence based medical care (EBM). All reports are free of charge and can be downloaded. Printed publications can also be ordered via a web shop, directly linked to a distribution agency.

Other web services. During 2008, SBU further develop their web services, including project databases for HTA networks, interactive patient cases, web-based practice surveys, and web-designed conference invitations.

Cochrane Library. Since 2005, SBU has a subscription for the Cochrane Library. The subscription, which is free of charge for all Swedish users via SBU's website, aims to serve as a scientific knowledge bank and to facilitate medical decisions.

Answers to questions from the public. SBU responds to e-mail questions from the public, patients and relatives regarding the content of the reports and SBU's working methods.

Information Service. SBU has an information service that responds to questions from the health care. This can concern for instance how the results should be used in the planning of treatment.

‘Synthesis products’

234

SBU publishes yearly annual reports. On the website, a strategy plan for future directions is available. Besides, the Journal ‘Science and Practice’, published four times a year, offers a more popular compilation of the development within the field. One of the editions is in English and is published in relation to the INAHTA international congress.

Involvement of stakeholders in dissemination

Stakeholders are not systematically and directly involved in the dissemination of products but do take their own initiatives to spread information.

Monitoring of policy developments

The marketting department conduct surveys among policy-makers and among practitioners regarding status, trust, knowledge of the institute etc. In general, SBU are satisfied w3ith the influence they have and the credibility they have built up within policy-making

3.2. Stakeholders addressed by outputs Ways in which stakeholders are addressed by outputs

SBU maintain direct contact with the Ministry of Health and Social Affairs and tries to invite important stakeholders, such as the Social Committee of the Ministry of Health and Social Affairs for information of SBU’s activities.

Involvement of stakeholders in developing outputs

Sometimes, stakeholders are involved in developing outputs, in particular in relation to the Alert products. The report is sent to relevant stakeholders before its finalization, for them to comment on. However, stakehodlers are not involved ni the expert groups.

SBU is therefore not engaged in any lobbying as a way to remain impartial. However,

Stakeholders, base don their interest, take their own initiative to use the results of SBU reports to develop on products.

Involvement of stakeholders in disseminating outputs

SBU tries to engage the project group in disseminating information, both the SBU staff and the experts. The experts are crucial considering their expertise in the field. The SBU project groups and the networks of local SBU ambassadors across Sweden also initiate and promote local and regional efforts as to help assuring that the reports are used by decision makers and that the findings are applied in clinical practice. Local and regional involvement is important for an effective dissemination and implementation of SBU's findings. They have a special interest in disseminating information about evidence based medicine. They use the findings from SBU reports to influence clinical policy and practice at the regional and local levels.

A range of professional organizations in health care, insurance organizations, and other interested parties also participate in implementing the findings of SBU assessments.

3.3. Actual take-up of outputs Monitoring of the actual take-up of outputs

SBU is working continuously with monitoring the actions that might have the greatest impact. Together with the Swedish Central Statistical Bureau (SCB), SBU has made several target group surveys, as a basis for their communication strategy.

A study from 2005 showed that the majority (59-76 percent) of the operational managers, doctors and regional politicians in Sweden are made aware of the contents of the SBU reports via the SBU's summaries and conclusions. In 2008, a ‘reader value survey’ was carried out the Journal of Science & practice among physicians with a management responsibility, general physicians, regional boards, regional politicians, local health care boards, hospital boards, and nurses. The survey showed that the Journal continues to be an important and valued channel for disseminating knowledge coming from SBU's research results and for promoting

235

implementation of evidence-based health care. The majority of those receiving the Journal furthermore share it with others at their workplace.

Monitoring has also taken place of the extent to which the results from SBU reports published during the year have been referenced and discussed in detail in the SBU's Journal of Science & practice. Also the extent to which articles from the Journal of Science & practice have been quoted in other media and have been used in medical teaching and in educations at medical colleges have been monitored.

4. Contribution to the decision-making process 4.1. Familiarity with research projects Monitoring of the knowledge of projects of specific stakeholders

SBU has partly monitored the knowledge of their projects among specific stakeholders through surveys among specific stakeholders, with questions on the extent to which they are familiar with SBU and their reports. These type of surveys are no longer conducted for two reasons: First, the survey as well as the general impression show that most respondents are already familiar with SBU whereby these types of surveys are less important. Secondly, the response rate was shown to be very low.

4.2. Research utilization SBU affect the health care in two ways:

First, other authorities, like the National Board of Health and Welfare and the Dental and Pharmaceutical Benefits Agency, (TLV) use SBU's reports for its policies and decisions. The literature reviews produced by SBU found the basis for the Medical Products Agency’s treatment recommendations and for the national guidelines of the National Board of Health and Welfare.

Secondly, the health care itself follows the conclusions drawn from the SBU projects. The literature reviews produced by SBU found the basis for regional and local health care programs.

Since SBU doesn’t have any legal power, they can, however, only encourage policy makers and health care providers to utilize research results by offering high quality and credible reports and by upholding its reputation as competent and trustworthy.

Monitoring of whether policy makers are influenced by the findings

There are no specific measures for monitoring whether policy makers are influenced by SBU’s research findings, although the work of SBU is clearly reflected in the political debate. However, through the surveys measuring the familiarity among stakeholders of SBU and through cooperation with the National Board of Health and Welfare, SBU keeps informed of the extent to which their reports influence policy-making and practice.

Monitoring of the actual use of findings?

The mission statement state that SBU should describe the impact of their activities on the health care practice, including the dental care. SBU should also make an assessment of use of the Alert reports within the health care.

SBU is generally monitoring the concerned practice before and after a report has been published through so called practice studies. These are carried out in two different ways:

Usually, monitoring takes place through questionnaire surveys to the concerned the heath care services. Questions are asked regarding the current practice around methods applied and advice given etc. These surveys are then followed up after a report has been published.

More exceptionally, registered data used. Opportunities to conduct register studies has been greatly improved since the establishment of a National Register. Register data includes information on operations, prescribed pharmaceuticals, etc. Through comparing such data before and after a report has been published, changes in practice can be monitored. The data also provides an almost complete picture of differences in practices between different fields and different parts of the country. The establishment of a quality register also facilitates a

236

stronger use of practice studies, not least because they often contain more data on treatment strategies and indications for treatment.

Many examples in recent years have shown that the publication of reports influence the practice. Some recent examples is given in section 5.1.4:

Monitoring of the type of use)> Monitoring of the reasons for not using the findings from a project

Through the boards, in which representatives from the different parts of the health care as well as academicians, are included, SBU gets the necessary knowledge about reasons behind changing or nor changing certain practices and procedures. The Board can thus be seen as an instrument for keeping up to date with the utilization of the research findings.

If the desired changes are not to be seen, extra measures may be taken in form of re-publication or writing of articles.

5. Conclusions 5.1. Main challenges in achieving a high level of impact.

Challenges identified by SUB:

The biggest challenge when it concerns the utilization of research findings is to discard practices or procedures. SBU does not have the legal power to influence this but can influence through the trust they enjoy among stakeholders and target groups;

The medical knowledge is rapidly developing (more than 1.4 million new research articles are published each year within the health care field), which requires SBU to quickly evaluate and update their reports. The challenge is to regularly update the reports on the basis on new knowledge in the field. SBU tries to go through old reports after a few years and sometimes other stakeholders bring to their awareness the need to update certain reports. There needs to be resources put aside for such updates.

Internationally, more and more literature reviews are being made which are important to consider within the Swedish health care. The challenge is to avoid double work, make use of others’ material, and ensuring a balanced quality between various research institutes.

The need for health-economic basis for prioritization within the health care increases.

Collaboration with organizations at local, regional and national level in order to bring out the available knowledge needs to be strengthened. SBU is allocating significant resources on dissemination. However, considering the extensiveness of the health care field, the resources are marginal.

In many projects, SBU identifies knowledge gaps where there is a need for more research. It is important for universities, colleges and research institutes to be aware of such future research needs.

The work with highly qualified experts is crucial for the success of SBU’s work. A challenge is that these experts are in general very busy, having their major job elsewhere, and can only devote limited time to the projects. In recent years, there has been difficulties with recruiting experts to the project groups. To compensate for this, SBU aims at standardize some aspects of the project work.

5.2. Changes in strategy during the existence of the agency? One change during the existence of the agency is they have stopped the programme for recruiting local ambassadors. This programme was necessary at an earlier stage to build up relationships and create awareness of the work of SBU on a local level. Today, these structure are already in place and the programme is no longer necessary.

The newer products, such as SBUkommenterar and the quick answers are also a way to adjust the products to the demand.

In 2006, SBU began to develop a new strategic and long-term plan that would set the future direction of SBU. The strategy, which was adopted in 2007 (an update of

237

the plan is currently under progress and will be finished sometime in November), suggests 21 measures, as to strengthen the impact of the SBU's works. The proposals were divided into three main aspects:

cooperation between authorities and organisations on national, regional and local level.

strengthening medical evaluation in Sweden

Improving the internal efficiency, including project process and reporting

The 21 suggestions, which illustrate the challenges experienced and measures to meet these challenges, include:

Collaboration with other authorities and organizations

Enhanced cooperation with authorities & Knowledge Centres

Enhanced cooperation with professional organizations

Collaboration with research councils, universities and colleges

Developing and strengthening medical evaluation in Sweden

SBU performs all systematic literature and knowledge reviews in order to streamline tasks and avoid duplications, including more applied research serving for other authorities, such as systematic statements for the National Board of Health and Welfare on which they can base their guidelines. More resources has also been set off for making reviews on behalf of the Dental care.

More Swedish health-economic studies

SBU begins quick evaluations and starts up an information service

SBU summarizes foreign evaluations to make foreign findings more accessible for the Swedish audience

SBU disseminates evidence-based knowledge through IT systems

Enhancing internal efficiency

Making the process for selection of evaluation projects more systematic and coherent for all types of products. Selection of projects must be planned together with other authorities and on the basis of the needs in the health care.

Adopting an international system of evidence grading

New project organisation including less work load on the experts

Less and more clear conclusions

Wide dissemination of SBU's summaries rather than dissemination of whole reports. The summaries will be tailor-made and adjusted to the needs of the target groups, requiring analysis of needs and desired of different target groups.

Make sub-reporting of large projects as to make faster the reporting of important research conclusions

Enhanced international publishing

Enhanced use of register data for practice studies as to decrease the work load of of the health care

International cooperation on methodological issues as to strengthen the methodological and specialised competencies within SBU as as to ensure a better international comparability of medical evaluations

More health-economical studies within the existing SBU framework

Developing system for continuous updating of the state of knowledge

Broadening the representation of stakeholders within the SBU's Lay board

Facilitate the recruitment of experts

All in all, these strategic proposals mean a greater commitment of SBU.

5.3. Important lessons for the agency in achieving high levels of impact?

In general, SBU is vary satisfied working together with external experts instead of doing the work in-house. Through the external experts, the work is from start to

238

end integrated in health care practice and the best expertise is used. The disadvantage is that it takes longer to carry out the work.

Besides the measures suggested above, two specific measures for increasing the impact are interesting to mention.

One efficient measure for promoting the utilization of research results, specifically mentioned by SBU, is to show differences in medical practices and quality between different parts of the country. Differences between practices in different parts of the country can indicate irrational use of healthcare resources and illustrate that a clinical practice in one part of the country is not consistent with scientific research. Local comparative data is also a way to get a message across via the media. The logic behind this strategy is that local comparative data stimulate local engagement and leads to action. In such work, it is natural to collaborate with other local and regional authorities and organizations.

During 2008, SBU developed and intensified their efforts to write clearly. All staff have attended a special training in how to write in an efficient, simple, and easy understandable language. Scientific knowledge about effective communication and implementation has also been compiled and discussed, as a basis for work on a new communications strategy.

5.4. Interesting findings from impact studies on the agencies activities

Many examples in recent years have shown that the publication of reports influence the practice. Below some examples are given.

A SBU report from 2000 stated that not sufficient knowledge exist about the most effective treatment for concussion. The two existing alternatives were on the one hand monitoring in the hospital or on the other hand computer tomography and early release. A SBU study published in 2006 showed that there was no difference in efficacy between the two. SBU's report showed that the total cost can be reduced by one third by using the strategy of computer tomography and early release instead of hospitalization. One year after publication, the number of days spent in hospital after concussion decreased by 1 500 care days and health care resources of 40 million could be released and used for other health care interventions.

The project on early fetal diagnosis recommended that SBU information should be provided by the so-called two-step model (general offer of information followed by information only to those explicitly asking for it). A combined method was further highlighted as the best practice that would also help reducing the number of invasive tests. After two years, approximately half of the regional districts had developed the two-step model, one third had introduced the combined test, and over half had reduced the number of invasive tests.

SBU's assessment of pre-operative procedures showed that the routine X-ray examinations of heart, lungs, ECG or clinical chemical tests of healthy individuals without a history of relevant diseases are of no use. The conclusions of the report quickly affected the practice and lead to direct savings of approximately SEK 235 million per year for the health care.

Purchase of equipment for measuring bone mineral density decreased after a SBU study showed that there is no scientific basis for recommending bone density measurement for general or targeted screening.

The Health care has begun phasing out light therapy after a SBU report showed that there is scientific evidence that light therapy is effective against depression

When SBU in 2004 evaluated a new approach to public hearing screening of newborn, the coverage in the country was around 25 percent. After the publishing of the SBU report, an estimated 70-75 per cent of the newborns in Sweden are being screened with the evaluated method.

239

NATIONAL INSTITUTE FOR HEALTH RESEARCH (NIHR)

Draft


Projectnumber: b3561

The responsibility for the contents of this report lies with Research voor Beleid. Quoting of numbers and/or text as an explanation or support in papers, essays and books is permitted only when the source is clearly mentioned. No part of this publication may be copied and/or published in any form or by any means, or stored in a retrieval system, without the prior written permission of Research voor Beleid. Research voor Beleid does not accept responsibility for printing errors and/or other imperfections

1. Introduction 1.1. Mission and goals

The National Institute for Health Research (NIHR) was established in April 2006 to carry forward the vision, mission and goals outlined in the health research strategy 'Best Research for Best Health' of the English government.

The mission of the NIHR is: to create a health research system in which the NHS supports outstanding individuals, working in world-class facilities, conducting leading-edge research, focused on the needs of patients and the public.

Five goals have been defined. These are:

Establish the NHS as an internationally recognised centre of research excellence.

Attract, develop and retain the best research professionals to conduct people-based research.

Commission research focused on improving health and social care.

Strengthen and streamline systems for research management and governance.

Act as a sound custodian of public money for public good.

1.2. Target audience (including stakeholders) The target audience can be deduced from a set of operational statements in the NIHR progress report 2008/09.xcviii

First, the NIHR exists to maintain and manage the research, research staff and research infrastructure of the NHS in England as a national reserach facility.

It also exists to enable the NHS to support outstanding individuals - leaders and collaborators - working in both the NHS and universities, conducting leading-edge research, and to establish the NHS as the preferred environment in which to carry out multi-centre clinical research in partnership with and for industry and non-commercial organisations.

But perhaps most important NIHR exists to benefit patients, society, the NHS and all other stakeholders by developing the NHS's reputation as the best environment for collaborative research in the public interest.

By way of a summary, the target audience consist of:

Researchers within the NHS

Researchers within universities

Reserarchers within the industy and non-commercial

Patients

General public

xcviii Department of Health, Research and Development Directorate. Delivering Health Research: National Institute

for Health Research Progress Report 2008/09. Available at: www.nihr.ac.uk/publications. See also the website of the HIHR: www.nihr.ac.uk.

240

Managers and clinicians within the NHS

Table 1 Total budget of the NIHR

NIHR funding for 2008/2009

Infrastructure; Goal 1: NHS as an centre for research excellence

Flexibility and Sustainability 62,0

Clinical Research Networks 157,5

Biomedical Research Centres 113,0

Biomedical research Units 13,1

Patient Safety and Service Quality Research Centres 2,4

Clinical Research Facilities 12,0

Experimental Cancer Medicine Centres 3,3

Technology platforms 3,6

Ad-hoc support and excess treatment costs 8,6

Research Design Service, helping researchers with their applications 9,5

Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) 5,7

Other (including clinical academics) 6,5

Infrastructure total 397,2

Faculty; Goal 2: Attract, develop and retain the best research proffesionals

Integrated academic trainees-including clinical Fellowships, lectureships and awards 33,0

Fellowship and Senior investigators 13,1

Other (including management and academic units) 3,5

Faculty total 49,6

Research programmes; Goal 3: Commission research

Health Technology Assessment 33,1

Service Delivery and Organisation 9,6

Programme Grants for Applied Research 19,8

Research for Patient Benefit 7,6

Invention for Innovation (i4i) and Research for Innovation, Speculation and Creativity 6,7

Reviews (Cochrane, Centre for Reviews and Dissemination (CRD)) 11,8

School for Primary Care Research 3,2

Horizon Scanning and Methodology 2,6

Other, 21,6

Research programmes total 116,0

Systems; Goal 4 Strengthen and streamline systems

Busting bureaucracy information systems 5,0

Research ethics committees and system 10,4

Other 2,8

Systems total 18,2

Transition funding for NHS infrastructure 207,2

NIHR capital allocations 31,2

Total NIHR budget 819,4 Source: HIHR. Delivering Health Research: Progress Report 2008/09, p. 113.

1.3. Funding of the agency In 2008/09 the NIHR has a budget of ₤ 819.2 million. This total budget will increase in the next years up to approximately ₤ 1,000 million. The budget is an earmarked part of the total budget for the NHS and as such part of the budget of the Ministry of Health.

Table 1 presents an overview of the total budget divided by the four operational goals of the NIHR.

241

Staffing

The staffing of the bureau of the NIHR at the Ministry of Health consists of approximately 45 people. The work of the bureau regards organising the funding and making up the contracts with the participants in the NIHR. A part of the work of this bureau is outsourced to a private organisation: the Central Commission Facility(CCF). This concerns 20 to 25 people.

The NIHR Central Commissioning Facility is a primary gateway for NHS clinicians and related health researchers seeking funding from the NIHR and the Department of Health

Organisation model

The NIHR is a network organisation which consists of four main parts that coincide with the goals of the HIHR, as can be seen from table 1.

Infrastructure

Faculty

Research

Systems

Figure 1 presents an overview of the organisation model of the NIHR.

Infrastructure

Infrastructure is about building world class facilities necessary to perform high quality research and encourage innovation across the country. NIHR provides the necessary infrastructure at very stage to ensure that research is turned into safe and effective benefits for patients as quickly as possible. All the infrastructure supports collaboration with industry.

Faculty

The NHS faculty encompasses all the researches in the NHS, universities and other organisations who are supported by the NIHR. This includes individuals at every stage of their carreer and at al points along the research pipeline. The faculty includes those who generate research ideas, those who lead research, those who make research happen and those who help turn te resulting discoveries into improved health and quality of life.

Research

NIHR commission and fund research into health and social care that is essential to produce evidence to enable professionals, policy makers and patients to make informed decisions and provide the means to turn new interventions into better care.

Systems

To provide better patient care through research, efficient, robust and above all transparent ethical and governance systems are needed. With the systems programme ethical and research management systems are refined to meet this challenges.

The main starting point for the organisation model is that the NIHr needs to cover 100% of the UK population. This implies that a model based on a geographical equality.

However sometimes this is difficult to reach, because of the nature of a specific programme. For instance, to apply for a clinical research centre it is needed to prove world class excellence in five biomedical fields. Consequently this programme is concentred at the big universities.

242

Figure 1 The NIHS organisation


In principle every kind of organisation is allowed to submit proposals. Most important organisations are:

HNS organsiations

academic organisations

other research institutes

health care industry

professional bodies

patient organisations How many percent of the proposals is submitted by each organisation in 2008?

There are no actual data available about this question. Brief description of the way research is prioritised.

There are three starting points with regard to prioritising research. Research must be:

relevant to patients in the NHS

of high scientific quality

abstained from by other parties.

This last starting point refers to a situation of market failure in funding research. Research has the potential to be publicly beneficial, but non other party wants to fund the research because it is out of their direct interest. It is interesting that other parties often join funcing a research project once it has received a HIHR budget. In general the contributions of industry are some 25%.

The selection process is guided by a number of principals. These are:

Accountability

The NIHR is committed to using public money efficiently, to providing value for money and to communicating the processes and results of the research

Delivery

All research has to be able to be translated into deliverable outcomes for NHS patiens.

243

User involvement

The public, service users and carers are active contributers to all the projects at all the stages of the research process.

Peer review

Application are rigorously evaluated by independent peer review.

Preparation

Schemes and programmes are in place to help researchers to plan their applications and to get approval without undue bureaucracy.

Transparency

Clear and transparent processes at every step of the research pathway are being used. from deciding what research to fund to costing it, finding it, publishing it and disseminating its results.

Collaboration

A collaborative approach is adapted, with partners ranging from the Medical Research Council, to research charities, universities, the life sciences industry and others.

There are three funding rounds a year and ten regional funding committees. Brief description of the way (procedures) the research projects are selected.

The NIHR uses a variety of funding mechanisms, each appropriate to the needs of the individual research programmes. The guiding principle here are the above mentioned principles

Reviewing committees exits of lay people amongst which NHS managers, patients and other members of the general public; peers and board members. These committees are representing the stakeholders, the scientific community and the general public.

Type of proposals: are they policy or practice oriented?

There are no data available of the type of proposals, but from the total budget of the NIHR research program of ₤ 116 million, ₤ 33 is reserved for HTA and ₤ 20 million for applied research.

Broadly speaking there are to levels to which the research is directed: the strategic level and the decision making level. The strategic level involves research that is conducted for the R&D directorate of the Ministry of Health. This is research with a long term perspecyive. The decision making level involves practitioners and policy makers

2.2. Involvement of stakeholders during research process Description on if and how stakeholders are generally involved before, during and

after the research.

Involving stakeholders in the research process from the planning stage to the implementation is at the heart of the NIHR. Two examples will be given here to show this.

The first example are the nation schools of primary and social care. As part of the commitment to transform health and social care, the NIHR funds two national research schools for primary care and social care research. These schools bring together top academics and practitioners to conduct leading-edge research to benefit patients and to improve their experiences of health and social care in their local communities.

The second example is one of the projects within the Research programmes: the INVOLVE programme with a relatively small budget of ₤ 0.8 million. This program supports active public involvement in research working closely with researchers, funders and the public to help ensure that patients and the public have a say at all stages from planning to implementation. The budget for the INVOLVE programme will increase to ₤ 1.0 in 2009/10.

2.3. Impact intended

244

Does the agency explicitly formulate the intended impact of a research project in advance?

Yes, this is done for all research projects as can be notices from the delivery principle to fund research

Does the agency explicitly define the primary target audience(s) aimed for?

Yes. For instance, within the ‘Programme grants for applied research’ the research findings are intended to provide the NHS with evidence to help improve management of chronic debilitating health conditions such as mental health, heart disease, diabetes, arthritis, dementia, stroke and children’s health problems.

3. Dissemination 3.1. Dissemination activities performed What happens generally after a research project is finished and the report is

published?

To turn research into routine practise nine regionally working organisations have been set up. These are the Collaboration for Leadership in Applied Health Research and Care (CLAHRC’s) organisations. These bring together professionals from the universities and surrouding NHS organisations to test new treatments and ways of working in specific clinical areas, and to see if they are effective and appropriate for everyday NHS use. Where potential improvements are identified, CLAHRC’s will help NHS staff to incorporate them into their daily working practices. In the first year 2008/09 the CLAHRC’s have been putting their infrastructure in place working out their research agenda and creating partnerships in their region.

In what way are the findings disseminated?

In the Derbyshire and Lincolnshire CLAHRC there are diffusion fellows active. These are people like consultants, senior nurses and others who work with the CLAHRC staff to do the research and who are enthusiastic about pushing the evidence into practice. Also managers and health care professionals such as GP and district nursing teams are trained to put the evidence into practice. The budget for the CLAHRCs was in 2008/09 ₤ 6 million, it will increase to ₤ 18 million in 2009/10

Furthermore there is the Centre for Reviews and Dissemination (CRD).With funding worth of ₤ 2 million, the CRD, based at the University of York, is helping to ensure that those who need to know findings of research get to hear about them and can make sense of them in the context of their everyday practice

Are products tailored to the primary target group ?

Yes, see the above description of the working of the CLAHRC’s

Two other faculties are worth mentioning here, because they are directed towards specific target groups. These are the Schools for Primary Care Research and for Social Care Research. These schools bring together top academics and practitioners to conduct leading-edge research to benefit patients in their local communities. The School for Primary Care Research supports more than 145 researchers investigating how primary care professionals can prevent, diagnose and manage common conditions such as infections, diabetes, cancer, and mental health problems. The first task of the school of Social Care research is to raise awareness of the need for social care research, because such research is often hampered by the belief that social care interventions are just common sense.

In what ways products are obtainable?

Lots of different ways. One of the most important is the UK PubMed Central. This is a free archive of life sciences journals. It also provides a manuscript submission system for grantees from the participating funding organisations. The NIHR is one of these organisations. The starting point for funding research by the NIHR is that research findings need to be published and need to be freely available for everyone. This is implemented with a condition for funding that the results need to be published in the UK PubMed Central.

Does the agency develop ‘synthesis products’

Yes Does the agency cooperate with stakeholders to disseminate findings?

Yes, see the description of the working of the CLAHRCs

245

Does the agency monitor policy developments?

Yes. For instance the CRD works closely with those informing and developing policy such as the Department of Health and the Office of the Chief medical Officer to produce independent, policy-relevant systematic reviews. It is also one of the academic centres undertaking reviews commissioned by the NIHR HTA programme and NICE

3.2. Stakeholders addressed by outputs In what way are stakeholders addressed by outputs?

There are regional road shows directed at NHS people. NIHR also organises colloquia and discussions,

Are stakeholders involved in developing outputs?

Yes. One of the most important aspects here is that beforehand formats of reporting the results are designed together with the stakeholders. This improves the effectiveness of the process of disseminating the research results.

3.3. Actual take-up of outputs Does the agency monitor the actual take-up of outputs?

Yes, by audits of research output, by ex post evaluations, by guideline quotation systems and by the design of a monitor system. This is very much work in progress.xcix

4. Contribution to the decision-making process 4.1. Familiarity with research projects Does the agency monitor if specific stakeholders have knowledge of projects?

yes. The NIHR organises perception audits amongst practitioners, academics and members of industry and other organisations.

4.2. Research utilization Does the agency monitor if policy makers are influenced by the findings?

Yes. This monitoring takes place by ‘Policy review programmes. These programmes are also directed at the level of the ‘Strategic Health Authority’.

The actual use of findings and the type of use?

Yes. By case studies and setting up a library of impact.

5. Conclusions Some general questions:

What have been the main challenges in achieving a high level of impact.

One of the main challenges is connected with the non-linearity of the research process. In general this process is unpredictable

Another main challenge is the evaluation of the use of evidence in policy and in practice.

Did the agency change strategy during its existence?

No. the NIHR exists since 2006. It is still in a situation of passing out of transition. One can say that in 2008/09 the HIHR as really taken off.

What are important lessons for this agency in achieving high levels of impact?

One of the main lessons is that the bottleneck of improving the health case system is not the amount of research and the production of knowledge. What will save

xcix El Turabi, A. Performance framework in practice: Trying to link performance to impact. Presentation. New

frontiers in evaluation of impact of medical research. Signula, Sweden, 8th May 2009.

246

most lives en will improve quality the most is the dissemination or already established knowledge.

It is generally recognized, also by the English Treasury, that this will bring value for money. That is why the budget for the NIHR will increase to a level of ₤ 1 billion in the next years.

Other important findings are related to the organisation of the NIHR. The NIHR can be characterised as a network organisation that is governed from a specific centre. This gives the possibility to address market failures in the research process. The NIHR does not want to intervene in research processes that will be taken up by private parties. It concentres on these branches of research that can’t be left to the market.

A consequence of this is that the NIHR takes unnecessary competition between researchers and research institutes out of the system. But a very important starting point and leading principle for the governance of the NIHR is that decisions need to be taken at a level as low as possible.

Study into the impact of the Belgian Health Care Knowledge ...

Documents

Transcript of Study into the impact of the Belgian Health Care Knowledge ...