EDUCATIONAL INTERVENTIONS TO PROMOTE CLINICAL

REASONING: A BEME SYSTEMATIC REVIEW

SYSTEMATIC REVIEW PROTOCOL

Nanos gigantum humeris insidentes1.

1. REVIEW GROUP MEMBERS AND AFFILIATIONS

Professor Judy Mckimm. Dean of Medical Education at College of Medicine, Swansea University

Dr. Ana L. Da Silva (lead reviewer). Lecturer in Medical Education Swansea University

Dr. Paul Garrud Assistant Director of Medical Education, Division of Medical Sciences & Graduate Entry

Medicine, School of Medicine, University of Nottingham

Dr. Daniel Mclaughlin, Director of Undergraduate Studies, Division of Medical Sciences & Graduate Entry

Medicine, School of Medicine, University of Nottingham

Dr. Jean McKendree, Senior Lecturer in Medical Education, Hull York Medical School, University of York

Dr. Janine Henderson, Programme Director MB BS, Hull York Medical School, University of York

Dr Anna Hammond, Director of Communication Skills Teaching, Hull York Medical School, University of York

Dr. Simon Gay, Clinical Lecturer in Medical Education, School of Medicine, Keele University

Dr. Maggie Bartlett, Clinical Lecturer in Medical Education, School of Medicine, Keele University

Dr. Laurence Atkinson, Assistant Director of General Practice Studies at Cambridge School of Clinical Medicine

Dr. Claire Vogan, Director of Student Support at College of Medicine, Swansea University

Dr. Steve Durning, Professor of Medicine and Pathology Director, Uniformed Services University of the Health

Sciences F. Edward Hebert School of Medicine (reviewer and lead of Clinical reasoning assessment BEME

review)

All members of the group will undertake an active role in the current review. Tasks performed by each

individual member (e.g. review of RoB, search, synthesis) will be allocated based upon the members’ interest

and expertise.

CONTACT DETAILS OF LEAD REVIEWER

College of Medicine, Grove Building, Swansea University, Singleton Park Swansea SA2 8PP

Tel. 01792 606547

Twitter:AnalindaS Email: a.l.sergiodasilva@swansea.ac.uk

1.1 WIDER ADVISORY GROUP

Experts in the field will be approached to take part in this review as an wider advisory group upon protocol

approval by BEME reviewers.

1.2 SOURCES OF FUNDING

Currently this review has no external sources of funding. Funding from available grants will be sought to

support particular activities such as training sessions, preparation of outputs, group meetings and

dissemination of findings to a wider audience (e.g. stakeholders, educators).

1 John of Salisbury, Metalogicon (1159 pp. 167). Meaning: One who discovers by building on previous discoveries.

1

2. BACKGROUND AND CONTEXT

‘Our job as educators is to continue to evolve our teaching methods in the hope that our students become

more efficient and more accurate problem solvers and make fewer cognitive errors’ (Kassirer, 2010, p.1124).

The importance of clinical reasoning for the competency of healthcare professionals (Epstein, 2007; Higgs et

al., 2008; Higgs, 1992) and the quality of care they provide to patients has long been established (Stiegler et

al., 2011; Graber et al., 2005; Norman & Eva, 2010). The fascination with revealing and understanding the

mental processes by which experts can successfully diagnose and treat patients, are embedded in four

decades of prolific research (Norman, 2005). However, only recently more attention has started to be devoted

to how educational interventions could contribute to promote this thought process. This is of great

importance, as is our role as educators to ensure students and trainees are given the learning opportunities

necessary to develop into successful and safe clinicians (Kassirer, 2010, p.1124). That can only be effectively

done if based upon the best available evidence.

The past research agenda across healthcare disciplines in has focused largely on understanding how expert

clinicians solve clinical cases, mainly in comparison with novices (Elstein, 2009; Rikers & Verkoeijen, 2007),.

During the last forty years many competing explanations and conflicting hypotheses have been suggested.

Many of these hypotheses still endure today however, some consensus has emerged. Research identified a

strong link between clinical knowledge and clinical reasoning (Patel et al., 1990; Woods et al., 2007; Boshuizen

et al., 1992) and the impact of experience in clinical environments (Schmidt et al., 1990; Norman & Eva, 2005;

Norman, 2006). It seems that experts use less knowledge, but possibly of different nature, in a more

expeditious way, based on the construction of elaborate mental representations (schemas or scripts) of the

association between signs, symptoms, diagnoses and prognosis derived from their clinical experience

(Norman, 2005; Elstein et al., 1978; Schmidt et al., 1990; Charlin et al., 2000; Norman & Eva, 2005; Charlin et

al., 2007; Eva et al., 2002). Such mental representations allow experts to rapidly and accurately recognise

similarities (patterns) between old and new cases, and act accordingly. This process was designated as ‘pattern

recognition’ or ‘non-analytic reasoning’ (NAR) (Norman et al., 2007). Novices, lacking such representations,

have to rely on a more analytic, slower and more conscious system (Gobet, 2007; Croskerry, 2009; Reiter et al.,

2002) which is possibly more error prone (Sherbino et al., 2012). This explanation, based on the Dual-Process

Theory (DPT) (Evans, 1984), is one of the most widely used in the field (Norman et al., 2009).

Several authors have shown that patient characteristics, the environment and individual characteristics all

have an impact on clinical reasoning (Durning et al., 2012; Eva et al., 2010; Higgs & Jones, 2000; Smith et al.,

1991; Ajjawi & Higgs, 2012). Some recent views suggest that, as a highly context-specific type of reasoning,

clinical reasoning maybe better understood in the guise of a non-positivist paradigm. Ajjawi & Higgs, for

example, adopted a hermeneutic phenomenological approach to research professionals’ journeys and

trainees’ perceptions of their development of clinical reasoning (Ajjawi & Higgs, 2012, 2007, 2008). More

recently, Durning & Artino suggested ‘Situativity Theory’ as a/the conceptual framework to understand clinical

reasoning (Durning et al., 2012; Durning & Artino, 2011). Situativity theory is defined as ‘theoretical

frameworks which argue that knowledge, thinking, and learning are situated (or located) in experience’

(Durning & Artino, 2011, p.188) encompassing elements from situated cognition (Brown et al., 1989), situated

learning (Lave & Wenger, 1991) and other theories, that share the view that knowledge cannot be separated

of its context and agents (Durning & Artino, 2011). It is still ‘early days’ in terms of fully undestanding how such

viewpoints will help to answer the many questions about clinical reasoning, particularly how its propositions

will endure further research and practice. But expectations are certainly high, especially as these authors are

also conducting a BEME review (Durning, et al. 2013 – BEME protocol), likely to render useful and information

to all of those interested in this area. Two decades have passed since the Custers et al(1996) review into

clinical reasoning and almost one since the review by Norman(2005). Both are landmark reviews in the field,

and during this time many changes occurred calling for the need to re-examine this research area.

2

More so, because in past decades educational interventions to promote clinical reasoning were frequently

neglected or limited to some vague suggestions but recently there appears to be a growing interest in this area

(for example: Ark et al., 2007, 2006; Bowen, 2006; Struyf et al., 2005: Irby & Bowen, 2004). This shift in the

research area, allied to an growing interest in this topic by educators, shown, for example, in the high number

of citations rendered by Eva (2005) paper entitled “what every clinical teacher needs to know about clinical

reasoning” highlights the need for the present BEME review.

Today, the literature includes a vast range of suggestions regarding educational interventions to promote

clinical reasoning. Some examples are: individualized feedback; small group discussions of cases;

demonstration sessions; structure clinical placements; virtual learning patients and online cases; simulation

sessions; reflective practice/reasoning; cognitive forcing and cognitive debasing strategies (Harris et al., 2011;

Dequeker & Jaspaert, 1998; McMillan, 2010; Windish et al., 2005; Durak et al., 2007; Struyf et al., 2005;

Kurzenhäuser & Hoffrage, 2002; Elizondo-Omaña & López, 2008; Delany et al., 2013; Kuiper et al., 2008;

Rencic, 2011; Croskerry et al., 2013). This recent proliferation makes the task of choosing between

interventions complex if no comprehensive summary exists. BEME reviews, as high quality systematic reviews

in medical education, address this problem by providing medical schools and educators with a synthesis of the

best available evidence, allowing for informed choices to be made (Patrício & Vaz Carneiro, 2012).

Additionally, there seems to be also a growing number of thorough research studies designed to formally

evaluate the effect of such suggestions (Cook et al., 2010; Cook & Triola, 2009; Stieger et al., 2009; Borleffs et

al., 2003; Radomski & Russell, 2010; Ark et al., 2007; Bowen, 2006; Graber, 2003; Croskerry, 2003; Croskerry et

al., 2013; Mamede et al., 2008; Sherbino et al., 2012; Trowbridge et al., 2013; Thompson & Stapley, 2011).

These studies are an encouraging sign and provide confidence that there is sufficient high quality research on

educational interventions to make possible the present systematic review.

Finally, although clinical reasoning captured the interest and research endeavors of researchers across many

different research fields (e.g. computer sciences, cognitive psychology, artificial intelligence, sociology) and

different health professions, this body of knowledge is fractured by disciplinary boundaries. Such disciplinary

and professional silos make research findings less accessible to healthcare institutions and teachers faced with

the difficult decision of choosing what educational activities to implement.

In summary, the interest in educational interventions used to promote of clinical reasoning both from

practice and research supports the need and timely nature of the current review, maximizing the potential for

both its usefulness for practice and possible impact on shaping the future research in this field.

KEY DEFINITIONS:

The present review shares the vision of BEME reviews as an “up-to-date summary of evidence that will

enable teachers to make practical decisions and assist policy makers in their choices about the funding of

education” while simultaneously contributing to the body of knowledge in healthcare education (Hammick,

2005, p.33).

The key definitions below will guide the present review. Those definitions were carefully chosen to ensure

the present review aligns with other reviews in the field, primarily the work being carried out by the BEME

group led by Professor Steve Durning (BEME protocol- Durning et al. 2013).

Clinical Reasoning:

Definitions of clinical reasoning are varied, often lacking necessary consensus, nevertheless some common

components can be identified: i) clinical reasoning is a type of thinking process2; ii)it is present at all stages of

the clinical cycle (from selection of information to treatment/prognosis); iii) it leads to actions and decisions

about patient care (context specificity); iv)it involves interplay of factors associated with the patient, situation

2 Thinking is defined here according to Holyoak & Morrison (2012) as “the systematic transformation of mental representations of

knowledge to characterize actual or possible states of the world, often in service of goals.” (Holyoak & Morrison, 2012, p.15)

3

and the individual; v) it is susceptible to heuristics and bias; vi) it is a fundamental part of healthcare

competency (Higgs et al., 2008; Durning et al., 2011; Eva et al., 2007). Such components seem to be common

across different healthcare professions (Jensen et al., 2008). Echoing these identified communalities and the

definition used the contemporary BEME review cited above, for the purposes of the present review clinical

reasoning will be defined as the thinking process by which healthcare professionals select, interpret/analyze

and combine information in order to make decisions and take actions about a patient in each clinical situation.

Additionally, as a thinking process, clinical reasoning will be susceptible to cognitive bias and heuristics.

Therefore any educational intervention, aimed at reducing the risk of such bias or error at any stage of the

clinical cycle (diagnosis, clinical investigations, treatment, management, review) can be understood as

promoting clinical reasoning (Graber et al., 2012; Norman, 2009).

Educational interventions:

For the purposes of the current review, educational interventions will be defined as planned activities with

clearly defined outcomes that are designed to lead to a positive change in a person’s behaviour by creating

structured opportunities for knowledge, skills and attitudes/behaviours to be developed (Gordon & Findley,

2011). These can take place face-to-face either in classroom or clinical environments (real or simulated),

online or using a combination of multiple environments. Equally, these can be aimed at large, small groups or

take place as individualised sessions (Dent & Harden, 2005).

Clinical reasoning is a desirable outcome of any healthcare degree, and therefore is a tacit outcome of many

of the learning opportunities provided within those programmes. For the purposes of the present review all

educational interventions considered must explicitly state clinical reasoning promotion as a desired outcome

(see inclusion/exclusion criteria section).

Additionally, studies using clinical reasoning as one of the outcome measures to investigate overall

differences between curricula types will not be included. Such studies were reviewed elsewhere (e.g. Newman

et al., 2003, Colliver, 2000) with no ubiquitous conclusion regarding the link between clinical reasoning and a

particular curriculum type (Newman et al., 2003). Furthermore it is now understood that variability exists

within the way different schools implement the same type of curriculum. Such variability is often unreported

in the research conducted, which compromises the usefulness of such studies for the aims of the current

review.

It is, however, expected that studies retrieved by our searches will report information about the type of

curriculum as part of the description of their educational context. That information will be included in the

coding sheets to inform discussion of findings. This will narrow the focus of the review without compromising

useful information increasing the feasibility and usefulness of the recommendations (Hammick, 2005).

4

3. REVIEW QUESTION(S), OBJECTIVES AND KEY WORDS

Review Question(S): The current review will address the following question: Which educational interventions contribute towards

promoting clinical reasoning?

Within this main questions, the following sub-questions will be considered:

Which educational interventions are suggested to promote clinical reasoning?

What are their characteristics (environment, participants, content, educational context(e.g.

curriculum type), research context resources required) ?

What are the conceptual frameworks they are based upon?

Do educational intervention vary across professions? How?

What other elements, external to the interventions, can impact the success of such activities?

Review Objectives: In line with the purposes of BEME reviews, the present review main objective are:

1. Provide educators and researchers with a synthesis of the available evidence regarding educational

intervention used to promote clinical reasoning, reporting on quality of the research/findings, type of

studies conducted and type of outcome evaluated (adapted from Barr et al., 2000)

2. Make recommendations for educators and institutions, based upon the best available evidence,

regarding which educational interventions contribute towards the development of clinical reasoning.

These recommendations will included an identification of key characteristics of such interventions

(e.g. type of intervention, profession, participants level, educational context, research methods,

conceptual frameworks underpinning clinical reasoning teaching, resources required for

implementation) in order to allow for informed choices to be made.

3. Identify research trends and gaps, and make recommendations for future research in the field.

Review Key Words: clinical reasoning; diagnostic reasoning; health professions education; inter-professional education, medical

education; educational interventions; cognitive interventions; cognitive error; clinical judgment; clinical

problem solving; clinical decision making.

5

4. SEARCH SOURCES AND STRATEGIES

Search Strategy: Database searches will be performed accordingly to BEME guidelines (Haig & Dozier, 2003). Databases will

be searched using the search terms identified below, using both keyword, MESH terms, and full article search

when available. Conference Proceedings Citation Index- Science (CPCI-S) and Conference Proceedings Citation

Index Social Science & Humanities (CPCI-SSH) and websites of international medical education conferences

(mainly AMEE, RIME Conference-AACM, ASME, APMEC) will also be searched. This search will be limited to a

two year period (2012-2014), as it is reasonable to assume that, older studies, if of enough quality, would

probably have been published elsewhere and therefore will be identified by the journal search. Backward and

forward citation searches may be performed to test the quality of the search strategy, and exceptionally, to

identify relevant references not retrieved in the initial search. Hand searches of particular relevant articles

identified during the article retrieval and extraction process may also be performed (Academic Medicine,

Nursing Education, Medical Education, Medical Teacher, BMJ Patient Safety, and others). Only primary

research will be included in this review. The authors and editors of journals may be approached to obtain data

not available on the article or the supplementary materials.

Databases:

ERIC, Medline, PubMed, Scopus, Psychinfo, Pubmed, Google Scholar, EMBASE, CINAHL.

Time-span of the literature reviewed:

A pilot search conducted in SCOPUS database yielded 1477 articles (search 1 and 2 including duplicates and

all types of articles) from those only 5 were published before 1994, supporting the argument that the interest

in educational interventions is relatively recent compared with research on other aspects of clinical reasoning

(e.g. assessment) (Appendix 5). Therefore search will focus on the last two decades of research, between 1994

and 2014. Backward citation searching may identify studies outside this period in such cases, decisions about

inclusion/exclusion will be made upon their quality and only high quality studies will be exceptionally included.

Language and geography:

Keyword searches will be performed in English but no further language restrictions will be applied. In the

unlikely event of articles in other language that not being retrieved in our searches, drawing on the members

of the group’s language skills, articles in French, Spanish, Portuguese will be reviewed in original language,

other may be translated (using Google translator) if required. If necessary specialist translation service may be

used if translation performed by Google translate does not retrieve the necessary quality required to review of

the study (e.g. translation from Chinese).

No geographical restrictions will be applied. During phase 1, with the support of the librarians in the lead

institution (Swansea University), the search strategy presented here will be iteratively refined and improved,

however, based upon the pilot results it can be anticipated that in phase 1 between 500 to a maximum of

1000 records of potential relevant articles will be handled.

Table 1: Search Terms*

Search terms 1(S1)

clinical reasoning OR clinical decision making OR clinical problems solving OR diagnostic reasoning OR competency OR Diagnostic thinking OR Expert reasoning OR Critical thinking OR Critical analysis OR Situation awareness OR

AND

Promotion OR Teaching OR Pedagogy OR development OR improvement OR learning; OR teaching OR education OR educational OR interventions OR

LIMIT TO (database filters)

Population: Healthcare

professionals (full list below) Time

Period:

6

Schema OR Illness Scripts OR System 1/Pattern recognition OR System 2/hypothetic deductive reasoning OR Forward reasoning OR Backward reasoning OR Bayesian reasoning OR Probabilistic reasoning

educational activities OR educational strategies OR educational methods OR Expertise OR Competence OR

1994 to 2014 Language:

all

Search terms 2(S2)

cognitive bias OR cognitive errors OR heuristics

AND

Reduction OR educational OR interventions educational

OR activities OR educational strategies

7

5. STUDY SELECTION CRITERIA

Population: Initially all domains listed below and levels of healthcare education will be included in the searches. Based on

the quality (criteria below) of articles retrieved, discussion section may focus mainly on domains yielding a

higher number of results to be able to provide a consistent picture of the evidence available. All retrieved

studies will be reported in the findings section.

Evidence of educational interventions at all training levels, undergraduate/initial training to postgraduate

and continuum professional development will be included. Evidence will be classified accordingly to the levels

of training and a ’multiple levels’ category will be included to account for those studies presenting evidence of

educational activities that extend across multiple levels.

List of healthcare professions included: (alphabetical order)

1. Dentistry

2. Medicine

3. Nursing

4. Occupational Therapy

5. Osteopathy

6. Paramedic science

7. Pharmacology

8. Veterinary

Interventions:

Any educational intervention in accordance to the definition used for the purposes of this review with the

explicit aim of promoting clinical reasoning will be included in accordance to the inclusion/exclusion criteria

below will be included.

Study types:

No study will be excluded based on their methodology. All primary research studies reporting educational

interventions at all levels of Cook, Bordage, & Schmidt (2008) framework (description, justification and

clarification).

Studies research design will be classified accordingly to the classification developed by Barr et al.(2000)

(Appendix 3). No grey literature will be included.

Retrieved studies will be rated against the quality criteria in appendix 2.

Outcomes:

Outcomes will be classified according to the Kirkpatrick model adapted by Barr et al. (2000) to use for

systematic review in healthcare education. In order to achieve a holistic view of the research in this field,

identify potential gaps and areas for future research, no study will be excluded on the grounds of outcome

type reported. Such conclusions will be reported in our initial characterization of the field.

The strength of the conclusions identified in each study will be rated on a numerical scale in line with BEME

guidance and a previous systematic review of educational interventions (Gordon & Findley, 2011).

In order to identify messages regarding the effectiveness of educational interventions, studies reporting

highest quality and strength of conclusions will be analysed separately, with more detail and greater emphases

will be placed in their findings. In this analysis outcomes will be also discussed in the light of the findings of

Durning et al.2013 BEME review on assessment of diagnostic reasoning, to ensure consistency between both

BEME reviews. It is expected that this review will be mainly of a narrative nature. However if enough high

quality quantitative studies using with comparable measures are retrieved analysis of weighted effect sizes (at

8

95% confidence interval) possibly using a random-effects model will be calculated and considered as the

primary measure of effect has previously done by other systematic review in health profession education

(Cook et al., 2010).

Inclusion/exclusion criteria:

All of the below Inclusion conditions need to be met for article inclusion:

Clearly defined educational activities/strategies AND

Explicitly aim to develop clinical reasoning OR reduce cognitive bias OR expertise from cognitive

perspective AND

Involve healthcare professionals listed above AND

Provide a detailed description of the intervention to allow for it to be reproduced by others

OR/AND

Be primary research AND Report evaluation data (at any level of Kirkpatrick model adapted by Barr

et al., 2000.)

Exclusion conditions driven from the above inclusion criteria (excluded if)

NOT reporting on clearly identified educational intervention explicitly aimed at promoting clinical

reasoning;

Other reviews (if of high quality these maybe used for backward citation searching);

Articles reporting assessment of clinical reasoning (as that is covered in other BEME review) without

explicit mentioning of educational intervention to teach/develop it;

Any research reporting on design, validation or psychometric characteristics of assessments clinical

reasoning (as that is covered in other BEME review);

Articles using clinical reasoning as one of the outcome measure in curriculum comparisons:

Any research reporting on educational activities to develop reasoning and not CLINICAL reasoning;

Studies describing interventions to reduce cognitive bias or support decision making that are not

educational nature, such as changes in the workplace, introduction of computerized decision

making systems, use of electronic healthcare records or development of mathematic algorithms

to improve test interpretation.

9

6. PROCEDURE FOR EXTRACTING DATA

Data extraction will take place in two phases: first an iterative identification and screening of records in the

databases will take place. Title and abstracts will be scrutinized and a decision made about inclusion/exclusion

(phase 1); the second phase will involve assessment of eligibility and the final decision on inclusion. Full

articles and supplementary materials will be retrieved from databases, their eligibility for review assessed, and

those included coded and rated based on their methodological quality and strength of their findings (phase 2).

Distinction will be made between those to be included in the overall synthesis and those high-quality papers

which will be analysed and reported in greater detail. If necessary, during phase 2, backward citation

searching and hand searching may be used to identify and retrieve articles if not identified during initial

searches. Coding sheets are presented in Appendix 4 and 5.

The review lead, with the support of the librarians and a research assistant at the host institution (Swansea

University) will be responsible for the initial searches.

Reference manager software will be used to store and manage the retrieved references. Two independent

reviewers will independently, blindly review each record and decide on its inclusion or exclusion, providing

reasons for their choice. In cases of disagreement, consensus will be reach by discussion between reviewers.

Cohen's kappa coefficient (k) will be calculated. Full article extraction will be performed.

Phase 2 will follow a similar process. Reviewers will be paired and each pair will make an independent

blinded assessment of the papers eligibility for the study. As previously, consensus will be reached by

discussion between reviewers. The full papers to be included will then be redistributed for coding and quality

assessment based upon the members’ interests and methodological expertise.

The lead reviewer will be responsible for collating the information and maintaining the databases during all

steps of the review.

If the number of studies reporting quantitative measures of outcomes justifies, Rev.Man5 and GradePro may

be used to analyzed and report data of such nature.

TRAINING

Training needs will be identified on an individual basis and reviewers will be directed towards the

appropriate online training available from Campbell Collaboration (http://www.campbellcollabo

ration.org/resources/training.php),COCHRANE LEARNING (www.cochranelearning.com) and other specific

learning resources by The Cochrane Collaboration, and other sources of online training on particular aspects of

the review (e.g. assessment of rigor quality of qualitative studies). Funding may be sought for training sessions,

or development of online training packs, in such case these will be made freely available to others conducting

other BEME review s training.

10

7. SYNTHESIS OF EXTRACTED EVIDENCE – PRISMA

Reporting and presentation of findings will follow the adapted the PRISMA (Preferred Reporting Items for

Systematic reviews and Meta-Analyses) guidance (Liberati et al., 2009; Moher et al., 2009) and research

questions posed. The final summary of the findings will be structured according to the three objectives of the

review stated earlier.

All studies will be classified according to Cook, Bordage, & Schmidt (2008) framework into description,

justification and clarification studies.

All studies will be analyzed using content analysis (thematic analysis) (Denzin & Lincoln, 2000) in order to

provide a summary of themes associated with definitions of clinical reasoning use by different healthcare

professions3, description of the interventions, the theoretical frameworks underpinning such activities. The

aim here will not be to look for effectiveness but rather provide details about the nature and salient features

and context of such interventions.

Justification and clarification studies (quantitative and qualitative) will be categorised by the type of

educational activity reported, methodological choices and outcome reported using the Kirkpatrick model as

adapted by Barr et al.(2000). Special attention been paid to the description of the heterogeneity between

studies and possible different between different healthcare professions, such aspects will be discussed

separately within the discussion section . Qualitative studies will be synthesised using thematic analysis and

used to inform discussion of quantitative data (Bearman & Dawson, 2013).

It is expected that this review will be mainly of a narrative nature. However if enough high quality quantitative

studies are found with comparable measures (allowing standardization of the mean to be performed) are

retrieved these will be analysed separately. In this analysis, weighted effect sizes (at 95% confidence interval)

possibly using a random-effects model will be calculated and considered as the primary measure of effect has

previously done by other systematic reviews in healthcare education (Cook at al. 2009). In these studies

heterogeneity will be analysed using I2 statistic as described by (Liberati et al., 2009).

High quality studies will be discussed in more detail in order to answer the main research questions as it is

appropriate to place greater confidence in their findings. If possible, depending on retrieved data, for this

group of interventions the review will also consider the resources required to implement such interventions.

The PRISMA checklist (Appendix 1) will be used to monitor changes and developments carried out during the

review phases and a detailed record of potential sources of bias and limitations of the review will be kept and

presented along with its findings.

8. PROJECT TIMETABLE

The project timetable is presented below in Table 2. The timeline reflects the processes described in previous sections and, importantly, allows for conclusions from the Durning et al. 2013 BEME on assessment of diagnostic reasoning (due to be published by January 2015) to be taken in consideration and inform the assessment of the outcomes and data analysis and synthesis.

9. CONFLICT OF INTEREST STATEMENT

No conflicts of interest to report.

10. PLANS FOR UPDATING THE REVIEW

Updating the review will be undertaken in accordance with BEME guidelines.

11. CHANGES TO THE PROTOCOL

Any changes to the protocol will be recorded on the adapted PRISMA checklist (appendix 1) along with the

rationale for their need.

3 This will be done in collaboration with the Dr.Durning review group. One of the aims of this group is to investigate

definitions of clinical reasoning in medicine.

11

Table 2: Project timetable

2015

Jul

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sear

ch a

nd

dat

a ex

trac

tio

n

Phase 1:

Testing of data extraction form (coding

sheet) already ongoing

Search optimisation

Identification

Screening: title + abstract

Testing of data extraction process

Full article retrieval

Phase 2 (full article):

Assessment of eligibility and data

extraction

Quality and RoB assessment

Dat

a

anal

ysis

Discussion of findings meeting

Data analysis and synthesis

Rep

ort

an

d

pu

blic

atio

n

Preparation of report and manuscripts

PRISMA sections

1 -17

PRISMA

sections 17-

29

Manuscript and report

Presentation of findings at AMEE

12

12. REFERENCES

Ajjawi, R. & Higgs, J. (2012) Core components of communication of clinical reasoning: a qualitative study with experienced Australian physiotherapists. Advances in health sciences education : theory and practice. [Online] 17 (1), 107–119. Available from: doi:10.1007/s10459-011-9302-7 [Accessed: 2 December 2012].

Ajjawi, R. & Higgs, J. (2008) Learning to reason: a journey of professional socialisation. Advances in health sciences education : theory and practice. [Online] 13 (2), 133–150. Available from: doi:10.1007/s10459-006-9032-4 [Accessed: 13 February 2013].

Ajjawi, R. & Higgs, J. (2007) Using Hermeneutic Phenomenology to Investigate How Experienced Practitioners Learn to Communicate Clinical Reasoning. 12 (4), 612–638.

Ark, T.K., Brooks, L.R. & Eva, K.W. (2006) Giving learners the best of both worlds: do clinical teachers need to guard against teaching pattern recognition to novices? Academic medicine : journal of the Association of American Medical Colleges. [Online] 81 (4), 405–409. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16565197.

Ark, T.K., Brooks, L.R. & Eva, K.W. (2007) The benefits of flexibility: the pedagogical value of instructions to adopt multifaceted diagnostic reasoning strategies. Medical education. [Online] 41 (3), 281–287. Available from: doi:10.1111/j.1365-2929.2007.02688.x [Accessed: 2 September 2010].

Barr, H., Freeth, D., Hammick, M., Koppel, I., et al. (2000) Evaluations of interprofessional education: a United Kingdom review for health and social care. [Online] (August). Available from: http://www.opengrey.eu/item/display/10068/551729 [Accessed: 13 March 2014].

Bearman, M. & Dawson, P. (2013) Qualitative synthesis and systematic review in health professions education. Medical education. [Online] 47 (3), 252–260. Available from: doi:10.1111/medu.12092 [Accessed: 13 May 2014].

Borleffs, J.C.C., Custers, E.J.F.M., van Gijn, J. & ten Cate, O.T.J. (2003) “Clinical reasoning theater”: a new approach to clinical reasoning education. Academic medicine : journal of the Association of American Medical Colleges. [Online] 78 (3), 322–325. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12634216.

Boshuizen, H.P.A.P., Schmidt, H.G.G., Hp, B. & On, S.H.G. (1992) On the role of biomedical knowledge in clinical reasoning by experts, intermediates and novices. Cognitive Science: A Multidisciplinary Journal. [Online] 16 (2), 153–184. Available from: doi:10.1016/0364-0213(92)90022-M [Accessed: 6 September 2011].

Bowen, J.L. (2006) Educational strategies to promote clinical diagnostic reasoning. The New England journal of medicine. [Online] 355 (21), 2217–2225. Available from: doi:10.1056/NEJMra054782 [Accessed: 28 September 2011].

Brown, J.S., Collins, A., Duguid, P. & Seely, J. (1989) Situated Cognition and the Culture of Learning. Educational Researcher. [Online] 1832–42. Available from: doi:10.2307/1176008.

Buckley, S., Coleman, J., Davison, I., Khan, K.S., et al. (2009) The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Medical teacher. [Online] 31282–298. Available from: doi:10.1080/01421590902889897.

13

Charlin, B., Boshuizen, H.P.A., Custers, E.J. & Feltovich, P.J. (2007) Scripts and clinical reasoning. Medical education. [Online] 41 (12), 1178–1184. Available from: doi:10.1111/j.1365-2923.2007.02924.x [Accessed: 6 July 2011].

Charlin, B., Tardif, J. & Boshuizen, H.P. (2000) Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Academic medicine : journal of the Association of American Medical Colleges. [Online] 75 (2), 182–190. Available from: http://www.ncbi.nlm.nih.gov/pubmed/10693854 [Accessed: 4 November 2010].

Cook, D. a, Erwin, P.J. & Triola, M.M. (2010) Computerized virtual patients in health professions education: a systematic review and meta-analysis. Academic medicine : journal of the Association of American Medical Colleges. [Online] 85 (10), 1589–1602. Available from: doi:10.1097/ACM.0b013e3181edfe13 [Accessed: 21 February 2014].

Cook, D.A., Bordage, G. & Schmidt, H.G. (2008) Description, justification and clarification: a framework for classifying the purposes of research in medical education. Medical Education. [Online] 42 (2), 128–133. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18194162.

Cook, D.A. & Triola, M.M. (2009) Virtual patients: a critical literature review and proposed next steps. Medical education. [Online] 43 (4), 303–311. Available from: doi:10.1111/j.1365-2923.2008.03286.x.

Croskerry, P. (2009) A universal model of diagnostic reasoning. Academic medicine : journal of the Association of American Medical Colleges. [Online] 84 (8), 1022–1028. Available from: doi:10.1097/ACM.0b013e3181ace703.

Croskerry, P. (2003) The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them. Academic Medicine. [Online] 78 (8), 775–780. Available from: doi:10.1097/00001888-200308000-00003.

Croskerry, P., Singhal, G. & Mamede, S. (2013) Cognitive debiasing 2: impediments to and strategies for change. BMJ quality & safety. [Online] 22 Suppl 2ii65–ii72. Available from: doi:10.1136/bmjqs-2012-001713 [Accessed: 3 March 2014].

Custers, E.J.F.M., Regehr, G. & Norman, G.R. (1996) Mental Representations of Medical Diagnostic Knowledge: A Review. Acad Med. 71 (10), S55–S61.

Delany, C., Golding, C. & Bialocerkowski, A. (2013) Teaching for thinking in clinical education: Making explicit the thinking involved in allied health clinical reasoning. Focus on Health Professional Education A Multidisciplinary Journal. [Online] 14 (2), 44. Available from: http://search.informit.com.au/documentSummary;dn=344079003268797;res=IELHEA.

Dent, J.A. & Harden, R.M. (2005) A practical guide for medical teachers. University of Michigan, Elsevier Churchill Livingstone.

Denzin, N.K. & Lincoln, Y.S. (2000) The Sage handbook of qualitative research. 2nd edition. Norman K. Denzin & Yvonna S. Lincoln (eds.). [Online]. Thousand Oaks, California, Sage Publications, Inc.

Dequeker, J. & Jaspaert, R. (1998) Teaching problem-solving and clinical reasoning: 20 years experience with video-supported small-group learning. Medical education. [Online] 32 (4), 384–389. Available from: http://www.ncbi.nlm.nih.gov/pubmed/9743800.

Durak, H.I., Caliskan, S.A., Bor, S. & Van der Vleuten, C. (2007) Use of case-based exams as an instructional teaching tool to teach clinical reasoning. Medical teacher. [Online] 29 (6), e170–4. Available from: doi:10.1080/01421590701506866 [Accessed: 23 August 2010].

14

Durning, S., Artino, A.R., Pangaro, L., van der Vleuten, C.P., et al. (2011) Context and clinical reasoning: understanding the perspective of the expert’s voice. Medical education. [Online] 45 (9), 927–938. Available from: doi:10.1111/j.1365-2923.2011.04053.x [Accessed: 24 August 2011].

Durning, S.J. & Artino, A.R. (2011) Situativity theory: a perspective on how participants and the environment can interact: AMEE Guide no. 52. Medical teacher. [Online] 33 (3), 188–199. Available from: doi:10.3109/0142159X.2011.550965 [Accessed: 19 February 2014].

Durning, S.J., Artino, A.R., Boulet, J.R., Dorrance, K., et al. (2012) The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Advances in health sciences education : theory and practice. [Online] 17 (1), 65–79. Available from: doi:10.1007/s10459-011-9294-3 [Accessed: 31 July 2013].

Elizondo-Omaña, R.E. & López, S.G. (2008) The development of clinical reasoning skills: a major objective of the anatomy course. Anatomical sciences education. [Online] 1 (6), 267–268. Available from: doi:10.1002/ase.57.

Elstein, A., Shulman, L.S. & Sprafka, S.A. (1978) Medical Problem Solving: an Analysis of Clinical Reasoning. Cambridge, MA, Harvard Un. Press.

Elstein, A.S. (2009) Thinking about diagnostic thinking: a 30-year perspective. [Online] 14 Suppl 17–18. Available from: doi:10.1007/s10459-009-9184-0.

Epstein, R.M. (2007) Assessment in medical education. The New England journal of medicine. [Online] 356 (20), 2109; author reply 2110. Available from: doi:10.1056/NEJMra054784.

Eva, K.W. (2005) What every teacher needs to know about clinical reasoning. Medical education. [Online] 39 (1), 98–106. Available from: doi:MED1972 [pii] 10.1111/j.1365-2929.2004.01972.x.

Eva, K.W., Hatala, R.M., Leblanc, V.R. & Brooks, L.R. (2007) Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information. Medical education. [Online] 41 (12), 1152–1158. Available from: doi:10.1111/j.1365-2923.2007.02923.x.

Eva, K.W., Wood, T.J., Riddle, J., Touchie, C., et al. (2010) How clinical features are presented matters to weaker diagnosticians. Medical education. [Online] 44 (8), 775–785. Available from: doi:10.1111/j.1365-2923.2010.03705.x [Accessed: 19 July 2011].

Eva, K.W.W., Norman, G.R.R., Neville, A.J.J., Wood, T.J.J., et al. (2002) Expert-Novice Differences in Memory: A Reformulation. Teaching and Learning in Medicine. [Online] 14 (4), 257–263. Available from: http://www.tandfonline.com/doi/abs/10.1207/S15328015TLM1404_10 [Accessed: 8 December 2011].

Evans, J.S. (1984) Heuristic and analytic processes in reasoning. British Journal of Psychology. [Online] 75451–468. Available from: doi:10.1111/j.2044-8295.1984.tb01915.x.

Gobet, F. (2007) A pattern-recognition theory of search in expert problem solving. Thinking Reasoning. [Online] 3 (4), 291–313. Available from: http://dx.doi.org/10.1080/135467897394301.

Gordon, M. & Findley, R. (2011) Educational interventions to improve handover in health care: a systematic review. Medical education. [Online] 451081–1089. Available from: doi:10.1111/j.1365-2923.2011.04049.x.

Graber, M. (2003) Metacognitive training to reduce diagnostic errors: ready for prime time? Academic medicine : journal of the Association of American Medical Colleges. [Online] 78 (8), 781. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12915364.

15

Graber, M.L., Franklin, N. & Gordon, R. (2005) Diagnostic error in internal medicine. Archives of internal medicine. [Online] 165 (13), 1493–1499. Available from: doi:10.1001/archinte.165.13.1493.

Graber, M.L., Kissam, S., Payne, V.L., Meyer, A.N.D., et al. (2012) Cognitive interventions to reduce diagnostic error: a narrative review. BMJ quality & safety. [Online] 21 (7), 535–557. Available from: doi:10.1136/bmjqs-2011-000149.

Haig, A. & Dozier, M. (2003) BEME guide no. 3: systematic searching for evidence in medical education--part 2: constructing searches. [Online] 25 (5), 463–484. Available from: doi:10.1080/01421590310001608667 Q8AP0TL08X7J33LY [pii].

Hammick, M.R. (2005) Evidence-informed education in the health care science professions. Journal of veterinary medical education. [Online] 32399–403. Available from: doi:10.3138/jvme.32.4.399.

Harris, A., Boyce, P. & Ajjawi, R. (2011) Clinical reasoning sessions: back to the patient. The clinical teacher. [Online] 8 (1), 13–16. Available from: doi:10.1111/j.1743-498X.2010.00407.x.

Higgs, J. (1992) Developing Clinical Reasoning Competencies. Physiotherapy. [Online] 78 (8), 575–581. Available from: doi:10.1016/s0031-9406(10)61202-3.

Higgs, J. & Jones, M.A. (2000) Clinical Reasoning in Health Professions. 2nd edition. Joy Higgs & Mark A. Jones (eds.). [Online]. Elsevier Health Sciences. Available from: http://books.google.com/books?id=YfJmzMbX9jIC&pgis=1 [Accessed: 29 November 2010].

Higgs, J., Jones, M.A. & Christensen, N. (2008) Clinical Reasoning in Health Professions. 3rd edition. Elsevier.

Holyoak, K.J. & Morrison, R.G. (2012) Thinking and reasoning: A reader’s guide. In: K J Holyoak & Robert G Morrison (eds.). Oxford Handbook of Thinking and Reasoning. [Online]. New York, U.S.A., Oxford University Press.

Irby, D.M. & Bowen, J.L. (2004) Time-efficient strategies for learning and performance. The Clinical Teacher. [Online] 1 (1), 23–28. Available from: doi:10.1111/j.1743-498X.2004.00013.x.

Jensen, G., Resnik, L. & Haddad, A. (2008) Expertise in Clinical Reasoning. In: Joy Higgs, Mark A. Jones, S Loftus, & N Christensen (eds.). Clinical Reasoning in the Healthcare Professions. 3rd edition. Philadelphia/US, Elsevier.

Kassirer, J.P. (2010) Teaching Clinical Reasoning : Case-Based and Coached. Academic Medicine. 85 (7), 1118–1124.

Kim, S.Y., Park, J.E., Lee, Y.J., Seo, H.-J., et al. (2013) Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. Journal of clinical epidemiology. [Online] 66 (4), 408–414. Available from: doi:10.1016/j.jclinepi.2012.09.016 [Accessed: 12 March 2014].

Kuiper, R., Heinrich, C., Matthias, A., Graham, M.J., et al. (2008) Debriefing with the OPT model of clinical reasoning during high fidelity patient simulation. [Online] 5Article17. Available from: doi:10.2202/1548-923X.1466.

Kurzenhäuser, S. & Hoffrage, U. (2002) Teaching Bayesian reasoning: an evaluation of a classroom tutorial for medical students. Medical teacher. [Online] 24 (5), 516–521. Available from: doi:10.1080/0142159021000012540 [Accessed: 9 September 2011].

16

Lave, J. & Wenger, E. (1991) Situated learning: Legitimate peripheral participation. [Online]. Available from: doi:10.2307/2804509.

Liberati, A., Altman, D., Tetzlaff, J., Mulrow, C., et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. BMJ (Online). [Online] Available from: doi:10.1136/bmj.b2700 [Accessed: 14 March 2014].

Mamede, S., Schmidt, H.G. & Penaforte, J.C. (2008) Effects of reflective practice on the accuracy of medical diagnoses. Medical education. [Online] 42 (5), 468–475. Available from: doi:10.1111/j.1365-2923.2008.03030.x [Accessed: 2 September 2010].

McMillan, W.J. (2010) Teaching for clinical reasoning - helping students make the conceptual links. Med Teach. 32 (10), e436–42.

Moher, D., Liberati, A., Tetzlaff, J. & Altman, D.G. (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. [Online] 339 (august), 332–336. Available from: doi:10.1136/bmj.b2535 [Accessed: 11 March 2014].

Newman, M., Learning, C.C.S.R.G. on the effectiveness of P.B. & University, M. (2003) A pilot systematic review and meta-analysis on the effectiveness of Problem Based Learning. [Online]. p.74. Available from: doi:0 7017 0158 7.

Norman, G. (2009) Dual processing and diagnostic errors. Advances in health sciences education : theory and practice. [Online] 14 Suppl 1 (0), 37–49. Available from: doi:10.1007/s10459-009-9179-x.

Norman, G.R. (2006) Building on experience--the development of clinical reasoning. The New England journal of medicine. [Online] 355 (21), 2251–2252. Available from: doi:10.1056/NEJMe068134.

Norman, G.R. (2005) Research in clinical reasoning: past history and current trends. Medical education. [Online] 39 (4), 418–427. Available from: doi:10.1111/j.1365-2929.2005.02127.x.

Norman, G.R., Barraclough, K., Dolovich, L. & Price, D. (2009) Iterative diagnosis. BMJ. [Online] 339 (sep22 1), b3490–b3490. Available from: doi:10.1136/bmj.b3490.

Norman, G.R. & Eva, K.W. (2010) Diagnostic error and clinical reasoning. Medical education. [Online] 44 (1), 94–100. Available from: doi:10.1111/j.1365-2923.2009.03507.x [Accessed: 2 September 2010].

Norman, G.R. & Eva, K.W. (2005) Does clinical experience make up for failure to keep up to date? Evidence Based Medicine. [Online] 10 (3), A8–9. Available from: doi:ACPJC-2005-142-3-A08 [pii].

Norman, G.R., Young, M. & Brooks, L.R. (2007) Non-analytical models of clinical reasoning: the role of experience. Medical education. [Online] 41 (12), 1140–1145. Available from: doi:10.1111/j.1365-2923.2007.02914.x [Accessed: 9 March 2012].

Patel, V.L., Evans, D.A. & KAUFMAN, D.R. (1990) Reasoning strategies and the use of biomedical knowledge by medical students. Medical Education. [Online] 24 (2), 129–136. Available from: doi:10.1111/j.1365-2923.1990.tb02511.x [Accessed: 2 September 2010].

Patrício, M. & vaz Carneiro, A. (2012) Systematic reviews of evidence in medical education and clinical medicine: is the nature of evidence similar? Medical teacher. [Online] 34 (6), 474–482. Available from: doi:10.3109/0142159X.2012.682186 [Accessed: 28 January 2014].

17

Radomski, N. & Russell, J. (2010) Integrated case learning: teaching clinical reasoning. Advances in health sciences education : theory and practice. [Online] 15 (2), 251–264. Available from: doi:10.1007/s10459-009-9194-y [Accessed: 30 August 2010].

Reiter, H.I., Eva, K.W., Hatala, R.M. & Norman, G.R. (2002) Self and peer assessment in tutorials: application of a relative-ranking model. Academic Medicine. [Online] 77 (11), 1134–1139. Available from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=12431928.

Rencic, J. (2011) Twelve tips for teaching expertise in clinical reasoning. Medical Teacher. [Online] 33 (11), 887–892. Available from: doi:10.3109/0142159X.2011.558142.

Rikers, R.M.J.P. & Verkoeijen, P.P.J.L. (2007) Clinical expertise research: a history lesson from those who wrote it. [Online] 41 (12), 1115–1116. Available from: doi:doi:10.1111/j.1365-2923.2007.02920.x.

Schmidt, H.G., Norman, G.R. & Boshuizen, H.P.A. (1990) A Cognitive Perspective on Medical Expertise: Theory and Implications. Acad Med. 65 (10), 611–621.

Sherbino, J., Dore, K.L., Wood, T.J., Young, M.E., et al. (2012) The relationship between response time and diagnostic accuracy. Academic medicine : journal of the Association of American Medical Colleges. [Online] 87 (6), 785–791. Available from: doi:10.1097/ACM.0b013e318253acbd [Accessed: 3 February 2013].

Smith, M., Higgs, J. & Ellis, E. (1991) Factors influencing clinical decision making. 89–100.

Stieger, S., Praschinger, A., Kletter, K. & Kainberger, F. (2009) Diagnostic grand rounds: A new teaching concept to train diagnostic reasoning. European journal of radiology. [Online] In Press,. Available from: doi:10.1016/j.ejrad.2009.05.015.

Stiegler, M.P., Neelankavil, J.P., Canales, C. & Dhillon, a (2011) Cognitive errors detected in anaesthesiology: a literature review and pilot study. British journal of anaesthesia. [Online] 108 (December 2011), 229–235. Available from: doi:10.1093/bja/aer387 [Accessed: 17 January 2012].

Struyf, E., Beullens, J., Van Damme, B., Janssen, P., et al. (2005) A new methodology for teaching clinical reasoning skills: problem solving clinical seminars. [Online] 27 (4), 364–368. Available from: doi:RK674L2J7875NNLX [pii] 10.1080/01421590500046411.

Thompson, C. & Stapley, S. (2011) Do educational interventions improve nurses’ clinical decision making and judgement? A systematic review. International Journal of Nursing Studies. [Online]. 48 pp.881–893. Available from: doi:10.1016/j.ijnurstu.2010.12.005.

Trowbridge, R.L., Dhaliwal, G. & Cosby, K.S. (2013) Educational agenda for diagnostic error reduction. BMJ quality & safety. [Online] 22 Suppl 2ii28–ii32. Available from: doi:10.1136/bmjqs-2012-001622 [Accessed: 15 February 2014].

Windish, D., Price, E., Clever, S., Magaziner, J., et al. (2005) Teaching medical students the important connection between communication and clinical reasoning. Journal of General Internal Medicine. [Online] 20 (12), 1108–1113. Available from: http://dx.doi.org/10.1111/j.1525-1497.2005.0244.x.

Woods, N.N., Brooks, L.R. & Norman, G.R. (2007) The role of biomedical knowledge in diagnosis of difficult clinical cases. [Online] 12 (4), 417–426. Available from: doi:10.1007/s10459-006-9054-y.

18

APPENDIX 1: PRISMA CHECKLIST ADAPTED FROM LIBERATI ET AL. (2009)

Milestones in the review/updates to planed review sections Description of the current review details (planned at protocol stage)

Were any changes to the

protocol requires? Provide rational

Phase 1

Phase 2

Title 1 Educational Interventions To Promote/Teach Clinical Reasoning: A Best Evidence In Medical Education Systematic Review (BEME) Abstract

Structured summary 2

We will provide a structured summary of the background, objectives, data sources, study eligibility criteria, populations, types of educational activities reviewed, health professions covered, study appraisal/quality indicators and synthesis methods, results, limitations, conclusions and implications for both future research and practice. Summary of outcomes will be grouped by type of educational activity to increase readability and usefulness of the results.

Introduction

Rationale 3 A brief rationale is provided in pages 2 and 3 of the current protocol. This section will be expanded and updated in the final report to account for possible developments, mainly those identified in another BEME review by Durning et al.2013 currently in progress.

Objectives 4 The objectives of the current review are stated on page 5 of the protocol. Methods

Protocol and registration 5 All materials produced by the current review will be available on the BEME website and written according to BEME guidance.

Eligibility criteria 6

Eligibility criteria is presented in the current protocol (page 7). No study will be excluded on the basis of its methodology or type of outcome measured, rather those will be coded and included in the review as an overview of the research in the field.

Information sources 7

Databases searched are included in section 4 along with a description of the search strategies used. The current review will focus on the period between 1994 and 2014/2015. Studies looking at all healthcare professions will be included in initial search, however if necessary, for feasibility reasons, the articles retrieved will be limited to the professions where a higher number of studies has been conducted (likely to be medicine, nursing, dentistry, veterinary, physiotherapy). This will be clearly stated and reported in the report of our review.

Search 8 Search strategy, terms and databases searched are presented on page 6 of the protocol. Study

selection 9 All study records from phase 1 and retrieved studies in phase 2 will reviewed by two independent reviewers, consensus will be achieved by discussion. The process for selecting studies screening, eligibility and decision about inclusion in the systematicreview is

19

described on page 9 of the protocol and further details will be provided in the final report. Coding sheets are presented in appendix 4. A final diagram will be presented identifying the number of records and articles identified, screened, assessed for eligibility and included in the review.

Data collection process 10

Method of data extraction, including data extraction forms (appendix 3) and any changes made to the testing phase will be reported. Software to collate data from the paired reviewers, to calculate inter-rater reliability (Cohen’s Kappa) will be identified. It is not anticipated that data from the studies will need to be obtained from investigators, however if required, the authors and editors of journals may be approached to obtain data not available on the article or the supplementary materials.

Data items 11

All classification items used to code the studies are presented in appendix 3 and 4. Any simplifications (categorisation of data or standardisation of measures) made will be presented and justified.

Risk of bias in individual studies 12

Quality of the studies will be assessed by the checklist (appendix 2) adapted from a previous BEME reviews (Buckley et al., 2009) No studies will be excluded on the basis of their methodological quality, except for those not reporting any quality measures. Quantitative, high quality studies’ risk of bias (RoB) will be assessed by an independent reviewer according to either the Cochrane Risk of Bias for Randomized or Non-Randomized studies (Kim et al., 2013). The results will be reported as a separated section within findings section.

Summary measures 13

It is expected that this review will be mainly of a narrative nature. An initial analysis of the definitions, salient features and frameworks of interventions will be carried out by content analysis (thematic analysis). If enough high quality quantitative justification and clarification studies are found with comparable measures (allowing standardization of the mean to be performed) are retrieved these will be analysed separately. In this analysis, weighted effect sizes (at 95% confidence interval) possibly using a random-effects model will be calculated and considered as the primary measure of effect has previously done by other systematic reviews in healthcare education (Cook at al. 2009).

Planned methods for analysis 14

All included studies (quantitative and qualitative) will be categorised by the type of educational activity reported, methodological choices and outcome reported using the Kirkpatrick model as adapted by Barr et al.(2000). Qualitative studies will be synthesised using thematic analysis and used to inform discussion of quantitative data (Bearman & Dawson, 2013). If enough studies retrieved allow for a meta-analysis to be performed (which we do not anticipate) heterogeneity will be analysed using I

2 statistic as described by (Liberati et al., 2009).

Risk of bias across studies 15

Possible risk of bias likely to impact on the review will be identified and discussed. At this early stage of the review, it is possible to identify a few possible risks we account for: i) publication bias and author (as pilot search shows a very high percentage of the papers has been published by only a few journals/authors therefore more likely to be susceptible to this type of bias) ii)poor indexing, may mean searches are difficult iii) possible selective reporting of findings. Careful coding of the research retrieved will be made to account for such possible sources of bias and to identify others during the review process, the steps taken will be described in the final reports.

Additional analyses 16 If during the review process additional analyses are necessary the methods used will be described in detail in this section.

Results

Study selection 17

Selection criteria and search strategies are described in page 7 of the protocol. At later stages of this review this section will report numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage. This will be present as a follow diagram in accordance to PRISMA guidance.

20

Study characteristics 18

Coding sheets were designed to allow characterisation of different types of studies and their methodological choices (See coding sheet/extraction form appendix 3). No studies will be excluded on due to methodological choices or quality (as long as reporting quality measures).

Risk of bias within studies 19

Study quality and risk of bias will be assessed as described in the protocol. Additionally, comment boxes will be included to allow reviewers to include extracts from the originals to support their judgement.

Results of individual studies 20

For all the outcome levels considered (Kirkpatrick model) and educational intervention type, per study, a simple summary of characteristics will be presented. Additionally, if enough high-quality quantitative studies are retrieved to justify a meta-analysis, statistical data will be presented as weighted effect estimates and a confidence intervals forest plot.

Synthesis of results 21

The summary of results will be based upon the defined aims of the current review as described in the protocol. Summary of results, both quantitative and qualitative, will take a tabular format, along with narrative description of the findings. For any meta-analyses, if done, confidence intervals and measures of consistency (I2 heterogeneity) will be reported.

Risk of bias across studies 22

Identified quality queries and risk of bias across studies will be reported. If any meta-analysis are conducted and heterogeneity levels between studies are high those should be explored using adequate methods (e.g. funnel plot).

Additional analysis 23

Currently no additional analyses are planned, but it may be that during the review process those are found to be necessary, there results will be reported here.

Discussion/conclusion

Summary of evidence 24

A summary of the main findings will be guided by the aims of the review. It will provide a overview table with results from the holistic analysis of the research in this field and the strength of the evidence found. From the subgroup of high-quality studies, those in which it is more appropriate to place greater confidence, a more detailed description of results will be provided, by type of educational intervention in relation to the key groups included in the review (e.g. different healthcare professions, levels of training), contexts and if possible the resources required for implementation. Finally, the relevance to stakeholders (such as healthcare educators and providers, institutions and policy makers) educators and researchers will be considered and recommendations drawn (items 27 and 28).

Limitations 25

A discussion of the limitations of the current review will be provided under this section. Here we will also report on a evaluation of the review process, focusing on key questions about the process of the review and how significant for practice were those findings.

Conclusions 26

In this section we will provide a general interpretation of the results in the context of other evidence, and implications for future research.

Recommendations For Practice* 27

These sections were added to the PRISMA checklist based upon BEME recommendations that the reviews conducted in this context should aim to provide guidance to those in practice. In these section we will include key messages for both practice and research in this field.

Recommendations For Research* 28

Funding 29

External funding maybe be sought to support particular activities (e.g. meetings, research support, database management). Any sources of funding, support from the institutions and possible role of funders will be described under this section.

21

* Added to original PRISMA checklist.

22

APPENDIX 2: QUALITY CHECKLIST

Adapted from Buckley et al. 2009 BEME review.

Quality Indicator Question/descriptor:

Ge

ne

ral (

all a

rtic

les)

Research question Is the research question or hypothesis clearly stated? YES / NO /UNCLEAR

Study participants Is the participant group appropriate for the study being carried out (number, characteristics, selection, and homogeneity)? YES / NO /UNCLEAR

Sampling strategy is clearly outlined. YES / NO /UNCLEAR

Methodological choices/Type of study

The methodology used is clearly stated as well as the reasons for the choice made with reference to the research questions and aims of the research. YES / NO /UNCLEAR

‘Data’ collection methods

Chosen methods are coherent and follow logically from the research questions and methodological choices made. YES / NO /UNCLEAR

Are the methods used (qualitative or quantitative) reliable and valid for the research question and context? YES / NO /UNCLEAR

Completeness of ‘data’

Applies to both qualitative and quantitative studies. Have subjects dropped out? Is the attrition rate less than 50%? YES / NO /UNCLEAR

For questionnaire based studies, is the response rate acceptable (60% or above?) YES / NO /UNCLEAR

Control for confounding

Have multiple factors/variables been removed or accounted for where possible? YES / NO /UNCLEAR

Analysis of results Are the statistical or other methods of results analysis used appropriate?

YES / NO /UNCLEAR

Reporting of results Are all measures/results of interest reported? (or is there a selective reporting of outcomes?) YES / NO /UNCLEAR

Limitations Study discusses/recognises possible sources of bias and how those were addressed? YES / NO /UNCLEAR

Conclusions Low--------------------------------------------------------------------------------- High

1 2 3 4 5

23

1 – No clear conclusions can be drawn. Not significant 2 – Results ambiguous, but there appears to be a trend. 3 – Conclusions can probably be based on the results. 4 – Results are clear and very likely to be true. 5 – Results are unequivocal.

Reproducibility Could the study be repeated by other researchers? YES / NO /UNCLEAR

Prospective Does the study look forwards in time (prospective) rather than retrospective? YES / NO /UNCLEAR

Peer review Has the paper been peer reviewed? YES / NO /UNCLEAR

Ethical issues Were all relevant ethical issues addressed YES / NO /UNCLEAR

Was ethical approval sought for the study? YES / NO /UNCLEAR

Triangulation Were results supported by data from more than one source? YES / NO /UNCLEAR

Qu

anti

tati

ve s

tud

ies

(on

ly, l

eav

e b

lan

k fo

r q

ual

itat

ive

stu

die

s)

Hypotheses Are clear and theory driven? YES / NO /UNCLEAR

Instruments psychometric characteristics

Uses valid instruments/tests to measure outcomes? YES / NO /UNCLEAR

Reports validity? YES / NO /UNCLEAR

Reports reliability data on the instruments used? YES / NO /UNCLEAR

Sampling Reports power statistics and/or sampling strategy used ? YES / NO /UNCLEAR

Confounding variables

Study presents adequate confirmation and consideration of possible confounding variables? YES / NO /UNCLEAR

Are there any confounding variable not accounted for that may have impacted the study findings. YES / NO /UNCLEAR

Replicability Would this study be replicable by others? YES / NO /UNCLEAR

Does this article report a

RCT > Use Risk of Bias Assessment tool for randomized Studies (RoBANS)

Cohort study, Non-randomized controlled trial, Case-control study or a Before-after study > Use Risk of Bias Assessment tool for randomized Studies (RoBANS) (Kim et al. 2013)

24

Qu

alit

ativ

e s

tud

ies

(on

ly)

Trustworthiness Are the procedures taken in order to ensure trustfulness clearly described? YES / NO /UNCLEAR Transferability Does the study discuss how the same ideas could be applied

more widely in an adequate manner ? YES / NO /UNCLEAR

Dependability Does the study discuss how dependability was achieved ?(audit trails, rich documentation, triangulation, etc.) YES / NO /UNCLEAR

Confirmability Does the study identifies strategies used to control for researcher bias (e.g. researcher self-reflection on potential bias, peer review, or consultations) ? YES / NO /UNCLEAR

Credibility Are the procedures taken in order to ensure credibility clearly described?

Triangulation, or multiple sources of data as evidence. YES / NO /UNCLEAR

Does the study report any of the following measure to increase quality and rigour of its findings:

Member checks, or arranging for those who provided data to evaluate the conclusions. YES / NO /UNCLEAR Saturation, or continuous data collection to the point where more data add little to regularities that have already surfaced. YES / NO /UNCLEAR Peer review, or consultation with experts. YES / NO /UNCLEAR Audit trail, or the detailed record of data collection and rationale for important decisions. YES / NO /UNCLEAR Thick description, or providing rich detail of the context of the study. YES / NO /UNCLEAR Plausible alternatives, or the rationale for ruling out alternative explanations and accounting for discrepant (negative) cases. YES / NO /UNCLEAR Inter-coder or inter-observer agreement (Cohen’s k) or and code-recode consistency. YES / NO /UNCLEAR

25

APPENDIX 3: KIRKPATRICK’S EVALUATION OF OUTCOMES

Adapted from : Gordon & Findley, 2011 and Barr et al., 2000.

Outcome level Description:

Level 1: Reaction Learners’ views and satisfaction with the educational interventions. Can include but are not limited to learns' views on organization, presentation, content, teaching methods, and aspects of the instructional organization, materials, quality of instruction.

Level 2a: modification of attitudes/perceptions

Outcomes relate to changes in attitudes or perceptions of the learns' towards the intervention's aims (e.g. learners' perceived impact on confidence in clinical reasoning ability; perception of diagnostic accuracy; perceptions of preparation for practice).

Level 2b Modification of knowledge/'skills'

Knowledge outcomes relate to the acquisition of concepts, principles or conceptual models(e.g. system 1 vs. system 2); 'Skills' outcomes relate to the impact on the ability to solve problems or engage in reasoning/ thinking about similar cases to the ones used in the intervention and/or within the same/similar context (e.g. results in high-fidelity simulation scenario - MI case- as outcome of high-fidelity simulation interventions using common MI typical presentations).

Level 3 Behavioral change Outcome here should document a transfer of learning to the workplace or application of newly acquired knowledge/clinical reasoning abilities in practice, or in a different context from the context of the interventions (e.g. performance in simulation scenario- MI case- used as a outcome measure of taught small group case discussions on models of clinical reasoning; evaluation of students differential diagnosis and clinical decisions in general practice placements as outcome measure of case based discussions).

Level 4a Change in organisational practice

Wider changes in the organizational delivery of care, attributable to an educational program.

Level 4b Benefits to patient/clients

Any improvement in the health & well being of patients/clients as a direct result of an educational program.

26

APPENDIX 4: DRAFT CODING SHEETS

Phase 1 - Abstract Screening Checklist (adapted from Gordon & Findley, 2011) Reference Number: Reviewer: Search Method: Electronic search Hand search Database: Citation (BEME style): DOI: URL:

Population: Are participants healthcare professionals? YES / NO If NO exclude

Level of training (report):

Does this study look at an educational intervention offered to the population?

YES / NO If NO exclude

Was clinical reasoning (decision-making, clinical problem solving, or reduction of cognitive bias) an explicit outcome/aim of this intervention?

YES / NO If NO exclude

Is this article reporting an assessment clinical reasoning without mentioning educational interventions (e.g. comparisons between novices and experts)

YES / NO If YES exclude

Type of study (Cook et al., 2008),

Description Justification Clarification

Decision: Include Exclude If not clear from above provide reason with reference to the inclusion/exclusion criteria.

What level of outcome are reported? (Kirkpatrick’s hierarchy)

Select Level 1: Reaction Level 2a: modification of attitudes/perceptions Level 2b Modification of knowledge/'skills' Level 3 Behavioural change Level 4a Change in organisational practice Level 4b Benefits to patient / clients Not clear from abstract

27

Data Extraction form Phase 2 full articles meeting eligibility criteria (Adapted from Gordon & Findley, 2011 and Barr et al., 2000). Reference Number: Reviewer: Search Method: Electronic search Hand search Database: Citation (BEME style): DOI: URL: Type of study: Level of outcome: (Filled in phase 1)

Background/ Question / Objective

Has a review of the literature been described? YES / NO

there a clearly defined and well described objective to the study?

YES / NO

Research design

Is the design appropriate to answer the research question?

YES / NO

Is the study design reported? YES / NO

Place an S for Stated or I for Implied in the box:- Audit Interrupted time series Survey Before and after studies Cross sectional studies Measures taken over period of time after the intervention Post intervention studies Randomized Non-randomized Case series Cohort studies Retrospective Prospective

Action research Longitudinal studies. Observational Case studies; Mixed-methods case (select above) Qualitative studies (state type of design) ___________________________________ Other please state here type of study ___________________________________

Were the learner characteristics reported (If NO continue to intervention)

YES/NO

Age Mean: SD:

Level Undergraduate Postgraduate CPD

Healthcare profession (state)

Gender Distribution:

Number of participants

Intervention

Definition of clinical reasoning (please use original text when possible):

28

Conceptual framework used and reference if provided:

Is the educational intervention clearly described (could it be replicated)?

YES / NO

Are the required resources described? YES / NO

Please record details of educational intervention (please provide details and if possible quotes from the original article to support your classification)

Outcome Single outcome (clinical reasoning) Multiple outcomes (one being clinical reasoning)

Environment 1 Face-to-face Online Mixed

Environment 2 Clinical Simulation High fidelity/Low fidelity Non-clinical Mixed (please state)

Group/participants Small group Large group Individualised Mixed

Content Clinical cases Clinical reasoning conceptual frameworks/research (specify) Biomedical knowledge Mixed

Duration (e.g. 45m session): Frequency: Time period (e.g. one month, one semester, one year):

Other described factors, external to the intervention, that impact the outcome - limitations (if necessary quotes from the original article to support your classification):

Others provided details of the intervention:

Report quality assessment measures in Quality Checklist

29

APPENDIX 5:

Below are the overall results retrieved from a pilot search using SCOPUS database. These results below are

to be understood as a first test of the search terms and strategy only.

Figure Ap5.1: Number of articles (all types considered) retrieved by year per each of the searches considered

Table Ap5.1: Types Of Articles And Subject Areas Of Papers Retrieved Per Search

Types of articles S1 Types of articles S2

Research articles 792 (70%) Research articles 204 (59%)

Reviews 215 Reviews 89

Subject area (all types considered) Subject area (all types considered)

Medicine 586 Medicine 149

Social Sciences 461 Social Sciences 109

Psychology 286 Psychology 160

Nursing 234 Nursing 28

Health Professions 78 Health Professions 9

Computer Science 73 Computer Science 9

Dentistry 26 Dentistry 2

Veterinary 11 Veterinary 1

Pharmacology, Toxicology and

Pharmaceutics 6

Pharmacology, Toxicology and

Pharmaceutics 1

Decision Sciences 5 Decision Sciences 7

Multidisciplinary 1 Multidisciplinary 1

Arts and Humanities 34

Neuroscience 31

Biochemistry, Genetics and

Molecular Biology 11

0

20

40

60

80

100

120

140

S1

S2

30

Table Ap5.2: Search terms (test of search terms and ‘limit to ’vs ‘exclude’)

Search 1 Search 2

(ALL (clinical reasoning OR clinical decision making OR clinical problems solving OR diagnostic reasoning AND development OR improvement OR learning OR teaching OR education OR educational OR interventions educational activities OR educational strategies OR educational methods OR Expertise) AND ( EXCLUDE(SUBJAREA,"ARTS" ) OR EXCLUDE(SUBJAREA,"BUSI" ) OR EXCLUDE(SUBJAREA,"NEUR" ) OR EXCLUDE(SUBJAREA,"ENGI" ) OR EXCLUDE(SUBJAREA,"ARTS" ) OR EXCLUDE(SUBJAREA,"BUSI" ) OR EXCLUDE(SUBJAREA,"NEUR" ) OR EXCLUDE(SUBJAREA,"ENGI" ) OR EXCLUDE(SUBJAREA,"MATH" ) OR EXCLUDE(SUBJAREA,"BIOC" ) OR EXCLUDE(SUBJAREA,"ECON" ) OR EXCLUDE(SUBJAREA,"AGRI" ) OR EXCLUDE(SUBJAREA,"ENVI" ) OR EXCLUDE(SUBJAREA,"CENG" ) OR EXCLUDE(SUBJAREA,"IMMU" ) OR EXCLUDE(SUBJAREA,"CHEM" ) OR EXCLUDE(SUBJAREA,"EART" ) OR EXCLUDE(SUBJAREA,"MATE" ) OR EXCLUDE(SUBJAREA,"PHYS" ) OR EXCLUDE(SUBJAREA,"MATH" ) OR EXCLUDE(SUBJAREA,"BIOC" ) OR EXCLUDE(SUBJAREA,"ECON" ) OR EXCLUDE(SUBJAREA,"AGRI" ) OR EXCLUDE(SUBJAREA,"ENVI" ) OR EXCLUDE(SUBJAREA,"CENG" ) OR EXCLUDE(SUBJAREA,"IMMU" ) OR EXCLUDE(SUBJAREA,"CHEM" ) OR EXCLUDE(SUBJAREA,"EART" ) OR EXCLUDE(SUBJAREA,"MATE" ) OR EXCLUDE(SUBJAREA,"PHYS" ) ) )

(ALL (cognitive bias cognitive errors OR heuristics clinical problems solving OR diagnostic reasoning AND Reduction OR educational interventions OR educational activities OR educational strategies ) AND ( LIMIT-TO(SUBJAREA,"PSYC" ) OR LIMIT-TO(SUBJAREA,"MEDI" ) OR LIMIT-TO(SUBJAREA,"SOCI" ) OR LIMIT-TO(SUBJAREA,"NEUR" ) OR LIMIT-TO(SUBJAREA,"NURS" ) OR LIMIT-TO(SUBJAREA,"PSYC" ) OR LIMIT-TO(SUBJAREA,"MEDI" ) OR LIMIT-TO(SUBJAREA,"SOCI" ) OR LIMIT-TO(SUBJAREA,"NEUR" ) OR LIMIT-TO(SUBJAREA,"NURS" ) OR LIMIT-TO(SUBJAREA,"HEAL" ) OR LIMIT-TO(SUBJAREA,"DECI" ) OR LIMIT-TO(SUBJAREA,"DENT" ) OR LIMIT-TO(SUBJAREA,"MULT" ) OR LIMIT-TO(SUBJAREA,"PHAR" ) OR LIMIT-TO(SUBJAREA,"VETE" ) OR LIMIT-TO(SUBJAREA,"HEAL" ) OR LIMIT-TO(SUBJAREA,"DECI" ) OR LIMIT-TO(SUBJAREA,"DENT" ) OR LIMIT-TO(SUBJAREA,"MULT" ) OR LIMIT-TO(SUBJAREA,"PHAR" ) OR LIMIT-TO(SUBJAREA,"VETE" ) ) )