Specialty Certificate Examinations Specialty Certificate Examinations (SCEs): Where Are We Now?(SCEs): Where Are We Now?

Oliver J CorradoOliver J CorradoLeeds General Infirmary,Leeds General Infirmary,

Former Lead SCE Geriatric Former Lead SCE Geriatric MedicineMedicine

Chair SAC Geriatric MedicineChair SAC Geriatric Medicine

Specialty Certificate Examinations Specialty Certificate Examinations (SCEs)(SCEs)

• Background to SCEs and Pilot Background to SCEs and Pilot ExaminationExamination

• The Development of SCEsThe Development of SCEs

• Where we are now?Where we are now?

Why Do We Need Another Exam?Why Do We Need Another Exam?

• As part of new curriculum PMETB require As part of new curriculum PMETB require all StRs registering on or after 1 August all StRs registering on or after 1 August 2007 to have formal assessment of 2007 to have formal assessment of competence including work placed based competence including work placed based assessments eg mini-CEX and SCEassessments eg mini-CEX and SCE

• To ensure that specialists dealing with To ensure that specialists dealing with patients have the appropriate knowledge patients have the appropriate knowledge of their specialtyof their specialty

Pilot ExaminationPilot Examination

2006 JCHMT in partnership with specialist societies ran KBA2006 JCHMT in partnership with specialist societies ran KBApilots in 4 specialties pilots in 4 specialties (cardiology, neurology, geriatric med and dermatology)(cardiology, neurology, geriatric med and dermatology)

Question writers limited training, small question bank, for Question writers limited training, small question bank, for geriatric medicine pilot standard setting not robust geriatric medicine pilot standard setting not robust

100 best-of-five questions in 3 hours, paper based exam, 100 best-of-five questions in 3 hours, paper based exam, sat on 2 dates in May 2006 a week apartsat on 2 dates in May 2006 a week apart

448 sat exam (411 SpRs (77% total SpRs), 37 consultants) 448 sat exam (411 SpRs (77% total SpRs), 37 consultants)

Best-of-five QuestionBest-of-five Question

An 82-year-old womanAn 82-year-old woman had a 3 year history of urinary had a 3 year history of urinary frequency, with a sudden desire to urinate at times frequency, with a sudden desire to urinate at times associated with incontinence, but no dysuria. She was on no associated with incontinence, but no dysuria. She was on no medication and there were no abnormal findings on medication and there were no abnormal findings on

examination. Mid examination. Mid stream urine specimen was negative.stream urine specimen was negative.

The most likely cause for her symptoms is:The most likely cause for her symptoms is:

AA atonic bladderatonic bladderBB autonomic neuropathyautonomic neuropathyCC detrusor instabilitydetrusor instabilityDD pelvic floor weaknesspelvic floor weaknessEE overflow incontinenceoverflow incontinence

Question: Courtesy of Adam Harper !Question: Courtesy of Adam Harper !

A 92- year-old woman was admitted to hospital with pneumonia. HerA 92- year-old woman was admitted to hospital with pneumonia. Herdaughter was concerned her mother may develop Clostridium difficile daughter was concerned her mother may develop Clostridium difficile infection. infection. What is the greatest risk factor for her developing Clostridium difficileWhat is the greatest risk factor for her developing Clostridium difficileinfection? infection? 

A a matching tie and cufflink set A a matching tie and cufflink set 

B cramming 6 beds into an area designed for 4 B cramming 6 beds into an area designed for 4 

C having insufficient nursing staff so discharge is delayed by 1 C having insufficient nursing staff so discharge is delayed by 1 week week 

D sharing a commode with 6 other patients D sharing a commode with 6 other patients 

E sharing a sphygmomanometer cuff with 10 other patients E sharing a sphygmomanometer cuff with 10 other patients   

Answer: A

Pilot Evaluation and CommentsPilot Evaluation and Comments

• Questions? : About right 64%Questions? : About right 64%

• Right amount of time? : About right 70%Right amount of time? : About right 70%

• Familiar with question format? : Yes 80% Familiar with question format? : Yes 80%

CommentsComments

Poor questions: ambiguous, too long, negative (eg “which is Poor questions: ambiguous, too long, negative (eg “which is least likely”) and double negative questionsleast likely”) and double negative questionsToo many questionsToo many questionsToo many on falls, inappropriate orthopaedic questionsToo many on falls, inappropriate orthopaedic questionsSome questions not relevant to Scottish lawSome questions not relevant to Scottish law

Pilot EvaluationPilot Evaluation

Evaluation of pilot by JCHMT positiveEvaluation of pilot by JCHMT positive

However for pilot KBA in Geriatric Medicine Cronbach’s However for pilot KBA in Geriatric Medicine Cronbach’s coefficient coefficient

alpha (measure of reliability) 0.67 alpha (measure of reliability) 0.67

(for high stakes exam should be 0.80 +, preferably 0.9+) (for high stakes exam should be 0.80 +, preferably 0.9+)

(Spearman-Brown formula if 200 questions used alpha 0.81) (Spearman-Brown formula if 200 questions used alpha 0.81)

But But nono standard setting was undertaken in Geriatric standard setting was undertaken in Geriatric

Medicine.Medicine.

Development of SCEsDevelopment of SCEs

Federation of Royal Colleges proposed introducing this format as KBA for 13 Federation of Royal Colleges proposed introducing this format as KBA for 13 major medical subspecialties in partnership with specialist societies split major medical subspecialties in partnership with specialist societies split 75% v 25%. KBA called SCE in November 200875% v 25%. KBA called SCE in November 2008

Initial ProposalsInitial Proposals

2 diets a year 200 b-o-f questions, on-line (using Pearson Vue (DVLA theory)2 diets a year 200 b-o-f questions, on-line (using Pearson Vue (DVLA theory)

Use infrastructure and expertise of MRCP (UK) to administer/organiseUse infrastructure and expertise of MRCP (UK) to administer/organiseexamexam

Candidates with MRCP who pass SCE awarded MRCP (specialty) those without Candidates with MRCP who pass SCE awarded MRCP (specialty) those without Diploma. No differentiation in post-nominal Award for UK trainees from Diploma. No differentiation in post-nominal Award for UK trainees from

others.others.

Cost £800 if sat in UK, £1000 if taken in an overseas centre Cost £800 if sat in UK, £1000 if taken in an overseas centre

BGS Newsletter March 2007BGS Newsletter March 2007

Development of SCEDevelopment of SCE

Established:Established:

Question Writing Group - 25 BGS members (in response to advert and Question Writing Group - 25 BGS members (in response to advert and email to Regional and Council reps) broad interests, non-academics email to Regional and Council reps) broad interests, non-academics and some academics, representation from all 4 nations, all trained andand some academics, representation from all 4 nations, all trained andnot paid!not paid!

Examination Board - 10 members including Chair and Examination Board - 10 members including Chair and secretary of SACsecretary of SAC

Standard Setting Group – several members with expertise in standard Standard Setting Group – several members with expertise in standard setting undergraduate examinations.setting undergraduate examinations.

Relationship of 3 GroupsRelationship of 3 Groups

Question Writing Group

Questions

Modify

Reject

Bank

ClinicalExamination

Board

Select 200+ questions for paperfor paper

Standard Setting Group “Anghoff Method”

Paper

Where Are We Now?Where Are We Now?

1.1. Successfully negotiated award title change from Diploma to Successfully negotiated award title change from Diploma to

Certificate (avoids confusion with DGM). For UK trainees award will Certificate (avoids confusion with DGM). For UK trainees award will

convert to MRCP (specialty) at time CCT.convert to MRCP (specialty) at time CCT.

2. Not been able to reduce cost of exam (£800 if taken in UK, 2. Not been able to reduce cost of exam (£800 if taken in UK, £1000 if £1000 if

taken overseas) but negotiated free resits 2008-10 taken overseas) but negotiated free resits 2008-10

3. One “diet” (exam) to be held per year (not 2)3. One “diet” (exam) to be held per year (not 2)

4. 4. 200 b-o-f questions (2 papers), same day, computer-based test, 200 b-o-f questions (2 papers), same day, computer-based test,

Pearson Vue 12 centres (centres in all 4 UK nations) Pearson Vue 12 centres (centres in all 4 UK nations)

Where Are We Now?Where Are We Now?

5. Gastroenterology held exam 24 June 2008. Geriatric 5. Gastroenterology held exam 24 June 2008. Geriatric Medicine 4 March 2009, followed by Nephrology, Medicine 4 March 2009, followed by Nephrology, Respiratory Med and Diabetes/EndoRespiratory Med and Diabetes/Endo

6. Dermatology and Medical Oncology September 2009. Med 6. Dermatology and Medical Oncology September 2009. Med Onc, Rheumatology and Acute Medicine 2010Onc, Rheumatology and Acute Medicine 2010

7. Next exam in Ger Med March 2010 date to be finalised7. Next exam in Ger Med March 2010 date to be finalised

8. 5 sample questions on MRCP website – need log in 8. 5 sample questions on MRCP website – need log in

Eligibility Criteria and CurriculumEligibility Criteria and Curriculum

• Non UK candidates must have MRCPNon UK candidates must have MRCP

UK candidates ST3+, ideally pass it before UK candidates ST3+, ideally pass it before PYAPYA

• SCE based on JRCPTB Curriculum in SCE based on JRCPTB Curriculum in Geriatric MedicineGeriatric Medicine

• SCE regulations include exam “blueprint”SCE regulations include exam “blueprint”

Area %

Health Status; Health Promotion; Disease Prevention 3

Gerontology and Illness of Old Age 6

Rehabilitation; Poor Mobility; Intermediate Care/Community Practice; Assessment Scales; Discharge Planning; Continuing Care; Health Care Organisation

5

Ethical Issues; Medico-legal Aspects; End of Life Planning 4

Acute and Chronic Disease (Cardiovascular 5; Respiratory Disease 4; GI Disease 4; Endocrine Disease 3; Renal Disease 3; Neurology 5; Sensory Impairment 2; Dermatology 1; Musculo-Skeletal 3; Anaemia/Haematology 2; Infection 5)

37

Falls 7

Delirium and Dementia 6

Continence 5

Palliative Care 4

Orthogeriatrics and Osteoporosis 4

Old Age Psychiatry (other than Delirium and Dementia) 2

Stroke and TIA 7

Tissue Viability; Thermoregulation and Nutrition 3

Clinical Pharmacology and Therapeutics and Safe Prescribing 7

100

Specialty Overall pass rate%

UK pass rate%

Geriatric Medicine (n=15) 100 100

Nephrology 57.58   81

Respiratory Medicine 60.00   89

Neurology 80.00   87.5

Endocrinology & Diabetes 38.46   64

ConcernsConcerns

1.1. What happens to trainees who cant pass the What happens to trainees who cant pass the exam?exam?

2.2. Why cant UK based trainees who pass the exam Why cant UK based trainees who pass the exam but are ineligible for a CCT (ie pursuing CESR but are ineligible for a CCT (ie pursuing CESR route to specialist certification) have MRCP (Ger route to specialist certification) have MRCP (Ger Med) Med)

Useful WebsitesUseful Websites

•www.bgs.org.uk

•www.jrcptb.org.uk (curriculum, training) (curriculum, training)

•www.mrcpuk.org/KBA

• www.doctors.net.ukwww.doctors.net.uk

AcknowledgementsAcknowledgements

Thanks toThanks to• Question writersQuestion writers

• Examination Board in particular Mike Vassallo Examination Board in particular Mike Vassallo (Secretary) (now SCE Lead)(Secretary) (now SCE Lead)

• Standard Setting GroupStandard Setting Group

• SAC Geriatric Medicine and BGS ETC SAC Geriatric Medicine and BGS ETC

• Trainee reps and traineesTrainee reps and trainees