1. OSCE workshop Dr Ed Parry-Jones RCGP

2. Who am I ? edwardparry-jones@nhs.net 3. Overview of workshop topics What is an OSCE Blueprinting Case writing Paperwork Mark scheme location Equipment Role players Examiners The big day Standard setting Case assurance / metrics and review Building and maintaining case bank 4. OSCE Objective Structured Clinical Examination 5. Objective - all candidates presented with same test Structured - the marking scheme for each station is structured Clinical Examination - test of skills, behaviour, attitudes and application of knowledge 6. Why use OSCEs in clinical assessment? careful specification of content observation of wide sample of activities increased reliability interactions between examiner and student are structured 7. Validity a qualitative issue, testing the RIGHT things in the RIGHT way Reliability- Technical aspects of the test, consistency, accuracy, ability to predict who is competent or incompetent . Same level of test wherever or whenever it is taken 8. OSCE stations 3 categories: Observed Examiner present Non-observed interpretation of clinical data e.g. video, CT scan - no examiner Rest or Preparatory station 9. Types of OCSE stations history taking/ counselling /explanations 10. Types of OCSE stations Examination skills 11. Types of OCSE stations Data interpretation 12. Resources Simulated patients (trained) Real patients Volunteers (not trained) Models Video/ DVD of patients Films/ scans Equipment for tests/ procedures 13. Blueprinting Blueprint written and clinical assessments against each other Content of the assessment should align with curriculum learning objectives Use appropriate assessment methods to test different aspects of curriculum Create a matrix - competencies along one axis, system or specialism along the other Sample widely across the whole matrix 14. Learning outcome 2.1 Evaluate a patient according to the bio-psycho social approach 2.2 Formulate and execute, in consultation with the patient, a mutually acceptable, cost- effective management plan 2.3 Provide comprehensive, continuing care throughout the life cycle incorporating preventative, diagnostic, therapeutic, palliative and rehabilitative interventions TOTAL NO OF QUESTIONS Key skills to be examined Interpretation of investigations (e.g. x- rays, ECG, blood tests, etc.) x2 Diagnostic procedures (e.g. LP, lymph node biopsy etc.)x2 Therapeutic skill or procedures (e.g. injection joint) x2 Emergency management and skills (e.g. resuscitation, primary survey, airway management) x2 Anaesthetic skills (e.g. spinal anaesthetic, use of local anaesthetic, GA) x1 Brief counseling (e.g. breaking bad news, behavior change) x2 Question based on portfolio x1 Weight Stations 4 5 3 Trauma & Orthopedics 2 HIV/AIDS, TB and Malaria 2 Womens and Child Health 2 Surgery & Anaesthetics 2 ENT, eye, skin, mental health 2 General adult medicine 2 TOTAL NO OF QUESTIONS 15. Case writing n plan the construct that the station is testing and write it out as a single sentence e.g. to assess the ability of a candidate to explain colonscopy clearly, accurately and succinctly to a patient e.g. to assess the competence of a candidate in examining a patients thyroid gland n refer to your construct while you write 16. Paperwork scheme Learning Outcome Station Title Skill tested Context and focus for the station What the author(s) feel the candidate has to do to pass the station 17. Marking Scheme n Think about whether to use 2/1/0 2 = done well 1 = adequate 0 = poor/ not done n ..or 1/0 1 = Performed competently 0 = poorly performed/ not done OR other schemes ? Check lists and / or Global score 18. Location 19. Equipment 20. Role Players 21. Examiners 22. The Big Day 23. Standard setting Standards are based on judgments about examinees performances against a social or educational construct e.g. competent practitioner or student ready for graduation 24. The Standard Setting Problem Test Result Pass Fail Competent Incompetent 25. Classification Scheme Relative methods based on judgments about groups of test takers Absolute methods based on judgments about test questions based on judgments about the performance of individual examinees Compromise methods Livingston, S.A. & Zeiky, M.J. (1982) Passing scores: a manual for setting standards of performance on educational and occupational tests Educational Testing Service, Princeton 26. Types of Standards Relative standards/norm referenced methods: based on a comparison among the performances of examinees a set proportion of candidates fails regardless of how well they perform e.g. the top 84% pass Eg Borderline group, regression Absolute standards/criterion referenced methods: based on how much the examinees know candidates pass or fail depending on whether they meet specified criteria e.g. examinees must correctly answer 70% of the questions Eg Angoff 27. Norm-referenced standard Test score distribution 30 % 50 % 80 % 28. Criterion referenced standard 50 % Test score distribution (average group) Test score distribution (good group) Test score distribution (poor group) 29. Case assurance / metrics Measures of reliability not validity Cronbach Alpha co-efficient Central to producing a reliable examination is the idea that every question (or OSCE station)should contribute in the same direction towards the final mark or result. Thus the marks that candidates get on each item / question / case should broadly correlate positively with each other and with the total mark Standard error of measurement the confidence interval around a score 30. Case / question banks Benefit of saving time writing cases / questions Allow development of cases / questions based on metrics Useful for developing a regional / national exam Problems of security 31. Thanks and acknowledgements Sandy Mather, Tom Owen and John Howard at the RCGP Angela Hall and Richard Wakefield