Postgraduate Medical Training – Evaluation and Audit

Copenhagen Nov 2013

Professor Wendy ReidMedical Director Health Education England

Past- Vice-President, Education, RCOG

© Royal College of Obstetricians and Gynaecologists

UK Specialty Training & Education Programme

A model of clinical competence

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S67.

Knows

Shows how

Knows how

Does

Pro

fess

ion

al a

uth

enti

city

Cognition =knowledge

Behaviour = Skills + attitude

UK Specialist training programme

• Basic – years 1&2, part 1 MRCOG• Intermediate – years 3,4&5, part 2 MRCOG• Advanced – years 6&7, requires 2 ATSMs

minimum, career development and ‘independent’ competencies

• 19 core modules, subject based, includes professional skills and leadership

Basic Training

• Exposure to the specialty• Basic emergency obstetric and gynaecology

skills• Understanding role of the doctor• Team work – multi professional, develop

leadership skills• Pass Part 1 MRCOG

Intermediate training

• Builds on basic skills• Leadership – clinical, administrative• Competences for normal practice i.e. day to day

obstetrics, emergency gynae and core gynae skills

• Pass Part 2 MRCOG• Workplace-based assessments• More clinical responsibility, labour ward

leadership and acute gynaecology, develop interests and choose advanced modules

Advanced training

• Core continues throughout programme!• Advanced Training Skills Modules

(minimum x2)• Designed to produce a workforce for the

service and give individuals scope to develop clinical expertise in specific area

• New ATSMs in development, some academic, some ‘professional’

Advanced training skills modules (ATSMs)

• Fetal Medicine• Benign Vaginal Surgery• Advanced Labour Ward

Practice • Advanced Lap surgery for the

excision of benign disease• Benign Gynaecological

Surgery: Laparoscopy• Labour Ward Lead • Benign Gynaecological

Surgery: Hysteroscopy• Maternal Medicine• Colposcopy • Advanced Antenatal Practice

• Vulval Disease• Acute Gynaecology and Early

Pregnancy• Abortion Care • Gynaecological Oncology• Sexual Health• Subfertility and Reproductive

Endocrinology• Menopause• Urogynaecology• Paediatric and Adolescent

Gynaecology• Benign Abdominal Surgery• Medical Education• Domestic violence

Workplace Based Assessments

• All trainee grades in UK• Varied names but similar principles• Ongoing challenge of ‘formative vs

summative’• Monitoring through Royal Colleges/Faculties

Testing formats

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S67.

Knows

Shows how

Knows how

Does

Pro

fess

ion

al a

uth

enti

city

Written/ Computer based assessment

Performance/hands on assessment

Drivers for WBA

• New curricula – trainees need to prove ‘competence’

• GMC- the regulator ( and the public) want explicit evidence of competence

• Professional examinations do not test ‘real life’ skills and performance

• Learning from other systems• One way of evaluating quality of training

UK experience of WBA

• Began with Foundation Programme (years 1&” after graduation)

• Launched 2005, integrated assessment process

• Regardless of post or geography• Outcomes collated by Sheffield University• Each training area (Deanery) informed of

‘outliers’• Large cohort• Centralised faculty training

Specialty training

• From end of F2 to CCT• New curricula, launched August 2007• Assessment tools based on FP• Many ‘in development’ and specialty specific• Trainees in mixed programmes, mostly using log

books to capture evidence of progress• Most curricula mandate ‘minimum numbers of

assessments’• Summarised annually in ARCP (previously RITA)

Challenges of WBA in Specialty Training

• Does it really measure the doctors?• Are we sure we are measuring the right

things?• How often do they need to be done?• Are they a good measure of continued

competence?• How do we involve patients?• How do we ensure trainers are trained and

have the time to do WBAs properly?• To provide QA takes large numbers – poor

reliability

Other tools for QA of Training

• Longitudinal analysis of MRCOG results – cohort comparison, demographic data required

• Trainee doctor ‘user’ surveys• Trainee feedback at end of training episodes• Population wide survey of trainee doctors by

the GMC

Whole QA system

Formal requirements in UK for Training QA

• Royal College annual report to GMC – specialty specific

• Deanery (regional) annual report of Education and Training – all specialties

• Trainees must complete the Annual GMC survey

• All curriculum changes assessed by GMC• All examination changes and examination data

submitted to the GMC

Whole QA system

• “The GMC expects medical schools and deaneries to demonstrate compliance with the standards and requirements that it sets. To do this, they will need to work in close partnership with the medical Royal Colleges and Faculties, NHS trusts and health boards and other LEPs. This means that QM should be seen as a partnership between those organisations because it is only through working together that medical schools, deaneries, Royal Colleges and Faculties, with LEPs, can deliver medical education and training to the standards required.” (GMC Quality Improvement Framework, para. 29)

Whole QA system

• “The GMC quality assures medical education and training through the medical schools and deaneries but day-to-day delivery is at LEP level. This delivery involves medical staff, medical education managers, undergraduate and postgraduate medical centre staff, other health professions and employers. Clinical placements, student assistantships, individual foundation programme and specialty including GP training are delivered through careful supervision and assessment by specialists in the relevant discipline advised and overseen by regional and local staff from the UKFPO, the Academy of Medical Royal Colleges and the relevant medical Royal College or Faculty.” (GMC Quality Improvement Framework, para. 46)

Role of medical royal colleges

• Set curriculum and workplace-based assessments for trainee doctors according to GMC standards

• Set criteria for progression between stages of training• Engage with a range of stakeholders to assure quality of

training, particularly 16 UK deaneries• Provide fora for making policy, sharing best practice and

developing training requirements as clinical practice develops• Provide specialist faculty development• Assure the quality of individual trainees (recommendation for

CCT/CESR, MRCOG)

Role of medical royal colleges

• Colleges can also raise concerns about patient or trainee safety directly with the GMC or CQC

• Colleges work together on national medical education policy through Academy

Governance

• College committees agree national policy on various aspects of specialty education (e.g. exams, curriculum, ARCP)

• Network of College Tutors coordinate training in individual Trusts

• Specialist educational management and leadership roles created in Colleges (e.g. committee chair)

• Heads of Deanery Specialty Schools jointly appointed with Colleges

• Colleges report to GMC via Annual Report

QA processes

• ARCP – colleges send specialist assessor to assure deanery process for progressing trainees

• Quality visits – colleges provide specialist assessor on request to join deanery visit team

• CCT/CESR(CP) and CESR – recommendation of individual doctor to GMC for inclusion on specialist register

• Examination – standard-setting• Curriculum approval – changes to curricula approved

by GMC

Data on quality• GMC Trainee Survey• ARCP outcome data – summary of achievements

annually for every trainee• Examination data• Colleges’ own surveys (e.g. Training Evaluation Form)• Reports from external assessors on local/regional QA

processes (ARCP and quality visits)• Increasingly linked with quality of care and patient

safety reviews

GMC Trainees’ survey – O&G Perspectives

• Three specific elements -o How O&G trainees compare with other specialties.o How the results from this year for specific

questions compare with those in previous years (looking at the areas previously considered).

o Specialty Specific QuestionsTotal number of trainees responding 49000 (95%)

Trainee Evaluation Forms

• Not mandatory• Might work effectively if based on MSF ‘360’

feedback• Should be real-time tool for local training

quality management• Best discriminator is ‘would you recommend

this job to a friend?’

Programme Groups

  National This Report

Programme Group Mean Min Q1 Median Q3 Max Lower CI Upper CI N Mean Lower CI Upper CI N

ACCS 79.66 24 72 80 92 100 78.75 80.57 1114 79.66 78.75 80.57 1114Acute Internal Medicine 81.72 20 76 84 92 100 81.32 82.12 4766 77.6 75.97 79.24 302Allergy 81.72 20 76 84 92 100 81.32 82.12 4766 82.8 74.7 90.9 10Anaesthetics 82.68 20 76 84 92 100 82.16 83.2 2409 82.58 82.05 83.11 2358Anaesthetics F1 75.46 20 68 76 84 100 75.12 75.79 7077 89.92 88.5 91.33 198Anaesthetics F2 78.67 20 72 80 88 100 78.33 79.01 7138 87.79 86.32 89.27 232Audio vestibular medicine 81.72 20 76 84 92 100 81.32 82.12 4766 81.6 74.11 89.09 15Cardiology 81.72 20 76 84 92 100 81.32 82.12 4766 81 79.75 82.26 550Cardio-thoracic surgery 83.67 20 76 84 96 100 83.19 84.14 3514 82.78 79.28 86.28 95Chemical pathology 84.93 20 80 84 96 100 83.93 85.92 672 80.45 76.81 84.09 62Child and adolescent psychiatry 86.46 20 80 88 96 100 85.75 87.18 1232 87 85.32 88.67 211Clinical genetics 81.72 20 76 84 92 100 81.32 82.12 4766 86.78 83.84 89.73 46Clinical neurophysiology 81.72 20 76 84 92 100 81.32 82.12 4766 85.22 80.66 89.77 23Clinical oncology 84.53 20 76 84 96 100 83.85 85.2 1332 82.3 80.79 83.81 285Clinical pharmacology and therapeutics 81.72 20 76 84 92 100 81.32 82.12 4766 78.86 71.5 86.22 21Clinical radiology 84.53 20 76 84 96 100 83.85 85.2 1332 85.13 84.38 85.89 1047CMT 74.55 20 64 76 84 100 74 75.11 2730 74.55 74 75.11 2730Community Sexual and Reproductive Health 78.59 20 68 80 88 100 77.93 79.25 1915 78.46 71.5 85.43 13Core Anaesthetics 85.28 20 80 88 96 100 84.48 86.08 1052 85.28 84.48 86.08 1052CPT 81.77 24 76 84 92 100 81.06 82.47 1529 81.77 81.06 82.47 1529CST 74.52 20 64 76 88 100 73.67 75.38 1463 74.52 73.67 75.38 1463Dermatology 81.72 20 76 84 92 100 81.32 82.12 4766 84.29 82.38 86.21 191Emergency medicine 80.15 28 72 80 92 100 78.98 81.32 550 80.15 78.98 81.32 550Emergency Medicine F1 75.46 20 68 76 84 100 75.12 75.79 7077 87.81 86.05 89.57 169Emergency Medicine F2 78.67 20 72 80 88 100 78.33 79.01 7138 82.45 81.75 83.14 1199

O&G Programme Group Comparison

Supervision (1)How would you rate the quality of (clinical) supervision in this post?

Did you have a designated educational supervisor (the person responsible for your appraisal) in this post? Yes

2012 – 99.3% (2011 – 99.5%, 2010 – 99.5%, 2009 – 99.8%)

In this post did you have a training/learning agreement with your educational supervisor, setting out your respective responsibilities? Yes

2012 – 86.4% (2011 – 91.9%, 2010 – 92.6%, 2009 – 91.1%)

In this post did you use a learning portfolio? Yes

2012 – 92.4% (2011 – 94.7%, 2010 – 89.9%, 2009 – 91.2%)

In this post were you told who to talk to in confidence if you had concerns, personal or educational? Yes

2012 – 71.2% (2011 – 77.7%, 2010 – 72.2%, 2009 – 68.8%)

Excellent

Good

Fair

Poor

Very Poor

22.49%

49.22%

22.61%

4.31%

1.38%

23.74%

49.78%

20.08%

5.09%

1.31%

28.31%

48.95%

18.02%

3.81%

0.91%

30.86%

50.05%

15.83%

3.00%

0.26%

O&G Trainees 2012O&G Trainees 2011O&G Trainees 2010O&G Trainees 2009

Supervision (2)Did you have a formal meeting with your supervisor to talk about your progress in this post?

Did you have a formal assessment of your performance in the workplace in this post?

Yes, and it was useful

Yes, but it wasn't useful

No, but this will happen

No, but I would like to

72.34%

10.04%

13.83%

3.79%

79.71%

11.82%

6.65%

1.81%

82.17%

11.94%

4.17%

1.71%

81.13%

10.15%

6.47%

1.63%

O&G Trainees 2012 O&G Trainees 2011O&G Trainees 2010 O&G Trainees 2009

Yes, and it was useful

Yes, but it wasn't useful

No, but this will happen

No, but I would like to

63.91%

7.60%

20.22%

8.27%

74.59%

8.17%

10.30%

6.94%

79.38%

8.94%

6.50%

5.17%

76.97%

7.26%

8.94%

4.84%

O&G Trainees 2012 O&G Trainees 2011O&G Trainees 2010 O&G Trainees 2009

Access to Training (1)How would you rate the practical experience you were receiving in this post?

In this post, how often have you worked beyond your rostered hours?

Excellent

Good

Fair

Poor

Very Poor

17.40%

42.34%

30.86%

7.60%

1.79%

20.02%

44.69%

26.53%

7.11%

1.64%

22.23%

47.53%

25.13%

4.09%

1.02%

28.39%

42.95%

22.50%

5.10%

1.05% O&G Trainees 2012O&G Trainees 2011O&G Trainees 2010O&G Trainees 2009

Daily

Weekly

Monthly

Rarely

Never

15.07%

37.44%

16.15%

27.45%

3.89%

12.42%

36.71%

17.72%

28.17%

4.98%

13.47%

38.15%

16.43%

28.03%

3.92%

12.72%

46.90%

16.82%

21.50%

2.05% O&G Trainees 2012O&G Trainees 2011O&G Trainees 2010O&G Trainees 2009

Access to Training (2)How confident are you that this post will help you acquire the competencies you needed at that particular stage of your training?

How good or poor was access to each of the following in your post? (2012 question only)

O&G Trainees 2012

O&G Trainees 2011

O&G Trainees 2010

O&G Trainees 2009

1.68%

1.53%

2.90%

2.63%

7.89%

9.10%

10.89%

12.62%

16.40%

17.57%

18.27%

20.63%

47.42%

48.95%

48.91%

47.31%

26.60%

22.85%

19.04%

16.81%

Very confidentFairly confidentNeutralNot very confidentNot at all confident

Library

Online journals

E-learning resources

Internet access

Space for private study

Equipped rooms for group teaching

Simulation facilities

46.90%

46.32%

53.21%

51.79%

31.91%

53.68%

31.91%

24.24%

12.78%

14.14%

22.77%

6.36%

11.72%

9.41%

Very Good

Good

Working Beyond Competence (1)In this post how often did you feel forced to cope with clinical problems beyond your competence or experience?

In this post how often, if ever, were you supervised by someone who you felt wasn't competent to do so?

In this post how often have you been expected to obtain consent for procedures where you feel you do not understand

the proposed interventions and its risks?

O&G Trainees 2012

O&G Trainees 2011

O&G Trainees 2010

O&G Trainees 2009

0.42%

0.85%

0.98%

1.32%

2.21%

6.03%

6.24%

6.16%

4.89%

8.93%

10.28%

10.11%

48.37%

57.70%

54.60%

55.74%

44.11%

26.49%

27.90%

26.67%

NeverRarelyMonthlyWeeklyDaily

O&G Trainees 2012

O&G Trainees 2011

O&G Trainees 2010

O&G Trainees 2009

0.42%

0.68%

0.82%

0.72%

1.58%

1.82%

2.13%

2.81%

3.36%

3.81%

4.38%

4.19%

28.50%

32.58%

33.64%

32.66%

66.14%

61.11%

59.03%

59.63%

NeverRarelyMonthlyWeeklyDaily

O&G Trainees 2012

O&G Trainees 2011

O&G Trainees 2010

O&G Trainees 2009

0.11%

0.06%

0.11%

2.48%

0.21%

0.64%

0.61%

17.93%

1.10%

1.85%

2.06%

11.75%

16.56%

20.28%

22.20%

34.83%

79.71%

77.17%

75.01%

33.01%

NeverRarelyMonthlyWeeklyDaily

Working Beyond Competence (2)

Yes and they were accessible

Yes, but they were not easy to access

No, but there was usually someone I could contact

No, there was no-one I could contact

87.14%

6.60%

6.03%

0.23%

82.78%

8.34%

8.34%

0.55%

86.32%

6.72%

6.96%

0.00%

O&G Trainees 2009 O&G Trainees 2010 O&G Trainees 2011

In this post did you always know who was providing your clinical supervision when you were working? (2009 – 2011 inclusive)

In this post did you always know who your available senior support was during on call (2012)

88.70%

6.89%

3.89% 0.37% 0.16%

Yes - accessibleYes - not easy accessNo - but someone to con-tactNo - no one to contactN/A

Undermining – 2012 (1)

How often, if at all, have you been the victim of bullying and harassment in this post?

How often, if at all, have you witnessed someone else being the victim of bullying and harassment in this post?

In this post, how often if at all, have you experienced behaviour from a consultant/GP that undermined your professional confidence and/or self esteem?

0.63%

0.47%

0.53%

1.74%

2.58%

2.00%

2.00%

2.42%

2.79%

3.73%

7.26%

5.47%

18.82%

24.76%

26.81%

64.51%

51.95%

54.15%

8.57%

10.57%

8.25%

Prefer not to answer

Never

Less often than once per month

At least once per month

At least once per fortnight

At least once per week

Every Day

Undermining 2012 (2)

Overall• 96.0% of trainees said they had never

been bullied and/or harassed in their post, or if they had, it happened less than once a month. 1.1% said it happened every day or at least once per week (n=48,512).

• 1.6% said they had witnessed someone else being the victim of bullying and/or harassment in their post every day or at least once per week (n=48,464).

• 92.4% said they had never experienced behaviour from a consultant or GP that undermined their professional confidence and/or self-esteem or, if they had, it happened less than once a month. 1.7% said it happened every day or at least once per week (n=48,785).

O&G

• The equivalent figures for O&G are 83.33% and 2.37%.

• The equivalent figure for O&G is 3.05%.

• The equivalent figures for O&G are 80.96% and 2.53%.

(1902 respondents for O&G)

O&G versus other specialties

Next Steps

• Specialty specific questions to be further analysed• Breakdown by training level may also be available –

need to discuss with GMC.• Will be involved with preparation for 2014 survey.• Must publish more quickly• An updated trainers survey has been discussed –

believe something may be in place within 12 months.• TEF – a potential method of triangulation?• RCOG has appointed Workplace Advisory Officer to

combat undermining

Who does what in governance of training?

• Education Board -RCOG• Heads of Schools – joint between college and

local regional Postgraduate Dean• Local Education and Training Boards

(Deaneries)• Individual hospitals (Local Education Providers,

LEPs) – DMEs or Clinical Tutors• Individual doctors through trusts or

organisations

Future developments• Outcome of review of QA system by GMC in Autumn 2013

(note: GMC became regulator in 2010 for PG medical education)

• Growing recognition of need to clarify role of colleges in QA• Increased focus on sharing data between deaneries, colleges

and GMC• Increased emphasis on the role of educational and clinical

supervisors/trainers with consequent impact on service delivery

• Impact of national policy changes, e.g. Shape of Training, new English healthcare structure

• Role of HEE, relationship with Colleges, GMC, devolved nations