Andrew ConnollyChair, MCNZ

The outcomes & effectiveness of revalidation in New Zealand for non-certified general practitioners

Definitions• “Non-certified” are those general registrants

who have not achieved vocational (specialist) registration and are not actively in an accredited specialist training program– Not restricted to General Practice, but vast

majority are working in this field• Does not apply to PGY-1• Will not apply to PGY-2 from 2016

Why do it?• Council legally responsible for ensuring

practicing doctors are competent• Long-established CPD requirements but a

lack of standardisation, consistency• College programs all accredited via

Australian Medical Council & MCNZ• Council recognised the lack of a general

registrant mandated program was a risk

Council decisions:• Electronic format (“e-portfolio”)• Mandatory unless “grandfathered” by existing

successful participation in an accredited scheme

• Mix of annual and triennial requirements– RPR & MSF three-yearly

• Prospective research evaluation by independent company

The Program• Introduced in May 2012

– Essentials quiz– Collegial relationship with meetings mandated– Professional development plan– Audit– CPD record– Regular practice review: began July 2013– Multi-source feedback

The numbers• Currently some 1800 participants• Very few non-compliant but Council has

escalation policy including suspension - has been necessary in a very small number

• Most radical has been the Regular Practice Review (RPR)

• RPR is prime focus of research• Numbers are small, but knowledge is growing

• A formative assessment via the Best Practice Advocacy Centre’s Inpractice programme

• Focus is on quality improvement by helping individual doctors identify areas of improvement and build them into professional development plan- A mandatory requirement for doctors registered in a

general scope of practice- Reviewers visit the participating doctors in their

practices for a one day visit involving review of notes, observation and discussion

• Will be prospectively studied

The Regular Practice Review (RPR) Programme

The Regular Practice Review (RPR) Programme

• Spans July 2014 to 2020 and

focuses on the outcomes of RPR• Collects data from participating

doctors and the reviewers

Methodology• Survey at 2 weeks post RPR• Repeated at 12 months post RPR

– Early cohort of follow-ups– Later cohort of follow-ups

• “Early” and “Later” cohorts refer to RPR pre or post July 2014– Early cohort did not have 2 week survey– Considerable on-going communication with the

profession about purpose & importance– Later cohort also saw emphasis on reviewer skills

Doctors participating in the evaluation

• Surveys of participants two-weeks and twelve-months post-RPR

Later cohort17 of 31 doctors (55% response)

Early cohort:32 of 45 doctors (71% response)

Early cohortCompleted

RPR Jan-May 2014

Later cohortCompleting

RPR from June 2014 - ongoing

Two-week post-RPR survey Twelve-month post-RPR survey

Did not complete RPR post survey - not yet

implemented

Completed twelve-month survey

Closed

Completed two-weeksurvey

159 of 236 doctors (67%) - Ongoing

Completed twelve-month survey

Ongoing

The Participants

Doctor characteristicsTwo-week

respondents(n = 159)

12 month early cohort

(n = 32)

12 monthlater cohort

(n = 17)

RoleGeneral practice 68% 91% 76%

Other 32% 9% 24%

Trained inNew Zealand 36% 34% 35%

Other 64% 66% 65%

Years in practice

0-10 45% 47% 18%

11-20 43% 38% 76%30+ 12% 16% 6%

English first language

Yes 77% 84% 100%No 23% 16% -

Improvements for patients: 2 weeks

• Just under half strongly agreed or agreed that participating in RPR had improved the care they deliver to their patients (44%) and improved their practice in other ways (49%)

Changes to practice: Two weeks after RPR

• Nearly half (46%) said they had already made changes to their practice as a result of RPR

• A further 13% said they intended to make changes

Overall views on RPR: 2 weeks

• The majority (71%) agreed that the practice visit was a positive experience

• Over half found the report useful (63%) and would positively recommend RPR to colleagues (56%)

Examples of immediate changes• Improved records and note taking – most commonly mentioned

Ensuring appropriate documentation of clinical notes. Going deeper into patient history beyond presenting complaint.

• The consultation – style and interaction with patientsTried to change consultation style, trying to prioritise patient questions.Communicating more effectively with patients

• Review of prescribing and ordering lab testsI am a bit more critical about which lab tests I order.•Improving cultural competenceTaking specific interest in Māori and Pacific cultural aspects of patients and trying to integrate them in consultations.

Changes to professional development plans (PDPs) at two weeks

• Doctors evenly divided between those who had and had not made changes to their PDP

• One-third (34%) had discussed their PDP with the reviewer

• Just over half (52%) agreed that their professional development plan is a useful tool to improve their practice

Changes to practice: Twelve months after RPR

59%

19%

I have made changes to my practice

Early cohort

Later cohort

• Early group lacked knowledge of the process & of the importance of the program; later reviewers were more skilled

• Improved relationship with collegial supervisor• Small numbers, but encouraging

Improvements for patients: Twelve months

• The later cohort were much more likely to agree that RPR had improved care (53% compared to 18%)

Changes to PDP: 12 months – Early cohort

• Similar proportions – about one-fifth – reported making changes to their PDP, the way they manage it, and that changes have made it more useful

Changes to PDP: 12 months – later cohort

• Later cohort was much more likely to agree that RPR led them to make changes to their PDPs and that those changes made their PDPs more useful

Overall views on RPR: Twelve months

• Later cohort much more likely to view RPR as useful or very useful - 53% compared to 31%

• Later cohort much more likely to recommend RPR to their colleagues – 65% compared to 28%

What about the RPR reviewers?

• Almost every reviewer provided an example of a benefit to themselves. Some benefits were personal:

Seeing a variety of different GPs in their practices and consulting has been extremely interesting.

• Online survey and interviews in 2014 – to be repeated in 2015/16. All nineteen reviewers responded in 2014.

Reviewer comments• I learn from the positives I observe in sitting through

consults of my colleagues and use it in my own practice.

• It has also been an opportunity to chew the fat over both ordinary and contentious issues and to hear fresh and diverse perspectives. I too feel under the looking glass by those I review so it puts pressure on me to ensure I am really up to date so I appear credible.

Summary• Early experience is encouraging• Better knowledge of the purpose and

importance of revalidation appears crucial to sustained success

• Reviewer skills and understanding are important

Other Groups• Obstetrics & Gynaecology – voluntary• General Practice specialists – voluntary• Orthopaedics- mandatory

– 95% pre-RPR were opposed– 95% post-RPR were in favour

• Greatest benefits in work/life balance, scope of practice

Conclusions to Date• Early indications of value of RPR

– higher proportions reporting sustained benefits• Positive influence on professional

development• Improved collegial relationships• Positive influence on practice • Positive influence on patient care

Next Steps• Continue to highlight the importance of revalidation• Continued 2-week and 12 month follow-ups• As numbers grow we will be able to firm our

conclusions– Including begin to look at competence concerns

• Further survey of reviewers• Map progress into vocational training• Ultimately see processes adopted by other

vocational groups