Creating an Evaluation Process by Dr. Kathryn Rugen
Transcript of Creating an Evaluation Process by Dr. Kathryn Rugen
VETERANS HEALTH ADMINISTRATION
• Describe the structure of the VA Centers of Excellence in Primary Care Education NP Residency Programs
• Explain the development of the NP Residency competency tool
• Appraise the outcomes of the NP Residency• Identify next steps
1
Objectives
VETERANS HEALTH ADMINISTRATION
VA Centers of Excellence in Primary Care Education: NP ResidencyEligibility and Recruitment• New graduates of adult-gerontology primary care or family nurse
practitioner program from an accredited master’s or doctor of nursing practice (DNP) program within the past year
• Obtain board certification and state APRN licensure in 90 days of starting • All the CoEPCEs have been successful in recruiting NP students who have had
clinical practicums in CoEPCE to continue in their NP residency program. Program Attributes• 12 months long supported with a trainee stipend which includes benefits for
healthcare, vacation and sick leave• Majority of time in primary care with own panel, specialty and inpatient
rotations• Focus on interprofessional collaborative practice, leadership, scholarship, QI• Co-precepting in last 6 mo
2
VETERANS HEALTH ADMINISTRATION
• Demonstrate program effectiveness
• Standardization across 5 sites
• Document competence in 7 domains
• Prepare for site accreditation
NP Competency Tool
VETERANS HEALTH ADMINISTRATION
– AACN/CCNE Masters and DNP Essentials– AACN/NONPF Adult-Gerontology Nurse Practitioner
Core Competencies – NCQA PCMH Standards– Core Competencies for Interprofessional
Collaborative Practice (IPEC)– ACGME competencies– VA top outpatient diagnoses– COE education core domains – Entrustable Professional Activities
Development
VETERANS HEALTH ADMINISTRATION
• Iterative process – VA NP experts at each site and MD education
consultant
– Post-graduate NP trainee reviewed and offered suggestions
– Solicitied input from experienced and new NPs throughout VA Primary Care
Content validity
VETERANS HEALTH ADMINISTRATION
• Clinical competency in planning and managing care• Leadership• Interprofessional team collaboration• Patient-centered care• Shared decision making• Sustain relationships• Quality improvement and population management
Domains
VETERANS HEALTH ADMINISTRATION
• Assess, diagnose, treat and manage health conditions commonly seen in primary care
• Conditions prevalent in the Veteran population - Post-Traumatic Stress Disorder (PTSD), Traumatic Brain Injury (TBI), military sexual trauma, suicidality, and hepatitis C virus
• Perform a comprehensive history and physical exam, construct pertinent differential diagnoses, order appropriate screening diagnostic tests and appropriate medications, perform comprehensive medication review and reconciliation, present clear, concise and organized patient case
Clinical Competency
VETERANS HEALTH ADMINISTRATION
• Lead team huddles, case conferences, team meetings, quality improvement projects, shared medical/group appointments
• Apply leadership strategies that support collaborative practice and team effectiveness
Leadership Competency
VETERANS HEALTH ADMINISTRATION
• Develop own professional identity and ability to explain one’s role to patients, families and other professions
• Use respectful language, understand and appreciate contribution of other team members
• Function as a resource to other professions, maintain open communication with team members
• Safely transition patients among team members • Seek feedback from team members, constructively
manage disagreements with team • Engage in continuous professional and interprofessional
development to enhance team performance
Interprofessional Collaboration Competency
VETERANS HEALTH ADMINISTRATION
• Communicate with patients between office visits by telephone, secure messaging and telehealth monitoring
• Elicit patient’s values, preferences, and cultural beliefs • Identify, accommodate, and customize care for patients
with language, cognitive, functional or cultural barriers • Assess and provide education to empower patients to
self-manage their chronic conditions• Track and coordinate care by ensuring follow-up on
messages, tests, consults, and care at outside facilities • Engage other health professionals in shared patient-
centered problem-solving • Use motivational interviewing to help change health
related behaviors
Patient Centered Care Competency
VETERANS HEALTH ADMINISTRATION
• Use active listening skills and open-ended question during a patient visit
• Counsel and support patients in their self-management of chronic diseases
• Facilitate patient’s participation in healthcare decisions using decision aids
• Engage patient in advanced care planning • Activate community resources for patients or population
needs • Engage patients as care team members in tracking and
coordinating care • Share accountability with other professions, patients and
communities for outcomes relevant to prevention and health care
Shared Decision-Making Competency
VETERANS HEALTH ADMINISTRATION
• Devise, follow, review and adjust a longitudinal care plan to meet the patient’s needs
• Develop and sustain a respectful and trusting relationship with the clinic staff, the faculty, their peer learners and their patients/families
• Give timely, sensitive instructive feedback to others about their performance on the team
• Respond respectfully to feedback from others
Sustained Relationships Competency
VETERANS HEALTH ADMINISTRATION
• Access and interpret clinic performance data • Improve care through Plan-Do-Study-Act cycles • Perform root cause analyses and reflect upon critical
incidents (medical error, near miss, preventable emergency room visits or readmissions)
• Query registries to determine the health status and care needs of the entire practice and/or specific populations of interest i.e. all diabetic patients
• Reflect on individual and team performance and introduce strategies for improvement
QI/Population Health Competency
VETERANS HEALTH ADMINISTRATION
• NP resident and mentor complete competency tool at 1, 6, and 12 months (total 69 items)
• Rate on 0-5 scale– 0= not observed or not performed– 1= observes task only– 2= needs direct supervision– 3= needs supervision periodically– 4= able to perform without supervision– 5= able to supervise others- aspirational!NP resident responds to open ended questions
Methods
VETERANS HEALTH ADMINISTRATION
• Evaluation questions: – identify items and domains NP residents are strongest and
weakest – determine how NP residents progress over time – determine agreement between trainee and mentor ratings
• Descriptive statistics to evaluate the distributional characteristics of each item and domain, the impact of the time on trainee and mentor
• T-test and general linear models to assess relationship between NP resident and mentor ratings over time
Analysis
VETERANS HEALTH ADMINISTRATION
Number of NP Post-Graduate Trainees
2011-2012
2012-2013
2013-2014
2014-2015
2015-2016
2016-2017
Boise - 1 2 4 2 2Cleveland - - 2 4 4 4SanFrancisco - 3 3 5 3 5West Haven 3 4 4 5 6
(1post DNP)6
Seattle - - 1(1 post-DNP)
1 2(1post DNP)
2
Houston - - - - - 2Los Angeles - - - - - 2
Total 3 8 12 19 17 23
VETERANS HEALTH ADMINISTRATION
CharacteristicAge
n reportingMean (years)SDRange
1034.19.4
27-59Gender (n, %)n reportingFemale Male
3832 (84.2%)6 (15.8%)
Prior CoE student (n, %) n reportingYesNo
3819 (50.0%)19 (50.0%)
Retained in VA After Training (n, %)n reportingYesNo
5924 (40.6%)
35 (59.4%) (2 Yale geriatric fellowship, 1 VAQS)Retained in VA Primary Care (n, %)n reporting 8/24 (33.3%)
Cardiology pain clinic VAQS Women's Health
Demographics
VETERANS HEALTH ADMINISTRATION
CharacteristicAverage Years of RN experience prior to NP ResidencyNaverageRangeMedianMode
385.4 years
0-31 years51
Type of NP programNgraduate entryBSN to MSN (traditional)
3817 (44.7%)21(55.3%)
Demographics
VETERANS HEALTH ADMINISTRATION
Subscale
Trainee Ratings Faculty Ratings
1 month6
months12
monthsp-value 1 month
6 months
12 months
p-value
Clinical Competency in Planning/Managing Care
nMeanSDRange
372.75.56
1.71-3.85
343.41.46
2.28-4.25
353.751.430 -
5.00
<.000137
2.94.60
1.86-4.59
343.68.49
2.89-5.00
364.42.50
3.50-5.00
<.0001
Clinical Competency
VETERANS HEALTH ADMINISTRATION
Clinical Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Mea
n
Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
Subscale
Trainee Ratings Faculty Ratings
1 month6
months12
monthsp-value 1 month
6 months
12 months
p-value
LeadershipnMean SD Range
371.451.35
0-4.85
342.411.58
0-5.00
353.131.56
0-5.00
<.0128
2.641.231.00-4.33
293.63.67
2.00-5.00
364.44.55
3.20-5.00
<.0001
Leadership Competency
VETERANS HEALTH ADMINISTRATION
Leadership Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
2.1 Lead PACTteam huddle
2.2 Lead caseconference
2.3 Lead teammeeting using
conflictmgmt/resolution
2.4 Lead groupeduc activities for
pts/fam, PACTteam, peers
2.5 Lead PACTteam qualityimprovement
project
2.6 Leadshared/groupmedical appts
2.7 Applyleadership
strategies tosupport
collaborativepractice/teameffectiveness
Mea
n
Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
Subscale
Trainee Ratings Faculty Ratings
1 month6
months12
monthsp-value 1 month
6 months
12 months
p-value
Interprofessional Team Collaboration
nMean SD Range
373.48.65
2.33-4.77
344.06.57
2.66-5.00
354.101.53
0-5.00
<.0236
3.65.70
2.00-5.00
334.13.42
3.44-5.00
364.59.42
3.89-5.00
<.0002
Interprofessional Collaboration Competency
VETERANS HEALTH ADMINISTRATION
Interprofessional Collaboration Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Mea
n
Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
Subscale
Trainee Ratings Faculty Ratings
1 month 6 months12
monthsp-value 1 month 6 months
12 months
p-value
Patient Centered-CarenMeanSD Range
373.21.68
1.71-4.71
343.94.52
2.85-5.00
354.041.51
0-5.00
<.00236
3.43.69
2.29-5.00
334.08.51
2.86-5.00
364.61.46
3.71-5.00
<.0002
Patient Centered Care Competency
VETERANS HEALTH ADMINISTRATION
Patient Centered Care Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
4.1 Communicatewith pt between
office visits byphone, secured
email,MyhealtheVet
4.2 Elicit pt values,preferences, and
cultural beliefregarding care
4.3 Identify,accommodate, andcustomize care forpts with language,
cognitive,functional or
cultural barriers
4.4 Assess/provideeducation to
empower the ptsto self-manage
chronic conditions
4.5Track/coordinate
care for ptsensuring follow-upon messages, tests,
consults, care atother facilities
4.6 Engage healthprofessionals,
appropriate to thespecific care
situation, in sharedpt centered
problem solving
4.7 Usemotivationalinterviewing
Mea
n
Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
Subscale
Trainee Ratings Faculty Ratings
1 month 6 months 12 months p-value 1 month 6 months 12 months p-value
Shared Decision Making
nMean SD Range
373.17.71
1.28-4.57
343.86.55
2.28-5.00
353.961.52
0-5.00
<.00337
3.42.72
2.14-5.00
343.99.53
2.71-5.00
364.63.46
3.71-5.00
<.0001
Shared Decision Making Competency
VETERANS HEALTH ADMINISTRATION
Shared Decision Making Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Mea
n
Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
SubscaleTrainee Ratings Faculty Ratings
1 month 6 months12
monthsp-value 1 month 6 months
12 months
p-value
Sustained Relationships
nMean SD Range
373.46.69
2.00-4.83
344.01.43
3.16-5.00
354.041.53
0-5.00
<.0437
3.62.70
2.00-5.00
344.15.44
3.40-5.00
364.70.44
3.67-5.00
<.0001
Sustained Relationships Competency
VETERANS HEALTH ADMINISTRATION
Sustained Relationships Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
6.1 Devise, follow,review, adjust
longitudinal care planto meet assisgned pt
panel needs
6.2 Develop/sustainrespectful and trustingrelationship with clinic
faculty, preceptor,mentor
6.3 Develop/sustainrespectful and trustingrelationship with peer
trainees
6.4 Develop/sustain arespectful & trusting
relationship with PACTteam (includes clinic
staff)
6.5 Develop/sustain arespectful & trusting
relationship withpts/families
6.6 Give timely,sensitive, instructivefeedback to others
about theirperformance on team
Mea
n Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
SubscaleTrainee Ratings Faculty Ratings
1 month 6 months 12 mos p-value 1 month 6 months 12 mos p-value
Quality Improvement/Pop Management
nMean SD Range
312.071.51
0-5.00
342.821.59
0-5.00
353.261.54
0-5.00
<.000127
2.531.201.00-5.00
313.43.72
1.75-5.00
344.2669
2.40-5.00
<.0001
QI/ Population Management Competency
VETERANS HEALTH ADMINISTRATION
QI/ Population Management Competency
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
7.1 Access/interpretperformance data
7.2 Improve care via Plan-Do-Study-Act cycles
7.3 Perform root causeanalyses and reflect upon
critical incidents
7.4 Query registries todetermine the healthstatus/needs of entirepractice/population of
interest
7.5 Reflect onindividual/team
performance & introducestrategies for improvement
Mea
n Mentor_1m
Mentor_6m
Mentor_12m
Trainee_1m
Trainee_6m
Trainee_12m
VETERANS HEALTH ADMINISTRATION
• At 1 month, 24 out of 28 items were rated between 2 and 3 (2= needs direct supervision; 3=needs supervision periodically) only four items were rated greater than 3 by the NP Residents.
• Four items rated higher than 3 were “perform comprehensive history and physical exam” (3.48), “perform medication reconciliation” (3.54) and “management of hypertension” (3.13) and “management of obesity” (3.35).
• At the 12 month time point all items were rates higher than 3 and seven items out of 28 were rated higher than 4 (able to perform without supervision) by the NP Residents
• The seven items rated 4 or higher were “perform comprehensive history and physical exam” (4.17), “order appropriate consults” (4.11), “perform medication reconciliation” (4.14) “management of hypertension” (4.08), “management of obesity” (4.11) “management of gastroesophageal reflux” ( 4.02), and “management of osteoarthritis” (4.00).
• At the 12 month time point the mentors ratings were all above 4 (4=able to perform without supervision) except for two items, “management military sexual trauma” (3.58) and “ management of traumatic brain injury” (3.66).
Item Analysis -Clinical Competence
VETERANS HEALTH ADMINISTRATION
Item Analysis: Leadership Competency
• NP Residents rated themselves between 1 and 2 (1= observes only; 2= requires direct supervision) on each item within the leadership competency domain at 1 month.
• At 12 months, NP Residents ratings were between 2.5 and 3.5 (2= requires direct supervision; 3= needs partial supervision); two items that remained low at the 12 month time point were “lead PACT huddles” (2.68) and “lead shared medical appointments” (2.80).
• Mentors ratings at 12 months on the leadership domain were 4 (4=able to perform without supervision) or above except on one item, “lead team meeting using conflict management/resolution” (3.41).
34
VETERANS HEALTH ADMINISTRATION
Item Analysis: QI/Population Mgmt
• Both NP Residents and mentor ratings were low at 1 month; between 1 and 2 (1= observes only; 2= requires direct supervision) on each item except NP Residents higher rating on “reflect on individual/team performance and introduce new strategies for improvement” (2.43).
• By 12 months, both NP Residents and mentor ratings were 3 (3= needs partial supervision) and above, with the exception of “perform root case analyses and reflect upon critical incidents” (NP residents= 2.94; mentors=2.80)
35
VETERANS HEALTH ADMINISTRATION
• At 1 month highest ratings for both NP residents and mentors were interprofessional collaboration, sustained relationships, patient centered care, and shared decision making domains
• These domains remained the highest rated at the end of 12 months by both NP residents and mentors
• Domains that were rated lower at 1 month by both NP residents and mentors were clinical competency, leadership, and quality improvement.
• All domains improved significantly over the 12 month period (p< .002 - .0001)
Findings
VETERANS HEALTH ADMINISTRATION
• At 12 months, mentor ratings were at a 4 or higher on each domain
• Interprofessional collaboration remained the highest rated domain at 12 months for the NP residents and sustained relationship was the highest rated by the mentors
• Quality improvement/population management was the lowest rated domain at 12 months
• Ratings were comparable at 1 and 6 months, but statistically different at 12 months in several domains
• NP residents generally rated themselves lower than their mentors
• COE general program evaluation findings show that NP residents are highly satisfied with the program
Results
VETERANS HEALTH ADMINISTRATION
Psychometrical Analysis
• Internal consistency calculated by NP resident and mentor for each domain and each time point
• Cronbach’s alpha ranging from 0.82-0.96
38
VETERANS HEALTH ADMINISTRATION
• List 2 things you do well• List 2 things you would like to improve• Set 2 short term goals that you can achieve in the next 3
months• Set 1 long-term goal that can be achieved by the end of
the residency• Describe how you will know if you achieved these goals• Describe any potential opportunities/obstacles you
might encounter as you try to reach these goals• Describe your strategies for achieving these goals• Other comments
Qualitative Questions
VETERANS HEALTH ADMINISTRATION
• Mapped to 7 competency domains plus personal attribute category• Most comments in clinical and personal attribute categories
– Clinical: comprehensive care, clinical guidelines, medication management, behavior modification skills, pathophysiology, care planning, diabetes management
– Personal attributes: time management, manage full load of patients, confidence, comfortable, work-life balance, organized, seeing more patients in less time, competence, knowledgeable primary care provider
• Overall agreement (between 3 raters) was .54 (Kappa statistic, this value indicates Moderate agreement) after first pass rating
• 100% after second pass rating
Qualitative Analysis
VETERANS HEALTH ADMINISTRATION
Final Program Evaluation• What did you like best about the NP residency program?• What did you like least about the NP residency program?• Now that your NP residency is completed, how confident and proficient do you feel in
your role as an advanced practice provider? Why?• Tell us the top 3 things we can do to improve the NP residency.• Were the program expectations for your performance, too high, too low, or just right?
How?• What specific experience or skill during your residency helped to advance your
transition towards a confident, proficient advanced practice provider?• What additional specific experience or skill that was NOT provided in the NP residency
would have helped to advance your transition towards a confident, proficient advanced practice provider?
• Thinking back, tell us why you selected to participate in a NP residency.• If you had to do it over again, would you participate in a VA Primary Care NP
residency?• Tell us if you would like to seek employment in VA, Why? Why not?
41
VETERANS HEALTH ADMINISTRATION
Final Program Evaluation• Please rate the following: 5= excellent, 4= good, 3= fair. 2= poor, 1 = very poor• Your preceptors• The staff• Your learning experience with trainees from other professions• Your ability to lead interprofessional teams• Your ability to work in an interprofessional collaborative practice setting• Your ability to implement quality improvement strategies• Your ability to manage chronic diseases• Your ability to develop differential diagnoses• Your ability to use shared decision-making with the patient and family• Your ability to develop sustained relationships with the patient and family
42
VETERANS HEALTH ADMINISTRATION
• Expand the use of the tool to the 2 new VA Centers of Excellence (Houston and West Los Angeles) and other VA and private sector sites
• Publish quantitative and qualitative outcomes• Conduct further psychometric analysis
Next Steps