BY: CHANDLER HICKS Civil War Computer Competency By: Chandler Hicks.
Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program...
Transcript of Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program...
![Page 1: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/1.jpg)
Program Improvement through Program Evaluation
Patricia Hicks, Program Director, CHOPAdam Pallant, Program Director, Brown
![Page 2: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/2.jpg)
Disclosures
Drs. Hicks and Pallant have no relevant financial relationships to disclose nor conflicts of interest to resolve.
![Page 3: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/3.jpg)
Objectives for the Presentation
• Review ACGME Program Requirements regarding Program Evaluation and Program Improvement.
• Review models and strategies used in program evaluation
• Share examples and lessons learned
![Page 4: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/4.jpg)
Outline• Overview of purpose of program evaluation
• Review of ACGME program evaluation requirements
• Examine a conceptual model for program evaluation
• Review potential sources of outcome data
![Page 5: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/5.jpg)
Why do Program Evaluation?
• Inform various stakeholders about the effectiveness of the “program”• Program = curriculum
• Program = entire program
• Program = smaller aspect of program
• To address outcomes of learners in the aggregate
• To establish new goals (needs assessment)
![Page 6: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/6.jpg)
It’s The Journey-Not the Destination!
Where do I begin?
What are the rules?
What should I expect of myself?
What should I expect of others?
How might I feel about this?
![Page 7: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/7.jpg)
http://www.acgme.org/Outcome/
![Page 8: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/8.jpg)
Get to Know The ACGME!
Peruse the program requirements
•Common/ Institutional
•PediatricLearn how to use the PDF search box!
![Page 9: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/9.jpg)
New Rules: New Challenges:New Opportunities-2011
528 V.C. Program Evaluation and Improvement530 V.C.1. The program must document formal, systematic
evaluation of the curriculum at least annually. The program must monitor and track each of the following areas:
a) resident performance;
b) faculty development;
c) graduate performance, including performance of program graduates on the certification examination; and
d) program quality
![Page 10: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/10.jpg)
Specifically:V.C.1.d)(1) Residents and faculty must have the opportunity to evaluate the program confidentially and in writing at least annually, and
V.C.1.d).(2) The program must use the results of residents’ assessments of the program together with other program evaluation results to improve the program.
![Page 11: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/11.jpg)
V.C.2. If deficiencies are found, the program should prepare a written plan of action to document initiatives to improve performance in the areas listed in section V.C.1.
The action plan should be reviewed and approved by the teaching faculty and documented in meeting minutes.
![Page 12: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/12.jpg)
Using program requirements to improve your program
![Page 13: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/13.jpg)
ACGME Program Requirements
V.C. Program Evaluation and Improvement1. The program must document formal, systematic
evaluation of the curriculum at least annually. The program must monitor and track each of the following areas:a) resident/fellow performance
b) faculty development
c) graduate performance
d) program quality
![Page 14: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/14.jpg)
ACGME Program Requirements
V.C. 2. If deficiencies are found, the program should prepare a written plan of action to document initiatives to improve performance. The action plan should be reviewed and approved by the teaching faculty and documented in meeting minutes.
ACGME Program Requirements: Categorical Pediatrics, V.C., p 35ACGME Program Requirements: Subspecialties of Pediatrics, V.C., p 20-21
![Page 15: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/15.jpg)
What the????
Take a Deep Breath…think of what you’re already doing!
You get to choose how, where, when, and what YOU want to do!
Don’t let the perfect get in the way of the good!
![Page 16: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/16.jpg)
Do you already do this?
Do you survey anyone?Residents, faculty,
applicants, graduates…
Do you have a curriculum committee?
Do you speak at faculty meetings?
Do you work with an advisory group?
Do you teach your advisors how to advise?
Do you get feedback about your program?
Has you program ever had a problem that you have tried to fix???
![Page 17: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/17.jpg)
Draw on the wisdom and energy of others to assist you
Wisdom of the ages…•When someone has a problem -ASK THEM TO HELP TO FIX IT!
•Residents often know best how to solve problems that impact them directly!
•Forgive yourself when the best laid plans do not work!
![Page 18: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/18.jpg)
Common PIF Questions
“Describe the approach used for program evaluation.”
Survey
Electronic Feedback
Board Scores
Patient, nurse feedback
Written evaluations
Consultant
Internal Reviews
Other ideas???
![Page 19: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/19.jpg)
Common PIF QuestionsDescribe the improvement efforts…currently
undertaken in the program based on feedback from the ACGME resident survey.
This is ahidden gift.
![Page 20: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/20.jpg)
ACGME Resident Survey Questions
8. Do you have the opportunity to confidentially evaluate your overall program in writing or electronically at least once a year?
15. Have residents/fellows had the opportunity to assess the program for purposes of program improvement?
Please consider looking at ALL of the data over time…are you improving on scores of “teaching”, “service vs education”, etc…
![Page 21: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/21.jpg)
Task 1 – evaluation purpose or need• For whom is the evaluation being
conducted?
• What is the focus of the evaluation?
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
Task Oriented Conceptual Model for Program Evaluation in GME
![Page 22: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/22.jpg)
Task 2 – What is the focus of the evaluation?• Overall training program• Component of the training program
• Rotation• Event• Project• Curriculum
• Person• Faculty (teachers)• Leadership (PD, etc.)
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
Task Oriented Conceptual Model for Program Evaluation in GME
![Page 23: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/23.jpg)
Task 3 – determine evaluation methodology• When will the evaluation be conducted?
• What setting will you conduct the evaluation?
• How will the data be collected?
• What type of data analysis will be utilized?
A COMBINATION OF METHODOLOGIES IS RECOMMENDED
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
Task Oriented Conceptual Model for Program Evaluation in GME
![Page 24: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/24.jpg)
Task 4 – Present (report) evaluation results• Who are the stakeholders who should receive
results?
• In what forum should the results be presented
• How should results be interpreted or presented
• When should the results be presented?
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
Task Oriented Conceptual Model for Program Evaluation in GME
![Page 25: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/25.jpg)
Task 4 – Present (report) evaluation results• Who are the stakeholders who should receive
results?
• In what forum should the results be presented
• How should results be interpreted or presented
• When should the results be presented?
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
Task Oriented Conceptual Model for Program Evaluation in GME
![Page 26: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/26.jpg)
Task 5 – Documentation of evaluation results• Content specific reports
• Interpretation or just raw data?
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
Task Oriented Conceptual Model for Program Evaluation in GME
![Page 27: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/27.jpg)
Task 6 – Action plan• Assignments of follow-up and/or changes should align
with stakeholders, directors, responsible parties
• Timelines
• Measurable milestones to assess achievement of action/change
ASSESSMENT DRIVES LEARNING
PROGRAM EVALUATION DRIVES CHANGE, DEVELOPMENT, IMPROVEMENT
Task Oriented Conceptual Model for Program Evaluation in GME
Musick, D. A Conceptual Model for Program Evaluation in GME Acad Med 2006;81(8):759-765
![Page 28: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/28.jpg)
Other considerations in design and implementation of program evaluation
• Feasibility
• Culture of change – institutional or group readiness
• Knowledge of evaluation methods
![Page 29: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/29.jpg)
Methodologies available for program evaluation – development needed !!
• Counting of things
• Performance on low or high stakes examinations
• Performance on standardized cases (real, simulated, standardized patients, etc)
• Rating, direct observations, checklists
• Surveys
![Page 30: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/30.jpg)
“Not everything that counts can be counted and not everything that can be counted counts”
-Albert Einstein
![Page 31: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/31.jpg)
What sources of outcome data do I have for my program?
![Page 32: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/32.jpg)
References to Documentation: You are already collecting data!
Pediatric Program Requirements32 references to required documentation
11 in the companion document
Subspecialty Requirements16 references to required documentation
8 in the companion document
![Page 33: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/33.jpg)
Evaluations = Outcome Data
Use individual learner data to evaluate learner and help them progress and graduate
Use aggregate learner data to evaluate the program and help it improve the delivery of education
![Page 34: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/34.jpg)
Using feedback from assessments and surveys to improve PROGRAM outcomes
Program evaluation• Aggregate data of/from individuals
• Evaluations specific to the program• Annual evaluation by residents and faculty
• Graduate surveys
• ACGME Resident Survey
• “Milestones report”?
![Page 35: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/35.jpg)
Sources of outcome data
General• Current ACGME citations• Procedural skill documentation• Scholarly productivity of faculty• Patient volume, variety of diagnoses• In-training exam scores• Conference attendance data• Duty hours logs• Board take and pass rates• ACGME Resident Survey
![Page 36: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/36.jpg)
Sources of outcome data
Semiannual reviews with residents/fellows• Feedback from trainee
• Completion of core curriculum, other assignments
• Individual learning plan needs
• Quality improvement activities
• Fatigue, burnout, professionalism
• Moonlighting
• Career plans
![Page 37: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/37.jpg)
Sources of outcome data
Evaluations• Faculty of trainee
• Trainee of faculty (required annually)
• Trainee of program (required annually)
• Faculty of program (required annually)
• Others of trainee (direct observation, multisource feedback)
• Internal review by GMEC (mid-accreditation cycle)
![Page 38: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/38.jpg)
Sources of outcome data
Scholarship oversight committee• Membership, meeting frequency
• Scholarly work product of fellow• Identification of a mentor
• Progress on project
• Quality of project
• Grants
• Presentations and Publications
![Page 39: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/39.jpg)
Lessons Learned through Program Review
![Page 40: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/40.jpg)
Adam’s True Confessions
Not effectiveSomewhateffective Effective
Very effectiveExtremelyeffective No opinion/ don't
know
S1
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
Feedback Value-faculty
Series1
![Page 41: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/41.jpg)
North…Rainbow
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
2007-2008 supervision
NorthstarRainbow
![Page 42: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/42.jpg)
0
5
10
15
20
25
30
Adam's Review (residents) 2008-9
1 (poor)2345 (excellent)
![Page 43: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/43.jpg)
Attempt to Restructure Coverage for a Specialty Service
• Float in call was arranged for a Q4 specialty service where fewer than 4 residents were needed for coverage during the day.
• Feedback was immediate and came through many sources of data• Residents not familiar with the specialty service felt
unprepared to accept sign-out
• Duty hour violations
• Unhappiness amongst team members
![Page 44: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/44.jpg)
Asthma Education
Goal: Get interns to know and use the asthma pathways correctly in their CC setting
Curriculum: Taught pathways with ppt, case-based examples
Outcome: Residents performed 98% correct on multiple choice test about pathway use
![Page 45: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/45.jpg)
Asthma Education
Other outcome: Chart review revealed:
• Only 30% of interns correctly identified classification of asthma severity
• Poor documentation regarding questions asked to determine severity
• Pathway not followed for 50% of those who were classified correctly
![Page 46: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/46.jpg)
Now that we have told you what you can learn, let’s talk about some strategies
![Page 47: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/47.jpg)
The Simplest Example: Medical Knowledge
ReceiveITE exam
scores
Review at Residency Committee
meeting
Assess factors contributing
to the Program’s ITE
scores
Develop and Document an Action Plan
Implement the Action
Plan
![Page 48: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/48.jpg)
Raising the research bar in fellowshipsScholarship Oversight
Committee reports
Monitor grants, presentations, publications at Department
Fellowship Committee meeting
Assess contributing
factors
Compare programs (internal/external)
Develop and implement an action plan
Set benchmarks
Improve core curriculumTime projects to grant/abstract deadlinesFaculty development for research mentors
![Page 49: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/49.jpg)
Comprehensive Internal Review of Rotations in Pediatrics (ChIRRP)
CHIRRP Reports: Action
plan availablefor rotation director and ACGME site
visitor
Review by Residency Education
Committee
Review by Program Director
Summary by committee reviewer
Rotation evaluations
Evaluations of faculty
Curriculum review
document
![Page 50: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/50.jpg)
Comprehensive Internal Review of Rotations in Pediatrics
• Review of rotation curricula
• Review of faculty teaching
• Comparison of teaching among rotations
• Opportunity to make improvements at a rotation level
![Page 51: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/51.jpg)
One Approach to Program Evaluation Meetings
• Meet twice a year: 2-3 hours • Residents/chief residents
• Associate PDs
• PD
• Directors of various groups leading resident education (Continuity Clinic director, key educational faculty in other areas)
• Discuss evaluation data collected within the past 6 months
![Page 52: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/52.jpg)
Suggested Format of Program Evaluation Report
• Data (from many sources) reviewed by the group
• Results interpreted and discussed
• Additional group input/perspective
• Problem areas identified and listed
• Interventions suggested/action plan designed
![Page 53: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/53.jpg)
• Action plan leader and timeline identified
• Report/update meeting time established
• Minutes distributed
• Timeline for action plan monitored
• Results monitored
Format of Program Review Meetings
![Page 54: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/54.jpg)
Action Item Grid: Monitored by the PD and the Coordinator
Action Plan for Pediatric Residency Program Improvement
Identified problem
requiring action
Action requested
Timeline for result or report
back to PD
Responsible party for
response/action/reporting
Not enough surgical specialty education
Create subspecialty surgical rotation, with emphasis on ped surg, urology, ENT
3 months to establish new rotation with curriculum, evaluation tool and learning activities specified
APDs with Chief of Surgery or his designee (Dr. Hicks to initiate request)
![Page 55: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/55.jpg)
Summary
• Look at program evaluation as an opportunity to improve your program, not just something you have to do for the ACGME
• Start with existing data, then add to it
• Document the process, plans, outcomes!
![Page 56: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/56.jpg)
Resources to Learn More about Program Evaluation
![Page 57: Program Improvement through Program Evaluation€¦ · Program Evaluation Patricia Hicks, Program Director, CHOP Adam Pallant, Program Director, Brown. Disclosures Drs. Hicks and](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c62c21066715c087622a9/html5/thumbnails/57.jpg)
Questions and Practical Answers