Overview of Institutional Assessment

Institutional Assessment and Educational Initiatives

David S. Roffman, PharmD BCPSAssociate Dean for Academic Affairs

Institutional Assessment and Educational Initiatives

David S. Roffman, PharmD BCPSAssociate Dean for Academic Affairs

© 2006 University of Maryland School of Pharmacy. All rights reserved.

Institutional AssessmentInstitutional Assessment

Goal: To institute a comprehensive, iterative process of continuous improvement evaluating key issues of critical importance, regardless of a seemingly endless supply of data.

Components and Considerations: Institutional assessment must be both qualitative and quantitative, contain longitudinal data to detect trends over time, and incorporate peer comparisons and benchmarks when available.



P1 P2 P4P3 AlumniFaculty Staff Preceptors

Institutional

Programmatic

LearningOutcomes

Curriculummapping

ePortfolios

GraduatingStudentSurvey

OrientationSurvey

ContinuingStudentClimateSurvey

FacultySurvey

StaffSurvey

AlumniSurvey

PreceptorSurvey

CoordinatedP1 exams

OSCEs

NAPLEXresults

Course evaluations



EXAMPLES

1. Institutional Level: course evaluations

2. Programmatic Level: curriculum mapping

3. Learning Outcomes Level: OSCEs & portfolios


1. Institutional Level Assessment1. Institutional Level Assessment

Standard course evaluation questions: was the syllabus clear, did the coursemaster address class-wide issues, was the textbook helpful, were prerequisite courses adequate preparation for this course…

Shady Grove: “Asynchronous availability of lecture material was

beneficial to my formal learning experience in this course.”

Student feedback indicated a need to improve on presentation quality of asynchronous lectures delivered via MediaSite, so CNS support staff created “Building a Better Presentation,” a workshop for faculty, post-docs and teaching assistants.


2. Programmatic Level Assessment2. Programmatic Level Assessment

At a programmatic level, the revised PharmD curriculum is being mapped for science content and terminal performance outcomes.

Mapped 198 items in 4 areas of science: • basic biomedical – immunology, biochemistry…• pharmaceutical – pharmacology, biopharmaceutics…• social/administrative – ethics, practice management…• clinical science – drug information, medication safety…

At what level of emphasis – introduced, reinforced, or emphasized

Have we covered everything that needs to be covered in a comprehensive PharmD curriculum, and are there any hidden gaps? The faculty will determine if or how those areas should be addressed.


3. Learning Outcomes Assessment3. Learning Outcomes Assessment

Terminal performance outcomes (TPOs) form the “strategic plan” for the PharmD curriculum. These are the 63 “goals” we set for our students to reach by the time they graduate. Each course supports these goals with their own course objectives and assessments.

TPOs are aligned with the 3 areas of CAPE* Outcomes:

1.Pharmaceutical care (60% of our TPOs)

2.Health care system (20% of our TPOs)

3.Population (20% of our TPOs)

*Center for the Advancement of Pharmacy Education


Educational Initiatives - OSCEsEducational Initiatives - OSCEs

Objective Structured Clinical Exams, or OSCEs, are a means of measuring student progress toward the terminal performance outcomes.

OSCEs are standardized simulations of clinical experiences using professional actors. They are used in Canadian medical/pharmacy licensing exams and in USMLE.

Traditional assessment methods are not as useful for evaluation of clinical skills, including clinical knowledge and interpersonal and problem-solving skills.



EXAMPLE: P2 Fishbowl Learning Objectives: After completing pre-learning activity assignments and participating in the session, student groups will be able to do the following:

• After watching a simulated patient counseling encounter, develop an analytical checklist of performance and/or knowledge items that should have been demonstrated by the “student” during the encounter.

• After re-watching a simulated patient counseling encounter, rate “student” performance using an analytical checklist and global impression scale.

• Interview a simulated patient to collect a past medical history, medication history, and focused review of systems.

• Practice and observe a variety of verbal and non-verbal communication behaviors such as active listening, empathy, use of lay terminology, and re-direction of interview.


History of OSCE at MarylandHistory of OSCE at Maryland

• Summer 2006 – team sent to Toronto for 3-day training workshop

• 2006-2007 academic year – OSCE piloted in PCPPM course

• 2007-2009 – OSCE expanded to years 2-4 of the curriculum, blueprint developed based on program TPOs, and practice opportunities integrated into courses

• 2009 and beyond – development of interdepartmental committee to direct OSCE activities, further expand blueprint, add additional practice opportunities


Proposed OSCE Activities (housed within Abilities Lab sequence)Proposed OSCE Activities (housed within Abilities Lab sequence)

Year 1– TOSCE fall semester– TOSCE + 3-station OSCE spring semester



Year 4– 5-station OSCE



Graded OSCE

STEP 1

Student is presented with a case scenario



Graded OSCE

STEP 2

Student meets with standardized patient (professional actor) and student interviews patient, performs task/skill in stations, such as

– Symptom analysis– Medication history– Patient interviewing– Vitals



Graded OSCE

STEP 3: Patient evaluates student using a standard rubric


Educational Initiatives - PortfoliosEducational Initiatives - Portfolios

Electronic student portfolios were established in the Fall 2009 to generate a sense of awareness and self-evaluation that leads to continuous professional development. In the future, preceptors and faculty will also be able to comment and review.

Two main components:

1. Students will store meaningful “artifacts” throughout their education, and be able to look back at previous entries and see their growth.

2. Each semester, students self-evaluate their competency on the terminal performance outcomes, identify knowledge gaps and plan for improvement.


Educational Initiatives – Contemporary IssuesEducational Initiatives – Contemporary Issues

Contemporary Issues is a concept that was identified by the Curriculum Committee during the redesign process: students needed more exposure to relevant current events that affect the profession.

Students are asked to attend 2 educational forums and 2 social events each semester, then write up guided reflections which are stored in their electronic portfolios as “artifacts.”

• Purpose of educational forums is to inspire and intrigue their professional interests with “hot topics” in healthcare (interdisciplinary where possible)

• Purpose of social events is to explore the more informal side of their pharmacy education, and build important skills such as interpersonal communication, cultural awareness and collegiality

Overview of Institutional Assessment

Education

Transcript of Overview of Institutional Assessment