Student Learning Assessment Report

MARYMOUNT UNIVERSITY DOCTOR OF PHYSICAL THERAPY DEGREE PROGRAM

SUBMITTED BY: CATHY S. ELROD, PT, PHD DATE: SEPTEMBER 30, 2013

EXECUTIVE SUMMARY All of the Program’s Learning Outcomes

Upon graduation from Marymount University’s DPT program, the student will be able to:

Learning Outcome Year of Last

Assessment

Year of Next Planned

Assessment 1. function independently managing patients with a wide variety of simple or complex

conditions; 2011-2012 2014-2015

2. perform skilled physical therapist examinations, interventions, and clinical reasoning proficiently and consistently;

2011-2012 2014-2015

3. apply best available scientific evidence, clinical judgment, and patient preferences in physical therapy patient management;

2012-2013

4. manage a full-time physical therapist’s caseload to achieve resource-efficient and patient-effective outcomes;

2013-2014

5. exhibit care, compassion, and empathy in delivering physical therapy services sensitive to individual, cultural, and social differences;

2012-2013

6. implement a self-directed plan for professional development and lifelong learning.

2013-2014

2

WHERE AND HOW ARE DATA AND DOCUMENTS USED TO GENERATE THIS REPORT ARE BEING STORED

The data used to analyze these learning outcomes is taken from the several different sources. These sources include a written assignment, a final comprehensive practical exam, the CPI-Web, and student self-evaluations. The results of the written assignment and self-evaluations are stored in the faculty instructor’s office. The results of the practical exams are stored in the Department Chair’s office. CPI-Web is an on-line database that houses summative student evaluation data input by clinical faculty during clinical practicum experiences. Relevant data for this report were identified within CPI-Web, and then downloaded into an Excel workbook for analysis. The data is stored in the Marymount University (MU) Doctor of Physical Therapy (DPT) program share drive. Access is limited to DPT faculty and staff.

HOW THE PROGRAM’S OUTCOMES SUPPORT MARYMOUNT’S MISSION, STRATEGIC PLAN, AND RELEVANT SCHOOL PLAN

The mission of the Doctor of Physical Therapy (DPT) degree program is to prepare generalist practitioners to deliver best available physical therapist practice for improving movement, function and health across diverse individuals and communities. The program employs a dynamic learning-in-context environment that is warm and welcoming. Critical thinking, professionalism, respecting individual differences, and adherence to ethical practices ground all classroom, clinic, and community activities. The mission of the university states, “Marymount University is an independent Catholic university that emphasizes academic excellence at the undergraduate and graduate levels. Committed to the liberal arts tradition, the University combines a foundation in the arts and sciences with career preparation and opportunities for personal and professional development. Marymount is a student-centered learning community that values diversity and focuses on the education of the whole person, promoting the intellectual, spiritual, and moral growth of each individual. Scholarship, leadership, service, and ethics are hallmarks of a Marymount education.” As the mission indicates that Marymount is committed to career preparation and professional development while focusing on the whole person (intellectual, spiritual, moral), the student learning outcomes developed by the PT faculty attempt to show how the DPT program supports this Mission. Not only do our learning outcomes identify the skills necessary to practice in the career of physical therapy (#1-4), they also measure the moral sensitivity to recognize and understand the diversity of individuals (#5) and professional development (#6). The strategic plan, and thus School plan, has been evolving at Marymount University with the transition to a new President. It is the intent of the PT faculty to ensure that student learning outcomes also take into consideration the strategic plan of the University and of the Malek School of Health Professions. Over the next academic year, the PT faculty will review and reflect on the new University strategic plan and its relationship to our student learning outcomes.

3

DESCRIPTION OF THE ASSESSMENT PROCESS USED INCLUDING STRENGTHS, CHALLENGES AND PLANNED IMPROVEMENTS

In the spring of 2012, the PT faculty reviewed the two learning outcomes scheduled for assessment during the 2012-2013 academic year and chose the aspect of each outcome we wanted to focus on for this annual review. A strength of our process is all core PT faculty work together during a scheduled faculty meeting to determine the best way to assess the chosen learning outcomes. Since all faculty are present, the entire curriculum and course work can be discussed, ensuring that all potential assessment methods are considered. Since the tools that are available for assessment are not always quantitative in nature it becomes a challenge to optimally evaluate the outcomes. Thus, the faculty and Chair will continue to reflect on what methods were successful, what were not, and work diligently to ensure that relevant data is collected.

HOW THE PROGRAM IMPLEMENTED ITS PLANNED IMPROVEMENTS FROM LAST YEAR

Specifically from last year’s report, it was identified that some curricular revisions would be beneficial. We indicated that we wanted to integrate clinical reasoning theory and discussions related to expert decision making into an earlier course on case report writing. Starting the fall of 2013, this change occurred. We also stated that we wanted to separate pediatrics into it’s own course that would have an enhanced focus on clinical reasoning in this population and redesign an existing course to have a greater focus in geriatrics. Both of these curricular changes will be presented to the University Graduate Studies Committee this academic year. We also indicated that we would develop a new comprehensive practical examination. Upon reflection at a faculty meeting in May, it was determined that a new exam was not necessary. It was decided that the grading rubric should be revised to better capture what the faculty was trying to assess. This rubric is being redesigned during the fall of 2013. Finally, the use of the portfolio for collecting information on student performance and self-evaluation is also being re-assessed by the faculty of the whole. We are looking at its purpose, design, and use as an assessment tool. We plan on finishing this evaluation by the end of the 2013-2014 academic year.

RESPONSE TO LAST YEAR’S UNIVERSITY ASSESSMENT COMMITTEE REVIEW OF THE PROGRAM’S LEARNING ASSESSMENT REPORT

The comments that were provided last year were helpful and appreciated. We had not given consideration into including students in the dialogue about what they would consider to be the most relevant and valuable material to be collected in their portfolios. We will solicit student feedback to incorporate into our ongoing discussions about the purpose and use of the portfolio this year. We will also gather student feedback on why they believe they only possess an “advanced intermediate” skill as it will help us identify any curricular or programmatic weaknesses.

4

STUDENT LEARNING OUTCOME ASSESSMENT

Learning Outcome #3: Apply best available evidence, clinical judgment, and patient preferences in physical

therapy patient management

Is this outcome being reexamined? Yes No

ASSESSMENT METHOD #1

ASSESSMENT TOOL

Final Assignment in PT 733: Evidence Based Practice II: Applying Evidence in the Clinical Environment

During the spring semester of the second year of the DPT program, students participated in the PT 733, Evidence Based Practice II course. The three concepts that make up this learning outcome: 1) apply best evidence, 2) apply clinical judgment, and 3) apply patient preferences are the cornerstones of evidence-based practice (EBP). The final assignment for PT 733 required students to utilize all three concepts in a written assignment that documents their decision-making about PT management choices for an individual patient (see Appendix 1 for a copy of the assignment). In the assignment, students address 12 different factors that affect management of a unique ‘paper patient’ who has many medical, personal, and environmental factors complicating decision-making about their prognosis and plan of care. Additionally, students provided two short evidence-supported summaries of topics applicable to the case. The one item deemed most representative of each concept was identified and the responses of all students (n=32) on these items were examined against a grading rubric to judge the response as excellent, acceptable, or unacceptable.

ASSESSMENT ACTIVITY

Direct Outcome Measure Performance Standard Data Collection Analysis

Direct measure #1: Final assignment in PT 733 (see above for explanation of assignment). The three concepts/categories for assessment are:

1. applying best evidence

85% of students will score at or above the acceptable level. The standards for what is considered acceptable and above are listed below with a more detailed description of the data/analysis.

Data was collected from spring 2013 final written assignment.

See the table below with the results for each category/concept. For applying best evidence, 87.5% of the students were at or above acceptable. For using clinical judgment, 85% of the students were at or above acceptable. For incorporating patient preferences, 91%

5

Direct Outcome Measure Performance Standard Data Collection Analysis

2. using clinical judgment 3. incorporating patient

preferences

of the students were at or above acceptable.

Summary of Responses in Each Category

Category Excellent Acceptable Not acceptable

1. Applying best evidence* 27 29 8

2. Use clinical judgment** 13 14 5

3. Incorporating patient preferences** 17 12 3

*Based on 64 submissions (2 per student) **Based on 32 submissions (1 per student)

CONCEPT 1: ‘APPLY BEST AVAILABLE EVIDENCE’

This assessment is based on a review of the two short evidence-based summaries that were part of the final assignment. The directions to students for the assignment were “Take two topics of your choice applicable to this case and provide a short evidence-based and clinician friendly summary of key evidence. Each summary should be concise, evidence-based, and easy to ‘digest’ with a quick 2-minute read.”

A Score of:

Excellent: Included 1) recent references, 2) references from scientific journals and high quality sources (NOT general web pages or wikepedia-type references), 3) was well-focused on a clinically important topic, and 4) demonstrated accurate application of best evidence.

Acceptable: Fully met criteria 1 and 2 above, partially met criteria 3 and 4

Unacceptable: Significant flaws in two or more categories

6

INTERPRETATION

The responses are summarized in the table above; 87.5% of scores were either excellent or acceptable. This exceeds the expectation that at least 85% of student responses should be at least at the ‘acceptable’ level. Of the 8 unacceptable responses, three were judged unacceptable because they did not apply the summary to the topic. They appeared to have misinterpreted the assignment and provided a critique of one specific journal article without any application to the project or clinical decision-making. Five others either used significantly outdated references as the basis for their summaries or used websites and/or textbooks exclusively to justify their summaries. Clearer instructions for the activity may have avoided misinterpretation in some people.

CONCEPT 2: USING CLINICAL JUDGMENT

This assessment is based on the review of the item “What is the patient’s prognosis in terms of physical functioning?” Based on course information, clinical judgment involves analysis and synthesis of patient information (medical, personal, environmental) to make a thoughtful and realistic judgment about expected changes in physical functioning; and, ideally, the ability of PT to influence this outcome. Using the rubric identified below, a score of 14-16 was considered as excellent, a score of 10-13 was acceptable and a score of 9 or below was deemed to not be acceptable.

Grading rubric for this assessment item

Excellent = 4 Accurate, well stated

Acceptable = 3 Minor inaccuracies, incomplete

Unacceptable = 2 or lower Information not provide or assumptions are incorrect

States prognosis for physical functioning 4 3 2-0

Considers impact of primary diagnosis 4 3 2-0

Consider impact of comorbid medical conditions

4 3 2-0

Considers psychosocial factors, as applicable

4 3 2-0

7

INTERPRETATION

85% of students achieved a score at or above ‘acceptable’ on this item which meets expectations. Most students with an unacceptable score did not consider the impact of co-morbidities or applicable psychosocial factors. They focused exclusively on the primary diagnosis (which can be misleading).

CONCEPT 3: CONSIDERING PATIENT PREFERENCES

This assessment is based on the review of the item “What are the patient’s objectives for the visit? Why is the information important?” The following served as the grading rubric guidelines:

Excellent: Overtly identifies patient preferences and objectives for the PT visit and indicates that patient preferences are a significant factor in driving patient management decisions

Acceptable: Discusses factors generally viewed as patient preferences, motivations, or goals and gives evidence in the rationale section that they saw a link between patient preferences and motivations and the direction of the PT management of the patient

Not acceptable: Does not address patient preferences, motivation, or goals

INTERPRETATION

As displayed in the summary table above, 91% of students achieved a score of acceptable or excellent on this item. This exceeded the expectation that at least 85% of students would achieve an acceptable or excellent score. The three students who provided unacceptable responses did not provide any evidence that patient-centered factors were considered in their decision-making. They focused exclusively on what they (the therapist) believed was important for the visit.

8

ASSESSMENT METHOD #2

ASSESSMENT TOOL

Final Comprehensive Practical Exam – Capstone Activity

During the spring 2013 semester of the final year of the DPT program, all students in the graduating class undertook a final comprehensive practical examination where a mock patient was created in each of the major clinical areas (Orthopedics, Neurorehabilitation and Acute Care). Students were required to randomly select a patient case, prepare and perform an evaluation of that case and demonstrate an initial treatment session based on the responses elicited through the evaluation. Two faculty members acted in tandem, one as the mock patient and the other as an examiner. The mock patient responded to the student’s questions with appropriate signs and symptoms responses designed to lead the student towards a definitive diagnosis and a potential plan of care. The other faculty member acted as the external examiner questioning the student along the way to ascertain their clinical thinking process and ensuring that they understood the value of the activities they were demonstrating.

Despite their distinct roles in the evaluation, both faculty members contributed to the final assessment of the student performance, one through how the performance looked and the other through how the performance felt. At the completion of the evaluation and treatment, students were required to complete appropriate documentation on the mock patient they had just worked with. This documentation was incorporated into the final determination of success or failure of the examination. Each student examination was videotaped for later review and to allow unsuccessful students to review their performance and learn from their errors.

Each student was evaluated using a standardized rubric during the examination (see Appendix 2). Each session was video-taped to allow for additional review by faculty initially and subsequently by the unsuccessful student to clearly identify unsuccessful performance. By filming each session, additional faculty, especially subject experts, were able to review the session after the fact to determine if the performance was acceptable or not. Unsuccessful students reviewed their performance and discussed their performance with the faculty examiners prior to retaking the examination.

9

ASSESSMENT ACTIVITY

Direct Outcome Measure Performance Standard Data Collection Analysis

Direct measure #2: Final comprehensive practical examination. The exam covers all aspects of clinical practice and is written to ensure that students can effectively and safely manage diverse patient conditions.

100% of the students will pass the examination.

Data was collected from spring 2013 grading rubric for the final comprehensive practical exam (see Appendix 2).

82% of the students were successful at the first attempt, and 100% were successful upon completion of the second attempt.

INTERPRETATION

Thirty-three students attempted the examination with twenty-seven being successful at the first attempt. This meant that 82% of students were successful at the first attempt and, following review of the video recordings of unsuccessful attempts accompanied by any remedial study, all students were successful after a second attempt. This mock patient experience, while only simulating real life, allowed the faculty to observe the clinical judgment processing of the students in a clinical setting. With faculty acting as both patient and examiner, faculty were able to know exactly what condition they were demonstrating and were therefore better able to follow the student reasoning. It allowed the faculty who was acting as the patient to express emotions and answer questions to determine how well the student incorporated the patient’s behaviors and unique attributes into his/her management of the patient. Reviewing the video recording was an enlightening experience for both the student and faculty and added to the conversation of why a student thought a particular way about that case presentation.

EXTENT THIS LEARNING OUTCOME HAS BEEN ACHIEVED BY STUDENTS The direct measures indicate that the students are, overall, applying best evidence, clinical judgment, and patient preferences to successfully and effectively treat patients. The faculty wanted to assess the students’ demonstration of these concepts during the second year of the program to see if the students are starting to integrate them into patient management, thus, giving the students (and faculty) time to continue to build upon the students’ knowledge throughout the remaining 2 semesters and 2 summers in the program. Through a written assignment using a theoretical patient scenario, we attempted to capture this information. The student met our threshold for success. Upon examining why

10

some of the students were unsuccessful, we concluded that they are applying best evidence, clinical judgment, and patient preferences. There seemed to be some misinterpretation of the assignment, rather than a true lack of knowledge.

We saw that the students continued to integrate these attributes throughout the remaining time in the DPT program, as they all successfully passed a comprehensive practical exam. Unfortunately, not all students demonstrated them on the first attempt. The seven students who did not pass on the first attempt either performed an activity that was considered unsafe, was inefficient in the organization of their interaction with the patient and thus ran out of time, or did not recognize/incorporate important patient information into their patient management strategies. Upon reviewing their videotaped performance and discussing their areas of weakness, the students successfully passed the second attempt.

This was the first year that this examination was given which may explain some of the variance in pass rate. Both the faculty and students learned from this ‘first time’ activity and will hopefully be ‘better’ prepared to more accurately assess student performance next year.

Program strengths: A strength in this outcome assessment is the data was captured during two different years (second and third) in the program. It was also captured in two different formats, a written assignment and skills-based practical examination.

Areas for improvements: However, the expectations for both activities may not have been clearly shared with the students leading to some inaccuracy in the student’s work/performance. As it was the first year for the final comprehensive examination, there are process issues that need to be improved. The faculty also agreed that the grading rubric should be revised to better capture what we want to assess during the student’s performance.

PLANNED CURRICULAR OR PROGRAM IMPROVEMENTS FOR THIS YEAR BASED ON ASSESSMENT OF OUTCOME 1. The assignment in PT 733 will be revised to clarify the wording that explains what is expected of the student. 2. In PT 733, class discussions will revolve around real patient scenarios that the students will bring back from the physical therapy clinics

they are placed in that semester. These new discussions will hopefully better link the three cornerstones of physical therapy practice (measured in this outcome) with a greater understanding of how physical therapists’ think and use information.

3. The grading rubric for the final comprehensive practical exam will be revised to better capture what the faculty is trying to assess. 4. The expectations for student performance will be more clearly articulated to the students in preparation for the final comprehensive

practical examination. More avenues for discussion and practice with faculty feedback will be explored to see if it helps student performance.

11

Learning Outcome #5: Exhibit care, compassion and empathy in delivering physical therapy services sensitive

to individual, cultural and social differences

Is this outcome being reexamined? Yes No

ASSESSMENT METHOD #1

ASSESSMENT TOOL

The faculty chose to determine if the students could appropriately exhibit care, compassion and empathy in the delivery of physical therapy services and be sensitive to individual, cultural and social differences. These skills are the essence of how physical therapist practice should be carried out. These components, while essential in how physical therapists engage with their patients, pose some potential problems with regard to their evaluation. The faculty determined that these attributes should be assessed while the student interacted with patients. During the final clinical practicum at the end of the DPT program, students are assessed by clinical faculty/instructors (CI) on their performance. The tool that is used is the Clinical Performance Instrument (CPI-Web). The CPI-Web (see detailed description below) asks questions about and requires assessment of measures of Professional Practice, which include the characteristics being assessed in this outcome. The students were also asked to complete the CPI-Web to self-reflect on how they saw themselves meet the criteria identified on the tool. They were also required to keep a regular reflective log of their experiences in the clinic and to identify how they felt as they worked with their patients.

Clinical Performance Instrument (CPI-Web)

The CPI-Web is the primary student evaluation instrument used to quantify student performance in the clinical environment against entry-level expectations of a licensed physical therapist. This proprietary tool was developed by the American Physical Therapy Association. It underwent extensive psychometric analyses of content throughout its development. The majority of physical therapy academic programs in the United States and Canada use this tool to assess student outcomes. The CPI-Web contains eighteen (18) distinct evaluative criteria that cross the spectrum of behaviors and actions required of a physical therapist in clinical practice. Each person inputting data into a CPI-Web tool must first complete an on-line course and certification examination to confirm basic knowledge and competency using the instrument. Data entered into the CPI-Web is immediately accessible to the Program and is easily downloaded for analyses.

The CPI-Web is a summative evaluation instrument. Both the student and his or her clinical instructor (CI) input data into the CPI-Web at midterm and completion of each clinical practicum experience. Data include Likert rankings and narrative comments. The Likert scale anchors with “beginning” on the left, or low end of the scale, and projects to “beyond entry-level” on the upper scale. Entry-level performance, which is positioned just below “beyond-entry-level”, is the expected student outcome on each criterion.

12

This report uses direct measures of CI assessment of student clinical performance and indirect measures of student self-assessment of clinical performance. This data was extracted from documented student performance that occurred during final, fulltime clinical practicum experience in August 2013. These students graduated from MU following this clinical practicum. These data are measures of MU-DPT student entry-level performance.

Definition: Entry-Level Performance (CPI-Web)

Capable of functioning without guidance or clinical supervision managing patients with simple or complex conditions.

Consistently proficient and skilled in simple and complex tasks for skilled examinations, interventions, and clinical reasoning.

Consults with others and resolves unfamiliar or ambiguous situations.

Capable of maintaining 100% of a full-time physical therapist’s caseload in a cost effective manner

Definition: Advanced-intermediate Performance (CPI-Web)

Requires clinical supervision less than 25% of the time managing new patients or patients with complex conditions and is independent managing patients with simple conditions.

Consistent and proficient in simple tasks and requires only occasional cueing for skilled examinations, interventions, and clinical reasoning.

The student is capable of maintaining 75% of a full-time physical therapist’s caseload.

Limitations of the CPI-Web as an Assessment Tool Students are supervised throughout their clinical experiences by a clinical instructor (CI) who is not a core faculty member; and each student has a unique clinical instructor. Although the expectation is that the online training program for completing the tool enhances the tool’s reliability in making judgments about achieving entry-level performance, there is still great variability based on settings and CI philosophy. The definition of entry-level is complex and the number of concepts embedded in each of the 18 overarching criterion can be large. If a student is deemed lacking in any aspect of a criterion, he/she will be graded below entry-level performance on all aspects of it, which makes it difficult to tease out specific areas of weakness.

Program faculty continues to support a graduate outcome goal that states: 100% of MU-DPT graduates will be rated entry-level in each evaluative criterion in the CPI-Web upon completion of their final clinical practicum. This is controversial in the professional community because students do not have the “real” opportunity to practice “without supervision.”

13

CPI-Web Criteria Used for this Student Learning Outcome

Criterion (CPI-Web Reference #)

Definition

2 Demonstrates professional behavior in all situations.

4 Communicates in ways that are congruent with situational needs.

5 Adapts delivery of physical therapy services with consideration for patients’ differences, values, preferences, and needs.

ASSESSMENT ACTIVITY

Direct Outcome Measure #1 Clinical Performance Instrument (CPI-Web)

Performance Standard Data Collection Analysis

Advanced-intermediate

Entry-Level Beyond Entry-Level

Criterion #2: Demonstrates professional behavior in all situations

100% of students will be rated “entry-level” on criterion items 2, 4 and 5 at the completion of the third and final clinical practicum

Data was collected from the CPI-Web CI evaluation of student performance

2 26 4

Criterion #4: Communicates in ways that are congruent with situational needs

1 28 3

Criterion #5: Adapts delivery of physical services with consideration for patient’s differences, values, preferences and needs

1 28 3

INTERPRETATION

The results of Clinical Instructor’s evaluation of student performance via the CPI-Web show that by the culmination of the final clinical placement, immediately prior to the graduate entering the workforce, 100% of the students had achieved advanced intermediate level with

14

approximately 95% of students being at entry level or above for these skills. Individual comments from student Clinical Instructors also bear out that students are successfully achieving this outcome measure. (These comments are available on request.)

Indirect Outcome Measure #1 Clinical Performance Instrument (CPI-Web)

Performance Standard Data Collection Analysis

Advanced-intermediate

Entry-Level Beyond Entry-Level

Criterion #2: Demonstrates professional behavior in all situations

100% of students self-report performance on criteria 2, 4, and 5 at the completion of the third and final clinical practicum.

Data was collected from the CPI-Web self-evaluation of final performance

1 30 1

Criterion #4: Communicates in ways that are congruent with situational needs

3 29 0

Criterion #5: Adapts delivery of physical services with consideration for patient’s differences, values, preferences and needs

1 30 1

INTERPRETATION

The results of self-evaluation show that by the culmination of the final clinical placement, immediately prior to the graduate entering the workforce, 100% of the students believed they were at advanced intermediate level with all but 1 student (professional behaviors and consideration of differences, values, preferences, needs) and 3 students (communication) believing they are at or above entry-level. These results were comparable to the clinical instructors view of their performance.

15

ASSESSMENT METHOD #2

ASSESSMENT TOOL

Student Self-Evaluation During Clinical Practicum II Student Learning/Mission Trip

During the spring of the third year of the program, the DPT students have the option to participate in a two week service learning opportunity in Costa Rica. While in Costa Rica, they provide physical therapy services to diverse vulnerable poor populations. In order to help better understand the student’s experiences outside of direct skill acquisition, a self-evaluation tool was developed (see Appendix 3). This tool focuses on the professional behaviors the DPT faculty expect of the students.

Three different aspects of the tool were evaluated for this outcome.

1. #2: Communication - The ability to communicate effectively (i.e. verbal, non-verbal, reading, writing, and listening) for varied audiences and purposes.

2. #4: Interpersonal Skills – The ability to interact effectively with patients, families, colleagues, other health care professionals, and the community in a culturally aware manner.

3. #6: Professionalism – The ability to exhibit appropriate professional conduct and to represent the profession effectively while promoting the growth/development of the Physical Therapy profession.

Indirect Outcome Measure #2 Self-Evaluation Tool

Performance Standard Data Collection Analysis

Intermediate Entry-Level Post-Entry Level

#2. Communication #4. Interpersonal Skills #6. Professionalism

100% of students self-report performance of each professional behavior at Entry-Level

Student self-evaluations of Service Learning Experience in Costa Rica (n=23)

3 17 3

2 17 4

0 17 6

16

INTERPRETATION Out of the 23 students who participated in the service learning opportunity in Costa Rica, at least 87% indicated they were at or above entry-level for these behaviors. As this practicum is only their second clinical internship (they have a 12 week final clinical practicum still to come) it is not surprising that some of the students felt below entry-level in communication and interpersonal skills. This experience occurred in a country where Spanish is the primary language spoken and most of the students do not speak Spanish, making communication even more challenging. Upon reflection and in reality, entry-level should not be expected, at least in the categories of communication and interpersonal skills, at this stage of their development into a graduate practicing as a physical therapist. Thus, the faculty were pleased with the results but recognize the need to redesign the tool. Additional Comment Another example of the DPT students recognizing the importance of these behaviors can be seen in the portfolios that the student develops throughout the program. During the final clinical practicum, students are required to write weekly blog posts in their portfolio that are reflections of the types of health conditions that they treated that week and also how they felt as they worked with their patient populations. Through these posts during the summer of 2013, across a class of 32 students, there are examples of students who really understood the role that many physical therapists play as they empathized with patients in very difficult circumstances. One member of the graduating class made this note in her blog from week eight, “Once you gain your patient's trust, it’s crazy how they open up to you. Some stories are funny and very entertaining, while some breaks your heart. On another note, they also trust that we can cure them or end their misery - which unfortunately might not be the case all the time. But all through this clinical, I've learned to maintain a good positive composure with my patients even if they don't reciprocate the same way. Even though, I don't always feel 100% thorough in my skills, I have learned that my attitude while I am with the patient takes it a long way and gain me their trust. My CI mentioned to me that I can have the greatest manual skills and have the best exercise tool kit' but if I don't have a good rapport with the patient, all my efforts could go in vain.” We think that this simple statement, made three-quarters of the way through the final practicum clearly demonstrates the understanding that the student has gained of the importance of these softer skills.

EXTENT THIS LEARNING OUTCOME HAS BEEN ACHIEVED BY STUDENTS The measures used to evaluate this outcome indicate that the students are exhibiting care, compassion and empathy in the delivery of physical therapy services and are sensitive to individual, cultural and social differences. As these behaviors are a critical component of patient care it is important to help ensure the students are aware of them and practice in a manner that is humanistic and understanding. The challenge arises in determining how to measure them. For this year we used the clinical instructors’ evaluation of the student immediately prior to him or her graduating from the program and entering practice. The tool is used consistently throughout the nation but is dependent on each faculty member’s interpretation of what entry-level means (there are descriptors to help minimize subjectivity). Despite these potential shortcomings, it is a valuable tool in determining if the student is ready to leave the academic environment and become a licensed physical therapist. Overall

17

our students were deemed to be at entry-level or above. The few students who were identified as advanced-intermediate clearly demonstrated these behaviors; they just needed less than 25% assistance with complex patients. The interaction with a complex patient can be daunting as so many factors have to be taken into consideration; this complexity can overwhelm students as they don’t have experience to draw on to help them distinguish between necessary and irrelevant data. We know that with mentorship in their first job, students quickly emulate these professional behaviors when managing the complex patient. The self-evaluations clearly show that the students value these attributes and believe that they are demonstrating them. We believe that self-awareness regarding compassion and caring across all types of patient differences is just as important as someone explaining to them why they should embody them.

Program strengths: A strength in this outcome assessment is the data was captured during the final clinical internship after which the students graduate and enter practice. There are categories on the assessment tool (CPI-Web) that allows for the information to be evaluated. As it is based on an evaluation of the student interacting with patients, it captures what we are trying to measure.

Areas for improvements: The self-evaluation tool that was used to help capture these attributes was based on ‘entry-level’ practice although the students were only in their second (of three) clinical internships during which entry-level practice is not expected. Even though we would like these characteristics to be demonstrated 100% of the time with 100% of the patients, we know and it was shown that communication skills can be continually improved upon. We need to determine what are the best ways to measure these ‘skills’ so we can continue to help our students who believe they are weaker in them grow. Finally, we did not attempt to discriminate between individual, social, and cultural differences to determine if there is one component that is more challenging for the students.

PLANNED CURRICULAR OR PROGRAM IMPROVEMENTS FOR THIS YEAR BASED ON ASSESSMENT OF OUTCOME 1. The self-evaluation of students’ perceptions of their professional behaviors while in Costa Rica form will be revised. 2. The faculty will meet to discuss what additional tools could be used to measure this student learning outcome. 3. The faculty will meet to discuss if it is important to differentiate out individual, social, and cultural differences to determine if the

students are having more difficulty in one particular area. If so, then curricular changes could potentially occur to address the area of weakness.

18

Appendix 1

PT 733- 2013 30% of grade will focus on the ‘clinical facts’ (left side column) 40% interpretation and clinical reasoning associated with these facts (right side column) 15% of grade will focus on EBP summary topic #1 15% of grade will focus on EBP summary topic #2

FINAL PROJECT: APPLYING EVIDENCE AND CONTEXTUAL FACTORS TO MAKE INFORMED DECISIONS IN THE PRESENCE OF MULTIMORBIDITY

Due date: May 6, 2013 BUT, once you are assigned a specific case on Feb 18, you should be thinking about the details of this project during each class session! Start ‘filling in the blanks’ as we talk about related materials during class sessions; and most every class session during the second half of the semester will have content that can be applied to this final project.

Goals 1. Locate, interpret, and apply relevant evidence to make clinical recommendations about the management of complex patients 2. Develop strategies to modify ‘generic’ clinical guidelines for application to complex patients. 3. Reflect on class discussion and personal experience about the benefits and drawbacks of practice guidelines and summarized literature for managing

older adults with multimorbidity.

Overview of the assignment 1. A list of comorbid conditions as well as personal and environmental factors, are listed on a separate excel spreadsheet (posted on blackboard). These

factors add context to a patient’s primary diagnosis of either ‘heart failure’ or ‘lumbar stenosis.’ 2. In class on February 18, each group will craft 8 unique cases. Each case is a patient with the same primary condition (heart failure or lumbar spinal

stenosis) but substantively different from each other in terms of contextual and comorbid factors. Each patient should have 3-5 comorbid conditions; and at least 2 personal and 2 environmental factors listed on the spreadsheet.

3. Each student will take one of these cases and use it across the remainder of the semester as a vehicle to interpret the key elements of the initial patient management encounter emphasizing how these individual factors influence the recommendations you make about the management of this particular patient. Your reflections on the science, clinical expertise, and patient values and motivations should be evident in the explanations.

4. Use the case examples at the end of the AGS article1 as a general guide to content organization. Note: The AGS article is organized primarily for

physicians. Thus, the specific content of each section will be geared to the needs of the physician; and yours should be geared to the needs of the PT. 5. In addition, you will take two topics of your choice applicable to this case and provide a short evidence-based and clinician friendly summary of what

you believe the PT should know about the topic. Each summary will be concise, evidence-based, and easy to ‘digest’ with a quick 2-minute read. One topic should focus on a personal factor of the patient and the other should focus on either an environmental factor or a comorbid condition. Think about it as a brief but authoritative summary available to you when at clinic for a quick refresher on the topic.

19

Recommendation: Talk with older adults, caregivers, PTs in the community, other health care providers, and community advocates. In their experience, what are the key values, perceptions, and motivators of patients and families that the PT should explore? What are the biggest barriers and facilitators to obtaining optimal functional status? What should the PT have their antenna ‘up’ for (What words of wisdom would they give you?)? What do they recommend you do to overcome these barriers and best utilize the facilitators. Reflect on these insights and include examples of how their insights contributed to your choices in your assignment.

1. American Geriatrics Society Expert Panel on the Care of Older Adults with Multimorbidity. Guiding principles for the care of older adults with multimorbidity: an approach for clinicians. JAGS. 2012;60(10):E1–E25.

WRITTEN FORMAT FOR PATIENT SUMMARY The questions are taken primarily from the AGS clinical Guideline article (2012). Figure one and the two case examples at the back of the article should guide the content. Your assignment differs in that the clinical ‘facts’ and the explanation / interpretation of the facts are divided into 2 columns. Format:

in the left hand column, summarize ‘medical record’ information and or key information gathered from the patient (as relevant to the section) ;

in the right hand column, explain and defend why the information you provided is important. How does it contribute to the practice decisions in this situation? Focus particularly on factors that require deviation from a ‘generic’ guideline. What was your clinical reasoning? Reflections on what surprised you or new insights gained.

Clinical ‘Facts’ Interpretation of importance of key ‘facts’ and reflections on them

Current concerns and objectives for the visit Include brief description of pt, primary medical diagnosis, his or her concerns/problems, and patient’s perception of the major objective of the visit. Identify the type of clinic also (outpt, SNF, etc)

Review of current medical conditions

List of current medical conditions

Any insights into how these may interact with each other and or impact PT assessment and plan of care?

List of current interventions (medications + any others) used to manage the health conditions. What fitness and healthy lifestyle activities does the

pt engage in regularly?

How do you gather information about medications? Strategies to be sure you have info about all medications? Gathering info about health behaviors? What are potential issues to screen for as part of history/examination? (FYI: later section asks more details @ likely impact of the meds on function)

20

Review of systems summary ( basic PT screen with key functional impairments and limitations to be

further investigated; issues outside scope of PT; review of any medical data available to help delineate impact of the chronic conditions)

Note: What are you particularly alert for as potential problems given what you are seeing at this time?

Is the individual comfortable with, and adherent to, the clinical management plan? (for current interventions)

Note: What questions might you ask this pt to glean information about the pt’s health beliefs, adherence, understanding of their conditions, attitudes about health care that might guide treatment planning?

What evidence is available regarding the patient’s current interventions? a) Focus on evidence that suggests a medication could affect functional

ability or PT approaches; b) are the non-medication conservative management approaches

adequately supported in the literature? (exercise, diet, relaxation, etc)

What does this all ‘say’ to you? How might this all interact with the primary referring diagnosis (main medical reason for coming to PT)? Why is it important? What should you ‘do’ with the information?

What are the preferences of the Individual and his or her family (if family/caregiver role is relevant)

You will be ‘making this up’ in terms of providing a description of the pt’s preferences (as if you had interviewed the patient). Be thoughtful and reflective about what the preferences might be.

Note: Comment on why you thought these preferences were realistic (apply your knowledge of common concerns and resources of older adults combined with prognosis and family dynamics, etc)

What is the patient’s prognosis in terms of physical functioning? How did you come to this prognosis? How confident are you in it? Why?

What patient problems will the PT plan of care focus on and what treatment goals will you set to address these problems? (FYI: there is NOT a parallel section in the AGS case)

Given the complex nature of this patient, what factors were most influential in your decisions about the plan of care? Does anything vary substantively from the ‘generic’ guidelines for managing the primary condition? Why?

Communicate and discuss with patient/family the PT recommendations for the approach to management. State what you anticipate you would say to the patient and what you would write in their medical record.

How might psychosocial and communication factors of the pt/family influence strategies used to discuss prognosis and plan of care andeEngage in collaborative decision-making? Reflect on factors that influenced your decision to use this approach and describe the approach?

Describe the PT interventions you plan to deliver Include the frequency and number of visits, HEP, pt education efforts, progression, etc (FYI: there is no real parallel in the AGS article)

Reflect on and discuss why you chose these interventions. What influenced your decisions?

21

What modifications in the program may be needed? Adapting to negative response, no response, or lack of adherence?

How do you go about monitoring for these responses? (ie, what do you build into your plan to be able to pick up on these things?). What do you do about it?

Clinician summary Topic 1: _____________________________________________ (limited to 250 words excluding references) Clinician summary Topic 2: ________________________________________ (limited to 250 words excluding references)

FACTORS THAT WILL CONTRIBUTE TO THE GRADING OF THE PROJECT 1. Reflective and thoughtful use of clinical reasoning and evidence-based principles in adapting a patient management encounter to accommodate

individual patient differences and preferences 2. Effective and accurate use of evidence to support clinical decisions 3. Clear, concise, clinician friendly format and language

22

Appendix 2

FINAL COMPREHENSIVE PRACTICAL EXAM

General Topic Areas within Topic The student…

PASS/FAIL

PLANNING Given written case…

Recognizes Red Flags (contraindications, precautions)

Identifies initial hypotheses based on initial data? (areas of focus, key points)

Prioritizes subjective interview questions

EXAMINATION & EVALUATION Review of Systems

Accurately identifies all components of the Review of Systems Tests & Measures

Prioritizes and justifies “best” Tests & Measures

Accurately administers Tests & Measures

Accurately interprets PT Examination data

DIAGNOSIS/PROGNOSIS/DESIRED OUTCOME Based on differential discussion, accurately labels/identifies the principal diagnosis/patient’s problems

Presents a prognosis grounded by interpretation of data

Identifies the desired patient outcome using functional performance language

Selects an “optimal” outcome measure

INTERVENTION Prioritizes and justifies appropriate intervention(s)

Effectively carries out the intervention(s)

Re-assesses the patient after intervention(s)

Accurately instructs/educates patient (HEP, etc.) and confirms patient understanding

PLAN OF CARE/TREATMENT PROGRESSION Analyzes patient case information to determine effective plan of care

Justifies rationale for plan of care/treatment progression

DOCUMENTATION Concisely and completely documents a note for the PT session within the allotted time

Final Grade: PASS FAIL

Examiners: _________________________________________________________________________________________

Students must PASS all sections of the examination. SAFETY issues identified by faculty in ANY section result in mandatory failure / discontinuation of exam.

23

Appendix 3

Marymount University Clinical Practicum II-Service-Learning/Mission Experience

STUDENT SELF-EVALUATION Student:

Facility: Costa Rica Dates:

Clinical Instructor:

Signature:

Please check the predominant level at which you view yourself as performing along each of the professional behaviors listed in the table below. Use the attached Professional Behaviors

Student Self Assessment Guide to assist you in determining your current performance level.

Professional Behavior

Intermediate

Level Entry-Level Post-Entry Level

1. Critical Thinking - The ability to question logically; identify, generate and evaluate elements of logical argument; recognize and differentiate facts, appropriate or faulty inferences, and assumptions; and distinguish relevant from irrelevant information. The ability to appropriately utilize, analyze, and critically evaluate scientific evidence to develop a logical argument, and to identify and determine the impact of bias on the decision making process.

Comments:

2. Communication - The ability to communicate effectively (i.e. verbal, non-verbal, reading, writing, and listening) for varied audiences and purposes.

24

Professional Behavior

Intermediate

Level Entry-Level Post-Entry Level

Comments:

3. Problem Solving – The ability to recognize and define problems, analyze data, develop and implement solutions, and evaluate outcomes.

Comments:

4. Interpersonal Skills – The ability to interact effectively with patients, families, colleagues, other health care professionals, and the community in a culturally aware manner.

Comments:

5. Responsibility – The ability to be accountable for the outcomes of personal and professional actions and to follow through on commitments that encompass the profession within the scope of work, community and social responsibilities.

Comments:

6. Professionalism – The ability to exhibit appropriate professional conduct and to represent the profession effectively while promoting the growth/development of the Physical Therapy profession.

Comments:

25

Professional Behavior

Intermediate

Level Entry-Level Post-Entry Level

7. Use of Constructive Feedback – The ability to seek out and identify quality sources of feedback, reflect on and integrate the feedback, and provide meaningful feedback to others.

Comments:

8. Effective Use of Time and Resources – The ability to manage time and resources effectively to obtain the maximum possible benefit.

Comments:

9. Stress Management – The ability to identify sources of stress and to develop and implement effective coping behaviors; this applies to interactions for: self, patient/clients and their families, members of the health care team and in work/life scenarios.

Comments:

10. Commitment to Learning – The ability to self direct learning to include the identification of needs and sources of learning; and to continually seek and apply new knowledge, behaviors, and skills.

Comments: