All Capstone Projects Student Capstone Projects Fall 2015 ...
Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams
Transcript of Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams
![Page 1: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/1.jpg)
1ExamSoft Assessment Conference | #ASSESS2015
ExamSoft Assessment Conference
#ASSESS2015
![Page 2: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/2.jpg)
2
Restructuring a capstone course based on longitudinal assessment of module exams
Karen L. Kier, Ph.D., M.Sc., R.Ph., BCPS, BCACPProfessor of Clinical Pharmacy
Director of AssessmentRaabe College of PharmacyOhio Northern University
![Page 3: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/3.jpg)
3
ONU and Capstone
• Ohio Northern University – Small private university– College of Pharmacy has 1000 of the 3300 students on
campus– Pharmacy is a direct entry 0-6 program
• Capstone course– Designed in 1998 as part of the transition from a BSPh to a
PharmD– Course was designed to prepare students for 1 year of
rotations (APPE)– Pretest/Posttest design for one measure of material retention
![Page 4: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/4.jpg)
4
ONU Curricular Design
• 2 ½ year pre-pharmacy curriculum with a Profession of Pharmacy sequence each semester
• Spring semester 3rd year module introduction• 4th and 5th year practice modules• Modules designed as a team-based approach with
module coordinator• Integrated materials in module including medicinal
chemistry, pharmacology, pharmacokinetics, pharmacogenetics, and therapeutics
![Page 5: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/5.jpg)
5
Module Assessment
• Exams• OSCE– Rubric assessment– Skill check-off
• Skill-based assessments– Rubric assessment– Skill check-off
• Case-based assignments– Rubric assessment
![Page 6: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/6.jpg)
6
ExamSoft
• ExamSoft instituted in spring semester 2013 for the capstone course – Pretest/Posttest on ExamSoft – Categories and Strength & Opportunities (S&O) report
utilized with exams– Pretest results used to determine course materials as in the
past• Entire 4th and 5th year required to use ExamSoft within
all modules in fall 2013– First cohort to have entire module curriculum on ExamSoft
![Page 7: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/7.jpg)
7
ExamSoft
• Category creation– OSU Medical School Modified Bloom’s taxonomy– Each exam categorized by Director of Assessment
for consistency– Categories include course, material subject,
writing professor, modified Bloom’s– Each question mapped to ACPE 2009 (now 2016),
CAPE 2013, ONU Ability Based Outcomes (ONU ABOs)
![Page 8: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/8.jpg)
8
Topic Discussion
• Category Selection?
![Page 9: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/9.jpg)
9
ExamSoft
• Strength and Opportunities Reports– Students trained to access reports– Rationale release is a primary incentive to look
• Rationales– Faculty development session on use of rationales
to promote a process of remediation– Students to not get exams back only rationales
![Page 10: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/10.jpg)
10
OSU Modified Bloom’s
• 4 areas of modified Bloom’s– Factual– Conceptual– Procedural– Metacognitive
• ONU Exams– 25% Factual– 45% Conceptual– 5% Procedural– 25% Metacognitive
![Page 11: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/11.jpg)
11
ONU Exam History
• Scantron technology used in the past– Item analysis run for every exam– Module teams review item analysis for adjustments– Review for potential questions that could be revised– Point biserial
• ExamSoft technology– Same methodology as scantron– Item analysis reviewed for each exam– Point biserial
![Page 12: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/12.jpg)
12
Pre-test/Posttest Design
• Various methods have been employed for designing the exams
• These designs were intended to modify course materials to improve deficiencies
• Methods included:– PSAP questions– Module coordinators selected questions at
random– Implementation of ExamSoft Exams
![Page 13: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/13.jpg)
13
Capstone
• Idea to use ExamSoft as a basis for pretest selection
• Had first complete cohort• Capstone coordinator and Director of
Assessment felt that ExamSoft could create a better evidence-based selection of questions
• Department chairs agreed• Held module coordinator meeting
![Page 14: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/14.jpg)
14
Module Coordinator Meeting
• Director of Assessment provided ExamSoft metrics from each of their modules
• Item analysis available for every exam in every module
• Category reports by modules showing student performance by subtopics within module
• Category reports included modified Bloom’s for level of questions
• Capstone coordinator gave each module point values• Module coordinators selected questions
![Page 15: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/15.jpg)
15
Module Coordinator Meeting
• What to do with results?• How do we remediate material?• How did this compare to past measures?• Decided to use a combination of Moodle and
ExamSoft to deliver materials and to track student performance
• Decided to use S&O report to require students to look at materials– Red areas were mandatory remediation– All students had access to all remediation materials
![Page 16: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/16.jpg)
16
Module Coordinators
• Each module created a remediation site in Moodle to allow students to review materials
• Quizzes were available to test the review materials
• Module coordinators used ExamSoft results to design course materials (dynamic process) for the current capstone course in spring semester
![Page 17: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/17.jpg)
17
Topic Discussion
• Selection of questions?• Categories?• Difficulty?• Weakness in subtopics versus missing one
question in an area?– Local anesthetics versus depression
![Page 18: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/18.jpg)
18
Pretest
• Creation of pretest• Students took in January 2015 prior to
Pharmacy Administration module, Special Populations module.
• Students knew results in January with S&O report available to them. Rationales were released.
• Categories were simplified to major topic areas.
![Page 19: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/19.jpg)
19
Pretest Results
• Class average 32% (compared to averages in the past of 55-63%)
• What does this mean? How to interpret results?
• Capstone coordinator and Director of Assessment went to discuss results with class in Feb 2015 and expectations for posttest
![Page 20: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/20.jpg)
20
Remediation
• 100% of the students completed all Moodle assignments and quizzes for S&O red areas
• 93.7% of the students completed all Moodle assignments and quizzes regardless of S&O area achievement
• All module coordinators dynamically modified their in class materials based on results of the pretest exam
![Page 21: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/21.jpg)
21
Topic Discussion
• Remediation topics? Methods?• How could ExamSoft analytics help in this
process?• What ExamSoft analytics could be used?
![Page 22: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/22.jpg)
22
Posttest
• Posttest average 58%– The high score was an 89%– Analytics of posttest followed the analytics for student
performance on exams throughout the last 2 years– Tracking cohort performance demonstrated strong
students continued to excel and the weaker students still struggled
• Is this good or yikes???• Students below one standard deviation of the
mean had to remediate the posttest.
![Page 23: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/23.jpg)
23
Curricular Summit
• COP had a curricular summit in May• Evaluating all aspects of the curriculum• Evidence from ExamSoft was used extensively
to think through our process• How to manage weak areas prior to a student
proceeding in curriculum?• Incentives for life-long learning
![Page 24: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/24.jpg)
24
Curricular Summit
• Longitudinal assessment made more readily available to students
• Assessments prior to students moving to next year
• Put more active learning into curriculum to reinforce material prior to capstone
![Page 25: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/25.jpg)
25
Topic Discussion
• Rationale Development• Helpful?• How do you make sure students are reviewing
rationales and learning from them?• Complement rationales with other content• Fair game questions for any module?? • Pull ExamSoft questions in to modules from
other modules?
![Page 26: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/26.jpg)
26
Application Topic Discussion
• How could this be used at your institution?• How could these analytics change what you
do now?• What can we do better?
![Page 27: Restructuring a Capstone Course Based on Longitudinal Assessment of Module Exams](https://reader034.fdocuments.in/reader034/viewer/2022042716/55c8b8b1bb61ebd4318b469a/html5/thumbnails/27.jpg)
27
Conclusion
• ONU continues to evolve with this process• Faculty perceptions• Student perceptions• Provided some great evidence for our faculty on
how to improve our testing and curriculum process
• Provided great curriculum dialogue• Rewards• Barriers