©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer...
-
Upload
ernest-lindsey -
Category
Documents
-
view
214 -
download
0
Transcript of ©2003-2005 Subjective Metrics, Inc. 05.11.20051 waypoint online assessment and structured peer...
1©2003-2005 Subjective Metrics, Inc.05.11.2005
waypointonline assessment and structured peer review
COAS: Drexel University
Andrew J. McCannvisiting professor of english, drexel university
founder and president, subjective metrics, inc.www.gowaypoint.com
215.713.9393
2©2003-2005 Subjective Metrics, Inc.05.11.2005
Agenda
• Background
• Introduce waypoint
– Brief overview
• Drexel applications
• Live Demonstration
– Evaluate
– Manage
– Libraries
3©2003-2005 Subjective Metrics, Inc.05.11.2005
Problem: Feedback
• Handwritten and/or manually typed
• “Lost”
• Little accountability
• Consistency issues
• No data
• Needed: not AI – but a technological tool
4©2003-2005 Subjective Metrics, Inc.05.11.2005
Waypoint: Evaluation Tool
• Encourages pre-written feedback:
clearer, more detailed explanations
of key concepts
• Facilitates sharing of assessments
amongst instructors
• Quantifies evaluations by skill
• Archives all feedback
Web-based:
• cross-platform
• backed up
• zero maintenance
5©2003-2005 Subjective Metrics, Inc.05.11.2005
Waypoint Process
6©2003-2005 Subjective Metrics, Inc.05.11.2005
Develop Assessment
7©2003-2005 Subjective Metrics, Inc.05.11.2005
Develop Assessment
8©2003-2005 Subjective Metrics, Inc.05.11.2005
Receive Paper or Exam(Hard copy shown here)
9©2003-2005 Subjective Metrics, Inc.05.11.2005
Receive Paper or Exam(Hard copy shown here)
10©2003-2005 Subjective Metrics, Inc.05.11.2005
Evaluate Against Skills
11©2003-2005 Subjective Metrics, Inc.05.11.2005
Evaluate Against Skills
12©2003-2005 Subjective Metrics, Inc.05.11.2005
Respond to Student
13©2003-2005 Subjective Metrics, Inc.05.11.2005
Respond to Student
14©2003-2005 Subjective Metrics, Inc.05.11.2005
Snapshot Analysis
15©2003-2005 Subjective Metrics, Inc.05.11.2005
Snapshot Analysis
16©2003-2005 Subjective Metrics, Inc.05.11.2005
Longitudinal Cohort Analysis
Organizational Behavior: Research Competency1 to 4 scale; 110 students evaluated
0
10
20
30
40
50
60
70
1 2 3 4
# T
eam
s Feb '04Mar '04
Jun '04
17©2003-2005 Subjective Metrics, Inc.05.11.2005
Process Summary
Organizational Behavior: Research Competency1 to 4 scale; 110 students evaluated
0
10
20
30
40
50
60
70
1 2 3 4
# T
eam
s Feb '04Mar '04
Jun '04
18©2003-2005 Subjective Metrics, Inc.05.11.2005
Drexel Applications
• tDec Humanities (Dr. Valarie Arms)– Evaluation of all major writing assignments (600 students)
• Writing Center (Harriet Millan)– Customized assessments for WITs
• College of Engineering (Kevin Scoles & Adam Fontecchio)– Evaluation of lab reports with WITs and TAs
• College of Business (Frank Linnehan)– Structured response to student writing & accreditation data
generation
• Engineering Management (Mike Scheuerman)– Peer review
19©2003-2005 Subjective Metrics, Inc.05.11.2005
Demonstration
• www.gowaypoint.com• Evaluate
– “Final Report”• Peer Review
– “Final Report”• Manage
– Quantification– Sorting– Data analysis
20©2003-2005 Subjective Metrics, Inc.05.11.2005
Additional Features
• Self-assessment
• Collaborative assessment
– Multiple instructors can contribute to evaluation
21©2003-2005 Subjective Metrics, Inc.05.11.2005
Questions and Discussion
22©2003-2005 Subjective Metrics, Inc.05.11.2005
Testimonials
• “I've been doing this for twenty years and have my own system of grading and commentary…and found little need to improve. And then Waypoint came along. Somehow--and I really can't even explain it--but my grading time has been cut down by more than half and my students are actually thanking me for the in-depth commentary.”– Professor Ken Bingham
• “Usually I have trouble criticizing a peer’s paper if I’m not given certain criteria to judge. During this peer review, I was actually focused and excited about judging a peer’s paper.”– Erin Williams, COE 2008
• “If this peer review program was available in high school, I would have probably done a lot better in English.” – Ed Itaas, COE 2008
23©2003-2005 Subjective Metrics, Inc.05.11.2005
Testimonials
• “An innovative teacher-friendly, student-friendly, efficient approach to grading writing—the most creative and time-saving method for evaluating writing I have ever found or used.”
– Gayle, a high school English teacher with 35 years’ experience
• “I don’t know how I ran a writing program for three years without it.”
– Harriet Millan, Director of the University Writing Program
24©2003-2005 Subjective Metrics, Inc.05.11.2005
Quantitative Data
• Anonymous survey of Peer Review process
– 114 freshman engineers: 51 use waypoint, 63 the ‘old’ method
1-5 Scale (1 – strongly disagree, 5 – strongly agree)
wp (N=51)
Mean
‘old’ (N=63)
Mean
P value
(t-test, 95% CL)
1. I received clear and helpful criticism of my draft.
3.90 3.52 p<0.025
2. Evaluating other students’ papers helped me better understand the assignment and the play.
3.86 3.48 p<0.02
3. I made significant changes to my first draft based on peer review feedback.
3.81 3.25 p<0.005
4. I made significant changes to my first draft independent of peer review feedback.
3.49 3.43 p=0.38