Tests & Quizzes (aka SAMigo) Keli Amann, User Experience Specialist Christine Doherty, User Support...

Tests & Quizzes (aka SAMigo)

Keli Amann, User Experience SpecialistChristine Doherty, User Support Specialist

Stanford UniversityJune 17, 2010

• Feature Overview• 2.7 Highlights• 2.8 Roadmap• Assessment in 3akai• Q&A

Agenda

211th Sakai Conference - June 15-17, 2010

What are SAMigo’s Features:


• Assessment Authoring• Assessment Publishing• Assessment-Taking Management• Assessment Grading• Question Pools Management• Assessment-Type Management

• Online test• Self-study questions• Homework, problem-sets• Paper, code, or project submission• Language drills• Quick knowledge probes• Surveys• Others???

Pedagogical Flexibility


2.7 Highlights

11th Sakai Conference - June 15-17, 2010 6

Main Page


Assessments

Pending Assessments

Published Assignments

Active

Inactive

New AssessmentsCreate assessment ORImport assessment


2.62.6

2.72.7

Main Page: Create

Main Page, Pending


Assessments

Pending Assessments


Active

Inactive


Main Page: Pending


2.62.6

2.72.7

Main Page: Published


Assessments

Pending Assessments


Active

Inactive


Main Page: Published


2.62.6

2.72.7

*OOTB, Edit (not pictured) only appears if students haven’t started

Main Page: Columns


2.62.6

2.72.7

Edit Assessment Page


Assessments

Pending Assessments


Active

Inactive


Edit Assessment

Assessment Settings

Publish

Preview

Auto-scored• Multiple Choice (single and multiple correct)• Survey (multiple choice, different rating scales available)• Fill in the Blank• Numeric Response• Matching• True/False

Manually scored• Short Answer/Essay• Audio Recording• File Upload

About Question Types



2.62.6

2.72.7

Edit Assessment Page: Add Part


2.62.6

2.72.7

Edit Assessment Page: Preview, Publish

About Settings• Configures delivery of assessments (how they appear to

students)

• Options include: • display/organization• late handling• number of submissions • feedback/grading options • high security and time limit for high stakes testing

• Most settings are editable after publishing, but it is still important for instructors to review carefully


Improvements to Preview

• Fill in the Blank and Numeric Response text fields and Matching selection menus made functional

• Timer (inaccurate in Preview mode) replaced with graphic/explanation

• Submit for Grading and Exit confirmation pages included


Preview screen


Partial Credit for Multiple Choice Questions

Allows instructor to enter percentages of credit to give student for selections other than correct answer

• Correct answer is set to 100% automatically; can not be edited

• Reset links allow instructor to go back to default grading logic or reset percentage values to 0

• Validation requires value to be between 0 and 100 (inclusive); no negative numbers allowed

• Applies to Single Correct Answer only

• Can not be used in conjunction with Negative Marking

• Feature needs to be turned on in Sakai Properties; not OOTB

• Contributed by Rice University


Print Assessment (with or without feedback)

• Print action appears in Actions menu on main authoring page and as link in Edit Assessment page

• Available for both pending and published assessments

• Options include:• Showing short answer key• Showing feedback (question-level and selection-level)• Showing intro and part titles• Changing font size of document• Printing as HTML or downloading a PDF

• Feature needs to be turned on in Sakai Properties; not OOTB • Contributed by Universidad Politecnica de Valencia


Email Notification• Option to send site members (or group members) email notification upon

publishing or republishing assessment

• Publishing without notification is set as the default to prevent accidental spamming

• Important settings are pre-populated in notification:• Assessment title• Available date, due date• Group released to (entire site, section(s)/group(s) or anonymous users) • Assessment URL• Time limit (if applicable)• Number of submissions allowed• Grading and feedback settings

• Option to include a personal message


Editable Parts in Published Assessments

• In 2.6, only questions and settings were editable in published assessments

• In 2.7, Parts are editable in the following ways:• Title• Information (longer text field)• Attachments (add or remove)

• Operations that are more likely to create discrepancies in scoring are not editable:• Random Draw options• Question Ordering (when not using Random Draw)


File Upload in Question Feedback

Option to upload files as feedback for each question; multiple files per question allowed

• Available on Questions page or student’s full submission page (under Scores)

• Update button must be clicked in scoring page after file uploaded with attachment helper

• Useful for returning commented papers to students

• Note: File size now displayed next to all file uploads (student submissions and files uploaded by instructor)


Improvements for No Submission Records

• Applies to Total Scores page when student identities can be seen by graders

• Comments: • Disabled for students with no submissions• ‘What’s This’ link to clarify purpose of Comments field

• Adjustment Scores:• If adjustment score is entered for student with no submission, the score is

now sent to the Gradebook and to the Export file• Useful for handling offline submissions (e.g., students experiencing technical

difficulties)


Improvements in Delivery Mode

• Changes in button labels• “Save and Continue” is now Next• “Save for Later” is now Exit

• Changes in button configuration• Navigation buttons to left, Save/Exit/Submit buttons to right• Submit for Grading is rightmost button on the last question page

• Auto-Save and Save (on demand)

• Improved button layout in audio recording applet



2.6 Button Configuration in Delivery Mode2.6 Button Configuration in Delivery Mode


2.7 Button Configuration in Delivery Mode2.7 Button Configuration in Delivery Mode

Auto-Save and Save (on demand)

• Auto-Save:• Saves student work without interruption (no screen refresh)• Runs at configurable interval (set in Sakai Properties)• Resets session timeout for inactivity• Useful for long assessments on one page or high stakes testing

• Save (on demand):• Allows students to save anytime while remaining on same page• Useful for answers that require significant time (e.g., essay questions)

• Contributed by Rutgers University


Improved Button Layout in Audio Recording Applet

• Record and Stop are two separate buttons now, replacing toggle • Reduces chances of students erasing recording by accident


Students View All Submissions/Scores

• In 2.6, only the submission with the recorded score (highest or last) displays in the student’s Submitted Assignments list

• In 2.7, the student’s Submitted Assignments list will show all submissions/scores by default

• Students can collapse the Submitted Assignments list to show only recorded scores

• Useful for self-study or practice exam scenarios; students can see feedback for multiple submissions

• Contributed by Universidad Politecnica de Valencia


Performance Improvements

• Performance has been improved in the following areas:

• Generation of Statistics for high number of submissions (~500)

• Loading of Questions page (under Scores) for File Upload and Audio Recording questions

• Downloading of student submission files (from File Upload questions)


2.8 Roadmap (independent)

64

• Respondus integration

• Record Average Score for multiple submissions*• Allow students to use Rich-Text Editor for Short Answer/Essay

question type**

• Remove 4000 character limit for student response in Short Answer/Essay question type**

• Allow instructors to enter zero score for unsubmitted assessments

• Allow instructors to show students scores only, without showing any other feedback (e.g., questions/responses)

• More to be discussed

* Based on contributions from Rice University** Based on contributions from Longsight

Assessment in Sakai 3.0?

65

User Research and Concept Development: Fall 09-Summer 2010

• No such thing as Tests & quizzes; functionality distributed

• All learning activities initiated from (syllabus schedule, to do, repository) in same basic way

• Activities stored in repository for sharing or reuse: find in “drawer” or repository

Work ongoing for

• instructor evaluation

• student views of syllabus, to do, grades

http://confluence.sakaiproject.org/display/UX/ Sakai+3.0+capabilities+for+learning+activities

Questions?


Contacts

Questions and feedback: [email protected]

Lydia Li, Stanford University Project Manager [email protected]

Homepage:http://confluence.sakaiproject.org/confluence/display/SAM


Appendix

Review of capabilities of Sakai 2.x


Organize & Publish Assessments


• Pending Assessments• Starting page for assessment authoring• Can be created using markup text (aka Quick Create)• Can be exported/imported in QTI XML format• Can create duplicates easily • When ready, published via Settings

• Published Assessments for student assessment-taking• When students respond, Scores action appears• Can edit published assessments and most settings • Can remove published assessments

Auto-scored• Multiple Choice (single and multiple correct)• Survey (multiple choice, different rating scales available)• Fill in the Blank• Numeric Response• Matching• True/False

Manually scored• Short Answer/Essay• Audio Recording• File Upload

Question Types


• At all levels of assessment authoring:• Assessment/Part Headers • Questions• Selections• Feedback

Rich Text & Attachments


Quick Create

• Efficiency created by reducing number of clicks, pages loading and saving

• Useful for transferring assessments created in other applications or LMS (e.g., MS Word, WebCT), no need to paste into different fields

• Supports Multiple Choice, Fill in the Blank, Short Answer/Essay, and True/False question types

• Rich text, attachments, and additional questions can be added later, through assessment authoring page


Import/Export

• Supports QTI v 1.2

• Export types: IMS QTI-compliant XML or IMS Content Packaging (for use with attachments)

• Useful for:• Copying assessment to different site• Archiving assessments offline


Settings• Configures delivery of assessments (how they appear to

students)

• Options include: • display/organization• late handling• number of submissions • feedback/grading options • high security and time limit for high stakes testing

• Most settings are editable after publishing, but it is still important for instructors to review carefully


Publishing• Release assessment to students

• Copy made from pending assessment – pending assessment can be modified without affecting published version

• Can set Available Date in future -- useful for instructors who want to do bulk of setup early

• Can retract any time, making assessment unavailable to all students

• Can remove published assessment

• Can edit published assessment


Taking an Assessment (Students)

• Take an Assessment section• List determined by available, due, and retract dates,

plus number of allowed submissions• Can submit when done or save a work in progress

• Submitted Assessments section• Provides submission confirmation with timestamp• Depending on settings, students can review scores,

feedback, grader’s comments, statistics


Grading

• Ability to grade all questions for one student or grade all responses to one question

• Manual score adjustments can be made to individual questions or entire assessment

• Comments can be made for individual questions or entire assessment

• Ability to send scores to Gradebook

• Ability to export scores/responses to Excel

• Ability to calculate statistics and histograms


Question Pools

• Organize questions into multiple subpools

• Move/copy/remove multiple questions at a time

• Author questions from within Pool Manager

• Import questions from IMS QTI-compliant XML file

• Randomly draw from pools for an assessment

• Question pools belong to an individual user, available across all user’s sites

• Ability to share question pools with other site members


Assessment Types

• Create assessment “types” for different disciplines, different uses

• Hide complexity by selecting which settings are visible to the instructor.

• Pre-set defaults by making setting choices for the type (e.g., Self-study always has immediate feedback)


Assessment Creation Workflow

July 2009 10th Sakai Conference - Boston, MA, U.S.A.

Tests & Quizzes (aka SAMigo) Keli Amann, User Experience Specialist Christine Doherty, User Support...

Documents

Transcript of Tests & Quizzes (aka SAMigo) Keli Amann, User Experience Specialist Christine Doherty, User Support...