Jackie Stapleton Sandra Keys University of Waterloo January 30, 2008.

Jackie StapletonSandra Keys

University of WaterlooJanuary 30, 2008

OverviewPart 1CMS and the online quiz – review of the literatureOn-line quiz options – UW ACE / Angel

Part 2 Features of good question designHands on exercise

2OLA Super Conference 2008: Creating On-line Quizzes

Advantages/Disadvantages Advantages : Easy (relatively) to create and distribute in CMS environment Feedback – instant, automatic, consistent, Digital format preferred by new generation of students/active

learners (Johnson, GM, 2006)

Disadvantages: Lack of quiz security Surface understanding assessment only Technical problems

Johnson, G. M., & Johnson, J. A. (2006). Learning style and preference for online learning support: Individual quizzes versus study groups. 18th Annual World Conference on Educational Multimedia, Hypermedia, and Telecommunications, Orlando, FL. 1861-1868.


What does the literature say?The next series of slides will highlight some interesting

case studies and outcomes of research studies in the library and education literature.


Assess beyond surface learningDr. Diana Quinn and Ian Reid, University of South Australia

“Quizzes can be developed in ways that they play less of an assessment role, and more of a teaching role. “

Provide an outline of several quiz designs which claim to:Support personal construction of understandingSimulate group feedbackPromote self assessment

Quinn, D and Reid, I. (2003). Using innovative online quizzes to assist learning,

Accessed on October 21, 2007 at http://ausweb.scu.edu.au/aw03/papers/quinn/paper.html.

.


CMS and other quiz software:RH Locklin and PR Howell, Pennsylvania State University Used Angel for practice and pre test quizzes, Questionmark

for end of unit quizzes

Open book vs closed book quizzes (different delivery options)Student satisfaction survey revealed:

80% of students preferred open book quizzes over closed book exams

Majority believed that they would obtain a higher grade with an open book format (this did not prove to be true)

90% of students claimed to read the feedback

Locklin, R. H., & Howell, P. R. (2005). The design, implementation and assessment of an on-line, open-book quizzing environment for an introductory materials science course. Materials Research Society Symposium Proceedings, 909, 23-41.


Innovative training opportunitiesKathryn Skhal and Catherine Thureson, University of Iowa Library Used CMS to create an online training program for library science

students who staff the reference desk. Included article discussion forum, online quizzes, ‘challenging

questions’ module

Outcomes: Social community for reference students who often work in isolation Students posted many more questions to the ‘challenging question’

forum. Librarians have become much more efficient in the use of the CMS and

able to promote their skills to faculty

Skhal, K. J., & Thureson, C. (2007). ICONic training: Use of a course management system to provide continual reference student education. Medical reference services quarterly, 26(2), 15-25.


Not cheating – group collaboration! Amanda Fales-Williams et al, Iowa State University College of

Veterinary MedicineOnline pre exam quiz in WebCT for veterinary anatomy courseQuestions were designed to be challenging, did not expect

student to be able to answer the question on their first attempt

Instructor noted an increased rate of identical answers Not cheating, but group collaboration

Fales-Williams, A., Kramer, T., Heer, R., & Danielson, J. (2005). A quiz becomes a multidirectional dialogue with web-based instructional tools for an anatomical pathology rotation. Journal of veterinary medical education, 32(1), 144-149.


Creating a quiz in CMSI consider it a three step process:

1. Think about the learning objectives of your instructional material and then the purpose of your quiz.

Assessment of learned material? Learning/teaching opportunity?

2. Create the questions Many different question types.

3. Decide on delivery options


Question types

Fill in the blank, multiple fill in the blankTrue/False

Drop down listOrderingMatching

• Multiple choice, multiple select


Other quiz design features• Section heading

• Images, links, other html features can appear in a question stem or answer


FeedbackAdvantages: instant, automaticGeneral feedback for an entire questionSpecific feedback for each individual answer option


Delivery options:Creating the questions is only the first step. There are many

other decisions to make before you can implement the quiz.

Access – Who? When?Question deliverySubmission options Quiz security Marking

Does your CMS offer different features?


Accessibility considerations:Results of test study with JAWS reader:

Technical issues: Multiple choice questions were the most successful,

both in clarity and technical issues Drop down boxes were not readable Fill in the blank (edit boxes) caused problems

moving to the next question

Quiz Design issues: Place fill in the blank at end of question Stem should be a complete question.


My experiences:What I have learned:

Spend more time on question design View quiz more as a learning tool Expect technical problems and student issues

Next steps:Study how quiz delivery affects various factors such as:

How many students attempt to complete quiz Student achievement Student satisfaction


Writing QuestionsNo matter what assessment tool is chosen, the key is to

write sound test items

Nothing new; Long-standing traditionPlethora of information on writing “good” multiple choice

questions

But do we do it right?


Types of QuestionsTrue/FalseMultiple ChoiceFill in the Blanks

General Advantages and Guidelines Type-Specific Advantages, Disadvantages, and GuidelinesExamples


General AdvantagesVersatilityScoring accuracy and economyReliabilityEfficiency


General GuidelinesTest significant material and specialized knowledge, not

trivial detailsFocus on a single idea in each questionBe precise and correct in your languageRewrite or use examples to discourage rote memorizationAvoid negatively worded statements and particularly

double negatives

Ignore any guideline if you have a good reason to do so!


True/False QuestionsAdvantages

Only two reasonable answers Especially useful for testing misconceptions Expressible in a few words

Disadvantages Subject to guessing Less discriminating Lead to verbatim memorization due to tendency to write trivial

items


True/False QuestionsGuidelines

Make use of popular misconceptions or beliefs Provide sufficient information to judge the truth If expressing a relationship, give the correct part first and vary

the second part Do not use qualifiers (always, never, usually, often, etc.) Keep items of approximately equal length throughout the test Use somewhat more false than true statements Randomize the sequence

The Dana Porter Library is sinking

e.g., According to ...


It never rains in California

True/False Questions

Problems Negative (not) Clarity (which article?) Vocabulary (reference list)

Solutions Write as a positive Specify which article or other clarification Ensure that the vocabulary is consistent with the lesson

Question: Kreb’s article does not have a reference list

New question: Kreb’s article, discussed in Lesson 3, has a reference list


Multiple Choice QuestionsAdvantages

Diagnosis Control of difficulty Reduction of guessing Freedom from response sets

Disadvantages Difficult and time-consuming to write Subject to clueing Tendency to write items requiring factual knowledge rather

than higher-level skills and understanding Performance can be influenced by unrelated factors


Multiple Choice QuestionsGeneral Guidelines

Must have a definite answer Don’t give answer away by including irrelevant clues Don’t use language which may be offensive Have items reviewed by other knowledgeable people


Multiple Choice QuestionsTwo parts

Item stem (the question) Item options (the responses)

Item Stem Guidelines Single, concise statement Write as a question or an incomplete statement Most of the reading should be in the stem Include in the stem any words that would be repeated in each

option


Multiple Choice Questions Item Options Guidelines

Keep the number of options consistent List options in logical order Ensure each option is mutually exclusive Keep length of options the same Keep options parallel in form and grammatically consistent

with the stem All distracters should be plausible and equally attractive Use distracters which are correct but do not answer the

question Use distracters obtained from short answer/completion

responses


Multiple Choice Questions

Problems Negative (not) Vocabulary (synonomous) Too many options(7) Improbable option(c) “all of the above”; “A and D”

Question:Which of these is not synonomous with “Krebs Cycle”?A.Tricarboxylic acid cycleB.Citric acid cycleC.Rinse cycleD.Trioxaloacetic acid cycleE.Citrate cycleF.All of the aboveG.A and D


Multiple Choice Questions

Solutions Write as a positive Change synonomous Eliminate extraneous

distracters Include “select all that

apply” statement

New question:The “Krebs cycle” is a common name for which of the following terms? (select all that apply)A.Tricarboxylic acid cycleB.Citric acid cycleC.Trioxaloacetic acid cycleD.Citrate cycle


Fill in the BlanksAdvantages

Difficult to guess answers Relatively easy to construct Useful for definitions

Disadvantages Not as easy to score

Guidelines Avoid long or complex sentences Omit only one or two key words at the end of the sentence


Fill in the Blanks

Problems WAY too many blanks! More than one idea

The ____ cycle, also known as the ____ or the _____, is a series of _____ of central importance in all _____ that use _____ as part of _______.


Fill in the BlanksSolution

Rewrite question with just one or two blanks, preferably at the end

Make it two questions to cover both the name and a main idea

1. The citric acid cycle is also known as the tricarboxylic acid or the _____ cycle.

2. The citric acid cycle is a series of enzyme-catalysed chemical reactions in all living cells that use _____ as part of cellular respiration.


Hands on exerciseEach table has been given a sample quiz question. Please review as a group and make any changes which you

think will improve the question.

Things to consider:Is there a better way to phrase the question? Is library

jargon an issue?Does the question follow the suggested format for that

question type?


Sample quiz question #1:What activities are allowed in the library? Please check all

that apply. a) Silent studyb) Low conversation at group tablesc) Birthday party with hats and whistlesd) Eating your lunch from the cafeteria

Answer: a and b

Note: Is humour appropriate/inappropriate in this question/in any question?


Sample quiz question #2The boolean operator ______ should be used to combine two

concepts in a search box. Answer: AND


In conclusion

Whether you choose online or in print, make your quiz a good one!

Cyrus Wu, UW co op student, 2008


Jackie Stapleton Sandra Keys University of Waterloo January 30, 2008.

Documents

Transcript of Jackie Stapleton Sandra Keys University of Waterloo January 30, 2008.