Creating Tests for Higher Order...
Transcript of Creating Tests for Higher Order...
Creating Tests for Higher Order Thinking*
Thinking Schools Academy
*Content informed by:Gareis, C. R., & Grant, L. W. (2008). Teacher-made assessments.: How to connect curriculum, instruction, and student
learning. Larchmont, New York: Eye on Education
Th
ink
ing
Sch
oo
ls A
cad
emy
Lesson Objectives:
1. The learner will (TLW) understand that tests must assess lower- and higher-order thinking
(DOK1 - 10% emphasis)
2. TLW use a Table of Specifications to create lesson objectives, identify thinking level for each objective, and determine the lesson emphasis for the objective
(DOK 3 - 50% emphasis)
3. TLW understand guidelines for writing test questions (DOK 2 - 30% emphasis)
4. TLW will be introduced to the use of rubrics
(DOK1 - 10% emphasis)
Th
ink
ing
Sch
oo
ls A
cad
emy
Th
ink
ing
Sch
oo
ls A
cad
emy
Table of Specifications for this Lesson Thinking (Rigor) Level
Lesson Objectives: DOK1Recall
DOK2Application
DOK3Strategic Thinking
1. The learner will (TLW) understand that tests must assess lower- and higher-order thinking. 10% emphasis
High
2. TLW use a Table of Specifications to create lesson objectives, identify thinking level for each objective, and determine the lesson emphasis for the objective.
50% emphasis
High High
3. TLW understand guidelines for writing test questions.
30% emphasis
High Medium
4. TLW will be introduced to the use of rubrics.
10% emphasis
High
Lesson Objective #1
The learner will (TLW) understand that tests must assess lower- and higher-order thinking
(DOK1 - 10% emphasis)
Th
ink
ing
Sch
oo
ls A
cad
emy
4
The Planning Teaching & Learning Assessment Cycle
Planning
Establish What to Teach; Plan the Test; Create the Lessons
Teaching & Learning
Teach the Lessons while Checking for Learning
(Formative Assessment)
AssessmentAdminister the Test
(Summative Assessment)
Th
ink
ing
Sch
oo
ls A
cad
emy
Test Student Thinking…
• To be a Thinking School we must test a wide range of thinking beyond the ability to memorize a correct answer.
• A good test assesses both lower- and higher-order thinking.
Th
ink
ing
Sch
oo
ls A
cad
emy
Test Both Lower- and Higher-Order Thinking
Th
ink
ing
Sch
oo
ls A
cad
emy
DOK3
Strategic
Thinking
DOK4
Create
DOK2
Application
DOK1
Recall LowerOrder
HigherOrder
Lesson Objective #2
TLW use a Table of Specifications to create lesson objectives, identify thinking level for each objective, and determine the lesson emphasis for the objective (DOK 3 - 50% emphasis)
(TLW means “The Learner Will”)
Th
ink
ing
Sch
oo
ls A
cad
emy
Planning Teaching and Assessment
• Before teaching, decide the following:
• What objectives will be taught and how much of the unit will each objective require (10%, 25%, 50%?)
• What will be the emphasis of each objective in the lesson? (high, medium, low)
• At what DOK level will the objective be taught? (DOK 1, 2, 3, 4)
• How will objectives be assessed? Th
ink
ing
Sch
oo
ls A
cad
emy
USING A TABLE OF SPECIFICATIONSA table of specifications is a tool for planning the lesson and the test by:
1. Creating Lesson objectives
2. Identifying thinking (DOK or rigor) levels for each objective
3. Planning the test
Th
ink
ing
Sch
oo
ls A
cad
emy
The Table of Specifications is a planning tool which links lesson objectives, rigor, and testing
Th
ink
ing
Sch
oo
ls A
cad
emy
Assessment
Rigor
Lesson Objectives
Table of Specifications
• A planning tool
• Addresses three questions:
• What is the content to be taught?
• At what thinking level will the content be taught?
• What will be the emphasis of the content?
Th
ink
ing
Sch
oo
ls A
cad
emy
Table of Specifications
Th
ink
ing
Sch
oo
ls A
cad
emy
Content
Thinking Level by
DOK (Emphasis)
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
15
Table of Specifications
Th
ink
ing
Sch
oo
ls A
cad
emy
Content
Thinking Level by DOK and
Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
1. Human archeological history (50% of test)
Describe hunter-gatherer societies
Identify locations
Identify tools
High
Emphasis
High
Emphasis
High
Emphasis
16
Table of Specifications
Th
ink
ing
Sch
oo
ls A
cad
emy
Content
Thinking Level by DOK and
Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
1. Human archeological history (50% of test)
Describe hunter-gatherer societies
Identify locations
Identify tools
High
Emphasis
High
Emphasis
High
Emphasis
2. Describe adaptations to climate changes
(25% of test)
Medium
Emphasis
Medium
Emphasis
Low
Emphasis
17
Table of Specifications
Th
ink
ing
Sch
oo
ls A
cad
emy
Content
Thinking Level by DOK and
Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
1. Human archeological history (50% of test)
Describe hunter-gatherer societies
Identify locations
Identify tools
High
Emphasis
High
Emphasis
High
Emphasis
2. Describe adaptations to climate changes
(25% of test)
Low
Emphasis
Medium
Emphasis
High
Emphasis
3. Describe eventual domestication of plants
and animals (25% of test)
Medium
Emphasis
Medium
Emphasis
Low
Emphasis
18
Activity:
• For this activity you will design your own Table of Specification for a lesson that you teach
Th
ink
ing
Sch
oo
ls A
cad
emy
19
Specifying your lessonDirections: Step 1 - for a lesson you teach, write 3 content objectives. Step 2: When thinking of the entire lesson, what percentage of lesson emphasis will each objective take? (all objectives should sum to 100%.)Step 3: for each DOK or rigor level, circle the emphasis for each objective.
Th
ink
ing
Sch
oo
ls A
cad
emy
20
Content
Objectives(The total of all content objectives should equal 100%)
Thinking Level by DOK
And Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
Content Objective #1:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #2:
% of Lesson Emphasis: _______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #3:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Specifying your lessonDirections: Step 1 - for a lesson you teach, write 3 content objectives. Step 2: When thinking of the entire lesson, what percentage of lesson emphasis will each objective take? (all objectives should sum to 100%.)Step 3: for each DOK or rigor level, circle the emphasis for each objective.
Th
ink
ing
Sch
oo
ls A
cad
emy
21
Content
Objectives(The total of all content objectives should equal 100%)
Thinking Level by DOK
and Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
Content Objective #1:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #2:
% of Lesson Emphasis: _______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #3:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
1. Begin by breaking your lesson down into
objectives. Write up to 3 objectives for your lesson
Specifying your lessonDirections: Step 1 - for a lesson you teach, write 3 content objectives. Step 2: When thinking of the entire lesson, what percentage of lesson emphasis will each objective take? (all objectives should sum to 100%.)Step 3: for each DOK or rigor level, circle the emphasis for each objective.
Th
ink
ing
Sch
oo
ls A
cad
emy
22
Content
Objectives(The total of all content objectives should equal 100%)
Thinking Level by DOK
and Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
Content Objective #1:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #2:
% of Lesson Emphasis: _______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #3:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
2. What percentage of the total lesson will be emphasized
with this objective?
1. Begin by breaking your lesson down into
objectives. Write up to 3 objectives for your lesson
Specifying your lessonDirections: Step 1 - for a lesson you teach, write 3 content objectives. Step 2: When thinking of the entire lesson, what percentage of lesson emphasis will each objective take? (all objectives should sum to 100%.)Step 3: for each DOK or rigor level, circle the emphasis for each objective.
Th
ink
ing
Sch
oo
ls A
cad
emy
23
Content
Objectives(The total of all content objectives should equal 100%)
Thinking Level by DOK
and Emphasis
Recall
(DOK1)
Application
(DOK2)
Strategic
Thinking
(DOK3)
Content Objective #1:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #2:
% of Lesson Emphasis: _______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Content Objective #3:
% of Lesson Emphasis: ______
(Out of 100%)
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
Circle one:
Low/Medium/High
2. What percentage of the total lesson will be emphasized
with this objective?
1. Begin by breaking your lesson down into
objectives. Write up to 3 objectives for your lesson
3. For each objective, indicate
the emphasis as low, medium, or high by
DOK
Lesson Objective #3
• TLW understand guidelines for writing test questions (DOK 2 - 30% emphasis)
Th
ink
ing
Sch
oo
ls A
cad
emy
24
Making a Test: Matching DOK to type of Test Question
Th
ink
ing
Sch
oo
ls A
cad
emy
Question Types
Rigor Level
Multiple
Choice
True
False
Fill in the
Blank Matching
Short
Answer Essay
DOK1 Recall
DOK2 Application
DOK3 Strategic
Thinking
Writing Test Items
Th
ink
ing
Sch
oo
ls A
cad
emy
True/False Rules: (typically DOK1 questions)
#1. Only one idea should be in a True/False statement.
#2. Make the statement absolutely true or absolutely false.
#3. Avoid absolute qualifiers like always and never.
#4. Avoid using negatives in a True-False statement.
#5. Always avoid the use of opinion in statement items.
Matching Rules: (typically DOK1 questions)
#1. Use homogeneous content in a matching set.
#2. Place item to be matched on the right with the longer
responses on the left.
#3: Keep the matching set short.
#4: Use an uneven number of items to match or allow responses to be
used more than once.
#5. Order items in a logical manner.
Writing Test Items - Example
Th
ink
ing
Sch
oo
ls A
cad
emy
Which depth-of-knowledge level is application?
A. DOK1 (distractor item)
B. DOK2 (correct response)
C. DOK3 (distractor item)
D. DOK4 (distractor item)
Writing Multiple Choice Questions(can be DOK1, 2 or 3, but not 4)
Th
ink
ing
Sch
oo
ls A
cad
emy
Question Stem Rules
#1. Be sure the problem is written clearly.
#2. Do not give away the answer in the question stem.
#3. Write the stem from a positive perspective.
#4. Qualifiers such as most likely, best, least, and most should be
emphasized.
Rules for Writing Choices
#5. Write answers that are parallel in grammar and length.
#6. The answer choices should be plausible
#7: Try to avoid using “None of the above” and “All of the above.”
#8. Put your answers in a logical order.
#9. Do not put clues to the answer in your response choices.
#10. Be certain that there is only one correct response.
Writing Higher Order Questions
• A goal of the Thinking Schools Academy is to teach and assess higher order thinking
• Higher order (DOK3) thinking can be assessed using multiple choice, fill-in-the-blank, and short-answer questions.
Th
ink
ing
Sch
oo
ls A
cad
emy
5 Principles for Writing DOK3 Multiple Choice Items
1. Match the content with a rigor or thinking level (DOK 1, 2, 3, 4)
2. Use novelty
3. Focus on content that is complex
4. Use an extended prompt
5. Provide stimulus material
Th
ink
ing
Sch
oo
ls A
cad
emy
Principle #1: Match the Content with Cognitive Level
Th
ink
ing
Sch
oo
ls A
cad
emy
Learning Objective: Analyze the factors influencing where hunter-gatherer societies locate. (DOK3 learning objective)
Poor Question Item:Which factor below would NOT be used to determine where a hunter-gatherer community might locate?
A. Availability of waterB. Proliferation of edible plantsC. Amount of retail shoppingD. Acceptable climate
Better Question:A group of hunter-gatherer leaders were considering whether to relocate their community. One leader pointed out that the wild animals they hunted for food had significantly decreased in number while another suggested that a different location with better natural beauty would make life more enjoyable.
Which of the following most likely had the greatest influence on their decision to relocate the community?
A. Better natural beautyB. Cleaner airC. A decrease in wild animalsD. A safer community
Principle #2: Introduce Novelty
• To test DOK1 knowledge, use straight forward questions that will be familiar to students.
• To assess higher order thinking, use novelty that will be less familiar to students.
• The previous slide is an example where a situation was created (the meeting of hunter-gatherer leaders) to introduce novelty
• The novelty requires students to analyze or evaluate the situation based on their previous learning
Th
ink
ing
Sch
oo
ls A
cad
emy
Principle #3: Focus on Complex Content
• Simple content focuses on one dimension or just a few components.
• A list of dates is simple material even though they may be difficult to memorize.
• An explanation of events involving the dates could be complex.
• Simple content is assessed at DOK1 & 2 levels
• Complex content is assessed at DOK3 levels
Th
ink
ing
Sch
oo
ls A
cad
emy
Example of a DOK1 and DOK3 Multiple Choice Question for the Same Lesson Objective
Th
ink
ing
Sch
oo
ls A
cad
emy
Lesson Objective: The student will understand the variables that contribute to a sustainable hunter-gatherer community.
DOK1 Question:Which of the following IS NOT a variable contributing to a successful hunter-gatherer community?A. WaterB. Edible plantsC. Hospitable climateD. Well-paved roads
DOK3 Question:The typical spring and summer has resulted in much less rainfall than normal. This will most likely result in:A. A hot summerB. Fewer healthy plants and
vegetables to harvestC. A large harvest of edible plants
and fruitD. Trees which have grown faster
than normal
Activity:
• Discuss with your “elbow partner” what distinguishes DOK1 from DOK2 from DOK3 test questions?
• Make a bullet list of your conclusions
Th
ink
ing
Sch
oo
ls A
cad
emy
Lesson Objective #4
• TLW will be introduced to the use of rubrics
(DOK1 - 10% emphasis)
Th
ink
ing
Sch
oo
ls A
cad
emy
Developing a Rubric to Assess Complex Tasks
• A rubric lists the scoring criteria for a task
• Rubrics are used to decrease subjectivity in grading a complex task
• Articulating essential criteria helps the teacher to make grading more objective
Th
ink
ing
Sch
oo
ls A
cad
emy
An Example of a Rubric to Grade Bicycle Riding
Th
ink
ing
Sch
oo
ls A
cad
emy
Indicator 1 2 3 4 Score
Balance Immediately falls off bike
Can maintainbalance for up
to 5 meters
Generallymaintains
balance
Balance is rarely lost
Steering No steering ability
Steers in a wobbly manner over very short
distances
Maintains good control in most
riding circumstances
Steering is lost only under
extreme circumstances
Pedaling Is not able to pedal
Can pedal softly for up to 5
meters
Can pedal with little thought; can
easily adjust pedaling speed
Expert at pedaling in a
variety of conditions
Control Cannot maintain control of
bicycle
Control is very limited
Can generally maintain control
of bicycle
Maintainscontrol in all
terrain conditions
Total Score:(Add the scores for balance, steering, pedaling, and control)
Applications for Rubrics
• Scoring a writing assignment
• Grading a performance
• Scoring a research report
• Evaluating a speech
• Evaluating the creation of something new such as a piece of art, a product improvement or innovation T
hin
kin
g Sc
ho
ols
Aca
dem
y
Activity: Design Your Own Rubric
Th
ink
ing
Sch
oo
ls A
cad
emy
Indicator 1 2 3 4 Score
Total Score:(Add the scores for each row)
Summary
• Assessments must match the curriculum
• Curriculum must include Higher Order Thinking (DOK3)
• Decide the percentage of curriculum dedicated to DOK1, DOK2, and DOK3
• Decide the emphasis for each curriculum objective (low, medium, high)
• Use the Table of Specifications as a planning tool
Th
ink
ing
Sch
oo
ls A
cad
emy