Assessing Higher Order Thinking Skills
description
Transcript of Assessing Higher Order Thinking Skills
Assessing Higher Order Thinking Skills
Vickie Mohnacky April 19, 2011
Assessing Higher Order Thinking Skills
• To identify personal, social and academic student outcomes in the PreK-Grade 12 Programming Standards
• To align student outcomes with different types of assessments
Today’s Objectives
What do you want your students to learn as a result of gifted
education?
• http://www.twiddla.com/528716
What do I value the most? (As a teacher – as a human being)
• Following directions?• Timeliness?
What do we value most?
• Speed?• Accuracy?• Knowledge of content?
Test #1 Following Directions
LastName: _________________ 1stName:_____ __ Date: _________ Period: ___ _________________________________________________________________________________ INSTRUCTIONS Read very carefully. Read over and understand this entire paper before doing any of it. You may ask for clarifications. 1. Stand up from your chair and stand behind it. Stay that way until a teacher initials
this line. 2. Sit down on your chair, facing its desk top. 3. Count the number of students in the room and write the number here ____. 4. Put this paper on top of your head and leave it there until the teacher initials this line. 5. On the line below, write in the next two numbers in the series of numbers. 1, 3, 5, 7, 6. At the bottom of the paper, write and solve this math problem: 1/2 + 3/8 = ? 7. How many male humans in this room? ____. 8. In the blank space below question 6, draw a stick-figure of a person. 9. Add up the number of teachers plus the number of female students in this room. Write your answer here . 10. Sit on your chair with your legs to the side, and the right side of your body towards
the your desk’s top. Stay that way until a teacher initials this line. INSTRUCTIONS Do not do any of the numbered questions. Instead, fill in the blanks at the top of the paper (just below the title). Next, turn your paper over. Sit and watch the fun. If a student asks you why you are not doing it, just say “I don’t want to look foolish.”
What do we value most?Complex thinking
Originality Research Understanding systems Seeing connections Self Knowledge Perspective
Abstract Concepts Leadership Civic Responsibility Empathy
How will we use the information?
Teachers’ responses:
How will we use the information?
Grade?Class rank?Determine entrance to program?Embarrass the student?Punishment?Predict success?Determine coursework?Measure growth/progress in learning?Plan instruction?
Assessment Uses
• Determines what the student already knows;
• Determines readiness
Pre-assessment
• A process during learning• As the learning is forming
Formative Assessments
• An event after learning• Sums up learning
Benchmark – Summative Assessments
Not high stakes/Not for accountability.
Used to adjust instruction/ improve student learning.
Not for report card grades.
Examples:
• Teacher informal questioning
• Warm-ups
• Homework
• K-W-L Chart
Formative AssessmentAn assessment FOR learning. Occurs while learning is forming.
Not high stakes.
Examples
• End of chapter tests Acuity http://wvde.state.wv.us/oaa/acuity.php DIBELS Benchmark testing
An assessment OF learning – Interim points
Benchmark/Interim Assessment
Not high stakes.
Used to adjust overall curriculum, programming
More Examples:
• Attitude scales
• Interest Inventories
• Established rubrics for class product and performances
• Critical thinking tests/checklists (Cornell, Watson-Glaser, Test of Critical Thinking)
• AP/IB course taking and performance
An assessment OF learning – Interim points
Benchmark/Interim Assessment
High stakes.
Reporting and accountability
Used to evaluate the overall effects of programming
Examples
• State Standardized Test (WESTEST 2)
•NAEP (National Assessment of Educational Perf.)
•PISA (Program of International Student Assessment
• Act Explore/ACT/SAT
An assessment OF learning; Sums up; Occurs at set points
Summative Assessment
Difference in Formative and Summative Assessments?
How they are used
Level of student
involvement
Critical Consumers
Responsible Producers
What are the ultimate student outcomes?
Brian Housand
What is the ultimate outcome?
Solve
Houston, we have a problem
• http://www.youtube.com/watch?v=pbBS8sHrDgA&NR=1
Rick Wormeli 2006Fair Isn’t Always Equal
The point is that we have to be clear in our
objectives before we can differentiate instruction and properly assess our students’ attainment of
those objectives.”
Look at Your Data
• What does it measure?• Will it give you the information you need?
2010 Gifted Education Programming Standards
Students identified with gifts and talents demonstrate important learning progress as a result of programming and services (Assessment Criterion 2.5)
Standards Focus on Student Outcomes
Learning and Development Student Outcomes
• Demonstrate self-knowledge of interests, strengths, identities and needs.
• Possess developmentally appropriate understanding of how they learn and grow and recognize the influences of their beliefs, traditions and values on their learning and behavior
• Demonstrate an understanding and respect for similarities and differences between themselves and their peer group and others in the general population.
1.1 – 1.3 Self-Understanding
Learning and Development Student Outcomes
• Access resources from community to support cognitive and affective needs.
• Receive support from family and communities who understand similarities and differences with respect to the development and characteristics of advanced and typical learners.
1.4 – 1.5 Awareness of Needs
Learning and Development Student Outcomes
• Benefit from meaningful and challenging learning activities.
• Recognize preferred approaches to learning and expand their repertoire.
• Identify future career goals that match their talents and abilities and resources needed to meet goals.
1.6 – 1.8 Cognitive and Affective Growth
Curriculum Planning and Instruction Student Outcomes
• Demonstrate advanced and complex learning as a result of using multiple, appropriate and ongoing assessments
• Demonstrate important learning progress as a result of programming and services.
2.4 -2.5 Assessment
Curriculum Planning and Instruction Student Outcomes
• Demonstrate growth commensurate with aptitude.
3.1 Curriculum Planning
Curriculum Planning and Instruction Student Outcomes
• Become more competent in multiple talent areas and across dimensions of learning.
• Develop their abilities in their domain of talent and/or area of interest.
3.2 – 3.3 Talent Development
Curriculum Planning and Instruction Student Outcomes
• Become independent investigators.• Develop knowledge and skills for living
and being productive in a multicultural, diverse and global society.
3.4 - 3.5 Instructional Strategies
Curriculum Planning and Instruction Student Outcomes
• Benefit from programming that provides a variety of high quality resources and materials.
3.6 Resources
Learning Environments Student Outcomes
• Demonstrate growth in personal competence and dispositions for academic and creative productivity (self-awareness, self-advocacy, self-efficacy, confidence, motivation, resilience, independence, curiosity, risk taking).
4.1 Personal Competence
Learning Environments Student Outcomes
• Develop social competence manifested in positive peer relationships and social interactions.
• Demonstrate personal and social responsibility and leadership skills.
4.2 – 4.3 Social Competence - Leadership
Learning Environments Student Outcomes
• Value their own and other’s language, heritage, and circumstance.
• Possess skills in communicating, teaming and collaborating with diverse individuals and across diverse groups.
• Use positive strategies to address social issues, including discrimination and stereotyping.
4.4 Cultural Competence
Learning Environments Student Outcomes
• Develop competence in interpersonal and technical communication skills.
• Demonstrate advanced oral and written skills, balanced biliteracy or multiliteracy, and creative expression.
• Display fluency and technologies that support effective communication.
4.5 Communication Competence
• Specify clearly and exactly what it is you want to assess
• Design questions and tasks that require students to demonstrate this skill or knowledge
Understanding By Design (UBD)
1. Identify desired results (what the student will know)
2. Determine acceptable evidence (how will the student demonstrate knowledge)
3. Plan instruction; Implement the design
“I began by seeing assessment as judging performance, then as informing teaching, and finally as informing learning.”
• Carol Ann Tomlinson
Types of Assessment
• Standard - • Selected Response (Multiple Choice)• Constructed Response (Short Answer)• Fill in the Blank• True/False• Performance-based or Product- • Rubrics• Rating scales• Checklists
High School Entrance Exam – 1928http://wvde.state.wv.us/1928/
Standardized – Large Scale
•Mass produced•Mass graded
Balanced Assessment System
•Content – Knowledge•Reasoning•Performance Skill•Product
Standard/Objective:
Type: Knowledge Reasoning Performance Skill Product
Learning TargetsWhat are the knowledge, reasoning, skill or product targets underpinning the standard/objective?
Knowledge Targets Reasoning Targets Performance Skill Targets
Product Targets
What knowledge would students need to master this skill?
What reasoning proficiencies (if any) would students need to master this skill?
What performance skills (if any) would students need to practice to master this skill?
What products (if any) would students need to practice creating to master this skill?
Example from real life:
Drive with skill.Type: Knowledge Reasoning Skill Product
Learning TargetsWhat are the knowledge, reasoning, skill or product targets underpinning the standard/objective?
Knowledge Targets Reasoning Targets Performance Skill Targets
Product Targets
•Know the law•Understand informal rules of the road•Understand what different parts of the car do•Read signs and understand what they mean•Understand what “creating a danger” means•Understand what “creating a hazard” means•Other?
•Analyze road conditions, vehicle performance, and other driver’s actions•Compare/contrast this information with knowledge and past experience•Synthesize information and evaluate options to make decisions on what to do next•Evaluate “Am I safe?” and synthesize information to take action if needed.
•Other?
•Driving actions such as: steering, shifting, parallel parking, looking, signaling, backing up, braking, accelerating, etc.•Fluidity/automaticity in performance driving actions.
•Other?
None
Since the ultimate type of target is a performance skill, there are no embedded product targets
Standard/Objective:
SS.O.06.05.04: Interpret quotes of famous Americans from various periods of historyType: Knowledge Reasoning Performance Skill Product
Learning TargetsWhat are the knowledge, reasoning, skill or product targets underpinning the standard/objective?
Knowledge Targets Reasoning Targets Performance Skill Targets
Product Targets
What knowledge would students need to master this standard?
What reasoning proficiencies would students need to master this standard?
None? None?
Standard/Objective:
SC.O.7.1.11: Construct . . . charts, graphs and tables for various purposes . . .Type: Knowledge Reasoning Performance Skill Product
Learning TargetsWhat are the knowledge, reasoning, skill or product targets underpinning the standard or benchmark?
Knowledge Targets Reasoning Targets Performance Skill Targets
Product Targets
What knowledge would students need to master this standard?
What reasoning proficiencies would students need to master this standard?
What performance skills would students need to practice to master this standard?
What products would students need to practice creating to master this standard?
Informal QuestioningConcept Questions - •Why do we care about this? Why are we studying this?•Open ended•Broad, over-arching, outcome oriented questions•Will motivate students and target higher-order thinking . What is the big concept you are trying to uncover? Do the standards have any “big concept” words?
Content Questions –•What are the facts? What are the procedures?• Closed ended.• Will help strengthen and develop students’ understanding of the larger questions.
Informal QuestioningConcepts:
Conflict – How can conflict be resolved? (How could this particular conflict have been resolved?)Change - How can we cope with change?Friendship – What does it mean to be a friend?Freedom (of speech) – Is pure freedom of speech desirable in today’s world? Why or why not? http://www.esldiscussions.com/index.html Questions Resource
Teachers who do not specifically plan classroom discussion
questions ahead of time to tap particular higher-order thinking
skills, but rather ask extemporaneous questions “on
their feet,” are likely to ask recall questions.
Susan M. Brookhart, 2010
ChecklistsChecklist of items - the least complex form of assessment. Yes or No – Is it present or not? There is no value attached to the performance. All elements weighted the same.
Skill
On-task throughout time period
Participates in class discussions
Collaborates with other students
Score
To determine a score, the total number of checked items or the percentage of total possible. No quality is attached unless specified in the item. Example, “three paragraphs required” or “neatly” or “500 words.”Meaning is then attached to the score. For example, what is the minimum score that would be considered proficient?
ChecklistsExample of basic speech skills:
The student:
Maintains eye contact with the audience
Speaks loudly enough to be heard in all parts of the room
Enunciates clearly
Stands up straight (does not shift from foot to foot)
Does not go over the allotted time
Has notes
Uses notes sparingly
Score
Rating ScalesGraphic rating scale on a continuum – example :
Never Seldom Sometimes Usually Always
•Turns in lessons on time
•Uses correct capitalization
•Completes projects
Rating ScalesNumerical rating scale – example : 1 to 10 with 10 being the highest
Performance Rating
Time on task
Participation in class discussion
Developing a plan/Setting goals
Perseverance during project
Total:
Rating ScalesNumerical rating scale with descriptions example:
1 = typical for grade/age 3. Quite advanced for grade/age2 = above average for 4 = Remarkable for grade/age grade/age (1 in 50)
Rating
Learns quickly
Shows power of concentration
Enjoys “adult” conversations
Has many and/or intense interests
Asks many questions
Invents, creates
Total:
RubricsA quality is attached to each skill. Start with the skill you want to assess.Example: Does the student reason inductively from the examples to arrive at a clear, accurate description of physical and chemical changes (example from book)?
Completely and clearly – Response give clear evidence of reasoning from the examples.
Partially – Response is accurate, but reasoning from examples isn’t clear or is only partial.
No – Response does not demonstrate reasonable conclusions from the examples.
RubricsA quality is attached to each skill. Very generalized example of SCALE: Scale refers to numerical or word ratings.
5 4 3 2 1
• Exceptional or Distinguished
• Above Mastery
• Proficient or Mastery
• Below Mastery or Proficient
• Limited or Novice
Scoring Performance Assessments: Scoring is not always simple and straightforward. Students generate their own responses.
Rubrics
Developing Competent Proficient
• Preparation
• Organization of the portfolio
Another example. Very generalized example of Portfolio (Paulson & Paulson, 1996)
Rubrics•may have holistic areas or specific dimensions.•may be used for self-monitoring and self-assessment•may be used in authentic contexts•may be used formatively, summatively or as interim assessments
Rubrics
Developing Competent Proficient
• Reflection • Does not accurately self assess strengths and weaknesses
Accurately self assesses strengths and weaknesses
Focuses on what and how the knowledge or skill was learned
• Goal setting • Goals vague; takes no responsibility for planning work
Accepts responsibility/ describes progress toward goals
Identifies future goals that link to strengths and weaknesses
A quality is attached to each skill. More descriptive example of Portfolio (Paulson & Paulson, 1996)
Rubrics
Skill Criteria Criteria Criteria
•Accuracy of Information
100% accurate information
Inaccurate information
•Craftsmanship Well organized, logical/ clear (strong word choices, good sentence variety, powerful images)
•Reasoning Skills •Reasons are relevant and to the point the writer is making. Justifications
Define the highest performance level first. A three-level rubric is common, but five levels allow for some “gray areas.”
4 3 2 1 Score
Fluency I can think of many ideas.
I can think of some ideas
If I get some help, I can think of ideas
I have a hard time thinking of ideas
Flexibility I notice what is surprising and unusual
I notice unusual things around me
When someone reminds me, notice
I hardly ever notice unusual things
Evaluation I know several ways of deciding
I can tell which ideas are worth working on
With help, I can tell which ideas worthwhile
I cannot tell which ideas are worthwhile
Risk-taking I like to try new ideas
I try new ideas
Sometimes I try new ideas
I do not try new ideas
Seeking Challenges
Goal setting (etc.)
Goal setting Goal setting (etc.)
I do not set goals
Elaboration When I have good idea, I add details to make great
I can usually add details to make better
Sometimes, I can think of way to make better
I do not know how to make better
Criteria Exemplary (4-5) Good(2-3)
Needs Improvement(0-1)
Initial Questions Questions are probing and help clarify facts
All questions may not be relevant
Few or no questions formulated
Understanding the problem
Clearly defines the problem
Statement has some vagueness or missing information
Problem defined incorrectly
Seeking information
Identifies several sources of information
Relies on few sources Not clear as to what is needed
Risk-taking I try new ideas Sometimes I try new ideas
I do not try new ideas
Integration of knowledge
Effectively applies previous knowledge
Applies limited amount of prior knowledge
Unable to connect previous knowledge
Criteria 2 1 0
Thesis (judgment of credibility, identification of assumption or persuasive tactic, etc.)
Thesis is clear, is complete, and answers the question posed by the problem or task.
Thesis is clear and at least partially answers the question posed by the problem or task.
Thesis is not clear or does not answer the question posed by the problem or task.
Evidence Evidence is accurate, relevant and complete.
Evidence is mostly clear, relevant and complete.
Evidence is not clear, relevant or complete.
Reasoning and clarity
The way in which the evidence supports the thesis is clear, logical and well explained.
The way in which the evidence supports the thesis is mostly clear and logical. Some explanation is given.
The way in which the evidence supports the thesis is not clear, is illogical or is not explained.
General Rubric for Critical Thinking Involving Judgment
The Crow and the PitcherA crow, dying of thirst, came upon a
pitcher which had once been full of water. When the crow put his beak into the mouth of the
pitcher, he found that only very little water was left in it, and he could not reach far enough down to get at it. He tried and tried, but at last had to
give up in despair.Then a thought came to him. He took a
pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the
pitcher. Then he took another pebble and dropped it into the pitcher. At last he saw the
water rising toward him, and after casting a few more pebbles into the pitcher, he was able to
drink and save his life.
The Crow and the Pitcher
A 3
A 2
A 1
Consequences and Implications What would have happened if the crow had done the
following:a) Kept putting his beak in the bottle?b) Flown away?c) Broken the bottle?d) Waited for rain?
Cause and Effect What caused the water to reach the crow? What overall
effect did it have on the crow?
Sequencing What steps did the crow use to get water? Outline them
below in order:1. _________________2. _________________3. _________________4. _________________
The Crow and the PitcherC
1
C 3
C 2
Main Idea/Theme
Inference
What made the crow successful in getting a drink of water? Why did his plan work?
Characterization
What are the crow’s most important qualities? What other characters have you read about that show similar life qualities? How were their situations similar or different from the crow’s situation?
What main idea(s) did you get from this story?
Assessment/Response Form
Name:____________________________________________ Date: ____________
Scoring: 0 = Needs Improvement 1 = Satisfactory 2 = Exceeds Expectations Students score their work in the top part of the scoring box.
Teachers score the work in the bottom part of the scoring box.
Student Score: 0 1 2 Teacher Score: 0 1 2
Student Score: 0 1 2 Teacher Score: 0 1 2
Student Score: 0 1 2 Teacher Score: 0 1 2
Student Comments: Is there anything you would like your teacher to know about your work on this reading?
Selection Title: ________________________________________________________
Circ
le On
e: A3
B3
C3 D
3 Ci
rcle
One:
A2
B2
C2
D2
Circ
le On
e: A1
B1
C1 D
1
Teacher Comments:
There’s not a book I read about that was similar to this one. The crow’s most important qualities are water and food.
The crow was successful because when you put the pebbles in the water, the water came up. That helped him to get water. He had to get pebbles and keep doing that until water came up.
Crows get thirsty. Crows are smart. The crow kept putting pebbles in the water until it could drink. I didn’t even know that.
Main Idea
Inference
Characterization
Learning – Thinking Skills Rubric
Assess How?
Susan M. Brookhart (2010)
Many people may be surprised that higher-order thinking can be assessed with “well-written” multiple choice test items.
NAEP (reasoning with data) The table below shows information about the weather in four cities on the same day.
In which city did snow most likely fall at some time during the day? A. City 1 B. City 2 C. City 3 D. City 4
City 1 City 2 City 3 City 4
High Temperature 65 ◦ 80 ◦ 48 ◦ 25 ◦Low Temperature 56 ◦ 66 ◦ 38 ◦ 10 ◦Precipitation – Rain or Snow (inches)
2 inches 0 inches 1 inch 1 inch
WESTEST 2 Do you use your phone to access your e-mail? Do you make a lot of calls on your cell phone? Do you spend a lot of time on your cell phone? Do you IM with your friends? Have you ever thought about reading a book on your phone? It’s all the rage in Japan! Cell phone novels, called keitai shosetsu, are becoming increasingly trendy in Japan and are starting to gain popularity in China and South Korea. The spread of cell phone novels is mainly due to their appeal to young adults, the same audience with whom blogging, IMing, text messaging, and cell phone use are all the rage. Since cell phone screens are so small, only about a …
What is the effect of the series of rhetorical questions at the beginning of this article?A. They draw readers into the article by making them think about their answers to thequestions.B. They illustrate the point that so many people now have cell phones at their disposal.C. They point out that the article’s main purpose is to talk about the many uses of cell phones.D. They show how American readers can become successful novel writers using their cell phones.
“How to Assess Higher Order Thinking Skills” (example from book)
Which of the following scenarios describes behavior that is legal because of the First Amendment?A. Mr. Jones threw a rock through the front window of Mr. Smith’s house.
Around the rock was tied a paper that called Mr. Smith nasty names.
B. Mr. Jones waited until Mr. Smith left for work one morning, then got in his car and followed him, honking and yelling.
C. Mr. Jones doesn’t trust his neighbor, Mr. Smith. Jones believes Smith is a dangerous person and a threat to the peace of the neighborhood. Therefore, Mr. Jones buys a gun.
D. Mr. Jones wrote a letter to the editor of the local paper. Mr. Smith heads a local environmental committee, and Mr. Jones called his position “disastrous.”
“How to Assess Higher Order Thinking Skills” (example from book)
Essay questions assessing the ability to make deductive conclusions: 1. Select one of the amendments in the Bill of Rights of the U.S. Constitution. Describe a specific example of one of the rights in the Bill of Rights. The example can be from a real event or something you make up yourself, but it must be a clear illustration of one of the rights in the Bill of Rights. Tell the story of your example. Then explain which right your story exemplifies, from which amendment, and tell why.
Criteria for feedback or rubric:
• Appropriate identification of a particular right and amendment.• Appropriateness of example.• Appropriateness of evidence.• Soundness of reasoning and clarity of explanation.
“How to Assess Higher Order Thinking Skills” (example from book)
Questions assessing critical thinking involving judgment: 12. The poster shown above was made during the First World War.
What was the poster designed to do?
A. Make people feel that it would be easy to win the war.B. Make people feel guilty for thinking that war is harmful.C. Get people to join the army by making them feel responsible for starting the
war.D. Get people to join the army by appealing to patriotic feelings.
“How to Assess Higher Order Thinking Skills” (example from book)
Questions assessing critical thinking involving judgment: 12. The poster shown above was made during the First World War.
What was the poster designed to do? Explain how you came to this conclusion.
Criteria for feedback or rubric:• Clear, appropriate statement of the main point.• Appropriateness of evidence.• Soundness of reasoning and clarity of explanation.
Multiple Choice
Examples from PISA – Program for International Student Assessment
http://nces.ed.gov/surveys/pisa/items.asp?sub=yes
Test Items from ACT
http://www.actstudent.org/sampletest/index.html
• What is the primary aim of assessment?
• Who will use the information?
• How will the information be used?
In Review:
What other measures are there?
How do we measure the football program?
How do we measure the music/dance program?
What other measures are there?
Competitions/Games
Exhibits
Bulletin Boards
Websites
What other measures are there?
Portfolio – of successes/best work.
Knowing that the “failures” along the way were lessons learned on the way to success.
Resources
• Julia Roberts NAGC’s WOW series• Susan K. Johnson NAGC’s WOW series• Brookhart, Susan M. (2010) Assess Higher-Order Thinking
Skills• Dweck, Carol (2010) Even Geniuses Work Hard,
Educational Leadership, September 2010, Vol. 68 No. 1 www.ascd.org
• Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association
• Wormeli, Rick (2006) Fair Isn’t Always Equal; Assessing & Grading in the Differentiated Classroom, p. 14, Stenhouse Publishers, Portland, Maine.
ResourcesNew Online IEP Form: http://wvde.state.wv.us/osp/forms.html
Policy 2419 at http://wvde.state.wv.us/policies.html
Resources for teachers at http://wvde.state.wv.us/osp/gifted.html
Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership September 2010, Vol. 68 No. 1 www.ascd.org
Pink, Daniel (2006) A Whole New Mind, Riverhead Books Published by the Penguin Group. New York, NY.
Reis, Sally & Renzullli, Joseph Curriculum Compacting: A Systematic Procedure for Modifying the Curriculum for Above Average Ability Students The National Research Center on the Gifted and Talented - University of Connecticut .
Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association
Van-Tassel-Baska, Joyce (2003) Content-Based Curriculum for High-Ability Learners, Waco, TX, Prufrock Press, Inc.