Practical Alignment for Item / Test Developers:

Revised Bloom’s Taxonomy (RBT) and

Depth of Knowledge Levels (DOK)Staff Trainings, 2009

Jerrie W. Brown, Content Lead for Social Studies Assessments

North Carolina State University

Center for Urban & Community Services

Technical Outreach for Public Schools (TOPS)

“Lack of excellence in American schools is not caused by ineffective teaching, but mostly by misaligning what teachers teach, what they intend to teach, and what they assess as having been taught.”

-Prof. S. Alan Cohen, 1987 University of San Francisco

“Alignment also helps ensure the validity of test results. If standards and tests are not aligned, results might provide a misleading picture of how well students are attaining standards.”

-Robert Rothman, 2003Annenberg Institute for School Reform

What is alignment?The No Child Left Behind Act of 2001 (NCLB) requires that each state align assessments to content standards.

Once the enacted (instruction), intended (state standards), and assessed (state tests) curricula have been measured, questions can be asked about the extent to which content is similar across them.

To the extent content is the same, they are said to be aligned.

J. Brown, NCSU, 2009

Defining Alignment

If the content assessed is exactly the same as the content represented in the standards, alignment is perfect.

There are two ways in which alignment can be less than perfect:

(1) Content in the standards may not be assessed, and (2) content assessed may not be in the standards.

J. Brown, NCSU, 2009

Alignment Procedure

1. Review alignment criteria of Revised Bloom’s Taxonomy (RBT) and Depth of Knowledge (DOK) levels

2. Rate standards3. Rate test items4. Aggregate individual ratings5. Determine degree of RBT-based alignment6. Determine degree of DOK-based alignment

J. Brown, NCSU, 2009

J. Brown, NCSU, 2009 6

Alignment DimensionsDOKDOK – Two Dimensions1. Depth of Knowledge Levels2. Content Topics or Strands

(Webb, 2002)

RBTRBT – Two Dimensions1. Cognitive Processes2. Knowledge Types

(Anderson, 2001)

6. Create

5. Evaluate

4. Analyze

3. Apply

2. Understand

1. RememberContent

Topic/Strand A

4. Extended Thinking

3. Strategic Thinking

2. Skill/Concept

1. RecallA. Factual

B. Conceptual

C. Procedural

D. Meta-Cognitive

ContentTopic/Strand B

ContentTopic/Strand C

(Note: achieves content validity) (Note: achieves content validity and instructional validity)

7

Revised Bloom’s Taxonomy (RBT) Table for _____________________________

(Subject / Grade Level)

KnowledgeDimension

(Lower Order = 0%) Cognitive Process Dimension (Higher Order = 0%)

1.Remember

2.Understand

3.Apply

4.Analyze

5.Evaluate

6.Create

A. FactualKnowledge (0%)

B.ConceptualKnowledge(0%)

C.ProceduralKnowledge(0%)

D.Meta-CognitiveKnowledge(0%)

Adapted from Anderson, Lorin W. and David R. Krathwohl, et al., (2001).

J. Brown, NCSU, 2009

8

Depth of Knowledge (DOK) Levels for _____________________________

(Subject / Grade Level)

ContentCategories

Depth of Knowledge Levels

CategoricalConcurrence

Yes / No

1.Recall

2.Skill/Concept

3.StrategicThinking

4.ExtendedThinking

A. Topic/Strand__________(Specify)

B.Topic/Strand__________(Specify)

C.Topic/Strand__________(Specify)

Adapted from Norm Webb, (2002).

J. Brown, NCSU, 2009

RBT’s Knowledge Dimension:

Major Types Definitions

A Factual The basic elements students must know to be acquainted with a discipline or solve problems in it

B Conceptual The interrelationships among the basic elements within a larger structure that enable them to function together

C Procedural How to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods

D Metacognitive Knowledge of cognition in general as well as awareness and knowledge of one’s own cognition

J. Brown, NCSU, 2009

RBT’s Knowledge Dimension: Major Types Subtypes

A Factual AA. Knowledge of terminologyAB. Knowledge of specific details and elements

B Conceptual BA. Knowledge of classifications and categoriesBB. Knowledge of principles and generalizationsBC. Knowledge of theories, models, and structures

C Procedural CA. Knowledge of subject-specific skills and algorithmsCB. Knowledge of subject-specific techniques and methodsCC. Knowledge of criteria for determining when to use appropriate procedures

D Metacognitive DA. Strategic knowledgeDB. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledgeDC. Self-knowledge

J. Brown, NCSU, 2009

RBT’s Cognitive Process Dimension:

Cognitive Process Categories

Definitions

1 Remember Retrieve relevant knowledge from long-term memory

2 Understand Construct meaning from instructional messages, including oral, written, and graphic communication

3 Apply Carry out or use a procedure in a given situation

4 Analyze Break material down into its constituent parts and determine how the parts relate to one another and to an overall structure or purpose

5 Evaluate Make judgments based on criteria and standards

6 Create Put elements together to form a coherent or functional whole; reorganize elements into a new pattern or structure

J. Brown, NCSU, 2009

RBT’s Cognitive Process Dimension: Cognitive Process Categories Cognitive Processes

1 Remember 1.1 Recognizing1.2 Recalling

2 Understand 2.1 Interpreting2.2 Exemplifying2.3 Classifying2.4 Summarizing

2.5 Inferring2.6 Comparing2.7 Explaining

3 Apply 3.1 Executing3.2 Implementing

4 Analyze 4.1 Differentiating4.2 Organizing4.3 Attributing

5 Evaluate 5.1 Checking5.2 Critiquing

6 Create 6.1 Generating6.2 Planning6.3 Producing

J. Brown, NCSU, 2009

Norm Webb’s Specific Alignment Criteria

Content FocusA. Categorical Concurrence: between goals and assessment is met if the same or consistent categories of content appear in both documents

B. Depth-of-Knowledge Consistency: at least 50% of the items corresponding to a goal had to be at or above the level of knowledge of the goal

C. Range-of-Knowledge Correspondence: fifty percent of the objectives for a goal must have at least one related assessment item

D. Balance of Representation: the degree to which one objective is given more emphasis on the assessment than another

Formal Alignment Studies and Depth of Knowledge Levels (DOKs)

J. Brown, NCSU, 2009

Depth of Knowledge LevelsNorm Webb (2002) Generic DOKs

Level 1: Recall– Recall of a fact, information, or procedure.

Level 2: Skill/Concept– Use information or conceptual knowledge, two or more steps, etc.

Level 3: Strategic Thinking– Requires reasoning, developing plan or a sequence of steps, some

complexity, more than one possible answer.

Level 4: Extended Thinking– Requires an investigation, time to think and process multiple

conditions of the problem.

J. Brown, NCSU, 2009

Subject-Specific DOKsScience Social Studies Math

Level 1Recall and Reproduction

Recall of Information

Recall

Level 2Skills and Concepts

Basic Reasoning

Skill/Concept

Level 3Strategic Thinking

Complex Reasoning

Strategic Thinking

Level 4Extended Thinking

Extended Reasoning

Extended Thinking

J. Brown, NCSU, 2009

Reading DOKs

J. Brown, NCSU, 2009

Level 1 :•Support ideas by reference to details in the text.•Use a dictionary to find the meaning of words.•Identify figurative language in a reading passage.

Level 2:•Use context cues to identify the meaning of unfamiliar words.•Predict a logical outcome based on information in a reading selection.•Identify and summarize the major events in a narrative.

Level 3:•Determine the author’s purpose and describe how it affects the interpretation of a reading selection.•Summarize information from multiple sources to address a specific topic.•Analyze and describe the characteristics of various types of literature.

Level 4: •Analyze and synthesize information from multiple sources.•Examine and explain alternative perspectives across a variety of sources. •Describe and illustrate how common themes are found across texts from different cultures.

Getting Started

Resources needed:– Knowledge types (a-d)– Cognitive processes (1-6)– Depth of knowledge levels (1-4)– Standards ratings forms– Test items ratings forms– RBT blank taxonomy tables

J. Brown, NCSU, 2009

Rating WA State Standards

J. Brown, NCSU, 2009

Plotting Standards

J. Brown, NCSU, 2009

KnowledgeDimension

(Lower Order = 0%) Cognitive Process Dimension (Higher Order = 0%)

1.Remember

2.Understand

3.Apply

4.Analyze

5.Evaluate

6.Create

A. FactualKnowledge (0%)

101, 5

9B.ConceptualKnowledge(0%)

2, 3

86

C.ProceduralKnowledge(0%)

74

D.Meta-CognitiveKnowledge(0%)

Rating ‘WASL’ Test Items

J. Brown, NCSU, 2009

Plotting Test Items

J. Brown, NCSU, 2009

KnowledgeDimension

(Lower Order = 0%) Cognitive Process Dimension (Higher Order = 0%)

1.Remember

2.Understand

3.Apply

4.Analyze

5.Evaluate

6.Create

A. FactualKnowledge (0%)

101, 5

9B.ConceptualKnowledge(0%)

2, 3

86

C.ProceduralKnowledge(0%)

74

D.Meta-CognitiveKnowledge(0%)

Individual and Group Results

Thank You for Participating!

23

FOR MORE INFORMATION:

Mr. Jerrie W. Brown, ConsultantNorth Carolina State UniversityCenter for Urban and Community ServicesTechnical Outreach for Public Schools1500 Blue Ridge RoadRaleigh, N. Carolina 27607

Tel: (919) 515-1125Email: Jerrie_Brown@ncsu.eduWebsite: http://www.cuacs.ncsu.edu