Content Analysis Workshop

John L. Smithson, Director, Measures of the Enacted CurriculumAlissa Minor, Projects Manager, Measures of the Enacted Curriculum

Wisconsin Center for Education Research, University of Wisconsin-Madisonjohns@wcer.wisc.edu

Minneapolis, MN Nov 7-9, 2007

Describing the Content of

Standards & Assessments

The Goal

To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.

The Goal

To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.

Content Descriptions

ContentDescriptions

AlignmentAnalyses

NeedsAssessment

ProgramEvaluation

MonitoringChange

CurriculumManagement

TeacherReports

ContentAnalyses

SECTaxonomy

Uses of Content Analysis Results

Descriptive: (Tile Charts and Content Maps)

• Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning.

• Predict student achievement gains• Control for content to examine other factors• As an outcome measure for change over time• Examine alignment of Standards & Assessments

Analytic: (Alignment)

To Describe Instructional Content

SEC utilizes a two-dimensional taxonomy based on:Topic

by Cognitive Demand

1

2

3

4

5

B C D E F

Categories of Cognitive Demand

Topics Memorize Conduct

Investigations Communicate Understanding

Analyze Information

Apply / Make Connections

Nature of Science

Science & Technology

Science, Health, Env.

Meas. & Calc. In Sci.

Comp. Of Living

Systems

Botany

The Content Matrix

Categories of Cognitive Demand

Topics Memorize Conduct

Investigations Communicate Understanding

Analyze Information

Apply / Make Connections

Nature of Science

Science & Technology

Science, Health, Env.

Meas. & Calc. In Sci.

Comp. Of Living

Systems

Botany

… adding levels of relative emphasis yields a 3-D construct

MemorizePerform

CommunicateConjecture

Connect0

0.020.040.060.08

0.1

Operations

Number Sense

Measurement

Geometric Concepts

Algebraic Concepts

Data Analysis

Instructional Technology

MemorizePerform

CommunicateConjecture

Connect

State J Grade 8Mathematics Assessment

Content Map Data Displays

To Facilitate Comparisons

Uses of Content Analysis Results

Descriptive: (Tile Charts and Content Maps)

• Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning.

• Predict student achievement gains• Control for content to examine other factors• As an outcome measure for change over time• Examine alignment of Standards & Assessments

Analytic: (Alignment)

Alignment as a Quantity

1.000.00 0.500.25 0.75

0.27 (Avg. Alignment: Test to Standard)

Range of Alignment: Test to Standard)

(Based on results for 10 states, across grades 4, 6 and 8: SEC Collaborative 2003)

0.14 0.40

Aligning Tests to Standards

State U Grade 8 Mathematics Alignment: Test to Standard (0.23)

Instructional Alignment

1.000.00 0.500.25 0.750.03 0.31

Avg.0.05 0.29

Based upon results for 168 teachers, across 3 states: MSP PD Study 2004

Fine Grain

Min. Max.0.17

Avg.Min. Max.

Instruction to Standards & Assessments

Standards

Assessments

0.19

Time 0 Time 1 Time 2

Time 0 Time 1 Time 2

Learning Gains by Course Type

Learning Gains Controlling for Content

RegentsAlgebraStretch RegentsMath A/B/UCSMPGen. Mth. / Pre-alg.

From:Upgrading High School Mathematics Instruction,(Gamoran, Porter, Smithson, & White, 1997),EEPAv19n4

12

11

10.5

12

9.5

10

11.5

12

11

10.5

12

9.5

10

11.5

Explaining variation in student learning gains

Alignment Index:Instruction to Standards

MathematicsAcross 4 Districts

Alignment Analyses for School Improvement

Using alignment as an outcome measure

(Measuring change in alignment over time)

Treatment 99

Control 124

Leaders 16

Counts

Content Analysis Procedures

Exploring theDimensions of Content

But first….

Let’s take a 10 minute break!

The Two Dimensions Of Content

What students should know[Topics]

And…

Be Able to Do [Expectations for student performance]

Describing the Cognitive Domain

3

4

5

6

How Many Categories?

Bloom’s

SEC

DOK (Webb)

SCASS Science

Dimensions of Knowing & Inquiry

Acquire Use Extend

(From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student StandardsScience Project, Council of Chief State School Officers, 1997)

Depth of Knowledge

* Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.

Skill/Concept

Recall

StrategicThinking

ExtendedThinking

Level

1

2

3

4

Skill/ConceptRecallStrategicThinking

ExtendedThinking

Acquire Use Extend

Exploring Cognitive Demand

Cognitive Demand (or Expectations for Student Performance)

Skill/ConceptRecallStrategicThinking

ExtendedThinking

Acquire Use Extend

MemorizePerform

ProceduresDemonstrate

Understanding

Conjecture,Generalize

Prove

Solve non-routine/ makeconnections

MemorizeConduct

InvestigationsCommunicateUnderstanding

AnalyzeInformation

Apply concepts/make

connections

RecallPerform

ProceduresAnalyze/

InvestigateEvaluate

Generate/Demonstrate

Cognitive Demand (or Expectations for Student Performance)

Skill/ConceptRecallStrategicThinking

ExtendedThinking

Acquire Use Extend

MemorizePerform

ProceduresDemonstrate

UnderstandingAnalyze

InformationEvaluate/Apply

Recall Understanding Application Analyze Evaluate Create

Exploring Cognitive Demand

CgD Immersion Activity• Organize into Groups/Tables• Each Table w/ CgD Pie• Each Person w/ Cgd Descriptors

Step 1: Place CgD cards on Pie Slices face-down

Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same sliceif Group Agrees ... discuss key wordsif not … Discuss … operational definition to distinguish

Step 3: Discuss disagreementsif consensus reached put in envelope / if not, set aside

Content Analysis Materials

• Cognitive Demand List• Topics Lists• Comments & Suggestions Worksheet• Coding Forms• Documents to be analyzed

Cognitive Demand Lists

• Five categories of cognitive demand• Slightly different for each subject• Each category is defined by a list of descriptors• The list of descriptors are not exhaustive

• Each category stands on its own• Each category has an associated letter (B-F)

Dimension A: Content Topics

Topics List (In your packet of material)

Organized at two levels:Content Areas

(16 for Mathematics)(27 for Science)(14 for ELAR)

Topics (identified by number)(182 Mathematics Topics)(211 Science Topics)(114 ELAR Topics)

Plus: non-specific & other

Topics Lists

• Topics Lists• Mathematics• Science• English Language Arts & Reading

• Cover grades K-12• Organized into Content Areas• Topics & Content Areas have an associated #

Comments & Suggestion Worksheet

• One for each reviewer - more available• Use to:

• Record coding conventions/decision rules• Suggest/identify additional topics not listed• Suggest/identify additional CGD descriptors• Provide other comments & suggestions

• Be sure to turn in at end of workshop (and with mail-in materials, as necessary).

Coding Forms

• Assessment Coding Forms• Benchmark Coding Forms

• Each is used to record content descriptions

• Each content description consists of• A topic number• A cognitive demand category letter

Rater: ________ Subject:__NC Math Gr 3 Benchmark Test (SF)_ Form: Test 1 Page of

Itm. Desig./Nbr. Itm. Desig./Nbr.TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 TPC1 CGD1 TPC2 CGD2 TPC3 CGD3

1 40

2 41

3 42

4 43

5 44

6 45

7 46

8 47

9 48

10 49

11 50

12 51

13 52

14 53

15 54

16 55

17 56

18 57

19 58

20 59

21 60

22 61

23 62

24 63

25 64

26 65

27 66

28 67

29 68

30 69

31 70

32 71

33 72

34 73

35 74

36 75

37 76

38 77

39 78

Content Code 2 Content Code 3Content Code 1 Content Code 2 Content Code 3 Content Code 1

Assessment Coding Forms

Rater: ________ Document:__NAEP Grade 8____Itm. Desig./Nbr. Itm. Desig./Nbr.

Number PropertiesNumber Sense TPC1 CGD1 TPC2 CGD2 TPC3 CGD31a 4a

1b 4b

1d 4c

1e 4d

1f

1g 5a

1i 5b

1j 5c

Estimation TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 5d2a

5e2b

5f2c

Measurement

2d Measuring Physical Attributes TPC1 CGD1 TPC2 CGD2 TPC3 CGD3

1bNumber Operations TPC1 CGD1 TPC2 CGD2 TPC3 CGD3

3a 1c

3d 1g

3e 1h

3f 1j

3g 1k

Properties of Number & Operations

TPC1 CGD3

CGD3CGD1 TPC2

CGD1 TPC2 CGD2 TPC3

CGD2 TPC3

Content Code 1 Content Code 2 Content Code 3 Content Code 1 Content Code 2 Content Code 3Ratios & Proportional Reasoning

TPC1

Standards Coding Forms

Standards Coding Forms

Coding Procedures

TPC CGDContent Descriptions arerecorded using pairs ofcodes to identifyparticular intersectionsTopic & CognitiveDemand Categories.

B200203

C

Cognitive Demand Categories arerepresented by letters B-F.

Topics are identifiedusing numbers.

D290

TPC CGD

Large Grain Topic Descriptions:To reference vague content descriptions thatcould cover most topics in a given contentarea, use the content area code (ends in 00)to indicate the relevant content area. (Thereare 7 content areas in K-8 Mathematics and25 content areas in K-8 Science.)

200Grain Size can berepresented using thefollowing conventions:

Coding Conventions

Should none of five categories of cognitivedemand listed on the Cognitive DemandCoding Sheet appear relevant, (e.g. thestandard statement makes no reference tocognitive demand, use the letter ‘Z’. Thisshould only occur rarely, if at all.

Non-SpecificCognitive Demand:

Coding Conventions

TPC CGD200 Z

Coding Conventions

Non-specific Content TPC CGD

0

Z

999 Z

Non-specific target subject 0

EOut of subject area

Cognitive Demand only

(*Please make note on your comments form.)

Non-specific Topics: TPC CGDB290

293 DOther: (rater-defined*)

Other: (topic not listed)

Coding Conventions

Practice Coding Exercise

Content Analyzing Assessments(Three code maximum)

Practice Coding Exercise

Content Analyzing Standards(Six code maximum)

The Content Analysis Process

‘Coding’ Teams of 4-5 Content Experts

Independent Coding by each Analyst w/ Group Discussion

Goal for Process: Generalizability not Inter-rater Reliability

Should not be necessary to discuss every item – select by team

Pick-up and return documents / coding sheets to Alissa

Sign & return to Alissa non-disclosure forms

A neutral content language

Topics by Cognitive Demand

The intended curriculum: State content standards—What students should learn

The enacted curriculum: What teachers teach

The learned curriculum: Student outcomes based on school learning

The assessed curriculum: State (and other) assessments—tested learning

Content Analysis Workshop