Content Analysis Workshop
description
Transcript of Content Analysis Workshop
![Page 1: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/1.jpg)
Content Analysis Workshop
John L. Smithson, Director, Measures of the Enacted CurriculumAlissa Minor, Projects Manager, Measures of the Enacted Curriculum
Wisconsin Center for Education Research, University of [email protected]
Minneapolis, MN Nov 7-9, 2007
Describing the Content of
Standards & Assessments
![Page 2: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/2.jpg)
The Goal
To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.
![Page 3: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/3.jpg)
The Goal
To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.
![Page 4: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/4.jpg)
Content Descriptions
ContentDescriptions
AlignmentAnalyses
NeedsAssessment
ProgramEvaluation
MonitoringChange
CurriculumManagement
TeacherReports
ContentAnalyses
SECTaxonomy
![Page 5: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/5.jpg)
Uses of Content Analysis Results
Descriptive: (Tile Charts and Content Maps)
• Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning.
• Predict student achievement gains• Control for content to examine other factors• As an outcome measure for change over time• Examine alignment of Standards & Assessments
Analytic: (Alignment)
![Page 6: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/6.jpg)
To Describe Instructional Content
SEC utilizes a two-dimensional taxonomy based on:Topic
by Cognitive Demand
1
2
3
4
5
B C D E F
![Page 7: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/7.jpg)
Categories of Cognitive Demand
Topics Memorize Conduct
Investigations Communicate Understanding
Analyze Information
Apply / Make Connections
Nature of Science
Science & Technology
Science, Health, Env.
Meas. & Calc. In Sci.
Comp. Of Living
Systems
Botany
The Content Matrix
![Page 8: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/8.jpg)
Categories of Cognitive Demand
Topics Memorize Conduct
Investigations Communicate Understanding
Analyze Information
Apply / Make Connections
Nature of Science
Science & Technology
Science, Health, Env.
Meas. & Calc. In Sci.
Comp. Of Living
Systems
Botany
… adding levels of relative emphasis yields a 3-D construct
![Page 9: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/9.jpg)
![Page 10: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/10.jpg)
MemorizePerform
CommunicateConjecture
Connect0
0.020.040.060.08
0.1
Operations
Number Sense
Measurement
Geometric Concepts
Algebraic Concepts
Data Analysis
Instructional Technology
MemorizePerform
CommunicateConjecture
Connect
State J Grade 8Mathematics Assessment
Content Map Data Displays
![Page 11: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/11.jpg)
![Page 12: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/12.jpg)
To Facilitate Comparisons
![Page 13: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/13.jpg)
Uses of Content Analysis Results
Descriptive: (Tile Charts and Content Maps)
• Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning.
• Predict student achievement gains• Control for content to examine other factors• As an outcome measure for change over time• Examine alignment of Standards & Assessments
Analytic: (Alignment)
![Page 14: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/14.jpg)
Alignment as a Quantity
1.000.00 0.500.25 0.75
0.27 (Avg. Alignment: Test to Standard)
Range of Alignment: Test to Standard)
(Based on results for 10 states, across grades 4, 6 and 8: SEC Collaborative 2003)
0.14 0.40
Aligning Tests to Standards
State U Grade 8 Mathematics Alignment: Test to Standard (0.23)
![Page 15: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/15.jpg)
Instructional Alignment
1.000.00 0.500.25 0.750.03 0.31
Avg.0.05 0.29
Based upon results for 168 teachers, across 3 states: MSP PD Study 2004
Fine Grain
Min. Max.0.17
Avg.Min. Max.
Instruction to Standards & Assessments
Standards
Assessments
0.19
![Page 16: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/16.jpg)
Time 0 Time 1 Time 2
Time 0 Time 1 Time 2
Learning Gains by Course Type
Learning Gains Controlling for Content
RegentsAlgebraStretch RegentsMath A/B/UCSMPGen. Mth. / Pre-alg.
From:Upgrading High School Mathematics Instruction,(Gamoran, Porter, Smithson, & White, 1997),EEPAv19n4
12
11
10.5
12
9.5
10
11.5
12
11
10.5
12
9.5
10
11.5
Explaining variation in student learning gains
![Page 17: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/17.jpg)
Alignment Index:Instruction to Standards
MathematicsAcross 4 Districts
Alignment Analyses for School Improvement
Using alignment as an outcome measure
(Measuring change in alignment over time)
Treatment 99
Control 124
Leaders 16
Counts
![Page 18: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/18.jpg)
Content Analysis Procedures
Exploring theDimensions of Content
But first….
Let’s take a 10 minute break!
![Page 19: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/19.jpg)
The Two Dimensions Of Content
What students should know[Topics]
And…
Be Able to Do [Expectations for student performance]
![Page 20: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/20.jpg)
Describing the Cognitive Domain
3
4
5
6
How Many Categories?
Bloom’s
SEC
DOK (Webb)
SCASS Science
![Page 21: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/21.jpg)
Dimensions of Knowing & Inquiry
Acquire Use Extend
(From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student StandardsScience Project, Council of Chief State School Officers, 1997)
![Page 22: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/22.jpg)
Depth of Knowledge
* Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.
Skill/Concept
Recall
StrategicThinking
ExtendedThinking
Level
1
2
3
4
![Page 23: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/23.jpg)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
Exploring Cognitive Demand
![Page 24: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/24.jpg)
Cognitive Demand (or Expectations for Student Performance)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
MemorizePerform
ProceduresDemonstrate
Understanding
Conjecture,Generalize
Prove
Solve non-routine/ makeconnections
MemorizeConduct
InvestigationsCommunicateUnderstanding
AnalyzeInformation
Apply concepts/make
connections
RecallPerform
ProceduresAnalyze/
InvestigateEvaluate
Generate/Demonstrate
![Page 25: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/25.jpg)
Cognitive Demand (or Expectations for Student Performance)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
MemorizePerform
ProceduresDemonstrate
UnderstandingAnalyze
InformationEvaluate/Apply
Recall Understanding Application Analyze Evaluate Create
![Page 26: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/26.jpg)
Exploring Cognitive Demand
CgD Immersion Activity• Organize into Groups/Tables• Each Table w/ CgD Pie• Each Person w/ Cgd Descriptors
Step 1: Place CgD cards on Pie Slices face-down
Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same sliceif Group Agrees ... discuss key wordsif not … Discuss … operational definition to distinguish
Step 3: Discuss disagreementsif consensus reached put in envelope / if not, set aside
![Page 27: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/27.jpg)
Content Analysis Materials
• Cognitive Demand List• Topics Lists• Comments & Suggestions Worksheet• Coding Forms• Documents to be analyzed
![Page 28: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/28.jpg)
Cognitive Demand Lists
• Five categories of cognitive demand• Slightly different for each subject• Each category is defined by a list of descriptors• The list of descriptors are not exhaustive
• Each category stands on its own• Each category has an associated letter (B-F)
![Page 29: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/29.jpg)
![Page 30: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/30.jpg)
![Page 31: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/31.jpg)
![Page 32: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/32.jpg)
Dimension A: Content Topics
Topics List (In your packet of material)
Organized at two levels:Content Areas
(16 for Mathematics)(27 for Science)(14 for ELAR)
Topics (identified by number)(182 Mathematics Topics)(211 Science Topics)(114 ELAR Topics)
Plus: non-specific & other
![Page 33: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/33.jpg)
Topics Lists
• Topics Lists• Mathematics• Science• English Language Arts & Reading
• Cover grades K-12• Organized into Content Areas• Topics & Content Areas have an associated #
![Page 34: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/34.jpg)
Comments & Suggestion Worksheet
• One for each reviewer - more available• Use to:
• Record coding conventions/decision rules• Suggest/identify additional topics not listed• Suggest/identify additional CGD descriptors• Provide other comments & suggestions
• Be sure to turn in at end of workshop (and with mail-in materials, as necessary).
![Page 35: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/35.jpg)
Coding Forms
• Assessment Coding Forms• Benchmark Coding Forms
• Each is used to record content descriptions
• Each content description consists of• A topic number• A cognitive demand category letter
![Page 36: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/36.jpg)
Rater: ________ Subject:__NC Math Gr 3 Benchmark Test (SF)_ Form: Test 1 Page of
Itm. Desig./Nbr. Itm. Desig./Nbr.TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
1 40
2 41
3 42
4 43
5 44
6 45
7 46
8 47
9 48
10 49
11 50
12 51
13 52
14 53
15 54
16 55
17 56
18 57
19 58
20 59
21 60
22 61
23 62
24 63
25 64
26 65
27 66
28 67
29 68
30 69
31 70
32 71
33 72
34 73
35 74
36 75
37 76
38 77
39 78
Content Code 2 Content Code 3Content Code 1 Content Code 2 Content Code 3 Content Code 1
Assessment Coding Forms
![Page 37: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/37.jpg)
Rater: ________ Document:__NAEP Grade 8____Itm. Desig./Nbr. Itm. Desig./Nbr.
Number PropertiesNumber Sense TPC1 CGD1 TPC2 CGD2 TPC3 CGD31a 4a
1b 4b
1d 4c
1e 4d
1f
1g 5a
1i 5b
1j 5c
Estimation TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 5d2a
5e2b
5f2c
Measurement
2d Measuring Physical Attributes TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
1bNumber Operations TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
3a 1c
3d 1g
3e 1h
3f 1j
3g 1k
Properties of Number & Operations
TPC1 CGD3
CGD3CGD1 TPC2
CGD1 TPC2 CGD2 TPC3
CGD2 TPC3
Content Code 1 Content Code 2 Content Code 3 Content Code 1 Content Code 2 Content Code 3Ratios & Proportional Reasoning
TPC1
Standards Coding Forms
![Page 38: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/38.jpg)
Standards Coding Forms
![Page 39: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/39.jpg)
Coding Procedures
TPC CGDContent Descriptions arerecorded using pairs ofcodes to identifyparticular intersectionsTopic & CognitiveDemand Categories.
B200203
C
Cognitive Demand Categories arerepresented by letters B-F.
Topics are identifiedusing numbers.
D290
![Page 40: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/40.jpg)
TPC CGD
Large Grain Topic Descriptions:To reference vague content descriptions thatcould cover most topics in a given contentarea, use the content area code (ends in 00)to indicate the relevant content area. (Thereare 7 content areas in K-8 Mathematics and25 content areas in K-8 Science.)
200Grain Size can berepresented using thefollowing conventions:
Coding Conventions
![Page 41: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/41.jpg)
Should none of five categories of cognitivedemand listed on the Cognitive DemandCoding Sheet appear relevant, (e.g. thestandard statement makes no reference tocognitive demand, use the letter ‘Z’. Thisshould only occur rarely, if at all.
Non-SpecificCognitive Demand:
Coding Conventions
TPC CGD200 Z
![Page 42: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/42.jpg)
Coding Conventions
Non-specific Content TPC CGD
0
Z
999 Z
Non-specific target subject 0
EOut of subject area
Cognitive Demand only
![Page 43: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/43.jpg)
(*Please make note on your comments form.)
Non-specific Topics: TPC CGDB290
293 DOther: (rater-defined*)
Other: (topic not listed)
Coding Conventions
![Page 44: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/44.jpg)
Practice Coding Exercise
Content Analyzing Assessments(Three code maximum)
![Page 45: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/45.jpg)
Practice Coding Exercise
Content Analyzing Standards(Six code maximum)
![Page 46: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/46.jpg)
The Content Analysis Process
‘Coding’ Teams of 4-5 Content Experts
Independent Coding by each Analyst w/ Group Discussion
Goal for Process: Generalizability not Inter-rater Reliability
Should not be necessary to discuss every item – select by team
Pick-up and return documents / coding sheets to Alissa
Sign & return to Alissa non-disclosure forms
![Page 47: Content Analysis Workshop](https://reader035.fdocuments.in/reader035/viewer/2022062723/56813c39550346895da5b956/html5/thumbnails/47.jpg)
A neutral content language
Topics by Cognitive Demand
The intended curriculum: State content standards—What students should learn
The enacted curriculum: What teachers teach
The learned curriculum: Student outcomes based on school learning
The assessed curriculum: State (and other) assessments—tested learning
Content Analysis Workshop