Learning Progressions Learning Progressions in Science

Context

Conference in Iowa City, IA in June 2009Edited book based upon conference papers & conference papers & synthesis discussionsFocused on challenges in 4 g 4strands of learning progressions work :

DefiningDefiningAssessingModeling

National Science FoundationAward No. DRL-0824432

Using

Potential Role of Learning Progressions (LPs)

Grounding assessments in research on student glearning

Aligning curriculum, instruction, and assessment around “big ideas”

Focusing on a smaller number of “big ideas” with ti it d l lcontinuity across grade levels

Describing student learning in terms of qualitatively changing knowledge and practice rather than changing knowledge and practice rather than mastery of a checklist of standards or benchmarks

Consideration of Assessment at a Range of Levels

Classroom Gotwals, Songer, & Bullard: Classroom-Level Research

District

StateState

Alonzo, Neidorf, & Anderson: Large-Scale Assessment

National

International

Using Learning Progressions to Using Learning Progressions to Inform Classroom Assessmento C ass oo ssess e t

A melia Wenk Gotwals , Michigan State Universi ty, g y

Nancy Butler Songer & Lea Bul lard, Universi ty of Michigan

Overview

Potential Role of Learning Progressions in g gClassroom Assessment

ChallengesDesigning Items for Formative Use

Coding Non-conforming Student Responses

C l i d R d tiConclusions and Recommendations

Role of Learning Progressions for Classroom AssessmentAssessment

Provide coherence between curriculum, assessment ,and professional development

Provide a “vision” for teachers to see how students may develop more sophisticated ideas

Design assessment opportunities to gather evidence f h t d t f ll l i iof where students fall on a learning progression

Use evidence to inform future instruction

Fundamental Challenges/Questions

How can we design classroom assessment items such gthat students at a range of placements on a learning progression have opportunities to provide evidence

f h t th k d d ?of what they know and can do?

Wh t d ith t d t th t d t What can we do with student responses that do not “fit” into our hypothesized learning progression?

What can these responses tell us about:What can these responses tell us about:The students?

The learning progression?

h iThe assessment items?

Deep Think Learning Progression

Three grade levels4th grade• 4th grade

• 5th grade• 6th grade

Th i t t d Three interconnected content strands• Ecology (E)• Classification ( C )• Biodiversity (B)

Focus on scientific Focus on scientific explanations• Claim

Evidence• Evidence• Reasoning(Songer, Kelcey, & Gotwals, 2009)

Scientific Explanations

Claim: Assertion or l i i conclusion answering

scientific question

E id S i tifi d t Evidence: Scientific data to support the claim

Reasoning: Justification Reasoning: Justification linking claim and evidence using scientific evidence using scientific principles

( l i 8 h (Toulmin 1958, Kuhn 1993, Lee 2003, Songer, Kelcey & Gotwals, 2008)

Explanation difficultiesp

Claims: difficulty articulating clear claims based Claims: difficulty articulating clear claims based on data; don’t back up claims with evidence or reasoning

Evidence: what counts as evidence (appropriate); sufficient evidence; convincing evidence

Reasoning: doesn’t explicitly link evidence to claim; not articulated

Simplified Scientific Explanation Learning Progression Simplified Scientific Explanation Learning Progression ((SongerSonger, , KelceyKelcey & Gotwals, 2009; Gotwals & & Gotwals, 2009; Gotwals & SongerSonger, 2010), 2010)

Level 4: Student constructs scientific explanationp

Level 3: Student makes a claim and backs it up with sufficient and appropriate evidence but does not use reasoning to tie the evidence to the claimLevel 2: Student makes a claim and backs it up with evidence but the evidence is insufficient orevidence, but the evidence is insufficient or inappropriateLevel 1: Student makes a claim but does not back it up pwith evidence

Level 0: Student does not make a claim or makes an incorrect claim

Assessment DesignAssessment Design

What are features of ideal LP assessments? What are features of ideal LP assessments?

gg

What are features of ideal LP assessments? What are features of ideal LP assessments?

More than end point tasks; e.g., assessments that are sensitive to progress towards end goalp g g

Assessments that provide a range of information of middle knowledge (vs. all or nothing) g ( g)

Assessments that provide information for a range of ability and age levels

Assessments that articulate the when of middle knowledge development (e.g., continuous)

Types of ItemsTypes of Items

4 levels of items4 levels of items

1 Minimal Items: Multiple choice1. Minimal Items: Multiple choice

2. Intermediate Level Items: Content and Explanation ScaffoldsExplanation Scaffolds

3. Intermediate Level Items: Explanation OR Content Scaffolds onlyy

4. Complex Items: No Scaffolds

Type 2: Content & Explanation ScaffoldsType 2: Content & Explanation ScaffoldsTYPE OF ORGANISM WHAT THEY USE FOR ENERGY

small fish water lily large fish small fish, water lily h ll fi h l fi h iheron small fish, large fish, insects

Write a scientific explanation for the following questionScientific Question:  Is the large fish a producer or consumer?

Hint:Think about how producers and consumers get 

energy.

Make a CLAIM:Write a sentence that answers the scientific question 

Give your REASONING:Write the scientific concept or definition that you thought about to make your claim 

Hint:Think about the definition of the scientific term 

Give your EVIDENCE:Look at your data and find two pieces of evidence that help answer the scientific question

you used.

Hint:Think about 

where the largethe scientific question.1.  2. 

where the large fish get its energy.

What Information LP Assessment System ProvideWhat Information LP Assessment System Provideyy

Assessment items & coding rubric based on our Assessment items & coding rubric based on our learning progression allowed us to gather rich evidence of students’ abilities to formulate scientific explanations with and without scaffolding

Using the same scaffolding and coding rubric over multiple years allowed us to trace students’

i progress over time

Student responses that do not “fit”Student responses that do not “fit”Student responses that do not fitStudent responses that do not fit

Student responses that do not “fit”Student responses that do not “fit”pp

ClaimClaim

Student responses that do not “fit”Student responses that do not “fit”Student responses that do not fitStudent responses that do not fit

ReasoningReasoning

Conclusions and Recommendations

ConclusionsImportance of assessment tasks that are sensitive to progress towards an end goal and nuances in middle knowledge

Additional researchMaking sense of responses that do not fit neatly into the LPMaking sense of responses that do not fit neatly into the LP

Continued research on scientific explanation formulation across content areas and grade bands

h d id h f i fTeachers use LPs to decide what formative assessment format to give to individual students

Using Learning Progressions to Using Learning Progressions to Inform Large-Scale Assessmento a ge Sca e ssess e t

Alicia C. Alonzo, Michigan State Universi ty

Teresa Neidorf , American Inst i tutes for Research

Charles W. Anderson, Michigan State Universi ty

CCSSO National Conference on Student Assessment, June 21, 2010

Fundamental Challenge/Question

How might learning progression (LP)-based How might learning progression (LP) based items fit into a large-scale testing apparatus which has been designed to make different which has been designed to make different claims about student achievement?

Challenges Recommendations

C t ti F hContrasting purposes: Policy decisions

Contrasting practices:

For researchers

For test-developersg pTechnical decisions

p

Contrasting Purposes: Policy Decisions

Current large-scale assessment systemsg yClaims about students’ mastery of particular benchmarks for their grade level (often expressed in terms of achievement levels)levels)

LP approachClaims about where student performances fit in a succession of pincreasingly sophisticated ideas and practices (which may span multiple grade levels)

LP levels articulate ideas and practices characteristic of students LP levels articulate ideas and practices characteristic of students at a given level of sophistication (including non-canonical ways of thinking) with respect to a particular “big idea”

Contrasting Practices: Technical Decisions

Item Development

Pilot Testingg

Item Analysis and Evaluation

Design of Operational Assessmentsg p

Scoring and Reporting

Item Development

Current Large-Scale Assessment SystemsCurrent Large-Scale Assessment Systems

LP ApproachLP ApproachAssessment SystemsAssessment Systems

Linear process Iterative process

Framework (LP)Framework o ( )o

Individual items designed Sets of items designed to

Assessment Student Responses Assessment

to make judgments about students’ mastery of particular knowledge/skills

evaluate the sophistication of students’ thinking with respect to the topic of a

i l p g /

Traditional item types and scoring procedures

particular LPLP-specific item types

and/or scoring procedures

Possible LP-Based Item Types

Ordered Multiple Choice (OMC) (Briggs, Alonzo, Wilson, & Schwab, 2006)p ( )Appear similar to typical MC items, but options are mapped directly to levels of a LP

M lti l T F lMultiple True-FalseStudents are presented with a list of objects or properties and are asked to determine whether each of these meets a given criteriong

Constructed Response

Choose & ExplainpStudents are asked to choose from a set of options (such as in a multiple-choice format) and then to explain their choice

Pilot Testing

Pilot testing offers opportunities for data collection g ppfrom relatively large, representative sample of the target population

LP-Specific RequirementsMultiple pilot-testing phases

Pilot test design in which students respond to sets of LP based Pilot-test design in which students respond to sets of LP-based items

LP-based items cannot be randomly distributed through test b kl tbooklets

Larger pilot samples may be required

Item Analysis & Evaluation

Sets LP-based items must be analyzed to inform yevaluation of the items and the LP

Criteria for evaluating items for inclusion in typical large-scale tests are well established but may not be appropriate or adequate for evaluating LP-based itemsitems

Item difficulty

Item discrimination

Frequency distributions

Inter-rater reliability

iff i l i f i i ( )Differential item functioning (DIF)

Design of Operational Assessments

Issues and challenges in incorporating LP-based items into current assessment systems

Content coverage

Item presentation

Computer-adaptive testing (CAT) may be a promising vehicle for incorporating LP-based itemspromising vehicle for incorporating LP based items

Scoring & Reporting

LP-Specific ReportingLP-Specific ReportingIncorporating LP-Based Items into the Main ScaleIncorporating LP-Based Items into the Main ScaleItems into the Main ScaleItems into the Main Scale

Scaling (cut-score establishment) requires

LP-based items may include content not establishment) requires

cross-grade analyses

Different

include content not included in the assessment framework f i d l l

measurement/ reporting models may b i d f

for a given grade levelScore mapping or rescoring may be required

be required for LP-based frameworks

Reliable sco es ma be

Implications for reporting trends

Implications item release Reliable scores may be difficult to achieve

Implications item release policies

Recommendations

For researchers

For test developersp

Recommendations: Researchers

Research to develop well-grounded LPsCurriculum-neutralTargets grade levels typically assessed by large-scale testsIncludes students representative of target populations Includes students representative of target populations (rather than convenience samples)Longitudinal (rather than cross-sectional)

Additi l hAdditional researchTo explore/evaluate alternative measurement modelsTo inform design of LP-based assessmentsgTo explore impact of LP-based test designTo develop CAT administration of LP-based itemsTo inform reporting methodsTo inform reporting methods

Recommendations: Test Developers

Clear instructions and training for item-writers and gitem-scorers

Additional cycles of revision and pilot testing (with particular requirements)

Alternative methods for evaluating item quality

Alternative measurement models for score reporting

For More Information

LeaPS conference & book

http://education.msu.edu/projects/leaps/p // /p j / p /

Amelia W. Gotwals

gotwals@msu.edu

Alicia C. Alonzo

alonzo@msu edualonzo@msu.edu