We Get By With A Little Help From Our Friends:

A Case Study of Collaborative Assessment Development

February 2014

Denis Jarvinen, Ph.D.Strategic Measurement and Evaluation, Inc.

Presentation Will Be Available At:CAPFoundationinc.org

The Requirement

• The State of Ohio implements a teacher evaluation system that requires evidence of student growth within all courses

• Need to measure student achievement in areas not tested by the Ohio testing program

• Science and Social Studies are two critical content areas

What Are The Options?

Purchase off-the-shelf tests that claim to meet district needs

OR

Each district can build their own unique science and social studies assessments

ORMultiple districts can work together

to create a common set of assessments

What Factors Influence the Decision?

Time

Money

Quality

Which Approach Provides the Best Overall Value to the District?

Professional Development Connection to Classroom

What Factors Influence the Decision?

Approach Time Money Quality PD ClassroomUse Off-The-Shelf Tests + ? ? - -

Develop on a District Basis - - - ? ?Collaborate with Other Districts = + ++ + +

Factor

Districts Come Together

Twenty-Two (22) Ohio districts came together to form an assessment development collaborative.

Legal Entity: The CAP Foundation

Purpose of the CAP Assessments

* Provide data to meet Ohio teacher evaluation requirements * Bring teachers into the process to improve assessment literacy * Enhance teacher understanding of the Ohio strands and content

statements for tested subjects * Deliver customized reports to meet district-specific reporting needs

Building the CAP Assessments

Building the CAP Assessments

Building the CAP Assessments

Adding Partners Cleveland State University – Accounting and Granting Credits Strategic Measurement – To Provide Expert Guidance

Establishing Membership

Defining Assessment Content – Test Blueprint

Holding Meetings: 4 Full-Day Meetings Item Writer Training Item Development Item Review (Pre-Field Test) Item Data Review (Following Field Test)

Building the CAP Assessments

Building the Test Blueprint

Establishes the Emphasis Assigned to Each Strand & Content Statement

Set Qualifications for Participants

Survey Participating Districts

Share Results – Collect Feedback

Finalize Emphasis Content Statement-by-Content Statement

Building the CAP Assessments

CAP Social Studies Content Blueprint: Grades 5 to 8

STRAND Grade 5 STRAND Grade 6 STRAND Grade 7 STRAND Grade 8History 26% History 21% History 52% History 51%Geography 33% Geography 33% Geography 18% Geography 17%Government 17% Government 17% Government 16% Government 18%Economics 24% Economics 29% Economics 14% Economics 14%

Topic* Grade 5 Topic* Grade 6 Topic* Grade 7 Topic* Grade 8Historical Thinking and Skills

5%Historical Thinking and Skills

7%Historical Thinking and Skills

5%Historical Thinking and Skills

5%

Early Civilizations 11% Early Civilizations 14% Early Civilizations 9%Colonization to Independence

15%

Heritage 9%Spatial Thinking and Skills

9%Feudalism and Transitions

20% A New Nation 12%

Spatial Thinking and Skills

8% Places and Regions 9% First Global Age 18% Expansion 8%

Places and Regions 9% Human Systems 16%Spatial Thinking and Skills

5%Civil War and Reconstruction

11%

Human Systems 17%Civic Participation and Skills

9% Human Systems 13%Spatial Thinking and Skills

5%

Civic Participation and Skills

8%Roles and Systems of Government

9%Civic Participation and Skills

7% Human Systems 12%

Roles and Systems of Government

9%Decision Making and Skills

9%Roles and Systems of Government

9%Civic Participation and Skills

8%

Decision Making and Skills

6% Scarcity 9%Decision Making and Skills

5%Roles and Systems of Government

11%

Scarcity 5% Markets 7% Scarcity 4%Economic Decision Making and Skills

3%

Production and Consumption

5% Financial Literacy 5% Markets 5%Production and Consumption

3%

Markets 5% Markets 3%Financial Literacy 5% Financial Literacy 3%

Building the CAP Assessments

Building the CAP Assessments

Meeting 1: Item Writer Training

Social Studies (80+ teachers)Science (70+ teachers)

Professional Development – How to Write High-Quality Items Initial Peer Review – Content is King Homework Assignment

Building the CAP Assessments

Building the CAP Assessments

Meeting 2

Item Writer Training – Peer Review & Edits

Building the CAP Assessments

Original Item

What was the cause of settlement in California as part of Manifest Destiny?

A)The discovery of gold.*B)The profitable fur trade.C)Available fertile land to expand the cottonocracy.D)Creation of a transcontinental railroad.

G8 – CS.15: Describe the movement of people, products and ideas that resulted in new patterns of settlement and land use and analyze its impact on the political and economic development of the US.

Building the CAP Assessments

Which MOST influenced the settlement of California in the 1840s and 1850s?

A)the discovery of gold*B)the profitable fur tradeC)the farming of cottonD)a new cross-country railroad

Building the CAP Assessments

Meeting 3: Item Review

SME Editors and Graphic Artists Setup Final Items

Teacher Ratings

Alignment to Content Statement Item Quality Bias Final Edits

Field Test Of All Items

Building the CAP Assessments

Meeting 4: Field Test Item Data Review

SME Prepares Statistical Item Summaries for All Items

Teachers Taught to Interpret Item Statistics

Final Ratings Clear Way for Items to Make Final Pretest or Posttest Forms

Building the CAP Assessments

Multi-District Collaboration Can Be Successful

Teacher Participation is Professional Development

Data Links Assessments to Instruction