Assessing K-12 Students’ Learning Progression of Carbon Cycling Using Different Types of Items 2010 NARST PresentationWritten by: Jing Chen and Charles W. Anderson (Michigan State University), Jinnie Choi, Yong Sang Lee, Karen Draney (University of California, Berkeley)Culturally relevant ecology, learning progressions and environmental literacyLong Term Ecological Research Math Science PartnershipApril 2010Disclaimer: This research is supported by a grant from the National Science Foundation: Targeted Partnership: Culturally relevant ecology, learning progressions and environmental literacy (NSF-0832173). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Assessing K-12 Students’ Learning Progression of Carbon Cycling Using Different Types of Items

Jing ChenCharles W. Anderson

Michigan State University

Jinnie ChoiYong Sang LeeKaren Draney

University of California, Berkeley

PURPO

SE OF TH

IS STUDY

PURPOSE•analyze how well ordered multiple choice (OMC), multiple True or False

(MTF) items differentiate students among achievement levels by comparing students’ responses to these items to their responses to the same problems set in open-ended (OE) formats

• inform the development of OMC and MTF items to measure students’ achievement levels more reliably

OMC and MTF items

• OMC item-- each of the possible answer choices is linked to a particular developmental level of student understanding for the construct being measured. (Briggs, Alonzo, Schwab, & Wilson, 2006).

• MTF item-- a set of true or false questions asks students to judge True or

False.

RESEARCH Q

UESTIO

NS

• How well can OMC and MTF items diagnose students’ achievement levels?

• How well can our mixed types of items assess students’ performances?

• Carbon cycle learning progression framework

• Related studies

LEVELS Explaining Progress variable

LEVELS Naming Progress Variable

4 Linking processes with matter and energy as constraints

4 Scientific statements

3 Changes of Molecules and Energy Forms with Unsuccessful Constraints

3 Scientific words of organic molecules, energy forms, and chemical change

2 Force-dynamic accounts with hidden mechanisms

2.5 Easier scientific words with mixed meanings

2 Hidden mechanism words

1 Macroscopic force-dynamic accounts

1.5 Easier hidden mechanism words

1 Words about actors, enablers, and results

• Test scores based on OMC items have greater validity than test scores based on TMC items, without sacrificing reliability. There a weak to moderate positive correlation between students’ scores on OMC items and their scores on traditional MC items (Briggs, Alonzo, Schwab, & Wilson, 2006).

• Compare to OE items, OMC items appear to provide more precise diagnoses of students’ learning progression levels and to be more valid, eliciting students’ conceptions more similarly to cognitive interviews (Alonzo & Steedle, 2008)

• Inconsistency in students’ responses to items in different formats, students performed differently on multiple-choice and short-answer items (Steedle, 2006)

THEO

RETICAL FRAMEW

ORK

Participants

Assessments

• students in grade 4 to 12

• 1044 assessment (454 pre, 550 post, 181 elementary, 377 middle, 446 high)

• collected during 2008-2009

• 48 items in total

• include 8 OMC, 8 MTF items and their paired OE items in this study (32 in total)

• matter energy transformation in 5 macroscopic events and large scale events.

RESEARCH M

ETHO

D

Data Analysis

• OMC items cross-tabulation Spearman rank-order correlation between OMC and OE levels

• MTF items cross-tabulation point-biserial correlation between MTF and OE responses

• For all items IRT partial credit model; IRT analysis qualitative evaluation by a group of researchers

RESEARCH M

ETHO

D

RESULTS O

F OM

C ITEMS

BREAD ITEM

A loaf of bread was left uncovered for two weeks. Three different kinds of mold grew on it. Assuming that the bread did not dry out, which of the following is a reasonable prediction of the weight of the bread and mold together?

A) The mass has increased, because the mold has grown. (Level 1)

B) The mass remains the same as the mold converts bread into biomass. (level 2)

C) The mass decreases as the growing mold converts bread into energy. (level 2)D) The mass decreases as the mold converts bread into biomass and gases. (level 3)

OE response levels OMC response levels

1 (A) 2 (B,C) 3 (D)

0 1 3 4

1 42 15 2

2 1 24 32

3 0 0 12

RESULTS O

F OM

C ITEMS

Correlation between OMC scores and paired OE item scores

OMC ITEM

PAIRED OE

SAMPLE SIZE

CORRELATION

WTLOS WTLOS_N 306 .272**

WTLOS_E 306 .342**

BODYTE BODYTE_N 138 .313**

BODYTE_E 138 .476**

BREAD BREAD_N 137 .581**

BREAD_E 137 .651**

MATCH MATCH_N 283 .217**

MATCH_E 283 .258**

LEAVES LEAVES_N 306 .300**

LEAVES_E 306 .320**

STOREN STOREN_E 287 .360**

TROP TROP_E 269 .411**

LIGHT LIGHT_E 289 .219**

TOTAL TOTAL_N .654**

TOTAL_E .755**

• There are weak to moderate positive correlations between students’ OMC levels and their OE levels.

• The correlation between overall OMC scores and overall OE naming scores is . 654, and the correlation between overall OMC scores and overall OE explaining scores .755.

** correlation is significant at the .01 level (2-tailed)

RESULTS O

F OM

C ITEMS

Reliability of OMC and OE items

OMC (5 items administrated at high school form C)

OE (5 paired items administrated at high school form C)

Cronbach’s alpha(N of cases =138)

.441 .704

Expected reliability for a longer test (using spearman-brown prophecy formula)

0.703(a test with 15 OMC items)

RESULTS O

F OM

C ITEMS

Item difficulties for OMC items, paired OE items (naming and explaining)

******Test information comparison (one test includes 8 OMC items only, the other included 8 paired OE items only)

• Compare the item difficulties for 8 OMC and 8 paired OE items (both naming and explaining), OMC items are easier than OE naming, which is easier than OE explaining.

• The test that contains 8 OE items will get more information at the higher ability range compare to a test that includes 8 paired OMC items.

RESULTS O

F MTF ITEM

S

• Students’ choices to “sunlight”, “air”, and “plant create their own energy” do not have significant correlation to their levels in the paired OE item. • Students who circled N for water (water0), N for nutrients (nutrients0) are more likely to be at higher levels compare to students who circled Y for these 2 indicators.

PLANT ENERGY: Which of the following are sources of energy for plants? Circle yes or no for each of the following:a). Water Yes / Nob). Light Yes / Noc). Air Yes / Nod). Nutrients in soil Yes / Noe). They make their own energy Yes / NoExplain what you think is energy for plants

MTF ITEMS (INDICATORS)RESU

LTS OF M

TF ITEMS

MTF ITEMS A B C D E F G

TREEG ROWTH SUN WATER AIR SOIL .

HUMAN ENERGY SUN WATER NUTRIENTS FOOD EXER CO2 O2

N N Y N N N

PLANT ENERGY SUN WATER AIR NUTRIENTS OWN ENERGY

N N

HUMAN GROW SUN WATER AIR FOOD .

RUN FOOD WATER AIR ENERGY SLEEP

Y

CARBON SOIL PLANT CO2 INSECT ENERGY

ANIMAL WINTER HEAT WATER+GAS

WASTE OTHER MATERIALS

Y

STONE WARM RUN CLOTH FOOD

Y

• The classical item discrimination index shows the correlation between students’ scores on this item and their total scores. There is a moderate to strong correlation between students’ score on most of the items to their total score. • The red dot items are the OMC items. They generally show lower correlation between students’ scores on the OMC item and their total scores compare to the OE items.

RESULTS O

F ALL ITEMS

RESULTS O

F ALL ITEMS

Average person location for each score

• All the items are fitting well to the model (for each item, the weighted MNSQ

is within the range from .77 to 1.33). • This graph indicate that our assessment items can fairly differentiate students with different abilities among levels.

CON

CLUSIO

NS and IM

PLICATION

SConclusions

• OMC items are generally easier than OE items, OMC items do not measure students at high ability levels as precise as OE items do

• OMC items have weak to moderate correlation with OE items that measure the same content• lower reliability compare to OE items. • The choices in many MTF items did not associate with students’ levels in OE format• The entire assessment differentiate students among levels well

1. Though it’s hard to write OMC options at higher achievement levels without using “science-y” terminologies, we still need to develop options for students at high level.

2. The design of MTF items need to be informed by more research. So the options should be better indicators that actually differentiate students among levels.

3. The number of OMC items in a test should be relatively large in order to get equivalent amount of test information (test reliability) compare to a test with OE items or with mixed types of items. Implications

Thank you for your attention!

a b c d

1 8 5 11 17

2 3 1 65 10

3 0 0 10 1

4 0 0 1 0

WH

Y STUDY TH

IS? • OMC and MTF items can be more easily used in large-scale assessments compare to OE items

•OMC, MTF items are informed by educational research, options in OMC items represent typical students’ understandings, from the least to most scientifically sophisticated accounts, found in science education research.

• OMC and MTF items are more diagnostic compare to traditional MC and T or F items, can be used to provide teachers quick feedbacks.

• however, OMC and MTF items do not give students opportunity to create their own responses

• given the advantages and limitations of the OMC and MTF items, the analysis of how well the OMC and MTF items can diagnose students’ achievement levels can direct the future use of OMC and MTF items.