Gary Weiser Dissertation - Columbia University
Transcript of Gary Weiser Dissertation - Columbia University
DevelopingNGSS-AlignedAssessmentstoMeasureCrosscuttingConceptsinStudent
ReasoningofEarthStructuresandSystems
By
GaryWeiser
Submittedinpartialfulfillmentofthe
requirementsforthedegreeofDoctorofPhilosophy
undertheExecutiveCommitteeoftheGraduateSchoolofArtsandSciences
COLUMBIAUNIVERSITY
2019
©2019
GaryWeiser
Allrightsreserved
Abstract
DevelopingNGSS-AlignedAssessmentstoMeasureCrosscuttingConceptsinStudent
ReasoningofEarthStructuresandSystems
GaryWeiser
Thepasttwodecadesofresearchonhowstudentsdeveloptheirscienceunderstandingsas
they make sense of phenomena that occur in the natural world has culminated in a
movementtoredefinescienceeducationalstandards.Theso-calledNextGenerationScience
Standards(orNGSS)codifythisnewdefinitionintoasetofdistinctperformanceexpectations,
whichoutlinehowstudentsmightrevealtowhatextenttheyhavesufficientunderstanding
ofdisciplinarycoreideas(DCIs),sciencepractices(SEPs),andcrosscuttingconcepts(CCCs).
Thelatterofthesethreedimensionsisuniquebothinbeingthemostrecenttothefieldand
inbeingtheleastsupportedbypriorscienceeducationresearch.Morecrucially,asapolicy
document,theNGSSalonedoesnotprovidethesupportsteachersneedtobringreformsto
theirclassrooms,particularlynotsummativeassessments.Thisdissertationaddressesboth
ofthesegapsusingacombinationofquantitativeandqualitativetechniques.First,Ianalyze
differentialcategorizationofproblemsthatrequirerespondentstoengagewiththeirCCC
understandings via confirmatory factor analysis inference. Second, I use a set of Rasch
modelstomeasurepreliminarylearningprogressionsforCCCsevidentinstudentactivity
withinacomputer-assistedassessmentexperience.Third,Ianalyzestudentartifacts,think-
aloud interviews, and post-task reflective interviews via activity theory to adapt the
progression into a task model in which students explain and predict aspects of Earth
systems.Theculminationof these threeendeavorsnotonlysets forthamethodology for
researchingCCCsinawaythatismoreintegrativetotheotherdimensionsoftheNGSS,but
alsoprovidesaframeworkfordevelopingassessmentsthatarealignedtothegoalsofthese
newstandards.
i
TABLEOFCONTENTS
TABLEOFFIGURES,GRAPHS,ANDTABLES iii
ACKNOWLEDGEMENTS iv
CHAPTER1 1
INTRODUCTION 1RESEARCHQUESTIONS 5
CHAPTER2 7
LITERATUREREVIEW 7DIMENSIONSOFTHENGSS 7ANCHORINGLEARNINGANDASSESSMENTCONTEXTS 14ELEMENTSOFRIGORINTHEDEVELOPMENTOFASSESSMENTS 19DESIGNINGTHEDAT-CROSSASSESSMENTSUITE 23DEVELOPINGCOGNITIVELABPROTOCOLS 28FRAMEWORKFORANALYSIS 31
CHAPTER3 37
METHODOLOGY 37DATACOLLECTION 38DATAANALYSISMETHODOLOGY 41ETHICSANDREFLEXIVITY 46
CHAPTER4 47
RESULTS 47SUMMARYSTATISTICS 47RESEARCHQUESTION1 48RESEARCHQUESTION2 51RESEARCHQUESTION3 55
CHAPTER5 62
FINDINGSANDDISCUSSION 62RESEARCHQUESTION1 62RESEARCHQUESTION2 64
ii
RESEARCHQUESTION3 71MAKINGAMENDMENTSTOTHEECOSYSTEMTASKSTORYBOARD 73IMPLICATIONSFORTHEDESIGNOFTASKSINNEWSYSTEMCONTEXTS 74LIMITATIONSTOMYFINDINGS 75
CHAPTER6 77
CONCLUSION 77
REFERENCES 79
APPENDICES 93
APPENDIXA:TASKMODELFORASSESSMENTOFECOLOGICALSYSTEMS,STRUCTURES,ANDFUNCTIONS–ADAPTEDFROMMS-LS2-3,MS-LS2-4,&MS-LS2-5 93APPENDIXB:HYPOTHESIZEDLEARNINGPROGRESSIONS 105APPENDIXC:ECOSYSTEMTASKITEMSWITHDESIGNRATIONALES 114APPENDIXD:POST-TASKINTERVIEWPROTOCOL 128APPENDIXE:DAT-CROSS:BACKGROUNDQUESTIONNAIRE 130APPENDIXF:DATACOLLECTIONMATRIX 132APPENDIXG:ETICCODINGSCHEMEFORINTERVIEWDATA 133APPENDIXH:RUBRICFORASSESSINGCONSTRUCTEDRESPONSES 134APPENDIXI:RSCRIPTFORSTRUCTURALEQUATIONANDRASCHMODEL 146APPENDIXJ:DEFINITIONSOFSOMEKEYTERMS 148
iii
TableofFigures,Graphs,andTables
Figure2.1FundamentalsoftheECDValidityArgument.AdaptedfromMislevyetal.(2017)........................................................................................................................................................................................21 Figure2.2DiagramofEcologicalSystemsTaskModelDevelopment............................................27 Figure2.3Question2ofDAT-CROSSwithItemDesignRationale,below....................................29 Figure2.4ExampleofaWrightMap.AdaptedfromGotwals&Songer(2013).........................33 Figure2.5ExampleofEngestrom’s(2000)ActivityTheoryModelinStudentWork.............34 Figure3.1Question5ofDAT-CROSSEcosystemStoryline.................................................................40 Figure3.2ExampleofActivityTheoryModelatPlayinClassroomWork..................................44 Figure4.1HypothesizedFactorStructureofDAT-CROSSEcosystemAssessment..................50 Figure4.2ResultingCorrelationsfromCFAAnalysis............................................................................51 Figure4.3Person-Item(Wright)MapofStructureFunctionItems...............................................52 Figure4.4Person-Item(Wright)MapofSystemsandSystemModelItems...............................54 Figure4.5DiagramofActivityDisruptionWhenStudentInteractswithModelingTool.......58 Figure4.6DiagramofActivityDisruptionDuringStudentInteractionwithSimulation.......59 Figure4.7DiagramofActivityDisruptionDuringStudentConstructedResponses................61 Figure5.1DAT-CROSSQuestion5..................................................................................................................65 Figure5.2Usability32'sresponsetoDAT-CROSSQuestion5..........................................................66 Figure5.3:Usability41'sresponsetoDAT-CROSSQuestion5..........................................................67 Figure5.4Usability7'sresponsetoDAT-CROSSQuestion5.............................................................68 Figure5.5DAT-CROSSQuestion10..............................................................................................................69 Figure5.6ExampleQuestionfromWaterUseSystemStoryline......................................................75
Table3.1Inter-raterReliabilityRatingsforConstructedResponses............................................41 Table4.1AveragePerformanceonAssessmentItemsbySubgroup.............................................47 Table4.2FitStatisticsofStructureandFunctionItems.....................................................................53 Table4.3FitStatisticsofSystemsandSystemModelItems..............................................................55
iv
Acknowledgements
MyexceedingthanksgoouttoProf.EmdinandProf.Andersonfortheircontinuingexpertiseandguidancealongmyjourneytothedissertation.AdditionalthanksareduetoProf.RivetfortheinitialpushthatsetmyworkinmotionandtoProf.Mensahforseeingitattheend.ThisworkwouldnothavebeenpossiblewithoutthementorshipandfriendshipofLeiLiuandtherestoftheDAT-CROSSteam.Iextendtothemsincereappreciation.Lastly,Iwanttothankmyfamilyandfriendswhosecontinueddedicationmadethisworkpossible.The project described henceforth was partially supported by the U.S. Department ofEducation. Institute of Education Sciences, National Center for Education Research,throughgrantnumberR305A170456.ThecontentissolelytheresponsibilityoftheauthoranddoesnotnecessarilyrepresenttheofficialviewsoftheInstituteofEducationSciences.
1
Chapter1
INTRODUCTION
TheNextGenerationScienceStandards(NGSS)takeanovelapproachtothewaysthe
science education community thinks about what counts as acceptable expressions of
students’ science understandings (Krajcik & Merritt, 2012). Previous reform documents
suchastheAmericanAssociationfortheAdvancementofScience’sBenchmarksforScience
Literacy(henceforthAAAS;AAAS,1993)describedexpectationsintermsofwhatstudents
shouldknowratherthanwhatstudentscando(Duschl,Schweingruber,&Shouse,2007).The
NGSS reforms, however, describe expectations for student understanding in terms of a
performance,somethingstudentsneedtodoasawayofexpressingcompetenciesinuseof
relevant disciplinary core ideas, science and engineering practices, and crosscutting
concepts(NationalResearchCouncil,2012).Thoughthesestandardsarenot, themselves,
assessments,theyplayanessentialroleininformingassessmentdesign(DeBarger,Penuel,
Harris,&Kennedy,2015)byhelpingtoconstrainpossibleperspectivesontheassessment
triangle: Cognition, Observation, and Interpretation (Pellegrino, Chudowsky, & Glaser,
2001).Thethree-dimensionalviewofscienceunderstanding(Duschletal.,2007)whichfuse
disciplinary core ideas (DCIs), crosscutting concepts (CCCs), and science-engineering
practices(SEPs)detailedintheNationalResearchCouncil’s(henceforthreferredtoasthe
NRC) A Framework for K-12 Understanding (NRC, 2012) (henceforth referred to as
Framework) sets expectations for how the community should think aboutwhat students
know (the cognition corner of the assessment triangle model; Pellegrino et al., 2001).
Similarly,theexpectations,themselves,makesalientwhatstudentunderstandingofscience
should look like (the observation corner of the triangle model; Pellegrino et al., 2001).
2
However, where assessment observations should be found, and how to interpret
observationsmade(Pellegrinoetal.,2001),remainunderspecifiedinreformliterature.
Uniquely underspecified in existing literature are supports for eliciting and
interpreting observations of the crosscutting concepts (CCCs), which has left some
questioningwhythedimensionappearsatall(Osborne,Rafanelli,&Kind,2018).Whilethe
subtlety of CCCs is undeniable, they have immense power in shaping how students use
languagetoexplainphenomena(Weiser,Lyu,&Rojas-Perilla,2017)and,byextension,how
assessorsvaluetheresponsesstudentsprovide.Nonetheless,ifmyphilosophicalpositionis
that student reasoning of the crosscutting concepts adds value to their science
understandings in a way that the other dimensions do not, then failing to capture that
reasoning is failing to capture the full richness of their performances. It is, therefore, a
continuingchargeplacedontheNGSS-supportingscienceeducationcommunitytodesign
assessmentsthataligntoallpartsofthenewstandards.Inthefollowingsections,Idescribe
aframeworkthattheevaluationcommunitymightusetodesignassessmentsfortheNGSS,
fillinginthegapsontheassessmenttrianglealongtheway.
Thisresearchstudyto followsaparadigmknownasdesign-basedresearch,which
emergedfromthelearningsciencesduringthelate1990s(Barab&Squire,2004).Unlike
clinicalinterventionresearch,design-basedresearchmakesatradeoff;i.e.,sacrificingtypical
treatmentcontrolsinfavorofdatathatcanonlybecollectedinthebustleoftherealworld,
butnolessinasystematicandevidence-basedway(Hoadley,2004).DiSessaetal.(2004)
describethegoalsoftheseendeavorsas‘ontologicalinnovation’,inwhichtheresearchadds
somethingnewtoanimportanttheorythatcouldonlyhavebeenaddedwithdatacollected
fromthenonclinicalsetting.Partofthisgoalof innovation(asDiSessaandhiscolleagues
3
describeit)isanewformofvaliditydubbedconsequentialvaliditybyHoadley(2004)that
definesthequalitybothoftheoryandresearchintermsoftheirabilitytosolveaproblem
that actually exists in the real world. Simply put, a good educational theory ought to
successfullysupportgooddesignprinciplesthatinturnshouldsuccessfullysupportagood
educational product or practice. The framers of the Next Generation Science Standards
certainlyhaveanabundanceofgoodeducationaltheory(setforthexceptionallybyDuschl,
2008) and evidence-centered design is a growing set of design principles that are often
describedintermsoftheiralignmenttothesenewstandards(DeBargeretal.,2016).The
timenowcomestoengagewiththeseprinciplesinordertoproduceneededassessmentand,
insodoing,innovateon(asPellegrino,Chudnowsky,andGlaserputit)howbesttoknow
thatstudentsknow(Pellegrinoetal.,2001).
The validity argument that foregrounds the evidence-centered design framework
hingesontheabilityoftheassessmenttosuccessfullysupportaparticularclaimregarding
whatstudentsknow(Mislevyetal.,2017).However,beforeevidencebackingaclaimcanbe
elicited,designersfirstneedtodefinewhatclaimsareusefulforrelevantstakeholders;such
asteachers,students,administrators,andpolicymakers(Debargeretal.,2016).Whilethe
broadstrokespresentedintheNextGenerationScienceStandards(NGSS;NGSSLeadStates,
2013)maybesufficient foradministrators lookingtomakeoverarchingclaimsaboutthe
science proficiency of students in their local purview (Pellegrino, 2013), teachers and
students requiremore detailed information about their present understandings that are
capableofbeingmappedtoalearningtrajectorythatendsatsomeproficiencygoal(Harris
etal.,2016). In ideal cases,prior literatureprovidesempirically-backedmodels forwhat
claimsoughttobemadeatcertaingradebands(Duncan&Rivet,2013).Forcross-cutting
4
concepts(CCCs),however,thelackofresearchintowhatstudentsknowastheybuildoverall
sciencecompetencymakesthedeterminationofappropriateclaimsdifficult.Forthisreason,
theprimaryresearchquestionofmydissertationis:Whatdoesstudentdemonstrationsofthe
crosscuttingconceptual reasoningaspectof their three-dimensional scienceunderstandings
looklikeatdifferentlevelsofoverallscienceunderstanding?
Toaddressthisbroadquestion,aprojecttitled:DevelopingAssessmentsandToolsto
Support the Teaching and Learning of Science Crosscutting Concepts (DAT-CROSS) was
created in collaboration between the Educational Testing Service (ETS) and Indiana
University.Theprojectwassupportedbyagrant(R305A170456)fromtheNationalCouncil
for Education Research (a division of the U.S. Department of Education’s Institute of
EducationSciences)todevelopanempirically-validatedprogressionforCCCunderstanding
(particularly regarding theCCC’sSystemsand SystemModels andStructure andFunction;
NGSSLeadStates,2013c).BuildingonmypreviousworkingrelationshipswithseveralETS
researchers,aswellasoneofmypriordevelopedexpertiseonCrosscuttingconcepts(Rivet
et al., 2016;Weiser et al., 2017), Iwas invited to design an assessment suite capable of
meeting the learning progression goals of the project. As with many IES-funded grant
projects,theDAT-CROSSendeavorisdesignedtocontinueacrossseveralyears,culminating
inmaterials that canbe readilyprovided to teachers to formativelyassess students’CCC
understandings.Thedataformyresearchquestionderivedfromthefirsttwoyearsofthe
project,inwhichtheteaminvestigatedthe‘usability’(Zaharias&Poylymenakou,2009)of
the assessment in order to validate the chosen tasks (Johnstone, Bottsford-Miller, &
Thompson,2006)andtominimizetheroleofnon-focalskills(suchasELAproficiencyor
5
computer-usecompetency)inmediatingstudents’abilitytoengagewiththetasks(Hoadley,
2004).
IntheprocessofworkingontheDAT-CROSSproject,itbecameclearthatpartofthe
challengeinthedesignofassessmentsforcrosscuttingconceptswasthelackofclarityover
howtheconceptsweredistinctfromoneanother.Asrecentlyas2018,researchersinscience
educationhaveexpresseddoubtsovertheutilityofevenattemptingtomeasurecrosscutting
conceptsin-use(Osborne,Rafanelli,&Kind,2018).Answeringmyinitialresearchquestion
entailed, first, answering a brand-new question: Is there evidence that the crosscutting
concepts of the NGSS are distinct constructs that can be measured as students use them?
Simultaneously, the goal of design research is not merely to confirm that the designed
product works as intended. The goal of design research, particularly when student
assessment is involved, is to identify all the impediments tomaking a locally functional
solutionwhilekeepingtoexistingdesignprinciples(Joseph,2004).Thisgoalmanifesteda
thirdresearchquestion,describedbelow,thatfocusesonthechallengesfacedbymysubjects
astheyinteractedwithassessmenttaskelements.
ResearchQuestions
Therearethreeresearchquestionsforthisstudy:
1. IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthat
can be measured as students use them? Answers to this question undergird the
possibilityofameasurementmodel.
2. Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectof
their three-dimensional scienceunderstandings look likeatdifferent levelsofoverall
6
science understanding?Answers to this questionwill inform the development the
evidencemodel.
3. What challenges do middle-school-aged subjects face in presenting their CCC
understandingswhileengagingwithan interactivesuitedesignedtotargetSystems-
Models and Structure-Function dimensions of their three-dimensional science
understandings?WhileCCCsrepresentanimportantconstituentofstudents’science
reasoning,thedimensionisoftenthesubtlestconstituentofagivenperformance.As
students face challenges in the task, determining that their barriers stem from
underdevelopedCCCunderstanding iscritical todeterminingtheusefulnessof the
assessment.
7
Chapter2
LITERATUREREVIEW
DimensionsoftheNGSS
The first step in understanding the rationale for this thesis study comes from
unpackingthemajordimensionsoftheNextGenerationScienceStandards(NGSS).
PerformanceExpectations:ThestructureoftheNGSS.
TheNextGenerationScienceStandardsarecomposedofperformance-based tasks
whichcombineall threedimensionsofsuccessfulscienceeducation: a)disciplinarycore
ideasthatmakeuptherelevantsciencecontent(e.g.,therelationshipbetweenenergyand
forces),b)scientificpracticesthattieinstructiontoactivityofthescientificcommunity(e.g.,
developingandusingmodels),andc)crosscuttingconceptsthatreflectthecommonalities
ofthetaskandmainideastoallsciencedisciplines(e.g.,causeandeffect)(Krajcik&Merritt,
2012;Krajciketal.,2014;NRC,2012).Thesynthesisofthesethreedimensionsatarelevant
grade-bandresultsinaperformanceexpectation(orPE)statedlikethis:“HS-PS3-5.Students
whodemonstrateunderstandingcan:developanduseamodelof twoobjects interacting
throughelectricormagneticfieldstoillustratetheforcesbetweenobjectsandthechanges
inenergyoftheobjectsduetotheinteraction.”(NGSSLeadStates,2013a).Thisreflectsa
radically altered view of the nature of science, and its relation to science education,
comparedtothatoftheNationalScienceEducationStandards(NationalResearchCouncil,
1996)andtheBenchmarksforScientificLiteracy(AAAS,1993).Bothfocusedondeveloping
inquiry practices and on replacing students’ alternative conceptions with those AAAS
consideredmoreaccurate (Kesidou&Roseman,2002).Whereearliernational standards
8
suggested that students have sufficient knowledgewhen they “know” particular science
content,theNGSSstatesthatstudentsrevealtheirknowledgeby“doing”ataskthatrequires
useofthatknowledge(Krajciketal.,2014).
DisciplinaryCoreIdeas
Disciplinarycoreideas(DCIs)arethedomain-specificcontentofscienceknowledge
writ-large,best exemplifiedby the informationcontainedwithin textbooksor flashcards.
IdeaslikeNewton’slawsortheprocessofnaturalselectionareexamplesofthisdimension
that represent key building blocks to modern scientific understandings (Krajcik, 2015).
These building blocks have been featured in science standards for about as long as the
conceptofeducationalstandardshaveexisted(DeBoer,1991).Unsurprisingly,thehistoric
salienceofthecontent,nowcategorizedasDCI,toooftendominatesintheclassroomatthe
expense of the other dimensions of the NGSS (Stroupe, 2015). Successfully planning
instructionaimedtowardsachieving thegoalsof theNGSSentailsacarefulbalancingact
betweenhelpingstudentsdevelopsophisticationaroundthesecoreideasandhelpingthem
understandthewaysthesecoreideasareactuallyusedbyscientists(Krajciketal.,2014).
Withoutadditionalsupportsforteachers,particularlyinassessingstudentunderstandings
oftheothertwodimensionsoftheNGSS,itissuspectedthatinstructorswilldefaulttoafocus
onDCIs(Pellegrinoetal.,2014).
ScienceandEngineeringPractices
Much of the recent literature on science learning that emerged from the learning
science fieldhascenteredonhowknowledgegetsused(Brown,Collins,&Duguid,1989;
CognitionandTechnologyGroupatVanderbiltUniversity,1992;Edelson,2001;Krajcik&
Merritt,2012;Nersessian,2002;Stroupe,2015).Thoughdescribedinearlierstandardsin
9
terms of scientific inquiry (DeBoer, 1991; Edelson, 2001), the various means by which
scientists(andbyextensionsciencestudents)engagewiththeirknowledgeandbuildnew
understandingarefeaturedastheseconddimensionoftheNGSS:ScienceandEngineering
Practices (SEPs) (Krajcik & Merritt, 2012). There are eight such practices: a) asking
questionsanddefiningproblems;b)developingandusingmodels;c)planningandcarrying
out investigations; d) analyzing and interpreting data; e) using mathematical and
computationalthinking;f)constructingexplanationsanddesigningsolutions;g)engagingin
argument from evidence; and h) obtaining, evaluating and communicating information
(Osborne,2014).Whileeachofthesepracticeshaveappearedinalitanyofpasteducational
standards(Osbourne,2014),priorresearchhasbeenunevenonwhatthosepracticeslook
like both at the expert and novice level (Stroupe, 2015). Overwhelmingly, research has
centeredonthedevelopmentofsciencemodels(Pluta,Chinn,&Duncan,2011;Schwarzet
al., 2009; Schwarz &White, 2005; Svoboda & Passmore, 2011) and the construction of
scientificarguments(Berland&Reiser,2009;Osborneetal.,2016;Sampson&Clark,2009).
Whilethisfocusisareasonablefunctionofthenatureofthescientificenterprise(McComas,
Clough, & Almazroa, 2002), it means that there is a dearth of empirically-validated
instructionalapproachestohelpingstudentsdeveloppracticalskillsalongmanyoftheother
SEPs.
CrosscuttingConcepts.
Crosscuttingconceptsareacomparativelynewadditiontothecorecomponentsof
sciencestandards(firstappearingaspartof the“unifyingconceptsandprocesses” in the
NationalScienceEducationStandardsproducedbytheNRCin1996[p.104])andrepresent
themes that underlay the scientific enterprise (Rivet et al., 2016) in all domains and
10
disciplines.Theseseventhemesare:a)patterns;b)causeandeffect;c)scale,proportionand
quantity;d)systemsandsystemsmodels;e)energyandmatter;f)structureandfunction;
andg)stabilityandchange(NGSSLeadStates,2013b).TheNGSSpridesitselfonitsempirical
backing,anditsframersroutinelyclaimthatthescopeandsequenceencouragedbytheNGSS
anditssupportingdocumentswasderivedfromevidencearoundhowstudentslearnscience
concepts(Duschl,2008;Duschletal.,2007;NRC,2012).Whilethisclaimismostlytruefor
disciplinarycoreideas,andpartlytrueofscienceandengineeringpractices,thereisawell-
establishedlackofevidenceforcrosscuttingconcepts(Rivetetal.,2016).Forthisreason,
somecriticshaveclaimedthatCCCsmakeapooradditiontothenewstandards(Osborneet
al.,2018),butIdisagreewiththeassessmentthatthecurrentabsenceofevidenceforCCCs
instudentworkisevidencethatthedimensionisnotequallyimportantinstudentlearning
(Weiseretal.,2017).
As in the case of SEPs, what prior research does exist regarding student
understandingoftheconceptsatplayfortheCCCdimensionoftheNGSSislimitedtojusta
fewoftheenumeratedthemes.Whilemuchhasbeenwrittenaboutmatterandenergy(Jin
&Anderson,2012;Neumann,Viering,Boone,&Fischer,2013;Stevens,Delgado,&Krajcik,
2009), this research generally treats such concepts as bounded by the uses that are
particulartoasciencedomain(akintoDCIs).Oneofthefewinstancesinwhichresearchinto
CCCs seems to consider the researched construct as a concept that might apply across
contexts iswith regards to students’useof systemsandsystemsmodels.Examining this
literatureacrossbiological,physical,andearthsciences(Breslynetal.,2016;Gunckeletal.,
2012;Jin&Anderson,2012:Mohanetal.,2009;Songeretal.,2009),Ifoundfourgeneral
pathwaysbywhichsophisticationinsystemsthinkingbuildsovertimeasfollows.
11
1. System Phenomena – As students build sophistication in their thinking around
systems, they are better able to describe phenomena as a system of many
simultaneousinteractions.
2. System Components – As students build sophistication in their thinking around
systems,theyarebetterabletobreakdownthecomponentsofasystemintotheir
constituent,dynamicparts.
3. System Relationships – As students build sophistication in their thinking around
systems, they are better able to describe the relationships between previously
identifiedcomponentsofthesystem.
4. System Boundaries – As students build sophistication in their thinking around
systems,theyarebetterabletodefinetheboundariesofthesystemandtrackthe
flowsofinputsandoutputsacrossthoseboundaries.
Thesepathwayswilllaterserveaskeyindicatorsofprogressusedinthedevelopmentofthe
tasksthatmakeupmyresearchendeavor(seeAppendicesAandB).
GiventhephilosophicandepistemicnatureoftheCCCs,Ibelieveitismorelikelythat
appropriateinstrumentsforfindingCCCsstillneedtobedeveloped(Rivetetal.,2016).In
previous research, my team found a set of four roles that students expressed in their
understandingof CCCswhen constructing scientific explanations (Weiser et al.,2017) as
listedbelow.
1. CCCsasLenses–TheroleoftheCCCistohighlightsalientfeaturesthatmaynotbe
immediatelyobviousduetoscale,scope,orsize.
2. CCCsasBridges–TheroleoftheCCCistodrawconnectionsbetweentwoentities,
facilitatingtransferofunderstanding.
12
3. CCCsasLevers–TheroleoftheCCCistocombineseveral,relatedideas,entities,or
representations in order to make understanding take on a new form towards a
particulargoal.
4. CCCs as Rules – The role of the CCC is to validate a representation’s utility in
explainingorpredictingthenaturalworld.
Whicheverrolestudentsemploy, itradicallyalterstherelationshipbetweentheevidence
usedintheexplanationandthesubject-phenomenarelationship.BythinkingaboutCCCsin
termsoftheseroles,IbelieveIcanconstructassessmentsthatarebothbetterabletoelicit
evidence of CCCs, and better contextualize that evidence in terms of students’ three-
dimensionalscienceunderstandings.
LearningProgressions
OneoftheadvertisedfeaturesoftheNGSSistheirbasisinempiricallyderived(Duschl
etal.,2007)researchintohowstudentslearn.ExemplifiedintheworkofGotwalsandSonger
(2013),learningprogressions(LPs)arethedominantframeworkforidentifyingwhatthat
empirical evidence looks like (Corcoran,Mosher, & Rogat, 2009).While discussions that
appear in Berland and McNeill’s (2010) work paint a very rosy picture of the positive
implicationsoftheLPframework,thereisalsomuchdiscussionfocusingonthenotionofa
“messy middle” (Gotwals & Songer, 2013, p. 599) where the pathways between well-
established stepping-stone ideas are less clear. The dominant perspective on learning
progressions and learning science research (Lehrer et al., 2001; Rivet & Krajcik, 2008)
suggestthatprogressionsareanchoredbythecontextsofaparticularphenomenon,making
ithardtogeneralizeonelearningprogression(forexample,onmoonphases[Plummer&
Krajcik,2010])toanotheraboutadifferenttopic,suchasNewton’slaws(Alonzo&Steedle,
13
2009).Someexistingprogressionsattempttobeapplicableacrossdisciplinarycontextsby
focusingonsciencepractices,asinthecaseofBerlandandMcNeill(2010);oronconcepts
thatappearsinallsciencedomains,asinthecaseofNeumannetal.(2013).However,such
progressions are few and far between (Duschl et al., 2011). LPs continue to be a useful
paradigm for research and theory, but incorporation into instructional practice requires
greaterelucidation.
ContemporaryCritiques
No reformmovement iswithout its constructive critics, andwhile therehasbeen
longstanding agreement that past standards have failed to live up to their lofty goals
(Eisenhart,Finkel,&Marion,1996;Lee,1997),thebeststepforwardalwaysremainsatopic
ofdebate.FortheNGSS,anearlycritiqueofitsemphaseswasthediminishedfocusonthe
nature of science,whichwas initially absent from the standards until its inclusion as an
addendum document (Appendix H of the NGSS [NGSS Lead States, 2013]). For science
educatorswhoarguedthatnatureofscienceisacriticalcontentelementthatneedstobe
explicitly discussed in classrooms (Abd-El-Khalick, Bell, & Lederman, 1998; Lederman&
Zeidler,1987),relegatingnatureofsciencecontenttoanappendixthatonlyappearsonsome
performanceexpectationswouldleadteacherstoavoidsuchtopicswhenweighedagainst
their other instructional responsibilities (Lederman& Lederman, 2014). Coupled to this
discussionwerecritiquesoverthepresenceofnewdimensionsandtheunevendegreeto
which existing learning progressions could describe how these dimensions grew in
sophisticationasstudentsmature(Duschl,Maeng,Sezen,2011).Returningtothenotionof
consequentialvalidity,thetruetestofthetheoriesundergirdingtheNGSSwillbethedegree
towhich instructional-materialdevelopers(includingassessmentdesigners)canproduce
14
thetoolscapableofhelpingstudentsreachthesupposedgoalsofthenewstandards(Furtak,
2017).
Concernsof equity in science learninghave alsobeena locusof critical reviewof
educational standards reform. As researchers like Lee (1997) and Brown (2005)
investigatedduringtherolloutand,later,enactmentofTheBenchmarksforScienceLiteracy
(AAAS,1993),thedefinitionsfortermslike“scienceliteracy”(Lee,2005)becomedefinitions
forwhohasaccesstoeffectresources.AsstatesbegintheprocessofimplementingtheNGSS,
anewgenerationofresearcherslikeMutegi(2011)andElam-Respass(2018)arewondering
hownewstandardswillaffecthistoricallyunder-representedteachersandstudents.Equity
intheNGSS,asasetofstandardsdrivenbystudentdoingoverstudentknowing,isuniquely
tiedtoassessmentequity(Rodriguez,2015).Nowmorethanever,alignmenttoboththetext
and the goals of the NGSS requires the design of assessments responsive to students’
linguisticskillsandsocially-mediatedcognitions(Mislevy,2016).
AnchoringLearningandAssessmentContexts
Theconceptoflearningascognitiveapprenticeship,developedbyBrown,Collins,and
Duguid(1989),tellsusthatlearningrepresentsamaster-novicerelationshipbetweenthe
teacherand thestudent inwhich thestudentbecomesenculturated inauthenticpractice
throughthedoingofrelevantactivity,muchastheapprenticeofoldpracticedhiscraftunder
thetutelageofamastercraftsman.Tothisend,theyarguethatalllearningissituatedinthe
contextinwhichitwastaught,withthiscontextbeingkeytothelearner’sabilitytotransfer
conceptsfromthelearningenvironmenttonewscenarios(J.Brownetal.,1989).Criticalto
developingmasteryoftransferis“authenticity.”Authenticityreflectsthedegreetowhicha
15
cognitive tool that develops during learning in the classroom context is, later, usable to
studentsasexpertpractitionersmightwieldit.TouseanexampledevelopedbyBrownetal.
(1989),ifsomeoneusesahammerasapaperweight,itisavaliduseofthetool;but,itisnot
an authentic use. In contrast, developing the use of a hammer in context of building a
birdhouserepresentsauthenticpracticeofusingthetoolasacarpenterorcraftsmanwould
useit.Theimportantdifferenceisthatourearlierexamplewouldnotbeextendabletomany
otherinstancesinwhichahammermightbeuseful,whileourlatterexampleprovidesuseful
knowledgeaboutthekindsofscenariosinwhichauthenticuseofahammermaybecalled
for.Inthemostfundamentalsense,everyinstanceinwhichatoolmaybeusedauthentically
hassomecommonalitywitheveryotherpossibleinstanceofauthenticuse.Contextualizing
instructionintermsofthisauthenticusehelpsthenovicelearnerrecognizecommonalities,
whichassistsinbothintegration(Rivet&Krajcik,2008)andtransfer(Yilmaz,Eryilmaz,&
Geban,2006)ofnewlearning.
However,cognitiveapprenticeship ismoreadescriptionofwhat itmeansto learn
thananexplicitmeansofevaluatingthequalityofagivenlearningenvironment.Thereis
somesense inwhichall learningenvironments are instancesof cognitiveapprenticeship
(Brown et al., 1989) and, moreover, such instances are not inherently sufficient for
developing a successful learning environment. Extending their theory on cognitive
apprenticeship, Lave andWenger (1991) presented amore complex analysis of learning
environments; i.e., legitimateperipheralparticipation(LPP),whichexamineslearningnot
onlyasapprenticeship,butasameansofmovingperipheralmembersofacommunityof
practiceintofullparticipants.Legitimateperipheralparticipationrepresentsananalytical
perspective for evaluating cognitive apprenticeship environments for their ability to
16
producedesirablechangesnotjustinnovices’abilities,butintheirrolewithinacommunity
ofpracticeandintheirself-identityinrelationtothatcommunity.Thisisnosmalltask,even
a well-established apprenticeship can fail if the novices do not perceive themselves as
movingfromperipheralparticipationtocentralparticipation(Lave&Wenger,1991).
IntheensuingdecadesofresearchfollowingthatofLaveandWenger,keyideasfrom
the frames of cognitive apprenticeship and legitimate peripheral participation have
continuedtoproveuseful.Examiningtheroleofscienceidentity(asinfluencedbypeerand
by teacher evaluation of language use), Brown et al. (2005) reiterate the importance of
identityshift(Lave&Wenger,1991).Understandinguseoftheestablishedtechnologyofa
community of practice is needed in all learning environments, and LPP presents such
understandingthroughtheconceptoftransparency.Transparencyrepresentsaconjunction
oftheabilitytorecognizetheuseofaparticulartoolandthescenariosinwhichithasutility
(visibility)andtheabilitytousethattoolinordertoachieveadesiredend(invisibility).Lave
andWenger(1991)presenttheexampleofawindow,whichoneusestolookoutsidefrom
theinsideofaconfinedspace(itisinvisible);butsimultaneously,wemustrecognizeitas
somethingwecanlookoutofinordertodoso(ithasvisibility).Atoolmustbetransparent
tothelearnerforthemtoeffectivelyuseitinbothperipheralandcentraltasks.Themedical
school studentmust be able to use a stethoscope both in their novice, peripheral stages
working with dummies and in their central role on actual humans once they become
residents.Failurestoseeacognitivetoolbothinitspresent,noviceuse,andhowexperts
similarlyuseitcanoftenhavedisastrousresultsforstudents’viewsontheutilityoftheir
scienceknowledge(Sandoval,2005).Thisidearemergedincritiqueofthen-presentscience
standards by Eisenhart, Finkel, and Marion (1996) who highlighted the importance of
17
couchingwhat is taught in the light of activities students find relevant to them and, by
extension,arebetterabletoseethedesiredend(s)ofapieceofscienceknowledge.Authority
also matters in terms of classroom discourse. Legitimate learning sets itself apart from
traditional instruction inwhich themaster will talk about a practice, opting instead for
encouragingperipheralmemberstotalkwithintheestablishedrulesofthecommunity(Lave
&Wenger,1991).Ford(2008)reiteratesthisideainsuggestingthatateacher’sauthority
shouldextendfromtheirabilitytohelpstudentsengagewiththescientificdiscipline(and
itsrules)ratherthanfromtheirpositionwithintheschoolhierarchy.
RichPerformanceTasks
For assessment development, all this researchhas served to show that the things
students have learned are much more contextually dependent than might have been
suspectedbypriorreformmovements(whichfocusedmoreonwhatscientistsknowthan
whatstudentknow;Eisenhartetal.,1996).Consequently,assessmenttasksthatmeasure
learningoughttomirrorthecomplexityofthelearningprocessbyacceptingawidearrayof
potentialperformancesthatallsuccessfullymeetthecriteriaofatask(Mislevy,2017).Such
tasks are known as ‘rich performance tasks’ and are well aligned to the philosophical
underpinnings of the NGSS (Gorin & Mislevy, 2013). Unfortunately, they are also more
resource intensive (in capital, cognitive load, and time) than more traditional tasks.
Determininganddesigning learningtoachievetheappropriatebalancebetweenrichness
anddomainrangeremainsatopicofcontinuedresearch(Weiser&Liu,2018).
Socio-scientific issues, cases where there is a salient intersection between the
understandingsgeneratedbyscientistsandongoinghumanbehavior(Kitcher,2010),are
productive contexts for creating rich tasks (Morin, Simonneaux, & Tytler, 2017).
18
Unfortunately, students struggle to readily engage with their science knowledge once a
politicallenshasbeenappliedtoaproblemcontext(Morin,Simonneaux,&Tytler,2017)and
teachersarehesitanttowadeintoacontextthatmayproverifewithpoliticalpitfalls(Kilinc,
Demiral,&Kartal,2017).Designingtowardsasocio-scientificcontextforanassessmentmay
provefruitfulforelicitingrobustevidence(whichmaybeneededwhenthetargetconstruct
is sosubtleasareCCCs [Weiseretal.,2017]),but careshouldbe taken tochoose those
contextswheretheinstructoralreadyhasstrong,positiveaffect(Kilincetal.,2017).
ConnectiontotheDimensionsoftheNGSS.
TheNextGenerationScienceStandardsrepresentamajorshiftawayfromBruner’s
inquiry(DeBoer,1991)andtowardsmoretransformative,social-constructionbasedactivity
(Blumenfeld et al., 2000). This shift, in addition to the redevelopment of benchmarks as
performance expectations (Krajcik et al., 2014) represents clear attempts to establish
technologic transparency (Lave & Wenger, 1991) within science content and to induce
knowledgecyclesamongpeersandnear-peersthroughdiscourse,cooperation,andcritique
(Lave &Wenger, 1991; NRC, 2012). Along these dimensions, the NGSS represent a step
forward fromProject 2061 in establishing legitimate peripheral participationwithin the
classroom.However, thechoicetopresentthesestandardsasbeingwhollydifferentiated
fromcurriculummaterials(Duschletal.,2007),whileconducivetoteacher-studentcentered
instructionaldesign,alsoplacesasizableburdenoninstructorstoredeveloptheirlessons
withoutasmuchinstructionalsupportasAAASprovided(AAAS,2000).
19
ElementsofRigorintheDevelopmentofAssessments
ValidityandGeneralizability
Assessmentsexistintermsoftheirvalidity(Pellegrinoetal.,2001):dotheymeasure
whattheyclaimtomeasure?Inorderformyassessmentstohavecontentvalidity,theyneed
toprovidethesupportsthatwillallowassessorstomeasurestudents’CCCunderstandings.
However,thethree-dimensionalframeworkofstudentsciencelearningthatisthebasisfor
the NGSS (NRC, 2012) is clear in its assertion that evaluators cannot measure any one
dimensioninisolation,becausetheothertwodimensionsalwaysaffectthecontextofthe
evaluation.Thus,assessmentsdesignedtoaligntotheNGSSneedaveryparticulartypeof
structuralvalidityinwhichtherelationshipsbetweenthethreedimensionsareconsidered.
This is not a simple task and the central concept ofmy dissertation research is that all
currentlyexistingassessmentslackthisformofvalidity,becausetheydonotconsiderthe
roleofCCCs.Generalizabilityisanotherchallengeformyresearchproject.Alldesignprojects
arebuiltupondevelopingsolutionsforlocalproblems;thisisespeciallytrueofframeworks,
likeactivitytheory,whichtakeethnographicapproaches.BarabandSquire(2004)suggest
areconceptualizationofgeneralizabilitythattakestheformofwhattheycall“consequential
validity” (p. 8). For consequential validity, the designed solution is not the end goal of
research.Rather, it isatoolforevaluatingatheoryofdesign(Cobb,Zhao,&Dean,2009).
Therefore, my research seeks generalizability not by suggesting that the particular
assessment I have designed should be used in novel contexts, but rather by showing a
successful method for design (Clarke & Dede, 2009) of science assessments that makes
conclusionsaboutstudents’three-dimensionalunderstandingsinamorestructurallyvalid
way.
20
TheAssessmentTriangle
Theassessmenttriangle(Pellegrino,Chudowsky,&Glaser,2001)sitsatthecenterof
allclaimsregardingtheconsequentialvalidityofanypsychometricinstrument.Thegoalfor
anyassessmentistomakeaconclusionaboutalatentcognitiveprocessfromacollectedset
ofobservableevidence.Whileevidencefromobservationderivesfromcognitionprocesses,
theyarenotthecognitionitself.Aninterpretationschememustbeusedtoconvertfromthe
observations to the desired claims. These three corners of the assessment triangle
(cognition,observation,andinterpretation)needtoagreeiftheassessmentistobevalid–
observablesshouldbetargetedtobestelicitrelevantevidence,theinterpretationscheme
shouldbeabletoaccountforallmeaningfulobservations,andtheclaimsaboutcognition
shouldbelimitedtothosewhereobservablescouldbereasonablygenerated.
Evidence-CenteredDesign
Fromtheevidence-centereddesignperspective,assessmentservesasanopportunity
forstudentstoprovideevidenceofwhattheyknow,andasanopportunityforassessorsto
collect,evaluate,anddrawconclusionsfromthatevidence(DeBargeretal.,2015;Mislevy&
Haertel, 2007). Mislevy and Haertel (2007) suggest that an early step in developing
assessmentistheconstructionofaconceptualassessmentframeworkwhichdictateswhat
youareinterestedinaboutasubject(thestudentmodel),howyouwillcreateopportunities
for subjects to provide assessors with evidence (the task model), and what collected
evidenceneededfortheanalysislookslike(theevidencemodel).Theevidencemodelcanbe
further subdivided into an observationmodel that dictates what evidence will count as
applicable,andameasurementmodelthatdictateshowobservationswillbeconvertedinto
a form suitable for analysis (Mislevy & Haertel, 2007). The evidence model (and its
21
subdivisions),whichseemstocorrelatewiththeunderspecifiedinterpretationcornerofthe
assessmenttriangle(Pellegrinoetal.,2001),isthefocusofmyresearch.
Modelsoftheevidence-centereddesign(ECD)process(Figure2.1)frequentlytakea
formsimilar to thatofToulminarguments (Mislevy,2017) inorder tohighlighthowthe
validityofanassessmentisnotsomemetriclikeprecisiontobequantified,butratherisan
ongoing argument that the instrument is successfully supportinguseful claimsabout the
subjects(Baxter&Mislevy,2004).
Figure2.1Fundamentalsof theECDValidityArgument.Adapted fromMislevyetal. (2017)“Assessing model-based reasoning using evidence-centered design: A suite of research-baseddesignpatterns.”Cham,Switzerland:SpringerInternationalPublishing.ECDandNGSSPerformanceExpectations
Thegoalofassessmentineducationistomeasurecurrentlevelsofproficiencyina
particulardomain(inthiscase,science)inordertosupportfuturegrowth.TheNGSShave
22
redefinedwhatitmeansforstudentstobeproficientinscience(Pellegrino,2013)bylisting
a particular set of claims regarding what proficient students can do (elucidated in the
EvidenceStatementswhichaccompanyeachperformanceexpectation[NGSSLeadStates,
2013]). Thismakes the PEs of NGSS (or a bundle of them)well suited to the evidence-
centereddesignapproach,whichframesassessmentsastheinstrumentsthatelicitevidence
insupportofproficiencyclaims(DeBargeretal.,2016).OncewebundleasetofPEs,soalso,
wehaveselectedasetofclaimstomakeaboutstudents;i.e.,knownasthestudentmodel
(Mislevy&Haertel,2006).FromtherethePEscanbeunpackedintocomponentdimensions
aspartofadomainanalysis(Harrisetal.,2016)thatseekstooutlinethemanywaysdifferent
partsof thedimensional constructs integrateas studentsdemonstrate three-dimensional
understanding.Thenextstepistoconstructanevidencemodel(Mislevy&Haertel,2006)
capableofdelineatingvaluableevidenceofstudentproficiencyfromevidenceattributable
tothenon-focalknowledge,skills,orabilitiesthatagiventaskmightrequire.Ourevidence
model takes the form of a design pattern (Mislevy & Haertel, 2006), which highlights
recurrent themes in the assessment of any one PE from the bundle linked by a viable
sequenceof science/engineeringpractices.Finally,weconstructa taskmodel (Mislevy&
Haertel,2006) thatdefines theway the item/tasksetwillelicitevidence in the lightofa
provided scenario. In later subsections, Iwill further elaborate on the use of ECD in the
creationoftaskstargetingCCCconstructsfromabundleofperformanceexpectations.
EthicsofAssessment
PartoftheToulmin-esquevalidityargumentmadeviatheevidence-centereddesign
approach to assessment development is accounting for alternative explanations by
establishingthatthetasksusedwereappropriateforboththeusecase(Gorin&Mislevy,
23
2013)andthepopulation(Cohen&Lotan,1995).Studentswithminimalexperiencewiththe
terminologythatispresentedwithinataskmaystruggletoengagewithtaskitems(Nobleet
al.,2012).Thisstruggle,therefore,maybearesultofalackoflanguageratherthanalackof
conceptual understanding (Brown et al., 2005). Equity in education considers what is
sociallyrelevant tostudents (Freire&Macedo,1995)so that theyhave thebestpossible
opportunities to learn. Likewise, equity in testing considers what is socially relevant to
students(Wiliam,2010)sothattheyhavethebestpossibleopportunitytopresentwhatthey
know to assessors.To assuage some, thoughnot all, concerns allDAT-CROSS taskswere
passedthroughanequityandfairnessreviewprocessthatconsideredcontextrelevanceand
ELAdifficultyindeterminingtheappropriatenessofthetasks.Additional
DesigningtheDAT-CROSSAssessmentSuite
Crosscutting concepts are broad themes that consistently appear in similar forms
across disciplinary boundaries and, despite differences in conceptual content, seem to
perform similar roles within student understanding across contexts (Rivet et al., 2016).
These roles are subtle and frequently lose salience to assessors when compared to the
students’ more obvious disciplinary core idea competency. Assessing CCCs, therefore,
requiresrichperformancetasks(Gorin&Mislevy,2013)thatallowstudentstobringawider
swath of personal experiences and understandings to bear during the activity. Mislevy
(2017)providessomeinsightintohowrichtaskscanstillbeimplementeddespitetheirhigh
cognitiveload.Althoughtheperformanceofanyonestudentonanyonerichperformance
taskmaybetoomessytosuccessfullymakeindividual-levelconstruct-relevantconclusions,
it is possible touse repeatedmeasures of the construct acrossmany contexts andmany
24
individuals that combine (or overlap) to produce a clearer picture of students’ CCC
understandings.
DAT-CROSSwasdesignedtousethreesuchrichtasks,eachwithastorylinearounda
familiar system: infestation of a farm by a dangerous insect (ecological systems), the
shrinkingwateraccessofafarmtown(earth-hydrologysystems),andtheconsequencesof
starch-richdietsonhumanhealth(humanbodysystems).Eachstorylineisgroundedinreal
problemsthatscientistsseektosolve(Brownetal.,1989)thatstudentscancontextualize
(Noble et al., 2012) in light of their own experienceswith insects, waterways, and food
consumption.ThefollowingparagraphsexemplifythedesignprocessthatIusedtodevelop
theecologicalsystemsstoryline.
Once I selected the ecological system of the farm as being a viable context for
anchoring the assessment, thenext step is to find appropriateperformance expectations
(PEs)thattargettherelevantCCCconstructs,sciencepractices,andcore ideasapplicable
bothtothescenarioandtotheobjectiveoftheresearchproject(theCCCs‘systemandsystem
models’and‘structureandfunction’;NGSSLeadStates,2013c).IselectedabundleofMS-
LS2-3,MS-LS2-4,andMS-LS2-5(NGSSLeadStates,2013a)whichjointlydescribeataskof
modelingenergyandmatterflowswithinanecosystem,howthoseflowscanbeaffectedby
changestothephysicalcomponentsoftheecosystem,andhowhumansmightengineera
solutionthatseekstomaintainapre-existingecology.ItshouldbenotedthatthePEsofthis
bundledonotfullyaligntothedesiredCCCconstructs(acommonproblemduetouneven
distributionof CCCs across thePEsof theNGSS).However, addendumdocuments to the
NGSSfocusingonCCCs(NGSSLeadStates,2013c)notethatCCCsrarelyexistinisolationand
25
canfrequentlybegroupedthematically(i.e.‘energyandmatter’canbeausefullensinthe
understanding of systems or ‘stability and change’ can be critical to the continued
functioningofaparticularstructure).AppendixAhighlightshowthePEswereadaptedfrom
theiroriginalformtoplacegreateremphasisonthenewfocalCCCs.
Inprocessofmakingsenseofthephenomenon(Krajcik&Merritt,2012)regarding
the consequences of an insect infestation to a pre-existing ecology, and the steps a local
community might take to protect their farmland, students will naturally present their
understandingsofsystems(Breslyn,etal.,2016;Gunckel,etal.,2012;Jin&Anderson,2012:
Mohan,etal.,2009;Songeretal.,2009;Yoonetal.,InPress),foodwebs(Griffiths&Grant,
1985;Hogan,2000;Hokayem,Ma,&Jin,2015),argumentation(Berland&McNeill,2010),
and modeling (Schwarz et al., 2009). The prior literature on practices, DCI learning
progression,andexpecteduseoftheCCCSystemsandSystemsModelsservedtosettheclaims
regardingwhatitmeansforstudentstobescientificallyliterateintermsofthereal-world
scenario.Thisactedasaformofdomainanalysis(Mislevy&Haertel,2006)thatfacilitated
creationofataskmodeldescribingabroadsetoftasksthatcouldpotentiallybeconstructed
forthisresearchstudy;onlyasubsetofthosepossiblewereproduced.Figure2.4outlines
theprocessbywhichthethreeselectedPEswereunpackedandtranslatedintoataskmodel
(describedinAppendixA)appropriatetowardstheCCC-assessinggoalsoftheDAT-CROSS
project.
Oneofthebiggestchallengesassociatedwiththedesignprocesswashowtomaintain
thesalienceofCCCsevenafterintegratingthemorewell-supporteddimensionsoftheNGSS.
Unlike the integration process used by Harris et al. (2016), which combined all three
26
dimensionsatthesametimefollowingtheirdomainanalysis,Istaggeredtheintegrationof
theDCI.Thispermittedgreater in thedesignprocess toward the interactionofSEPsand
CCCs,beforethesalienceoftheDCIoverwhelmedthedesignprocess.
27
Figure2.2DiagramofEcologicalSystemsTaskModelDevelopment
28
DevelopingCognitiveLabProtocols
Inassessmentdevelopment,theroleofcognitivelabs–inwhichstudentsoftarget
ageand/orcontentfamiliarityworkthroughthetaskwhile“thinkingaloud”(Johnstoneet
al.,2006)–istovalidatetestelementsthatwillnotbereadilyaccessibleoncethetaskmoves
into production stages. It is of critical importance that test itemshave construct validity
(Peterson et al., 2017). However, in multiple choice questions especially, it may not be
possible to know if the reasons why students select a particular response is actually
reflectiveofreasoningwithintherelevantconstruct(Howelletal.,2013).Simultaneously,
thetargetaudienceofthefinalproductneedstobeviewedasusableanduseful(O’Brienet
al.,2008).Accordingly,thefinalresearchprotocolwasmanifestedintwohalves:acontent
validation half and a reflective use half. These two halves, described in subsequent
subsections,arepresentedinAppendicesCandD.
Aprotocolforcontentvalidation.Howelletal.(2013)describetworesearchgoals
addressed by the cognitive interview method. The first goal is confirming items are
appropriatelykeyedsuch that thestatedbestanswer is theonlyone thatstudentscould
successfullyjustifywithinthecontextofthequestion.Thesecondgoalistoreviewresponses
todistractoroptionstoensuretheyreflectareasonedguessratherthantheresultofsome
non-focal barrier. This requires the support of “design rationales” (Howell et al., 2013),
whichdescribeboththeintendedreasoningthateachtaskofanassessmentistargeting,and
theplausible reasonings thatwould reflect awell-reasoned inaccuracy rather thanause
issue. The answer to these questions support what Snow and Katz (2009) call the
“interpretiveargument,” a stageof theevidence-centeredvalidityargument inwhich the
designercanarguethattheinterpretationofevidencecollectedduringthetaskisactually
29
usefultowardsclaimsaboutwhatstudentsknoworcando.Petersonetal.(2017)notesa
cog-lab research cycle in which good understanding of the goals of each item supports
identificationofbothsuccessesandbarrierstostudentperformance.Below,Iexemplifyan
itemdesignrationaleforquestion2oftheDAT-CROSSassessment:
Figure2.3Question2ofDAT-CROSSwithItemDesignRationale,below
Key: Corn–Producer,Goat–PrimaryConsumer,
Cornrootworm–PrimaryConsumerHuman–PrimaryConsumerandSecondaryConsumer
ItemDesignRationale Themajordecisionstobemadeinthedesignofitemsisaddressedbelow.Whatistheassessmentitemasking?Thisitempresentsavarietyoftrophicrolesorganismscantakeonwithinanecosystemandasksrespondentstocorrectlyassociatestoryelements(corn,goats,people,andcornrootworms[CRWs])withthoseroles.What information is important? It is important to pay attention to respondents’ability to differentiate primary and secondary consumption, identifywhich kind oforganismsareconsideredproducers,andtoseeCRWsasinvasiveconsumersratherthankeydecomposers.
30
Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistforsomeselectdistractorresponses.Furtherrationalesmayneedtobeinvestigatedwithinterviewprobes.
1. The corn are producers – Corn produces edible biomass by convertingmatter(carbonintheairandwaterfromair/soil)byusingenergyfromthesun.
2. Thegoatsareprimaryconsumers.–Thegoatdirectlyconsumesthecorn.3. ThepeopleofTownsvillearebothprimaryconsumersandsecondary
consumers.–Humansconsumecornbothbydirectlyeatingitandbyeatingorganisms(thegoats)thateatthecorn.
4. The people of Townsville are one of either primary or secondaryconsumers-Humansconsumecorneitherbydirectlyeatingitorbyeatingorganisms(thegoats)thateatthecorn.(Note:checkrespondent’sreadingofthestoryandwhethertheynotedthatbothconsumptionsoccur).
5. The corn rootworms are primary consumers. – The CRWs directlyconsumethecorn.
6. Thecornrootwormsaredecomposers.–TheCRWscausethecorntodie(decompose).
In addition to outlining the tasks undertaken by students, Appendix C includes my
documentationofreasoninganddesignrationalethatundergirdseachresponseoption.
A protocol to inspect usability barriers. Nonetheless, it is not enough that
questions are construct relevant. Theymust also be perceived as authentic by students
(O’Brienetal.,2008)andaligntowhattheyshouldbeabledoattheirstageofdevelopment
(Johnstone et al., 2006). Inauthentic tools will feel frustrating and foreign, potentially
limitingabilityorwillingnesstoengagewithatask;whiletoolsthatarebeyondthescopeof
students’skillswillnotprovideconstruct-relevantinformation.Bensonetal.(2002)provide
aseriesofheuristicsforidentifyingusability,particularlyincomputer-assistedtasks(like
DAT-CROSS), which served to inform a second, semi-structured protocol. During these
interviews,Iaskedstudentstoreflectontaskelementsandidentifychallengesintaskclarity,
ability to error check, and their comprehension of the content in the task storylinewith
questionssuchas: “Was thevideouseful tohelpyouunderstand the relationships in the
31
system?”;“Basedonthevideo,whatwasthepurposeofthesimulation?”;and“Whatdidyou
learn from the video?” A more thorough detailing of the usability reflection protocol is
providedinAppendixD.
FrameworkforAnalysis
ConfirmatoryFactorAnalysis
Factor analysis is a critical tool in this process of supporting a validity argument
(Kline,1994).Factoranalysisuseslinearalgebratobreakdownthescoresassignedtoeach
itemintoalessernumberoflatentvariableswhichshould(iftheinstrumenthasconstruct
validity) map to the target constructs. Confirmatory factor analysis (as opposed to
exploratory factoranalysis) isused to ‘validate’ a setof constructs;whereas, exploratory
factor analysis is used to analyze data to potentially discover underlying constructs or
factors.
RaschModelsandWrightMaps
Whilemanyanalysisapproachesfitwithinthelearningprogressionparadigm(Rivet
&Duncan,2013),acurrentdominantmeansofinterpretingstudentresponsesinorderto
empirically supportahypothesizedprogressionmakesuseof theRaschmodel (Briggs&
Alonzo,2012).TheRaschmodel is a special classof item-response-theorymodelswhich
supposesthattheonlymeaningfulfactorsindeterminingthelikelihoodthatstudentigets
task item n correct is the relativemagnitudes of the student’s latent ability (δi) and the
difficultyoftheitem(βn).AsinEquation1,below,thesetwofactorscombinetodictatealogit
model.
32
Pr{$%& = 1} = +,-./01 + +,-./0 (3456789:2.1)
TheRaschmodelcanbeamendedtoconsiderpartialcredit(asinGotwals&Songer,
2013) and some examples exist inwhich a greater number of item-parameters (such as
discriminability)areincluded(asinWilson&Draney,2005),buttheprimaryend-goalofall
RaschmodelsistheWrightMap.Sincethetwofactors(abilityanddifficulty)aremodeled
ontoasharedintervalscale,wecangraphthemalongasharedaxis(e.g.,Figure2.2),whereby
each item (the right side of the map) sits at the same height at which a student of
corresponding ability should have a 50% chance of getting the item correct. By coding
questionstoparticularconstructlevels,theWrightmapcanprovidesomevisualevidenceof
patternsinthedifficultyofitemsrelativetotheconstructlevels(Rivet&Kastens,2012).In
the figure below (adapted from Gotwals and Songer [2013]), questions of code E5
consistentlysitbelowquestionsofcodeC8,suggestingthatconstructE5(regardingability
todetectscientificevidence)iseasierthanconstructC8(regardingabilitytodetectscientific
claims). This visual evidence can be further supported by ANOVA of difficulty scores to
determineifobserveddifficultydifferencesarestatisticallysignificant(Lee&Liu,2010).
33
Legend:The orange histogram bars indicate the number of accurate responses for eachevaluation item ina seriesof increasingdifficulty, someofwhichare indicatedby labelslistedatthebaseofthehistogram.Figure 2.4 Example of a Wright Map. Adapted from Gotwals & Songer (2013). Validityevidenceforlearningprogression-basedassessmentitemsthatfusecoredisciplinaryideasandsciencepractices.JournalofResearchinScienceTeaching,50(5),597–626.ActivityTheory
Developing an observationmodel requires deep thinking regarding what kind of
evidenceassessorscanobserveandwhatkindofevidence isuseful toobserve.Acareful
consideration of contextual factors can help assessors make these needed distinctions
betweenthemanyobservations.Fromthesituatedactionperspective(Nardi,1996),context
plays the most important role in transforming a potential assessment space into an
actualizedassessmentspace,oftentransformingthesespacesinwaystheassessorcannot
possiblypredict. Incontrast, thedistributedcognitionperspective(Nardi,1996)suggests
thatassessmentspacesexistasinstancesofsharedcognitionbetweenassessors(whoput
thinkingintomakingaproblemsolvable)andsubjects(whoputthinkingintoproducingthe
solution).Activitytheorysitsasasortofmiddlegroundbetweenthetwo(Nardi,1996)by
34
implyingthatwhilecontextultimatelydirectsthesubject’scognition,andthatcognitioncan
beconstrainedbytheassessortowardsaparticulargoal.Engestrom(2000),asthefounder
of activity theory, describes the framework as a method for analyzing the interaction
betweensubjects,theobjectoftheirperformances,theinstrumentsortoolsusedtooperate
ontheobjects,therulesforacceptableperformance,thenormsofthecommunity,andthe
division of labor among relevant parties. These six factors, which all contribute to the
achievement of a desired outcome, are frequently depicted as vertices of interconnected
triangles.Studentwork,too,isaformofactivity–depictedinhypotheticaltriangleformas
Figure2.3.
Figure2.5ExampleofEngestrom’s(2000)ActivityTheoryModelinStudentWork
Jonassen and Rohrer-Murphy (1999) explains how activity theory can serve as a
framework for evaluating and developing learning environments. While they focus on
activitytheoryasamechanismforevaluatingdesigninterventions,theideascancarry-over
to evaluationof studentperformances (as in theactivity theory-baseddesignprocessby
SparksandDeane,2015).Successfulstudentperformancesinvolveeffectivelyusingamodel
35
asatoolfortransformingtheobjecttoalignwiththeperformancegoal,activitytheorycan
beusedtounderstandwhentheseperformancesaredisrupted.Cohen(1998)suggeststhat
these disturbances are a function of mismatches between expectations of subjects and
objects, usually due to the differing contexts by which the two live. Anchoring context
(Cognition andTechnologyGroup atVanderbilt, 1992)may, therefore, prove effective in
constrainingcontextwhilemakingexplicittostudentsthekindsofperformancesdeemed
acceptablebythecommunity.
Ibelievethefindingsthatemergedbyemployingtheactivitytheoryframeworkwere
invaluableindirectingmuchoftheanalysisintohowtheDAT-CROSSassessmentmightbe
improved.Indeed,muchhasbeenwrittenontheaffordancesofactivitytheory(Nardi,1996;
Roth&Tobin,2002).Sincetheearly2000s,however,increasedattentionhasbeenplacedon
adapting activity theory to account for the roles culture and history play (Nussbaumer.
2012).Thecultural-historicalactivitytheorymodel(CHAT)considersthatwhatscriptsfor
activityexistforthesametoolsdevelopunderdifferentcontextsandmanifestindifferent
formsundertheinfluenceofboththelivedexperiencesofthesubjectbutalsothecollective
experiencesoftheculturethesubject inhabits.Morenuance isrequired, therefore, inthe
claimsImakeregardingthekindsofstudentswhomightbewillingtoperformusabilitytests
foracompanythatspecializesinassessmentdesign.
Accordingly, I cannot claim that participants are representative of the general
populationof futureusersbutIcansupposethattheseparticipantsareat leastbest-case
use-participantsfortheirgradeband.Thatis,anyusabilitychallengesfacedbythisbest-case
sample of users are likely to be faced by most other students, and possibly in a more
demandingway.Moreover,fortheveryfirstimplementationofanovelresearchenterprise,
36
itisoftenusefultobeginwithmoretractableparticipantstobetterestablishafoundation
forbroaderfutureresearch.Itislefttofutureresearchendeavorstoexaminehowstudents
useCCCsnotonlyacrossdisciplines(assetforthbytheNRC[2014]),butalsoacrosscultural
contexts.
37
Chapter3
METHODOLOGY
Inthischapter,Idescribetheproceduresentailedinbothcollectingandanalyzingthe
dataformydissertationresearchendeavor.Iuseddifferentformsofdataand,byextension,
employeddifferentanalyticproceduresforeachofmythreequestionsinterwovenamongst
eachotherinwhatCreswellwouldcallan“analyticspiral”(Creswell,2009,p.183).Below,I
restatemythreeresearchquestions.
1. IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthat
canbemeasuredasstudentsusethem?
2. Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectof
their three-dimensional scienceunderstandings look likeatdifferent levelsofoverall
scienceunderstanding?
3. What challenges do middle-school-aged subjects face in presenting their CCC
understandingswhileengagingwithan interactivesuitedesignedtotargetSystems-
Models and Structure-Function dimensions of their three-dimensional science
understandings?
A more succinct overview of these methods can be found in the data collection matrix
(AppendixF).
Addressing these research questions required two forms of data: a) themultiple-
choiceresponsesdirectlyproducedbyrespondentswhentheyengagedwiththetestsuite,
and b) selected interview responses obtained from the respondents’ narrative as they
participatedinathink-aloudtaskthatinvolvedareviewofallthecomponentsoftheactivity
(Hoadley,2004).The firstof thesedatasets isquantitative innaturewhile thesecond is
38
qualitative;thus,all-togetherrequiringamixed-methodsapproachcommonlyemployedin
thedesign-basedresearchparadigm(Brown,1992).However,unlikesomenarrativeforms
ofqualitativeresearchthatemphasizethesemanticmeaningsofthenarrativeasarecordof
theindividual’s livedexperiences(as inthecaseofethnography,casestudy,orgrounded
theory (Creswell, 2009); the goal of these interviews is to confirm that theprompts and
stimuliused in the assessment are successfully inducing students toprovideevidenceof
relevantcognitiveprocesses.Ideally,theassessmentsituationestablishesapositiveclimate
for activity, where the student is not faced with significant barriers from the non-focal
processesinvolvedinthedoingoftasks(i.e.,thesubjects’knowingthecorrectbuttonsto
clickonscreentologtheirdesiredmultiple-choiceselection,etc.).Priorlivedexperiences
play a role – as they do for all activities of life – in how students both engage with
assessments-writ-large and how they do so in the specific, somewhat rare context of
usabilityinterviews.However,theanalysisofsuchimpactsremainsatopicoffuturestudy.
DataCollection
ParticipantsandExperimentalSetting.
Thecognitivelabsthatmakeupthedatacollectionformydissertationresearchtook
placeatETSheadquartersinPrinceton,NJduringtheautumnof2018.TheDAT-CROSSteam
recruited45,middle-school-agechildren,primarilybyreachingouttoETSstaffwhohave
appropriately aged offspring. Middle-schoolers are the target participants, because they
representthetargetageofboththeDAT-CROSSassessmentsuiteandthegradebandofthe
NGSSperformanceexpectationsusedduringthedevelopmentprocess.Ultimately,about15
participantsfromeachofgrades6,8and10wererecruited,generatingaspreadinstudent
39
performances.ParticipantswereremuneratedfortheirtimewithaVisagiftcardatanETS
standardrateof$30.
Regardingmyparticipants,47%percentwereFemale,44%werewhite,38%were
AsianorAsian-Indian,11%wereAfrican-AmericanorBlack,whiletheremaining7%were
Hispanicorofotherracial/ethnicbackground.Amore
Process/Procedures
The formatof theDAT-CROSS suite isdetailed inAppendixC, includingquestions
usedduringtheassessment(whichwereamixtureofmultiplechoice,multipleselect,drag-
and-drop, and subject-constructed style items). Each question has a dedicated stimulus
designedtomapsuccessfulcompletionofthetasktothekeyprogressindicators,whilestill
beingcapableofmappingpotentialdistractorstolowerlevelsofthelearningprogression.
Duringtheassessment,subjectsengagedwithastorylinethatrequiredthemtointeractwith
simulationsandmodels inorder todraw theevidenceneeded to support socio-scientific
arguments(likethoseofEggertandBögeholz,2010).Theseargumentstakethegeneralform
ofhowbesttoimprovethefunctioningofanengineeredsolutiontoalocalproblem(likean
infestationofcrop-destroyinginsectstoafarm).Figure3.1exemplifiesonesuchquestion
(Question5)fromtheactivity.
Duringthisphase,theassentingparticipants(consentalsoprovidedfromtheirlegal
guardian)willbeaskedtocompletethetaskswhileverbalizingtheirthinkingabouteachof
their choices. Hammer (1994) has noted that even advanced students struggle to
spontaneouslyengageinthesetypesofinterviews,soaDAT-CROSSteammember(though
40
nottheauthorofthisproposal)waspresenttoguidethesubjectsthroughtheactivitiesin
ordertohelpparticipantsrecognizeinstancesinwhichsubjectthinkingmayhaveoccurred.
Figure3.1Question5ofDAT-CROSSEcosystemStoryline
Questiontext:Jonahandthescientistswanttoknowhowthecornrootwormswillaffecttheamountofcornthatsurviveuntilharvest.Usethetooltomakeamodelforthefoodwebtoshowtherelationshipbetweenthecorn,thegoats,theSun,andthecornrootworms.Key:Sun->Corn;Corn->Goat;Corn->Rootworm.
Each of the subjects responded in a ‘think aloud’ format of the assessment suite.
Additional questions were posed by the researcher targeting places where subjects
struggled(Hoadley,2004).Thiswascrucialtoestablishingevidenceoftheusabilityofthe
assessmentsuite,so that the future iterationsof thedesigncouldmoreeffectivelydevise
assessmentswithminimal interruptions by a proctor to explain aspects of the task. The
think-aloudprocesswasmoretimeconsumingthananon-verbalparticipationapproach–
closertotwohoursofparticipation,ratherthanone.
41
I scored (with theassistanceof fellow researchers) all non-constructed responses
according to their key – listed below each item in figures 2.5, 3.1, and the remainder in
AppendixC.Iscoredeachconstructedresponse(orCR)question(Questions10,12,and13)
accordingtotherubricinAppendixG,andthenIassignedvarioussubsetsoftheresponses
tofiveassistantraters.ForeachofthethreeCRs,35ofthe45responseswereratedbyat
least three people. Since constructed response scores are ordinal, and measured by
nonunique sets of multiple raters, a weighted kappa (Maclure &Willett, 1987) seemed
appropriate. I alsoconsideredsomeothermeasuresof interrater reliability suggestedby
Hallgren(2012),includingpercentagreementandintraclasscoefficient.Table3.1describes
theseratingsforeachconstructedresponsequestion.
Table3.1Inter-raterReliabilityRatingsforConstructedResponses
Question PercentAgreement
Krippendorf’sAlpha
Fleiss’Kappa(weighted)
Intra-ClassCoefficient
Question10 90.5% 0.506 0.501 0.603Question12 90.3% 0.444 0.439 0.533Question13 91.6% 0.608 0.603 0.671
I should also note thatmy role as a consultant precluded the proctoring of these
usability/cognitivelabs.Myanalysisinthisresearchstudywasderivedfrompseudonymized
dataoftheselabs,simultaneouslydiminishingpossiblesourcesofbiasthatImightotherwise
bringintothethink-aloudprocess,butalsolimitingmyagencyoverwhatquestionswere
askedbythelabproctors.
DataAnalysisMethodology
Theoverallgoalofmyresearchquestionsistodevelopaneffectiveevidencemodel
forstudents’understandingsofcrosscuttingconcepts,particularlyregardingtwooftheCCCs
42
Systems and System Models and Structure and Function (NGSS Lead States, 2013c). As
mentionedintheintroductorychapter,however,developinganevidencemodelrequiresthe
ability isolate relevant observations of CCC reasoning from observations of the other
knowledges,skills,orabilitiesthatareevokedbythetask(addressedbyResearchQuestion
3)while also differentiating evidence reflecting progressing levels of CCC understanding
(addressedbyResearchQuestions1and2).Addressingallthreeresearchquestionsrequires
amixofbothquantitativeandqualitativeanalyticalmethods.
QuantitativeAnalysis–ResearchQuestions1and2
DatagatheredtoaddressResearchQuestions1and2formthequantitativeaspectof
the research to be answered via factor analysis (Kline, 1994) and item response theory
(Gotwals&Songer,2013).Factoranalysisservesasthefirststepinconstructvalidation,in
which we examine if variation in performance across a broad set of questions can be
accountedforintermsofalessernumberoflatentdimensions(Kline,1994).Ideally,this
analysisrevealsthatavariablethatcutsacrosscontexts(muchasCrosscuttingConceptsdo)
isasignificantexplainerofoverallperformance.Usinganitem-responseRaschmodel,Ithen
placebothstudentsandCCCperformancealongthesamedimension,reflectinganincrease
initemdifficultyassociatedwithcomplexityofCCCunderstandingelicitedbythetask(the
equationforthismodelcanbefoundinChapter2ofthisthesis).
QualitativeAnalysis–ResearchQuestion3
ResearchQuestion3fitssquarelywithinaphenomenologicalparadigm(asdefined
byCreswell,2009).LiketheapproachtakenbyDuncanandTseng(2011)andbyHoganand
Fisherkeller(1996),phenomenologicalstudyregardingtheexperienceofinteractingwith
curricular or assessment materials can serve as productive frames for evaluating the
43
usabilityofeducationalmaterials.Bothgroupsofresearchersmadeuseofeticcoding(where
thematiccodesaregeneratedpriortoanalysis;Bricker&Bell,2008)tocheckalignmentof
studentworktothedesignedgoalsoftheinstructionalmaterialsandemiccodes(inwhich
themesareallowedtoemergeintheprocessofanalysis)inordertoprobeinstancesinwhich
misalignmentwasfound.Iderivedeticcodesformyanalysisfrompriorpublishedresearch
onthenatureofCCCs(Rivetetal.,2016;Weiseretal.,2017)aswellasknownavenuesof
progressionsinsystemthinking(Breslynetal.,2016;Gunckeletal.,2012;Jin&Anderson,
2012:Mohanetal.,2009;Songeretal.,2009).ThesecodescanbefoundinAppendixG.
Hermeneutics. Translating the words of students into conclusions about their
thinking involves hermeneutic analysis (Eger, 1992) in which the language choice of
students in their crafting of arguments and explanations is evidence for their level of
understanding.Eger(1992)exemplifiesthisbynotingthattheclaimthatanobject‘has’a
forcemaysound likeanobject ‘exerts’or ‘isacteduponby’a force, thusentailingavery
different understandings of Newton’s laws. As in the work of Rivet et al. (2016), such
analysesdrawtheirarrayofconclusionsfromsubtledifferencesinwording(allowingfora
hostofalternativeinterpretationsofthedata).Inordertofurthersupportmyanalysis,the
findings were analyzed quantitatively in terms of their alignment to the hypothesized
evidencemodelandmembercheckedwiththeotherfacilitatorsoftheusabilitystudy.
Activity theoryasananalytical framework.Within theprevioussections, Ihave
describedthebroadproblemofdesigningassessmentappropriatetotheNGSS(DeBargeret
al., 2015) and have outlined activity theory as a framework for thinking deeply about
potential solutions to that problem. Figure 2.3 from the literature review (reprinted as
44
Figure3.2,below)exemplifiesonepossibleactivitytheorytrianglethatdemonstratesthe
interactionbetweentheresponses/activitiesofthestudentsandtheevidence:
Figure3.2ExampleofActivityTheoryModelatPlayinClassroomWork
Thesetrianglesservetoinformtheevidencemodeloftheconceptualassessmentframework
(Mislevy&Haertel, 2007) for this research, and highlight a clear problem space for the
designofanassessment.Mydesigninterventionrequired:(a)assessmentwhichmakesthe
evidencedictatedbytheactivitytheorymodelsalient(Mislevy&Haertel,2007)and(b)tools
for the teacher thathelps themmeasure thatevidence (Joe,Tocci,Holtzman,&Williams,
2013;vanEs&Sherin,2002).
Roth and Tobin (1992), in their use of activity theory to improve teacher
preparedness programs, describe several different forms of contradictions that impede
successfulengagementinaparticularactivity.Ofthese,twoareofparticularrelationtothis
research project: (a) differences between the language of the classroom and that of the
student,and(b)differencesinmotivebetweenassessmentactivityandtheactivityofexpert
scientists.Intheformercontradiction,thegreatestchallengetoelicitingevidenceofCCCsis
45
gettingstudentstounderstandandrecognizeCCCuseintheirownwork.CCCslikepatterns
andsystemshavemeaningsinthesciencesthataredistinctfromtheireveryday,English-
language meanings (NGSS Lead States, 2013b). Because assessment materials are
linguistically constrained, students receive some prompt in English and, then, have to
interpretwhatthatmeansrelativetotheirscienceknowledge(Lee,Quinn,&Valdes,2013).
Inpilotstudies,lookingforevidenceofstudent’sCCCunderstandings–asinWeiseretal.
(2017)–studentsroutinelyaskedforclarificationofthemeaningofpatternsandsystems.
Thisputresearchintoadilemma:clarifymeaningatthecostofpossiblyimposingourown
understandingsoftheCCCsontostudents,orrefusetoclarifymeaningatthecostofstudents’
abilitytoprovideevidenceoftheirreasoning.Bycombiningfieldnotesoftheconfederate
thinkaloudfacilitatorswiththepost-taskinterviewquestions,Iusedactivitytheoryasalens
toexamineifthelanguageoftheassessmentwassuccessfulinelicitingstudents’displayof
their CCC understandings. In the second-referenced contradiction (differences inmotive
between assessment activity and the activity of expert scientists), students generate
epistemic cleavages between their own way of knowing and the ways of knowing
appropriatetoscientists(furtherexpoundedinSandoval,2005).Activitytheoryservesasa
useful framework for ensuring that the finalized, three-dimensional assessment is an
effectivestepping-stonetowardsthepracticeofexpertscientists(Stroupe,2015).Forboth
contradictions of interest, activity theory is not the framework for how the assessor
interpretsevidence,butratherhowitsolicitsevidence(thetaskmodelofDeBargeretal.,
2015).Returningtothenotionsofvalidity,Ihavediscussedinaprevioussection,activity
theory serves as a framework for analyzing field notes, student responses, and teacher
probestoensurethattheassessmentshaveconstructvalidity(wherebytheconstructisthe
46
activitydonebypracticingscientists)andconsequentialvalidity(wherebytheassessment
activitiesdonothaveglowingcontradictionswhichinhibittheiruseintheclassroom).
EthicsandReflexivity
Theapproachusedtoanswermyresearchquestionshasseveralfeaturesthatpose
both general and unique ethical challenges falling into three broad categories: (a) the
concerns associated with any study carried out on human subjects, (b) the concerns
associated with remunerating subjects for their participation, and (c) the concerns
associatedwiththedisconnectbetweenmyroleasdataanalyzer/researchcoordinatorand
theroleofotherDAT-CROSSteammemberswhowerethePIsoftheusabilitystudy.Ofthe
foremost, this study was approved by the appropriate Institutional Review Boards as
fulfillingtheirrequirementsandstandards.
47
Chapter4
RESULTS
SummaryStatistics
Table4.1presentsperformanceaverages foreachassessment item(listedas sub-
itemswhererelevant)oftheDAT-CROSSecosystemtask(AppendixC)categorizedbyboth
gradeandgender.Thefirstitem,Q1,wasunscoredasitservedasanactivatorthatcouldnot
bekeyedtosomebest-possible-response.Differencesbetweengradesreflectsomeevidence
ofalearningprogression,whiletheabsenceofgenderdifferencesissurfaceevidencethat
itemswerenotgenderbiasedincontentorstructure.
Table4.1AveragePerformanceonAssessmentItemsbySubgroup
OVERALLAVG G6AVG
G8AVG
G10AVG
MALEAVG FEMALEAVG
Q2CORN
.91(.28)
.88(.33)
.94(.25)
1.00(0.00)
1.00(0.00)
.81(.40)
Q2GOAT
.62(.48)
.71(.47)
.44(.51)
.82(.4)
.71(.46)
.52(.51)
Q2PEOPLE
.04(.21)
0.00(0.00)
0.00(0.00)
.18(.4)
.04(.2)
.05(.22)
Q2ROOTWORM.04(.21)
.06(.24)
0.00(0.00)
.09(.30)
.08(.28)
0.00(0.00)
Q2SUM1.62(.77)
1.65(.70)
1.38(.62)
2.09(.83)
1.83(.70)
1.38(.80)
Q3.93(.25)
.88(.33)
.94(.25)
1.00(0.00)
1.00(0.00)
.86(.36)
Q4.33(.47)
.24(.44)
.38(.50)
.36(.50)
.25(.44)
.43(.51)
Q5.39(.49)
.35(.49)
.20(.40)
.73(.47)
.39(.49)
.38(.5)
Q6PT1.73(.44)
.71(.47)
.75(.45)
.82(.4)
.71(.46)
.76(.44)
Q6PT21.00(0.00)
1.00(0.00)
1.00(0.00)
1.00(0.00)
1.00(0.00)
1.00(0.00)
Q6SUM1.73(.44)
1.71(.47)
1.75(.45)
1.82(.4)
1.71(.46)
1.76(.44)
Continuednextpage
48
Table4.1AveragePerformanceonAssessmentItemsbySubgroup,continued
OVERALLAVG G6
AVGG8AVG
G10AVG
MALEAVG
FEMALEAVG
Q6SUM1.73(.44)
1.71(.47)
1.75(.45)
1.82(.4)
1.71(.46)
1.76(.44)
Q7CORN.96(.21)
.88(.33)
1.00(0.00)
1.00(0.00)
.96(.2)
.95(.22)
Q7GOATS
.87(.34)
.76(.44)
.88(.34)
1.00(0.00)
.83(.38)
.90(.30)
Q7PEOPLE
.33(.47)
.41(.51)
.25(.45)
.27(.47)
.25(.44)
.43(.51)
Q7SUM2.16(.67)
2.06(.90)
2.13(.50)
2.27(.47)
2.04(.69)
2.29(.64)
Q8.6(.49)
.65(.49)
.5(.52)
.73(.47)
.67(.48)
.52(.51)
Q9.18(.38)
0.00(0.00)
.38(.50)
.18(.40)
.17(.38)
.19(.40)
Q101.38(.85)
1.06(.75)
1.63(1.02)
1.55(.69)
1.25(.85)
1.52(.87)
Q11.02(.15)
0.00(0.00)
0.00(0.00)
.09(.30)
.04(.20)
0.00(0.00)
Q121.11(.77)
.94(.90)
1.13(.62)
1.36(.81)
1.08(.78)
1.14(.79)
Q131.16(.89)
.94(.83)
1.25(.86)
1.45(1.04)
1.21(.93)
1.1(.89)
TOTAL11.6(2.95)
10.47(3.26)
11.63(2.45)
13.64(2.25)
11.63(2.92)
11.57(3.12)
Note1:Standarddeviationsinparenthesis.Boldtextsignifiesstatisticallysignificantdifference(α=0.05)fromgrade6scores
ResearchQuestion1
IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthatcanbemeasuredasstudentsusethem?
Researchquestion1addressedtheissueifcrosscuttingconceptslikeStructureand
Function(StFinFigures4.1and4.2)orSystemsansSystemModels(SyMinFigures4.1and
4.2) are different and measureable constructs that can be assessed. For DAT-CROSS,
questions2 through8weredesigned to target theStFCCC.Thesequestionsentailed the
followinggeneralaspects:a)constructingamodeltounderstandthetrophicstrutureofthe
49
farm,b) identifying theroleof thecornasa trophicproducer,andc)predictinghowthe
introductionofanewconsumer(theWesterncornrootworm,D.virgifera)mightthreaten
thattrophicstructure.
Followingthosequestions,studentsinteractedwithasimulationthatdemonstrated
theoutcomeofimplementingatwoalternativestrategiesforcontrollingtheinfestationof
thefarm.Throughtthatinteraction,studentsrespondedtoQuestions9through13,which
weredesignedtotargettheSyMCCC.Thesefivequestionsentailedthefollowingaspects:a)
understanding the consequencs of not implementing any infestation control strategy, b)
comparingandexplainingtheefficacyofthedemonstratedcontrolstrategies,andc)making
arecommendationregardinghowthecontrolstrategyoughttobeusedinthefuture.
Includinggradeasapossiblefactorinfluencingperformance,Ihypothesizefromthe
designoftheDAT-CROSSassessmentfactorstructureillustratedinFigure4.1.
Figure4.2describes the resulting standardizedparameters from theconfirmatory
factor analysis. The number within each arrow represents the standardized correlation
betweenvariables.
50
Figure4.1HypothesizedFactorStructureofDAT-CROSSEcosystemAssessment.
Thehighlight of this analysis is the lackof correlationbetweenStF andSyMwith
confidence interval [-0.070, 0.033]. Following the convention outlined by Garver and
Mentzer (1999), a standardized inter-construct correlation that excludes 1 within its
confidenceintervalisevidencefordiscriminabilityofconstructs.
Legend:Grd=Student's Grade-year; StF=Structure and Function; SyM=Systems and SystemModels; Q#=the scorestudentsgotonanitemnumber#.Squareboxessignifyobservedvariableswhilecirclesconveylatentvariables
51
Figure4.2ResultingCorrelationsfromCFAAnalysis
ResearchQuestion2
Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectoftheirthree-dimensionalscienceunderstandings look likeatdifferent levelsofoverallscienceunderstanding?
PreliminaryLearningProgression.FollowingtheresultsoftheCFAanalysis,Iused
separateRaschpartialcreditmodelstoestimatestudentperformanceonboththeStructure-
Function-targeting and Systems-and-System-Models-targeting items. These resulted in
Wright Maps (Figures 4.3 and 4.4, respectively) and fit statistics (Tables 4.2 and 4.3,
respectively).Ingeneral,using45participantsisonthelowendofacceptableforvalidating
aRaschmodel(deAyala,2010),sotheseresultsshouldbeviewedaspreliminaryandasa
meansoffindingavenuesofimprovementratherthanproofuntoitself.
Legend:Grd=Student's Grade-year; StF=Structure and Function; SyM=Systems and SystemModels; Q#=the scorestudentsgotonanitemnumber#.Squareboxessignifyobservedvariableswhilecirclesconveylatentvariables
52
Figure4.3Person-Item(Wright)MapofStructureFunctionItems
FromFigure4.3,thefirststandoutareQuestions2(inwhichstudentsidentifythe
trophiclevelofvariousorganismsonthefarm)and7(inwhichstudentsmakeaprediction
abouttheoutcomeoftheoncomingrootworminfestation).TheRaschpartialcreditmodel
(adaptedfromEquation2.1fromtheliteraturereview)estimatesthatitwaseasiertoget
fourresponsesoutoffourcorrectinQuestion2,thanitwastogetthreeresponsescorrect
and, similarly, estimates that it was easier to get two responses out of three correct in
Question7,thanitwastogetjustone.Thesestrangeresultsbearoutinthefitstatisticsas
Question2hadanoutfitTbeyondacceptablelevels(deAyala,2010).Questions2and7were
alsosomewhatuniqueinthattheypromptedmultipleselectresponses(i.e.,studentswere
askedtoselectallthatapply).
Legend:Top:TheordinatedimensionisahistogramofnumberofstudentsinrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).Bottom: The ordinate dimension lists Structure-Function-targeting questions (with partial scores asapplicable)inrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).
53
Table4.2FitStatisticsofStructureandFunctionItems
CHISQ DF P-VALUE OUTFITMSQ
INFITMSQ
OUTFITT
INFITT
Q2_SUM 22.588 44 0.997 0.502 0.469 -2.59 -2.86Q3 21.636 44 0.998 0.481 0.802 -0.78 -0.34Q4 54.053 44 0.142 1.201 1.147 1.07 1.16Q5 30.923 43 0.915 0.703 0.741 -2.02 -2.49
Q6_SUM 34.631 44 0.843 0.77 0.838 -1.01 -0.99Q7_SUM 31.003 44 0.93 0.689 0.725 -1.43 -1.26Q8 54.678 44 0.13 1.215 1.174 1.38 1.39
Overall,theSystemsandSystemModelsitemsseemtobehavewellwithsomedecent
evidence.Forexample, the responses toDAT-CROSSQuestion9 (inwhichstudentswere
asked to describe patterns in the data produced from the simulation as it simulated the
efficacyofthefirststrategyforcontrollingtherootworminfestation–addingnewpredators)
andtheconstructedresponsequestions(questions10,12,and13inAppendixC)alignsto
the hypothesized learning progression (detailed inAppendixB). That is, Question 9was
designedtoaligntolevel4ontheSystemslearningprogression(inwhichstudentsareable
tomakepredictionsaboutthefuturestateofasystem),aboutequaltothetargetalignment
oftheconstructedresponsesthatearnedathree-pointscorebasedonthescoringrubrics
(furtherdetailedinAppendixH).
54
Figure4.4Person-Item(Wright)MapofSystemsandSystemModelItems.
InFigure4.4,wefindexactlythatthealignmentbehaviorbearsoutinthemodeled
itemdifficulty(the latentdimensionaxis).Forexample,Question9hasanearequivalent
itemdifficultymeasureasthree-pointscoresinQuestion10(thefirstconstructedresponse
questioninwhichstudentsareaskedtoexplainwhythepredator-introductionstrategythat
was also the subject of Question 9 was not as effective as predicted). Examining the fit
statistics (Table 4.3), further examination is needed to understandwhy Question 12 (in
whichstudentsexplainwhyasecondcontrolstrategy–plantingatrapcrop–waseffective
at reducing the impacts of the infestation) has poor outfit T, and why students found
Question11sodifficult(whichwassimilarinformtoQuestion9butregardingtheefficacy
ofthesecond,trap-cropstrategyratherthanthefirst).
Legend:Top:TheordinatedimensionisahistogramofnumberofstudentsinrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).Bottom:TheordinatedimensionlistsSystems-and-System-Models-targetingquestions(withpartialscoresasapplicable)inrelationtoLogitsofdifficulty/abilityontheabscissa(theLatentDimensionsharedbyitemsandpeople).
55
Table4.3FitStatisticsofSystemsandSystemModelItems
CHISQ DF P-VALUE OUTFITMSQ
INFITMSQ
OUTFITT
INFITT
Q9 29.343 42 0.93 0.682 0.973 -0.51 -0.05Q10 37.621 42 0.663 0.875 0.846 -0.57 -0.74Q11 19.178 42 0.999 0.446 0.827 0.2 0.04Q12 23.946 42 0.989 0.557 0.562 -2.45 -2.42Q13 35.926 42 0.734 0.835 0.83 -0.75 -0.82
TheseRaschmodelsallowedmetoisolatesixsubjects(threeeachfromStFandSyM)
whoperformedatthebest,worst,andmedianofparticipants.Theirresponsesbothduring
thethinkaloudandinthereflectiveinterviewareexaminedinthediscussionchapter.
More on the Structure and Function and the Systems and Systems Models
Constructs.Togain furtherevidence that theStructureandFunction construct isunique
compared to the Systems and SystemModels construct, I compared themodeled student
abilityscores(deltafromEquation2.1,alongthelatentdimensionaxisfromFigures4.3and
4.4) fromtheirperformanceonthequestionsfromeachconstruct. If theconstructswere
equivalent, we would expect that the distribution among the population of each was
equivalent. Using a Wilcoxon signed-rank test (Kerby, 2014) to account for potentially
nonparametricvalues,thedistributionsofstudents’abilityinthetwoconstructsrejectedthe
nullhypothesisofequivalentdistributions(W=830,p<<.01).
ResearchQuestion3
What challenges do middle-school-aged subjects face in presenting their CCCunderstandingswhileengagingwithan interactivesuitedesignedtotargetSystems-Models and Structure-Function dimensions of their three-dimensional scienceunderstandings?
UnderstandingDisruptionsinStudentPerformance
56
InlightofthefindingsfromResearchQuestion2,inwhichcertainitemswerefound
to be poorly functioning, understanding the non-construct relevant challenges students
facedbecomesallthemorecritical.ToanswerResearchQuestion3,IreturntoEngestrom’s
(2000)activitytheorymodelasameansforanalysis.Withinhisframework,thestudentwho
weremysubjectshavedevelopedascriptforbeingassessedintheirscienceclassroomthat
hasbeencrafted,muchliketheconstructivistschema,inresponsetotheirpriorexperiences.
Astheyworkedwithinthenovelcontext(understandingecosystems)oftheassessmentand
inanovelsetting(onacomputeroutsideoftheirnormalclassroom),studentsmayfindthat
theirpriorscriptsareunproductiveorcounterproductive–leadingtoadisruptionintheir
engagementwiththetask.Inthefollowingsubsections,Ioutlinethreepatternsofdisruption
thatcommonlyoccurredasstudentsworkedwiththeassessment–eachdiagrammedusing
activitytheorytriangleswithredlinkages.
Disruption1:ParingtheFoodWeb.Thefirstsetofactivitiesinthefarmecosystem
assessment was designed to introduce students to modeling ecosystems as a set of
relationships betweenorganisms thatmight grow in complexity as neworganismswere
addedintothemodel.Whilestudentshadminimaldifficultyconstructingtheinitialmodel
(Q3 of the task as described in Appendix C) with 44 of 45 correctly connecting model
elements, theyexpressedgreater challenge inunderstanding the roleof themodelwhen
asked to incorporate the rootworm into their prior creation. Usability Participant 15
expressestheirconfusion...
Facilitator: So,thefirstisaboutthefirstmodelthatyoucreated.So,wasiteasytointeractwiththecomponents?
Usability15:Yes.Itwaseasy,butIthinkyoushouldlikeexplainmoreaboutwhatyouwantustodonexttime.Like,Igotthepartthatyouwantedustoshowliketherelationshipbetweenthem,butlikewhatkindofrelationship?Likethefoodchain
57
relationshiporlike,isitlikethesunfeedsthecornorisitthegoateatsthecorn?Thecorntakesfromthesunorisitthesunfeedsthecorn?Thecornfeedsthegoat?Isitbackwardsorisitforwards?
Student’schallengesintheseearlymodelscouldpropogateintolateritemsdespiteleveling
(inwhich studentswere shown the correct answer as part of interactionwith the task).
UsabilityParticipant30initiallyexplainswhytheharvestmenstrategymightnothavebeen
effective…
Usability30:Adding theharvestmendidnothelp increase thepercent corn yieldbecausethenumberofrootwormeggswerestillincreasingalso.
Only on later reflection, during the post-task interview portion, did they describe the
phenomenonusingideasfromthemodel…
Facilitator: So,wehavethesecondpartthatitwasrelatedtotheharvestmen.So,whatdidyounoticeintermsfromthedatafromthesimulationvideoandfromthechartandthegraphs? Whydoesitremain? Thinkthewayyouhavetoexplaintosomeone.Usability30:The corn harvested was going down but it wasn’t going down asgraphicallyasbefore.Andthenumberofrootwormeggswhereinashighasbefore.Facilitator: Doyouthinkthatthisstrategywaseffective?Usability30:Notreally,becausethey’restilllosingcornyieldandtherootwormeggsarestill,like,theyarestillincreasingandmultiplyasmuch.Facilitator: Whydoyouthinkthat thenumberofharvestmen introducedbythescientistwasthesameeveryyearfromyearthreetofive?Usability30:Because they want the harvestmen becoming -- they want theharvestmenhavingthepredatorsalsoandtakingoverthe--disturbingthefoodchain.Facilitator: Okay.Whatdotheharvestmendointhisecosystem?Usability30:Theygatheruptherootworms.Facilitator: Andhowdotheygetridoftherootworms?Usability30:Ithinktheywouldeatthem.
I call this pattern (as diagrammed in Figure 4.5) “paring the foodweb”, in which
studentspaidselectiveattentiontoideasfromthemodelingtaskwhichdidnottransferinto
makingsenseofthesimulation.
58
Figure4.5DiagramofActivityDisruptionWhenStudentInteractswithModelingTool
Disruption2:FailuretoInteractwithEvidenceAvailableintheSimulation.As
studentsmoveintothesecondphaseofthetask,theyareaskedtointeractwithsimulated
implementationsofthecontrolstrategiesinordertocollectevidenceabouteachstrategy’s
efficacy.Drawingasuccessfulconclusionentailedmergingevidenceavailablefromseveral
sources – information provided to students before the simulation regarding how each
strategy might work, a visualization of organisms in the farm ecosystem interacting
according to thecontrol strategy,adata table indicating thepopulationnumbersofeach
organismbyyear,andasetofgraphs(seeAppendixC).However,inpractice,studentsoften
restrictedthemselvestoonlyonesetofevidence,usuallyasinglegraphorthedatatable.
Below,UsabilityParticipant27explainshowheusedtheevidenceinreasoningaboutthe
harvestmenstrategy.
Facilitator: Yeah. So, the term harvestmen, are you okay with that term?Harvestmen?
Usability27:Well,Ididn’treallyknowitbefore. Facilitator: Okay,butwhenyouseethis,it’skindofbug Usability27:Itjustseemedlike,(00:42:50)isjustabug. Facilitator: Yeah,okay.So,forthesecondvideo,doyourecallwhatitwastryingto
tellyou?It’sabouttheharvestmen.
59
Usability27:Howtheygrewwhenaddedwhenlikeyouaddedtheharvestmenwiththerootworms.
Facilitator: Yeah. Usability27:Ifeellikeitdidn’treallymakeachangethatmuch. Facilitator: So,hereisthedatatablehere. Usability27:Istilldidn’tlookatthedatatable.Ilookedatthegraphs. Facilitator: Youstilllookedatthegraphs? Usability27:Yeah. Facilitator: So,youarelookingatwhetherthe--Ithinkyoulookatthisgraphalot,
right? Usability27:Yeah. Facilitator: Youarelookingwhethertherootwormeggsaregrowingordecreasing. Usability27:Because in thisone, it increased(00:43:37) likewenta littlebitand
wentbackup. Facilitator: Stillincreased? Usability27:Yeah. Facilitator: Okay.Andforthe--didyou--canyouexplainwhydoyouthinkthe
harvestmenmethodsdidn’twork? Usability27:BecauseIdon’tthinktheharvestmenworkedbecausehere,ifyoulook
at--Ilookedatthisonealotbecauseitstartedupthishigh,butthenitlookslikeit’sslowlystartingtodecrease.
These types of behaviors (diagrammed in Figure 4.6) resulted in constructed responses
which were based on changes in the population of rootworms, or on changes in the
populationofcorn,butnotboth(correspondingtoalevel1scoreintherubric).
Figure4.6DiagramofActivityDisruptionDuringStudentInteractionwithSimulation
60
Disruption 3: Salience of Systemic Relationships. The final disruption pattern
(depictedinFigure4.7)camewhenstudentslackedrelevantpriorknowledgetoengagewith
keytaskcomponentsintheireverydaylifeexperiences.Muchasoccurredindisruption1,
studentswereunable todescribeecosystemfunctioning in termsofsystemrelationships
(like predation or energy usage); instead they resorted to accounts, best described as
laypersonterms.Forexample,UsabilityParticipant26describeshowtheharvestmenscare
awaytherootworms…
Facilitator: Okay, perfect. We’re moving next to the two control methods, theharvestmenand the alfalfa. This video and this videowas about theharvestmen.Basedonthedata,whatisthemaintakeawayfromthissituation?
Usability26:Thattheharvestmen,theyhelpedbutnotthatmuchbecauseIfeellikeifthere(00:47:30)hadmoreharvestmenandwehadmoreharvestmeneggslikethatthenthatwouldkindofhelpbecausetheywereonlyliketenandthenumberofeggswereincreasing.ItwashardfortheharvestmentogotoeachandhelpthecornsoIfeelthatifwehadmoreharvestmen,thatwouldhavebeeneasierandwehavehadmorecornharvested.
Facilitator: Howdotheharvestmenhelpthecorn? Usability26:I thinktheyhelpedbecauseI thinktheykindofscareawaythecorn
rootwormsbecausethey’rebiggerorlike,yeah.
While such lay accounts could sometimes allude to complex interaction dynamics
between the corn, the rootworm, and the control strategy, more often the result was
explanations that misread available information. Usability Participant 1 inaccurately
describesthetrapcropmethodasineffective,because“Thealfalfaincreasedtherootworms
morejustbyprovidingmoreshelterforthemtoplanttheireggs.”
61
Figure4.7DiagramofActivityDisruptionDuringStudentConstructedResponses
62
Chapter5
FINDINGSandDISCUSSION
Thedesign-basednatureofmyresearchquestions ledmetoadoptamethodology
akintootherdesign-basedresearch(Brown,1992)byblendingquantitativeandqualitative
elements.Mirroringthe“analyticspiral”describedbyCreswell(2009,p.183),myqualitative
data served as a means of phenomenologically validating the activities that produced
quantitativefindings,whilethequantitativedataservedasatooltoidentifyandtriangulate
newavenuesofqualitativeanalysis.Morespecifically,thequalitativedatafromthethink-
aloudsandreflectiveinterviewshelpedtosupporttheclaimthatstudentswereusingthe
relevantconstructsintheirreasoningwhilethequantitativedatafromstudentperformance
ontheassessmenthelpedidentifytargetassessmentitemsinneedofrevision;andtarget
students for whom a deeper analysis of their statements would be most fruitful. This
spiralinganalysishelpedidentifyplaceswheretheassessmentcouldberevisedaswellas
providedsomeguidanceforthedevelopmentofbrand-newitems.Inthefollowingsections,
Idiscusstheresultsofmystudybyresearchquestioninrelationtothisoverarchinggoalof
implementingCCC-consciousassessment.
ResearchQuestion1
IsthereevidencethatthecrosscuttingconceptsoftheNGSSaredistinctconstructsthatcanbemeasuredasstudentsusethem?
The highlight finding from the CFA used for Research Question 1 was the
indistinguishable-from-zerocorrelationbetweentheStructureandFunction(StFinFigures
4.1and4.2)andtheSystemsandSystemModels(SyMinFigures4.1and4.2).Theremaining
resultsfromtheCFAareonlymoderate,withsomequestionshavingmuchbettercorrelation
63
totherelevantCCCconstructthanothers.Whilethep-valueofthechi-squaregoodnessoffit
was much greater than 0.05, the root-mean-squared-error of approximation (Mueller &
Hancock,2015)wasjustbarelywithintheacceptablerangewitha90%confidenceinterval
of[0,0.11].Overall,theitemswerebettercorrelatedandmorereliableintheSystemand
System Models CCC than in Structure and Function. Overall, these findings point to two
implicationsfortheDAT-CROSSassessment:a)moreitemsareneededinordertoimprove
intra-construct reliability, and b) understanding of how to design systems-reasoning-
oriented assessment exceeds understanding of how to do so for assessments oriented
towardsStructure-and-Function-basedreasoning.
Implicationa)isbeingaddressedbydevelopingtwonewsetsofitemsunderaparallel
designprocessaswasusedfortheusabilitystudy’sset.Thesetwonewsetsaredesignedto
usetherelevantCCCsinsimilarwaysbutinnewcontexts(hydrologicsystemsandhuman-
bodysystems).Furtherdiscussiononthedevelopmentofthenewitemsappearslaterinthis
chapter.
Implicationb)reflects, inmyestimation,thecurrentstateofthescienceeducation
fieldonStructureandFunctioncomparedtoSystemsandSystemsModels.Itiscommon,in
existing learning progression literature, to see phenomena (and the learning thereof)
portrayedasasetofrelationshipsthatorganizeintoasystematsufficientlysophisticated
levelsofstudentreasoning(Breslynetal.,2016;Gunckeletal.,2012;Jin&Anderson,2012:
Mislevy,2016;Mohanetal.,2009;Songeretal.,2009).Thesephenomenacanspanmany
relevantdisciplines,fromgeoscience(Breslynetal.,2016)toecology(Hokayem,Ma,&Jin,
2015;Jin&Anderson,2012)tohydrology(Gunckeletal.,2012)andbeyond(Mislvey2016),
facilitatingtransferintonewcontextsorcontentareas.Incontrast,StructureandFunction
64
has not been the focus of asmuch science-education literature. The outcomeof the CFA
reveals thatwhile the design framework can design items that are Systems and Systems
ModelsorientedandcandesignitemsthatarenotSystemsandSystemsModelsoriented,more
revisionisnecessarytotransformthenotSystemsandSystemsModelsconstructintoonethat
canmoredirectlyelicitsobservationspertainingtoStructureandFunction.
ResearchQuestion2
Whatdoesstudentdemonstrationsofthecrosscuttingconceptualreasoningaspectoftheirthree-dimensionalscienceunderstandings look likeatdifferent levelsofoverallscienceunderstanding?
OneofthemainaffordancesoftheRaschmodelusedtoanalyzestudentperformance
istheabilitytoplacestudentsalongadimensionofabilitythatisofthesamescaleasthe
dimensionofitemdifficulty(Briggs&Alonzo,2012).Thatallowsmetoestimatestudent’s
position along the hypothesized learning progression (described in Appendix B). In the
followingsubsections Idiscussstudent think-aloudresponses toaselectedStructureand
FunctionitemandaselectedSystemsandSystemModelsitemforstudentswhohavethebest,
theworst,andthemedianabilityscoreinthecorrespondingconstruct.Returningtothefour
rolesofCCCs(Weiseretal.,2017)fromtheliteraturereview,Ianalyzehowtherolesselected
byeachstudentdifferedastheyrespondedtothesameitem.
StudentResponsesatDifferentStructureandFunctionProgressionLevels.
Question5fromtheDAT-CROSSassessment(Figure5.1)wasthefinalclickanddragitem,
in which students amend a prior model of the farm to account for the introduction of
rootwormstotheecosystem. Itwasalsomodeledtohaveadifficultyscoreofabout1.14
logits,makingitoneoftheharderitemsintheStructureandFunctionset.Theinteractive
elementsofthetaskmakeitaproductiveavenueforexaminingthink-aloudresponses.
65
Figure5.1DAT-CROSSQuestion5Note:QuestionKey:Sun->Corn;Corn->Goat;Corn->Rootworm
HighStructureandFunctionability(Usability32).Usability32wasa10thgrader
whohad thehighestmodeledability score at about2.34 logits (corresponding to a75%
chanceofgettingQuestion5correctviatheRaschequation[Equation2.1]).Hereisthem
thinkingaloudwithartifactillustratedinFigure5.2.
Usability32:So,nowthecornrootwormswillaffecttheamountofcornsurviveuntilharvest.Tomakeonefoodwebofcorn(00:07:26)corngoestothesunandthecornrootworms.Thesunpowerthecorn,sothesunisreallyfaraway,soit’sgottobeup(00:07:37).Andthisgoatandthisrootworm,they’rebothnext.Sothecornisgoinginto the goat, but this corn is also going into the rootworm, so, that makes theproblem.In this response,Usability32 can correctlydescribe the teleological functioneach
entityplayed(eitherbeneficiallyordetrimentally)andusetheirunderstandingofadiscrete
setofrelationshipstobuilduptothewholestructure.Fromtheperspectiveofthefourroles
of CCC reasoning (Weiser et al., 2017), this is an example of ‘CCCs as Levers’ employed
successfully.
66
Figure5.2Usability32'sresponsetoDAT-CROSSQuestion5
Interestingly, while Usability 32 had the best performance on the Structure and
Functionitems(Questions2through8),theirperformanceonthelater,SystemsandSystems
Models items (Questions 9 through 13) were middling at best – with all three of their
constructedresponsesscoringa1basedonthescoringrubric.
Median Structure and Function ability (Usability 41). Usability 41 was an 8th
graderwhohadthemedianmodeledabilityscoreatabout0.74logits(correspondingtoa
40%chanceofgettingquestion5correctvia theRaschequation [Equation2.1]).Here is
themreflectingontheirresponsewiththeirartifactillustratedinFigure5.3:
67
Figure5.3:Usability41'sresponsetoDAT-CROSSQuestion5
Usability41:Iknewrootwormwould--likethatmorepeoplewereusingthecorn.Itwasliketakingcornawayfromtheotherpeoplewhowereusingit.Facilitator:Okay.So,youthinkrootwormistakingcornaway?Usability41:Yeah.Itwaslikedecomposingit.Facilitator:Decomposingthat. Okay. So,isthisthemodelthatyoudrew?Orthearrowsaredifferent?Usability41:Ithinkthearrowsarealittledifferent.Facilitator:So,what’syourarrow? Yeah,yourdirection is fromrootwormto thecorn?Usability41:Yeah.Facilitator:Whatdoesthatmean?Usability41:Thattherootwormswerelikeeatingthecorn.Facilitator:Eatingthecorn.Okay.However,here,thecorntogoat,isalsogoateatingthecorn,whythearrowisdifferent?Usability41: I think Idid thatbecauseof thepreviousone. I’dmeantadifferentthing.
Rather than construct a coherent structure inwhich the arrows alwaysmean the
same thing, Usability 41 uses his ‘CCC as a Bridge’ (Weiser et al., 2017) –with discrete
connectionsbetweenrelevantentitiesthataresensible(giventheircontentknowledge)only
68
asasetofpieces.Identificationofdiscreterelationshipsisacommonthemeatlowerstages
ofmanyLPs(Mislevy,2016)includingmyhypothesizedone(AppendixB).
LowStructureandFunctionability(Usability7).Usability7wasa6thgraderwho
hadthelowestmodeledabilityscoreatabout-1.21logits(correspondingtoa9%chanceof
getting question 5 correct via theRasch equation [Equation 2.1]).Here is them thinking
aloudwithartifactillustratedinFigure5.4.
Figure5.4Usability7'sresponsetoDAT-CROSSQuestion5
Usability 7: I think that it reflects your relationships between – the feedingrelationshipsbecauseifyoudon’thavethesun,thenyoucan’thavecornandifyoucan’tcorn,thenthere’snothingtofeedthegoatssotheywoulddie.So,ifoneisifyoudon’thaveone,theotheronewoulddieornotdowell.
In theirmodel,Usability 7 seems to develop an organizing principle, that the sun
needstoexisttosupporttheotherorganisms,butthenstrugglestocoordinatethatprinciple
69
intoacoherentmodel.Interestingly,underthefour-roleframework(Weiseretal.,2017),
thisparticipant’sbehavior isassociatedwithusingthe ‘CCCasaRule,’which(inthetask
model)wasplacedatthehigherendofthelearningprogression.Still,inthecontextofthe
question, itwas inappropriateandledthestudenttocreateamodel inwhichthearrows
meantdifferentthingsacrossdifferententities.
StudentResponsesatDifferentSystemsandSystemModelsProgressionLevels.
Question 10 from the DAT-CROSS assessment (Figure 5.5, below) was the first
constructedresponseitem,inwhichstudentswereaskedtoexplainwhythecontrolstrategy
inwhichpredatorswereintroducedtopreyontherootwormswasnoteffective.
Figure5.5DAT-CROSSQuestion10
Asaconstructedresponse,Question10wasscoredbyarubric(AppendixH)meaning
thateachpotentialscore(from0to3points)isassociatedwithadistinctdifficultymeasure.
Specificallya1-pointscorehadadifficultyof-2.31logits,a2-pointscorehadadifficultyof
0.47logits,andthetopscore(3points)hadadifficultyof1.55logits.Theconstructednature
of students’ responses entails ‘intentionality’ (Jonassen & Rohrer-Murphy, 1999), which
makethemaproductiveavenueforthinkingaboutstudentactivity.
70
HighSystemsandSystemModels ability (Usability20).Usability20wasa10th
graderwhohadthehighestmodeledabilityscoreatabout2.19logits(correspondingtoa
99%,85%,and65%chanceofscoring1,2,or3-points,respectively,viatheRaschequation
[Equation2.1]).Hereistheirconstructedresponse:
Usability20:Thereisthesamenumberofharvestmenthroughyears3to5buttherootwormeggsstillincreasedinnumberoverall.Iftheharvestmencouldonlykeep91 cornstalks harvestable in year 3, they are not going to keep more cornstalksharvestableiftheyhavemorerootwormeggstodealwithinthenextyears.
Usability20’sabilitytodetectatrendinthesystemand,fromthattrend,predictafuture
stateisindicativeofthe‘CCCasaRule”role(Rivetetal.,2016)usedsuccessfully.
MedianSystemsandSystemModelsability(Usability45).Usability45wasa10th
graderwhohadthemedianmodeledabilityscoreatabout0.01logits(correspondingtoa
91%,39%,and18%chanceofscoring1,2,or3-points,respectively,viatheRaschequation
[Equation2.1]).Hereistheirconstructedresponse:
Usability45:AddingtheHarvestmeninthefeildeveryyeardidn'tchangethecornyeildbecasuetheynumberofharvestmenstayedthesameandtherootwormsonlyaddaptedtotheirnewenvirnment.Theharvestmennumberstayedthesamewhilethe rootworms had eggs each year and only continued this cycle. So because theharvestmenwereaddedinsteadofincreasingcornyeild,thecornyeildonlydecresedlessrapidly{sic}.
IncontrasttoUsability20’sabilitytoconstructorganizingprinciplesthatdictatedthestate
ofthefarmecosystem,Usability45makestheCCCactasalensbyidentifyingrelevantparts
thatcouldhelpsupportanexplanation.Whilethatrolewassuccessfulinidentifyingevidence
relevanttotheirclaim,weseethattheirresponsehasnounderlyingreasoninganddescribed
nomechanismbywhichtheharvestmenandtherootwormsarerelatedtoeachother.
71
Low Systems and SystemModels ability (Usability 36). Usability 36 was a 6th
graderwhohadthelowestmodeledabilityscoreatabout-3.75logits(correspondingtoa
19%,1%,and0.5%chanceofscoring1,2,or3-points,respectively,viatheRaschequation
[Equation2.1]).Hereistheirconstructedresponse:
Usability36:BecausethereweretoomanyRootwormsfortheHarvestMen{sic}tofightoff.
These sorts of partial responses are commonplace both in our assessment and in past
assessmentsinvolvingstudentexplanations(Beck,Bookbinder,Lee,&Rivet,2017)inwhich
somerelevantclaimismadewithoutcitinganymeaningfulevidenceorprovidingsufficient
reasoning.
ResearchQuestion3
What challenges do middle-school-aged subjects face in presenting their CCCunderstandingswhileengagingwithan interactivesuitedesignedtotargetSystems-Models and Structure-Function dimensions of their three-dimensional scienceunderstandings?RoleSelectionandDisruptionsinStudentActivity.
FromthediscussionofthefindingsforResearchQuestion2,itseemsthatwhilethe
Roles(Rivetetal.,2016)donotalwaysaligntolearningprogressionlevels(assupposedby
the taskmodel [Appendix A]), students’ selection of Role does influence their ability to
construct explanations for phenomena. This is consistent with my findings from prior
research (Weiser et al., 2017). It stands to reason, given the influenceof ‘Role Selection’
(Weiseretal.2017),toconceptualizethedisruptionsfromtheresultschapterinthelightof
the4Rolesandwhichoneshadbeenselectedwhendisruptionsoccurred.
Disruption1.Indisruption1,Ifoundthatstudents,despiteinteractingwiththefood-
web model during questions 3-6, rarely carried ideas from that set of tasks into their
72
explanationsfortheefficacyoftherootworm-infestation-mitigatingstrategies.Instead,they
wouldfocusonlyonthecurrentmaterialavailableonscreen,identifywhatmatteredfrom
that set, and attempt to construct an explanation from there (often missing out on the
relationships,mechanisms,orreasoningrequired).Here,studentsusedtheirCCCsaslenses
toidentifycomponentstheyfoundrelevanttothetask,butthenstruggledinbridgingthose
components together. This behavior is consistent with prior research regarding student
viewsonbehaviorsandfunctions(Hmelo-Silver,Liu,&Jordan,2009)inwhichstudentscan
make sense of components and the individual behaviors of components before they can
describehowthosebehaviorsimpactotherentities.
Disruption2.Indisruption2,studentsfailednotonlytocarryideasfrompastscreen
intolateritems,theyalsofailedtociteevidencethatwaspresentrightinfrontofthem.The
bestexplanationforthisdisruptionwasthattheydidnothavesufficientpriorknowledge
aboutrelevantsciencepractice,andbyextensioncouldnotdeterminehowthatcontentwas
meant toserveasevidence for theirexplanations.Fromaroleselectionperspective, this
disruption underscores the three-dimensionality of science understanding. Even when
studentsareabletoselectanappropriaterolefortheCCCs(e.g.,tousetheirunderstanding
of thesystemtoconstructhierarchy intherelationshipsofsystemcomponents),without
sufficientunderstandingoftherelevantdisciplinarycoreideasorscienceandengineering
practice,itisnotpossibletoperformtotheexpectationofthestandards.
Disruption3.Disruption3wasacounterpointtodisruption2.Wheredisruption2
occurred when students failed to cite evidence on screen, disruption 3 occurred when
evidence was cited but the explanation constructed followed a lay account of the
phenomenon. This kind of behavior is common when students have developed
73
sophistication in theirCCCsandSEPsbefore theyhavedoneso in theirDCIs(Becketal.,
2017).InUsability26’sresponsetoQuestion10(asdescribedinChapter4),theyareableto
effectivelyusetherightrole,CCCasLever,toconstructasophisticated,albeitlay,accountof
thepredationstrategy.The linguisticsuccessofUsability26,contrastedwith thecontent
failing,revealsthehowimportantlanguageistotheCCCs.Asnotedintheliteraturereview,
the CCCs, while important in unique ways to the scientific domain, also play out in the
everyday senseofpattern, causeandeffect, systems, etc.That is, studentsdevelop these
conceptsbothincoordinationwiththeirlearningexperiencesinthescienceclassroomand
intheireverydayexperienceswhentheymightnotbeattendingtoothersciencecontent.
MakingAmendmentstotheEcosystemTaskStoryboard
Followingmyanalysison theusabilityof theexistingDAT-CROSS items, revisions
wereput forth toaddressusageconcerns fromResearchQuestion3andmakeconstruct
elements(asdescribedinthehypothesizedlearningprogression)moresalient,particularly
intheitemsmeanttotargetStructureandFunction.Additionalhelpertextwasincludedin
Questions 2 and 7 tomake clear that students should click onmore than one option as
appropriate.Language,regardingtheideathatorganismsonthefarmserveafunctioninthe
overallfunctioningofthefarmyardecosystem,totextonscreenbetweenquestions2and3
andbetweenquestions3and4.Thattextprovidestheanswerstopreviousquestionssothat
all students have the necessary content knowledge even if they responded to previous
questionsincorrectly).Laterintheassessment,moreemphasiswasplacedonmakingsure
studentsunderstoodwhythe farmerwas introducingthedifferentcontrolstrategiesand
whatheexpectedeachstrategytodo.
74
ImplicationsfortheDesignofTasksinNewSystemContexts
InadditiontoinformingrevisionstotheexistingitemsofDAT-CROSS,myresearch
sought to inform larger design ideas that could affect future CCC-targeting assessment
developmentwork.InthelightoftheIESgrantandthefindingsfromResearchQuestion1,
newitemsweredesignedtoimproveintra-constructreliabilitywhilealsoaccountingforthe
‘cuttingacrossdisciplines’(NRC,2014,p.37)natureofthecrosscuttingconcepts.Fromthe
findingsofResearchQuestion2, itwas clear that studentsneededgreater in-assessment
supportforthinkingaboutthefunctionsofstructuralcomponentsandhowtheysupportthe
overallgoalofastructure.Thesesupportsnotonlyincludemorescreensbetweenquestions
to introduce content, but also questions that are targeted to the function and behavior
progressionpathwaysfromthelearningprogression(AppendixB).Figure5.6exemplifies
oneofthesenewquestions.
Tominimizeusagedisruptionsacrossthewholesetoftasks,improvedsupporttext
andinstructionalvideosweredevelopedtohelpexplainhowvariousinteractiveelements
worked.
75
Figure5.6ExampleQuestionfromWaterUseSystemStoryline
LimitationstoMyFindings
Myresearchforthisdissertationcamefromonlyasmallsliceofabroaderongoing
project, asusability testing is often the first step in theprocessof assessmentvalidation
(Zaharias & Poylymenakou, 2009). Unlike later stages, which may involve many more
students (and accordingly, less rich data), usability tests involve fewer participants (my
study had 45middle-school-aged students) thatmay be selected on amore intentional,
nonrandomcriteria.ThechoicetousewhatIhavepreviouslycalleda“bestcase”sample,
particularlyinlightoftheframeworkofactivitytheory,impactssomedistinctlimitationson
thegeneralizabilityofmyfindingsregardingtheexactusageproblemsallstudentswillface
wheninteractingwiththeDAT-CROSSSuite.Forexample,oneofthehighlightfindingsfrom
theuseofactivitytheorytoinvestigateresearchquestion3waspreliminaryevidencethat
manystudentshavenaïve,proto-conceptionsaboutcrosscuttingconceptsthathavebuiltup
76
inanevolutionaryreasoningstyle(asdescribedbyOsborneetal.,2018).Theseprimitive
notionsregardingstructuresandtheirfunctionsseempresentinthewordsthatstudents
usetodescribephenomenawheresuchconceptsareusefultowardsmakingmeaningofthe
world,perhapsservingasausefulleverforteachers.However,itisverylikelythatthe“best
case” studentswho participatedwere alsomost likely to attempt to vocalize even their
underdevelopedideaswithoutfearofjudgementbytheirpeers(whowerenotpresentin
the testing setting) or the facilitating adult. Indeed, suchwillingnesswas found in other
investigationsofCCCreasoningwherelikely-affluentpopulationsservedassubjects(Beck
etal.,2016).Incontrast,studentsofcoloroftenstruggletoputforththeirprimitiveideasfor
fear of the teacher-student and student-student interactions that can manifest (Brown,
2004).Itmaybethecaseinfutureuseswithmorevulnerablepopulationsofstudents,that
thesevocalizationsdonotoccur,hinderingeducators’abilitytocapitalizeonthem.
Anotherlimitationisstudents’baselinecontentfamiliarity.Giventherecentnessof
the NGSS (NGSS Lead States, 2013), few students have had any explicit andmeaningful
instruction regarding the crosscutting concepts. Thus, the preliminary evidence for a
learningprogressionseenfromtheresultofresearchquestion2canbebestthoughtofasa
progressionofinformallearninginthatallthelearningexperiencesrelatedtothecontent
and phenomena entailed in the progression have likely only occurred in informal, lived
experiencesettings.Studentsofsimilarlylittleformallearningopportunitiesregardingideas
likesystemsorstructureandfunctionbutofdifferentlivedexperiencesmayprogressalong
averydifferenttrajectory.Itis,therefore,crucialtounderstandmyfindingsonlyasaproof-
of-conceptforassessingCCCs.
77
Chapter6
CONCLUSION
WithintheNGSS, thesalienceofDCIsandSEPsoftenoverwhelmsthevalueof the
CCCs in displaying expected performance. Even among current research in developing
assessment for the NGSS (DeBarger et al., 2015), it is rare to see rubrics that value the
presence of CCCs in student reasoning beyond a binary condition. However, previous
research findings suggest that students’ explanationswere influencednot only bywhich
phenomenatheywereaskedtoexplain,butalsobytheCCCstheywereaskedtouseandthe
wording on the questions they were asked (Weiser et al., 2017). Simultaneously, the
languageoftheNGSSseemstoemphasizestudentsdoingoverstudentsknowing(NGSSLead
States,2013a;NRC,2012).Thescienceeducationcommunityneedsnewassessmentsthat
arebothdoing-orientedandcapableofdrawingevidenceofstudentunderstandingacross
allthreedimensionsofthescience-learningframework(Duschl,2008).Tothisend,activity
theory, when coupled with quantitative metrics, can be a powerful analytic tool in
understanding and improving the kind of work dynamics (Engestrom, 2000) seemingly
desiredbytheNGSS(Duschletal.,2007;NRC,2012).Insettinganovelobservationmodel
forelicitingevidencefromstudents,theATframeworkprovidesteacherandstudentalike
torecognizeopportunitiesforexpressionofthree-dimensionalunderstandingsthatmaynot
beclearlyavailabletothemforpracticalorsocio-culturalreasons(vanEs&Sherin,2002;
Farenga & Joyce, 1997; Jonassen & Rohrer-Murphy, 1999). As I described in previous
sections,activitytheoryfitswellintothedesignofassessments,especiallyonesdesignedto
measuresummativeknowledgeinteractively(Pea,1993)ortoprovideformativelearning
opportunitiestostudents(Black&Wiliam,2009).However,thetaskofaligningsystem-wide
78
assessmentstotheNGSSremains.Ibelievethatactivitytheory,asamethodfordeveloping
evidencemodelsforassessments(Mislevy&Haertel,2007)hasmerit,butworkstillneeds
tobedonetodevelopsuchmodelsatappropriatescaleforthispurpose.
79
References
Abd-El-Khalick,F.,Bell,R.L.,&Lederman,N.G.(1998).Thenatureofscienceandinstructionalpractice:Makingtheunnaturalnatural.ScienceEducation,82(4),417–436.doi:10.1002/(SICI)1098-237X(199807)82:4<417::AID-SCE1>3.0.CO;2-E
Alonzo,A.C.,&Steedle,J.T.(2009).Developingandassessingaforceandmotionlearningprogression.ScienceEducation,93(3),389–421.doi:10.1002/sce.20303
AmericanAssociationfortheAdvancementofScience(AAAS).(1993).Benchmarksforscienceliteracy.AdvancementOfScience.NewYork:OxfordUniversityPress.Retrievedfromhttp://www.project2061.org/publications/bsl/online/index.php
Barab,S.,&Squire,K.(2004).Design-basedresearch:Puttingastakeintheground.JournaloftheLearningSciences,13(1),1–14.doi:10.1207/s15327809jls1301_1
Baxter,G.P.,&Mislevy,R.(2004).TheCaseforanIntegratedDesignFrameworkforAssessingScienceInquiry.LosAngeles:TheNationalCenterforResearchonEvaluation,Standards,StudentTesting(CRESST),CenterforStudiesinEducation,UCLA.
Beck,A.,Bookbinder,A.,Lee,M.J.,&Rivet,A.E.(2017,April).IdentifyingEarlyProductiveSteppingStoneConceptionsofThree-dimensionalEarthScienceUnderstandingbyHighSchoolStudents.PaperpresentatNARST,SanAntonio.
Benson,L.,Elliott,D.,Grant,M.,Holschuh,D.,Kim,B.,Kim,H.,…Reeves,T.C.(2002).Usabilityandinstructionaldesignheuristicsfore-learningevaluation.InP.Barker&S.Rebelsky(Eds.),ED-MEDIA2002--WorldConferenceonEducationalMultimedia,Hypermedia&Telecommunications(pp.1615–1621).Denver,CO:AssociationfortheAdvancementofComputinginEducation(AACE).Retrievedfromhttps://eric.ed.gov/?id=ED477005
Berland,L.K.,&McNeill,K.L.(2010).Alearningprogressionforscientificargumentation:Understandingstudentworkanddesigningsupportiveinstructionalcontexts.ScienceEducation,94(5),765–793.doi:10.1002/sce.20402
Berland,L.K.,&Reiser,B.J.(2009).Makingsenseofargumentationandexplanation.ScienceEducation,93(1),26–55.doi:10.1002/sce.20286
Blumenfeld,P.,Fishman,B.J.,Krajcik,J.,Marx,R.W.,&Soloway,E.(2000).CreatingUsableInnovationsinSystemicReform :ScalingUpTechnology-EmbeddedProject-BasedScienceinUrbanSchools.EducationalPsychologist,35(3),149–164.doi:10.1207/S15326985EP3503
80
Breslyn,W.,McGinnis,J.R.,McDonald,R.C.,&Hestness,E.(2016).Developingalearningprogressionforsealevelrise,amajorimpactofclimatechange.JournalofResearchinScienceTeaching,53(10),1471–1499.doi:10.1002/tea.21333
Bricker,L.A.,&Bell,P.(2008).Conceptualizationsofargumentationfromsciencestudiesandthelearningsciencesandtheirimplicationsforthepracticesofscienceeducation.ScienceEducation,92(3),473–498.doi:10.1002/sce.20278
Briggs,D.C.,&Alonzo,A.C.(2012).Thepsychometricmodelingoforderedmultiple-choiceitemresponsesfordiagnosticassessmentwithalearningprogression.InA.C.Alonzo&A.W.Gotwals(Eds.),LearningProgressionsinScience(pp.293–316).Rotterdam:SensePublishers.doi:10.1007/978-94-6091-824-7_13
Brown,A.L.(1992).DesignExperiments:TheoreticalandMethodologicalChallengesinCreatingComplexInterventionsinClassroomSettings.JournaloftheLearningSciences,2(2),141–178.doi:10.1207/s15327809jls0202_2
Brown,B.A.,Reveles,J.M.,&Kelly,G.J.(2005).Scientificliteracyanddiscursiveidentity:Atheoreticalframeworkforunderstandingsciencelearning.ScienceEducation,89(5),779–802.doi:10.1002/sce.20069
Brown,J.S.,Collins,A.,&Duguid,P.(1989).Situatedcognitionandthecultureoflearning.EducationalResearcher,18(1),32–42.doi:10.3102/0013189X018001032
Clarke,J.,&Dede,C.(2009).DesignforScalability:Acasestudyoftherivercitycurriculum.JournalofScienceEducationandTechnology,18(4),353–365.doi:10.1007/s10956-009-9156-4
Cobb,P.,Zhao,Q.,&Dean,C.(2009).ConductingDesignExperimentstoSupportTeachers’Learning:AReflectionFromtheField.JournaloftheLearningSciences,18(2),165–199.doi:10.1080/10508400902797933
CognitionandTechnologyGroupatVanderbiltUniversity.(1992).Anchoredinstructioninscienceandmathematics:Theoreticalbases,developmentalprojects,andinitialresearchfindings.InR.A.Duschl&R.J.Hamilton(Eds.),Philosophyofscience,Cognitivepsychology,andEducationaltheoryandpractice(pp.244–273).NewYork:SUNYPress.
Cohen,E.(1998).Makingcooperativelearningequitable.EducationalLeadership,56(1),18–21.Retrievedfromhttp://web.b.ebscohost.com.ezproxy.elib10.ub.unimaas.nl/ehost/pdfviewer/pdfviewer?sid=d7594b1a-abc6-4dfd-9d76-a2cf6bc85224@sessionmgr115&vid=1&hid=101
81
Cohen,E.G.,&Lotan,R.A.(1995).ProducingEqual-StatusInteractionintheHeterogeneousClassroom.AmericanEducationalResearchJournal,32(1),99–120.doi:10.3102/00028312032001099
Corcoran,T.B.,Mosher,F.A.,&Rogat,A.(2009).LearningProgressionsinScience:AnEvidence-BasedApproachtoReformLearningProgressionsinScience:AnEvidence-BasedApproachtoReform.CPREResearchReports.Retrievedfromhttp://repository.upenn.edu/cpre_researchreports
Creswell,J.W.(2009).QualitativeInquiryandResearchDesign:ChoosingAmongFiveApproaches.Book(3rded.,Vol.2nded).LosAngeles:SAGEPublications.doi:10.1177/1524839915580941
Cronbach,L.J.(1951).Coefficientalphaandtheinternalstructureoftests.Psychometrika,16,297-334.
deAyala,R.(2010).Itemresponsetheory.InG.R.Hancock,R.O.Mueller,&L.M.Stapleton(Eds.),TheReviewer’sGuidetoQuantitativeMethodsintheSocialSciences(p.155).NewYork,NY:Routledge.
DeBarger,A.H.,Penuel,W.R.,Harris,C.J.,&Kennedy,C.A.(2016).BuildinganassessmentargumenttodesignanduseNextGenerationScienceassessmentsinefficacystudiesofcurriculuminterventions.AmericanJournalofEvaluation,37(2),174–192.doi:10.1177/1098214015581707
DeBoer,G.E.(1991).Ahistoryofideasinscienceeducation:Implicationsforpractice.NewYork:TeachersCollegePress.
DiSessa,A.A.,Cobb,P.,&Andrea,A.(2004).Ontologicalinnovationandtheroleoftheoryindesignexperiments.TheJournaloftheLearningSciences,13(1),77–103.doi:DOI10.1207/s15327809jls1301_4
Duncan,R.G.,&Rivet,A.E.(2013).ScienceLearningProgressions.Science,339(6118),396–397.doi:10.1126/science.1228692
Duncan,R.G.,&Tseng,K.A.(2011).Designingproject-basedinstructiontofostergenerativeandmechanisticunderstandingsingenetics.ScienceEducation,95(1),21–56.doi:10.1002/sce.20407
Duschl,R.(2008).ScienceEducationinThree-PartHarmony:BalancingConceptual,Epistemic,andSocialLearningGoals.ReviewofResearchinEducation,32(1),268–291.doi:10.3102/0091732X07309371
Duschl,R.(2012).Theseconddimension—Crosscuttingconcepts.ScienceandChildren,79(2),10–14.
82
Duschl,R.,Maeng,S.,&Sezen,A.(2011).Learningprogressionsandteachingsequences:areviewandanalysis.StudiesinScienceEducation,47(2),123–182.doi:10.1080/03057267.2011.604476
Duschl,R.,Schweingruber,H.A.,&Shouse,A.W.(2007).Takingsciencetoschool:LearningandteachingScienceingradesK-8.TakingSciencetoSchool.Washington,D.C.:NationalAcademiesPress.doi:10.17226/11625
Edelson,D.C.(2001).Learning-for-Use:Aframeworkforthedesignoftechnology-supportedinquiryactivities.JournalofResearchinScienceTeaching,38(3),355–385.doi:10.1002/1098-2736(200103)38:3<355::AID-TEA1010>3.0.CO;2-M
Eger,M.(1992).Hermeneuticsandscienceeducation:Anintroduction.ScienceandEducation,1(4),337–348.doi:10.1007/BF00430961
Eggert,S.,&Bögeholz,S.(2010).Students’useofdecision-makingstrategieswithregardtosocioscientificIssues:Anapplicationoftheraschpartialcreditmodel.ScienceEducation,94(2),230–258.doi:10.1002/sce.20358
Eisenhart,M.,Finkel,E.,&Marion,S.F.(1996).Creatingtheconditionsforscientificliteracy:Are-examination.AmericanEducationalResearchJournal,33(2),261–295.doi:10.3102/00028312033002261
Elam-Respass,T.(2018).EffectiveInstructionalPracticesthatEngagetheAfricanAmericanMaleinMiddleSchoolScience.UniversityofMaryland,CollegePark.
Engestrom,Y.(2000).Activitytheoryasaframeworkforanalyzingandredesigningwork.Ergonomics,43(7),960–974.doi:10.1080/001401300409143
Farenga,S.J.,&Joyce,B.A.(1997).WhatChildrenBringtotheClassroom:LearningScienceFromExperience.SchoolScienceandMathematics,97(5),248–252.doi:10.1111/j.1949-8594.1997.tb17270.x
Ford,M.(2008).Disciplinaryauthorityandaccountabilityinscientificpracticeandlearning.ScienceEducation,92(3),404–423.doi:10.1002/sce.20263
Freire,P.,&Macedo,D.P.(1995).Adialogue:Culture,langagueandrace.HarvardEducationalReview,65(3),377–403.doi:10.17763/haer.65.3.12g1923330p1xhj8
Furtak,E.M.(2017).Confrontingdilemmasposedbythree-dimensionalclassroomassessment:IntroductiontoavirtualissueofScienceEducation.ScienceEducation.doi:10.1002/sce.21283
Garver,M.andMentzer,J.(1999).Logisticsresearchmethod:Employingstructuralequationmodelingtotestforconstructvalidity.JournalofBusinessLogistics,20(1),33.https://doi.org/10.1002/(ISSN)2158-1592
83
Gorin,J.S.,&Mislevy,R.J.(2013).InherentmeasurementchallengesintheNextGenerationScienceStandardsforbothformativeandsummativeassessment.InInvitationalAssessmentSymposium(pp.2–39).Washington,D.C.
Gotwals,A.W.,&Songer,N.B.(2013).Validityevidenceforlearningprogression-basedassessmentitemsthatfusecoredisciplinaryideasandsciencepractices.JournalofResearchinScienceTeaching,50(5),597–626.doi:10.1002/tea.21083
Griffiths,A.K.,&Grant,B.A.C.(1985).Highschoolstudents’understandingoffoodwebs:Identificationofalearninghierarchyandrelatedmisconceptions.JournalofResearchinScienceTeaching,22(5),421–436.doi:10.1002/tea.3660220505
Gunckel,K.L.,Covitt,B.A.,Salinas,I.,&Anderson,C.W.(2012).Alearningprogressionforwaterinsocio-ecologicalsystems.JournalofResearchinScienceTeaching.doi:10.1002/tea.21024
Hallgren,K.A.(2012).Computinginter-raterreliabilityforobservationaldata:anoverviewandtutorial.TutorialsinQuantitativeMethodsforPsychology,8(1),23–34.https://doi.org/10.1016/j.biotechadv.2011.08.021.Secreted
Hammer,D.(1994).Epistemologicalbeliefsinintroductoryphysics.CognitionandInstruction,12(2),151–183.doi:10.1207/s1532690xci1202_4
Harris,C.J.,Krajcik,J.S.,Pellegrino,J.W.,&McElhaney,K.W.(2016).Constructingassessmenttasksthatblenddisciplinarycoreideas,crosscuttingconcepts,andsciencepracticesforclassroomformativeapplications.MenloPark,CA:SRIInternational.
Hewson,P.W.,&Hewson,M.G.A.B.(1984).Theroleofconceptualconflictinconceptualchangeandthedesignofscienceinstruction.InstructionalScience,13(1),1–13.doi:10.1007/BF00051837
Hmelo-Silver,C.E.,Liu,L.,&Jordan,R.(2009).Visualrepresentationofamultidimensionalcodingschemeforunderstandingtechnology-mediatedlearningaboutcomplexnaturalsystems.ResearchandPracticeinTechnologyEnhancedLearning,04(03),253–280.https://doi.org/10.1142/S1793206809000714
Hoadley,C.M.(2004).MethodologicalAlignmentinDesign-BasedResearch.EducationalPsychologist,39(4),203–212.doi:10.1207/s15326985ep3904
Hobson,A.(1993).OzoneandInterdisciplinaryScienceTeaching-LearningtoAddresstheThingsthatCountMost.JournalofCollegeScienceTeaching,23(1),33–37.
Hogan,K.(2000).Assessingstudents’systemsreasoninginecology.JournalofBiologicalEducation,35(1),22–28.doi:10.1080/00219266.2000.9655731
84
Hogan,K.,&Fisherkeller,J.(1996).Representingstudents’thinkingaboutnutrientcyclinginecosystems:Bidimensionalcodingofacomplextopic.JournalofResearchinScienceTeaching,33(9),941–970.doi:10.1002/(SICI)1098-2736(199611)33:9<941::AID-TEA1>3.0.CO;2-V
Hokayem,H.,Ma,J.,&Jin,H.(2015).Alearningprogressionforfeedbackloopreasoningatlowerelementarylevel.JournalofBiologicalEducation,49(3),246–260.doi:10.1080/00219266.2014.943789
Howell,H.,Phelps,G.,Croft,A.J.,Kirui,D.,&Gitomer,D.(2013).Cognitiveinterviewsasatoolforinvestigatingthevalidityofcontentknowledgeforteachingassessments.ETSResearchReportSeries,2013(1),i-97.https://doi.org/10.1002/j.2333-8504.2013.tb02326.x
Jin,H.,&Anderson,C.W.(2012).Alearningprogressionforenergyinsocio-ecologicalsystems.JournalofResearchinScienceTeaching.doi:10.1002/tea.21051
Joe,J.,Tocci,C.,Holtzman,S.,&Williams,J.(2013).Foundationsofobservation:Considerationsfordevelopingaclassroomobservationsystlemthathelpsdistrictsachieveconsistentandaccuratescores.METProjectPolicyandPracticeBrief:BillandMelindaGatesFoundation.Princeton,NJ.Retrievedfromhttp://www.metproject.org/downloads/MET-ETS_Foundations_of_Observation.pdf
Johnstone,C.J.,Bottsford-Miller,N.a.,&Thompson,S.J.(2006).Usingthethinkaloudmethod(cognitivelabs)toevaluatetestdesignforstudentswithdisabilitiesandenglishlanguagelearners.TechnicalReport44UniversityofMinnesota,NationalCenteronEducationalOutcomes.Retrievedfromhttp://www.cehd.umn.edu/nceo/OnlinePubs/Tech44/%5Cnhttp://education.umn.edu/NCEO/OnlinePubs/Tech44/
Jonassen,D.H.,Rohrer-Murphy,L.,&JonassenH.,D.(1999).Activitytheoryasaframeworkfordesigningconstructivistlearningenvironments.EducationalTechnologyResearchandDevelopment,47(1),61–79.doi:10.1007/BF02299477
Joseph,D.(2004).Thepracticeofdesign-basedresearch:uncoveringtheinterplaybetweendesign,research,andthereal-worldcontext.EducationalPsychologist,39(4),235–242.https://doi.org/10.1207/s15326985ep3904_5
Kanter,D.E.(2010).Doingtheprojectandlearningthecontent:Designingproject-basedsciencecurriculaformeaningfulunderstanding.ScienceEducation,94(3),525–551.doi:10.1002/sce.20381
Kelly,G.J.,McDonald,S.,&Wickman,P.-O.(2012).Sciencelearningandepistemology.InB.J.Fraser,K.Tobin,&C.J.McRobbie(Eds.),SecondInternationalHandbookof
85
ScienceEducation(pp.281–291).Dordrecht:SpringerNetherlands.doi:10.1007/978-1-4020-9041-7
Kerby,D.S.(2014).Thesimpledifferenceformula:anapproachtoteachingnonparametriccorrelation.ComprehensivePsychology,3(1),11.IT.3.1.https://doi.org/10.2466/11.IT.3.1
Kesidou,S.,&Roseman,J.E.(2002).Howwelldomiddleschoolscienceprogramsmeasureup?FindingsfromProject2061’scurriculumreview.JournalofResearchinScienceTeaching,39(6),522–549.doi:10.1002/tea.10035
Kilinc,A.,Demiral,U.,&Kartal,T.(2017).ResistancetodialogicdiscourseinSSIteaching:Theeffectsofanargumentation-basedworkshop,teachingpracticum,andinductiononapreservicescienceteacher.JournalofResearchinScienceTeaching,54(6),764–789.doi:10.1002/tea.21385
Kline,P.(1994).FactorAnalysisinTestConstruction.InAnEasyGuidetoFactorAnalysis(pp.125–139).NewYork:Routledge.
Krajcik,J.(2015).Three-dimensionalinstruction:Usinganewtypeofteachinginthescienceclassroom.TheScienceTeacher,82(8).
Krajcik,J.,&Merritt,J.(2012).EngagingStudentsinScientificPractices:Whatdoesconstructingandrevisingmodelslooklikeinthescienceclassroom?ScienceandChildren,49(7),10.
Krajcik,J.,Codere,S.,Dahsah,C.,Bayer,R.,&Mun,K.(2014).PlanninginstructiontomeettheintentoftheNextGenerationScienceStandards.JournalofScienceTeacherEducation,25(2),157–175.doi:10.1007/s10972-014-9383-2
Krajcik,J.,McNeill,K.L.,&Reiser,B.J.(2008).Learning-goals-drivendesignmodel:Developingcurriculummaterialsthatalignwithnationalstandardsandincorporateproject-basedpedagogy.ScienceEducation,92(1),1–32.doi:10.1002/sce.20240
Lakoff,G.,&Johnson,M.(1997).MetaphorsWeLiveBy.Theproductionofreality:essaysandreadingonsocialinteraction(2nded.).Chicago:UniversityofChicagoPress.
Lave,J.,&Wenger,E.(1991).Situatedlearning:Legitimateperipheralparticipation.NewYork:CambridgeUniversityPress.Retrievedfromhttps://books.google.co.uk/books?id=ZVogAwAAQBAJ&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false
Lederman,N.G.,&Lederman,J.S.(2014).Isnatureofsciencegoing,going,going,gone?JournalofScienceTeacherEducation,25(3),235–238.doi:10.1007/s10972-014-9386-z
86
Lederman,N.G.,&Zeidler,D.L.(1987).ScienceTeachers’ConceptionsoftheNatureofScience:DoTheyReallyInfluenceTeachingBehavior?ScienceEducation,71(5),721–734.doi:10.1002/sce.3730710509
Lee,H.S.,&Liu,O.L.(2010).Assessinglearningprogressionofenergyconceptsacrossmiddleschoolgrades:Theknowledgeintegrationperspective.ScienceEducation,94(4),665–688.doi:10.1002/sce.20382
Lee,O.(1997).Guesteditorial:Scientificliteracyforall:Whatisit,andhowcanweachieveit?JournalofResearchinScienceTeaching,34(3),219–222.doi:10.1002/(SICI)1098-2736(199703)34:3<219::AID-TEA1>3.0.CO;2-V
Lee,O.,Quinn,H.,&Valdes,G.(2013).ScienceandlanguageforEnglishlanguagelearnersinrelationtoNextGenerationScienceStandardsandwithimplicationsforCommonCorestatestandardsforEnglishlanguageartsandmathematics.EducationalResearcher,42(4),223–233.doi:10.3102/0013189X13480524
Lehrer,R.,&Schauble,L.(2006).CultivatingmodelBasedreasoninginscienceeducation.InR.K.Sawyer(Ed.),TheCambridgehandbookofthelearningsciences(pp.371–387).Cambridge,UK:CambridgeUniversityPress.
Lin,C.,&Hu,R.(2003).Students’understandingofenergyflowandmattercyclinginthecontextofthefoodchain,photosynthesis,andrespiration.InternationalJournalofScienceEducation,25(1loving2),1529–1544.doi:10.1080/0950069032000052045
Liu,L.,Rogat,A.,&Bertling,M.(2013).ACBALTMScienceModelofCognition:DevelopingaCompetencyModelandLearningProgressionstoSupportAssessmentDevelopment.
Maclure,M.,&Willett,W.C.(1987).Misinterpretationandmisuseofthekappastatistic.AmericanJournalofEpidemiology,126(2),161–169.https://doi.org/10.1093/aje/126.2.161
McComas,W.F.,Clough,M.P.,&Almazroa,H.(2002).Theroleandcharacterofthenatureofscienceinscienceeducation.InW.F.McComas(Ed.),Thenatureofscienceinscienceeducation:Rationaleandstrategies(pp.3–39).Dordrecht:KluwerAcademicPublishers.doi:10.1007/0-306-47215-5_1
Mislevy,R.J.(2016).Howdevelopmentsinpsychologyandtechnologychallengevalidityargumentation.JournalofEducationalMeasurement,53(3),265–292.https://doi.org/10.1111/jedm.12117
Mislevy,R.J.(2017).Resolvingtheparadoxofrichperformancetasks.InTestFairnessintheNewGenerationofLarge-ScaleAssessment(pp.1–46).Charlotte,N.C.:InformationAgePublishers.
87
Mislevy,R.J.,&Haertel,G.D.(2006).Implicationsofevidence-centereddesignforeducationaltesting.EducationalMeasurement:IssuesandPractice,25(4),6–20.doi:10.1111/j.1745-3992.2006.00075.x
Mislevy,R.J.,Haertel,G.,Riconscente,M.,WiseRutstein,D.,&Ziker,C.(2017).Assessingmodel-basedreasoningusingevidence-centereddesign:Asuiteofresearch-baseddesignpatterns.Cham,Switzerland:SpringerInternationalPublishing.doi:10.1007/978-3-319-52246-3
Mislevy,R.J.,Riconscente,M.M.,&Rutstein,D.W.(2009).Designpatternsforassessingmodel-basedreasoning(Large-ScaleAssessmentTechnicalReport6).MenloPark,CA:SRIInternational.
Mohan,L.,Chen,J.,&Anderson,C.W.(2009).Developingamulti-yearlearningprogressionforcarboncyclinginsocio-ecologicalsystems.JournalofResearchinScienceTeaching,46(6),675–698.doi:10.1002/tea.20314
Morin,O.,Simonneaux,L.,&Tytler,R.(2017).Engagingwithsociallyacutequestions:Developmentandvalidationofaninteractionalreasoningframework.JournalofResearchinScienceTeaching,54(7),825–851.doi:10.1002/tea.21386
Mueller,R.O.,&Hancock,G.R.(2015).Factoranalysisandlatentstructureanalysis:confirmatoryfactoranalysis.InJ.D.Wright(Ed.),InternationalEncyclopediaoftheSocial&BehavioralSciences(SecondEdi,pp.686–690).Elsevier.https://doi.org/10.1016/B978-0-08-097086-8.25009-5
Mutegi,J.(2011).Theinadequaciesofscience-for-allandthenecessityandnatureofasociallytransformativecurriculumapproachforAfricanAmericanscienceeducation.JournalofResearchinScienceTeaching,248(3),301–316.
Nardi,B.A.(1996).Studyingcontext:AcomparisonofActivityTheory,SituatedActionModels,andDistributedCognition.InB.A.Nardi(Ed.),ContextandConciousness:ActivityTheoryandHumanComputerInteraction(pp.69–102).Cambridge,MA:TheMITPress.doi:citeulike-article-id:1292510
NationalResearchCouncil.(1996).NationalScienceEducationStandards.Washington,D.C.:NationalAcademiesPress.doi:10.17226/4962
NationalResearchCouncil.(2011).SuccessfulK-12STEMEducation.Washington,D.C.:NationalAcademiesPress.
NationalResearchCouncil.(2012).AframeworkforK-12scienceeducation:Practices,crosscuttingconcepts,andcoreideas.Washington,D.C.:NationalAcademiesPress.Retrievedfromhttp://www.nap.edu/catalog/13165/a-framework-for-k12-science-education-practices-crosscutting-concepts-and
88
NationalResearchCouncil.(2013).MonitoringprogresstowardsuccessfulK-12STEMeducation:Anationadvancing?NationalAcademiesPress.Washington,D.C.:NationalAcademiesPress.doi:10.17226/13509
NationalResearchCouncil.(2014).DevelopingAssessmentsfortheNextGenerationScienceStandards.Washington,D.C.:NationalAcademiesPress.doi:10.17226/18409
Nersessian,N.J.(2002).Thecognitivebasisofmodel-basedreasoninginscience.InP.Carruthers,S.Stich,&M.Siegal(Eds.),TheCognitiveBasisofScience(pp.133–153).NewYork:CambridgeUniversityPress.doi:10.1017/cbo9780511613517.008
Neumann,K.,Viering,T.,Boone,W.J.,&Fischer,H.E.(2013).Towardsalearningprogressionofenergy.JournalofResearchinScienceTeaching,50(2),162–188.doi:10.1002/tea.21061
NGSSLeadStates.(2013a).NextGenerationScienceStandards:ForStates,ByStates.Washington,DC:TheNationalAcademiesPress.
NGSSLeadStates.(2013b).NextGenerationScienceStandards:ForStates,ByStates(AppendixE).Washington,DC:TheNationalAcademiesPress.
NGSSLeadStates.(2013c).NextGenerationScienceStandards:ForStates,ByStates(AppendixG).Washington,DC:TheNationalAcademiesPress.
Noble,T.,Suarez,C.,Rosebery,A.,O’Connor,M.C.,Warren,B.,&Hudicourt-Barnes,J.(2012).“Ineverthoughtofitasfreezing”:Howstudentsanswerquestionsonlarge-scalesciencetestsandwhattheyknowaboutscience.JournalofResearchinScienceTeaching,49(6),778–803.doi:10.1002/tea.21026
Nussbaumer,D.(2012).Anoverviewofculturalhistoricalactivitytheory(chat)useinclassroomresearch2000to2009.EducationalReview,64(1),37–55.https://doi.org/10.1080/00131911.2011.553947
O’Brien,H.L.,&Toms,E.G.(2008).Whatisuserengagement?Aconceptualframeworkfordefininguserengagementwithtechnology.JournaloftheAmericanSocietyforInformationScienceandTechnology,59(6),938–955.https://doi.org/10.1002/asi.20801.1
Osborne,J.(2014).Teachingscientificpractices:Meetingthechallengeofchange.JournalofScienceTeacherEducation,25(2),177–196.doi:10.1007/s10972-014-9384-1
Osborne,J.F.,Henderson,J.B.,MacPherson,A.,Szu,E.,Wild,A.,&Yao,S.Y.(2016).Thedevelopmentandvalidationofalearningprogressionforargumentationin
89
science.JournalofResearchinScienceTeaching,53(6),821–846.doi:10.1002/tea.21316
Osborne,J.,Rafanelli,S.,&Kind,P.(2018).Towardamorecoherentmodelforscienceeducationthanthecrosscuttingconceptsofthenextgenerationsciencestandards:theaffordancesofstylesofreasoning.JournalofResearchinScienceTeaching,55(7),962–981.https://doi.org/10.1002/tea.21460
Pellegrino,J.W.(2013).ProficiencyinScience:AssessmentChallengesandOpportunities.Science,340(6130),320–323.doi:10.1126/science.1232065
Pellegrino,W.,Chudowsky,N.,&Glaser,R.(2001).Thenatureofassessmentandreasoningfromevidence.InKnowingWhatStudentsKnow:TheScienceandDesignofEducationalAssessment(pp.37–54).Washington,D.C.:NationalAcademiesPress.Retrievedfromhttp://books.google.com/books?hl=en&lr=&id=t5mcAgAAQBAJ&pgis=1
Peterson,C.H.,Peterson,N.A.,&Powell,K.G.(2017).Cognitiveinterviewingforitemdevelopment:validityevidencebasedoncontentandresponseprocesses.MeasurementandEvaluationinCounselingandDevelopment,50(4),217–223.https://doi.org/10.1080/07481756.2017.1339564
Plummer,J.D.,&Krajcik,J.(2010).Buildingalearningprogressionforcelestialmotion:Elementarylevelsfromanearth-basedperspective.JournalofResearchinScienceTeaching,47(7),768–787.doi:10.1002/tea.20355
Plummer,J.D.,Palma,C.,Flarend,A.,Rubin,K.,Ong,Y.S.,Botzer,B.,…Furman,T.(2015).Developmentofalearningprogressionfortheformationofthesolarsystem.InternationalJournalofScienceEducation,37(9),1381–1401.doi:10.1080/09500693.2015.1036386
Pluta,W.J.,Chinn,C.A.,&Duncan,R.G.(2011).Learners’epistemiccriteriaforgoodscientificmodels.JournalofResearchinScienceTeaching,48(5),486–511.doi:10.1002/tea.20415
Quellmalz,E.S.,&Pellegrino,J.W.(2009).TechnologyandTesting.Science,323(5910),75–79.doi:10.1126/science.1168046
Ríordáin,M.N.,Johnston,J.,&Walshe,G.(2015).Makingmathematicsandscienceintegrationhappen:keyaspectsofpractice.InternationalJournalofMathematicalEducationinScienceandTechnology,47(September),1–23.doi:10.1080/0020739X.2015.1078001
Rivet,A.E.,&Kastens,K.A.(2012).Developingaconstruct-basedassessmenttoexaminestudents’analogicalreasoningaroundphysicalmodelsinEarthScience.JournalofResearchinScienceTeaching,49(6),713–743.doi:10.1002/tea.21029
90
Rivet,A.E.,&Krajcik,J.S.(2004).AchievingstandardsinUrbansystemicreform:Anexampleofasixthgradeproject-basedsciencecurriculum.JournalofResearchinScienceTeaching,41(7),669–692.doi:10.1002/tea.20021
Rivet,A.E.,&Krajcik,J.S.(2008).Contextualizinginstruction:Leveragingstudents’priorknowledgeandexperiencestofosterunderstandingofmiddleschoolscience.JournalofResearchinScienceTeaching,45(1),79–100.doi:10.1002/tea.20203
Rivet,A.E.,Weiser,G.,Lyu,X.,Li,Y.,&Rojas-Perilla,D.(2016).Whatarecrosscuttingconceptsinscience?Fourmetaphoricalperspectives.InC.K.Looi,J.L.Polman,U.Cress,&P.Reimann(Eds.),ProceedingsofInternationalConferenceoftheLearningSciences,ICLS(Vol.2).Signapore:InternationalSocietyoftheLearningSciences.doi:10.22318/icls2016.149
Rodriguez, A. J. (2015).What about a dimension of engagement, equity, and diversitypractices?Acritiqueofthenextgenerationsciencestandards.JournalofResearchinScienceTeaching,52(7),1031–1051.https://doi.org/10.1002/tea.21232
Roseman,J.,Kesidou,S.,&Stern,L.(1996).Identifyingcurriculummaterialsforscienceliteracy:AProject2061evaluationtool.RetrievedNovember27,2014,fromhttp://www.project2061.org/publications/articles/roseman/roseman2.htm
Roth,W.-M.,&Tobin,K.(2002).Redesigningan“Urban”TeacherEducationProgram:AnActivityTheoryPerspective.Mind,Culture,andActivity,9(2),108–131.doi:10.1207/S15327884MCA0902
Rutherford,F.J.,&Ahlgren,A.(1990).ScienceforAllAmericans.NewYork:OxfordUniversityPress.
Sandoval,W.A.(2005).Understandingstudents’practicalepistemologiesandtheirinfluenceonlearningthroughinquiry.ScienceEducation,89(4),634–656.doi:10.1002/sce.20065
Schwarz,C.(2009).Developingpreserviceelementaryteachers’knowledgeandpracticesthroughmodeling-centeredscientificinquiry.ScienceEducation,93(4),720–744.doi:10.1002/sce.20324
Schwarz,C.V.,&White,B.Y.(2005).MetamodelingKnowledge:DevelopingStudents’UnderstandingofScientificModeling.CognitionandInstruction,23(2),165–205.doi:10.1207/s1532690xci2302_1
Schwarz,C.V.,Reiser,B.J.,Davis,E.A.,Kenyon,L.,Achér,A.,Fortus,D.,...&Krajcik,J.(2009).Developingalearningprogressionforscientificmodeling:Makingscientificmodelingaccessibleandmeaningfulforlearners.JournalofResearchinScienceTeaching,46(6),632-654.
91
Shavelson,R.J.,Phillips,D.C.,Towne,L.,&Feuer,M.J.(2003).OntheScienceofEducationDesignStudies.EducationalResearcher,32(1),25–28.doi:10.3102/0013189X032001025
Snow,E.,&Katz,I.R.(2009).UsingcognitiveinterviewstovalidateaninterpretiveargumentfortheETSiSkillsTMassessment.CommunicationsinInformationLiteracy,3(2),99–127.
Songer,N.B.,Kelcey,B.,&Gotwals,A.W.(2009).Howandwhendoescomplexreasoningoccur?Empiricallydrivendevelopmentofalearningprogressionfocusedoncomplexreasoningaboutbiodiversity.JournalofResearchinScienceTeaching,46(6),610–631.doi:10.1002/tea.20313
Songer,N.B.,Lee,H.-S.,&McDonald,S.(2003).Researchtowardsanexpandedunderstandingofinquirysciencebeyondoneidealizedstandard.ScienceEducation,87(4),490–516.doi:10.1002/sce.10085
Sparks,J.R.,&Deane,P.(2015).CognitivelyBasedAssessmentofResearchandInquirySkills:DefiningaKeyPracticeintheEnglishLanguageArts.ETSResearchReportSeries,2015(2),1–55.doi:10.1002/ets2.12082
Stevens,S.Y.,Delgado,C.,&Krajcik,J.S.(2009).Developingahypotheticalmulti-dimensionallearningprogressionforthenatureofmatter.JournalofResearchinScienceTeaching,47(6),687–715.doi:10.1002/tea.20324
Stroupe,D.(2015).Describing“SciencePractice”inlearningsettings.ScienceEducation,99(6),1033–1040.doi:10.1002/sce.21191
Svoboda,J.,&Passmore,C.(2011).TheStrategiesofModelinginBiologyEducation.Science&Education,22(1),119–142.doi:10.1007/s11191-011-9425-5
vanEs,E.A.,&Sherin,M.G.(2002).LearningtoNotice:ScaffoldingNewTeachers’InterpretationsofClassroomInteractions.JournalofTechnologyandTeacherEducation,10(4),571–596.
Weiser,G.,&Liu,L.(2018,March).Adesignframeworkforthedevelopmentofscenario-basedassessmentsforsummativeassessmentsettings.PosterpaperpresentatNARST,Atlanta.
Weiser,G.,Lyu,X.,&Rojas-Perilla,D.F.(2017,April).WhatAreCrosscuttingConceptsinScience?FourMetaphoricalPerspectives.PaperpresentatNARST,SanAntonio.
Wiliam,D.(2010).Whatcountsasevidenceofeducationalachievement?Theroleofconstructsinthepursuitofequityinassessment,.ReviewofResearchinEducation,34(1),254–284.doi:10.3102/0091732X09351544
92
Wilson,M.,&Draney,K.(2005).Somelinksbetweenlarge-scaleandclassroomassessments:ThecaseoftheBEARassessmentsystem.YearbookoftheNationalSocietyfortheStudyofEducation,103(2),132–154.doi:10.1111/j.1744-7984.2004.tb00051.x
Yilmaz,S.,Eryilmaz,A.,&Geban,O.(2006).Assessingtheimpactofbridginganalogiesinmechanics.SchoolScienceandMathematics,106(6),220–230.doi:10.1111/j.1949-8594.2006.tb17911.x
Yoon,S.A.(2008).Anevolutionaryapproachtoharnessingcomplexsystemsthinkinginthescienceandtechnologyclassroom.InternationalJournalofScienceEducation,30(1),1–32.doi:10.1080/09500690601101672
Yoon,S.A.,Goh,S.-E.,&Park,M.(2018).TeachingandlearningaboutcomplexsystemsinK–12scienceeducation:Areviewofempiricalstudies1995–2015.ReviewofEducationalResearch,88(2),285–325.doi:10.3102/0034654317746090
Yore,L.,Bisanz,G.L.,&Hand,B.M.(2003).Examiningtheliteracycomponentofscienceliteracy:25yearsoflanguageartsandscienceresearch.InternationalJournalofScienceEducation,25(6),689–725.doi:10.1080/09500690305018
Zaharias,P.,&Poylymenakou,A.(2009).Developingausabilityevaluationmethodfore-learningapplications:Beyondfunctionalusability.InternationalJournalofHuman-ComputerInteraction,25(1),75–98.doi:10.1080/10447310802546716
93
APPENDICES
AppendixA:TaskModelforAssessmentofEcologicalSystems,Structures,andFunctions–AdaptedfromMS-LS2-3,MS-LS2-4,&MS-LS2-5
PerformanceExpectationsandTheirDimensions
Studentswhodemonstrateunderstandingcan:MS-LS2-3(Original) Developamodel todescribe the cyclingofmatter and flowof
energy among living and nonliving parts of an ecosystem.[Clarification Statement: Emphasis is on describing theconservationofmatterandflowofenergyintoandoutofvariousecosystems, and on defining the boundaries of the system.][AssessmentBoundary:Assessmentdoesnotincludetheuseofchemicalreactionstodescribetheprocesses.]
MS-LS2-3(Adapted) Develop a model to describe relationships among living andnonlivingpartsofanecosystemthat iscapableof tracking thecycling of matter and flow of energy among ecosystemcomponents.
MS-LS2-4(Original) Construct an argument supported by empirical evidence thatchanges tophysical or biological components of an ecosystemaffect populations. [Clarification Statement: Emphasis is onrecognizingpatterns indataandmakingwarranted inferencesabout changes in populations, and on evaluating empiricalevidencesupportingargumentsaboutchangestoecosystems.]
MS-LS-4(Adapted) Construct an argument supported by empirical evidence thatchangesthefunctioningofanecosystem(includingitsabilitytosupport population growth) can be driven by changes to thephysicalorbiologicalcomponentsoftheecosystem.
MS-LS2-5(Original) Evaluate competing design solutions for maintainingbiodiversity and ecosystem services. [Clarification Statement:Examplesofecosystemservicescouldincludewaterpurification,nutrient recycling, andpreventionof soil erosion.Examplesofdesign solution constraints could include scientific, economic,andsocialconsiderations.]
MS-LS2-5(Adapted) Evaluatecompetingdesignsolutionsforachievingsomehumanwant based on criteria that considermaintaining biodiversityandecosystemservices.
ScienceandEngineeringPractices
DisciplinaryCoreIdeas CrosscuttingConcepts
94
Developing and UsingModelsModelingin6–8buildsonK–5experiencesandprogressestodeveloping,using,andrevisingmodels to describe, test, andpredict more abstractphenomena and designsystems.
• Develop a model todescribe phenomena.(MS-LS2-3)
Engaging in Argument fromEvidenceEngaging in argument fromevidencein6–8buildsonK–5experiencesandprogressestoconstructing a convincingargument that supports orrefutes claims for eitherexplanations or solutionsaboutthenaturalanddesignedworld(s).
• Construct an oral andwritten argumentsupportedbyempiricalevidenceandscientificreasoning to supportor refute anexplanationoramodelforaphenomenonorasolution to a problem.(MS-LS2-4)
• Evaluate competingdesign solutionsbasedon jointly developedand agreed upondesign criteria. (MS-LS2-5)
LS2.B: Cycle of Matter andEnergy Transfer inEcosystems
• Foodwebsaremodelsthat demonstrate howmatter and energy istransferred betweenproducers, consumers,and decomposers asthe three groupsinteract within anecosystem. Transfersofmatter into and outof the physicalenvironment occur atevery level.Decomposers recyclenutrients from deadplantoranimalmatterback to the soil interrestrialenvironmentsortothewater in aquaticenvironments. Theatomsthatmakeuptheorganisms in anecosystem are cycledrepeatedly betweenthelivingandnonlivingpartsoftheecosystem.
LS2.C: Ecosystem Dynamics,Functioning,andResilience
• Ecosystems aredynamic in nature;their characteristicscan vary over time.Disruptions to anyphysical or biologicalcomponent of anecosystem can lead toshifts in all itspopulations.
SystemsandSystemsModels(Adaptation):
• Representing flows ofmatter and energywithin a larger,complexsystemcanbeaccomplished bymodeling flows withinand among relevantsub-systems. (MS-LS2-3)
• Models canbeused torepresent the impactsof changes in a sub-system on the largersystem particularlywhen evidence canonly be collected fromlimited aspects of thebroader phenomenon.(Might be adaptableintoStructure/Function ifwe add ESS2.A orESS2.CDCIs) (MS-LS2-4)
Structure and Function(Adaptation):
• The structure ofengineering solutionsare designed to serveparticular functionsthrough the physicalproperties of thematerialsusedandtheenvironment thesolution inhabits (MS-LS2-5)
EnergyandMatter(Original)• The transfer of energy
can be tracked asenergyflowsthroughanaturalsystem.
95
• Biodiversity describesthe variety of speciesfound in Earth’sterrestrial and oceanicecosystems. Thecompleteness orintegrity of anecosystem’sbiodiversity is oftenusedasameasureofitshealth.
LS4.D: Biodiversity andHumans
• Changesinbiodiversitycan influence humans’resources,suchasfood,energy,andmedicines,as well as ecosystemservices that humansrely on—for example,water purification andrecycling.(secondary)
ETS1.B:DevelopingPossibleSolutions
• There are systematicprocesses forevaluating solutionswith respect to howwell they meet thecriteriaandconstraintsof a problem.(secondary)
Stability and Change(Original)
• Small changes in onepartofasystemmightcause large changes inanotherpart.
Practice-orientedFocal KSAs (Note:“ability” here meanscapabilitytoreasonasdescribedaboutgivenDCIs and CCCs. Noclaim is made for“abilities” asdecoupled from DCIsandCCCs.)
Abilitytodevelopmodel-basedarguments:• Abilitytodevelopmodelsthatrepresentnaturaleventssystems,
aspectsofatheoryandevidence,ordesignsolutions.• Abilitytodeterminethecomponentsaswellasconnectionsand
relationshipsamongmultiplecomponentsof theevent,system,ordesignsolutiontoincludeinthemodelandthosetoomit.
• Abilitytodeterminescope,scale,andgrain-sizeofthemodel,asappropriatetoitsintendeduse.
96
Thissectionadaptedfrom Model-basedArgumentationDesign PatternDocument.
• Abilitytorepresentmechanisms,relationships,andconnectionsto explain the event, system, or design solution with multipletypesofmodels.
Abilitytoevaluatecompetingmodels:• Ability to evaluate the explanatory or predictive power of
competing models taking into account evidence and empiricaldataassociatedwithaphenomenonunderinvestigation.
• Abilitytoevaluatehowcompetingmodelsdescribemechanismsandprocessesrelatedtothetargeteventorsystem.
• Ability to collect evidence to reason qualitatively orquantitatively about concepts and relationships represented inmodels.
• Abilitytouseevidenceorempiricaldatatogenerateexplanationsandpredictionsaboutthebehaviorofascientificphenomenon.
• Abilitytoapplyscienceconceptsorprinciplestoreasonwhythedatasupportorrefuteanargument.
• Abilitytoselectappropriatemodelrepresentationsthataremostusefulforsupportingorrefutinganargument.
Abilitytorevisemodel-basedarguments:• Abilitytorevisemodelsinlightofempiricalevidencetoimprove
theirexplanatoryandpredictivepower.• Ability to apply alternative science concepts/principles to
supportanargument.• Ability to refine the data collection approach to improve the
appropriateness,accuracy,orsufficiencyoftheempiricaldata.Viable Indicators ofCross-DimensionalProgress (Note:“cross-dimensional”heremeansreasoningthat manifests in theprocess of engagingwith a practice in aparticularwayaboutaparticularphenomenon. Noclaim is made for“abilities” asdecoupledfromDCIs.)Thissectionadaptedfrom Design
ProgressinSystemsandSystemModelsSM.1. Lowest performing students – Ability to identify relevant
features but not provide linkages connecting severalfeaturesnorprovideexplicitreasoningwhyfeaturesareorarenotrelevant.SM.1.a. Fragmented identification of system components and
relationshipswithinandacrossscalesorscopesthatdonotmaptoeachother.
SM.1.b. Systemcomponentsmayberepresentedas“boxedin”bysystemboundaryandimmutabletochange.
SM.1.c. Limitedabilitytorepresenttheboundariesofthesystemandsystemmaybeinappropriatelycategorizedasopenorclosedtoexternalinputs.
SM.1.d. Identification of salient features may be based onfamiliarity (or similarity to familiarfeatures/phenomena)ratherthanonwhatbestsuitsthegoal.
97
Patterns for CCC xSEP
SM.1.e. Goal of model development is visuospatial illustrationphenomenon with overemphasis on representing “thelook” of the system rather than relationships amongentities.
SM.2. Lower Intermediate performing students – Able to createsingle entity-entity relationship connections. Able to findseveralinstancesinwhichthesameconnectionappears.SM.2.a. In model development, may identify multiple system
entities and relationships atmultiple system levels, butnot represent interdependence among those systemlevels.
SM.2.b. Inmodeluse,maybeabletodescribethatsystemlevelsareinterdependentwithoutbeingabletoprovidedrivingmechanism.
SM.2.c. Able to identify movement of energy or matter acrosssystem boundaries (though inciting act will rely onanthropomorphicagents).
SM.2.d. May be inconsistently able to create a chain of singlecause-singleeffectrelationshipswithinthesystemlinkedoneafterthenext.
i. Represented chains will always reinforce theeffect(positivefeedbackloopsonly).
SM.2.e. Reasoning for represented relationshipsmaybe just-soor overly emphasize anthropomorphic agents. Goal ofmodel development is to describe “theway things are”rather than to have some broaderargumentative/predictivegoal
SM.3. HigherIntermediateperformingstudents–Abletoidentifyhow micro-level causal relationships can produce higher-levelsystembehavior.Abletocontextualizeamodelintermsofsomeexplanatoryorargumentativegoal.SM.3.a. In models and explanations, indicate relationship
mappinginwhichmanycausescombinetoproduceanyoneeffect.
i. At the higher end of this benchmark, mappingmaybegintobecomemany-to-many
SM.3.b. In models and explanations, interdependentrelationships across system/subsystem levels arerepresentedinlimitedform.
ii. Student can identify/represent evidence at thesubsystem level for relationships at the systemlevel.
98
iii. Studentcanidentifyevidenceatthesystemlevelfor relationships that primarily exist at thesubsystemlevel.
SM.3.c. Can describe/represent both positive and negativefeedback loops in matter/energy (or generalizedinput/output)flows.
SM.3.d. Continues to explain emergent phenomena in terms ofcentral control or adherence to a coherent, rigidframework.
SM.3.e. Can explain how features of model support a broaderexplanatory goal and make choices regarding modelfeatureselectionpursuanttothatgoal.
SM.4. Highestperformingstudents–AbletorecognizeconstraintsandlimitationsofamodelSM.4.a. Able to identify feature mediators (e.g. temperature,
topography, available inputs) that may alter existingsystemrelationships.
SM.4.b. Able to identify subsystem relationships mediate (orcreatedynamism)amongsystemrelationships
SM.4.c. –MayalreadybeatthehighestlevelbySM.3.cSM.4.d. In models and explanations, students represent how
emergent properties at higher system levels manifestfromtherulesgoverninglower-levelsysteminteractionswithoutrequiringcentralizedagents/actors.
SM.4.e. Goal of the model is to support critique of providedarguments, suggestnew sourcesof evidence, andmakecomplex predictions about how relationships fit into abroadersystemcontext
ProgressinStructureandFunctionSF.1. Lowest performing students – Ability to identify relevant
features but not provide linkages connecting severalfeaturesnorprovideexplicitreasoningwhyfeaturesareorarenotrelevant.SF.3.a. Fragmented identification of structural features and
entityfunctionswithinandacrossscalesorscopesthatdonotmaptoeachother.
SF.3.b. In models and model use, students primarily identifymacro-scalestructuresorfunctionsandonlysomemicro-scale functions (micro-scale structure may not beexpressed).
SF.3.c. Identification of salient features may be based onfamiliarity (or similarity to familiar
99
features/phenomena)ratherthanonwhatbestsuitsthegoal.
SF.3.d. Goal of model development is visuospatial illustrationphenomenon
SF.2. Lower Intermediateperformingstudents (Bridges?)–Abletocreatesinglestructure-functionconnections(orchainofthatconnection).Abletofindseveralinstancesinwhichthesameconnectionappears.SF.2.a. In models and explanations, indicate structures and
functionsaremapped1:1SF.2.b. In models and explanations may identify multiple
structuresandfunctionsatmultiplesystemlevelsSF.2.c. Inmodelsandexplanations,failtoidentifyhowstructures
andinterrelationshipsamongstructuresaffordfunction.SF.2.d. Reasoning for structure may be just-so or overly
emphasizeanthropomorphicagents.SF.3. HigherIntermediateperformingstudents–Abletoidentify
howmacro-levelfunctionalpropertiesinteractwithmicro-levelstructuresandhowmodelcanbeusedtosupportgoalSF.3.a. Inmodels and explanations, indicate structure-function
mapping(maybemanytoone)SF.3.b. Inmodelsandexplanations,identifythatstructuresand
functionsareinter-relatedacrosssystemlevelsSF.3.c. Canusethemodeltomakeanargumentinsupportofa
drivingmechanismthatmakesstructuresaffordfunction.SF.3.d. Can explain how features of model support a broader
argumentative goal andmake choices regardingmodelfeatureselectionpursuanttothatgoal.
SF.4. Highestperformingstudents–AbletorecognizeconstraintsandlimitationsofamodelSF.4.a. Inmodels and explanations, identify structure-function
relationshipsaredynamicSF.4.b. Abletoidentifymediators(e.g.temperature,topography,
available inputs) that may alter existing structure-functionrelationships.
SF.4.c. Able to use model of structure and function to makepredictions about how relationships fit into a broadersystemcontext.
SF.4.d. Abletocritiqueamodeloraskaquestionofamodelthathighlights failure to reach anexplanatory/predictive/argumentativegoal.
100
DCI AssessmentTargets
Studentscan:LS2.B.a. Differentiatebetweenorganismsbasedonwhethertheyare
producers,consumers,ordecomposers.LS2.B.b. Describetherolenonlivingpartsofanecosystem(likewater,
air,andminerals)playinmediatingtheinteractionsbetweenproducers, consumers, and decomposers within anecosystem.
LS2.B.c. Tracksandquantifythecyclingofmatterandenergythroughvarious reservoirs (both living and nonliving) within anecosystem.
LS2.B.d. Describe relationships between the consumption ofmatterand energy by both consumer and producer organisms intermsofmatter/energyconservationprinciples.
LS2.C.a. Identify evidence that changes are occurring among thephysical or biological components of an ecosystem anddescribetherateatwhichsuchchangesareoccurring.
LS2.C.b. Identify evidence that changes are occurring to thefunctioningofanecosystem(particularlyitsabilitytosupportpopulation growth) and describe the rate at which suchchangesareoccurring.
LS2.C.c. Identifykeystoneecosystemcomponents/relationshipsthat,if changed, have disproportionately large effects on thecontinuedfunctioningofecosystem.
LS2.C.d. Describe mechanisms by which various ecosystemcomponentsact tosupportpopulationgrowth,quantify thestrength of correlations between those mechanisms andpopulation sizes, and the describe the resilience of thosemechanismstochange.
LS2.C.e. Usebiodiversityasameasureofthehealthofanecosystemandtheabilityofanecosystemtosupportpopulationgrowth.
LS4.D.a. Identifyvarious‘Ecosystemservices’inwhichhumansacttosupportthestability/resilienceofanecosystem.
LS4.D.b. Describe the tradeoffs associated with various ecosystemservicesinlightofalocalcontext(bothlocaldesiresandlocalconstraints).
LS4.D.c. Identify potential unintended consequences of a givenecosystem service endeavor and describe ways that canmitigatetheriskofunintendedconsequencesoccurring.
ETS1.B.a. Identify relevant constraints and criteria that apply todesigning an engineering solution to a problem faced by alocalcommunity.
ETS1.B.b. Describetradeoffsbetweensolvingonlyalimitednumberofthetotalsetofproblem-pieces.
101
ETS1.B.c. Evaluate the effectiveness of multiple (three or more)potentialdesignsolutions toanengineeringproblembasedonprovideddata.
ETS1.B.d. Identifyappropriateevidencesourcesthatcouldbeusedtoevaluatepotentialdesignsolutionsontheirabilitytomeetthecriteria of the problem definition while staying withinrelevantconstraints.
PossiblePhenomenaorContexts*
Potential criteria/constrains that may govern the viability of designsolutionsinclude:
• Relevantphysicalprinciples• Costofdevelopment• Featuresofthelocalgeography• Differentialfeaturesoflocalclimate• Trendsinpopulationgrowthorper-capitaresourceconsumption• Localandglobaltrendsinhumanbehavior
Examples ofIntegration ofAssessment TargetsandEvidence
1. Systems and System Models – Task provides a modelrepresentingfoodwebrelationshipsamongorganismswithinan ecosystem. Students are asked to identify consumers,producers, and decomposers as well as identify keystoneorganisms (whereby changes to their populations havedramaticeffectsontheotherpopulationsintheecosystem).
a. SuccessfulPerformance:Studentsareabletoidentifyand categorize organisms appropriately in light of aprovidedcontext(SM.1.a).
b. Unsuccessful Performance: Students suggest thatirrelevantsystemcomponentsaremostsalient(i.e.fisharekeystoneorganismsbecausehumanswillfeelsadiftheydieout)(SM.1.d).
2. SystemsandSystemModels–Taskexpandsonthefoodwebmodeltoincludethetrackingoftransferofenergyandmatterviatheproductionandconsumptionoforganisms’biomasses(thismaybeasimulatedtask).Studentsareaskedtoquantifymatter/energystoredindifferentreservoirs.
a. Successful Performance: Students are able to trackthe transfer of energy among ecosystem componentsvia a chain of solar source -> producer viaphotosynthesis -> primary consumer -> secondaryconsumer->etc(SM.2.c).Theymayexhibitconfusionabout the role of decomposers as part of a nutrienttransportcycle(SM.2.b&SM.2.d).
b. Unsuccessful Performance: Student fails to identifyenergy/matter transport pathways or fails to amendmodeltorepresenttransport(SM.1.c).
102
3. SystemsandSystemModels–Taskexpandson thecontextgroundingthe foodwebmodel toprovidedetailsofachangethat may be affecting the physical or biological subsystemcomponents of the ecosystem. Students are asked to reasonabout the higher-level effects these changes might have onexistingenergy/mattertransportpathways.(Notethat,atthislevel, students will still focus on centralized controlmechanisms).
a. SuccessfulPerformance:Studentsareabletoexpressbi-directional, interdependent relationships betweensubsystemandsystemcomponents(SM.3.b).Theyareabletousetheserelationshipstoexplainwhetherornottheecosystemisresilienttothechangeprovidedbythetask(SM.3.c).
b. UnsuccessfulPerformance:Student fails todescribeways in which changes to system components canresult in changes to existing food-web relationshipsand/orfailstodescribewaysinwhichchangestooneormore food-web relationshipsmay affect the entireweb(SM.2.a&SM.2.b).
c. Potentialmidpointperformance:Studentsisabletodescribe interdependentrelationships(SM.3.b)but isonlyabletorepresentpositivefeedbackloops(therebyalwaysconcludingthatthereisalackofresilienceintheecosystem)(SM.2.d.i).
4. Systems and System Models – Task re-contextualizes theecosystem changes in terms of risks to change (i.e. anunintendedconsequenceofanecosystemservice).Studentsareaskedtomakeanargumentwhichcontrastspotentialbenefitsof the ecosystem service against potential negativeconsequences(bothofactionandinaction).
a. Successful Performance:Student is able to describemediating factors that contribute or mitigate risk,including feedback loops, entity mediators, andrelational mediators in order to predict outcomes(SM.4.a & SM.4.b). Argument for/against ecosystemserviceshouldbegroundedinthatprediction(SM.4.e).
i. Alternative: Student is able to critique aprovided argument/prediction on the basis ofrelationalmediators(SM.4.e).
b. UnsuccessfulPerformance:Student fails todescribehow system features or entities may act to mediateother system relationships (i.e. fails to describe how
103
populationgrowthaffectsoverallresourceavailability)(SM.3.d). They make inaccurate or inappropriatepredictionsorfailtoarguehowmodelelementsexisttosupportthatprediction(SM.2.e).
5. Structure and Function – Task provides a description ofseveral potential engineering solutions each designed toprovide an ecosystem service in light of the needs of a localcommunity. Students are asked to model the criteria andconstraints of the problem definition, highlighting relevantaspectsofthelocalenvironment.(Modelsmaytaketheformofillustrations,conceptmaps,linguisticdescriptions,etc.)
a. Successful Performance: Student is able to identifyrelevant ecosystem structures and describe how theengineering solution fits into those pre-existingstructures.(SF.1.a&SF.1.d)
b. UnsuccessfulPerformance:Studentoveremphasizesfamiliar structures or functions (SF.1.c). Studentrepresentsonlystructuresoronlyfunction(SF.1.b).
6. Structure and Function – Task provides further detailsregardingtheproposedengineeringsolutionforanecosystemservice.Studentsareaskedtodrawconnectionsbetweenthestructure/function relationships that are common to severalproposeddesigns.
a. SuccessfulPerformance:Studentsuccessfullymapsaparticularengineeredstructuretothefunctionalgoalofthe design (SF.2.a).They are able to identify severalinstances in which a given structure-function pairingappearsinmanydifferentproposedsolutions.(SF.2.b)
b. Unsuccessful Performance: Student fails to form astructure-function pairing or fails to see that samerelationshipinotherdesigns(SF.1.a).
7. StructureandFunction–Taskprovidesadditionalcontextinwhicha localcommunity isseeking toevaluate theproposedsolutionsinordertoimplementonethatbestsuitstheirneeds.Studentsareaskedtobuildontheconnectionsfromthebridgetasktoidentifysourcesofevidencethatcouldbeusedtoassessandselectthebestdesign.
a. Successful Performance: Student is able to usecommonalities in structure-function relationshipsacrossdesignstoidentifypotentialsourcesofevidence(SF.3.c). Student is able to identify evidence thatmanifests at a different level than the structure-
104
functionrelationshiporacrosslevelsofthestructure-functionrelationship(SF.3.b).
b. Unsuccessful Performance: Student fails to developan argument for how evidence collected supports anassessment of the proposed design (SF.2.c). Studentfails to contextualize relationships in terms of thecriteriaandconstraintsboundingtheproblem(SF.2.d).
8. StructureandFunction–Taskprovidesanargument fororagainst the implementationof aparticular ecosystemservicevia the construction of some human-engineered structure.Students are asked to use a model of the functioning of thesolutiontocritiquetheproposedimplementation/constructionstrategy.
a. SuccessfulPerformance:Studenteffectivelyusesthemodel to identify mediating relationships that runcounter to the functioning of the engineering design(SF.4.b) and proposes amendments to theimplementation that may act to mitigate unintendedconsequences(SF.4.c).
b. UnsuccessfulPerformance:Studentfailstopresentanaccount of the functioning of the design solutionwhereby mediators transform a many-to-onerelationshipintoamany-to-manyrelationship(SF.3.a).
CommonMisconceptions*
AdditionalAssessmentBoundaries
*Notanexhaustivelist
105
AppendixB:HypothesizedLearningProgressionsSystemsandSystemsModels
Level 1 à Level 2 à Level 3 à Level 4 à Level 5 SM1. Aggregate
level phenomenon
SM1.1.A. Inaccurately identify phenomena of interest (i.e., perturbed state, equilibrium)
SM1.1.B. Inaccurately describe phenomenon using observable features and personal experience.
SM1.2.A. Accurately identify relevant phenomenon of interest. May inaccurately identify other phenomena.
SM1.2.B. Accurately describe the phenomenon using observable features and personal experience only. May be inconsistent in describing phenomenon.
SM1.3.A. Accurately identify relevant phenomenon of interest only.
SM1.3.B. Accurately describe the phenomenon using observable features and one hidden mechanism. May be inconsistent in describing phenomenon.
SM1.4.A. Already at highest
SM1.4.B. Accurately describe the phenomenon using observable features and several hidden mechanisms. May be inconsistent in describing the mechanisms.
SM1.5.A. Already at highest
SM1.5.B. Accurately describe the phenomenon using observable features and hidden mechanisms.
106
SM2. Components
SM2.1.A. Identify irrelevant and/or inaccurate components related to the phenomenon
SM2.1.B. No identification of sub-components
SM2.1.C. No description of components as being static and/or changing.
SM2.2.A. Accurately identify some components relevant to the phenomenon. May identify irrelevant components.
SM2.2.B. Identify sub-components but may be irrelevant and/or inaccurate.
SM2.2.C. Describe components as being static or unchanging (initial conditions).
SM2.3.A. Accurately identify all components relevant to the phenomenon.
SM2.3.B. Accurately identify all sub-component
SM2.4.A. Already at highest
SM2.4.B. Already at highest
SM2.4.C. Describe components (or initial conditions) as changing due to external and internal factors. Description include short-term and long-term changes and may be inaccurate.
SM2.5.A. Already at highest
SM2.5.B. Already at highest
SM2.5.C. Accurately describe components (or initial conditions) as changing due to external and internal factors. Description include short-term and long-term changes.
107
SM2.3.C. Describe
components (or initial conditions) as changing due to external factors. Description is limited to short-term change and may be inaccurate.
SM3. Relationships between components
SM3.3.A. Describe causal relationships between components
SM3.4.A. Accurately describe causal relationships between
SM3.5.A. Already at highest
SM3.5.B. Accurately mention bidirectional
108
and sub- components but may be irrelevant and/or inaccurate.
SM3.3.B. Identify unidirectional between components and sub- components but may be irrelevant and/or inaccurate
SM3.3.C. Identify and one to one relationships but may be irrelevant and/or inaccurate
SM3.3.D. Identify linear relationship between components and sub- components but may be irrelevant
components and sub- components.
SM3.4.B. Accurately identify unidirectional between components and sub- components.
SM3.4.C. Identify one to one relationships and one-to-many relationships but may be irrelevant and/or inaccurate.
SM3.4.D. Accurately identify linear relationships between components and sub- components
and/or cyclical relationship between components and sub- components.
SM3.5.C. Accurately identify one to one relationships and one-to-many relationships
SM3.5.D. Accurately identify linear and non-linear relationships between components and sub- components
109
and/or inaccurate
SM4. Mechanisms (processes/ relationship)
(e.g., emergence, feedback loops, adaptation, iteration, randomness).
SM4.3.A. Identify one central mechanism that explains observed phenomena. May be inaccurate.
SM4.3.B. Identify mechanisms at one level only (e.g., local only and no aggregate). May be inaccurate.
SM4.3.C. Does not identify random nature of mechanisms at the local or individual level
SM4.3.D. Describe mechanisms as static, even as the components change.
SM4.4.A. Identify multiple mechanisms that explains observed phenomena. May be inaccurate and/or continue to identify one central mechanism.
SM4.4.B. Identify mechanisms at multiple levels (e.g., local and aggregate). May be inaccurate.
SM4.4.C. Identify random nature of mechanisms at the local or individual level. May be inaccurate.
SM4.5.A. Accurately identify multiple mechanisms that interact together to give rise to observed phenomena.
SM4.5.B. Accurately identify mechanisms at local and aggregate levels.
SM4.5.C. Accurately identify random individual/local mechanisms that interact across multiple levels in the system.
SM4.5.D. Accurately describe mechanisms and components as changing dynamically in response to one another.
110
SM4.3.E. Does not make any predictions about system level behaviors based on mechanisms
SM4.4.D. Describe mechanisms and components as changing dynamically in response to one another. May be inaccurate.
SM4.4.E. Predictions about system level behaviors from mechanisms are linear in scope (e.g., single mechanisms as impacting system)
SM4.5.E. Predictions about system level behaviors from mechanisms are dynamic. (e.g., multiple interacting behaviors as impacting system)
SM5. Properties of System
SM5.3.A. Limited or unclear description of relevant boundaries
SM5.3.B. System may be inappropriately categorized as open or closed
SM5.4.A. Detailed description of the boundaries of the system. May be inaccurate.
SM5.4.B. System categorized as open or closed to external inputs. May
SM5.5.A. Accurate description of the boundaries of the system
SM5.5.B. System accurately categorized as open or closed to external inputs.
SM5.5.C. Understanding that
111
to external inputs.
SM5.3.C. Inaccurate understanding of equilibrium as simply meaning positive balance
include inaccuracies
SM5.4.C. Limited understanding that mathematical equilibrium includes stable & positive population only
mathematical equilibrium includes both unstable and stable, positive and extinction of populations (i.e., chaos is the norm)
StructureandFunction
Level 1 à Level 2 à Level 3 à Level 4 à Level 5 SF1. Structure
(Components)
SF1.1.A. Identify a single structure but not all of its components. Description is based on familiarity or visual similarity. May be inaccurate.
SF1.1.B. Describe structures as consisting of smaller components. May be inaccurate.
SF1.2.A. Accurately identify a single structure and its sub-structures
SF1.2.B. Accurately describe structures as consisting of smaller components.
SF1.3.A. Accurately identify multiple structures and including sub-structures
SF1.3.B. Already at highest
SF1.4.A. Already at highest
SF1.4.B. Already at highest
SF1.5.A. Already at highest
SF1.5.B. Already at highest
112
SF2. Function: Purpose of the structure
SF2.1.A. Describe function by inferring from observable conditions or surface details, but details are limited or significantly inaccurate.
SF2.2.A. Describe function by inferring from observable conditions or surface details, but details are limited.
SF2.3.A. Describe individual functioning of structures and sub-level structures but does not describe overall function as a product of sub-level conditions.
SF2.4.A. Describes the multiple functions of structures and sub-level structures. Infers function as a product of sub-level conditions. May be inaccurate.
SF2.5.A. Accurately describes the multiple functions of structures and sub-level structures. Accurately infers function as a product of sub-level conditions.
SF3. Behavior (Mechanisms): Attributes or specific states derived from the structure Examples: cell wall or membrane
SF3.2.A. Does not describe properties/behavior of structure
SF3.2.B. Does not describe conditions under which the properties of the structure support its function
SF3.2.C. Does not describe relationships between structural
SF3.3.A. Limited description of properties or behavior that are inferred from macro level structures only.
SF3.3.B. Limited description of conditions under which the properties of the structure supports its function
SF3.3.C. Limited or no description
SF3.4.A. Infer properties or behavior from articulating relationship among micro-macro structures and their function. May be inaccurate
SF3.4.B. Causal description of the conditions under which the properties of the structure supports its function. May be inaccurate
SF3.5.A. Accurately infer properties or behavior from articulating relationship between (sub)-structures and their function.
SF3.5.B. Accurate causal description of the conditions under which the properties supports its function.
SF3.5.C. Describe causal relationships between structural
113
components and overall function.
of causal relationships between structural components and overall function on one level only. May be inaccurate.
SF3.4.C. Describe causal relationships between structural components and overall function that are parallel and occur at multiple levels. May be inaccurate.
components and overall function that are parallel and integrated across multiple levels in relation to overall function of system
114
AppendixC:EcosystemTaskItemswithDesignRationales
QUESTION1:Whyworryabouttherootworminvasion?
Key:Unkeyed.ThisitemwasredesignedtoaskstudentstoranktheconcernsaccordingtotheirsalienceforthepeopleofFarmville.Question1:DesignRationale• What is the assessment item asking? This item introduces students to the relevantentities that will be at play over the course of the whole activity while probing theirunderstanding that disturbances to one element in a system can affect other systemcomponents.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto identify likelyoutcomesof therootworminvasion,particularlythatmultiplenegativeoutcomesmayoccur.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.
1. Thegoatswillnothaveenoughcorn toeat. –The corn rootworms (CRWs)directlydecreasestheamountofcornavailable.Goatseatthecorn,soareductionincorndecreasestheamounttheycaneat.
2. Thepeoplewillnothaveenoughcornasfood.–TheCRWsdirectlydecreasestheamountofcornavailable.PeopleinTownsvilleeatthecorn,soareductionincorndecreasestheamounttheycaneat.
3. Thecornrootwormswillspreadtootherfarms.–TheCRWsaremobileandcanmove fromfarmto farm(Note:checkrespondent’s thoughtsaboutcentralmechanismdescribedinthestory).
115
4. Jonahwill not have enough corn for earningmoney. –The CRWs directlydecreases the amount of corn available. Jonah sells excess corn to support hisfamily,soareductionincorndecreasestheamounthecansell.
5. Thecornplantswillnotsurviveandeventuallydieoff.–TheCRWsconsumethe corn, if left unchecked, the CRWs will consume all the corn (Note: checkrespondent’sthoughtsabouthumanabilitytopreventcollapse).
QUESTION2:Differentiatingtheroleoforganismsonthefarm
Key: Corn–Producer,Goat–PrimaryConsumer,Cornrootworm–PrimaryConsumer
Human–PrimaryConsumerandSecondaryConsumer
ItemDesignRationale• What is the assessment item asking? This item presents a variety of trophic rolesorganismscantakeonwithinanecosystemandasksrespondentstocorrectlyassociatestoryelements(corn,goats,people,andcornrootworms[CRWs])withthoseroles.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytodifferentiateprimaryandsecondaryconsumption,identifywhichkindoforganismsareconsidered producers, and to see CRWs as invasive consumers rather than keydecomposers.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.
7. Thecornareproducers–Cornproducesediblebiomassbyconvertingmatter(carbonintheairandwaterfromair/soil)byusingenergyfromthesun.
8. Thegoatsareprimaryconsumers.–Thegoatdirectlyconsumesthecorn.
116
9. The people of Townsville are both primary consumers and secondaryconsumers. –Humans consume corn both by directly eating it and by eatingorganisms(thegoats)thateatthecorn.
10. ThepeopleofTownsvilleareoneofeitherprimaryorsecondaryconsumers-Humansconsumecorneitherbydirectlyeatingitorbyeatingorganisms(thegoats) that eat the corn. (Note: check respondent’s reading of the story andwhethertheynotedthatbothconsumptionsoccur).
11. Thecornrootwormsareprimaryconsumers.–TheCRWsdirectlyconsumethecorn.
12. The corn rootworms are decomposers. – The CRWs cause the corn to die(decompose).
QUESTION3:Creatingthebaselineenergy-transfermodel
Key:Sun->Corn;Corn->GoatItemDesignRationale• What is theassessment itemasking? This itemprovides a selectionof elements thatoughttobeincorporatedviatheSageModelerinterface.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoincorporatetheappropriateentities,drawarrowsinappropriatedirection,theorderofelementincorporation,andanyerrorsmadealongtheway.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.
117
1. Sun->Corn->Goat–Thesunproducestheenergythatsupportstheproductionofplant(corn)matterthatisultimatelyconsumedbythegoat.
2. Goat->Corn->Sun–Thegoatsderiveenergyfromthecornwhichderiveenergyfrom the sun (Note: check on respondent’s understanding on themeaning ofarrowsandtheirdirectionality).
3. Sunnotinmodelornotconnectedtoothermodelelements–Foodwebsonlyshow the flow of matter, sunlight is not matter and, therefore, should not beincludedinthemodel(Note:checkonrespondentsreadingofthetaskprompt).
4. Goatconnectedviathesun–Thegoatsliveduringtheday(whenthesunisout);they would die if the sun was gone (reasoning may vary); the sun providesnecessarythermalenergytothegoats.
QUESTION4:Whatdomodelelementsrepresent?
Key:2ItemDesignRationale• What is theassessment itemasking? This item levels students from thepastmodel-creation taskandasks them to reflecton the informationabout theecosystem that themodelisattemptingtoconvey.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoidentifythemechanismsbywhichmodelelementsservearolewithintheecosystem(asrepresentedbythearrowsinthemodel).• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.
118
1. Transferofmatter in thesametrophic level.–All theorganismsare in thesamelevel(horizontally)thereforethematterisflowingwithinthatsinglelevel(Note:checkrespondent’sunderstandingofwhat“trophiclevels”are).
2. Transfer of energy in the ecosystem. –Energy in the ecosystem originatingfromtheSunmovesfromelementatthebackofthearrowtotheelementattheheadofthearrow.
3. Feedingrelationshipsbetweenorganisms.–ThegoatsonJonah’sfarmsurvivebyfeedingonthecorn.
4. Thepopulationsizeofeachorganism–Withintheecosystemthereisasun,acorn,andagoat.(Note:checkhowstudentswouldmodelecosystemifthefarmcontainedmanycornstalksormanygoats).
QUESTION5:Integratingthecornrootwormintothemodel
Key: Sun->Corn;Corn->Goat;Corn->Rootworm ItemDesignRationale• Whatistheassessmentitemasking?This itemprovidesthecornrootwormasanewelementtobeincorporatedintothepreviousmodelviatheSageModelerinterface.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoincorporatetheappropriateentities,drawarrowsinappropriatedirection,theorderofelementincorporation,andanyerrorsmadealongtheway.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.
1. Corn->CornRootworm–Cornisafood/energysourcefortherootworms.
119
2. CornRootworm->Corn–Therootwormsderiveenergy fromthecornwhichderiveenergyfromthesun(Note:checkonrespondent’sunderstandingonthemeaningofarrowsandtheirdirectionality).
3. CornRootwormconnectedtothegoats–TheCRWscompetewiththegoatsandcausethegoatpopulationtochange(Note:checkonrespondentsreadingofthetaskprompt).
QUESTION6:Whatdomodelelementsrepresent?
Key: 1-newfoodchain;foodweb
2-producerstoconsumers;consumerstoproducersItemDesignRationale• What is theassessment itemasking? This item levels students from thepastmodel-creationtaskandasksthemtoreflectonthechangestotheinformationcontainedwithinthemodelasafunctionoftheincorporationoftheCRW.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto relate thenewroleof theCRWwithin theecosystem toa change inwhereavailableenergyinthesystemsgoes.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.
1. Newfoodchain.–Previouslythemodelshowed1foodchain(sun->corn->goat)andnowitalsoshowsanotherfoodchain(sun->corn->CRW)aswell(Note:checkstudentreasoningaboutthedifferencebetweenmultiplefoodchainsandasinglefoodweb).
2. Foodweb.–TheintegrationoftheCRWsmeanswehavecreatedamodelinwhichmanyorganismsserveastheproducerfororconsumerofmanyotherorganisms.
120
3. Producers to consumers – The CRWs consume the corn and derive energyproducedfromitsgrowthprocess(photosynthesis).
4. Consumerstoproducers.–WhentheCRWsdie,theirbodiesprovidenutrientstothesoilthatcornneedstogrow(i.e.“circleoflife”naïveconception;Note:checkon respondent’s understanding on the meaning of arrows and theirdirectionality).
QUESTION7:Inferringtheconsequencesoftheinvasion
Key: Corn–decrease
Goat–decrease(ifcornistheonlyfoodavailableinthefarm)People–remainthesame
ItemDesignRationale• Whatistheassessmentitemasking?This itemasksrespondentstomakepredictionsaboutchangestopopulations(bothseenandunseen)livingintheecosystemasaresultoftheCRWinvasion.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto identify likelyoutcomesof therootworminvasion,particularlythatmultiplenegativeoutcomesmayoccur.Theyshouldbeabletobackthesepredictionswithastatementaboutevidenceavailableinthemodel.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexist for some select distractor responses. Further rationales may need to beinvestigatedwithinterviewprobes.
1. Cornpopulationdecreases.–Thecornrootworms(CRWs)consumethecorn,directlyreducingcornpopulationnumbers.
121
2. Cornpopulationincreases–WhentheCRWsdie,theirbodiesprovidenutrientstothesoilthatcornneedstogrow(i.e.“circleoflife”naïveconception;Note:checkonstudentconceptionsabouttimelinedisplayedinthemodel–thatthefoodwebisasnapshotintime).
3. Cornpopulationstaysthesame–Thecorndrawsitsenergyfromthesun,sonewconsumersdon’taffecthowmuchfoodthecorngets(orbyextensionthecornpopulation supportedby the ecosystem;Note: check respondent’s viewof theconnectionbetweenenergyacquisitionandmatterconsumptiondepictedinthemodel).
4. Goatpopulationdecreases–The invasionof theCRWsmeansthere isanewcompetitorforthecorn,decreasingtheamountofenergyavailabletosupportthecurrentgoatpopulation,andresultinginagoatpopulationdecrease.
5. Goat population stays the same –While the CRWs consume corn, goats aremobileandcanconsumeotherplantswiththeaidoffarmerJonah(Note:checkrespondent’sthoughtsaboutcentralmechanismdescribedinthestory).
6. Populationofpeopledecreases.–TheCRWsdirectlydecreasestheamountofcornavailable.Jonahandhisfamilyeatthecorn,soareductionincorndecreasestheamountheandhisfamilycanconsume(Note:checkrespondent’sthoughtsabouthumanabilitytopreventcollapse).
7. Population of people increases – Human populations increase because, onaverage,more children are born than die (Note: check respondent’s ability toconnectclaimmadeaboutpopulationininformationcontainedwithinthemodel)
8. Populationofpeoplestaysthesame.–Unlike thegoats,humanscanreadilyacquireenergyfromnewsourcesandarenotreliantoneatingcorn(hencewhytheywerenotnecessarytoincludewithinthefoodchain/web).
QUESTION8:Understandingtheconsequencesoftheinvasion
122
Key:2&4ItemDesignRationale• Whatistheassessmentitemasking?Thisitemprovidesavarietyofdatadisplaysfromthe results of the CRW-invasion simulation and asks students to reason about theconsequencesofnotimplementinganycontrolstrategies.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytolinkclaimsmadeabouttheimpactsoftheCRWinvasionwithevidencefromthedatadisplaysandreasoningaboutthemechanismsatplayduringtheinvasion.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.
1. Thenumberofcornplanted increased.–Since thecornrootworms(CRWs)directlydecreasestheamountofcornthatsurvives,thefarmerJonahshouldplantmore so that enough survives the season (Note: check respondent’s ability toconnectclaimmadeaboutpopulationinevidencecontainedinthedatadisplay)
2. Thenumberofcornharvesteddecreased.–Eachyearthenumberofcornthatsurvivesuntilharvestdecreasesperthedatainthegreencolumn.
3. Thecornrootwormsremainedthesame.–ThecorncanonlysupportafixednumberofCRWs, so theirpopulationwill be the sameover time (Note: checkrespondent’s ability to connect claim made about population in evidencecontainedinthedatadisplay).
4. Thenumberofcornrootwormeggsincreased.–AsCRWssurvivetheseason,theyareabletolaymorethan1eggperrootworm.ThiscausestheamountofCRWeggstorapidlyincrease.
123
QUESTION9:Determiningtheefficacyofharvestmenpredatorspt.1
Key:3,4,&5ItemDesignRationale• Whatistheassessmentitemasking?ThisitemprovidesavarietyofdatadisplaysfromtheresultsoftheCRW-invasionsimulationwhentheharvestmen-basedcontrolstrategywas implementedandasksstudents to reasonabout thechanges thatoccurred to farmpopulationsovertime.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytolinkclaimsmadeabouttheimpactsoftheCRWinvasionwithevidencefromthedatadisplaysandreasoningaboutthemechanismsatplayduringtheinvasion.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.
1. The%cornyieldremainedthesameeveryyear.–Implementingthecontrolstrategydecreases the growth rateof the corn rootworms (CRWs)population,improving corn yield percentage (Note: check respondent’s ability to connectclaimmadeaboutpopulationinevidencecontainedinthedatadisplay).
2. Thenumberofcornplantedincreasedeveryyear–Sincethecornrootworms(CRWs) directly decreases the amount of corn that survives, the farmer Jonahshouldplantmoresothatenoughsurvivestheseason(Note:checkrespondent’sabilitytoconnectclaimmadeaboutpopulationinevidencecontainedinthedatadisplay).
3. Thenumberofcornrootwormsincreasedeveryyear.–Despiteimplementingthe control strategy, the CRW population (as depicted in the pink column)continuedtoincrease.
4. The number of corn harvested decreased every year. – – Each year thenumberofcorn thatsurvivesuntilharvestdecreasesper thedata in thegreencolumn.
124
5. Thenumberofharvestmenremainedthesameeveryyear.–Theharvestmenare not native organisms to the farm and can only survive while there arerootwormsavailabletoeat.Duringthewinter,theydieoffandJonahmustreleaseanewsetof10eachspring.
QUESTION10:Determiningtheefficacyofharvestmenpredatorspt.2
ItemDesignRationale• Whatistheassessmentitemasking?Thisitemlevelsstudentssotheyrecognizerelevantpatterns inpopulationnumbers following the implementationof theharvestmen-basedcontrol strategy and asks them to provide some reasoning for why the strategy wasineffective.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto the criteria students generate for determining what counts as effective solutions(students might note that the harvestmen strategy is still better than nothing). Whatelementsofthesystem(andthesimulation)dostudentsfocuson?Dotheytiebacktheirresponsestothenot-simulatedgoatandhumanpopulations?• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfor some select naïve responses. Further rationales may need to be investigated withinterviewprobes.
1. Answerfocusesoninsufficientnumberofharvestmenaddedeachyear.2. Answerfocusesonthestrategystillbeingmoreeffectivethandoingnothingatall.3. AnswerfocusesontheavailableamountofenergytosupporttheCRWs.4. Answer focusesonvisual elements from thevideo (i.e. “theharvestmendidn’t
interactwiththebugsenough”)
125
QUESTION11:Determiningtheefficacyofthetrapcroppt.1
Key:1&3ItemDesignRationale• Whatistheassessmentitemasking?ThisitemprovidesavarietyofdatadisplaysfromtheresultsoftheCRW-invasionsimulationwhenthetrap-crop-basedcontrolstrategywasimplemented and asks students to reason about the changes that occurred to farmpopulationsovertime.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytolinkclaimsmadeabouttheimpactsoftheCRWinvasionwithevidencefromthedatadisplaysandreasoningaboutthemechanismsatplayduringtheinvasion.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistfordistractorresponses.
1. The%cornyieldremainedthesameeveryyear.–ImplementingthetrapcropdistractstheCRWswithlessnutritiousnon-cornalternatives,allowingthecornto grow tomaturitybeforedestructionby theCRWs (asdepicted in the greencolumn)whilealsolimitingtheCRWpopulation.
2. Thenumberofalfalfa-trap-cropplantedincreasedeveryyear.–Byplantinglotsofalfalfa,butnotharvesting it for thegoatsorhumans, itspopulationcanincrease(Note:checkthatstudentcanidentifyrelevantdatacolumns).
3. Thenumberofcornrootwormsdecreasedfromyear3toyear4.–TheCRWsdonotreceiveenoughnutritionfromthetrapcrop,inhibitingthemfromlayingenougheggstoreplacetheincidentCRWpopulation.
4. Thenumberofcornharvesteddecreasedfromyear3toyear4.–TheCRWsstillcontinuetoconsumethecorn,eveniftheyalsoconsumethealfalfa.Coupledwiththedecreaseincornplantedbyreplacingsomelandwiththenon-corntrap
126
crop,means that corn to harvestmay decrease (Note: check that student canidentifyrelevantdatacolumnsinrelevantyears).
QUESTION12:Determiningtheefficacyoftrapcroppt.2
ItemDesignRationale• Whatistheassessmentitemasking?Thisasksstudentstogenerateanargumentaboutthe efficacy of the trap-crop-based control strategy and asks them to provide somereasoningforwhythestrategywasorwasnoteffective.• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilityto the criteria students generate for determining what counts as effective solutions(studentsmight note that the trap crop strategy entails an immediate reduction in theamountofcornplanted/harvested).Whatelementsofthesystem(andthesimulation)dostudentsfocuson?Dotheytiebacktheirresponsestothenot-simulatedgoatandhumanpopulations?• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistforsomeselectnaïveresponses.Furtherrationalesmayneedtobeinvestigatedwithinterviewprobes.
1. Answer focuses on short term reduction, concluding the strategy wasineffective
2. Answerconcededshorttermreductionbutalsolong-termstabilization.3. Answerfocusesonthestrategystillbeingmoreeffectivethandoingnothing
atall.4. AnswerfocusesontheavailableamountofenergytosupporttheCRWs.5. Answerfocusesonvisualelementsfromthevideo(i.e.“thealfalfatakesup
toomuchspace”)
127
QUESTION13:Drawingaconclusionaboutthestrategies
ItemDesignRationale• Whatistheassessmentitemasking?Whatistheassessmentitemasking?Thisitemlevelsstudentssotheyrecognizerelevantpatternsinpopulationnumbersfollowingtheimplementationof the trap-crop-basedcontrol strategyandasks them toprovide somereasoning for why the strategy was effective (particularly several years afterimplementationbegan).• Whatinformationisimportant?Itisimportanttopayattentiontorespondents’abilitytoextendthereasoningandargumentfromQ12.• Whatistherationaleassociatedwitheachpossibleanswer?Notethatrationalesalsoexistforsomeselectnaïveresponses.Furtherrationalesmayneedtobeinvestigatedwithinterviewprobes.
6. Answer focuses on short term reduction, concluding the strategy wasineffective
7. Answerconcededshorttermreductionbutalsolong-termstabilization.8. Answerfocusesonthestrategystillbeingmoreeffectivethandoingnothing
atall.9. AnswerfocusesontheavailableamountofenergytosupporttheCRWs.10. Answerfocusesonvisualelementsfromthevideo(i.e.“thealfalfatakesup
toomuchspace”)
128
AppendixD:Post-TaskInterviewProtocol
Item2(optionalquestions-askonlyiftimeallows)
• HowdoesidentifyingtheroleofCORN,GOATSandCORNROOTWORMhelpscientiststhinkabouttheimpactsofthecornrootworm?
Item3(Required)
• WhydidyouaddelementX(sun,corn,goat)intothemodel?HowdidyouconnectittoelementY?
• WhydidyouleaveelementX(sun,corn,goat)outofthemodel?HowdidyouconnectittoelementY?
• Whatdothearrowsinthemodelrepresent?Howdidyoudecidetoplacetheminthedirectionyoudid(indicatedirectionbasedonnotes)?
• RelevantSageModelerusabilityquestions.o Howeasywasusingwiththemodelingtool?o Wasthetutorialvideohelpfultoteachyouwhattodo?o Wasitclearhowtofixamistake(ifmade)?o Anysuggestionsforimprovement?
ITEM5(Required)
• HowdidyoudecidetoconnecttherootwormstoelementY?(Alternative:Whydidyouchosenottoconnecttherootwormstoanyoftheothermodelelements?)
• Howdidaddingtherootwormtothemodelhelpthescientiststhinkabouttheimpactsoftheinvasiononthefarmecosystem?
ITEM7(Required)
• Howmighttheinvasionofthecornrootwormshavethestatedeffect?(Interviewershouldremindthestudentsoftheresponsestheyselected)
• Didyouusethemodeltomakeyourhypothesis?Ifso,howwasthemodelhelpful?VIDEO:ExaminingtheImpactoftheRootwormInvasion(Required)
• Wasthevideousefultohelpyouunderstandtherelationshipsinthesystem?• Basedonthevideo,whatwasthepurposeofthesimulation?Whatdidyoulearn
fromthevideo? ITEM8(Required)
• Usabilityquestionsregardingthedatatableandgraphs.o Whatdiddifferentcolumnsinthedatatablerepresent?o Foreachgraph,whatpatternsdidyousee?Whatistherelationshipbetween
thegraphandthedatatable?o Whatdidwemeanbyyieldpercentage?Whyisyieldpercentageauseful
measureofJonah’sFarm?
129
VIDEO:EnactingtheHarvestmenControlStrategy(Required)
• WasthevideousefultohelpyouunderstandhowHarvestmenControlStrategyworks?
• Basedonthevideo,whatwasthepurposeofthesimulation?Whatdidyoulearnfromthevideo?
ITEM10(Required)
• Whywereonly10harvestmenreleasedperyearonthefarm?• Whatotherdatamighthavebeenhelpfulindrawingaconclusionabouttheefficacy
oftheharvestmen-basedcontrolstrategy?VIDEO:EnactingtheAlfalfaTrapCropControlStrategy(Required)
• WasthevideousefultohelpyouunderstandhowtheAlfalfaTrapCropControlStrategyworks?
• Basedonthevideo,whatwasthepurposeofthesimulation?Whatdidyoulearnfromthevideo?
ITEM12(Required)
• WhydidplantingtheAlfalfadecreasetheamountofcornplanted?• Whatotherdatamighthavebeenhelpfulindrawingaconclusionabouttheefficacy
ofthetrap-crop-basedcontrolstrategy?ITEM13(Required)
• Whatinformationabouttheefficacyofthestrategywasaddedbylettingthesimulationrunforseveralmoreyears?
• Howdidyouusethevideosofthesimulatedcontrolstrategiestohelpyourespondtorelevantquestions?
OverallTaskFeedbackQuestions
• Didyoufeellikeyouhadalltheinformationneededtoengagewiththetask?Whatotherinformationmighthavebeenhelpful?
• Weretheexperiencesofthistasksimilartotasksyouhadalreadyengagedinwithinyourscienceclassroom?
• Doyouthinkthefarmscenariowasmeaningfultoyou?Ifnot,whattypesofscenarioswillbemoreengaging?
• Wereyouhopingthatyoucouldhavedirectinteractionswiththesimulationshowninthevideo?Ifyouwereabletointeractwiththesimulationshowninthevideo,whattestswouldyouliketorun?
[EndofSemi-StructuredProtocol][Iftime,interviewasksfortaskfeedbackoraboutinterestingobservationsmadeduring
thinkaloud]
130
AppendixE:DAT-CROSS:BackgroundQuestionnaire
Beforewegetstarted,pleaseanswerthefollowingquestions.
1. What grade are you currently in? ○ 6th grade ○ 8th grade ○ 10th grade
2. How old are you? _(text entry)_
3. What is your gender?
o Male o Female o Other__________ o Prefer not to answer
4. Which of the following best describes you? (Select all that apply.)
African American
White Asian Hispanic Pacific Islander Native American
Mixed Race Other_______________
Prefer not to answer
5. Have you received any instruction about systems? Explain and use examples if necessary
131
6. Have you received any instruction about system models? Explain and use examples if necessary
7. Have you received any instruction about structure and function? 8. Do you have any experience using online or virtual simulations in school?
132
AppendixF:DataCollectionMatrix
DataCollection AnalysisMethods
• Background information questionnaire (See
AppendixE)
• Assessment items that probe students to be
moreexplicitintheirCCCreasoningbutarenot
expectedtofitintoanactualteachingsetting.
o Mixture of Multiple Choice/Multiple
Select and Free Response Items that
examineaCCCacrossDCIsfromseveral
domains.(SeeAppendixC)
• In-situthinkaloudwithstudentsthatprovide
them greater opportunity to vocalize their
reasoning. This will allow us to form some
validity in our assessment rubric. (See
AppendixH)
• Post-task reflective interviews that aloud
students to discuss challenges working with
the task, amend their prior thinking, and
recommendchanges.(SeeAppendixD).
• Rubricforquestionnairethatcan
distinguish between differing
levels of sophistication in CCC
reasoning.
• Coded interviews to ensure that
the rubric aligns to the ways
students are thinking about the
assessmenttask.
• UsingRtogenerateRaschmodel
to confirm construct leveling in
studentsCCCreasoning.
• Structuralequationtodetermine
if different constructs are
divergent.
133
AppendixG:EticCodingSchemeforInterviewData
TheeticschemeforcodingdrawsfrompriorliteratureintothenatureofCCCs(Rivetetal.,
2016;Weiseretal.,2017)aswellasintoknownavenuesofprogressioninsystemthinking
(Breslynetal.,2016;Gunckeletal.,2012;Jin&Anderson,2012:Mohanetal.,2009;Songer
etal.,2009).
• CCCsasLenses–TheroleoftheCCCistohighlightsalientfeaturesthatmaynotbe
immediatelyobviousduetoscale,scope,orsize.
• CCCsasBridges–TheroleoftheCCCistodrawconnectionsbetweentwoentities,
facilitatingtransferofunderstanding.
• CCCsasLevers–TheroleoftheCCCistocombineseveral,relatedideas,entities,or
representations in order to make understanding take on a new form towards a
particulargoal.
• CCCs as Rules – The role of the CCC is to validate a representation’s utility in
explainingorpredictingthenaturalworld.
• System Phenomena – As students build sophistication in their thinking around
systems, they are better able to describe phenomena as a system of many
simultaneousinteractions.
• System Components – As students build sophistication in their thinking around
systems,theyarebetterabletobreakdownthecomponentsofasystemintotheir
constituent,dynamicparts.
• System Relationships – As students build sophistication in their thinking around
systems, they are better able to describe the relationships between previously
identifiedcomponentsofthesystem.
• System Boundaries – As students build sophistication in their thinking around
systems,theyarebetterabletodefinetheboundariesofthesystemandtrackthe
flowsofinputsandoutputsacrossthoseboundaries.
134
AppendixH:RubricforAssessingConstructedResponsesOverallGuidanceonUsingRubric
• EachCRitemshouldbescoredfrom0to3holistically.Thatis,whileguidanceisprovided
to assess the items in terms of the claim, evidence, and reasoning of their argument,
scoresshouldcorrespondtotheoverallqualityoftheperformance.
• Ateachscorelevelforeachiteminthisrubricyouwillseeaconceptualdefinition(in
black),guidanceforwhatelementsmustappearinordertobescoredatleastashighas
thecurrentlevel(inred),andanexampleasprovidedbyastudent(inblue).
• Duetolimitationsonthesophisticationofthetask,itisnotreasonabletoexpectstudents
todisplayabroadrangeofclaimswithintheirarguments.Scorelevels2and3mayhave
similarcriteriaforthequalityofargumentativeclaimsmadebystudentsintheiroverall
response.Thedistinctionbetweenascoreof2orof3restsonthesophisticationofthe
evidenceandreasoningprovidedaswellasthelackincorrectorinaccuratestatements.
Question10:Determiningtheefficacyofharvestmenpredatorspt.2
ScoreLevel ScoringCriteria3
(Thislevelbroadlyalignstolevel4
elementsonthelearningprogression)
Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).
o Studentresponsesatthislevelshouldclaimthatthefailureoftheharvestmenwasaresultofatleastoneofthefollowingaspectsofthesystem:
§ Therewasafinitenumberofharvestmenreleased,whichwas insufficient to consume the pre-existing number ofrootworms.
§ Therewasnotenoughtimeinaseasonfortheharvestmentoconsumeenoughrootworms.
o Additionalclaimscomparingtheefficacyoftheharvestmentothebaseline(inwhichnostrategyisimplemented)arepermittedeventhoughtheyarenotpartofthequestionprompt.
• Evidence(fromsystemcomponentsorrelationships,SM2/3):responsecitefrom changes to two or more system components that are accuratelyrelatedtotheuseoftherelevantcontrolstrategyandthedegreetowhichthosecomponentschanged(SM2.4.C+SM3.4.A).
o Studentmust cite evidence related to all three of the number ofharvestmen,thenumberofrootworms,andtheamountofcornthatsurvivestheseason.
135
o Response should include a statement of rate (i.e. how fast therootwormsareprocreatingand/orconsumingcorn).
o Studentmay also include a statement about rootworm survival–thatsomerootwormssurvivetheseasonwithoutbeingeatenbytheharvestmen.
o Additional evidence from the baseline (in which no strategy isimplemented)canbecitedwithoutpenalty.
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant mechanism related to a causal chain between theharvestmen,rootworms,andtheamountofcornharvested(SM4.5.D).Allthree components must be addressed. Reasoning provided must berelevanttotheclaimsstudentsmakeandtheevidencetheyhavecited.
o Student response must make reference to the followingrelationships:
§ Theharvestmenconsumesomeoftherootworms,butsomerootworms survive the season (student may but do nothavetoreferencelayingeggs).
§ If sufficientlymanyrootwormssurvive theseason(to layeggs fornextyear), thentheirpopulationcancontinue togrowevenastheharvestmenattempttoconsumethem.
§ Astherootwormpopulationgrows,theamountofcornthatsurvivesuntiltheharvestdecreases.
o Response should include a statement of rate attribution –attributing the rate (or changes to the rate) of one change inrelevant components (the corn yield and/or the rootwormpopulation)tothebehaviorofothersystemcomponents.
FromDatusability40:Theremightnothavebeenenoughharvestmentokeepthepopulationofcornrootwormsundercontrol,soeventhoughthepopulationofcornrootworms did not go up as rapidly as before, as shown in the graph data, thepopulation went up nevertheless, which also means that the number of cornharvestedstillwentdown.
2(Thislevelbroadlyalignstolevel3
elementsonthelearningprogression)
Studentresponsedemonstratesacceptableunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited, and their reasoning. The student response includes some incorrect orinappropriateelementsbutisstillsophisticated.
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).
o Studentresponsesatthislevelshouldclaimthatthefailureoftheharvestmenwasaresultofatleastoneofthefollowingaspectsofthesystem:
§ Therewasafinitenumberofharvestmenreleased,whichwas insufficient to consume the pre-existing number ofrootworms.
§ Therewasnotenoughtimeinaseasonfortheharvestmentoconsumeenoughrootworms.
136
o Additionalclaimscomparingtheefficacyoftheharvestmentothebaseline(inwhichnostrategyisimplemented)arepermittedeventhoughtheyarenotpartofthequestionprompt.
• Evidence (from system components or relationships, SM2/3): responsecitefromchangestotwoormoresystemcomponentsbutdoesnotprovidearelationshipbetweenthoseevidencesandarelevantcontrolmechanism(SM2.3.C).
o Studentmustciteevidencerelatedtoatleasttwoofthenumberofharvestmen,thenumberofrootworms,andtheamountofcornthatsurvivestheseason.
o Studentmay also include a statement about rootworm survival–thatsomerootwormssurvivetheseasonwithoutbeingeatenbytheharvestmen.
o Additional evidence from the baseline (in which no strategy isimplemented)canbecitedwithoutpenalty.
o Somestatedevidencecanbeinaccurateorinappropriate,providedthatsufficientlymuchaccurate/appropriateevidenceisalsocitedinsupportoftheclaim.
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant mechanism related to a causal chain between theharvestmen, rootworms, and the amount of corn harvested. All threecomponents must be addressed. Response includes a mixture of bothappropriateandinappropriatesystemproperties(SM3.4.A+SM4.4.D).
o Student response must make some reference to the followingrelationships:
§ Theharvestmenconsumesomeoftherootworms,butsomerootworms survive the season (student may but do nothavetoreferencelayingeggs).
§ If sufficientlymanyrootwormssurvive theseason(to layeggs fornextyear), thentheirpopulationcancontinuetogrowevenastheharvestmenattempttoconsumethem.
§ Astherootwormpopulationgrows,theamountofcornthatsurvivesuntiltheharvestdecreases.
o Responseincludesreferencestoappropriaterelationshipsbutfailsto accurately attribute changes in the number of rootworms oramountofcornharvestedtothoserelationships.
FromDatusability42:Ithinkaddingtheharvestmentothefieldfor5yearsdidnt{sic}helpbecause{sic}ifyoulooktheharvestmenpopulationonlyhit10inyear3whiletherootwormpopulationwasstartingatyear2withaninitialpopulationof18andafinalpopulationof53.Thisshowsthatthepopulationofrootwormswasalwaysgreaterthanthepopulationofharvestmenfromthestartmakingitharderfortheharvestmentokilloffanyrootworms.
1(Thislevelbroadlyalignstolevel2
elementson
Studentresponsedemonstratesinsufficientunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.Responsemaybebroadlyinaccurate,butstillarticulatetheexistenceofmultiplesystemcomponents(rootworms,rootwormeggs,corn,harvestmen,etc.)thatmayhavebeenrelevantduringthesimulation.
137
thelearningprogression)
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentinaccuratelydescribesefficacyofthecontrolmechanism(SM1.2.B).
o Studentresponsedirectlyanswersquestionbutdoesnotrefertoanyaspects/features/behavioroftheharvestmen.
§ May include statements like “Adding harvestmen to thefielddidnothelpincreasethecornyieldpercentage”
• Evidence (from system components or relationships, SM2/3): responsecitesfromchangestoonlyasinglesystemcomponent(SM2.2.A).
o Responseonlycitesfromchangesintheamountofcornharvestedas indicativeof theefficacyof theharvestmenORresponseonlycitesfromchangesinthepopulationofrootwormsasindicativeoftheefficacyoftheharvestmen.
o Studentmayrefertorootwormsurvivalbutonlyasevidenceoflackofefficacy,notasevidenceinsupportofarelationship(i.e.doesnotconnectsurvivaltofuturegrowthofrootwormpopulation).
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovidesarelevantmechanismrelatedtoasinglerelationshipbetweenanytwooftheharvestmen,rootworms,andtheamountofcornharvestedbutdoesn’taddressallcomponents(SM3.3.B).
o Response only describes a relationship between the harvestmenand the rootworms OR response only describes a relationshipbetweentherootwormsandthecorn.
FromDatusability9:Thenumberofcorndidnotincreasewhentheharvestmenwereputintothefield.Thisisbecuse{sic}therewerestillmanyrootwormeggsintheendoftheyear,andtheycan'tstoptherootwormscompletaly{sic}.
0(Thislevelbroadlyalignstolevel1
elementsonthelearningprogression)
Student response demonstrates little or no understanding of systems and theirresponse is missing argumentative elements. They may provide a simpledescriptionof the videoor restate information thatwaspreviouslyprovided tothemOrStudentresponseisofftopicorinappropriate.This score may also be assigned for any response that fails to meet minimumexpectationsforalevel1score.
• Student response indirectly answers question but does not include a“because”statement
FromDatusability7:Inthevideosimulater{sic}thehavestmen{sic}ateboththerootwormsanf{sic}thecornsothecornharvestedstilldecreased.
Question12:Determiningtheefficacyoftrapcroppt.2
ScoreLevel ScoringCriteria3
(Thislevelbroadly
Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.
138
alignstolevel4
elementsonthelearningprogression)
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).
o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawasa resultofat leastoneof the followingaspectsof thesystem:
§ Therootwormsatealfalfainsteadofeatingcorn.§ Eatingalfalfainsomewaypreventedtherootwormsfrom
layingeggso Additional claims comparing the efficacy of the alfalfa to the
baseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.
• Evidence(fromsystemcomponentsorrelationships,SM2/3):responsecitefrom changes to two or more system components that are accuratelyrelatedtotheuseoftherelevantcontrolstrategyandthedegreetowhichthosecomponentschanged(SM2.4.C+SM3.4.A).
o Studentmustciteevidencerelatedtoallthreeofthepresence,thenumberof rootworms, and the amountof corn that survives theseason.
o Response should include a statement of rate (i.e. how fast therootwormsareprocreatingand/orconsumingcorn).
o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is
implemented)canbecitedwithoutpenalty.• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):response
provides a relevant mechanism related to a causal chain between thealfalfa, rootworms, and the amount of corn harvested. All threecomponentsmustbeaddressed.Reasoningprovidedmustberelevanttothe claims students make and the evidence they have cited(SM4.4.B+SM2.5C).
o Studentresponsemustmakereferencetoallthreeofthefollowingrelationships:
§ Therootwormsconsumealfalfa inaddition to consumingcorn.
§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.
§ Astherootwormsconsumethealfalfainsteadofthecorn,thecornyieldimproveseitherbecausethealfalfaisbeingeateninplaceofthecornorbecauseconsumingthealfalfadecreasestherootwormpopulation(andtherebydecreasesthelossofcorn).
o Response should include a statement of rate attribution –attributing the rate (or changes to the rate) of one change inrelevant components (the corn yield and/or the rootwormpopulation)tothebehaviorofothersystemcomponents.
FromDatusability12:Ithinkthatitdoes,anditismoreeffectivethanmethodone,thisisbecauseeventhoughthestartwasrough,the%didstarttogoup.Alsotherewas a higher harvest when you look at the # of corn planted to the # of corn
139
harvested.Therootwormeggsalsostartedtodecrease.Ithinkthiswasbecausetheydidn'thaveenough time toplanteggsbecause theywereeating thealfalfainsteadofthecorn,whichishowtheygrow.Inthesimulation,therootwormswerebecomingadults in the fall innstead{sic}of thesummer,whichmeant that theycouldn'talllaytheireggs,makingtheproblemlessofanissue.
2(Thislevelbroadlyalignstolevel3
elementsonthelearningprogression)
Studentresponsedemonstratesacceptableunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited, and their reasoning. The student response includes some incorrect orinappropriateelementsbutisstillsophisticated.
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).
o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawasa resultofat leastoneof the followingaspectsof thesystem:
§ Therootwormsatealfalfainsteadofeatingcorn.§ Eatingalfalfainsomewaypreventedtherootwormsfrom
layingeggso Additional claims comparing the efficacy of the alfalfa to the
baseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.
• Evidence (from system components or relationships, SM2/3): responsecitefromchangestotwoormoresystemcomponentsbutdoesnotprovidearelationshipbetweenthoseevidencesandarelevantcontrolmechanism(SM2.3.C).
o Studentmustciteevidencerelatedtoallthreeofthepresence,thenumberof rootworms, and the amountof corn that survives theseason.
o Response should include a statement of rate (i.e. how fast therootwormsareprocreatingand/orconsumingcorn).
o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is
implemented)ortotheharvestmenmethodcanbecitedwithoutpenalty.
o Somestatedevidencecanbeinaccurateorinappropriate,providedthatsufficientlymuchaccurate/appropriateevidenceisalsocitedinsupportoftheclaim.
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant mechanism related to a causal chain between thealfalfa,rootworms,andtheamountofcornharvested.Allthreecomponentsmustbeaddressed.Responseincludesamixtureofbothappropriateandinappropriatesystemproperties(SM3.4.A+SM4.4.D).
o Studentresponsemustmakesomereferencetoatleastoneofthefollowingrelationships:
§ Therootwormsconsumealfalfa inaddition to consumingcorn.
140
§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.
§ Astherootwormsconsumethealfalfainsteadofthecorn,thecornyieldimproveseitherbecausethealfalfaisbeingeateninplaceofthecornorbecauseconsumingthealfalfadecreasestherootwormpopulation(andtherebydecreasesthelossofcorn).
o Responseincludesreferencestoappropriaterelationshipsbutfailsto accurately attribute changes in the number of rootworms oramountofcornharvestedtothoserelationships.
FromDatusability34:Growingalfalfahelpedkeepthe%ofthecornyieldashighaspossiblebecausewhenthealfalfawasneartotherootworms,therootwormsdied,whichdecreasedtheamountofcorntheycouldeat,becausetheirwaslessofthem.
1(Thislevelbroadlyalignstolevel2
elementsonthelearningprogression)
Studentresponsedemonstratesinsufficientunderstandingofsystemsandsystemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.Responsemaybebroadlyinaccurate,butstillarticulatetheexistenceofmultiplesystemcomponents(rootworms,rootwormeggs,corn,alfalfa,etc.)thatmayhavebeenrelevantduringthesimulation.
• Claim(aboutsystemphenomena,SM1):responseaddressesonlyoneofthe“reducethecornrootworm”or“keepthecornyieldhigh”componentsofthe question. Response shows that the student inaccurately describesefficacyofthecontrolmechanism(SM1.2.B).
o Studentresponsedirectlyanswersquestionbutdoesnotrefertoanyaspects/features/behaviorofthealfalfa.
§ Mayincludestatementslike“Plantingalfalfainthefielddidnothelpincreasethecornyieldpercentage”
• Evidence (from system components or relationships, SM2/3): responsecitesfromchangestoonlyasinglesystemcomponent(SM2.2.A).
o Responseonlycitesfromchangesintheamountofcornharvestedas indicative of the efficacy of the alfalfaORresponse only citesfromchangesinthepopulationofrootwormsasindicativeoftheefficacyofthealfalfa.
o Studentmayrefertothetrade-offbetweenlandusedtoplantcornandlandusedtoplantalfalfa,butonlyasevidenceoflackofefficacy,notasevidenceinsupportofarelationship(i.e.doesnotconnectsurvivaltofuturegrowthofrootwormpopulation).
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovidesarelevantmechanismrelatedtoasinglerelationshipbetweenanytwoofthealfalfa,rootworms,andtheamountofcornharvestedbutdoesn’taddressallcomponents(SM3.3.B).
o ResponseonlydescribesarelationshipbetweenthealfalfaandtherootwormsORresponseonlydescribesarelationshipbetweentherootwormsandthecorn.
FromDatusability41:Thealfalfaplantedinthecornfielddoeshelpthecornyieldashighasitcanbe.Andeveninyear3-4therewasanincreaseofcornharvested.
141
Howeverthereisthedownsideofnotplantingasmuchcornaspossiblebutthismethodseemedtoworkbetterthanthefirstone.
0(Thislevelbroadlyalignstolevel1
elementsonthelearningprogression)
Student response demonstrates little or no understanding of systems and theirresponse is missing argumentative elements. They may provide a simpledescriptionof the videoor restate information thatwaspreviouslyprovided tothemOrStudentresponseisofftopicorinappropriate.This score may also be assigned for any response that fails to meet minimumexpectationsforalevel1score.From Datusability9: Yes, becuse {sic} the results produce a more suggestingnumberthatthecornisharvestingmorewhenthereisAlfalfainit.Thisisbecuse{sic}theAlfafa{sic}isstrongerandcangetridofmorerootworms.
Question13:Drawingaconclusionaboutthestrategies
ScoreLevel ScoringCriteria3
(Thislevelbroadlyalignstolevel4
elementsonthelearningprogression)
Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).
o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawillorwillnotcontinueintofutureyearsbecauseofatleastoneofthefollowingaspectsofthesystem:
§ The alfalfa was successful in reducing the rootwormpopulationinyears3-5
§ Thecornrootwormpopulationincreasedwhenalfalfawasnotplanted
o Response should include a statement of prediction in which thestudent asserts that the strategy will/will not continue to beeffectiveinthefuture.
o Additional claims comparing the efficacy of the alfalfa to thebaseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.
• Evidence(fromsystemcomponentsorrelationships,SM2/3):responsecitefrom changes to two or more system components that are accuratelyrelatedtotheuseoftherelevantcontrolstrategyandthedegreetowhichthosecomponentschanged(SM2.4.C).
o Student must cite evidence related to all three of the presencealfalfa, the number of rootworms (or rootworm eggs), and theamountofcornthatsurvivestheseason.
142
o Response may also include a statement regarding balance orequilibrium(i.e.“heldtherootwormsatbay”).
o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is
implemented)ortotheharvestmenmethodcanbecitedwithoutpenalty.
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant dynamism (ability to change in the future) to theconsequencesofacausalrelationshipbetweenthealfalfa,rootworms,andtheamountof cornharvested.All threecomponentsmustbeaddressed.Reasoningprovidedmustberelevanttotheclaimsstudentsmakeandtheevidencetheyhavecited(SM4.5.D+SM4.5.E).
o Student response must make reference to both of the followingrelationships:
§ The rootworms population grows in the absence of acontrolstrategy
§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.
o Responsemust includea statementofpredictability inwhich thestudent asserts that evidence from prior years can be used toestimatethefuturestateofthesystem.
o Response may include a statement of dynamism in which thestudent asserts that aspects of the system (like number ofrootwormsoramountofcornsurvivingtoharvest)aresubjecttochangeinresponsetochangestothesystem(i.e.theintroductionofacontrolstrategy).
FromDatusability6:Jonahcancontinuetoplantalfalfainyear7.Hecancontinuethatbecauseithelpskeepingthecornhealthyandkeepingthecornrootwormsaslowaspossible. In thedata table it shows that fromyear1 toyear2 thecorndecreasedbyalotwhenhedidnt{sic}usethealfalfa.Butwhenheusedit,itactuallyincreased.Intheotherdatatableoftherootwormeggsitshowsthatfromyear1toyear2theeggsincreaseed{sic}whenhedidnt{sic}plantalfalfa.Butfromyear3toyear5therootwormsdecreased.Inconclusion.Jonahshouldusethealfalfainyear7.
2(Thislevelbroadlyalignstolevel3
elementsonthelearningprogression)
Student response demonstrates complex understanding of systems and systemmodelingwithinthethreemainpartsof theirargument: theclaim,theevidencecited,andtheirreasoning.
• Claim(aboutsystemphenomena,SM1):responseshowsthatthestudentcanaccuratelydescribeefficacyofthecontrolmechanismusingatleastoneunseen/hidden/underlyingmechanism(SM1.3.B).
o Studentresponsesatthislevelshouldclaimthatthesuccessofthealfalfawillorwillnotcontinueintofutureyearsbecauseofatleastoneofthefollowingaspectsofthesystem:
§ The alfalfa was successful in reducing the rootwormpopulationinyears3-5
§ Thecornrootwormpopulationincreasedwhenalfalfawasnotplanted
143
o Response should include a statement of prediction in which thestudent asserts that the strategy will/will not continue to beeffectiveinthefuture.
o Additional claims comparing the efficacy of the alfalfa to thebaseline (in which no strategy is implemented) or to theharvestmenmethodarepermittedeventhoughtheyarenotpartofthequestionprompt.
• Evidence (from system components or relationships, SM2/3): responsecitefromchangestotwoormoresystemcomponentsbutdoesnotprovidearelationshipbetweenthoseevidencesandarelevantcontrolmechanism(SM2.3.C).
o Studentmustciteevidencerelatedtoall threeof thepresenceofalfalfa, the number of rootworms (or rootworm eggs), and theamountofcornthatsurvivestheseason.
o Response may also include a statement regarding balance orequilibrium(i.e.“heldtherootwormsatbay”).
o Studentmayalsoincludeastatementaboutrootwormeggs.o Additional evidence from the baseline (in which no strategy is
implemented)ortotheharvestmenmethodcanbecitedwithoutpenalty.
o Somestatedevidencecanbeinaccurateorinappropriate,providedthatsufficientlymuchaccurate/appropriateevidenceisalsocitedinsupportoftheclaim.
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovides a relevant dynamism (ability to change in the future) to theconsequencesofacausalrelationshipbetweenthealfalfa,rootworms,andtheamountof cornharvested.All threecomponentsmustbeaddressed.Response includes a mixture of both appropriate and inappropriatepotentialrelationshipdynamics(SM3.4.B+SM4.4.E).
o Student response makes reference to only one of the followingrelationships:
§ The rootworms population grows in the absence of acontrolstrategy
§ Consumingalfalfa in somewayprevents corn rootwormsfromsurvivingorprocreating.
o Response may include a statement of dynamism in which thestudent asserts that aspects of the system (like number ofrootwormsoramountofcornsurvivingtoharvest)aresubjecttochangeinresponsetochangestothesystem(i.e.theintroductionofacontrolstrategy).
FromDatusability13:Therootwormsdecreasedintheamountoftimethealfalfawasbeingused.Thismethodwouldprobablycontinuetoworkbecause theeggcounthasalreadygonedown,andwhentheeggcountgoesdown,morecorncanbeharvested.Therewillbelessrootwormstoeatthecorn.
1(Thislevelbroadlyalignsto
Studentresponsedemonstratesinsufficientunderstandingofsystemsandsystemmodeling(?) within the three main parts of an argument: claim, evidence, andreasoning.
144
level2elementsonthelearningprogression)
• Claim(aboutsystemphenomena,SM1):responseincludesaclaimthatonlyaddresses the state of the alfalfa. Response shows that the studentinaccuratelydescribesefficacyofthecontrolmechanism(SM1.2.B).
o Response should include a statement of prediction in which thestudent asserts that the strategy will/will not continue to beeffective in the future but does not refer to anyaspects/features/behaviorofthealfalfa.
§ Mayincludestatementslike“Plantingalfalfainthefielddidnothelpincreasethecornyieldpercentage”
• Evidence (from system components or relationships, SM2/3): responsecitesfromchangestoonlyasinglesystemcomponent(SM2.2.A).
o Responseonlycitesfromchangesintheamountofcornharvestedas indicative of the efficacy of the alfalfaORresponse only citesfromchangesinthepopulationofrootwormsasindicativeoftheefficacyofthealfalfa.
o Studentmayrefertothetrade-offbetweenlandusedtoplantcornandlandusedtoplantalfalfa,butonlyasevidenceoflackofefficacy,notasevidenceinsupportofarelationship(i.e.doesnotconnectsurvivaltofuturegrowthofrootwormpopulation).
o Responsemaycitedirectlyfromthegraphswithoutconnectiontoacausalrelationshipormechanism.
• Reasoning(fromsystemrelationshipsandmechanisms,SM3/4):responseprovidesarelevantmechanismrelatedtoasinglerelationshipbetweenanytwoofthealfalfa,rootworms,andtheamountofcornharvestedbutdoesn’taddressallcomponents(SM3.3.B).
o ResponseonlydescribesarelationshipbetweenthealfalfaandtherootwormsORresponseonlydescribesarelationshipbetweentherootwormsandthecorn.
o Student overemphasizes slight variation in years 4, 5, and 6 toassertthatthecontrolstrategyisnolongereffective.
o Response may include an inaccurate statement of dynamism inwhichthestudentassertsthataspectsofthesystem(likenumberof rootworms or amount of corn surviving to harvest) will notchange despite changes to other aspects of the system (like thecessationofcontrolstrategyimplementation).
FromDatusability40: Ithinkthatgrowingalfalfainyear7tohelpcontrolcornrootwormswill not help, because although the number of corn rootworms diddecreasewhen the alfalfawas planted in years 3 and 4, the numbers began toincreaseagain inyear5,and if it continuesat thatsamerate,plantingalfalfa inyears6and7willnotcontinuetohelp.
0(Thislevelbroadlyalignstolevel1
elementson
Student response demonstrates little or no understanding of systems and theirresponse is missing argumentative elements. They may provide a simpledescriptionof the videoor restate information thatwaspreviouslyprovided tothemOrStudentresponseisofftopicorinappropriate.
145
thelearningprogression)
This score may also be assigned for any response that fails to meet minimumexpectationsforalevel1score.FromDatusability7: On the second chart rootworms goes up and down so hecannotpredictwhatwillhappennextyearandforthatreasonhecannotcontroltheamountofrootworms
146
AppendixI:RScriptforStructuralEquationandRaschModellibrary(psych) library(GPArotation) library(eRm) library(mixRasch) library(dplyr) library(haven) library(lavaan) library(QuantPsyc) library(MASS) library(Hmisc) library(readxl) library(semPlot) #Read in data DATXecosystemUsabilityData <- read_excel("Dissertation Stuff/DATX Score Analysis.xlsx") ItemScoresOnly<- dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,Q8:Q13) #checking for unidimensionality scree(ItemScoresOnly) pca(ItemScoresOnly, nfactors = 2,residuals=TRUE) #sem cfa model <- ' #measurement model StructureFunction =~ Q8 + Q7_SUM + Q6_SUM + Q5 + Q4 + Q3 + Q2_SUM SystemModels =~ Q13 + Q12 + Q11 + Q10 + Q9 #regressions StructureFunction ~~ SystemModels StructureFunction ~ Grade SystemModels ~ Grade ' fit<-lavaan::sem(model, data = DATXecosystemUsabilityData) summary(fit, standardized=TRUE) semPlot::semPaths(fit) semPlot::semPaths(fit,what="path",whatLabels = "par") #Cronbach Alpha psych::alpha(ItemScoresOnly) psych::alpha(dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,-Q8)) psych::alpha(dplyr::select(DATXecosystemUsabilityData,Q9:Q13)) #Rasch modeling-Combined RaschCombined<-eRm::PCM(dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,Q8:Q13)) #Average Difficulty and Threshold position thresholds(RaschCombined) #Item Category Curves plotICC(RaschCombined, mplot = TRUE, legpos = FALSE, ask = FALSE) #Person Item Map plotPImap(RaschCombined,sorted=TRUE,main="Person-Item Map of Combined Behavior Items")
147
#item-fit statistics itemfit(person.parameter(RaschCombined)) ######Examining Fit Statistics RaschCombinedPersons<-person.parameter(RaschCombined) RaschCombinedPersonResid<-residuals(RaschCombinedPersons) scree(RaschCombinedPersonResid) #some indication that unidimensionality assumption fails CombinedTheta<-RaschCombinedPersons$theta.table ####Repeat with SF Separated RaschSF<-eRm::PCM(dplyr::select(DATXecosystemUsabilityData,Q2_SUM:Q5,Q6_SUM,Q7_SUM,Q8)) #Average Difficulty and Threshold position thresholds(RaschSF) #Item Category Curves plotICC(RaschSF, mplot = TRUE, legpos = FALSE, ask = FALSE) #Person Item Map plotPImap(RaschSF,sorted=TRUE,main="Person-Item Map of Structure Function Items") #item-fit statistics itemfit(person.parameter(RaschSF)) ######Examining Fit Statistics RaschSFPersons<-person.parameter(RaschSF) RaschSFPersonResid<-residuals(RaschSFPersons) scree(RaschSFPersonResid) #some indication that unidimensionality assumption fails SFTheta<-RaschSFPersons$theta.table ####Repeat with SSM Separated RaschSSM<-eRm::PCM(dplyr::select(DATXecosystemUsabilityData,Q9:Q13)) #Average Difficulty and Threshold position thresholds(RaschSSM) #Item Category Curves plotICC(RaschSSM, mplot = TRUE, legpos = FALSE, ask = FALSE) #Person Item Map plotPImap(RaschSSM,sorted=TRUE,main="Person-Item Map of Systems and System Model Items") #item-fit statistics itemfit(person.parameter(RaschSSM)) ######Examining Fit Statistics RaschSSMPersons<-person.parameter(RaschSSM) RaschSSMPersonResid<-residuals(RaschSSMPersons) scree(RaschSSMPersonResid) #some indication that unidimensionality assumption fails SSMTheta<-RaschSSMPersons$theta.table #Wilcoxon Signed Rank Test for difference between Thetas wilcox.test(SFTheta$`Person Parameter`,SSMTheta$`PersonParameter`,paired = FALSE)
148
AppendixJ:DefinitionsofSomeKeyTerms
• CognitiveLabs(orCogLabsorUsabilityTests)–Inassessmentdesign,acriticalstepinthedevelopmentofacomplexinstrument(suchasthoseusedintheDAT-CROSSprojects) are guided interviews (typically think-alouds), surveys, and item-responseanalysesthatexaminethedegreetowhichtheinstrumentiseffectivelyengagingrespondentsinthekindsofcognitionrelevanttothemeasuredconstructwithoutposingunnecessarychallengesthatmaysystematicallyprohibitthesubjectfrom acting toward their true ability. Drawing on the focus of diminishing non-constructrelevantbarrierstoparticipants’useoftheinstrument,theprocessmayalsobereferredtoas‘usabilitytesting’(Zaharias&Poylymenakou,2009).
• Models– In teachingparlance,modelling isan instructionalbehavior (i.e. the teachermodels thebehaviorshewishes thestudent toenact).However,models in thispaperrefer to scientific models – means of representing science knowledge by mappingabstractscienceconceptsontopictures,words,orotherstructures.Modelsservemanypurposes in science and some philosophers argue that all science knowledge can bethoughtofintermsofthevariousrepresentationsscientistsusetoexplain,describe,andpredict the naturalworld.More onmodels in the NGSS can be found in Kracjik andMerritt(2012).
• Performance Expectations – The standards in the NGSS are not a mere listing ofimportantideasinscience(thoughthisdimensiondoesappearasDCIs)becauseacentralthemeoftheNGSSisthatwhatstudentsdoordonotknowandonlybeevaluatedintermsoftheoperationalizationofthatknowledge.PerformanceexpectationsintheNGSSdetailwaysstudentscanoperationalizetheirscienceknowledgeviasomeactivity.
• Practices – In this paper, “practices” refer to the suite of activities authentic to thescientificcommunityoutlinedintheScienceandEngineeringPractices(SEPs)dimensionoftheFramework(NRC,2012).Theseactivitiesrepresentthevariouswaysthatscientistsengagewiththeirscienceknowledge,andfeatureasadominantpartofthelanguageofeachperformanceexpectationoftheNGSS.
• Scripts–ThefociofinvestigationinEngestrom’s(2000)activitytheoryanalyticalframeworkarethedevelopedschemaforhowpeopleengageinvariousactivitiesthatmakeupbotheverydayandprofessional life.Engestromcalls these schemascripts.Muchlikeascriptforanactor,activityscriptsareconstructedpriortoanyspecific instanceof anactivity inorder to informhow toproceedas theactivityoccurs.Justastheproductionofaplaycanbedisruptedinsuchawayastopushanactor off-script, so too candisruptions in the learning settingpush a student offscript. While there can be value in disrupting a script for the purpose ofconstructivist learning, such disruptions are an impediment to effectivemeasurementofstudents’currentstatesofunderstanding.
149
• Storyboards – The development of rich performance assessments entails themovement through a design process that starts at a very general level (domainmodeling),movestoincludespecificsofwhatperformancesmightbepossible(taskmodels), and culminates as a sequence of stimuli (including written prompts,videos,andimages)andresponsecollections.Thisseriesofstimuliandresponse,thelastdesignstagebeforeataskbecomeoperational,comprisesadocumentcalledthe “storyboard” (NRC, 2014) which details the specifics of what students willinteractwithastheyengageinthetask.