Are Texas' English language arts and reading...

I S S U E S & A N S W E R S R E L 2 0 1 0 – N o . 0 9 1

At Edvance Research, Inc.

Are Texas’ English language arts and reading standards college ready?

I S S U E S&ANSWERS R E L 2 0 1 0 – N o . 0 9 1


AreTexas’Englishlanguageartsandreadingstandardscollegeready?

August2010

Preparedby

EricRolfhus,Ph.D.EdvanceResearch

GaryCook,Ph.D.WisconsinCenterforEducationResearch

JessicaL.Brite,M.S.EdvanceResearch

JeniferHartman,Ed.D.EdvanceResearch

WA

OR

ID

MT

NV

CA

UT

AZ

WY

ND

SD

NE

KSCO

NM

TX

OK

CO

AR

LA

MS AL GA

SC

NC

VAWV

KY

TN

PA

NY

FL

AK

MN

WI

IA

IL IN

MI

OH

VT

NH

ME

MO


Issues&Answersisanongoingseriesofreportsfromshort-termFastResponseProjectsconductedbytheregionaleduca-tionallaboratoriesoncurrenteducationissuesofimportanceatlocal,state,andregionallevels. FastResponseProjecttopicschangetoreflectnewissues,asidentifiedthroughlaboutreachand requestsforassistancefrompolicymakersandeduca-torsatstateandlocallevelsandfromcommunities,businesses,parents,families,andyouth.AllIssues&AnswersreportsmeetInstituteofEducationSciencesstandardsforscientificallyvalidresearch.

August2010

ThisreportwaspreparedfortheInstituteofEducationSciences(IES)underContractED-06-CO-0017byRegionalEduca-tionalLaboratorySouthwestadministeredbyEdvanceResearch.ThecontentofthepublicationdoesnotnecessarilyreflecttheviewsorpoliciesofIESortheU.S.DepartmentofEducationnordoesmentionoftradenames,commercialproducts,ororganizationsimplyendorsementbytheU.S.Government.

Thisreportisinthepublicdomain.Whilepermissiontoreprintthispublicationisnotnecessary,itshouldbecitedas:

Rolfhus,E.,Cook,H.G.,Brite,J.L.,andHartman,J.(2010).Are Texas’ English language arts and reading standards college ready?(Issues&AnswersReport,REL2010–No.091).Washington,DC:U.S.DepartmentofEducation,InstituteofEduca-tionSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouth-west.Retrievedfromhttp://ies.ed.gov/ncee/edlabs.

Thisreportisavailableontheregionaleducationallaboratorywebsiteathttp://ies.ed.gov/ncee/edlabs.

Summary REL 2010–No. 091

i

Are Texas’ English language arts and reading standards college ready?

This study compares alignment of the ACT and the American Diploma Proj-ect (ADP) national college readiness standards sets with the Texas Essential Knowledge and Skills for English lan-guage arts and reading (TEKS ELAR) standards for grades 9–12 and analyzes their cognitive complexity. It finds that a majority of the content in the ACT and ADP standards sets is addressed to some extent by the TEKS ELAR standards and that the TEKS ELAR standards demand higher levels of cognitive complexity than do the other two standards sets.

College readiness has recently emerged as a national issue, driven in part by repeated find-ings that many first-year college students are required to take remedial courses (for exam-ple, Provasnik and Planty 2008; Terry 2007). In response, several sets of national college readiness standards (content statements that define what students should know in specific areas) have been developed, such as the ACT College Readiness Standards (ACT, Inc. 2007) and the American Diploma Project (ADP) College and Workplace Readiness Benchmarks (Achieve, Inc. 2004). An emphasis on college readiness standards is also evident in the dis-tribution of American Reinvestment and Re-covery Act education funds (U.S. Department of Education 2009) and in the 2009 Common

Core State Standards Initiative, sponsored by the National Governors Association and the Council of Chief State School Officers, which is developing a national set of K–12 English language arts and mathematics standards that includes college readiness standards (Missouri Department of Elementary and Secondary Education 2009; South Carolina Department of Education 2009).

Although Texas has not participated in this national initiative, recent state legislation has focused on developing college readiness standards, vertically aligning the state’s K–12 curriculum to those standards through a logi-cal progression for teaching content in a subject area across grades, and raising state standards for student performance to move Texas into the top 10 states in college readiness by 2019/20 (Texas Legislature 2006, 2009). Thus, state leaders need to understand how the Texas Es-sential Knowledge and Skills (TEKS) standards for grades 9–12 relate to college readiness ex-pectations. To support this work, an alignment study was requested comparing the 2008 TEKS English language arts and reading (TEKS ELAR) standards (Texas Education Agency 2008) and two national English language arts college readiness standards sets, ACT and ADP.

The study assessed alignment on two di-mensions: content (the knowledge and skills

ii Summary

representedbyastandardsstatement)andcognitivecomplexity(thelevelofreasoningorcognitivedemandonstudentsrepresentedbyastandardsstatement).Twoquestionswereexamined:

• WhatpercentageofcontentstatementsintheACTandAmericanDiplomaProject(ADP)collegereadinessstandardssets(thebenchmarksets)alignfullyorpar-tiallywithcontentstatementsinthe2008TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreading(TEKSELAR)grade9–12standardsset(thecom-parisonset)?

• Foreachofthesestandardssets,whatisthedistributionofcontentstatementsacrossthefourlevelsofacognitivecom-plexity(cognitivedemand)scale?

Oncontentalignment,thestudyfindsthatamajorityofcontentintheACTandADPcollegereadinessstandardssetsisaddressedtosomeextentbytheTEKSELARstandards.Specifically,

• FourteenpercentofACTstatementsfullyalignand75percentpartiallyalignwithTEKSELARstatements.

• Forty-eightpercentofADPstatementsfullyalignand45percentpartiallyalignwithTEKSELARstatements.

• TheproportionofACTstatementsthatfullyalignwithTEKSELARstatementsvariesacrossACTcontentstrandsfrom5percentto29percent,andtheproportionthatpartiallyalignsvariesfrom55percentto89percent.

• TheproportionofADPstatementsthatfullyalignwithTEKSELARstatementsvariesacrossADPcontentstrandsfrom0 percentto67percent,andtheproportionthatpartiallyalignsvariesfrom22percentto75percent.

Theseresultsaredifficulttointerpretinisolation,astherearenouniversallyacceptedcriteriafordetermininggoodorpoorlevelsofalignment.Reportingthefindingsinrelationtoanotherstandards-to-standardsalignmentstudy(Rolfhusetal.2010)canprovidecontextforinterpretingthefindings.Offivepairwisecomparisons(threeinRolfhusetal.andtwointhecurrentstudy),theADP–TEKScomparisoninthecurrentstudyhasthehighestpercentageofbothfullyalignedcontentandcombinedfullyandpartiallyalignedcontent.TheACT–TEKScomparisoninthecurrentstudyranksfourthinfullyalignedcontentandsecondincombinedfullyandpartiallyalignedcontent.ThesetwostudiesindicatethatTEKSELARalignsmorecloselytoADPthananyoftheotherthreenationalEnglishlanguageartscol-legereadinessstandardsexamined.

TheTEKSELARstatementsdemandhigherlevelsofcognitivecomplexitythanbothbenchmarkcollegereadinessstandardssetsexaminedinthisstudyandthetwoadditionalstandardssets(CollegeBoard,StandardsforSuccess)examinedinRolfhusetal.(2010).Inthecurrentstudy,theADPandTEKSELARstandardssetsexhibitthemostsimilarities.Othernotablefindings:

• Eachofthefourlevelsofcognitivecom-plexity(recall,skill/concept,strategicthinking,andextendedthinking)wasrepresentedineachofthestandardssets.

iii Summary

• Themajorityofstatementsineachstan-dardssetwereratedatlevel3–strategicthinking(55percentforACTandADPand65percentforTEKSELAR).

• TEKSELARhasmorestatementsratedatlevel3–strategicthinkingandlevel 4–extendedthinkingthandoACTorADP.

Thestudyhastwokeylimitations.First,thedefinitionofpartialalignmentwasverybroad,coveringcasesofjustoneelementofanACTorADPstatementthatwasaddressed

byaTEKSELARstatementorstatementsaswellascaseswhenallbutoneofmul-tipleelementsofanACTorADPstatementwereaddressed.Second,thedeterminationofcontentalignmentandtheevaluationofstandardsincludedjusttwodimensionsforevaluatingalignment(contentandcogni-tivecomplexity).Otherdimensions,suchasbreadthandspecificity,mightprovideadditionalcontentdetailthatstatestandardswritingteamsorassessmentwritingteamscouldfinduseful.

August2010

iv Table of conTenTS

TAblE of conTEnTs

Whythisstudy? 1Collegereadinessstandards 1Alignmentresearch 2Regionalimportance 2Thecurrentstudy 3

Descriptionofbenchmarkandcomparisonsetsofcollegereadinessstandards 4Descriptionofstandardssets 4Descriptionofcognitivecomplexityframework 6

Findings 6Contentalignment 6Cognitivecomplexity 11

Discussionandconclusions 12

Studylimitationsandsuggestionsforfurtherresearch 12

AppendixA Methodology 14

AppendixB Reviewerqualificationsandrolesandinterraterreliabilityfindings 18

AppendixC Examplesoffullyandpartiallyalignedstatements 22

AppendixD Contentalignmentfindingsbystrand 24

AppendixE Nonalignedstandardsstatements 26

AppendixF Otherstandards-to-standardsalignmentstudyfindings 40

AppendixG Webb’scognitivecomplexityleveldescriptionsandexamplestatements 42

AppendixH Cognitivecomplexitybystrand 46

AppendixI Cognitivecomplexitycomparisonforfullyandpartiallyalignedstatements 48

Notes 54

References 56

Boxes

1 Keydefinitions 3

2 Studymethodologyandratingscaleforexaminingcontentalignmentandcognitivecomplexity 7

Figures

1 Roleofcontentstandardsintheeducationsystem 1

2 PercentageofACTstatementsalignedwithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstatementsateachlevelofcontentalignment,2009 9

3 PercentageofACTstatementsalignedwithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstatementsateachlevelofcontentalignment,byACTstrand,2009 9

Table of conTenTS v

4 PercentageofAmericanDiplomaProjectstatementsalignedwithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstatementsateachlevelofcontentalignment,2009 9

5 PercentageofAmericanDiplomaProjectstatementsalignedwithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstatementsateachlevelofcontentalignment,byADPstrand,2009 10

6 PercentageofACTandAmericanDiplomaProjectstatementsthatalignfullyorpartiallywithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstatements,2009 11

7 DistributionofcognitivecomplexityratingsacrossthefourlevelsoftheWebbdepthofknowledgescale,bystandardsset(percent),2009 11

A1 PairwisecomparisonmethodologyusingACTandAmericanDiplomaProjectstandardssetsasbenchmarksforalignmentwiththeTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsset,2009 14

A2 Exampleofthestructureofthefullalignmenttable,2009 15

A3 Exampleofthestructureofthecognitivecomplexityratingtable,2009 16

B1 Interraterreliabilityinthecurrentstudyandotherresearch,2009(percentagreement) 21

B2 Interraterreliabilityinthecurrentstudycomparedtootherresearch,2009(intraclasscorrelation) 21

F1 Alignmentstudyfindingsorderedbypercentageoffullyalignedstandardsstatements,2009 40

F2 Alignmentstudyfindingsorderedbypercentageoffullyandpartiallyalignedstandardsstatements,2009 41

F3 Alignmentstudyfindingsorderedbypercentageofstandardsstatementsratedatthecombinedhighestlevels

H2 PercentageofAmericaDiplomaProjectstandardsstatementsateachlevelofcognitivecomplexitybystrand,

H3 PercentageofTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards

I1 OverallcognitivecomplexitycomparisonfindingsforACT–TexasEssentialKnowledgeandSkillsforEnglish

I2 OverallcognitivecomplexitycomparisonfindingsforACT–TexasEssentialKnowledgeandSkillsforEnglish

I3 OverallcognitivecomplexitycomparisonfindingsforAmericanDiplomaProject–TexasEssentialKnowledge

I4 OverallcognitivecomplexitycomparisonfindingsforAmericanDiplomaProject–TexasEssentialKnowledge

ofcognitivecomplexity(3and4)ontheWebbdepthofknowledgescale,2009 41

H1 PercentageofACTstandardsstatementsateachlevelofcognitivecomplexity,bystrand,2009 46

2009 47

statementsateachlevelofcognitivecomplexitybystrand,2009 47

languageartsandreadingstandardsfullyalignedstatements,2009 48

languageartsandreadingstandardspartiallyalignedstatements,2009 50

andSkillsforEnglishlanguageartsandreadingstandardsfullyalignedstatements,2009 52

andSkillsforEnglishlanguageartsandreadingstandardspartiallyalignedstatements,2009 53

Tables

1 OverviewofthetwobenchmarkcollegereadinessstandardssetsandtheTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardssetforgrades9–12,2009 4

vi Table of conTenTS

B1 Contentalignmentinterrateragreementpriortoconsensusmeeting,2009 19

B2 Cognitivecomplexityinterrateragreementpriortoconsensusmeeting,2009 19

C1 Examplesoffullyalignedstandardsstatements,2009 22

C2 Examplesofpartiallyalignedstandardsstatements,2009 23

D1 AlignmentofACTstatementswithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsstatementsateachlevelofcontentalignment,byACTstrandandsubstrand,2009 24

D2 AlignmentofAmericanDiplomaProjectstatementswithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsstatementsateachlevelofcontentalignment,byADPstrand,2009 25

E1 ACTstatementsthatdidnotalignwithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards,byACTstrand,2009 26

E2 AmericanDiplomaProjectstatementsthatdidnotalignwithTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards,byADPstrand,2009 27

E3 TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsstatementsthatdidnotaligntoACTstatements,byTEKSELARstrand,2009 28

E4 TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsstatementsthatdidnotaligntoAmericaDiplomaProjectstatements,byTEKSELARstrand,2009 38

G1 Examplesofstandardsstatementsratedatcognitivecomplexitylevel1,2009 42




I1 DetailedcognitivecomplexitycomparisonfindingsforACT–TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsfullyalignedstatements,2009 49

I2 DetailedcognitivecomplexitycomparisonfindingsforACT–TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardspartiallyalignedstatements,2009 51

I3 DetailedcognitivecomplexitycomparisonfindingsforAmericanDiplomaProject–TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsfullyalignedstatements,2009 52

I4 DetailedcognitivecomplexitycomparisonfindingsforAmericanDiplomaProject–TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardspartiallyalignedstatements,2009 53

1 Why ThiS STudy?

This study compares alignment of the AcT and the American Diploma Project (ADP) national college readiness standards sets with the Texas Essential Knowledge and skills for English language arts and reading (TEKs ElAR) standards for grades 9–12 and analyzes their cognitive complexity. It finds that a majority of the content in the AcT and ADP sets is addressed to some extent by the TEKs ElAR standards and that the TEKs ElAR standards demand higher levels of cognitive complexity than do the other two standards sets.

Why ThIs sTuDy?

The1983publicationofA Nation at Riskcalledfor“schools,colleges,anduniversities[to]adoptmorerigorousandmeasurablestandards,andhigherexpectationsforacademicperformance”(NationalCommissiononExcellenceinEducation1983,ascitedinU.S.DepartmentofEducation2008,p.5).Thispublicationwaspartofanationalmove-menttodevelopchallengingcontentstandardsforinstructionforallstudents,alsoknownasstandards-basedreform.1

A2008RANDCorporationreviewnotesthatcarefullydefiningtheknowledgeandskillsthatstudentsshouldhaveatvariousgradelevelsisthefirstcriticalaspectofstandards-basedre-form(Hamilton,Stecher,andYuan2008).2Thesedefinedcontentstandardsthenbecomethebasisforaligningotherkeyelementsoftheeducationsystem(figure1).

Collegereadinessstandards

WhiletheadoptionofK–12standardsandalignmentofthekeyelementsoftheeducationsystemwasinitiallyvoluntary,federallegisla-tioneventuallymadethemmandatory,undertheImprovingAmerica’sSchoolsActof1994(1995)andtheNoChildLeftBehindActof2001(2002).All50stateshavenowadoptedK–12contentstandards.

Researchersandpolicymakershavebeguntofocusonthelackofverticalalignment(alogi-calprogressionforteachingcontentinasubject

Standards

Curriculum Assessment

figure 1

Role of content standards in the education system

Source:Webb2005.

2 are TexaS’ engliSh language arTS and reading STandardS college ready?

areaacrossgrades)betweenK–12andpostsecondarycurriculaandstandards.3Ahighpercentageoffirst-yearcollegestudentshavefailedtoacquiretheknowledgeandskillsrequiredforsuccessinentry-levelcollegecourses.Terry(2007)estimatesthat38percentofTexasstudentsenrolledattwo-yearpublicinstitutionsand24percentenrolledatfour-yearpublicinstitutionsinfall2006wererequiredtotakeremedialcourses.Nationally,dependingonthetypeofinstitution,15–29percentofstudentsenteringpost-secondaryeducationinfall2003

self-reportedtakingremedialcourses(ProvasnikandPlanty2008,p.11).4

SeveralrecentinitiativeshavesoughttodefinecollegeandcareerreadinessinthesamemannerasK–12education.Theresulthasbeenthedevelop-mentofseveralsetsofnationalcollegereadinessstandardsthatsummarizetheknowledgeandskillsrequiredbystudentstosucceedinentry-levelcollegecourses.Oneoftheprioritiesofthecurrentfederalinitiative,theAmericanReinvestmentandRecoveryAct(2009),isdevelopingrigorouscollegeandcareerreadinessstandards.5Inadditiontothisinitiative,theNationalGovernorsAssociationandtheCouncilofChiefStateSchoolOfficershaveintroducedtheCommonCoreStateStandardsInitiativetoassiststatesinestablishingsuchstan-dards.TheinitiativefocusesonthedevelopmentofasinglenationalsetofK–12curriculumstandardsverticallyalignedtocollegereadinessstandards(NationalGovernorsAssociation2009).Thisinitia-tiveisbeingassistedbynationalleadersincollegereadinessstandards:Achieve,Inc;ACT,Inc.;andtheCollegeBoard.BecauseTexasadoptedstatecol-legereadinessstandards(TexasCollegeandCareerReadinessStandards,orTCRS)in2008andhasaligneditsK–12standardsinEnglishlanguageartsandmathematicstotheTCRS,itisnotparticipat-ingintheCommonCoreStateStandardsInitiativeconsortiumofstatesatthistime.

because both recent

Texas state legislation

and the federal American

Reinvestment and

Recovery Act initiative

focus on the need

for rigorous college

readiness standards,

it is important for

Texas policymakers to

understand how the

newly adopted state

standards compare

with national college

readiness standards sets

Alignmentresearch

Whilealignmentresearchhasfocusedoncompar-ingtestitemswithcontentstandards,comparingastate’sstandardswithotherstandardssetshasbeenanimportantcomponentofstandardsrevi-sions.Themethodologiesdevelopedtoevaluatethealignmentofthekeyelementsofstandardssetsusedifferentdimensionsandcriteria,buttherearesimilaritiesinapproach(NäsströmandHenriksson2008;Porter2002;Porteret al.2007;Rothmanet al.2002;Webb1999,2002,2005).Whileresearchershavedefinedvariousdimen-sionsbywhichstandardscanbedescribedandaligned,suchasbreadth,depth,andspecificity(NäsströmandHenriksson2008;LaMarca2001;Rothman2004),LaMarca(2001,para.4)hascon-cludedthatthedimensionsofcontentknowledgeandcognitivecomplexityarethe“twooverarchingdimensions”ofalignment.Arecentstandards-to-standardsalignmentstudyexaminingfoursetsofcollegereadinessstandardsalongthesetwodimensionsfoundlevelsofalignmentof8–31percentand34–77percent,dependingonwhetherfullorpartialalignment(orbothtogether;seebox 1andappendixA)wasconsidered(Rolfhuset al.2010).6

Regionalimportance

BecausebothrecentTexasstatelegislationandthefederalAmericanReinvestmentandRecoveryActinitiativefocusontheneedforrigorouscol-legereadinessstandards,itisimportantforTexaspolicymakerstounderstandhowthenewlyad-optedstatestandardscomparewithnationalcol-legereadinessstandardssets.In2006theTexasLegislaturemandateddevelopmentofcollegereadinessstandardsandalignmentofthestate’sK–12curriculumtothosestandards(HouseBill1;TexasLegislature2006).Inresponse,theTexasHigherEducationCoordinatingBoard(2008)developedandadoptedtheTCRS.Inaddition,theTexasStateBoardofEducationadoptedare-visedsetofTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards(TEKSELAR)thatwastobeverticallyaligned

3 Why ThiS STudy?

box 1

Keydefinitions

Content.Theknowledgeandskillsexplicitlystatedorstronglyimpliedinastandardsstatement(forex-ample,“demonstrateknowledgeof18thand19thcenturyfoundationalworksofAmericanliterature”).Eachstandardssetcategorizesandlabelscontentdifferently;thisstudyusesthetermsstrand, substrand,andstandards statements.

Content alignment.Theidentificationofcontentinastatement(orstate-ments)fromonesetofstandards(acomparisonsetofstandards)asthesameascontentinastatementinanothersetorsetsofstandards(thebenchmarksets).

Cognitive complexity ordepth.Thecognitivedemandortypeofthink-ingrequiredtodemonstratetheknowledgeandskillsrepresentedbyastandardsstatement(forexample,thelevelofabstraction,numberofsteps,ortypeofreasoning;Rothman2004;Webb1997,1999,2002).Know-ingthelevelofcognitivecomplexityfacilitatesthedevelopment,attheappropriatelevelofdifficultyorrigor,ofstateassessmentitemsthatmeasurestudentperformancebasedontheexpectationsrepresentedbythestandards(NäsströmandHenriksson2008).UnderstandinghowthecognitivecomplexityoftheTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardscompareswiththeexpectationsofACTandAmerican

DiplomaProject(ADP)standardsisasimportantasunderstandingcontentalignment(NäsströmandHenriksson2008).

Strands. Clustersofcontent-relatedstatementsintheEnglishlanguageartsdomainofeachsetofstandards.Forexample,theADPcommunica-tionstrandcontainstheindividualstatements,“Giveandfollowspokeninstructionstoperformspecifictasks,toanswerquestionsortosolveproblems”(B1)and“Summarizeinformationpresentedorallybyoth-ers”(B2;Achieve,Inc.2004).Strandnamesvaryacrossthestandards,andtheorganizationofstatementsintostrandsandsubstrandscanhelptoidentifyareasofcontentemphasis.

withtheTCRS(TexasEducationAgency2008).HouseBill3directsthestatetodevelopcollegereadinessperformancestandardsanddirectsthestateeducationcommissionertoperiodicallyraisethestatestandardsforstudentachievementsothatnolaterthanthe2019/20schoolyearTexaswillrankamongthetop10statesnation-allyinstudentperformance(TexasLegislature2009).

Thecurrentstudy

Thisstudyexaminesthealignmentofthe2008TEKSELARstandardstotwosetsofnationallyusedEnglishlanguageartscollegereadinessstan-dards:ACTCollegeReadinessStandards(ACT,Inc.2007)andtheAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBench-marks(Achieve,Inc.2004).7ACTwasselectedbe-causeitistheonlycollegereadinessstandardssetwithanexplicitempiricalbasis;ACTscorerangesaremappedtospecificstandardsstatements,andACTscoresarelinkedtogradesinthefirstyearofcollege(ACT,Inc.2007).ADPstandards

wereselectedbecauseAchievehasworkedwith35states,includingTexas,ontheirstandards.Neitheroftheothertwonationallyusedstan-dardssets,theCollegeBoard(TheCollegeBoard2006)orStandardsforSuccess(Conley2003),haspubliclydocumentedanexplicitempiricallinkwithstudentperformanceordirectinvolvementwithstatestandardsdevelopmentonthisscale.Thisreportisintendedtoinformdecisionmak-ersaboutthealignmentofthe2008TEKSELARstandardstonationalcollegereadinessstandardssetsandinformstateeffortstorevisethestan-dards,asexpressedinHouseBills1and3.Placingthefindingsinthecontextofotherstandards-to-standardsalignmentstudies,suchasRolfhuset al.(2010),willassistpolicymakersinusingthefindings.

Twodimensionsofalignmentwereselectedforevaluation:generalcontentalignmentandcognitivecomplexity(seebox1fordefinitions).Otheralignmentcriteriawereexcludedbecausestakeholdersdidnotrequestinformationonthem.


Twoprimaryresearchquestionswereaddressedinthisreport:

• WhatpercentageofcontentstatementsintheACTandAmericanDiplomaProject(ADP)collegereadinessstandardssets(thebench-marksets)alignfullyorpartiallywithcontentstatementsinthe2008TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreading(TEKSELAR)grade9–12standardsset(thecomparisonset)?

• Foreachofthesestandardssets,whatisthedistributionofcontentstatementsacrossthefourlevelsofacognitivecomplexity(cognitivedemand)scale?

DEscRIPTIon of bEnchmARK AnD comPARIson sETs of collEgE READInEss sTAnDARDs

ThissectiondetailsthetwobenchmarksetsofEnglishlanguageartscollegereadinessstandards(ACTandADP)andtheTEKSELARstandardssetexaminedinthisstudyanddescribesthegoalsofthedevelopingorganizations,intendeduses,de-velopmentprocess,andstrandstructure.Table 1providesabriefoverview.

Descriptionofstandardssets

ACT college readiness standards.TheACTcollegereadinessstandardsareintendedtorepresent

Table 1

overview of the two benchmark college readiness standards sets and the Texas Essential Knowledge and skills for English language arts and reading standards set for grades 9–12, 2009

category year

published Publisher organization type

method/process to derive standards statements

english language arts strands

number of english

language arts standards

statementsa

acT 2007 acT, inc. Test publisher national curriculum Survey to inform test development— standards derived from test content

english reading Writing

191

american diploma Project (adP)

2004 achieve, inc.

education reform organization to promote postsecondary readiness

committees of postsecondary academic leaders and business leaders

communication informational text language literature logic media research Writing

62

Texas essential Knowledge and Skills for english language arts and reading standards

2008 Texas education agency

State organization oversees activities related to public schools in Texas

State board of education develops curriculum standards with input from teacher work groups and content experts

listening and speaking

oral and written conventions

reading research Writing

278

a.Themainreasonforthedifferentnumberofstandardsstatementsisthatthestandardsarewrittenatdifferentlevelsofspecificity,reflectingtheirgoalsandintendeduses.Whilespecificitywasnotevaluatedinthisstudy,differentlevelsofspecificitywouldnothaveamajorimpactoncontentmatch.Thecontentexpertswhoconductedthealignmentarefamiliarwiththeintentofthestatements,whichenabledthemtoalignstandardsstatementswrittenatdifferentlevelsofspecificity.Inaddition,therearemanymoreTEKSELARstatementsthanADPorACTstatementsbecauseTEKSELARstandardscovergrades9–12,notjustonelevel,asthetwootherstandardssetsdo.

Source:ACT,Inc.2007;Achieve,Inc.2004;TexasEducationAgency2008.

deScriPTion of benchmarK and comPariSon SeTS of college readineSS STandardS 5

theknowledgeandskillsandtypeofthinkingrequiredforstudentstosucceedinentry-levelcollegecourses(ACT,Inc.2007).Informationonstudentperformancerelativetothestandardscanassiststudents,parents,andteachersinidentifyingskilldeficitsforremediation.

TheACTcollegereadinessstandardsweredevelopedempiricallythroughamultistageprocessbyACT,Inc.staffandreviewedbyexperts(whomACTidentifiesas“nationallyrecognized”)fromhighschoolandpostsecondaryEnglishandreadingeducationdepartments(ACT,Inc.2007).First,ACTdevelopedapoolofassessmentitemstakenfromtheresultsoftheACTNationalCurriculumSurveys(K–12).Then,basedon40yearsofresearchonACTstudentassessmentdata,ACTstaffidentifiedscorerangesfromthedistributionofstudentscoresonACT’sEducationalandPlanningAssessmentSystemthatbestdifferentiatedstudents’levelsofachievementinfourcontentdomains:English,mathematics,science,andreading.ThecollegereadinessstandardsstatementsdrewonACTstaffexpertanalysisoftheknowledge,skills,andtypeofthinkingneededtorespondcorrectlytoassessmentitems.Finally,theindependentreviewersvalidatedthatthestandardsaccuratelypredictedstudentperformanceinthefirstyearofcollege.Becauseofthismethodofconstruction,ACTstandardsareempiricallylinkedtoassessmentscoresandaretheonlysetofcollegereadinessstandardslinkedtostudentachievement.TheACTEnglishlanguageartscollegereadinessstandardsaredividedintothreestrands(English,reading,andwriting)and16substrands(forexample,topicdevelopment,mainidea,organizingideas).Thestandardsstatementsareorganizedwithinthesesubstrands.

American Diploma Project college readiness standards.TheADPisanetworkofstatepolicymakersandotherleadersworkingtoalignandraisestatestandardsandassessmentstoalevelthatwillpreparestudentsforsuccessinpostsecondaryeducationandtheworkplace(Achieve,Inc.2004).Asofthiswriting,35stateshadjoinedtheADPNetwork(Achieve,Inc.2009a).

TheADPcollegereadi-nessstandardswerede-velopedbyAchieve,Inc.(2004),throughatwo-yearprocessthatsoughtinputfrombusinesslead-ersandpostsecondaryeducatorsfromfivestates(Indiana,Kentucky,Massachusetts,Nevada,andTexas).Curriculumexpertsuseddataonedu-cationpatternsassociatedwitheducationandcareeradvancementalongwithotherassessments(suchashighschoolexitexamsandpostsecondaryplace-menttests)toidentifytheessentialknowledgeandskillsstudentsneedforsuccessinpostsecondaryeducation.Panelsofcontentareaexperts,postsecondaryschoolfaculty,andNationalAllianceofBusinessindustryrepresentativesreviewedtheworkingdocuments.CollegereadinessstandardsforEnglishandmathematicsemergedfromthisresearchandareintendedtoserveasabasisforstateassessments.TheADPEnglishlanguageartscollegereadinessstandardsaredividedintoeightstrands:communication,informationaltext,language,literature,logic,media,research,andwriting.

Texas Essential Knowledge and Skills for English language arts and reading standards.TheTEKSELARstandardsareasetofK–12standardsthatdefinetheknowledge,skills,andlevelofcognitivecomplexityrequiredasstudentsprogressfromgradetogradeinTexaspublicschools.Thissetofverticallyalignedstandardsformsthebasisofcurriculaandassessmentinthestate.TheTEKSperiodicallyundergorevision;newstandardsfortheELARcontentdomainwereapprovedandadoptedin2008,replacingthe1997standards(TexasEducationAgency1997,2008).TheTEKSELARstandardsweredevelopedoverthreeyearswithinputfromtheStateBoardofEducation,teacher

because of the method

of construction, AcT

standards are empirically

linked to assessment

scores and are the only

set of college readiness

standards linked to

student achievement.

for the ADP standards,

curriculum experts

used data on education

patterns associated

with education and

career advancement to

identify the knowledge

and skills students

need for success in

postsecondary education


The majority of

statements in all

three AcT strands are

partially aligned with

TEKs ElAR. The English

strand has the largest

proportion of partially

aligned statements,

at 89 percent, while

the writing strand

has the smallest

share, at 55 percent

workgroups,andcontentexperts.End-of-courseexamsalignedtothecontentstandardsforgrades9,10,and11willbeadministeredbeginninginthespringof2011(TexasEducationAgency2009).Thecurrentstudyusedthegrade9–12Englishlanguageartsstan-dardsforthefourEnglishcoursesrequiredforgraduation(EnglishI–IV)forcomparison.8TheTEKSELARstandardsaredividedintofivestrands:listeningandspeak-ing,oralandwrittenconventions,

reading,research,andwriting.

Descriptionofcognitivecomplexityframework

Standardsstatementscommunicaterelevantcontentinformationandreflectthelevelofcogni-tivecomplexity(typeofthinking)demandedofstudentsthroughtheuseofspecificlanguageandkeyterms(Rothman2004;Webb1997,1999,2002).Thecognitivecomplexityrepresentedinastatementcaninfluencethedevelopmentofinstructionalmaterialsandassessments(Webb1997,pp.15–16).Forexample,statementsthatrequirestudentsonlyto“identify”or“recognize”certaincontentrepresentlowerlevelsofcognitivecomplexitythanstandardsthatrequirestudentsto“reasonwith,”“synthesize,”or“produce”complexmaterials.Thus,increatingormodifyingassess-mentitemsforcollegereadinesspurposes,itisimportanttoattendnotonlytocontent,butalsotothelevelofcognitivecomplexitythatstudentsareexpectedtoexpressthroughtheirknowledgeandskills(Webb1997,pp.15–16).

Theliteraturedescribesavarietyofmethodsforexaminingthecognitivecomplexityofstandards(forexample,Achieve,Inc.n.d.;Blank2002;Cook2005;Webb2002;seealsothereviewbyNäsströmandHenriksson2008).ThemethodologyadoptedinthisstudywasderivedfromWebb(1999)andWixsonet al.(2002).Webb(1999)wasthefirsttoexaminestandardsstatements(infourstates)usingthefour-leveldepthofknowledge(DoK)

scale:recall,skill/concept,strategicthinking,andextendedthinking.Thestudydetailedseveralcrite-riaforevaluatingalignmentofstatestandardsandassessmentsinmathematicsandscience.Webbemphasizedtheimportanceofevaluatingdepthofknowledgetoensurethatassessmentsmeasurestudentperformanceatthesamedepthasexpectedintheclassroom.WebbfoundthatasubstantialpercentageofitemsonstateassessmentswereratedatlowerDoKlevelsthancorrespondingobjectivesinthestatestandards.InasubsequentstudyratingtheobjectivesforgradeK–5readingstandardsoffourstatesusingtheDoKscale,Wixonet al.(2002)showedthatstatestandardssetscanbedifferenti-atedbycognitivecomplexityusingtheDoKscale.Forexample,inonestate80percentofstatementswereratedlevel 19(recall)whereasinanotherstate,just19percentwere.Otherdetailsofthestudymethodologyareinbox2andappendix A.

fInDIngs

Thissectiondescribesthecontentalignmentandcognitivecomplexityfindings.ThecompletecontentalignmentandcognitivecomplexitytablesareavailablefromRegionalEducationalLaboratorySouthwest.Thedegreetowhichthetwoindepen-dentreviewersagreedbeforetheymettodeterminethefinalconsensusratingisreportedinappendixB.

Contentalignment

Summaryfiguresonthepercentageofstatementsateachlevelofcontentalignmentarepresentedbelow.ThenumberofstatementsateachlevelofcontentalignmentarereportedinappendixD.ACT,ADP,andTEKSELARcontentstatementsthatdidnotalignarereportedinappendixE.

ACT and Texas Essential Knowledge and Skills for English language arts and reading content alignment.ThedegreeofalignmentbetweencontentintheACTstatementsandintheTEKSELARstatementsareshowninfigure2.Inmanycases,asingleACTstatementalignedwithmorethanoneTEKSELARstatement.

7 findingS

box 2

Studymethodologyandratingscaleforexaminingcontentalignmentandcognitivecomplexity

ThisstandardsalignmentstudywasconductedduringJune–August2009.

Methodology for aligning content. ThecontentalignmentmethodologyusedinapreviousseriesofRegionalEducationalLaboratorySouthweststudieswasadaptedforthisstudy(ShapleyandBrite2008a–e;Timmset al.2007a–e;Rolfhuset al.2010).Thesamethree-levelcontentalign-mentratingscaleandprocessforreconcilingindependentreviewerratingswereusedtocomputethepercentageofbenchmarkstandardsstatements(ACTandtheAmericanDiplomaProject,orADP)thatarefully,partially,ornotalignedwiththecomparisonstatementsintheTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008).The191ACTstatementsandthe62ADPstatementsweredesignatedinturnasthebenchmarksetandalignedwiththe278TEKSELARcompari-sonsetstatements.1Inmanycases,thecontentinasinglebenchmarkstatementalignedwiththecontentinmultipleTEKSELARstatementsorthecontentinasingleTEKSELARstatementalignedtothecontentinmultiplebenchmarkstatements.Separatecontentalignmenttableswerecreatedtoconductthesepairwisecomparisons,withthefirstcolumnpopulatedbyeithertheACTortheADPstatements.

TwoindependentreviewersusedthefollowingscaletoratethelevelofcontentalignmentbetweentheACTortheADPsetandtheTEKSELAR:

• Fully aligned.Allthecontentinabenchmark(ACTorADP)state-mentalignswithcontentinoneormorestatementsinthecompari-son(TEKSELAR)standardsset.

• Partially aligned.Someofthecontent(from1–99percent)inthebenchmark(ACTorADP)statementalignswithsomeofthecontentinthecomparison(TEKSELAR)standardsset.

• Not aligned.Noneofthecontentinthebenchmark(ACTorADP)statementalignswithanyofthecontentinthecomparison(TEKSELAR)standardsset.

Ifsimilarcontentwasfound,eachre-viewerindependentlyratedthelevelofalignmentasfullorpartial.Theprocesscouldresultinaone-to-onealignment(onebenchmarkstatementalignswithoneTEKSELARstate-ment)oraone-to-manyalignment(onebenchmarkstatementalignswithmultipleTEKSELARstate-ments).Contentalignmentwasevalu-atedindependentlyofthecognitivecomplexityratings.Finalalignmentsandratingsweredeterminedduringconsensusmeetingswiththeseniorreviewer.Anexampleofhoweachcontentalignmenttablewasstruc-turedandpopulatedisprovidedinfigureA1inappendixA.

Methodology for rating cognitive complexity.Cognitivecomplexitywas

assessedbycomparingthedistribu-tionofstandardsstatementsfromeachsetofstandardsacrossfourlevelsofcognitivecomplexity.Thecognitivecomplexityratingswerecompletedbeforethecontentalign-ment.Therewasnobenchmarkforthecognitivecomplexityrating.

Cognitivecomplexitywasassessedbytworeviewerswhoworkedindepen-dentlyusingWebb’s(2002)depthofknowledge(DoK)scaletoratethecognitivecomplexityofeachstate-ment(seeappendixGfordetails):

• Level 1–recall.Requiresstudentstousesimpleskillsorabilitiestoretrieveorrecitefacts.

• Level 2–skill/concept.Requiresalevelofcomprehensionandsubsequentprocessingacrossportionsoftexttomakeinfer-encesbeyondsimplerecallorrecitationofstatedfacts.

• Level 3–strategic thinking. Focusesonreasoning,planningskills,makingmorecomplexinferences,andapplyingideasfromthetext;studentsmaybeencouragedtoexplain,general-ize,orconnectideas.

• Level 4–extended thinking.Re-quiresinvestigationandhigherorderthinkingskillstoprocessmultiplesolutionstoagivenproblem.

Athree-columncognitivecomplex-ityratingtablewascreatedforeachstandardsset,witheachstandardsstatementinthefirstcolumn,the

(conTinued)


box 2 (conTinued)

Studymethodologyandratingscaleforexaminingcontentalignmentandcognitivecomplexity

cognitivecomplexitylevelinthesec samplesandratingsandprac contentalignmentlevelratingondcolumn,andreviewercomments ticedconductingalignmentand totheACTstatementbasedoninthethirdcolumn(foranexample, ratingactivitiesandreaching thecumulativecontentofalltheseefigureA2inappendix A).The consensus. alignedTEKSELARstatementstworeviewers’independentcognitive (fully,partially,ornotaligned).complexityratingswerediscussed • Step 3—rating cognitive com Consensusmeetingswereheldduringconsensusmeetingsheld plexity levels.Reviewersinde aftercompletionofeachACTunderthesupervisionofasenior pendentlyratedthecognitive strand.Thecyclewasrepeatedreviewer,andthefinalratingwas complexitylevelofeachTEKS untilallpossibleACTandTEKSdeterminedatthattime. ELARstatementusingtheWebb ELARstatementswerecompared

DoKscaletoensurefamiliarity andthecontentalignmentlevelsAlignment and rating processes. The withthecontents.TheRolfhus wererated.alignment(content)andrating(level et al.(2010)cognitivecomplexofcontentalignmentandcognitive ityratingsfortheADPandACT • Step 6—comparing and aligning complexity)processesconsistedofsix wereusedforthisstudysincethe ADP–TEKS ELAR content.Thesteps: methodologyandreviewteam sameprocessasinstep5wasfol

werethesame. lowedfortheADP–TEKSELAR• Step 1—selecting reviewers. contentalignment.

Thereviewerswhohadpartici • Step 4—achieving consensus on patedintherecentlycompleted cognitive complexity levels.After Forfurtherdetailsonmethodology,Rolfhuset al.(2010)study completingindividualcogni seeappendixesAandB.wereselectedasreviewersfor tivecomplexityratingsforallthecurrentstudy.Theywere TEKSELARstatements,thetwo Limitations of the study. ThemainfamiliarwiththeACTandADP independentreviewersmetwith limitationofthestudyisthedefistandardsstatements,therating theseniorreviewertocompare nitionofpartialalignment.Thescales,andtheindependent ratingsandachieveconsensus definitionisbroad,encompassingratingandconsensusprocess whereratingsdiffered. alignmentbetweenstatementswith(formoreinformationabout alittlesharedcontentandthosewithreviewerqualifications,seeap • Step 5—comparing and aligning alotofsharedcontent.ModificationspendixB). ACT–TEKS ELAR content.Using tothenumberanddefinitionoflevels

theACT–TEKSELARcontent ofalignmentmightresultindiffer• Step 2—training reviewers. alignmenttableandbegin entlevelsofconsensusacrossthe

Reviewersexaminedthestruc ningwiththefirstACTstate standardssets.ture,organization,andcontent mentinthefirstACTstrand,ofeachstandardssetbefore eachreviewerindependently Notetraining.Duringathree-hour andsystematicallysearched 1. Thisreportuses“alignedwith”torefer

trainingsession,thereviewers allTEKSELARstatementsfor totheextentofthecontentalignmentbetweenthebenchmarkstandardssetswereretrainedonthethree-level anycontainingallorpartof(ACTandADP)andtheTEKSELAR

contentalignmentscaleandthe thesamecontent.Onceallfully standardssetand“alignedto”toreferfour-levelcognitivecomplexity andpartiallyalignedTEKS tohowtheTEKSELARstandardsmapscale(Webb2002),andtheyre ELARstatementswereidenti ontothebenchmarksetsofstandardsviewedanddiscussedalignment fied,thereviewerassigneda (ACTorADP).

-

-

-

-

-

-

-

-

-

--

-

-

-

-

9

figure 2

Percentage of AcT statements aligned with Texas Essential Knowledge and skills for English language arts and reading statements at each level of content alignment, 2009

Partially aligned 75%

Fully aligned 14%

Not aligned 11%

Source: SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinACT,Inc.(2007)andTexasEducationAgency(2008).

ACT’sEnglishlanguageartsstandardsareorganizedintothreestrands(English,reading,andwriting)and191standardsstatements.ThepercentageofACTstatementsthatalignwithTEKSELARstatementsvariesacrossthethreestrands(figure3).ThemajorityofstatementsinallthreeACTstrandsarepartiallyalignedwithTEKSELAR.TheEnglishstrandhasthelargestproportionofpartiallyalignedstatements,at89percent(63of71),whilethewritingstrandhasthesmallestshare,at55percent(34of62).Thereadingstrandhasthesmallestshareoffullyalignedstate-ments(5percent),whilethereading(9of58)andwriting(10of62)strandshavethelargestsharesofstatementsthatarenotaligned,at16percent.

American Diploma Project and Texas Essential Knowledge and Skills for English language arts and reading content alignment.Thedegreeofalign-mentbetweenthecontentintheADPstatementsandintheTEKSELARstatementsisshowninfigure4.Intotal,94percentofADPstatements(58of62)alignfullyorpartiallywithcontentinoneormoreTEKSELARstatements:48percent(30

findingS

figure 3

Percentage of AcT statements aligned with Texas Essential Knowledge and skills for English language arts and reading statements at each level of content alignment, by AcT strand, 2009

Percent 100 Fully aligned Partially aligned Not aligned

89

79

75

55

50

29

25 16 16

8 53

0 Writing (62) Reading (58)English (71)

ACT strands (number of statements)

Note: ThenumberofstatementsateachlevelofalignmentbystrandisshownintableD1inappendixD.


figure 4

Percentage of American Diploma Project statements aligned with Texas Essential Knowledge and skills for English language arts and reading statements at each level of content alignment, 2009

Partially aligned 45%

Fully aligned 48%

Not aligned

6%

Note: Percentagesdonotsumto100becauseofrounding.

Source:SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinAchieve,Inc.(2004)andTexasEducationAgency(2008).


figure 5

Percentage of American Diploma Project statements aligned with Texas Essential Knowledge and skills for English language arts and reading statements at each level of content alignment, by ADP strand, 2009

67

Percent100 Fullyaligned Partiallyaligned Notaligned

7575

6057 57 56 55

50 5050 4543

40

3329

252225

1411 11

0 0 0 0 00Language(7) Communication(7) Writing(10) Research(5) Logic(9) Informationaltext(11) Media(4) Literature(9)

ADPstrands(numberofstatements)

Note:ThenumberofstatementsateachlevelofalignmentbystrandisshownintableD2inappendixD.


of62)alignfullyand45percentalignpartially;6percentofADPstatements(4of62)donotalignwithTEKSELARstatements.

TheADPEnglishlanguageartscollegereadinessstandardsareorganizedintoeightstrandswith62standardsstatements.TheADPlitera-turestrandhasthehighestproportionofstatementsfullyalignedwithTEKSELARstatements(67 percent,6of9;figure5).Thelanguageandcommunicationstrandshavethenexthighestproportionsoffullyalignedstatements(57percent,4of7).Acrossallstrands,theproportionofADPstatementsthatalignfullywithTEKSELARstatementsrangesfrom0percent(media)to67 percent(literature).

Themediastrandhasthehighestproportion(75 percent,3of4)ofADPstatementsthatalignpartiallywithTEKSELARstatements,followedbytheresearch(60percent,3of5)andlogic(56percent,5of9)strands.Acrossallstrands,theshareofADPstatementsthatalignpartiallywithTEKSELARstatementsrangesfrom22percent(literature,2of9)to75percent(media,3of4).

-

InfouroftheeightADPstrands(language,writing,research,informationaltext),allthecontentalignseitherfullyorpartiallywithcontentinTEKSELAR;11–25percentofthestatementsintheotherfourADPstrands(communication,logic,media,literature)donotalignwithTEKSELARstatements.

Summary of content alignment findings.ThecontentoftheADPcollegereadinessstandardssetismorecloselyalignedwiththeTEKSELARgrades9–12standardssetthanisthecontentoftheACTcollegereadinessstandardsset(figure6).Forty-eightpercentoftheADPstandardsstatementsalignfullywiththeTEKSELARstandardsset.(Recallthat100percentofthecontentinastatementmustbealignedtoachievearatingoffullalignment.)Another45percentofADPstatementsalignpartiallywiththeTEKS ELARstandardsset.(Partialalignmentincludesalignmentbetweenstatementswithverylittlesharedcontentandalignmentbetweenstatementswithextensivesharedcontent.)SixpercentofADPstandardsstatementsarenotaligned.

-

-

-

-

-

--

figure 6

Percentage of AcT and American Diploma Project statements that align fully or partially with Texas Essential Knowledge and skills for English language arts and reading statements, 2009

Percent100 Fullyaligned Partiallyaligned Notaligned

7575

4850 45

2514

116

0ACT(191) ADP(62)

Benchmarkstandardssets(totalnumberofstatements)

Note:Percentagesmaynotsumto100becauseofrounding.

Source:SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinACT,Inc.(2007),Achieve,Inc.(2004),andTexasEducationAgency(2008).

Incontrast,14percentoftheACTstandardsstate-mentsalignfullywiththeTEKSELARstandardsstatements,75percentalignpartially,and11percentdonotalign.

Cognitivecomplexity

Forthesecondresearchquestiononthecognitivecomplexitylevelsofthethreestandardssets,10

therewasnobenchmark,andallstatementsfromeachsetwereratedregardlessofwhetherthestate-mentscontainedalignedcontent.ThedistributionofcognitivecomplexitylevelratingsacrossthefourlevelsoftheWebbDoKscaleisshownforeachofthestandardssetsinfigure7.

AllfourlevelsoftheWebbDoKscalearerepre-sentedineachofthestandardssets.Morethanhalfthestandardsstatementsineachsetwereratedlevel3–strategicthinking.ACThasthehigh-estshareofstatementsratedatlevel1–recall(18percent)andthelowestshareatlevel4–extendedthinking(1percent).ADPhasthehighestshare

findingS 11

figure 7

Distribution of cognitive complexity ratings across the four levels of the Webb depth of knowledge scale, by standards set (percent), 2009

Percent100 Level1–recall Level2–skill/concept

Level3–strategicthinking Level4–extendedthinking

7565

55 55

50

31 27

2518 16 1413

41 20

ACT ADP TEKSELAR

Standardsset

Note: Percentagesdonotsumto100becauseofrounding.

Source:ForACTandAmericanDiplomaProject,Rolfhuset al.(2010);forTEKSELAR,summaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinTexasEducationAgency(2008).

ofstatementsratedatlevel2–skill/concept(31percent)andthelowestshareatlevel1–recall(2percent).TEKSELARhasthehighestshareofbothlevel3–strategicthinking(65percent)andlevel4–extendedthinking(14percent)statements.

TheproportionsofTEKSELARstatementsratedlevel 1(4percent)andlevel4(14percent)aremoresimilartotheproportionsforADP’slevel1(2 percent)andlevel4(13percent)ratingsthanforACT’slevel1(18percent)andlevel4(1percent)ratings.However,theproportionofTEKSELARlevel2ratings(16percent)isclosertothatofACT(27percent)thantothatofADP(31per-cent).Becausetheproportionoflevel3ratingsisidenticalforACTandADP(55percentforeach),theTEKS ELARproportion(65percent)differsequallyfrombothofthem.ThedistributionofWebbDoKratingsforeachstandardssetisalsoreportedbystrandinappendixH.

Ofthethreestandardssetsexamined,TEKSELARhasthehighestshareofstatementsratedlevel


3–strategicthinkingandlevel4–extendedthink-ing,whetherthelevelsareconsideredindividuallyortogether(seefigure7).Together,79percentofTEKSELARstatementswereratedlevel3orlevel4forcognitivecomplexity,comparedwith56percentforACTand68percentforADP.Inthedistributionofcognitivecomplexityratingsacrossthefourlevels,theTEKSELARstandardssetismoresimilartoADPthanitistoACT.

DIscussIon AnD conclusIons

ThemajorityofthecontentintheACTandADPcollegereadinessstandardssetsisaddressedtosomedegreebythecontentintheTEKSELARstandards.FourteenpercentofACTstatementsfullyalignand75percentpartiallyalignwithoneormoreTEKSELARstatements(seefigure6).Forty-eightpercentofADPstatementsfullyalignand45percentpartiallyalignwithTEKSELARstatements.

Theseresultsaredifficulttointerpretinisolation,astherearenouniversallyacceptedcriteriafordetermininggoodorpoorlevelsofalignment.Aswithmostqualitativeresearch,thesejudgmentsmustbemaderelativetootherstudiesthatuseasimilarmethodology.Whilemostalignmentresearchfocusesonthealignmentofassessmentitemsandcontentstandards,Rolfhuset al.(2010)isanotherstandards-to-standardsalignmentstudythatusedthesamemethodologyandratingscalesasthecurrentstudyandcanbeusedforcomparison.

fourteen percent of AcT

statements fully align

and 75 percent partially

align with one or more

TEKs ElAR statements.

forty eight percent of

ADP statements fully

align and 45 percent

partially align with

TEKs ElAR statements

Rolfhuset al.focusedonthreepairwisecomparisonsofnationalEnglishlanguageartscollegereadinessstandards,usingADPasthebenchmark.Offivepairwisecomparisons(threeinRolfhuset al.comparisonsandtwointhecurrentstudy),theADP–TEKScomparisoninthecurrentstudyresultedinthehighestpercent-ageofbothfullyalignedcontent

andcombinedfullyandpartiallyalignedcontent.TheACT–TEKScomparisoninthecurrentstudyranksfourthinfullyalignedcontentandsecondincombinedfullyandpartiallyalignedcontent.ThesetwostudiesindicatethatTEKSELARalignsmorecloselytoADPthananyoftheotherthreenationalEnglishlanguageartscollegereadinessstandardsexamined.(SeeappendixFforanad-ditionaldiscussion.)

Inaggregate,TEKSELARstatementsdemandhigherlevelsofcognitivecomplexitythanbothbenchmarkcollegereadinessstandardssetsexaminedinthisstudy.Inthedistributionofratingsacrossthefourlevels,theADPandTEKSELARstandardssetsexhibitmoresimilarities.IntheRolfhuset al.(2010)study,whichalsoexaminedcognitivecomplexitylevelsusingthesameratingscalesandmethodologyasthecur-rentstudy,theaggregatecognitivecomplexitylevelsofTEKSELARstatementsarealsohigherthanthoseidentifiedfortwoadditionalnationalcollegereadinessstandardssets(CollegeBoard,StandardsforSuccess).ThissuggeststhatTEKSELARstatementsrequirehigheraggregatecognitivecomplexitylevelsthanthefoursetsofnationalcollegereadinesssetsforEnglishlanguagearts.

sTuDy lImITATIons AnD suggEsTIons foR fuRThER REsEARch

Thedefinitionofpartialalignmentinthecurrentstudywasverybroad.AlignmentwasconsideredpartialwhetherjustoneelementofanACTorADPstatementwasaddressedbyaTEKSELARstatementorstatementsorwhetherallbutoneofmultipleelementsofanACTorADPstatementwasaddressedbyTEKSELARstatements.Readersareencouragedtoexaminethecompletealign-menttables(availableonrequest)tofurtherex-plorethedegreeofpartialalignmentforparticularACTorADPstatements.

Asecondlimitationisthatjusttwodimensionswereusedforevaluatingstandardsalignment

13 STudy limiTaTionS and SuggeSTionS for furTher reSearch

(contentandcognitivecomplexity),whereassomeresearchershaveusedmore(Rothman2004).Forexample,Webb(2005)ratesfivedimensions,twoofthemequivalenttothetwoemployedinthisstudyandthreethatarenot(rangeofknowledgecorrespondence,balanceofrepresentation,sourceofchallenge).Useofotherdimensionssuchasthesemightprovideadditionalcontentdetailthatstatestandardswritingteamsorassessmentwrit-ingteamscouldfinduseful.

OneavenueforfurtherresearchisanalignmentstudyusingthefinalCommonCoreStateStan-dards,astheywerenotavailableatthetimethecurrentstudywasconducted.Giventhegrowingimportanceofstudentperformancecomparisonsacrossstates—usingtheNationalAssessmentofEducationalProgress,assessmentssuchasAchieve’smultistatealgebrainitiative(Achieve,

Inc.2009b),orassessmentsdevelopedfromtheCommonCoreStateStandards—itwillbeimportantforTexaspolicymakerstounderstandhowtheirstandardsdifferfromthoseofotherstates.

Futurealignmentstudiescouldbenefitfromapply-ingdifferentdefinitionsofpartialalignment.Whilethecurrentstudyincludedjustasinglepartialalignmentrating,acontentalignmentscalewithmorethanthreelevelscouldincludepartialalign-mentratingsatvaryinglevels.Theresultswouldmorepreciselydescribethesimilaritiesbetweenthebenchmarkandcomparisonsetsofstandards.

Finally,astudyusingmorethantworeviewersmighthavegreaterreliability(Webb,Herman,andWebb2007,p.25),althoughthatremainsanempiricalquestion.


APPEnDIx A mEThoDology

Thisappendixdescribesthemethodologyandrat-ingscaleusedtoexaminecontentalignmentandtocomparethedistributionofstatementsacrossfourlevelsofacognitivecomplexityscale,aswellasthestepsofthealignment(content)andrating(contentalignmentandcognitivecomplexity)processes.

Aligningcontent

ThecontentalignmentmethodologyusedinapreviousseriesofRegionalEducationalLabora-torySouthweststudies(ShapleyandBrite2008a–e;Timmset al.2007a–e;Rolfhuset al.2010)wasadaptedforthecurrentstudy.TheTimmset al.andShapleyandBritestudiesinvolvedthealignmentofstateassessmentstandardsanditemspecificationswiththebenchmarkNationalAssessmentofEdu-cationalProgress(NAEP).11TheRolfhuset al.studyinvolvedthecontentalignmentofthreesetsofcol-legereadinessstandardssetswithafourthcollegereadinessstandardssetdesignatedasabenchmark.Thecurrentstudyemployedthesamethree-levelcontentalignmentratingscaleandprocessforrec-oncilingindependentreviewerratingstocomputethepercentageofbenchmarkstandardsstatementsintheACTCollegeReadinessStandards(ACT,Inc.2007)andtheAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBenchmarks(Achieve,Inc.2004)thatalignfully,partially,ornotatallwithcomparisonstatementsintheTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008).12

Becausethecurrentstudyexaminedthreesetsofstandards,thepairwisecomparisonapproachofthepreviousstudieswasadapted,withACTandADPdesignatedinturnasthebenchmarksetandalignedindividuallywithTEKSELAR,thecom-parisonset(figureA1).Inmanycases,thecontentinasinglebenchmarkstatementalignedwiththecontentinmultipleTEKSELARstatements,andthecontentinasingleTEKSELARstatementalignedtothecontentinmultiplebenchmarkstatements.13

figure a1

Pairwise comparison methodology using AcT and American Diploma Project standards sets as benchmarks for alignment with the Texas Essential Knowledge and skills for English language arts and reading standards set, 2009

TEKS ELAR 278

ACT 191

ADP 62

TEKS ELAR 278

Source: authors.

Twoseparatealignmenttableswerecreatedtoconductthesepairwisecomparisons,withtheleftcolumnpopulatedbyeithertheACTstandardsstatements(ACT–TEKSELARtable)ortheADPstandardsstatements(ADP–TEKSELARtable).Onestandardsstatementformsonerowofthealignmenttableforeachbenchmarkstandardsset.

TwoindependentreviewersexaminedcontentintheACTandADPstandardssetsforcontentsimi-lartothatintheTEKSELARstandardsset.Whensimilarcontentwasfound,eachreviewerindepen-dentlyratedthelevelofalignmentbetweenthetwostandardsstatementsasfullorpartial.Thecontentalignmentwasindependentofthecogni-tivecomplexityratings—thereviewersdidnotconsiderthecognitivecomplexityratingswhencomparingcontentstatementsorwhenratingthecontentalignmentlevel.

Levelsofcontentalignmentweredefinedasfollows:

• Fully aligned.Allthecontentinabenchmark(ACTorADP)statementalignswithcontentinoneormorestatementsinthecomparison(TEKSELAR)standardsset.

figure a2

Example of the structure of the full alignment table, 2009

american diploma Project TeKS elar content standard statement standard statements ratinga reviewer notes

a. language

a1. demonstra adP l of 110.31 b 13(d); 110.32 b 13(d); 110.33 b 13(d); 110.34 3 reviewers added TeKS 25 standard engli strand gh the b 13(d): edit drafts for grammar, mechanics, and because it states “students use of grammar, punctuation, spelling

content capitalization and spelling. 110.31 b 17(a); 110.32 b 17(a): use and underst rating as

determined the function of the following parts of speech i by expert context of reading, writing, and speaking: reviewers adP

content (i) more complex active and passive tenses and statement verbals (gerunds, infinitives, participles);

(ii) restrictive and nonrestrictive relative clauses; and TeKS elar (iii) reciprocal pronouns (e.g., each other, statement(s) that

one another) contain content that aligns with content in

speak clearly to the point using the conventions of . . .” for english i

expert reviewer comments

the adP statement

Source:Fullalignmenttable,availablefromRegionalEducationalLaboratorySouthwest.

15 aPPendix a. meThodology

• Partially aligned.Someofthecontent(1–99 percent)inthebenchmark(ACTorADP)statementalignswithsomeofthecontentinthecomparison(TEKSELAR)standardsset.

• Not aligned.Noneofthecontentinthebenchmark(ACTorADP)statementalignswithanyofthecontentinthecomparison(TEKSELAR)standardsset.

Amoredetaileddescriptionofthisthree-levelcon-tentalignmentratingscale,includingexamples,isprovidedinappendixC.Finalalignmentsandratingsweredeterminedduringaconsensusmeet-ingwiththeseniorreviewer.FigureA2showshowthecontentalignmenttableswerestructuredandpopulatedandillustratestheone-to-manycorrespondenceofabenchmarkstandardsstatementtoTEKSELARstandardsstatements.

Thebroaddefinitionofpartial alignment, rangingfromstatementswithverylittlesharedcontenttostatementswithalmostcompletesharedcontent,isalimitationofthisstudy.Modifyingthenumberanddefinitionoflevelsofalignmentcouldresultindifferentlevelsofconsensusacrossthestandardssets.

Ratingcognitivecomplexity

Cognitivecomplexitywasassessedbycomparingthedistributionofstandardsstatementsfromeachsetofstandardsacrossfourlevelsofcognitivecomplexity(Webb2002).Thecognitivecomplexityratingswerecompletedbeforethecontentalignment.Eachstatementfromeachsetofstandardswasratedindependently(therewasnobenchmark);asaresult,thereisnodirectionalitytothecognitivecomplexitycomparison.

Cognitivecomplexityratingswereassignedtoeachstatementbytwoindependentreviewers.Athree-columncognitivecomplexityratingtablewascreatedforeachstandardsset;eachstandardsstatementformedasinglerowinthefirstcolumnofthetable,andthecognitivecomplexitylevelcorrespondingtoeachstatementformedthesecondcolumnofthetable(figure A3providesanexample).IndividualreviewersworkedindependentlytoratethecognitivecomplexityofeachstatementusingWebb’s(2002)depthofknowledge(DoK)scale:

• Level 1–recall requiresstudentstousesimpleskillsorabilitiestoretrieveorrecitefacts.

--

--

--

-

-


• Level 2–skill/concept requiresalevelofcom-prehensionandsubsequentprocessingacrossportionsoftexttomakeinferencesbeyondsimplerecallorrecitationofstatedfacts.

• Level 3–strategic thinking focusesonreason-ing,planningskills,makingmorecomplexinferences,andapplyingideasfromthetext;studentsmaybeencouragedtoexplain,gener-alize,orconnectideas.

• Level 4–extended thinking requiresinvestiga-tionandhigherorderthinkingskillstobeabletoprocessmultiplesolutionstoagivenproblem.

AmoredetaileddescriptionoftheWebbDoKscale,includingexamples,isprovidedinappen-dix G.Thetwoprimaryreviewers’independentcognitivecomplexityratingswerediscussedduringconsensusmeetingsunderthesupervi-sionofaseniorreviewer,andthefinalratingwasdeterminedatthattime.AnexampleofhoweachcognitivecomplexityratingtablewasstructuredandpopulatedisprovidedinfigureA3.

TeKS elar standards

cognitive complexity

ratinga reviewer comments

oral and Written conventions

110.31 b 17 oral and Written co s/conventions. Students understand the function of and use the conventions of academic language when spea d writing. Students will continue to apply earlier standards with greater complexity. Students are expected to:

110.31 b 17(a) use and understand the function of the following parts of speech in the context of reading, writing, and speaking: (i) more complex active and passive tenses and verbals (gerunds, infinitives, participles); (ii) restrictive and nonrestrictive relative clauses; and (iii) reciprocal pronouns (e.g., each other, one another);

1

110.31 b 17(b) identify and use the subjunctive mood to express doubts, wishes, and possibilities;

3 This verb tense is not as common as others, and does indicate a deeper knowledge of grammar.

110.31 b 17(c) use a variety of correctly structured sentences (e.g., compound, complex, compound-complex).

3

figure a3

Example of the structure of the cognitive complexity rating table, 2009

Source: Cognitivecomplexitytable,availablefromRegionalEducationalLaboratorySouthwest.

expert reviewer comments

cognitive complexity rating as determined by expert reviewers

TeKS elar statement

TeKS elar strand

Stepsinthealignmentandratingprocess

Weeklyprogressmeetingswereheldbetweenthetwoteamsmanagingtheoverallstudy:theresearchteamthatwasresponsibleforthestudydesign,implementation,analysis,andreportingandthereviewteamthatconductedthecontentalignmentandcognitivecomplexityratings.Alsoduringthesemeetings,thereviewteamprovidedanycompleteddatatablestotheresearchteamforreview.

Step 1—selecting reviewers.Thestudymethod-ologyrequiredtwoindependentreviewerstoprovideratingsofcontentalignmentandcognitivecomplexityandaseniorreviewertosupervisecon-sensusdiscussions.Thereviewerswhohadpar-ticipatedintherecentlycompletedRolfhuset al.(2010)studywereselectedasreviewersforthecurrentstudy.14BecausetheRolfhuset al.studyin-volvedalignmentofEnglishlanguageartscollegereadinessstandards,includingtheACTandADP,andusedthesamemethodologyandratingscalesasthecurrentstudy,thereviewerswerefamiliarwiththeACTandADPstandardsstatements,

17 aPPendix a. meThodology

theratingscales,andtheindependentratingandconsensusprocess.InformationaboutreviewerqualificationsandrolesisprovidedinappendixB.Thereviewerswererecruitedandmanagedbyauniversity-basedorganizationthatspecializesinquantitativeandqualitativeresearch,programevaluation,andprofessionaldevelopmentforeducators.15

Step 2—training reviewers.Beforetraining,reviewersreceivedthethreesetsofstandardsusedinthisstudywithinstructionstoreviewtheirstructure,organization,andcontent.Dur-ingathree-hourtrainingsession,reviewerswereretrainedonthethree-levelcontentalignmentscaleandthefour-levelcognitivecomplexityscale(Webb2002).16Trainingconsistedofacarefulre-viewanddiscussionoftheresearchquestionsandrationaleforthestudy,eachoftheratingscalesandhoweachscalelevelwasdefinedanddiffer-entiatedfromtheothers,reviewanddiscussionofalignmentsamplesandtheirappropriateratings,andpracticeconductingalignmentandratingactivitiesandreachingconsensus.

Step 3—rating cognitive complexity levels.Follow-ingtraining,reviewersindependentlyratedthecognitivecomplexitylevelofeachTEKSELARstatementusingtheWebb(2002)DoKscalede-scriptions(seeappendixG),sothattheywouldbefamiliarwiththeTEKSELARstatementsbeforemakingcontentalignmentdecisions.CognitivecomplexityratingsfortheACTandADPsetsofstandardshadbeencompletedaspartoftheRolf-huset al.(2010)study.Sincethatstudyandthisoneusedthesamemethodologyandreviewteam,theADPandACTcognitivecomplexityratingsfromRolfhuset al.wereusedforthecurrentstudyaswell.

Step 4—achieving consensus on cognitive complexity levels.AftercompletingindividualcognitivecomplexityratingsforallTEKSELARstatements,thetwoindependentreviewersmetwiththeseniorreviewertocompareratingsandachieve

consensuswhereratingsdiffered.Theroleoftheseniorreviewerwastofacilitateconsensusandmakethefinaldecisionifconsensuscouldnotbereached.

Step 5—comparing and aligning ACT–TEKS ELAR content.UsingtheACT–TEKSELARcontentalignmenttableandbeginningwiththefirstACTstatementinthefirstACTstrand,eachreviewerindependentlyandsystematically17searchedallTEKSELARstatementsforcontentalignedtotheACTstatement.Thisreviewwasintendedtogivereviewersanoverallimpressionofcontentandstructure.Next,eachreviewerusedthecontentalignmenttabletoconductamoredetailedexami-nation,startingwithanACTcontentstatementandthensearchingTEKSELARforanystatementsthatcontainedallorpartofthesamecontent.

Thiswasanexhaustivesearch:allTEKSELARstatementswithaligningcontentwereincluded.OnceallfullyandpartiallyalignedTEKSELARstatementswereidentified,thereviewerassignedacontentalignmentlevelratingtotheACTstate-mentbasedonthecumulativecontentofallthealignedTEKSELARstatements(fully,partially,ornotaligned).

ConsensusmeetingsbetweentheindependentreviewersandtheseniorreviewerwereheldaftercompletionofanACTstrand—approximatelyeverytwoweeks.Afterachievingconsensus,thereviewersreturnedtoindependentstatementalignmentandratingonthenextACTstrand.ThiscyclecontinueduntilallpossibleACTandTEKSELARstatementswerealignedandthecontentalignmentlevelswererated.

Step 6—comparing and aligning ADP–TEKS ELAR content.TheADP–TEKSELARcontentalignmentwasconductedinthesamemannerastheACT–TEKSELARalignment,withreviewersindepen-dentlycompletingthefirsttwoADPstrandsbeforeholdingaconsensusmeetingwiththeseniorreviewer.


APPEnDIx b REvIEWER quAlIfIcATIons AnD RolEs AnD InTERRATER RElIAbIlITy fInDIngs

Thisappendixprovidesmoredetailonreviewerqualificationsandinterraterreliability.

Reviewerqualificationsandroles

ByusingthesamereviewersastheRolfhuset al.(2010)study,thecurrentstudywasabletotakeadvantageoftheirrecentexperiencewithalarge-scalealignmentofEnglishlanguageartscollegereadinessstandardsintendedfornationaluse.TheRolfhuset al.(2010)studyincludedtheACTCollegeReadinessStandards(ACT,Inc.2007)andtheAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBenchmarks(Achieve,Inc.2004)setsofstandardsandthesamemethodologyandratingscalesasappliedhere.Thereviewteamconsistedofaseniorreviewerandtwoprimaryreviewers.

TheseniorreviewerhasadoctoraldegreeinEnglisheducationandmorethan13yearsofcom-binedexperiencedesigningandteachingEnglishcoursesforgrades9–12,workshopsforK–12writ-inginstructionandotherwritingseminars,andpostsecondarylevelcourses.Theseniorreviewer’sresearchconcentrationiscompositionstudiesandwritingcentertheoryandpractice.Thetwoprimaryreviewersbothhavedoctoraldegreesincurriculumwithafocusonreadingeducation.Onehasnineyearsofexperienceteachingattheprimarylevel,sevenyearsatthepostsecondarylevel,sixyearsasareadingandEnglishlanguageartsspecialist,andthreeyearsworkingforastatedepartmentofeducation.Theotherprimaryreviewerhas14yearsofexperienceteachingattheprimarylevel,21yearsofexperienceatthepostsecondarylevel,andadditionalexperienceworkingwithvariousstateagencies.

Theseniorreviewerconductedtheinitialtraining,monitoredtheprogressofratings,heldconsensusmeetings,andwasreadytoserveasthefinaljudgeshouldconsensusnotbereachedonanyindividual

rating.Thetwoindependentreviewersconductedthealignmentandassignedtheratings.Inprac-tice,theseniorreviewerdidnotneedtointervenetoreachconsensusonanyfinalrating.

Interraterreliability:contentalignment

Standardsalignmentresearchisasubjectiveprocess,andtheuseofexpertjudgmentiscriti-cal.Multipleexpertsareusedsothattheuniqueperspectiveandknowledgeofeachindividualcontributestoresultsthatgeneralizebeyondoneindividual’sratings.However,theuseofmultipleratersdoesnotprovideanadvantageifthereislittleagreement.Lowlevelsofrevieweragreementmayindicateproblemswiththeratingsscales,reviewerqualifications,training,orothermeth-odologydecisions.Soitisimportanttoevaluateagreementamongreviewersasanindicatorofthequalityoftheresearchprocessandthepotentialgeneralizabilityofthefindings.

Theterminterrater reliabilityreferstothemeth-odsforsummarizingtheamountofagreementbetweenmultiplereviewers.Typically,thehigherthelevelofagreement,thegreatertheconfidencethattheassignedratingswouldbereplicatedbyothersfollowingthesameprocedures.Becausethisstudyemployedtwoexpertreviewerstomakeindependentjudgmentsusingasubjectiveratingscale,comparingtheratingscanprovideinformationabouttheinitialconsensusofthereviewers.However,becausethefinalratingsweredeterminedusingaconsensusmethod-ology,initialagreementordisagreementisnotcriticaltothefinalconsensusratingsandalignment.

Twoapproachestosummarizinginterrateragree-mentarereportedhere:percentagreementandintraclasscorrelation(tableB1).Percentagreementistheproportionofidenticalratingsassignedbybothreviewers.Becausethisapproachdoesnotconsiderthepossibilityofagreementbychanceorofratingsthatareclosebutnotanexactmatch,thestudyalsoreportstheintraclasscorrelation(ShroutandFleiss1979),whichassumesthateach

19 aPPendix b. revieWer qualificaTionS and roleS and inTerraTer reliabiliT y findingS

Table b1

content alignment interrater agreement prior to consensus meeting, 2009

benchmark Percent intraclass standards set agreementa correlationb

acT 65 0.74

american diploma Project (adP) 72 0.81

a.Overallpercentagreementinindependentalignmentratingspriortotheconsensusmeetingforthe62ADPbenchmarkstatementsandthe191ACTbenchmarkstatements.

b.CalculatedusingSPSS,version16.0(SPSS,Inc.2007)—twowayrandomeffectsmodel,absoluteagreement,averagemeasures.ThisisequivalenttoShroutandFleiss(1979)Case2,whichassumesthatthetworatersaredrawnfromapopulationofraters.

Source:SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinACT,Inc.(2007)andAchieve,Inc.(2004).

reviewerbringsmeasurementerrorintotheratingprocess.Theintraclasscorrelationalsoaccountsforsmalldiscrepancies,suchaswhenreviewer1assignsaratingofafullyalignedandreviewer2assignsaratingofpartiallyaligned.

Interraterreliability:cognitivecomplexity

Interraterreliabilityforcognitivecomplexityisreportedinthesamewayasforcontentalignment,withtwoexceptions.TableB2containsallthreestandardssetsandincludescognitivecomplex-ityratingsforeverystatementwithineachset,regardlessofwhetherstatementsalignedtoanystatementsfromthebenchmarkset.ThecognitivecomplexityratingsforACTandADParetakenfromRolfhuset al.(2010),18whichusedthesamemethodologyandreviewers.Thesamereview-erscompletedtheTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008)ratingsasaseparateactivity.Percentagree-mentmayappearlowerforthecognitivecomplex-ityratingsthanforthecontentalignmentratingsbecausetheWebbdepthofknowledge(DoK)scalehasfourlevelsandthecontentalignmentscalehasonlythree.Afour-levelscaleprovidesmoreopportunityforraterdisagreementthandoesathree-levelscale.

Table b2

cognitive complexity interrater agreement prior to consensus meeting, 2009

number of Percent intraclass Standards set statementsa agreement correlationb

acT 191 46 0.67

american diploma Project (adP) 59c 75 0.77

Texas essential Knowledge and Skills for english language arts and reading standards (TeKS elar) 278 68 0.75

a.Cognitivecomplexityratingswereconductedforallstatementsineachstandardsset.

b.CalculatedusingSPSS,version16.0(SPSS,Inc.2007)—twowayrandomeffectsmodel,absoluteagreement,averagemeasures.ThisisequivalenttoShroutandFleiss(1979)Case2,whichassumesthetworatersaredrawnfromapopulationofraters.

c.StatisticsforADParebasedonpairedratingsfor59of62statements.Reviewer1didnotassignratingstothreestatementspriortotheconsensusmeetingduetouncertaintyabouthowtoapplytheWebbDoKscaleto“softwarepresentations”andtwostatementsabout“explainingthemes”and“demonstratingknowledge”ofliterature.Thesestatementswerediscussedandconsensuswasreachedaswithallotherratings.Itcannotbeknownhowlackofthreeinitialratingsmayhaveaffectedfinalconsensusratingsoragreementrates.

Source:ForACTandADPexpertrateractivities,April–September2008(Rolfhuset al.2010).ForTEKSELAR,summaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinTexasEducationAgency(2008).

Interraterreliabilityresultsincontext

Becausethereislimitedresearchonstudiesofstandards-to-standardsalignment,itisdifficulttodrawdirectcomparisonswithagreementratesreportedinthealignmentliterature.Evenforatypicaltestitemtostandardalignmentstudy,therearenouniversalguidelinesforagreementratestobeconsideredgood.Agreementmustbecomparedwithsimilarstudies.Thisiscomplicatedbythefactthatresearchersoftenreportdifferentinterrateragreementstatistics(suchasintraclasscorrelation,percentagreement,Cohen’sKappa,andgeneralizabilitycoefficients),iftheyreportanythingatall.


TheWebbAlignmentTooltrainingmanual(Webb2005)providesroughguidelines.Asageneralrule,Webb(pp. 115–116)considersanintraclasscorrelationof0.70ormoretobe“adequate”and0.80orgreatertobe“good”;“pair-wisecompari-sons”(orpercentageagreement)oflessthan0.50areconsidered“poor,”whilehigherthan0.60is“reasonable”and0.70orhigheris“good.”Thesecategoriesarenotformalordefinitive,butusefulbenchmarksfromoneoftheleadersinalignmentresearch.NotethatWebb’sclassificationsusesixtoeightindependentraters,whileonlytwowereusedinthecurrentstudy.

AgreementratesareshownforthecurrentstudyandotherresearchinfigureB1(percentagree-ment)andfigureB2(intraclasscorrelations)forbothcontentalignmentandcognitivecomplexity.

Inthecurrentstudy,onlytheADPcognitivecom-plexityagreementratesdidnotmeetWebb’srea-sonableorgoodclassification(seefigureB1).ADPcognitivecomplexityratingwasthefirsttaskcom-pletedbythenewteamaspartoftheRolfhuset al.(2010)studyninemonthsbeforethecurrentstudy.Thismayhavecontributedtotheloweragreementratesbeforeconsensusinthatstudy,eventhoughthesameratingteamparticipatedinboth.

AllthreecomparisonstudiespresentedinfigureB1(thestudiesareneitherrepresentativenorexhaus-tive,butarerecentexamplesofagreementresultsfromleadingresearchers)hadhigherpercentagreementlevelsthanthecurrentstudy.WixsonandDutro(2002,p.94),reporting93percentagree-ment,usedtworaterstoalignthecontentofread-ingstandardsforprimarygradesfrom14statestoalimitedbenchmarklistof12contentstatementsdevelopedbytheresearchteam.SternandAhlgren(2002),reporting87percentagreement(usingafive-pointscale),trainedseventwo-memberteamstoratecontentalignmentfromninesciencetext-bookstoasubsetofsciencecontentbenchmarksfromProject2061(AmericanAssociationfortheAdvancementofScience2009).WixsonandDutro(2002,p.8)alsoreported94percentagreementonWebbcognitivecomplexityratingsforonestate’sgradeK–5readingobjectivesand80percentforthealignedassessmentitems.

Theintraclasscorrelationsforthecurrentstudy,withtheexceptionofACTcognitivecomplexity,arewithinWebb’s(2005)adequateorgoodranges(figureB2).

Theintraclasscorrelationagreementratesforthecomparisonstudiesvarywidely.Porteret al.(2007),inarecentstudyofacurriculum-to-standardsalignment,comparedEnglishlanguageartsstandardswiththecurriculumactuallytaughtintheclassroomatthreedif-ferentgradesintwostates.TheG-coefficients(equivalenttotheintraclasscorrelationsre-portedhere—seetablesB1andB2)fortworatersrangedfrom0.47to0.83.Theintraclasscorrelationsinthecurrentstudyarewithinthesamerange.Webb,Horton,andO’Neal(2002,p. 11)reportintraclasscorrelationsof0.36–0.92(M = 0.73)forWebbcognitivecomplexityrat-ingsoflanguageartsassessmentitems.Theintraclasscorrelationsinthecurrentstudyareatthehigherendoftherangesreportedinthesetwocomparativestudies.

WiththeexceptionofpercentagreementforADPcontentalignment,theresultsindicatethatinter-raterreliabilityinthecurrentstudywasconsistentwithsimilarresearchforbothcontentalignmentandcognitivecomplexityratings,atleastwithinthebroadrangeofagreementratesreportedbythesmallnumberofalignmentstudiesthatprovidethem.Highinterrateragreementisimportantinstudiesthatcomputeameanratingfromseveralraters(forexample,Webb,Herman,andWebb2007).Thecurrentstudydidnotcomputeameanfromtheindividualraters;rather,aconsensusapproachwasusedtodeterminethefinalratings.Nostudies,otherthanRolfhusetel.(2010),wereidentifiedthatappliedanidenticalmethodologyforstandards-to-standardscontentalignmentandthatalsoincorporatedcognitivecomplex-ityratings,sodirectcomparisonswithalargerresearchbaseofstudiesusingidenticalmethodsisnotpossible.Toreiterate,veryhighinitialagree-mentisnotcriticalbecausethisstudyusedaconsensusprocess,notameanofmultipleratingstodeterminefinalratings.Consequently,thelevelofagreementfoundinthepresentstudy’sinterimratingprocessisacceptable.

aPPendix b. revieWer qualificaTionS and roleS and inTerraTer reliabiliT y findingS 21

Current studycontent alignment

Current studycognitive complexity

Comparative studiesContent

alignmentCognitive

complexity

Webb (2005) criteria for percentageagreement among raters:

Good = 70 percent or greaterReasonable = 60 percent or greaterPoor = Less than 50 percent

ACTcognitive

complexity

ADPcognitive

complexity

TEKS ELARcognitive

complexity

Stern andAhlgren (2002)

contentalignment

Wixson andDutro (2002)

contentalignment

Wixson et al.(2002)

cognitivecomplexity

100

90

70

60

80

50

40

30

0

ACT–TEKS ELAR

contentalignment

ADP–TEKS ELAR

contentalignment

figure b1

Interrater reliability in the current study and other research, 2009 (percent agreement)

Source: Summary of reviewer ratings completed June–August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas Education Agency (2008); Rolfhus et al. 2010; Stern and Ahlgren 2002; Webb 2005; Wixson and Dutro 2002; Wixson et al. 2002.

1.00

0.90

Content alignment

Cognitivecomplexity

0.70

0.60

0.80

0.50

0.40

Webb (2005) criteria for intraclass correlation:

Good = 0.80 or greaterAdequate = 0.70 or greater

0.30

0.00

ACT–TEKS ELAR

contentalignment

ADP–TEKS ELAR

contentalignment

ACTcognitive

complexity

ADPcognitive

complexity

TEKS ELARcognitive

complexity

Porter et al.(2007)

contentalignment

Webb, Horton, andO’Neal (2002)

cognitivecomplexity

Current studycontent alignment

Current studycognitive complexity

Comparative studies

figure b2

Interrater reliability in the current study compared to other research, 2009 (intraclass correlation)

Source: Summary of reviewer ratings completed June–August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas Education Agency (2008); Porter et al. 2008; Rolfhus et al. 2010; Webb et al. 2002; Webb 2005.


APPEnDIx c ExAmPlEs of fully AnD PARTIAlly AlIgnED sTATEmEnTs

Fully alignedmeansthatstatementsintheTexasEssentialKnowledgeandSkillsforEnglishlan-guageartsandreadingstandards(TEKSELAR,TexasEducationAgency2008)alignedtoallportionsofastatementintheACTCollegeReadi-nessStandards(ACT,Inc.2007)ortheAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBenchmarks(Achieve,Inc.2004).Table C1showstwoexamplesoffullalignment.Inexample1,onestatementalignsfullytotheADPstatement.Thereviewernotesexplainthatthere

wasanexactmatchoflanguagesothereviewersdidnothavetoinferthelevelofalignment.Inexample2,thetwostatementsfromthecompari-sonsetwhenconsideredtogetheralignfullytotheADPstatement.

Thetermpartially alignedmeansthatstate-mentsintheTEKSELARalignedtoonlysomeoftheADPorACTstandard.TableC2providestwoexamplesofpartialalignment.Inexample1,thethreestatements(consideredtogether)partiallyaligntotheADPstatement.Thereviewernotesexplainthat“ADPreferstobroaduseofroots,affixes,andcognatestoreadunfamiliarwords,”whilethecomparisonstandardssetis

Table c1

Examples of fully aligned standards statements, 2009

Texas essential Knowledge and Skills for english language arts and

american diploma Project reading standards statements with benchmark statement full alignment to the adP statement reviewer notes

example 1

adP logic strand 110.33 b 9(b): distinguish between a complete match of language, didn’t e7. understand the distinction between inductive and deductive reasoning and need to infer match. a deductive argument (where, if analyze the elements of deductively the premises are all true and the and inductively reasoned texts and argument’s form is valid, the conclusion the different ways conclusions are is inescapably true) and inductive supported argument (in which the conclusion provides the best or most probable explanation of the truth of the premises, but is not necessarily true).

example 2

adP informational Text strand 110.31 b 9(a): summarize text and f4. distinguish between a summary and distinguish between a summary that a critique. captures the main ideas and elements of

a text and a critique that takes a position and expresses an opinion; 110.32 b 9(a): summarize text and distinguish between a summary and a critique and identify non-essential information in a summary and unsubstantiated opinions in a critique

Note:Statementidentifiercodes,suchasE7and110.32b9(A),wereusedtoidentifyspecificstandardstatements.ThecodesusedtoidentifyADPstatementsfollowedADP’scodingformat;forexample,“E”indicatesastatementinthelogicstrandand“7”indicatestheseventhstandardstatementinthatstrand.ThecodesusedtoidentifyTEKSstatementsfollowedTEKS’scodingformat;forexample,110.32indicatesthestandardisEnglishII;“b”indicatesthestatementiswithinTEKSknowledgeandskills;“9”indicatesthestandardistheninthstandardwithinTEKSknowledgeandskills;and“(A)”indicatesthestandardstatementisthefirststudentexpectationunderstandard9.


23 aPPendix c. examPleS of fully and ParTially aligned STaTemenTS

Table c2

Examples of partially aligned standards statements, 2009

Texas essential Knowledge and Skills for english american diploma Project language arts and reading standards statements benchmark statement with partial alignment to the adP statement reviewer notes

example 1 110.31 b 1(a); 110.32 b 1(a); 110.33 b 1(a); 110.34 b 1(a): The reviewers defined the adP language strand determine the meaning of grade-level technical academic interpretation of the word a3. use roots, affixes and english words in multiple content areas (e.g., science, cognate to include “meanings cognates to determine the mathematics, social studies, the arts) derived from latin, across languages” which meaning of unfamiliar words. greek, or other linguistic roots and affixes; would include i: id. They

110.31 b 1(d): describe the origins and meanings of foreign considered a rating of 3, words or phrases used frequently in written english (e.g., but 1a at each level is very caveat emptor, carte blanche, tete a tete, pas de deux, bon specific, specifying/limiting to appetit, quid pro quo) content specific vocabulary, 110.33 b 1(d): recognize and use knowledge of cognates in whereas adP refers to broad different languages and of word origins to determine the use of roots, affixes, and meaning of words cognates to read unfamiliar

words.

example 2 110.33 b 21(a); 110.34 b 21(a): follow the research plan to There is not enough adP research strand gather evidence from experts on the topic and texts written congruence in the language d2. gather relevant for informed audiences in the field, distinguishing between to make this a complete information from a variety reliable and unreliable sources and avoiding over-reliance on match. TeKS covers more of print and electronic one source; broadly with no direct sources, as well as from direct mention of observations, observation, interviews and interviews, and surveys. surveys.

Note:Statementidentifiercodes,suchasA3and110.31b1(A),wereusedtoidentifyspecificstandardstatements.ThecodesusedtoidentifyADPstatementsfollowedADP’scodingformat;forexample,“A”indicatesastatementinthelanguagestrandand“3”indicatesthethirdstandardstatementinthatstrand.ThecodesusedtoidentifyTEKSstatementsfollowedTEKS’scodingformat;forexample,110.31indicatesthestandardisEnglishI;“b”indicatesthestatementiswithinTEKSknowledgeandskills;“1”indicatesthestandardisthefirststandardwithinTEKSknowledgeandskills;and“(A)”indicatesthestandardstatementisthefirststudentexpectationunderstandard1.


morespecific.Inexample2,onlyonestatement thatthecomparisonstatementdoesnotincludefromthecomparisonstandardssetispartially “directmentionofobservations,interviews,andalignedtotheADPstatement.Thereviewersnote surveys.”


APPEnDIx D conTEnT AlIgnmEnT fInDIngs by sTRAnD

Thecontentalignmentanalysesexaminedwhetherstatementsinthebenchmarkstandardssets,ACTCollegeReadinessStandards(ACT,Inc.2007)andtheAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBench-marks(Achieve,Inc.2004)alignfully,partially,ornotatallwithcomparisonstatementsintheTexasEssentialKnowledgeandSkillsforEng-lishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008).Inmanycases,thecontentinasingleACTorADPstate-mentalignswithcontentinmorethanoneTEKSELARstatement.

ACTcontentalignmentfindings

ACT’sEnglishlanguageartsstandardsareorga-nizedintothreestrands(English,Reading,andWriting)containing16substrands(E-1throughW-5)and191standardsstatements.ThelevelofcontentalignmentbetweenACTandTEKSELARstatementsisshownintableD1.

AmericanDiplomaProjectcontentalignmentfindings

ADP’sEnglishlanguageartscollegereadinessstandardsareorganizedintoeightstrandsconsist-ingof62standardsstatements.ThelevelofcontentalignmentbetweenADPandTEKSELARcontentstatementsisshownintableD2.

Table d1

Alignment of AcT statements with Texas Essential Knowledge and skills for English language arts and reading standards statements at each level of content alignment, by AcT strand and substrand, 2009

fully aligned Partially aligned not aligned Total acT strand and substrand number number Percent number Percent number Percent

English 71 6 8 63 89 2 3

e-1: Topic development 11 4 36 5 45 2 18

e-2: organization 12 1 8 11 92 0 0

e-3: Word choice 13 0 0 13 100 0 0

e-4: Sentence structure 10 1 10 9 90 0 0

e-5: conventions of usage 11 0 0 11 100 0 0

e-6: conventions of punctuation 14 0 0 14 100 0 0

Reading 58 3 5 46 79 9 16

r-1: main ideas 12 2 17 9 75 1 8

r-2: Supporting details 12 1 8 11 92 0 0

r-3: Sequential, comparative, and 18 0 0 11 61 7 39 cause-and-effect relationships

r-4: meanings of words 7 0 0 6 86 1 14

r-5: generalizations and conclusions 9 0 0 9 100 0 0

Writing 62 18 29 34 55 10 16

W-1: expressing judgments 14 0 0 12 86 2 14

W-2: focusing on the topic 8 3 38 5 63 0 0

W-3: developing a position 10 3 30 3 30 4 40

W-4: organizing ideas 15 9 60 6 40 0 0

W-5: using language 15 3 20 8 53 4 27

All strands and substrands 191 27 14 143 75 21 11


Source:SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinACT,Inc.(2007)andTexasEducationAgency(2008).

25 aPPendix d. conTenT alignmenT findingS by STrand

Table d2

Alignment of American Diploma Project statements with Texas Essential Knowledge and skills for English language arts and reading standards statements at each level of content alignment, by ADP strand, 2009

fully aligned Partially aligned not aligned Total adP strand number number Percent number Percent number Percent

a. language 7 4 57 3 43 0 0

b. communication 7 4 57 2 29 1 14

c. Writing 10 5 50 5 50 0 0

d. research 5 2 40 3 60 0 0

e. logic 9 3 33 5 56 1 11

f. informational text 11 6 55 5 45 0 0

g. media 4 0 0 3 75 1 25

h. literature 9 6 67 2 22 1 11

All strands 62 30 48 28 45 4 6




APPEnDIx E nonAlIgnED sTAnDARDs sTATEmEnTs

ACTCollegeReadinessStandards(ACT,Inc.2007)andtheAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBenchmarks(Achieve,Inc.2004)standardsstatementsthatdonotalign

withTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008)areshownintablesE1andE2.TEKSELARstandardsstatementsthatdonotaligntoACTandADPstandardsstatementsarepresentedbystrandintablesE3andE4.

Table e1

AcT statements that did not align with Texas Essential Knowledge and skills for English language arts and reading standards, by AcT strand, 2009

Statement identifier Standards statement

R: Reading strand statements

r-1 main ideas and author’s approach

13-15-1 recognize a clear intent of an author or narrator in uncomplicated literary narratives

r-3 Sequential, comparative, and cause-and-effect relationships

13-15-2 recognize clear cause-effect relationships described within a single sentence in a passage

16-19-2 recognize clear cause-effect relationships within a single paragraph in uncomplicated literary narratives

20-23-3 identify clear cause-effect relationships in uncomplicated passages

24-27-4 understand implied or subtly stated cause-effect relationships in uncomplicated passages

24-27-5 identify clear cause-effect relationships in more challenging passages

28-32-3 understand implied or subtly stated cause-effect relationships in more challenging passages

33-36-3 understand implied, subtle, or complex cause-effect relationships in virtually any passage

r-4 meanings of words

16-19-1 use context to understand basic figurative language

E: English strand statements

e-1 Topic development in terms of purpose and focus

16-19-1 identify the basic purpose or role of a specified phrase or sentence

33-36-1 determine whether a complex essay has accomplished a specific purpose

W: Writing strand statements

W-1 expressing judgments

03-4-1 Show a little understanding of the persuasive purpose of the task but neglect to take or to maintain a position on the issue in the prompt

03-4-2 Show limited recognition of the complexity of the issue in the prompt

W-3 developing a position

03-4-1 offer a little development, with one or two ideas; if examples are given, they are general and may not be clearly relevant; resort often to merely repeating ideas

03-4-2 Show little or no movement between general and specific ideas or examples

05-6-1 offer limited development of ideas using a few general examples; resort sometimes to merely repeating ideas

05-6-2 Show little movement between general and specific ideas and examples

W-5 using language

03-4-1-a Show limited control of language by correctly employing some of the conventions of standard english grammar, usage, and mechanics, but with distracting errors that sometimes significantly impede understanding

(conTinued)

27 aPPendix e. nonaligned STandardS STaTemenTS

Table e1 (conTinued)

AcT statements that did not align with Texas Essential Knowledge and skills for English language arts and reading standards, by AcT strand, 2009


03-4-1-b Show limited control of language by using simple vocabulary

03-4-1-c Show limited control of language by using simple sentence structure

05-6-1-a Show a basic control of language by correctly employing some of the conventions of standard english grammar, usage, and mechanics, but with distracting errors that sometimes impede understanding

05-6-1-b Show a basic control of language by using simple but appropriate vocabulary

Note:ThecodesusedtoidentifyACTstatementspartiallyfollowedACT’scodingformatandweremodifiedbyresearcherstofacilitateuse.ThecodingschemeincludedanumberlettercombinationconveyingthescorerangeandlocationofthestandardstatementintheACTstandardsdocument.


Table e2

American Diploma Project statements that did not align with Texas Essential Knowledge and skills for English language arts and reading standards, by ADP strand, 2009


b. communication

b3. Paraphrase information presented orally by others.

e. logic

e2. identify false premises in an argument.

g. media

g4. apply and adapt the principles of written composition to create coherent media productions using effective images, text, graphics, music and/or sound effects—if possible—and present a distinctive point of view on a topic (for example, PowerPoint presentations, videos).

h. literature

h2. analyze foundational u.S. documents for their historical and literary significance (for example, The declaration of independence, the Preamble to the u.S. constitution, abraham lincoln’s “gettysburg address,” martin luther King’s “letter from birmingham Jail”).

Note: Statementidentifiercodes,suchasB1,wereusedinthestudytoidentifyspecificstandardstatements.ThecodesusedtoidentifyADPstatementsfollowedADP’scodingformat;forexample,“B”indicatesastatementinthecommunicationstrandand“3”indicatesthethirdstandardstatementinthatstrand.



Table e3

Texas Essential Knowledge and skills for English language arts and reading standards statements that did not align to AcT statements, by TEKs ElAR strand, 2009


English I

research/research plan

110.31 b 20 Students ask open-ended research questions and develop a plan for answering them:

110.31 b 20(a) Students are expected to brainstorm, consult with others, decide upon a topic, and formulate a major research question to address the major research topic.

110.31 b 20(b) Students are expected to formulate a plan for engaging in research on a complex, multi-faceted topic.

research/gathering sources

110.31 b 21 Students determine, locate, and explore the full range of relevant sources addressing a research question and systematically record the information they gather:

110.31 b 21(a) Students are expected to follow the research plan to compile data from authoritative sources in a manner that identifies the major issues and debates within the field of inquiry.

110.31 b 21(b) Students are expected to organize information gathered from multiple sources to create a variety of graphics and forms (e.g., notes, learning logs).

110.31 b 21(c) Students are expected to paraphrase, summarize, quote, and accurately cite all researched information according to a standard format (e.g., author, title, page number).

research/synthesizing information

110.31 b 22 Students clarify research questions and evaluate and synthesize collected information:

110.31 b 22(a) Students are expected to modify the major research question as necessary to refocus the research plan.

110.31 b 22(b) Students are expected to evaluate the relevance of information to the topic and determine the reliability, validity, and accuracy of sources (including internet sources) by examining their authority and objectivity.

110.31 b 22(c) Students are expected to critique the research process at each step to implement changes as the need occurs and is identified.

research/organizing and presenting ideas

110.31 b 23 Students organize and present their ideas and information according to the purpose of the research and their audience. Students are expected to synthesize the research into a written or an oral presentation.

110.31 b 23(d) Students are expected to uses a variety of evaluative tools (e.g., self-made rubrics, peer reviews, teacher and expert evaluations) to examine the quality of the research.

listening and speaking/listening

110.31 b 24 Students will use comprehension skills to listen attentively to others in formal and informal settings. Students will continue to apply earlier standards with greater complexity:

110.31 b 24(a) Students are expected to listen responsively to a speaker by taking notes that summarize, synthesize, or highlight the speaker’s ideas for critical reflection and by asking questions related to the content for clarification and elaboration.

110.31 b 24(c) Students are expected to evaluate the effectiveness of a speaker’s main and supporting ideas.

listening and speaking/teamwork

110.31 b 26 Students work productively with others in teams. Students will continue to apply earlier standards with greater complexity. Students are expected to participate productively in teams, building on the ideas of others, contributing relevant information, developing a plan for consensus-building, and setting ground rules for decision-making.

(conTinued)

2aPPendix e. nonaligned STandardS STaTemenTS 9




English II

reading/vocabulary development

110.32 b 1 Students understand new vocabulary and use it when reading and writing:

110.32 b 1 (c) Students are expected to infer word meaning through the identification and analysis of analogies and other word relationships.

reading/comprehension of literary text/theme and genre

110.32 b 2 Students analyze, make inferences and draw conclusions about theme and genre in different cultural, historical, and contemporary contexts and provide evidence from the text to support their understanding:

110.32 b 2(a) Students are expected to compare and contrast differences in similar themes expressed in different time periods.

110.32 b 2(b) Students are expected to analyze archetypes (e.g., journey of a hero, tragic flaw) in mythic, traditional and classical literature.

110.32 b 2(c) Students are expected to relate the figurative language of a literary work to its historical and cultural setting.

reading/comprehension of literary text/poetry

110.32 b 3 Students understand, make inferences and draw conclusions about the structure and elements of poetry and provide evidence from text to support their understanding. Students are expected to analyze the structure or prosody (e.g., meter, rhyme scheme) and graphic elements (e.g., line length, punctuation, word position) in poetry.

reading/comprehension of literary text/drama

110.32 b 4 Students understand, make inferences and draw conclusions about the structure and elements of drama and provide evidence from text to support their understanding. Students are expected to analyze how archetypes and motifs in drama affect the plot of plays.

reading/comprehension of literary text/fiction

110.32 b 5 Students understand, make inferences and draw conclusions about the structure and elements of fiction and provide evidence from text to support their understanding:

110.32 b 5(a) Students are expected to analyze isolated scenes and their contribution to the success of the plot as a whole in a variety of works of fiction.

110.32 b 5(d) Students are expected to demonstrate familiarity with works by authors from non-english-speaking literary traditions with emphasis on 20th century world literature.

reading/comprehension of literary text/sensory language

110.32 b 7 Students understand, make inferences and draw conclusions about how an author’s sensory language creates imagery in literary text and provide evidence from text to support their understanding. Students are expected to explain the function of symbolism, allegory, and allusions in literary works.

reading/comprehension of informational text/persuasive text

110.32 b 10 Students analyze, make inferences and draw conclusions about persuasive text and provide evidence from text to support their analysis:

110.32 b 10(b) Students are expected to analyze contemporary political debates for such rhetorical and logical fallacies as appeals to commonly held opinions, false dilemmas, appeals to pity, and personal attacks.

reading/comprehension of informational text/procedural texts

110.32 b 11 Students understand how to glean and use information in procedural texts and documents:

110.32 b 11(a) Students are expected to evaluate text for the clarity of its graphics and its visual appeal.

110.32 b 11(b) Students are expected to synthesize information from multiple graphical sources to draw conclusions about the ideas presented (e.g., maps, charts, schematics).

(conTinued)





reading/media literacy

110.32 b 12 Students use comprehension skills to analyze how words, images, graphics, and sounds work together in various forms to impact meaning. Students will continue to apply earlier standards with greater depth in increasingly more complex texts:

110.32 b 12(a) Students are expected to evaluate how messages presented in media reflect social and cultural views in ways different from traditional texts.

110.32 b 12(b) Students are expected to analyze how messages in media are conveyed through visual and sound techniques (e.g., editing, reaction shots, sequencing, background music).

110.32 b 12(c) Students are expected to examine how individual perception or bias in coverage of the same event influences the audience.

110.32 b 12(d) Students are expected to evaluate changes in formality and tone within the same medium for specific audiences and purposes.

Writing/writing process

110.32 b 13 Students use elements of the writing process (planning, drafting, revising, editing, and publishing) to compose text:

110.32 b 13(e) Students are expected to revise final draft in response to feedback from peers and teacher and publish written work for appropriate audiences.

Writing/literary texts

110.32 b 14 Students write literary texts to express their ideas and feelings about real or imagined people, events, and ideas. Students are responsible for at least two forms of literary writing:

110.32 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, interesting and believable characters, a range of literary strategies (e.g., dialogue, suspense) and devices to enhance the plot, and sensory details that define the mood or tone.

110.32 b 14(b) Students are expected to write a poem using a variety of poetic techniques (e.g., structural elements, figurative language) and a variety of poetic forms (e.g., sonnets, ballads).

110.32 b 14(c) Students are expected to write a script with an explicit or implicit theme and details that contribute to a definite mood or tone.

Writing/expository and procedural texts

110.32 b 15 Students write expository and procedural or work-related texts to communicate ideas and information to specific audiences for specific purposes:

110.32 b 15(c) Students are expected to write an interpretative response to an expository or a literary text (e.g., essay or review) that: (i) extends beyond a summary and literal analysis; (ii) addresses the writing skills for an analytical essay and provides evidence from the text using embedded quotations; and (iii) analyzes the aesthetic effects of an author’s use of stylistic and rhetorical devices.

110.32 b 15(d) Students are expected to produce a multimedia presentation (e.g., documentary, class newspaper, docudrama, infomercial, visual or textual parodies, theatrical production) with graphics, images, and sound that conveys a distinctive point of view and appeals to a specific audience.




110.32 b 20(b) Students are expected to formulate a plan for engaging in research on a complex, multi-faceted topic.

(conTinued)







110.32 b 21(a) Students are expected to follow the research plan to compile data from authoritative sources in a manner that identifies the major issues and debates within the field of inquiry.

110.32 b 21(b) Students are expected to organize information gathered from multiple sources to create a variety of graphics and forms (e.g., notes, learning logs).

110.32 b 21(c) Students are expected to paraphrase, summarize, quote, and accurately cite all researched information according to a standard format (e.g., author, title, page number).




110.32 b 22(b) Students are expected to evaluate the relevance of information to the topic and determine the reliability, validity, and accuracy of sources (including internet sources) by examining their authority and objectivity.



110.32 b 23 Students organize and present their ideas and information according to the purpose of the research and their audience:

110.32 b 23(d) Students are expected to synthesize the research into a written or an oral presentation that uses a variety of evaluative tools (e.g., self-made rubrics, peer reviews, teacher and expert evaluations) to examine the quality of the research.

110.32 b 23(e) Students are expected to synthesize the research into a written or an oral presentation that uses a style manual (e.g., modern language association, chicago manual of Style) to document sources and format written materials.



110.32 b 24(a) Students are expected to listen responsively to a speaker by taking notes that summarize, synthesize, or highlight the speaker’s ideas for critical reflection and by asking questions related to the content for clarification and elaboration.

110.32 b 24(c) Students are expected to evaluate how the style and structure of a speech support or undermine its purpose or meaning.

listening and speaking/speaking

110.32 b 25 Students speak clearly and to the point, using the conventions of language. Students will continue to apply earlier standards with greater complexity. Students are expected to advance a coherent argument that incorporates a clear thesis and a logical progression of valid evidence from reliable sources and that employs eye contact, speaking rate (e.g., pauses for effect), volume, enunciation, purposeful gestures, and conventions of language to communicate ideas effectively.


110.32 b 26 Students work productively with others in teams. Students will continue to apply earlier standards with greater complexity. Students are expected to participate productively in teams, building on the ideas of others, contributing relevant information, developing a plan for consensus-building and setting ground rules for decision-making.

(conTinued)





English III



110.33 b 1(c) Students are expected to infer word meaning through the identification and analysis of analogies and other word relationships.

110.33 b 1(d) Students are expected to recognize and use knowledge of cognates in different languages and of word origins to determine the meaning of words.

110.33 b 1(e) Students are expected to use general and specialized dictionaries, thesauri, glossaries, histories of language, books of quotations, and other related references (printed or electronic) as needed.



110.33 b 2(b) Students are expected to relate the characters and text structures of mythic, traditional, and classical literature to 20th and 21st century american novels, plays, or films.


110.33 b 3 Students understand, make inferences and draw conclusions about the structure and elements of poetry and provide evidence from text to support their understanding. Students are expected to analyze the effects of metrics, rhyme schemes (e.g., end, internal, slant, eye), and other conventions in american poetry.


110.33 b 4 Students understand, make inferences and draw conclusions about the structure and elements of drama and provide evidence from text to support their understanding. Students are expected to analyze the themes and characteristics in different periods of modern american drama.



110.33 b 5(d) Students are expected to demonstrate familiarity with works by authors in american fiction from each major literary period.

reading/comprehension of literary text/literary nonfiction

110.33 b 6 Students understand, make inferences and draw conclusions about the varied structural patterns and features of literary nonfiction and provide evidence from text to support their understanding. Students are expected to analyze how rhetorical techniques (e.g., repetition, parallel structure, understatement, overstatement) in literary essays, true life adventures, and historically important speeches influence the reader, evoke emotions, and create meaning.


110.33 b 7 Students understand, make inferences and draw conclusions about how an author’s sensory language creates imagery in literary text and provide evidence from text to support their understanding. Students are expected to analyze the meaning of classical, mythological, and biblical allusions in words, phrases, passages, and literary works.

reading/comprehension of informational text/expository text

110.33 b 9 Students analyze, make inferences and draw conclusions about expository text and provide evidence from text to support their understanding:

110.33 b 9(b) Students are expected to distinguish between inductive and deductive reasoning and analyze the elements of deductively and inductively reasoned texts and the different ways conclusions are supported.

(conTinued)







110.33 b 10(b) Students are expected to analyze historical and contemporary political debates for such logical fallacies as non-sequiturs, circular logic, and hasty generalizations.



110.33 b 11(b) Students are expected to translate (from text to graphic or from graphic to text) complex, factual, quantitative, or technical information presented in maps, charts, illustrations, graphs, timelines, tables, and diagrams.




110.33 b 12(b) Students are expected to evaluate the interactions of different techniques (e.g., layout, pictures, typeface in print media, images, text, sound in electronic journalism) used in multi-layered media.

110.33 b 12(c) Students are expected to evaluate the objectivity of coverage of the same event in various types of media.

110.33 b 12(d) Students are expected to evaluate changes in formality and tone across various media for different audiences and purposes.






110.33 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, suspense) and devices to enhance the plot, and sensory details that define the mood or tone.

110.33 b 14(b) Students are expected to write a poem that reflects an awareness of poetic conventions and traditions within different forms (e.g., sonnets, ballads, free verse).

110.33 b 14(c) Students are expected to write a script with an explicit or implicit theme, using a variety of literary techniques.



110.33 b 15(c) Students are expected to write an interpretation of an expository or a literary text that: (i) advances a clear thesis statement; (ii) addresses the writing skills for an analytical essay, including references to and commentary on quotations from the text; (iii) analyzes the aesthetic effects of an author’s use of stylistic or rhetorical devices; (iv) identifies and analyzes the ambiguities, nuances, and complexities within the text; and (v) anticipates and responds to readers’ questions or contradictory information.

(conTinued)





110.33 b 15(d) Students are expected to produce a multimedia presentation (e.g., documentary, class newspaper, docudrama, infomercial, visual or textual parodies, theatrical production) with graphics, images, and sound that appeals to a specific audience and synthesizes information from multiple points of view.




110.33 b 20(b) Students are expected to formulate a plan for engaging in in-depth research on a complex, multifaceted topic.



110.33 b 21(a) Students are expected to follow the research plan to gather evidence from experts on the topic and texts written for informed audiences in the field, distinguishing between reliable and unreliable sources and avoiding over-reliance on one source.

110.33 b 21(b) Students are expected to systematically organize relevant and accurate information to support central ideas, concepts, and themes, outline ideas into conceptual maps/timelines, and separate factual data from complex inferences.

110.33 b 21(c) Students are expected to paraphrase, summarize, quote, and accurately cite all researched information according to a standard format (e.g., author, title, page number), differentiating among primary, secondary, and other sources.




110.33 b 22(b) Students are expected to differentiate between theories and the evidence that supports them and determine whether the evidence found is weak or strong and how that evidence helps create a cogent argument.




110.33 b 23(d) Students are expected to synthesize the research into an extended written or oral presentation that uses a style manual (e.g., modern language association, chicago manual of Style) to document sources and format written materials.

110.33 b 23(e) Students are expected to synthesize the research into an extended written or oral presentation that is of sufficient length and complexity to address the topic.



110.33 b 24(a) Students are expected to listen responsively to a speaker by framing inquiries that reflect an understanding of the content and by identifying the positions taken and the evidence in support of those positions.

110.33 b 24(b) Students are expected to evaluate the clarity and coherence of a speaker’s message and critique the impact of a speaker’s diction and syntax on an audience.

(conTinued)

-






110.33 b 25 Students speak clearly and to the point, using the conventions of language. Students will continue to apply earlier standards with greater complexity. Students are expected to give a formal presentation that exhibits a logical structure, smooth transitions, accurate evidence, well-chosen details, and rhetorical devices, and that employs eye contact, speaking rate (e.g., pauses for effect), volume, enunciation, purposeful gestures, and conventions of language to communicate ideas effectively.

English Iv



110.34 b 1(c) Students are expected to use the relationship between words encountered in analogies to determine their meanings (e.g., synonyms/antonyms, connotation/denotation).

110.34 b 1(d) Students are expected to analyze and explain how the english language has developed and been influenced by other languages.

110.34 b 1(e) Students are expected to use general and specialized dictionaries, thesauri, histories of language, books of quotations, and other related references (printed or electronic) as needed.



110.34 b 2(a) Students are expected to compare and contrast works of literature that express a universal theme.

110.34 b 2(b) Students are expected to compare and contrast the similarities and differences in classical plays with their modern day novel, play, or film versions.


110.34 b 3 Students understand, make inferences and draw conclusions about the structure and elements of poetry and provide evidence from text to support their understanding. Students are expected to evaluate the changes in sound, form, figurative language, graphics, and dramatic structure in poetry across literary time periods.


110.34 b 4 Students understand, make inferences and draw conclusions about the structure and elements of drama and provide evidence from text to support their understanding. Students are expected to evaluate how the structure and elements of drama change in the works of british dramatists across literary periods.



110.34 b 5(d) Students are expected to demonstrate familiarity with works of fiction by british authors from each major literary period.

reading/comprehension of literary text/literary nonfiction

110.34 b 6 Students understand, make inferences and draw conclusions about the varied structural patterns and features of literary nonfiction and provide evidence from text to support their understanding. Students are expected to analyze the effect of ambiguity, contradiction, subtlety, paradox, irony, sarcasm, and overstatement in literary essays, speeches, and other forms of literary nonfiction.


110.34 b 7 Students understand, make inferences and draw conclusions about how an author’s sensory language creates imagery in literary text and provide evidence from text to support their understanding. Students are expected to analyze how the author’s patterns of imagery, literary allusions, and conceits reveal theme, set tone, and create meaning in metaphors, passages, and literary works.

(conTinued)





reading/comprehension of informational text/expository text

110.34 b 9 Students analyze, make inferences and draw conclusions about expository text and provide evidence from text to support their understanding:

110.34 b 9(b) Students are expected to explain how authors writing on the same issue reached different conclusions because of differences in assumptions, evidence, reasoning, and viewpoints.



110.34 b 10(b) Students are expected to draw conclusions about the credibility of persuasive text by examining its implicit and stated assumptions about an issue as conveyed by the specific use of language.



110.34 b 11(a) Students are expected to draw conclusions about how the patterns of organization and hierarchic structures support the understandability of text.

110.34 b 11(b) Students are expected to evaluate the structures of text (e.g., format, headers) for their clarity and organizational coherence and for the effectiveness of their graphic representations.




110.34 b 12(b) Students are expected to evaluate the interactions of different techniques (e.g., layout, pictures, typeface in print media, images, text, sound in electronic journalism) used in multi-layered media.

110.34 b 12(c) Students are expected to evaluate how one issue or event is represented across various media to understand the notions of bias, audience, and purpose.

110.34 b 12(d) Students are expected to evaluate changes in formality and tone across various media for different audiences and purposes.






110.34 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, a clear theme, complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, suspense), devices to enhance the plot, and sensory details that define the mood or tone.

110.34 b 14(b) Students are expected to write a poem that reflects an awareness of poetic conventions and traditions within different forms (e.g., sonnets, ballads, free verse).

110.34 b 14(c) Students are expected to write a script with an explicit or implicit theme, using a variety of literary techniques.

(conTinued)

37aPPendix e. nonaligned STandardS STaTemenTS






110.34 b 15(c) Students are expected to write an interpretation of an expository or a literary text that: (i) advances a clear thesis statement; (ii) addresses the writing skills for an analytical essay including references to and commentary on quotations from the text; (iii) analyzes the aesthetic effects of an author’s use of stylistic or rhetorical devices; (iv) identifies and analyzes ambiguities, nuances, and complexities within the text; and (v) anticipates and responds to readers’ questions and contradictory information.

110.34 b 15(d) Students are expected to produce a multimedia presentation (e.g., documentary, class newspaper, docudrama, infomercial, visual or textual parodies, theatrical production) with graphics, images, and sound that appeals to a specific audience and synthesizes information from multiple points of view.




110.34 b 20(b) Students are expected to formulate a plan for engaging in in-depth research on a complex, multi-faceted topic.



110.34 b 21(a) Students are expected to follow the research plan to gather evidence from experts on the topic and texts written for informed audiences in the field, distinguishing between reliable and unreliable sources and avoiding over-reliance on one source.

110.34 b 21(b) Students are expected to systematically organize relevant and accurate information to support central ideas, concepts, and themes, outline ideas into conceptual maps/timelines, and separate factual data from complex inferences.

110.34 b 21(c) Students are expected to paraphrase, summarize, quote, and accurately cite all researched information according to a standard format (e.g., author, title, page number), differentiating among primary, secondary, and other sources.




110.34 b 22(b) Students are expected to differentiate between theories and the evidence that supports them and determine whether the evidence found is weak or strong and how that evidence helps create a cogent argument.




110.34 b 23(d) Students are expected to synthesize the research into an extended written or oral presentation that uses a style manual (e.g., modern language association, chicago manual of Style) to document sources and format written materials.

110.34 b 23(e) Students are expected to synthesize the research into an extended written or oral presentation that is of sufficient length and complexity to address the topic.

(conTinued)







110.34 b 24(a) Students are expected to listen responsively to a speaker by framing inquiries that reflect an understanding of the content and by identifying the positions taken and the evidence in support of those positions.

110.34 b 24(b) Students are expected to assess the persuasiveness of a presentation based on content, diction, rhetorical strategies, and delivery.


110.34 b 25 Students speak clearly and to the point, using the conventions of language. Students will continue to apply earlier standards with greater complexity. Students are expected to formulate sound arguments by using elements of classical speeches (e.g., introduction, first and second transitions, body, and conclusion), the art of persuasion, rhetorical devices, eye contact, speaking rate (e.g., pauses for effect), volume, enunciation, purposeful gestures, and conventions of language to communicate ideas effectively.


110.34 b 26 Students work productively with others in teams. Students will continue to apply earlier standards with greater complexity. Students are expected to participate productively in teams, offering ideas or judgments that are purposeful in moving the team towards goals, asking relevant and insightful questions, tolerating a range of positions and ambiguity in decision-making, and evaluating the work of the group based on agreed-upon criteria.

Note:ThecodesusedtoidentifyTEKSstatementsfollowedTEKS’scodingformat;forexample,110.31indicatesthestandardisEnglishI;“b”indicatesthestatementiswithinTEKSknowledgeandskills;“20”indicatesthestandardisthe20thstandardwithinTEKSknowledgeandskills;and“(A)”indicatesthestandardstatementisathefirststudentexpectationunderstandard20.


Table e4

Texas Essential Knowledge and skills for English language arts and reading standards statements that did not align to America Diploma Project statements, by TEKs ElAR strand, 2009


English I



110.31 b 1(e) Student are expected to use a dictionary, a glossary, or a thesaurus (printed or electronic) to determine or confirm the meanings of words and phrases, including their connotations and denotations, and their etymology.


110.31 b 7 Students understand, make inferences and draw conclusions about how an author’s sensory language creates imagery in literary text and provide evidence from text to support their understanding. Students are expected to explain the role of irony, sarcasm, and paradox in literary works.



110.31 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, interesting and believable characters, and a range of literary strategies (e.g., dialogue, suspense) and devices to enhance the plot.



(conTinued)






English II



110.32 b 1 (d) Students are expected to show the relationship between the origins and meaning of foreign words or phrases used frequently in written english and historical events or developments (e.g., glasnost, avantgarde, coup d’état).



110.32 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, interesting and believable characters, a range of literary strategies (e.g., dialogue, suspense) and devices to enhance the plot, and sensory details that define the mood or tone.




English III standards



110.33 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, suspense) and devices to enhance the plot, and sensory details that define the mood or tone.




English Iv standards



110.34 b 1(d) Students are expected to analyze and explain how the english language has developed and been influenced by other languages.



110.34 b 14(a) Students are expected to write an engaging story with a well-developed conflict and resolution, a clear theme, complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, suspense), devices to enhance the plot, and sensory details that define the mood or tone.




Note:ThecodesusedtoidentifyTEKSstatementsfollowedTEKS’scodingformat;forexample,110.31indicatesthestandardisEnglishI;“b”indicatesthestatementiswithinTEKSknowledgeandskills;“1”indicatesthestandardisthefirststandardwithinTEKSknowledgeandskills;and“(E)”indicatesthestandardstatementisathefifthstudentexpectationunderstandard1.


-


APPEnDIx f oThER sTAnDARDs-To-sTAnDARDs AlIgnmEnT sTuDy fInDIngs

Because there are no universal criteria for determining what levels of alignment are poor or good, it is difficult to interpret the results of the current study in isolation. Interpreting the results relative to those of similar research pro-vides meaningful context for policymakers and other readers of this report. The Rolfhus et al. (2010) study provides such a context because it is a standards-to-standards alignment study that applied the same rating scales and methodol-ogy in comparing three sets of college readiness standards in English language arts19 to a fourth benchmark set, the American Diploma Project (ADP). As in the current study, it evaluated align-ment on two dimensions: content and cognitive complexity.

This appendix presents the findings of the current study with those of Rolfhus et al. (2010). Note that while the Texas Essential Knowledge and Skills for English language arts and reading (TEKS ELAR) standards are a set of standards for grades 9–12 vertically aligned with the Texas College and Ca-reer Readiness Standards, the other four sets are college readiness standards.

Comparison of content alignment findings

Figure F1 presents the five pairwise comparisons of the current study (ACT–TEKS ELAR and ADP–TEKS ELAR) and of Rolfhus et al. (2010) (ADP–ACT, ADP–College Board, and ADP–Standards for Success)20 ordered by the percentage of fully aligned benchmark statements. Figure F2 pres-ents the same data ordered by the percentage of combined fully and partially aligned benchmark statements.

Of the five comparisons, the ADP–TEKS ELAR content alignment had the highest percentage of fully aligned benchmark statements (48 percent) and the highest percentage of combined fully and partially aligned benchmark statements (93

percent). These findings show that ADP is more closely aligned with TEKS ELAR than with the other three sets of national English language arts college readiness standards (Rolfhus et al. 2010). Of the five comparisons, the ACT–TEKS ELAR content alignment had the fourth highest percent-age of fully aligned benchmark statements (14 percent) and the second highest percentage of combined fully and partially aligned benchmark statements (89 percent).

Comparison of cognitive complexity findings

Figure F3 presents cognitive complexity findings for the five standards sets ordered by percentage of statements rated at the combined highest levels of cognitive complexity (3 and 4) on the Webb (2002) depth of knowledge (DoK) scale.

Of the five sets of standards, TEKS ELAR has the highest percentage of statements rated at cognitive complexity level 4 (14 percent) and at

figure f1

Alignment study findings ordered by percentage of fully aligned standards statements, 2009

0

25

50

75

100

ADP–ACT

ACT–TEKS ELAR

ADP–CB

ADP–S4S

ADP–TEKS ELAR

Percent Fully aligned Partially aligned Not aligned

Comparison alignment study

661123326

814

2731

48

26

75

50

37

45

Note: Percentages may not sum to 100 because of rounding. ADP = American Diploma Project; TEKS ELAR = Texas Essential Knowl-edge and Skills for English language arts and reading; S4S = Standards for Success; CB = College Board.

Source: Rolfhus et al. 2010; summary of reviewer ratings completed June–August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas Education Agency (2008).

aPPendix f. oTher STandardS-To-STandardS alignmenT STudy findingS 41

the highest aggregate cognitive complexity level (79 percent). At 72 percent, College Board has the second highest percentage of statements rated at levels 3 and 4 combined. ACT, at 56 percent, has lowest percentage of statements rated at levels 3 and 4 combined.

figure f2

Alignment study findings ordered by percentage of fully and partially aligned standards statements, 2009

0

25

50

75

100

ADP–ACT

ADP–S4S

ADP–CB

ACT–TEKS ELAR

ADP–TEKS ELAR

Percent Fully aligned Partially aligned Not aligned

Comparison alignment study

663223116

831

27

14

48

26

37

50

75

45

Note: Percentages may not sum to 100 because of rounding. ADP = American Diploma Project; TEKS ELAR = Texas Essential Knowl-edge and Skills for English language arts and reading; CB = College Board; S4S = Standards for Success.


figure f3

Alignment study findings ordered by percentage of standards statements rated at the combined highest levels of cognitive complexity (3 and 4) on the Webb depth of knowledge scale, 2009

0

25

50

75

100

ACTS4SADPCBTEKS ELAR

Percent

Standards set

Level 1–recall Level 2–skill/conceptLevel 3–strategic thinking Level 4–extended thinking

11213

4

14

55

5355

68

65 27

18

21

14312516234

Note: Percentages may not sum to 100 because of rounding. TEKS ELAR = Texas Essential Knowledge and Skills for English language arts and reading; CB = College Board; ADP = American Diploma Project; S4S = Standards for Success.



APPEnDIx g WEbb’s cognITIvE comPlExITy lEvEl DEscRIPTIons AnD ExAmPlE sTATEmEnTs

Thefollowingcognitivecomplexitylevelde-scriptionsforreadingandwriting(withtheexceptionofthetableswithexamples)aretakenverbatimfromWebb’s(2002,pp.1–3)Cognitive Complexity Criteria: Language Arts Levels for Depth of Knowledgeandusedforinitialtrain-ingofreviewers.Boththereadingandwritingscalesarebasedonthefourlevelsdescribedinthemainbodyofthisreport:level1–recall,level2–skill/concept,level3–strategicthinking,andlevel4–extendedthinking.Reviewersusedeitherthereadingorthewritingscaletoratecognitivecomplexitybasedonthecontentofthestate-mentbeingrated.Consensusmeetingsamongthereviewteamrefinedhowthislanguageandterminologywasinterpretedduringtheratingprocess.ExamplesofstatementsfromthefoursetsofcollegereadinessstandardsinthisstudythatreviewersratedateachdepthofknowledgelevelareshownintablesG1–G4.

Level1

Reading (Webb 2002, p. 1).Level1(recall)requiresstudentstoretrieveorrecitefactsortousesimpleskillsorabilities.Oralreadingthatdoesnotincludeanalysisofthetextaswellasbasiccom-prehensionofatextareincluded.Itemsrequireminimalunderstandingoftextandoftenconsist

ofverbatimrecallfromtextorsimpleunderstand-ingofasinglewordorphrase.Someexamplesthatrepresentbutdonotconstitutealloflevel1performanceare:

• Supportideasbyreferencetodetailsinthetext.

• Useadictionarytofindthemeaningofwords.

• Identifyfigurativelanguageinareadingpassage.

Writing (Webb 2002, p. 2).Level1(recall)requiresthestudenttowriteorrecitesimplefacts.Thiswritingorrecitationdoesnotincludecomplexsynthesisoranalysisbutbasicideas.Thestudentsareengagedinlistingideasorwordsasinabrain-stormingactivitypriortowrittencomposition,areengagedinasimplespellingorvocabularyassessment,orareaskedtowritesimplesentences.StudentsareexpectedtowriteandspeakusingstandardEnglishconventions.Thisincludesusingappropriategrammar,punctuation,capitalization,andspelling.Someexamplesthatrepresentbutdonotconstitutealloflevel1performancefollow(seealsotableG1):

• Usepunctuationmarkscorrectly.

• IdentifystandardEnglishgrammati-calstructuresandrefertoresourcesforcorrection.

Table g1

Examples of standards statements rated at cognitive complexity level 1, 2009

Standards set Statement identifier Statement

acT r-2 13-15-1 Supporting details: locate basic facts (e.g., names, dates, events) clearly stated in a passage

american diploma a1 demonstrate control of standard english through the use of grammar, Project punctuation, capitalization and spelling

Texas essential 110.34 b 18 oral and written conventions/handwriting, capitalization, and punctuation: Knowledge and Skills for Students write legibly and use appropriate capitalization and punctuation english language arts and in their compositions. Students are expected to correctly and consistently reading standards use conventions of punctuation and capitalization


43 aPPendix g. Webb’S cogniTive comPlexiT y level deScriPTionS and examPle STaTemenTS

Level2

Reading (Webb 2002, p. 1).Level2(skill/concept)includestheengagementofsomementalprocess-ingbeyondrecallingorreproducingaresponse;itrequiresbothcomprehensionandsubsequentprocessingoftextorportionsoftext.Inter-sentenceanalysisofinferenceisrequired.Someimportantconceptsarecoveredbutnotinacom-plexway.Standardsanditemsatthislevelmayincludewordssuchassummarize,interpret,infer,classify,organize,collect,display,compare,andthedistinctionbetweenfactandopinion.Literalmainideasarestressed.Alevel2assessmentitemmayrequirestudentstoapplysomeoftheskillsandconceptsthatarecoveredinlevel1.Someexamplesthatrepresentbutdonotconstitutealloflevel 2performanceare:

• Usecontextcuestoidentifythemeaningofunfamiliarwords.

• Predictalogicaloutcomebasedoninforma-tioninareadingselection.

• Identifyandsummarizethemajoreventsinanarrative.

Writing (Webb 2002, pp. 2–3).Level2(skill/con-cept)requiressomementalprocessing.Atthislevel,studentsareengagedinfirstdraftwritingorbriefextemporaneousspeakingforlimitedpurposesandaudiences.Studentsarebeginningto

connectideasusingasimpleorganizationalstruc-ture.Forexample,studentsmaybeengagedinnotetaking,outlining,orsimplesummaries.Textmaybelimitedtooneparagraph.Studentsdemon-strateabasicunderstandingandappropriateuseofsuchreferencematerialsasadictionary,thesau-rus,orwebsite.Someexamplesthatrepresentbutdonotconstitutealloflevel2performancefollow(seealsotableG2):

• Constructcompoundsentences.

• Usesimpleorganizationalstrategiestostruc-turewrittenwork.

• Writesummariesthatcontainthemainideaandpertinentideasofareadingselection.

Level3

Reading (Webb 2002, pp.1–3).Deepknowledgebecomesmoreofafocusatlevel3(strategicthinking).Studentsareencouragedtogobeyondthetext;however,theyarestillrequiredtoshowunderstandingoftheideasinthetext.Studentsmaybeencouragedtoexplain,generalize,orcon-nectideas.Standardsanditemsatlevel3involvereasoningandplanning.Studentsmustbeabletosupporttheirthinking.Itemsmayinvolveabstractthemeidentification,inferenceacrossanentirepassage,orstudents’applicationofpriorknowledge.Itemsmayalsoinvolvemoresuper-ficialconnectionsbetweentexts.Someexamples

Table g2



acT r-1 16-19-1 main ideas and author’s approach: identify a clear main idea or purpose of straightforward paragraphs in uncomplicated literary narratives

american diploma a3 use roots, affixes and cognates to determine the meaning of unfamiliar Project words

Texas essential 110.31 b 21(c) Paraphrase, summarize, quote, and accurately cite all researched Knowledge and Skills for information according to a standard format (e.g., author, title, page english language arts and number) reading standards



Table g3



acT W-4 03-4-1 organizing ideas: Provide a discernible organization with some logical grouping of ideas in parts of the essay

american diploma d3 make distinctions about the credibility, reliability, consistency, strengths Project and limitations of resources, including information gathered from Web

sites

Texas essential 110.31 b 24(a) listen responsively to a speaker by taking notes that summarize, Knowledge and Skills for synthesize, or highlight the speaker’s ideas for critical reflection and by english language arts and asking questions related to the content for clarification and elaboration reading standards


thatrepresentbutdonotconstitutealloflevel3performanceare:

• Determinetheauthor’spurposeanddescribehowitaffectstheinterpretationofareadingselection.

• Summarizeinformationfrommultiplesourcestoaddressaspecifictopic.

• Analyzeanddescribethecharacteristicsofvarioustypesofliterature.

Writing (Webb 2002, p.3).Level3(strategicthink-ing)requiressomehigherlevelmentalprocessing.Studentsareengagedindevelopingcompositionsthatincludemultipleparagraphs.Thesecom-positionsmayincludecomplexsentencesandmaydemonstratesomesynthesisandanalysis.Studentsshowawarenessoftheiraudienceandpurposethroughfocus,organization,andtheuseofappropriatecompositionalelements.Theuseofappropriatecompositionalelementsincludessuchthingsasaddressingchronologicalorderinanarrativeorincludingsupportingfactsanddetailsinaninformationalreport.Atthisstagestudentsareengagedineditingandrevisingtoimprovethequalityofthecomposition.Someexamplesthatrepresentbutdonotconstitutealloflevel3perfor-mancefollow(seealsotableG3):

• Supportideaswithdetailsandexamples.

• Usevoiceappropriatetothepurposeandaudience.

• Editwritingtoproducealogicalprogressionofideas.

Level4

Reading (Webb 2002, p. 2).Higherorderthinkingiscentralandknowledgeisdeepatlevel4(extendedthinking).Thestandardorassessmentitematthislevelwillprobablybeanextendedactivitywithextendedtimeprovided.Theextendedtimeperiodisnotadistinguishingfactoriftherequiredworkisonlyrepetitiveanddoesnotrequireap-plyingsignificantconceptualunderstandingandhigherorderthinking.Studentstakeinformationfromatleastonepassageandareaskedtoapplythisinformationtoanewtask.Theymayalsobeaskedtodevelophypothesesandperformcomplexanalysesoftheconnectionsamongtexts.Someexamplesthatrepresentbutdonotconstitutealloflevel4performanceare:

• Analyzeandsynthesizeinformationfrommultiplesources.

• Examineandexplainalternativeperspectivesacrossavarietyofsources.

• Describeandillustratehowcommonthemesarefoundacrosstextsfromdifferentcultures.

-

45 aPPendix g. Webb’S cogniTive comPlexiT y level deScriPTionS and examPle STaTemenTS

Table g4



acT W-2 11-12-2 focusing on the topic: Present a critical thesis that clearly establishes the focus on the writer’s position on the issue

american diploma e8 analyze two or more texts addressing the same topic to determine how Project authors reach similar or different conclusions

Texas essential 110.34 b 23(c) develop an argument that incorporates the complexities of and Knowledge and Skills for discrepancies in information from multiple sources and perspectives while english language arts and anticipating and refuting counter-arguments reading standards


Writing (Webb 2002, p. 3).Higherlevelthinkingiscentraltolevel4(extendedthinking).Thestan-dardatthislevelisamulti-paragraphcomposi-tionthatdemonstratessynthesisandanalysisofcomplexideasorthemes.Evidentisadeepawarenessofpurposeandaudience.Forexample,informationalpapersincludehypothesesandsupportingevidence.Studentsareexpectedtocreatecompositionsthatdemonstrateadistinct

voiceandthatstimulatethereaderorlistenertoconsidernewperspectivesontheaddressedideasandthemes.Anexamplethatrepresentsbutdoesnotconstitutealloflevel4performanceis(alsoseetableG4):

• Writeananalysisoftwoselections,identify-ingthecommonthemeandgeneratingapurposethatisappropriateforboth.


APPEnDIx h cognITIvE comPlExITy by sTRAnD

ThisappendixpresentsthefindingsoncognitivecomplexitybystrandforACTCollegeReadi-nessStandards(ACT,Inc.2007),theAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBenchmarks(Achieve,Inc.2004),andtheTexasEssentialKnowledgeandSkillsforEng-lishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008).

ACTcognitivecomplexity

TheACTstrandsvaryinthedistributionofstate-mentsacrossthefourlevelsofcognitivecomplex-ity.Level3–strategicthinkingiswellrepresentedinACTandisthemostrepresentedlevelwithintheEnglishandwritingstrands;themajorityofthereadingstrandisrepresentedbylevel2–skill/concept(figureH1).Comparedwiththeothersetsofstandards,ACTdisplaysarelativelyhighper-centageofstatementsratedatlevel1–recall,rang-ingfrom8percentto34percentacrossstrands.ACTstrandsexhibitarelativelylowpercentage

figure h1

Percentage of AcT standards statements at each level of cognitive complexity, by strand, 2009


Level3–strategicthinking Level4–extendedthinking

7775

ReadingWritingEnglish

62

55

50

3429

25

1311 9820 00

ACTstrand


Source: SummaryofreviewerratingscompletedMay–September2008(Rolfhuset al.2010)drawingonstandardsstatementsinACT,Inc.(2007).

ofstatementsratedatlevel4–extendedthinking,rangingfrom0percentto2percentacrossstrands.OnereasonforthismaybethatthewordingoftheACTstrandsisverydetailedinordertofacilitatethedevelopmentofACTtestitems.Becauseofthisdetail,itmaybedifficulttoassesssomeofthemoreabstractconstructsdescribedunderlevel4–extendingthinking,resultinginthelowestpercentageoflevel4cognitivecomplexityratingsonthedepthofknowledgescaleamongthethreestandardssets.

AmericanDiplomaProjectcognitivecomplexity

Variationincognitivecomplexitywasalsoob-servedacrosstheeightADPstrands(figureH2).Morethanaquarterofthecontentinsevenoftheeightstrands(theexceptionbeinglanguage)wasatcognitivecomplexitylevel3–strategicthinking.Butthedistributionsoftheotherthreecomplexitylevelsdiffergreatlyacrossstrands.Forexample,level1–recallisrepresentedonlyinthelanguagestrand(14percent).Level2–skill/conceptisnotrepresentedbyeitherliteratureormediastrands,buthas71percentrepresentationinlanguage.Finally,statementsatthehighestlevelofcognitivecomplexityareabsentfromboththelanguageandwritingstrands;thegreatestrepresentationsoflevel4–extendedthinkingaredisplayedincommunication(29percent)andmedia(25percent).

TexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandardsstatementscognitivecomplexity

IneachoftheTEKSELARstrands,level3–strategicthinkingisrepresentedatthehighestpercentageofallfourWebblevels,withthehighestratesinthereading(84percent)andlisteningandspeaking(89percent)strands.Level1–recallisrepresentedatahighlevel(40percent)onlyintheoralandwrittenconventionstrand.Level2–skill/conceptappearsatthehighestpercentage(46percent)intheresearchstrand.Level4–extendedthinkingappearsatahighpercentage(42percent)onlywithinthewritingstrand.

47 aPPendix h. cogniTive comPlexiT y by STrand

figure h2

Percentage of America Diploma Project standards statements at each level of cognitive complexity by strand, 2009

Percent 100 Level 1–recall Level 2–skill/concept Level 3–strategic thinking Level 4–extended thinking

89

80

75 7175

55

50 43 45

404036

3329 29

25 20 20 2225

14 1411 9

0 0 0 0 0 0 0 0 0 0 0 0

Language Writing Literature Communication Research Logic Informational text Media

ADP strand


Source:SummaryofreviewerratingscompletedMay–September2008(Rolfhuset al.2010)drawingonstandardsstatementsinAchieve,Inc.(2004).

figure h3

Percentage of Texas Essential Knowledge and skills for English language arts and reading standards statements at each level of cognitive complexity by strand, 2009


Level3–strategicthinking Level4–extendedthinking84 89

75

5047 46

42 40 40

50

25 20

11117 5 43

0 0 0 0 00Reading Writing Oralandwritten Research Listeningand

conventions speaking

TEKSELARstrand


Source:SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinTexasEducationAgency(2008).


APPEnDIx I cognITIvE comPlExITy comPARIson foR fully AnD PARTIAlly AlIgnED sTATEmEnTs

Inadditiontothecognitivecomplexityanaly-sisconductedinresponsetoresearchquestion2,anotheranalysiswasconductedcomparingcognitivecomplexityratingsforthestatementsinthebenchmarkACTCollegeReadinessStandards(ACT,Inc.2007)andAmericanDiplomaProject(ADP)CollegeandWorkplaceReadinessBench-marks(Achieve,Inc.2004)standardssetsandtheTexasEssentialKnowledgeandSkillsforEnglishlanguageartsandreadingstandards(TEKSELAR;TexasEducationAgency2008)setsthatcontainalignedcontent.

Foreaseofreference,thisappendixusesthetermsACT–TEKS ELAR aligned statementsandADP–TEKS ELAR aligned statementstorefertothestatementsthatmakeupafullyorpartiallyalignedrelationship(theonefullyorpartiallyalignedACTorADPstatementandtheoneormoreTEKSELARstatementsthatfullyorpartiallyaligntothatACTorADPstatement).Whenmul-tipleTEKSELARstatementscombinetofullyorpartiallyaligntoasingleACTorADPstatement,eachindividualcomparisonstatementcontributedtothecalculationsreportedinthissection.Forex-ample,ifasingleACTstatementwasalignedfullywiththecumulativecontentoffourTEKSELARstatements,fourcomparisonswereconducted(thecognitivecomplexityleveloftheACTstatementwasindividuallycomparedwiththecognitivecomplexitylevelofeachofthefouralignedTEKSELARstatements).Becauseeachofthe278TEKSELARstatementscouldaligntomorethanonestatementineachbenchmarkset,therearemoreinstancesofTEKSELARstatementsalignedtobenchmarkstatementsthanthereareactualTEKSELARstatements.

ACTfindings

Cognitivecomplexityfindingsarepresentedforthe170(of191)ACTstatementsand174(of278)TEKSELARstatementsthatcontainfullyor

partiallyalignedcontent.Thecontentalignmentfoundthat27ofACT’s191statementsalignfullywithTEKSELARstatements,with227instancesinwhichTEKSELARstatementscontributedtothesefullalignments.Thecontentalignmentalsofoundthat143ofACT’s191statementsalignpartiallywithTEKSELARstatements,with777instancesinwhichTEKSELARstatementscontributedtothesepartialalignments.Cogni-tivecomplexitycomparisonswereconductedforeachinstanceofACT–TEKSELARfullyalignedstatementsandACT–TEKSELARpartiallyalignedstatements.TablesI1andI2presentthetotalnumberofTEKSELARalignedstatementsperACTstrand.

Cognitive complexity results for ACT–TEKS ELAR fully aligned statements.OverallresultsforthecognitivecomplexityanalysisoftheACT–TEKSELARfullyalignedstatementsarepresentedinfigureI1;detailedresultsarepresentedintableI1.

figure i1

overall cognitive complexity comparison findings for AcT–Texas Essential Knowledge and skills for English language arts and reading standards fully aligned statements, 2009

0

25

50

75

100

All ACT strands

Writing ReadingEnglish

Percent Above ACT level At ACT level Below ACT level

ACT strands

3

42

45

13

37

47

16

10043

53

Note: ThenumberofstatementsateachlevelofcognitivecomplexitybystrandisshownintableI1.Percentagesmaynotsumto100becauseofrounding.


Table i1

Detailed cognitive complexity comparison findings for AcT–Texas Essential Knowledge and skills for English language arts and reading standards fully aligned statements, 2009

Total number of fully fully aligned TeKS elar statements by acT aligned statements level of cognitive complexity

number above acT level at acT level below acT level of state TeKS

Strand and substrand ments acT elar number Percent number Percent number Percent

English 71 6 30 13 43 16 53 1 3

e-1: Topic development 11 4 17 8 47 9 53 0 0

e-2: organization 12 1 8 5 63 3 38 0 0

e-3: Word choice 13 0 0 0 0 0 0 0 0

e-4: Sentence structure 10 1 5 0 0 4 80 1 20

e-5: conventions of usage 11 0 0 0 0 0 0 0 0

e-6: conventions of 14 0 0 0 0 0 0 0 0 punctuation

Reading 58 3 14 14 100 0 0 0 0

r-1: main ideas 12 2 6 6 100 0 0 0 0

r-2: Supporting details 12 1 8 8 100 0 0 0 0

r-3: Sequential, 18 0 0 0 0 0 0 0 0 comparative, and cause-and-effect relationships

r-4: meanings of words 7 0 0 0 0 0 0 0 0

r-5: generalizations and 9 0 0 0 0 0 0 0 0 conclusions

Writing 62 18 183 68 37 86 47 29 16

W-1: expressing judgment 14 0 0 0 0 0 0 0 0

W-2: focusing on topic 8 3 23 8 35 10 43 5 22

W-3: developing a position 10 3 60 21 35 39 65 0 0

W-4: organizing ideas 15 9 84 36 43 24 29 24 29

W-5: using language 15 3 16 3 19 13 81 0 0

all strands and substrands 191 27 227 95 42 102 45 30 13

Note:BecauseanindividualTEKSELARstatementmayhavealignedtomultiplestatementsinthebenchmarkset,thetotalnumberofalignedTEKSELARstatementsvaries.Percentagesmaynotsumto100becauseofrounding.


49 aPPendix i. cogniTive comPlexiT y comPariSon for fully and ParTially aligned STaTemenTS

Acrossallstrands,forfullyalignedACT–TEKSELARstatements,theTEKSELARcognitivecomplexitylevelisabovetheleveloftheassociatedACTstatementin42percentoftheinstancesoffullyalignedstatements,atthelevelin45percentoftheinstancesoffullyalignedstatements,andbelowthelevelin13percentoftheinstancesoffullyalignedstatements.Theoneexceptionisthereadingstrand,wheretheTEKSELARcognitivecomplexitylevelisabovethecognitivecomplexity

leveloftheassociatedACTstatementin100 per-centoftheinstancesofACT–TEKSELARfullyalignedstatements.

Cognitive complexity results for ACT–TEKS ELAR partially aligned statements.Overallresultsforcog-nitivecomplexityanalysisoftheACT–TEKSELARpartiallyalignedstatementsarepresentedinfigureI2;detailedresultsarepresentedintable I2.AcrossallACTstrands,in49percentoftheinstancesof


ACT–TEKSELARpartiallyalignedstatements,thecognitivecomplexityleveloftheTEKSELARstatementwasabovethatoftheassociatedACTstatement;in40percentoftheinstancesofACT–TEKSELARpartiallyalignedstatements,theTEKSELARstatementwasatthecognitivecomplexityleveloftheassociatedACTstatement;andin11percentoftheinstancesofACT–TEKSELARpar-tiallyalignedstatements,thecognitivecomplexityleveloftheTEKSELARstatementwasbelowthatoftheACTstatement.AswiththefullyalignedACT–TEKSELARstatements,thehighestpercent-ageofTEKSELARstatementsabovethecognitivecomplexityleveloftheassociatedACTstatementsisfoundinthereadingstrand(70percent).

AmericanDiplomaProjectfindings

Cognitivecomplexityfindingsarepresentedforthe58(of62)ADPstatementsand255(of278)TEKSELARstatementsthatcontainfullyorpartiallyalignedcontent.Again,asingleTEKSELARstatementcouldaligntoseveralADPstatements.

figure i2

overall cognitive complexity comparison findings for AcT–Texas Essential Knowledge and skills for English language arts and reading standards partially aligned statements, 2009 Percent Above ACT level At ACT level Below ACT level

100

75

50

25

0 English Reading Writing All ACT

strands

11

40

17

39

30

25 494470

57

19

ACT strands

Note: ThenumberofstatementsateachlevelofcognitivecomplexitybystrandisprovidedintableI2.Percentagesmaynotsumto100becauseofrounding.


Therefore,theinstancesofTEKSELARalignedstatementsinmanycasesexceedthenumberoforiginalTEKSELARstatements.AswiththeTEKSELARandACTcognitivecomplexityanalyses,fullyandpartiallyalignedresultsarepresentedseparately.Resultsofthecontentalignmentindicatethat30ofADP’s62statementsalignedfullywithTEKSELARstatements;therewere232instancesinwhichTEKSELARstatementscontributedtothesefullalignments.Inaddition,thecontentalignmentindicatesthat28ofADP’s62statementsalignedpartiallywithTEKSELARstatements;therewere176instancesinwhichTEKSELARstatementscontributedtothesepartialalignments.CognitivecomplexitycomparisonswereconductedforeachinstanceofADP–TEKSELARfullyalignedstatementsandADP–TEKSELARpartiallyalignedstatements.ValuesintablesI3andI4presentthetotalnumberofinstancesofTEKSELARalignedstatementsperADPstrand.

Cognitive complexity results for ADP–TEKS ELAR fully aligned statements.Overallresultsforthecog-nitivecomplexityanalysisofADP–TEKSELARfullyalignedstatementsareinfigureI3;detailedresultsareintableI3.Acrossallstrands,in24percentoftheinstancesofADP–TEKSELARfullyalignedstate-ments,theTEKSELARcognitivecomplexitylevelisabovetheleveloftheassociatedADPstatement;in59percentoftheinstancesofADP–TEKSELARfullyalignedstatements,theTEKSELARcogni-tivecomplexitylevelisattheleveloftheassociatedADPstatement,andin16percentoftheinstancesofADP–TEKSELARfullyalignedstatements,theTEKSELARstatementisbelowthecognitivecom-plexityleveloftheassociatedADPstatement.

CognitivecomplexityfindingsvaryacrossADPstrands.Forexample,inlanguagestrandcom-parisons,62percentoftheTEKSELARstatementswereabovethecognitivecomplexityleveloftheassociatedADPstatements,while70percentoflogicstrandcomparisonsfoundtheTEKSELARstatementsbelowthecognitivecomplexityleveloftheassociatedADPstatements.NotethatnovaluesarereportedforthemediastrandbecausenoADPmediastatementsfullyalignwithTEKSELARstatements.

Table i2

Detailed cognitive complexity comparison findings for AcT–Texas Essential Knowledge and skills for English language arts and reading standards partially aligned statements, 2009

Total number of partially aligned Partially aligned TeKS elar statements

acT statements by level of cognitive complexity

number above acT level at acT level below acT level of state TeKS

Strand and substrand ments acT elar number Percent number Percent number Percent

English 71 63 216 53 25 123 57 40 19

e-1: Topic development 11 5 19 4 21 15 79 0 0

e-2: organization 12 11 56 20 36 19 34 17 30

e-3: Word choice 13 13 28 1 4 23 82 4 14

e-4: Sentence structure 10 9 43 10 23 16 37 17 40

e-5: conventions of usage 11 11 21 11 52 8 38 2 10

e-6: conventions of 14 14 49 7 14 42 86 0 0 punctuation

Reading 58 46 291 204 70 87 30 0 0

r-1: main ideas 12 9 68 68 100 0 0 0 0

r-2: Supporting details 12 11 73 73 100 0 0 0 0

r-3: Sequential, 18 11 81 43 53 38 47 0 0 comparative, and cause-and-effect relationships

r-4: meanings of words 7 6 15 13 87 2 13 0 0

r-5: generalizations and 9 9 54 7 13 47 87 0 0 conclusions

Writing 62 34 270 120 44 104 39 46 17

W-1: expressing judgment 14 12 63 24 38 39 62 0 0

W-2: focusing on topic 8 5 70 20 29 25 36 25 36

W-3: developing a position 10 3 38 21 55 17 45 0 0

W-4: organizing ideas 15 6 56 24 43 16 29 16 29

W-5: using language 15 8 43 31 72 7 16 5 12

all strands and substrands 191 143 777 377 49 314 40 86 11



51 aPPendix i. cogniTive comPlexiT y comPariSon for fully and ParTially aligned STaTemenTS

Cognitive complexity results for ADP –TEKS ELAR partially aligned statements.OverallresultsforthecognitivecomplexityanalysisofADP–TEKSELARpartiallyalignedstatementsarepresentedinfigureI4;detailedresultsarepresentedintableI4.Acrossallstrands,in22percentoftheinstancesofADP–TEKSELARpartiallyalignedstatements,thecognitivecomplexityleveloftheTEKSELARstatementsisabovethatoftheassoci-atedADPstatement;in73percentoftheinstancesofADP–TEKSELARpartiallyalignedstatements,theTEKSELARstatementisatthecognitive

complexityleveloftheassociatedADPstatement;andin6 percentoftheinstancesofADP–TEKSELARpartiallyalignedstatements,thecognitivecomplexityleveloftheTEKSELARstatementisbelowthatoftheassociatedADPstatement.

WhilefindingsvarysomewhatacrossADPstrand,ineverystrandthecognitivecomplexityleveloftheTEKSELARstatementisatthatoftheassociatedADPstatementinatleast56 per-centoftheinstances.Comparisonsforfiveoftheeightstrands(language,logic,informational


0

25

50

75

100

All ADP strandsLiteratureMediaaInformational textLogicResearchWritingCommunicationLanguage

Percent

ADP strand

16

6

50

50

70

50

4

40

12

Above ADP level At ADP level Below ADP level

59

84

2644

69

50

27

241046271062

figure i3

overall cognitive complexity comparison findings for American Diploma Project–Texas Essential Knowledge and skills for English language arts and reading standards fully aligned statements, 2009

Note: ThenumberofstatementsateachlevelofcognitivecomplexitybystrandisprovidedintableH3.

a.NoADPmediastatementsfullyalignwithTEKSELARstatements.


Table i3

Detailed cognitive complexity comparison findings for American Diploma Project–Texas Essential Knowledge and skills for English language arts and reading standards fully aligned statements, 2009

Total number fully fully aligned TeKS elar statements by adP aligned statements level of cognitive complexity

above adP level at adP level below adP level number of state TeKS

Strand and substrand ments adP elar number Percent number Percent number Percent

a. language 7 4 26 16 62 7 27 3 12

b. communication 7 4 10 1 10 5 50 4 40

c. Writing 10 5 55 15 27 38 69 2 4

d. research 5 2 18 1 6 8 44 9 50

e. logic 9 3 23 1 4 6 26 16 70

f. informational text 11 6 30 15 50 15 50 0 0

g. media 4 0 0 0 0 0 0 0 0

h. literature 9 6 70 7 10 59 84 4 6

all strands 62 30 232 56 24 138 59 38 16


Source: SummaryofreviewerratingscompletedJune–August2009drawingonstandardsstatementsinAchieve,Inc.(2004)andTexasEducationAgency(2008).

text,media,andliterature)foundthatallTEKSELARstatementsareaboveoratthecognitivecomplexityleveloftheirassociatedADPstatements.

Summary

Thefindingspresentedabovedemonstratethat,ingeneral,theTEKSELARstatementsarewrittenat

-

5aPPendix i. cogniTive comPlexiT y comPariSon for fully and ParTially aligned STaTemenTS 3

0

25

50

75

100

All ADP strandsLiteratureMediaInformational textLogicResearchWritingCommunicationLanguage

Percent

ADP strand

Above ADP level At ADP level Below ADP level

7 100 22644151279

21

25

93

73

6

94

56

69

15

74

15

75

figure i4

overall cognitive complexity comparison findings for American Diploma Project–Texas Essential Knowledge and skills for English language arts and reading standards partially aligned statements, 2009

Note: ThenumberofstatementsateachlevelofcognitivecomplexitybystrandisprovidedintableI4.Percentagesmaynotsumto100becauseofrounding.


Table i4

Detailed cognitive complexity comparison findings for American Diploma Project–Texas Essential Knowledge and skills for English language arts and reading standards partially aligned statements, 2009

Total number partially Partially aligned TeKS elar statements by adP aligned statements level of cognitive complexity

number above adP level at adP level below adP level of state TeKS

Strand and substrand ments adP elar number Percent number Percent number Percent

a. language 7 3 12 3 25 9 75 0 0

b. communication 7 2 14 0 0 11 79 3 21

c. Writing 10 5 34 4 12 25 74 5 15

d. research 5 3 13 2 15 9 69 2 15

e. logic 9 5 61 27 44 34 56 0 0

f. informational text 11 5 14 1 7 13 93 0 0

g. media 4 3 11 0 0 11 100 0 0

h. literature 9 2 17 1 6 16 94 0 0

all strands 62 28 176 38 22 128 73 10 6



thesameorahigherlevelofcognitivecomplexity TEKSELARstatementsthatcontributedtofullthantheACTorADPbenchmarkstatementsto alignmentsaswellasforthosecontributingtowhichtheyalign.Thisfindingwasconsistentfor partialalignments.


noTEs

1. Contentstandardsdefinetheknowledgeandskillsstudentsshouldhaveinspecificcontentdomainsastheyprogressfromkindergartenthroughgrade12.

2. Hamilton,Stecher,andYuan(2008,p.11)identifysixcriticalaspectsofstandards-basedreform:academicexpectationsforstudents,alignmentofthekeyelementsoftheeducationsystemtopromoteattainmentoftheseexpectations,assessmentsofstudentachievementtomeasureoutcomes,decentralizationtoschoolsofresponsibilityfordecisionsoncurriculumandinstruction,stateanddistrictsupportandtechnicalassistancetofosterimprovementofeducationservices,andrewardsorsanctionsofschoolsorstudentsbasedonmeasuredperformance(accountabilityprovisions).Thefirsttwoaspectsaremostdirectlyrelevanttothecurrentstudy.

3. VerticalalignmenthasbeenanimportantcriterioninjudgingthequalityofK–12standards(forexample,Stotsky2005).

4. Terry’s(2007)estimates,derivedfromenrollmentdataprovidedbytheTexasHigherEducationCoordinatingBoard,werecalculatedbydividingthenumberofstudentsenrolledinremedialcourseworkin2006(inbothtwo-andfour-yearinstitutions)bytotalenrollmentineachtypeofinstitution.TheProvasnikandPlanty(2008)estimateswerederivedfromanationallyrepresentativesamplesurveythataskedfirst-yearpostsecondarystudentstoself-reportwhethertheywereenrolledinremedialcourses(Cominoleet al.2007).

5. AmajorprogramforfundingtheAmericanReinvestmentandRecoveryActeducationagendaistheRacetotheTopFundcompetition,whichgivesprioritytostatesthathaveparticipatedintheCommonCoreStateStandardsInitiativeandagreedtoadopt

thosestandardswhentheyarefinalized(U.S.DepartmentofEducation2009).

6. Rolfhuset al.(2010)involvedthreepairwisecomparisonsofnationalcollegereadinessstandardssetsinEnglishlanguagearts—ACTCollegeReadinessStandards(ACT,Inc.2007),CollegeBoard(CollegeBoard2006),andStandardsForSuccess(Conley2003)—toafourthset,AmericanDiplomaProject(Achieve,Inc.2004)thatwasdesignatedthebenchmark.SeeappendixFforadditionalinformationaboutthefindingsfromthisstudy.

7. ThestudyaddressesonlytheEnglishlanguageartsandreadingdomain,asmathematicsalignmentinformationhasalreadybeenprovidedtotheTexasEducationAgencyattheagency’srequest.

8. EnglishI–IVaretheonlyEnglishlanguageartsandreadingcoursesrequiredforgraduation;soelectiveELARcoursesofferedingrades9–12arenotincludedinthisalignmentstudy.

9. WebbDoKlevel1–recallrequiresstudentstousesimpleskillsorabilitiestoretrieveorrecitefacts.Afulldescriptionofallfourlevelsisprovidedinthenextsection.

10. Inadditiontothecognitivecomplexityanalysisconductedinresponsetothesecondresearchquestion,asecondaryanalysiscomparingthecognitivecomplexitylevelsofstate-mentswithalignedcontentwasconducted.SeeappendixIforthedetailsandresultsofthistechnicalanalysis.

11. TheTimmsetal.(2007a–e)studiesalignedthesciencedomainsofthe2009NAEPassessmentstandardsanditemspecificationswithstateK–12assessmentstandards;theShapleyandBritestudies(2008a–e)alignedthemathematicsdomainsofthosesamesetsofassessmentstandardsanditemspecifications.

-

-

-

--

-

-

--

-

-

-

12. ThecurrentstudydidnotuseadditionalcodingemployedintheNAEPstudies;codesrepresentinghigher/lowergradealignmentarenotrelevantforcollegereadinessstandards,whichhaveonlyonelevel.

13. Inthisreport,“alignedwith”isusedwhenreferringtotheextentofthecontentalignmentbetweenthebenchmarkstandardssets(ACTandADP)andtheTEKSELARcomparisonset.“Alignedto”isusedwhenreferringtohowtheTEKSELARstandardsmaptothebenchmarksetofstandards(eitherACTorADP).

14. RatingactivitiesfortheRolfhuset al.(2010)studytookplaceduringMay–September2008;ratingactivitiesforthecurrentprojecttookplaceduringJune–August2009.

15. EducationalTraining,Evaluation,Assessment,andMeasurement(E-TEAM);CollegeofContinuingEducation,UniversityofOklahoma.

16. TheretrainingsessionwasintendedtoensurethatratingprocessfortheTEKSELAR

--

-

-

--

-

noTeS 55

statementswasconsistentwiththatofRolfhuset al.(2010).

17. Systematicallyheremeansthateachstan-dardstatementwasreadandevaluatedforacontentmatch,inthesameorderastheyappearedinthesourcedocument.Thiswasanexhaustivesearch,sothatevenifafullyalignedcontentmatchtothebenchmarkwasfoundimmediately,allremainingstandardsstatementsinthecomparisonsetwerealsoreadandevaluated.

18. RatingactivitiesforthepreviousprojectwerecompletedinSeptember2008;ratingactivitiesforthecurrentprojectoccurredinJune–August2009.AcompleteretrainingsessionwasheldtoensurethattheratingoftheTEKSELARstatementswasconsistentwiththatoftheACTandADPstandards.

19. ACTCollegeReadinessStandards(ACT);Col-legeBoard(CB);andStandardsforSuccess(S4S)

20. Thesetofstandardsusedasthebenchmarkineachpairwisecomparisonisidentifiedfirst.

http://eteam.ou.edu/

http://eteam.ou.edu/

http://www.occe.ou.edu/

http://www.occe.ou.edu/

http://www.ou.edu

http://www.ou.edu


REfEREncEs

Achieve,Inc.(n.d.).State alignment services.RetrievedApril3,2009,fromhttp://www.achieve.org/node/323

Achieve,Inc.(2004).Ready or not: creating a high school diploma that counts, the American Diploma Project. RetrievedMay4,2008,fromhttp://www.achieve.org/files/ADPreport_7.pdf

Achieve,Inc.(2009a).ADP network. RetrievedDecember9,2009,fromhttp://www.achieve.org/ADPNetwork

Achieve,Inc.(2009b).ADP assessment consortium.Re-trievedDecember9,2009,fromhttp://www.achieve.org/ADPAssessmentConsortium

ACT,Inc.(2007).ACT’s college readiness standards and college readiness benchmarks: helping to prepare every student for college and work.IowaCity,IA:ACT,Inc.

AmericanAssociationfortheAdvancementofScience.(2009).Project 2061. RetrievedNovember18,2009,fromhttp://www.project2061.org/

AmericanRecoveryandReinvestmentandActof2009,H.R.1,111Cong.(2009).

Blank,R.K.(2002).Usingsurveysofenactedcurriculumtoevaluatequalityofinstructionandalignmentwithstandards.Peabody Journal of Education,77(4),86–121.

TheCollegeBoard.(2006).College Board College Readiness Standards for College Success™ English Language Arts.TheCollegeBoard:TheCollegeBoard.

Cominole,M.,Wheeless,S.,Dudley,K.,Franklin,J.,andWine,J.(2007).2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06) Field Test Methodology Report [workingpaperseries].Wash-ington,D.C.:U.S.DepartmentofEducation,InstituteforEducationSciences,NationalCenterforEducationStatistics.RetrievedFebruary12,2010,fromhttp://nces.ed.gov/pubs2006/200601.pdf

CommonCoreStateStandardsInitiative.(2009).Common Core State Standards initiative frequently asked

questions.RetrievedDecember16,2009,fromhttp://www.corestandards.org/FAQ.htm

Conley,D.T.(2003).Understanding university success: a report from Standards for Success. Eugene,OR:Univer-sityofOregon,CenterforEducationalPolicyResearch.RetrievedFebruary11,2008,fromhttp://www.s4s.org/3_UUS_English.pdf

Cook,H.G.(2005).Milwaukee Public Schools alignment study of Milwaukee Public Schools’ learning targets in reading and math to Wisconsin Student Assessment System criterionreferenced test frameworks in reading and math (ResearchReport#0504).Milwaukee,WI:MilwaukeePublicSchoolsOfficeofAssessmentandAccountability.

Hamilton,L.S.,Stecher,B.M.,andYuan,K.(2008).Standardsbased reform in the United States: history, research, and future directions. Washington,DC:RANDCorporation.

ImprovingAmerica’sSchoolsActof1994.(1995).PublicLaw103–382.

LaMarca,P.M.(2001).Alignmentofstandardsandas-sessmentsasanaccountabilitycriterion.Practical Assessment, Research & Evaluation,7(21).RetrievedJuly20,2008,fromhttp://PAREonline.net/getvn.asp?v=7&n=21

MissouriDepartmentofElementaryandSecondaryEducation.(2009,August7).State Board of Education agrees to join “Common Core State Standards” project. RetrievedDecember19,2009,fromhttp://dese.mo.gov/news/2009/corestandards.htm

Näsström,G.,andHenriksson,W.(2008).Alignmentofstandardsandassessment:atheoreticalandem-piricalstudyofmethodsforalignment.Electronic Journal of Research in Educational Psychology, 16(6),667–690.RetrievedAugust1,2009,fromhttp://www.investigacion-psicopedagogica.org/revista/articulos/16/english/Art_16_216.pdf

NationalCommissiononExcellenceinEducation.(1983).A nation at risk: the imperative for educational reform.Washington,DC:U.S.DepartmentofEducation.

http://www.achieve.org/node/323

http://www.achieve.org/files/ADPreport_7.pdf

http://www.achieve.org/files/ADPreport_7.pdf

http://www.achieve.org/ADPNetwork

http://www.achieve.org/ADPAssessmentConsortium

http://www.achieve.org/ADPAssessmentConsortium

http://www.project2061.org/

http://nces.ed.gov/pubs2006/200601.pdf

http://nces.ed.gov/pubs2006/200601.pdf

http://www.corestandards.org/FAQ.htm

http://www.corestandards.org/FAQ.htm

http://www.s4s.org/3_UUS_English.pdf

http://www.s4s.org/3_UUS_English.pdf

http://PAREonline.net/getvn.asp?v=7&n=21

http://PAREonline.net/getvn.asp?v=7&n=21

http://dese.mo.gov/news/2009/corestandards.htm

http://dese.mo.gov/news/2009/corestandards.htm

http://www.investigacion-psicopedagogica.org/revista/articulos/16/english/Art_16_216.pdf



NationalGovernorsAssociation(2009).Fortynine states and territories join common core standards initiative.RetrievedDecember4,2009,fromwww.nga.org

NoChildLeftBehindActof2001.(2002).PublicLaw107–110.

Porter,A.C.(2002).Measuringthecontentofinstruction:usesinresearchandpractice.Educational Researcher. 31(7),3–14.

Porter,A.C.,Smithson,J.,Blank,R.,andZeidner,T.(2007).Alignmentasateachervariable.Applied Measurement in Education, 20(1),27–51.

Provasnik,S.,andPlanty,M.(2008).Community colleges: special supplement to The Condition of Education 2008 (NCES2008-033).Washington,DC:NationalCenterforEducationStatistics,InstituteofEducationSciences,U.S.DepartmentofEducation.

Rolfhus,E.,Decker,L.E.,Brite,J.L.,andGregory,L.(2010).A systematic comparison of the American Diploma Project English language arts college readiness standards with those of the ACT, College Board, and Standards for Success(Issues&AnswersReport,REL2010–No.086).Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssis-tance,RegionalEducationalLaboratorySouthwest.

Rothman,R.(2004).Imperfect matches: the alignment of standards and tests. Washington,DC:NationalAcad-emyofSciences.

Rothman,R.,J.B.Slattery,J.L.Vranek,andL.B.Resnick.(2002).Benchmarking and alignment of standards and testing.CSETechnicalReport566.LosAngeles,CA:UniversityofCalifornia,LosAngeles,GraduateSchoolofEducationandInformationScience,NationalCenterforResearchonEvaluation,Standards,andStudentTesting.

Shapley,K.,andBrite,J.(2008a).Aligning mathematics assessment standards: Arkansas and the 2009 National Assessment of Educational Progress (NAEP).(RELTechnicalBrief,REL2008–No.08).Washington,DC:

noTeS 57

U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.RetrievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs

Shapley,K.,andBrite,J.(2008b).Aligning mathematics assessment standards: Louisiana and the 2009 National Assessment of Educational Progress (NAEP).(RELTechnicalBrief,REL2008–No.09).Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.RetrievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs

Shapley,K.,andBrite,J.(2008c).Aligning mathematics assessment standards: New Mexico and the 2009 National Assessment of Educational Progress (NAEP).(RELTechnicalBrief,REL2008–No.11).Washington,DC:U.S.DepartmentofEducation,InstituteofEduca-tionSciences,NationalCenterforEducationEvalua-tionandRegionalAssistance,RegionalEducationalLaboratorySouthwest.RetrievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs

Shapley,K.,andBrite,J.(2008d).Aligning mathematics assessment standards: Oklahoma and the 2009 National Assessment of Educational Progress (NAEP).(RELTechnicalBrief,REL2008–No.10).Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.Re-trievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs

Shapley,K.,andBrite,J.(2008e).Aligning mathematics assessment standards: Texas and the 2009 National Assessment of Educational Progress (NAEP).(RELTechnicalBrief,REL2008–No.07).Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.Re-trievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs.

http://www.nga.org

http://ies.ed.gov/ncee/edlabs










Shrout,P.E.,andFleiss,J.L.(1979).Intraclasscorrelations:usesinassessingraterreliability.Psychological Bulletin,86(2),420–428.

SouthCarolinaDepartmentofEducation.(2009,Sep-tember 2).South Carolina becomes 48th state to join national Common Core Standards project. RetrievedDecember6,2009,fromhttp://www.ed.sc.gov/news/more.cfm?articleID=1301

SPSS,Inc.(2007).SPSS for Windows, Rel. 16.1.Chicago:SPSS,Inc.

Stern,L.,andAhlgren,A.(2002).Analysisofstudents’assessmentsinmiddleschoolcurriculummaterials:aimingpreciselyatbenchmarksandstandards.Journal of Research in Science Teaching,39(9),889–910.

Stotsky,S.(2005).The state of state English standards. Washington,DC:FordhamFoundation.RetrievedFeb-ruary11,2010,fromhttp://eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/1b/ab/31.pdf

Terry,B.D.(2007).The cost of remedial education.CenterforEducationPolicy:Austin,TX.RetrievedMay6,2008,fromhttp://www.texaspolicy.com/pdf/2007-09-PP25-remediation-bt.pdf

TexasEducationAgency.(1997).Texas Essential Knowledge and Skills.Austin,TX:TexasEducationAgency.

TexasEducationAgency.(2008).TexasEssentialKnowl-edgeandSkillsforEnglishLanguageArtsandRead-ing,TexasAdministrativeCode,§110-30-66(2008).RetrievedDecember21,2009,fromhttp://ritter.tea.state.tx.us/rules/tac/chapter110/ch110c.pdf

TexasEducationAgency.(2009).Texas Essential Knowledge and Skills (TEKS) processes/instructional materials/ college readiness standards (CRS) timelines.RetrievedDecember19,2009,from,http://ritter.tea.state.tx.us/teks/TEKSTimeline051109.pdf

TexasHigherEducationCoordinatingBoard.(2008).Texas college and career readiness standards.RetrievedJanuary6,2008,fromhttp://www.thecb.state.tx.us/

index.cfm?objectid=EADF962E-0E3E-DA80-BAAD2496062F3CD8.

TexasLegislature.(2006).HouseBill1,Texas79thLegis-lature,3rdCongressionalSession,Section5.01(2006)(enacted).

TexasLegislature.(2009).HouseBill3,Texas81stLegisla-ture,Section28.014(d)(2009)(enacted).

Timms,M.,Schneider,S.,Lee,C.,andRolfhus,E.(2007a).Aligning science assessment standards: Arkansas and the 2009 National Assessment of Educational Progress (NAEP)(Issues&AnswersReport,REL2007–No. 019).Washington,DC:U.S.DepartmentofEducation,Insti-tuteofEducationSciences,NationalCenterforEduca-tionEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.Retrievedfromhttp://ies.ed.gov/ncee/edlabs

Timms,M.,Schneider,S.,Lee,C.,andRolfhus,E.(2007b).Aligning science assessment standards: Louisiana and the 2009 National Assessment of Educational Progress (NAEP)(Issues&AnswersReport,REL2007–No. 020).Washington,DC:U.S.DepartmentofEducation,Insti-tuteofEducationSciences,NationalCenterforEduca-tionEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.Retrievedfromhttp://ies.ed.gov/ncee/edlabs

Timms,M.,Schneider,S.,Lee,C.,andRolfhus,E.(2007c).Aligning science assessment standards: New Mexico and the 2009 National Assessment of Educational Progress (NAEP)(Issues&AnswersReport,REL2007–No. 021).Washington,DC:U.S.DepartmentofEducation,Insti-tuteofEducationSciences,NationalCenterforEduca-tionEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.Retrievedfromhttp://ies.ed.gov/ncee/edlabs

Timms,M.,Schneider,S.,Lee,C.,andRolfhus,E.(2007d).Aligning science assessment standards: Oklahoma and the 2009 National Assessment of Educational Progress (NAEP)(Issues&AnswersReport,REL2007–No. 022);Washington,DC:U.S.DepartmentofEducation,Insti-tuteofEducationSciences,NationalCenterforEduca-tionEvaluationandRegionalAssistance,Regional

http://www.ed.sc.gov/news/more.cfm?articleID=1301

http://www.ed.sc.gov/news/more.cfm?articleID=1301

http://eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/1b/ab/31.pdf



http://www.texaspolicy.com/pdf/2007-09-PP25-remediation-bt.pdf

http://www.texaspolicy.com/pdf/2007-09-PP25-remediation-bt.pdf

http://ritter.tea.state.tx.us/rules/tac/chapter110/ch110c.pdf

http://ritter.tea.state.tx.us/rules/tac/chapter110/ch110c.pdf

http://ritter.tea.state.tx.us/teks/TEKSTimeline051109.pdf

http://ritter.tea.state.tx.us/teks/TEKSTimeline051109.pdf

http://www.thecb.state.tx.us/index.cfm?objectid=EADF962E-0E3E-DA80-BAAD2496062F3CD8






noTeS 59

EducationalLaboratorySouthwest.RetrievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs

Timms,M.,Schneider,S.,Lee,C.,andRolfhus,E.(2007e).Aligning science assessment standards: Texas and the 2009 National Assessment of Educational Progress (NAEP)(Issues&AnswersReport,REL2007–No. 011);Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.RetrievedFebruary11,2010,fromhttp://ies.ed.gov/ncee/edlabs

U.S.DepartmentofEducation.(2008).A nation accountable: twentyfive years after a nation at risk.Washington,DC:U.S.DepartmentofEducation.RetrievedApril8,2009,fromwww.ed.gov/rschstat/research/pubs/accountable

U.S.DepartmentofEducation(2009).Race to the Top Fund–Executive summary notice of proposed priorities, requirements, definitions, and selection criteria.RetrievedNovember19,2009,fromhttp://www.ed.gov/programs/racetothetop/executive-summary.pdf

Webb,N.L.(1997).Criteria for alignment of expectations and assessments in mathematics and science education. (ResearchMonographNo.6.)Washington,DC:CouncilofChiefStateSchoolOfficers.RetrievedDecember1,2008,fromhttp://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED414305

Webb,N.L.(1999).Alignment of science and mathematics standards and assessments in four states (ResearchMonographNo.18). Madison,WI:UniversityofWisconsin-Madison,NationalInstituteforScienceEducation.

--

-

-

Webb,N.L.(2002).Alignment study in language arts, mathematics, science, and social studies of state standards and assessments for four states. A study of the State Collaborative on Assessment & Student Standards (SCASS)TechnicalIssuesinLarge-ScaleAssessment(TILSA).Washington,DC:CouncilofChiefStateSchoolOfficers.

WebbN.L.(2005).Web alignment tool: training manual. Madison,WI:WisconsinCenterforEducationalResearch.

Webb,N.M.,Herman,J.,andWebb,N.L.(2007,Summer).Alignmentofscienceandmathematicsstate-levelstan-dardsandassessments:theroleofrevieweragreement.Educational Measurement: Issues and Practices,26(2),17–29.RetrievedSeptember15,2009,fromhttp://www3.interscience.wiley.com/journal/118495110/issue

Webb,N.L.,Horton,M.andO’Neal,S.(2002).An analysis of the alignment between language arts standards and assessments for four states.Madison,WI:Wiscon-sinCenterforEducationResearch.RetrievedMarch3,2007,fromhttp://facstaff.wcer.wisc.edu/normw/AERA%202002/Alignment%20Analysis%20Language%20Arts%20%20Four%20States%2031202.pdf

Wixson,K.K.andDutro,E.(2002).Standardsforprimary-gradereading:ananalysisofstateframeworks.The Elementary School Journal, 100(2),89–110.

Wixson,K.K.,Fisk,M.C.,Dutro,E.,andMcDaniel,J.(2002).The alignment of state standards and assessments in elementary reading (CIERAReport#3-024).AnnArbor,MI:UniversityofMichiganSchoolofEdu-cation,CenterfortheImprovementofEarlyReadingAchievement.



http://www.ed.gov/rschstat/research/pubs/accountable

http://www.ed.gov/rschstat/research/pubs/accountable

http://www.ed.gov/programs/racetothetop/executive-summary.pdf

http://www.ed.gov/programs/racetothetop/executive-summary.pdf

http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED414305

http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED414305

http://www3.interscience.wiley.com/journal/118495110/issue

http://www3.interscience.wiley.com/journal/118495110/issue

http://facstaff.wcer.wisc.edu/normw/AERA%202002/Alignment%20Analysis%20Language%20Arts%20%20Four%20States%2031202.pdf



Are Texas' English language arts and reading...

Documents

Transcript of Are Texas' English language arts and reading...