1

Montana Comprehensive Assessment System

Montana Office of Public Instruction

Linda McCulloch, Superintendent

May 2006

2

Welcome

Panelists– Sharon Houle, Measured Progress– Carol Taylor-Cann, Riverside Publishing– Bob Runkel, OPI—GAP Students– Lynn Hinch, OPI—English Language Proficiency– Donna O’Neill, OPI--NAEP Coordinator– Linda Peterson, OPI—Science Standards

– Moderator, Judy Snow, OPI--Assessment

3

Agenda

• MontCAS Overview– Judy Snow

• MontCAS—Updates on Other Assessments – Panelists

• MontCAS Issues and Details– Judy Snow

4

MontCAS Overview

• Assessment 101

• MontCAS Phases 1 and 2

5

Assessment 101

• Standardized Test• Norm-referenced test• Criterion-referenced

test• Performance levels• Alternate Assessment• LEP, ELL, ELP

• NCLB– No Child Left Behind– Adequate Yearly

Progress (AYP)

• Administrative Rules of Montana– ARM– Board of Public

Education (BPE)

6

Phase 1 Norm-referenced test (NRT)

—the Iowas—Riverside Publishing Company

Aligned to national standards Grades 4, 8, 11 Reading, language arts,

math, social studies, science Multiple choice Alternate assessment scales Funded by the state

Phase 2 Criterion-referenced test

(CRT)—Measured Progress Aligned to Montana Content

Standards Grades 3-8, and 10

(science in spring 2008) Reading and math Multiple choice, math short

answer and constructed response

Evidence based alternate Funded by NCLB

7

Other AssessmentsPanelists

8

Bob RunkelModified CRT

9

GAP Students

• In between CRT-Alternate and CRT• Modified performance standards• Feasibility Study

– Grade 5 students – Identified as special education students for the grade

4 CRT in spring 2006– Pilot math CRT based on modified performance

standards

• Tentative Window– September 11-19, 2006

10

Lynn Hinch English Language Proficiency Test

11

ELP

• English Language Proficiency– Required by NCLB– Grades K-12– Test Development through Mountain West

Consortium (MWAC)– Date TBA

12

NO CHILD LEFT BEHIND REQUIREMENTS

• Both Titles I and III require an annual assessment of English language proficiency of all K - 12 limited English proficient students

• Assessment must be aligned to state English language proficiency standards

• Assessment must report a separate score for listening, speaking, reading, and writing

13

TITLE III REQUIREMENTS

• States must establish Annual Measurable Achievement Objectives (AMAOs) that determine how LEP students are making progress in learning English, and whether students are attaining proficiency in English.

14

Status of English Language Proficiency Assessment

• Mountain West Assessment Consortium

• Test Development and Delivery

• RFP Process

• March '06 Request to the Department of Education for timeline extension

15

Donna O’NeillNAEP

16

NAEP

• National Assessment of Educational Progress– NAEP--Nation’s Report Card– Odd-numbered Years– Grades 4 and 8, Reading and Math– Required Sample

17

All About NAEP

• What is NAEP?

• Why is it important to Montana?

• Who takes the NAEP Assessment?

• What are the requirements for NAEP?

• New developments in Science & Writing

• What is the timeline for the NAEP Assessment?

18

NAEPNational Assessment

• The Nation’s Report Card• Measures academic achievement• Reports to the public what students know and can do:

– average scale scores– proficiency levels

• Formats vary by grade:– multiple choice – constructed-response

• Links performance to educational variables– Teacher background information– School characteristics

• NAEP is confidential

19

Montana

20

The Nation’s Report Card

21

Montana

22

Who?

• Random sample– 2,000 students per

• Subject• Grade

• Grades 4, 8 & 12

• Students with disabilities

• Limited-English proficient

23

NAEP Requirements

• Linked to Title I funding – Schools– Districts

• NAEP is designed – Minimal disruption of instruction– 90 to 120 minutes of each student’s time– Teachers are encouraged to observe– Field staff are many Montana teachers

24

New Developments

• Science Release– May

• Writing (8, 12)

• Pilot and Special Studies– National Indian Education Study (NIES)

• Grades 4, 8• NAEP data will be linked to survey data• Student, Teacher and Principal Questionnaires

25

NAEP Timeline

26

Key Dates

• Secure cooperation of districts and schools– Notify districts in early May– Notify schools of selection during May– Notify schools of assessment dates in June– Maintain communication with schools

throughout the process

27

Handouts

• Packet of Information– Facts About Montana Education– Historical information on results– NAEP Questions Tool– NAEP Data Explorer (researchers)

• http://nces.ed.gov/nationsreportcard/

28

Sharon HouleMeasured Progress

29

CRT Science Test

• Grades 4, 8, and 10

• Spring 2008

• Aligned to Revised Science Standards

30

Criterion-Referenced Test Where are we going?

Science--grades 4, 8 and 10• May 31, 2006

– National science content and bias reviews

• July 25– Montana science content and bias reviews

• Spring 2007– Field test

• Spring 2008– First Administration—CRT and CRT-Alternate

31

Linda Peterson

ACCREDITATION DIVISION

STANDARDS REVISION PROCESS

32

MontanaMontana StandardsStandards

(Intended)(Intended)

MontanaMontana StandardsStandards

(Intended)(Intended)

CurriculumCurriculum(Intended)(Intended)

CurriculumCurriculum(Intended)(Intended)

AssessmentAssessment(Assessed)(Assessed)(Learned)(Learned)

AssessmentAssessment(Assessed)(Assessed)(Learned)(Learned)

ProfessionalProfessionalDevelopmentDevelopmentProfessionalProfessionalDevelopmentDevelopment

InstructionInstruction(Enacted)(Enacted)

InstructionInstruction(Enacted)(Enacted)

5YCEP Continuous Improvement Process

5YCEP Continuous Improvement Process

5YC

EP

Co

nti

nu

ou

s Im

pro

vem

ent

Pro

cess

5YC

EP

Co

ntin

uo

us Im

pro

vemen

t Pro

cessM

on

tan

a S

urv

eys

of

the

En

acte

d C

urr

icu

lum

(M

SE

C)

– A

lig

nm

ent

Ass

essm

ent

Pro

cess

The Five-Year Comprehensive Education Plan (5YCEP) Building a Continuous Improvement Process for Montana Schools

33

Montana Board of Public Education

CHAPTER 54

CONTENT AND PERFORMANCE STANDARDS

10.54.2503 STANDARDS REVIEW SCHEDULE (1) Montana's content and performance standards shall be reviewed and revised on a five-year cycle beginning July 1, 2005.

Standards

CurriculumInstruction

Assessment

34

Montana Board of Public Education

Standards Revision 2005-2010

Purpose• Assure Montana citizens that its public schools are

providing all children of our great state with challenging academic expectations

• Revised standards clearly and consistently identify what students should know, understand and be able to do

• Revised standards provide a framework to help

guide local curriculum and instruction

35

Montana Board of Public EducationCriteria to Guide Standards Revision

Standards will be:

• academic in nature and content specific.

• challenging and rigorous.

• clear, understandable and free of jargon.

• measurable.

• address diversity, specifically fulfilling the commitment to implementing 20-1-501, Indian Education for All.

36

Montana Standards Review Schedule

Cycle I Science July 2005 – September 2006

Cycle II MathematicsCommunication Arts

TBD

Cycle III Social StudiesHealth EnhancementArtsTechnology

TBD

Cycle IV School CounselingLibrary MediaCareer and Technology EducationWorkplace CompetenciesWorld Languages

TBD

37

K-12 Science Standards

Montana Standards Framework

• Content Standards

• Benchmarks - Grades 4 and 8 and upon graduation

• Performance Descriptors

38

K-12 Science Standards

Performance Descriptors

• Advanced – superior performance• Proficient – solid academic competency• Nearing Proficiency – partial mastery of

prerequisite knowledge and skills• Novice – beginning to attain prerequisite

knowledge and skills

39

K-12 Science Standards

Content Standard 1

Students design, conduct, evaluate, and communicate processes and results of scientific investigations, and demonstrate thinking skills associated with this procedural knowledge

Benchmarks• Grade 4 (3) - describe and

communicate the results of scientific investigations

• Grade 8 (3) - review, communicate and defend results of investigations

• Upon Graduation – review evidence, communicate and defend results, and recognize that the results of a scientific investigation are always open to revision by further investigation

40

Performance DescriptorsContent Standard 1

Proficient• Grade 4 – with direction, completes a simple investigation with

identified variables, using appropriate tools and communicates results

• Grade 8 – identifies and communicates testable questions, plans and conducts experimental investigations and communicates results

• Upon Graduation – generates testable questions, constructs a plan for a controlled investigation, makes logical inferences based on observations, accurately interprets data by identifying the strengths and weaknesses in an investigation design, and communicates results.

41

K-12 Science Standards

Content Standard 3

Students demonstrate knowledge of characteristics, structures, and function of living things, the process and diversity of life, and how lining organisms interact with each other and their environment, and demonstrate thinking skills associated with this knowledge.

Benchmarks• Grade 4 – create and use a

classification system to group a variety of plants and animals according to their similarities and differences, (e.g., American Indian medicinal plants, American Indian dwellings)

• Grade 8 – create and use a basic classification scheme to identify plants and animals, (e.g., classification scheme based on American Indian use of indigenous plants for medicinal, food and other uses)

• Upon Graduation – recognize, generate and apply biological classification schemes to infer and discuss the degree of divergence using ecosystems, (e.g., American Indian food collections systems)

42

Performance DescriptorsContent Standard 3

Proficient• Grade 4 – identifies attributes of biotic (living) things and abiotic

(non-living) objects, including classification based on similarities and differences, basic structure and function, processes of each system

• Grade 8 – identifies and classifies biotic things and abiotic objects through application of common classification schemes; identifies the interdependence of life and the environment and explains how characteristics of living things change because of the environment

• Upon Graduation – organizes, classifies, and describes interactions of the biotic and abiotic parts of the biosphere as well as the natural history of interactions of life on Earth and uses these skills to solve related novel (to the student) problems

43

K-12 Science Standards Proposed Time Line

• March – May Gather Public Comments

• May 11 Present Update to BPE

• May 15 Presentation and Discussion with MACIE

• May – June Writing Team Work Sessions

• June Prepare draft standards for notice of hearing

44

K-12 Science Standards Proposed Time Line

• July Present to BPE draft Notice of Hearing and proposed time line

• August Notice of Hearing posted

• October Public Hearing

• November Final BPE Action Anticipated

• 2007 Implementation, Outreach, and Dissemination

45

K-12 Science Standards Revisions

Linda Vrooman Peterson

Accreditation Division Administrator

Office of Public Instruction

lvpeterson@mt.gov

(406) 444-5726

Draft K-12 Science Standards link:http://www.opi.mt.gov/pdf/standards/ScienceStandardsMarch06.pdf

46

Assessment Issues and DetailsStandardized

Administration

Accurate Data

Technical Adequacy

Impact on Instruction

47

Standard Administration = Accurate + Reliable Data

•Participation •Training•Test Security•Special Populations

•Accommodations•Alternate Assessments

48

Participation All Means All!

• Phase 1 Options– No

accommodations– If in IEP, 504, or

LEP plan• Standard or non

standard accommodations

• Alternate Assessment Scale

• Combination

• Phase 2 Options– No

accommodations– Standard

accommodations– If in IEP or 504

plan• Non standard

accommodations• CRT-Alternate

49

Training so that . . .

• everyone understands the procedures for administration.

• the test is administered in a comparable way in all locations across the state.

• quality control procedures are utilized when returning test materials.

50

Test Security

All test items and responses to those items in MontCAS are secure materials and may not be copied or duplicated in any way; CRT materials may not be retained in the school after testing is completed.

It is very important to review test security information with test administrators during test administration training.

51

OPI Test Security • Any concern about breaches in test security or

noncompliance with test administration procedures must be reported immediately to the principal and system test coordinator and to the State Assessment Director.   

• OPI Guidelines and Procedures for Test Security– This OPI publication outlines procedures for reporting

testing irregularities and should be made available to system superintendents, principals, and test administrators.

52

Standard accommodations do not change what we expect students to know and be able to do.

Test accommodations are based on individual student needs and classroom assessment practice; they are not for the use by an entire classroom or category of students.

They should not give students unfair advantages; rather they are meant to remove barriers that may exist due to a student’s learning style or disability.

Accommodations

53

Standard and Non StandardAccommodationsStandard • Do not change the

intent/content of the assessment

• Phase 1: Only by IEP, 504, or LEP plan

• Phase 2: By IEP, 504, LEP, or regular routine in the classroom

Non Standard• Change the

content/intent of the assessment

• Phase 1: Only by IEP,504, or LEP plan

• Phase 2: Only by IEP or 504 plan

54

Accurate Data• the information collected is of high quality

demographic informationenrollment informationdisability informationdecisions related to the use of

accommodations and the alternate assessment (CRT-Alternate)

55

2006 Data Check Points

OPI CRT Testing Cycle Enrollment Count• March 6, 2006 Window Opens• April 13, 2006 Data due to County Superintendent• April 20, 2006 Data due to OPI

Measured Progress CRT Online Demographic Data Verification• April 24 – May 19, 2006

OPI Comparability Study—June 2006

OPI Iowa Tests (NRT) Disaggregated Data Review• July 24 - August 4 , 2006

56

Technical Quality and Expertise

• Test Development

• Technical Advisory Committee (TAC)

• Technical Manuals

• CRT and CRT-Alternate Standard Setting

57

Montana Performance Levels

• Advanced• Proficient

• Nearing Proficiency• Novice

58

Standard Setting

• Standard setting is the process of determining cut points for performance levels for the CRT and the CRT-Alternate.

• Panels of Montana educators

• Facilitated by Measured Progress

Standard Setting

59

2006 Standard Setting Details

• For more information: http://www.opi.mt.gov/assessment/Phase2.html#TD

• To apply: – http://iservices.measuredprogress.org

• Travel, meals (those not provided during the sessions), and lodging reimbursements

• Renewal units and honoraria

60

2006 Standard SettingWhere and When

Red Lion Colonial Inn in Helena– June 21-22

• CRT, Reading and Math, grades 4, 8, and 10

– June 20-22• CRT-Alternate, Reading and Math, grades 3-8 and

10

– July 26-27• CRT, Reading and Math, grades, 3, 5, 6, and 7

61

Impact on Instruction

• Reporting

• Data Interpretation

• Tools

62

Reporting and Impact on Instruction

• Phase 1—NRT– NPR– NCE– Stanine– Performance Level

• Phase 2—CRT– Scaled Score

• Performance Level for each content area– Raw Score

• Number of points for each standard

63

Iowa Tests Interpretive Guide

2005

http://www.opi.mt.gov/PDF/Assessment/NRT/05InterpGuide.pdf

• Relationship of stanines, percentile ranks, and normal curve equivalents (p. 5)

• Comparison of NRT Scores (p. 6)

• Performance Profile (p. 12)

• Individual Customized Skills Report (p. 16)

64

CRT Results Interpretation

• 2005 Interpretive Guidehttp://www.opi.mt.gov/PDF/Assessment/CRT/05CRTCRTaltInterpGuide.pdf

– Roster and Item-Level Report (p. 6)

• Released Items– Items that scored are released.– http://www.opi.mt.gov/Assessment/index.html

• iAnalyze– A secure website where data gathered from the CRT

assessments are reported

65

Judy SnowState Assessment DirectorOffice of Public Instructionjsnow@mt.gov406-444-3656

Assessment link:http://www.opi.mt.gov/Assessment/index.html JUMP link:http://www.opi.mt.gov/Assessment/JUMP.html