CONCEPT’PRESENTATION’ - Masa · CONCEPT’PRESENTATION’ ... MobyMax’ 24 152 iReady’ 21...
36
January 4, 2015 Assessment Commi6ee Conference Call 1 CONCEPT PRESENTATION
Transcript of CONCEPT’PRESENTATION’ - Masa · CONCEPT’PRESENTATION’ ... MobyMax’ 24 152 iReady’ 21...
January 4, 2015 Assessment Commi6ee Conference Call
1
CONCEPT PRESENTATION
Agenda
I. Welcome II. Founda@on III. Assessment for Learning Study
-‐ MASA Survey Results and Recommenda@ons -‐ Study Purpose, Design and Legal -‐ What Results Look Like -‐ Study Time Line
IV. Site Visit to New Hampshire
2
FOUNDATION Missouri Associa7on of School Administrators
Show Me Accredita7on and Assessment Task Force
Goal: Iden%fy and recommend to the Missouri Commissioner of Educa%on a plan for accredita%on and assessment that emphasizes: • local control • con@nuous improvement • individual student growth with con@nued a6en@on to subgroup
achievement • right test, right @me • adaptability (flexible enough to meet current and future federal/
state guidelines • clarity of purpose (can be explained by a third grader to an adult
audience) • achieving Top 10 state status one student at a @me
3
ASSESSMENT ACCOUNTABILITY ACCREDITATION
Math/ELA Annually grades 3-‐8 Once in H.S Science once per grade span 3-‐5, 6-‐8, 9-‐12
Assessment Gradua7on Support boJom 5% of schools
State Control
• up to 7 states may submit a proposal to create an innova7ve state assessment system that includes the use of local assessments. A consor7um of up to 4 states is also permiJed. Grants may last from 3-‐5 five years. The reauthorized ESEA takes full effect in 17-‐18.
4
FOUNDATION FEDERAL ESSA REQUIREMENTS
Working from our Founda7on Build Bridge of Success
Create an assessment framework for next genera7on MAP which incorporates the following concepts and supports local control. • individual student growth with con@nued a6en@on to subgroup
achievement • right test, right @me • adaptability (flexible enough to meet current and future federal/state
guidelines
Working from research define the appropriate rela7onship between forma7ve, interim and summa7ve assessment? Define “proficient” in ways meaningful to the child, including: providing a clear, achievable learning target, suppor@ng goal se^ng with the development of personalized learning plans, providing feedback on depth of understanding, and informing the student, their parent(s)/guardian and teacher(s) about readiness for next stages in learning.
Assessment for Learning Study
6
7
Source: A Framework for Considering Interim Assessments; Marianne Perie, Sco6 Marion, Gong Na@onal Center for the Improvement of Educa@onal Assessment Feb 13, 2007
Summa7ve, Interim and Forma7ve Assessment
What interim assessments are currently used by MASA districts?
Assessment for Learning Study
8
MASA Survey Results & Proposed Assessments to Begin With
In late 2015, MASA conducted a survey, asking districts about the interim assessments being used.
9
Assessment Yes No
Star 73 103
Study Island 57 119
AIMSWeb 45 131
Acuity 38 138
Other 38 138
MobyMax 24 152
iReady 21 155
USA Test Prep 20 156
Discovery 18 158
NWEA-‐MAP 14 162
Evaluate 5 171
Ten Marks 5 171
Performance Coach 4 172
Galileo 2 174
Performance Series 0 176
The assessment list was reviewed to try and determine if there was a group that would naturally fit for the ini@al work of the proposed study. From this review, it is proposed that the following assessments be considered for the ini@al review: Star, Acuity, iReady, Discovery, NWEA-‐MAP, Evaluate, Galileo
Phase 1: February – May 2016 Purpose: Conduct research regarding the appropriate role of interim assessment in next genera@on MAP and MSIP. Ques7ons: How well do interim assessments predict performance on the MAP? (archival test data) How is interim assessment used in the learning process? (survey, extant research)
Assessment for Learning Study
10
Phase 1: February – May 2016 Par@cipa@on Requirements: -‐ Provide non-‐iden@fiable student level data for
interim assessment and MAP from the 2014-‐2015 school year according to prescribed fields on excel spreadsheet.
-‐ Adhere to study to legal requirements -‐ Meet deadlines for data submission -‐ Support other school districts involved in study as
needed.
INTERESTED? Par@cipate in webinar on (date), learn more and sign up.
Assessment for Learning Study
11
PROPOSED Process for Data Compilation, Analyses, and Reporting
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
Data from the 14-15 school year:
• District Name • Each row is a student • Grade level • Latest End-of-Year
(EOY) scale score from internal assessment ELA
• MAP ELA scale score • Latest EOY scale score
from internal assessment math
• MAP math scale score. • Random unique ID at
the student level, if the district wishes to take the findings back to the local level and apply them back to individual students.
D = District
*Note this sample is for one assessment (e.g., Acuity). This process would be applied for every assessment studied.
One Complete File per Interim
Assessment
• Correlation by grade level by content area
• Regression • Predict MAP Scale Score from
Interim Scale Score; • Attain adjusted predicted values
at the student level • Scatterplots of adjusted predicted
values against observed MAP scale scores
• Compare observed percent by achievement level against predicted.
• Correlation by grade level by content area
• Achievement level comparison by grade by content area (observed vs. predicted)
• Produced at the total, full group level, but also fully reproduced at the local, participating district level.
• Interim scale score/achievement level concordance reference table for users
Data Compilation Phase
Analyses Phase
Reporting Phase
*Note the individual level data could be returned to the district, if the district
wished to apply the findings back to individual student records. The district would
need to know the key (random student ID to
district ID) to make these joins as the researcher(s) doing this work would not want nor need access to
MOSIS IDs or District Student IDs.
12
What do the results of this study look like?
Assessment for Learning Study
13
Study Results: Actual Example
-‐ Study conducted December 2015 -‐ Student-‐level results from a commercially available
interim assessment were combined with MAP scale scores for the analysis to understand how well the interim assessments predict performance on the MAP.
Assessment for Learning Study
14
15
Assessment for Learning Study
ELA MATH
3rd Grade 352 362
4th Grade 366 394
5th Grade 365 371
6th Grade 359 355
7th Grade 357 349
8th Grade 340 214
TOTAL 2,139 2,045
Number of Cases by Grade Level by Content Area
16
Assessment for Learning Study ELA Correla7ons
Interim Feb 15-‐MAP
Interim March 15-‐
MAP
Interim Sum-‐MAP
Interim Feb 15-‐Interim March 15
Interim Feb 15-‐Interim Sum
Interim March 15-‐Interim Sum
3rd Grade .791 .789 .841 .763 .923 .953
4th Grade .680 .604 .694 .718 .933 .920
5th Grade .815 .806 .857 .790 .942 .950
6th Grade .733 .760 .798 .751 .932 .939
7th Grade .723 .799 .823 .728 .909 .948
8th Grade .758 .728 .805 .705 .923 .924
*All correla@ons significant at .05 level.
17
Assessment for Learning Study ELA ScaJerplots, MAP Scale – Interim Sum by Grade Level
Interim
Sum
Interim
Sum
Interim
Sum
Interim
Sum
Interim
Sum
Interim
Sum
18
Assessment for Learning Study ELA ScaJerplots, MAP Scale Observed– MAP Scale Predicted by Grade Level
Once the predicted MAP Scale Scores were a6ained, the next step was to apply the MAP Achievement Level scale score ranges to the observed and predicted MAP scale scores.
19
Assessment for Learning Study MAP Achievement Levels, English Language Arts
Below Basic
Basic Proficient Advanced
3rd Grade Below 2367 2367-‐2431 2432-‐2489 2490+
4th Grade Below 2416 2416-‐2472 2473-‐2532 2533+
5th Grade Below 2442 2442-‐2501 2502-‐2581 2582+
6th Grade Below 2457 2457-‐2530 2531-‐2617 2618+
7th Grade Below 2479 2479-‐2551 2552-‐2648 2649+
8th Grade Below 2487 2487-‐2566 2567-‐2667 2668+
Source: Missouri Department of Elementary and Secondary Educa@on. (June 2015). Missouri Assessment Program Grade-‐Level Assessments: Guide to Interpre%ng Results. Jefferson City, MO.
20
Assessment for Learning Study Comparing Observed vs. Predicted, ELA
21
Assessment for Learning Study Comparing Percent Top Two, Observed Vs. Predicted, ELA
22
Further disaggrega7on of the ELA observed levels against the predicted levels.
Assessment for Learning Study Addi7onal Suppor7ng Data Could Also Be Generated
23
Assessment for Learning Study Interim Sum Cut Scores for Predic7ng MAP ELA Performance Across Four Achievement Levels
3rd Grade
4th Grade
5th Grade
6th Grade
7th Grade
8th Grade
Below Basic 48 or lower
44 or lower
46 or lower
65 or lower
73 or lower
68 or lower
Basic 49-‐86 45-‐77 46-‐78 66-‐102 74-‐105 69-‐109
Proficient 87-‐120 78-‐114 79-‐123 103-‐144 106-‐146 110-‐161
Advanced 121 or higher
115 or higher
124 or higher
145 or higher
147 or higher
162 or higher
24
Assessment for Learning Study Math Correla7ons
Interim Feb 15-‐MAP
Interim March 15-‐
MAP
Interim Sum-‐MAP
Interim Feb 15-‐Interim March 15
Interim Feb 15-‐Interim Sum
Interim March 15-‐Interim Sum
3rd Grade .806 .789 .846 .777 .949 .935
4th Grade .762 .706 .782 .779 .959 .925
5th Grade .794 .798 .826 .856 .966 .961
6th Grade .786 .820 .860 .744 .931 .936
7th Grade .839 .860 .878 .873 .965 .970
8th Grade .739 .793 .816 .766 .935 .944
*All correla@ons significant at .05 level.
25
Assessment for Learning Study Math ScaJerplots, MAP Scale – Interim Sum by Grade Level
26
Assessment for Learning Study Math ScaJerplots, MAP Scale Observed– MAP Scale Predicted by Grade Level
Once the predicted MAP Scale Scores were a6ained, the next step was to apply the MAP Achievement Level scale score ranges to the observed and predicted MAP scale scores.
27
Assessment for Learning Study MAP Achievement Levels, Math
Below Basic
Basic Proficient Advanced
3rd Grade Below 2381 2381-‐2435 2436-‐2500 2501+
4th Grade Below 2411 2411-‐2484 2485-‐2548 2549+
5th Grade Below 2455 2455-‐2527 2528-‐2578 2579+
6th Grade Below 2473 2473-‐2551 2552-‐2609 2610+
7th Grade Below 2484 2484-‐2566 2567-‐2634 2365+
8th Grade Below 2504 2504-‐2585 2586-‐2652 2653+
Source: Missouri Department of Elementary and Secondary Educa@on. (June 2015). Missouri Assessment Program Grade-‐Level Assessments: Guide to Interpre%ng Results. Jefferson City, MO.
28
Assessment for Learning Study Comparing Observed vs. Predicted, Math
29
Assessment for Learning Study Comparing Percent Top Two, Observed Vs. Predicted, Math
30
Further disaggrega7on of
the Math observed levels against the
predicted levels.
Assessment for Learning Study Addi7onal Suppor7ng Data Could Also Be Generated
31
Assessment for Learning Study Interim Sum Cut Scores for Predic7ng MAP Math Performance Across Four Achievement Levels
3rd Grade
4th Grade
5th Grade
6th Grade
7th Grade
8th Grade
Below Basic 83 or lower
72 or lower
66 or lower
75 or lower
48 or lower
64 or lower
Basic 84-‐114 74-‐118 67-‐109 76-‐109 49-‐96 65-‐102
Proficient 115-‐149 119-‐157 110-‐139 110-‐135 97-‐135 103-‐131
Advanced 150 or higher
158 or higher
140 or higher
136 or higher
136 or higher
132 or higher
Correla7ons for Both Content Areas and MAP
ELA MAP—Interim Sum
Math MAP—Interim Sum
3rd Grade .841 .846
4th Grade .698 .782
5th Grade .857 .826
6th Grade .798 .860
7th Grade .823 .878
8th Grade .805 .816
32
Phase 2: Design Next Genera7on MAP Grades 3-‐8 (June 2016 – Spring 2017)
Local: standards based forma@ve and interim assessments with student as primary audience. State: standards based summa@ve end-‐of-‐grade level/grade span growth measures with student as primary audience and adults as an important second audience. Used to meet ESSA requirements. -‐ Innova7ve Design (e.g. adap@ve/EOC-‐Like/testlet) -‐ Mul7ple Administra7on Opportuni7es (e.g. not once and done as
the only op@on) -‐ Redefined Cut Score (e.g. proficiency set at true end of grade level expecta@on with considera@on of new nomenclature) -‐ Enhanced MOSIS (e.g. captures learning progression by ELA, Math,
Science and chronological grade level) 33
Assessment for Learning Study
Phase 2: Secure ESSA Innova7on State Assessment Pilot Grant* Year 1: Iden@fy a defined number of districts to implement innova@ve preK-‐12 approaches to assessment that lead to an understanding of the appropriate roles forma@ve, interim and summa@ve assessment play in suppor@ng college and career readiness for ALL students. Preference will be given to proposals with following design elements. -‐ Mul@-‐district mul@-‐region collabora@ve which includes, for research purposes, a
sufficient number of students from each subgroup. -‐ Agreement to collect, analyze and report data in prescribed formats. -‐ Shows promise for successful replica@on in a variety of contexts. -‐ Meets ESSA requirements while opera@onalizing the following principles: local
control, con@nuous improvement, individual student growth with con@nued a6en@on to subgroup achievement, right test, right @me, adaptability (flexible enough to meet current and future federal/state guidelines, clarity of purpose (can be explained by a third grader to an adult audience), achieving Top 10 state status one student at a @me.
-‐ Commits to par@cipa@on in a state-‐wide next genera@on MAP development group. * Federal Requirements not yet released and content above for discussion purposes only. 34
Assessment for Learning Study
Phase 2: Secure ESSA Innova7on State Assessment Pilot Grant* Year 2: Implement next genera@on MAP and conduct a state wide research project with sufficient par@cipa@on to understand growth effects of the new system across all subgroups. Informa@on from this study will be used to inform teachers, administrators, Boards of Educa@on, parents and community on best prac@ces for con@nuous improvement across all subgroups. The Missouri Assessment Framework will be adjusted based on data from this research.
Year 3: TBD • Federal Requirements not yet released and content above for discussion purposes only.
35
Assessment for Learning Study
New Hampshire Site Visit Manchester NH Feb 3 – 5
Host: Ray McNulty Key Ques7ons (sample) What is the rela@onship between Competency (Proficiency) Based Learning, School Improvement and State Accountability? How can competency (proficiency) based learning inform instruc@on and assessment design? How does the New Hampshire local, state and university level work inform next steps in Missouri?
Cost Es7mate: Hotel $200 a night plus plane fare and meals
36
