EQAO Alignment Analysis - University of Michiganmayess/Sarah_Mayes:_Teaching...The EQAO Primary and...

ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS

Mayes_03.29 EDUC783-Winter12

1

The EQAO Primary and Junior Mathematics Assessments: An Alignment and Content Analysis

Sarah Mayes

University of Michigan



2

The EQAO Primary and Junior Mathematics Assessments: An Alignment and Content Analysis

As the results of large-scale assessments gain increased attention at local, regional,

national, and international levels, it is becoming even more important that they be of the highest

quality possible (La Marca, 2001). The multiple roles that assessments play dictate factors that

must be considering when analyzing their effectiveness. One of these roles is to define curricula:

teachers, students, and other interested parties look to large-scale assessments as an indicator of

the content and cognitive skills that students are supposed to learn (Herman, 2004).

Curricula are reflected in many forms across an educational system, including official

regional standards, textbooks, the material that is taught, and assessments. Alignment describes

the relationship between two or more of these curricula and measures the extent to which they

agree. Researchers have studied the alignment between assessments and textbooks (Chandler &

Bronsan, 1995), between assessments and instruction (Porter, 2002), and between assessments

and standards (Roach, Elliott, & Webb, 2005; Webb, 1999; Wixson, Fisk, Dutro, & McDaniel,

2002) across various subjects.

Comparing standards and assessments has become increasingly important in recent years

as localities adopt various standards-based curricula and large-scale assessments purporting to

measure these curricula (Herman, Webb, & Zuniga, 2007). Achieving good alignment between

standards and assessments is not only essential for accountability (La Marca, 2001), validity of

test scores (Rothman, 2003), and providing information to the public (Herman, Webb, & Zuniga,

2007), but also for creating a coherent education system and sending consistent messages to

students and to teachers. If there is not good alignment, teachers may "teach to the test" and

cover only what is assessed (Herman, 2004). Further, some researchers believe that better

aligning tests with standards can increase student learning (Farenga, Joyce, & Ness, 2002) and



3

minimize the effects of factors such as socio-economic status, parental education, and home

support on students' performance (Mohamud & Fleck, 2010).

Assessments should not only be aligned with the standards they are supposed to measure,

but, like other aspects of the curriculum, they should also follow the practices of the academic

subjects which they assess. Precision and clarity, for instance, are vital parts of mathematical

practice communication; thus, tests evaluating mathematics should have questions that are

correct and instructions that are clear.

This paper reports the results of an analysis of ten recent elementary-level large-scale

assessments in Ontario administered by the Education Quality and Accountability Office

(EQAO). The research questions guiding this analysis were:

1. How well aligned to The Ontario Curriculum (Ontario Ministry of Education,

2005) are the EQAO Primary and Junior mathematics assessments?

2. Are the items on EQAO Primary and Junior mathematics assessments

mathematically correct?

As discussed above, the alignment and mathematical integrity of large-scale assessments is

important. The answers to these questions can provide guidelines for improving the EQAO

assessments while clarifying the current role of the assessments in Ontario’s education system.

The following section provides further background on The Ontario Curriculum (Ontario

Ministry of Education, 2005) and on EQAO necessary to understand the analysis conducted.

The Methods section outlines the details about how the analyses were conducted while the

Results section answers the research questions. Finally, the Discussion section lays out some

implications of this analysis and suggests how the quality of EQAO assessments might be

improved.



4

Background

The first part of this section provides background information about the provincial

mathematics curriculum in Ontario while the second part outlines the administration, purposes,

and structure of the EQAO assessments.

The structure of the Ontario mathematics curriculum

Students in Ontario generally attend elementary school from grades 1 through 8 and high

school from grades 9 through 12. The current version of the provincial mathematics curriculum

for elementary schools, The Ontario Curriculum, Grades 1-8: Mathematics (Ontario Ministry of

Education, 2005) was implemented in September 2005.1

There are two types of expectations in the Ontario mathematics curriculum. The Process

Expectations are the same for every grade level and include: problem solving; reasoning and

proving; reflecting; selecting tools and computational strategies; connecting; representing; and

communicating. The Specific (content) Expectations are divided into the following five strands:

Number Sense and Numeration; Measurement; Geometry and Spatial Sense; Patterning and

Algebra; and Data Management and Probability. Within each strand, expectations are further

classified into two to three sub-strands (see Table 1). The process expectations are to be

incorporated into each of the content expectations at a grade-appropriate level.

1 Ontario curriculum documents are available on the Ministry of Education’s website, http://www.edu.gov.on.ca.



5

Table 1

Strands and sub-strands in the of The Ontario Curriculum for grades 3 and 6

Strand Grade 3 Sub-strands Grade 6 Sub-strands NSN: Number Sense and

Numeration NSN1: Quantity

Relationships NSN2: Counting NSN3: Operational Sense

NSN1: Quantity Relationships

NSN3: Operational Sense NSN4: Proportional

Relationships M: Measurement M1: Attributes, Units, and

Measurement Sense M2: Measurement

Relationships

M1: Attributes, Units, and Measurement Sense

M2: Measurement Relationships

GSS: Geometry and Spatial Sense

GSS1: Geometric Properties GSS2: Geometric

Relationships GSS3: Location and

Movement

GSS1: Geometric Properties GSS2: Geometric

Relationships GSS3: Location and

Movement PA: Patterning and

Algebra PA1: Patterns and

Relationships PA2: Expressions and

Equality

PA1: Patterns and Relationships

PA2: Variables, Expressions, and Equations

DMP: Data Management and Probability

DMP1: Collection and Organization of Data

DMP2: Data Relationships DMP3: Probability

DMP1: Collection and Organization of Data

DMP2: Data Relationships DMP3: Probability

Although most of the Ontario curriculum document outlines specific expectations, topics

such as classroom assessment, accommodation of exceptional students, technology, and

integration of other subjects are also discussed. Neither this document nor other publications by

the Ministry of Education give any explicit guidance on the relative importance of strands, sub-

strands, or expectations. This implies that all strands, sub-strands, and expectations are equally

important; this assumption will be made for the alignment analysis described here.

Unless otherwise specified, in this document, "the curriculum" will refer to The Ontario

Curriculum, Grades 1-8: Mathematics (Ontario Ministry of Education, 2005).



6

The EQAO Assessments

The Education Quality and Accountability Office (EQAO), established in 1996, is a

Crown agency of the government of Ontario. EQAO's mandate is to conduct province-wide

assessments of performance in mathematics, reading, writing, and literacy. There are four

separate mathematics assessments conducted each year: the Primary Assessment of

Mathematics written at the end of grade 3; the Junior Assessment of Mathematics written at the

end of grade 6; and two Grade 9 Assessments of Mathematics taken at the completion of a grade

9 math course.2 These assessments test the math skills that students are expected to have gained

up to that point, as specified in the curriculum.

The Primary and Junior Assessments of Mathematics currently consist of 7 short-answer

and 29 multiple-choice questions each.3 Students writing the Primary assessment may not use a

calculator or manipulatives for the first six multiple-choice questions; apart from this, calculator

use is permitted throughout the other mathematics assessments.4

The EQAO assessments are criterion-referenced and use the same four-level rubric that is

used for all subjects and grade levels in Ontario (for example, level 3 denotes performance at the

provincial standard). The analysis of the results of the EQAO tests uses the Item Response

Theory model known as the 2 Parameter Logistic model, which takes into account both item

difficulty and the relationship between student scores on the test and responses to the item

(Kozlow, 2007).

Every student who writes an EQAO assessment is given an Individual Student Report

that states the overall level that they achieved on the assessment. Detailed provincial, school

2 Copies of the EQAO assessments are available on the agency’s website, http://www.eqao.com. 3 Different test structures have been used in the past but all assessments considered in this analysis are of this current form. 4 Schools are required to ensure that students have calculators during the tests.



7

board, and school reports are available on EQAO’s website. These reports include the number of

students within the province, school board, or school achieving each level on the assessment

along with a summary of the results of the student questionnaire.

According to the EQAO (2012), the results of these tests are used by the provincial

government to "make local schools and school boards accountable" and by administrators at all

levels "for effective system-improvement planning." Although the tests are not as high-stakes as

some similar American assessments and are in no way tied to funding, student placement, or

teacher evaluation, there is public pressure for schools and school boards to perform well.5

Method

This section will be organized as follows. First, the Sampling sub-section describes

which assessments were chosen to be part of the analyses and why. The Analysis subsections

detail the procedure for matching assessment items to expectations in the curriculum, the

textbook coding procedure, and the alignment criteria used. Finally, the Quality of Data sub-

section justifies two aspects of the assessment and textbook coding procedures.

Sampling

Although EQAO tests have been administered each year since 1998, the curriculum and

the structure of the tests has changed since that time. A revised Ontario Mathematics Curriculum

document was implemented in September 2005 and the Primary Assessment did not take its

current form in regards to the use of calculators and manipulatives until 2007. To ensure that all

tests evaluated had a similar structure, I restricted the analysis to the five Primary and the five

Junior mathematics assessments administered from 2007 through 2011.

I chose not to analyze the grade 9 mathematics assessments because the high school

5 The Fraser Institute, for example, uses the results of the EQAO assessments to rank high schools in Ontario.



8

mathematics curriculum is significantly different from the elementary curriculum. For example,

the strands and sub-strands of high school courses are course-specific and there are two different

grade 9 mathematics courses that students can take. As a result, the structure of the EQAO grade

9 mathematics assessments is not the same as the structure of the elementary assessments.

Analysis: Item Coding Process

I matched specific expectations from the grade 3 (respectively, grade 6) mathematics

curriculum to each item (or problem) from the Primary (respectively Junior) mathematics

assessments. More than one curriculum expectation was matched to some items if skills from

multiple expectations were needed to solve the problem or if significantly different strategies

could be used to solve the problem.6 Each match of an expectation to an assessment item is

called a hit for that expectation.

The following assumptions were made in the coding process.

• Students are likely to use the easiest or most direct method available that they know. For

example, students use calculators, when available, rather than doing a computation by

hand.

• Different students may use knowledge from different expectations to solve a given

problem. Consider, for example, question 30 from the 2007 Primary Assessment:

Steven earns $5 for every bundle of newspapers he delivers. He wants to buy a

game that costs $18. How many bundles of newspapers does Steven need to

deliver to earn enough money to buy this game?

A grade 3 student could use either skip counting or multiplication to solve this problem.

In the item matching, expectations corresponding to both methods would be matched to

6 The decision to match some items to multiple expectations is consistent with the alignment model developed by Webb (2005) that I use in this analysis.



9

this item even though any one student would only need to use one strategy to solve the

problem.

Assessment items were matched to expectations from grades 3 and 6 rather than to

expectations from all primary or junior grades; this choice was made for two reasons. First,

nearly all items on the Primary (Junior) Assessments were at the grade 3 (grade 6) level rather

than a lower grade level. Second, among those items that could have been matched to an

expectation at a lower grade level, there was almost always a grade 3 (grade 6) expectation that

included the expectation from a lower grade either explicitly or as a prerequisite.

The exception to this rule came from a set of expectations in the Location and Movement

sub-strand of the Junior curriculum. The following expectation appears in each junior grade with

the blanks filled in with translations grade 4, reflections in grade 5, and rotations in grade 6:

"identify, perform, and describe ______ using a variety of tools" (Ontario Ministry of Education,

2005). I decided to classify items requiring students to identify, perform, and describe

translations and reflections into the similar grade 6 expectation that included only rotations.

During the matching process it became clear that it was necessary to add one expectation

to the NSN3: Operations Sense sub-strand in the grade 3 curricula. This additional expectation

states "relate addition and subtraction to real-life situations.” This expectation did not appear in

the curriculum for any of the primary grades and is analogous to the grade 3 expectation: "relate

multiplication of one-digit numbers and division by one-digit divisors to real-life situations,

using a variety of tools and strategies" (Ontario Ministry of Education, 2005). While one might

argue that the ability to “relate addition and subtraction to real-life expectations” should be

considered part of the problem-solving process expectation, I felt that it was necessary to include

this expectation as a content standard due to the pervasiveness of this skill in the primary-level



10

assessments studied and to make the treatment of addition and subtraction consistent with that of

multiplication and division.

During the matching process, I flagged test items containing unclear wording, vague

instructions, or incorrect statements.

Analysis: Textbook Coding Process

Ontario's Trillium List includes all of the textbooks approved for use in the province's

classrooms.7 It includes three grade 3 mathematics texts and two grade 6 mathematics texts.

Textbook sections in one approved grade 3 text (Nelson Mathematics 3) and both approved grade

6 texts (Mathematics Makes Sense 6 and Nelson Mathematics 6) were counted and categorized as

focusing on one of the five content strands from the Ontario curriculum.8 Sections with a focus

in more than one strand were recorded as fractions to the nearest quarter of a section. Review

sections and sections focused on process standards without a specific content were excluded.

Analysis: Alignment Criteria Used

Each of the ten assessments was judged for alignment to the Ontario curriculum using the

following three criteria from the model developed Webb (2005): categorical concurrence, range-

of-knowledge, and balance of representation.

Categorical Concurrence

One way to judge the alignment of an assessment to a curriculum is to measure whether

the assessment measures all of the content categories that make up the curriculum. For example,

a test containing only questions on arithmetic is not aligned to the Geometry and Spatial Sense

strand. Webb says that the categorical concurrence criterion is met for a content category if the

assessment adequately measures it. The current analysis measures categorical concurrence at

7 The Trillium List can be viewed online at http://www.edu.gov.on.ca/trilliumlist/ 8 A ‘section’ refers to a section within a chapter as defined by the textbook authors.



11

both the strand and the sub-strand levels and assumes that at least there needs to be at least six

hits for a category in order for the assessment to have acceptable categorical concurrence with

that category.9 That is, six hits are needed to adequately assess whether a student has mastered

the given skill (Webb, 2005). Note that this criterion is more difficult to meet on the sub-strand

level than on the strand level.

Range-of-Knowledge

Assessments that are aligned with curricula should measure the same span of knowledge

as the curricula itself. For example, suppose that the curriculum has ten expectations in the

Measurement Relationships sub-strand and that only one of these expectations relates to finding

areas. If every item on an assessment drawing from the Measurement Relationships sub-strand

relates to finding areas, this assessment is not aligned with the curricula for the sub-strand

because it does not cover the same breadth as the curricula. The range-of-knowledge criterion is

met for a strand or a sub-strand on an assessment if the span of knowledge required to complete

the assessment is similar to the span of knowledge expected by the curricula. For this standard to

be met for a strand or sub-strand on an assessment, at least 50% of the expectations within a

strand or a sub-strand need to be hit by assessment questions.

Balance of Representation

A well-aligned assessment should test each of the expectations which are hit an equal

number of times. The balance of representation criterion measures how equally assessment

content is distributed among the standards that are hit. The balance index for an assessment with

respect to a given strand or sub-strand is given by

9 Recall that each match of an assessment item to a curriculum expectation is a ‘hit’ for that expectation.



12

where O is the total number of objectives hit within the strand or sub-strand, is the number of

hits corresponding to an expectation k, and H is the total number of hits within the strand or the

sub-strand. An index value of 1 indicates perfect balance and that hits are equally distributed

among the hit expectations while small index values indicate poor balance. Webb says that the

assessment satisfies the balance of representation criterion for a given standard or substandard if

the balance index is greater than or equal to 0.7.

The range-of-knowledge criterion and balance of representation criterion are

complementary and equally necessary to determine how well test items are distributed among

curriculum expectations. For example, consider a curriculum strand containing 20 expectations.

On one assessment, each of 8 expectations is hit exactly three times and no other expectations

are hit. In this case the range-of-knowledge correspondence is 40% (8/20) while the balance

index is 1. Now consider a second assessment where 15 of the 20 expectations are hit. One of

the expectations is hit 10 times while the remaining 14 expectations are hit only once. In this

case the range-of-knowledge is 75% (15/20) while the balance index is 0.7. Therefore, it is

possible to have assessments with very weak range-of-knowledge but perfect balance of

representation, as in the first case, and also assessments with strong range-of-knowledge and

weaker balance of representation, as in the second case.

Depth of Knowledge

Webb also includes a fourth alignment criterion, depth-of-knowledge, that compares the

complexity of knowledge stated in an expectation to the complexity of knowledge required to

complete a question on the assessment. To measure this criterion each content expectation must



13

be assigned a level according to whether it includes recall, skills/concepts, strategic thinking, or

extended thinking. However, the Ontario curriculum includes process expectations similar to

Webb’s depth-of-knowledge levels that are to be integrated into every content expectation and so

it is impossible to assign complexity of knowledge scores to the content expectations in the

curriculum. Therefore, this criterion was not used in the current analysis of EQAO tests.

Quality of the Data

Although I was the only reviewer matching expectations to test items, an Ontario teacher

completed the matching procedure for one Primary Assessment and one Junior Assessment.

This reviewer’s matches agreed with mine for all but one question; this indicates that the data

used for the alignment analysis was reasonably accurate. Further, this reviewer looked at all

items that I flagged for problematic mathematical content and agreed with my concerns on each

of the items.

In the textbook analysis, counting sections provided almost the same information as

counting by pages, because each of the textbooks had a standard section length (nearly all

sections in Nelson Mathematics were two pages while in Mathematics Makes Sense nearly all

sections were three pages). I chose to count sections because it is likely that teachers planning

lessons around a textbook will follow the sections in the textbook.

Results

This section presents the results of the analyses conducted on the Primary and Junior

EQAO tests. The first sub-section details the results of the alignment analysis, the second sub-

section details the results of the textbook analysis, and the third sub-section discusses the results

of the mathematical integrity analysis.



14

Alignment Analysis

Tables 2 and 3 summarize the results of the alignment analyses; detailed results are given in

Appendix. For each strand and sub-strand in the curriculum, these tables show the percentage of

the five Primary and the five Junior assessments satisfying each alignment criterion. The

following sub-sections, organized by alignment criterion, elaborate on the information found in

these tables.



15

Table 2

Summary of results of alignment analyses of EQAO Primary Assessments 2007-2011

Strands and Sub-Strands % of Assessments with

acceptable Categorical Concurrence

% of Assessments with acceptable Range of

Knowledge

% of Assessments with acceptable Balance of

Representation NSN: Number Sense and Numeration 100 80 100

NSN1: Quantity Relationships 0 0 100 NSN2: Counting 0 100 100

NSN3: Operational Sense 100 100 100 M: Measurement 100 20 100

M1: Attributes, Units, and Meas. Sense 60 40 100 M2: Measurement Relationships 0 60 100

GSS: Geometry and Spatial Sense 60 20 100 GSS1: Geometric Properties 0 20 100

GSS2: Geometric Relationships 0 0 100 GSS3: Location and Movement 0 80 100

PA: Patterning and Algebra 100 80 100 PA1: Patterns and Relationships 40 80 100

PA2: Variables, Expressions, and Equations 0 60 100 DMP: Data Management and Probability 40 60 100

DMP1: Collection and Organization of Data 0 20 100 DMP2: Data Relationships 0 40 100

DMP3: Probability 0 100 100

Note: N=5



16

Table 3

Summary of results of alignment analyses of EQAO Junior Assessments 2007-2011

Strands and Sub-Strands % of Assessments with acceptable Categorical

Concurrence

% of Assessments with acceptable Range of

Knowledge

% of Assessments with acceptable Balance of

Representation NSN: Number Sense and Numeration 100 0 100

NSN1: Quantity Relationships 0 20 100 NSN3: Operational Sense 0 0 100

NSN4: Proportional Relationships 0 100 100 M: Measurement 100 100 100

M1: Attributes, Units, and Meas. Sense 0 80 100 M2: Measurement Relationships 100 100 100

GSS: Geometry and Spatial Sense 100 100 100 GSS1: Geometric Properties 0 100 100

GSS2: Geometric Relationships 0 80 100 GSS3: Location and Movement 0 100 100

PA: Patterning and Algebra 100 100 100 PA1: Patterns and Relationships 40 80 100

PA2: Variables, Expressions, and Equations 0 60 100 DMP: Data Management and Probability 100 60 100

DMP1: Collection and Organization of Data 0 40 100 DMP2: Data Relationships 0 20 100

DMP3: Probability 0 60 100

Note: N=5



25

Total Hits

Because multiple expectations could correspond to a single test item, the total

number of hits was greater than 36, the number of assessment items. The total number of

hits ranged from 43 to 49 on Primary assessments and from 39 to 42 on Junior

assessments (see Table 4). The greater number of hits for the primary-level assessments

is due to two factors. First, less knowledge is assumed at the grade 3 level than at the

grade 6 level and expectations are more specific at the lower level. Second, six questions

on each of the Primary assessments are to be done without access to calculators so that an

additional expectation related to computation are often matched to these questions;

calculator use is permitted throughout the Junior assessment.

Table 4

Total number of hits on from the analysis of each assessment

Assessment Year 2007 2008 2009 2010 2011 Total # of Hits on Primary Assess. 46 44 43 43 49 Total # of Hits on Junior Assess. 40 42 40 39 39

Categorical Concurrence

Only the 2011 Primary Assessment achieved an acceptable level of categorical

concurrence for all of the strands; each of the other Primary assessments had at least one

strand with six or fewer hits (see Tables A1.1 to A1.5 in Appendix 1). All of the Primary

assessments had acceptable categorical concurrence levels for the Number Sense and

Numeration, Measurement, and Patterning and Algebra strands while only 3 of the 5 of

the tests had satisfactory levels for the Geometry and Spatial Sense strand and only 2 had

satisfactory levels for the Data Management and Probability Strand. Only three of the

sub-strands had acceptable categorical concurrence on any of the Primary assessments:



26

the Operational Sense, Attributes, Units, and Measurement Sense, and Patterns and

Relationships sub-strands (see Table 2).

All of the Junior assessments achieved acceptable levels of categorical

concurrence on all of the strands (see Tables A1.6 to A1.10 in Appendix 1). Only two of

the sub-strands had acceptable categorical concurrence on any of the Junior assessments:

the Measurement Relationships and the Patterns and Relationships sub-strands (see Table

3).

The main reason why categorical concurrence was rarely achieved on the sub-

strand level is simply that there were not enough assessment items to adequately measure

each of the thirteen sub-strands. Although multiple expectations were matched to some

questions so that the number of hits was greater than thirty-six, there were not enough

hits to adequately cover all of the sub-strands (see Table 4).

Range-of-Knowledge Correspondence

No Primary assessment met the range-of-knowledge criterion for all strands; the

number of strands for which the criterion was met ranged from one to four (see Tables

A1.11 to A1.15 in Appendix 1). Only one primary-level assessment met this criterion for

either the Measurement or the Geometry and Spatial Sense strands while all but one met

the criterion for the Number Sense and Numeration strand (Table 2). Satisfactory range-

of-knowledge was attained unevenly at the sub-strand level (see Table 3 for details).

No Junior assessment met the range-of-knowledge criterion for all strands (see

Tables A1.16 to A1.20 in Appendix 1). Two of the assessments attained a satisfactory

range-of-knowledge on 3 of 5 strands, failing to meet the criterion on the Number Sense

and Numeration and Geometry and Spatial Sense strands. The remaining three



27

assessments attained a satisfactory level on 4 of the 5 strands, only failing to meet the

criterion on the Number Sense and Numeration strand. See Table 3 for information about

range-of-knowledge attainment at the sub-strand level.

Balance of Representation

All assessments studied met the balance of representation criterion for all strands

and sub-strands. Balance indices ranged from 0.71 to 1 on the Primary assessments and

from 0.73 to 1 on the Junior assessments; see Tables A1.11 to A1.20 in the Appendix for

details.

Comparison of curricula defined by EQAO tests, The Ontario Curriculum, and

textbooks

A summary of the strand-by-strand content from the 2007 to 2011 Primary and

Junior assessments is shown in Figures A2.1 and A2.2 in Appendix 2. These charts also

display the content of the standards (as measured by number of expectations per strand)

and the content of the textbooks studied (as measured by number of sections per strand).

On the strand level, the content of each of the primary-level assessments was

closely aligned to both the textbook examined and to the grade 3 curriculum (Tables A2.1

and A2.2). The most notable differences in content were in the Number Sense and

Numeration strand, which was emphasized more on four of the primary tests than in the

curriculum, and in the Measurement strand, which was emphasized less on three of the

primary tests than in the textbook.

The content of each of the junior-level assessments was closely aligned with the

grade 6 curriculum on the strand level (see Table A2.3). There were no consistent

differences between content of the tests and the curriculum. There were, however,



28

significant differences between the content of the assessments and the grade 6 textbooks

examined (see Table A2.4; chi-square values ranged from 14.806 to 27.125 which were

significant at p=0.01). The primary discrepancy between the texts and the assessments

occurred in the Number Sense and Numeration strand: the textbooks emphasized this

strand far more than the assessments.

Mathematical Integrity of Assessment Items

Several of the assessment items use imprecise language or unclear instructions to

an extent that the question is either mathematically incorrect or it could be interpreted in

ways other than those considered in the Scoring Guide. Table 4 contains a summary of

such problematic items; I will explain the content of this table below. Overall, 18 out of

the 360 problems examined, or about 5%, had problematic mathematical content.

Table 4

Summary of assessment items with problematic content

Assessment Question Category of Concern

Area of Concern

1. 2007 Primary 7 Imprecise wording

All spinners could be generate the shown data

2. 9 Unclear instructions

Necessary ‘justification’ is unclear (see Scoring Guide)

3. 28 Pattern Cannot just extend pattern based on that shown in table

4. 2008 Primary 20 Imprecise wording

Confusing and imprecise wording

5. 30 Unclear instructions

Is not clear how many crayons are in each box

6. 2009 Primary 5 Pattern No description of rule to extend pattern 7. 7 Pattern Geometric situation unclear;

incomplete pattern rule 8. 29 Imprecise

wording Multiple shortest paths are possible; asks for “the” path

9. 2010 Primary 2 Pattern No description of rule to extend pattern 10. 2011 Primary 1 Pattern No description of rule to extend pattern 11. 9 Unclear

instructions Meaning of “on the grid lines” unclear



29

12. 2008 Junior 2 Imprecise wording

Should state “In what time could Joseph’s friend…”

13. 6 Pattern No description of rule to extend pattern 14. 28 Significant

mathematical error

Insufficient information given to be able to answer question; incorrect answers accepted (see Scoring Guide)

15. 2010 Junior 15 Pattern No description of rule to extend pattern 16. 26 Pattern No description of rule to extend pattern 17. 2011 Junior 1 Pattern No description of rule to extend pattern 18. 25 Pattern No description of rule to extend pattern

Small differences in wording are important in mathematics; this precision was lost

in some of the assessment questions, making statements incorrect. For example, question

29 of the 2009 Primary Assessment asks students to "draw the shortest path he can take

from school to the park and then to his house" when there is more than one shortest path

that satisfies the given conditions (see Figure A3.1). This type of mistake may confuse

students who see that there are multiple correct answers to the problem. Another

example of imprecise wording is found in question 7 on the 2007 Primary Assessment

which asks which spinner "could be" used when, in fact, all of the spinners could produce

the data in the table (see Figure A3.2); this question should instead ask which spinner is

most likely to produce the data in the table. A third example of imprecise wording is

found in question 2 of the 2008 Junior Assessment which asks "in what time does

Joseph's friend swim the race" when there are multiple times possible (see Figure A3.3);

the question should instead ask “which of the following could be the time that Joseph’s

friend swims the race.”

Other questions are problematic because the instructions or conditions in the

problem are vague. Although question 9 on the 2011 Primary Assessment asks students

to draw the shortest path "on the grid lines" it is not clear whether this refers to the actual



30

lines themselves or the entire area of the grid (see Figure A3.4); the Scoring Guide

reveals that students were required to draw a path that stays along the lines. Question 7

on the 2009 Primary Assessment simply states that "Sally is making triangles using

straws" but provides no information about how she is making those triangles or whether

she keeps making triangles in the same way (see Figure A3.5). The triangles may either

be disjoint, as the test writers assumed when writing this question, or they may share

sides; in fact, the corresponding expectation on forming sequences from geometric

sequences in the curriculum includes a similar problem where the shapes are assumed to

share sides. Other vague questions include problem 28 on the 2007 Primary Assessment

and question 30 on the 2008 Primary Assessment.

Short answer questions usually require students to justify their answer but what is

expected of a justification is unclear and at times uneven. For example, both question 9

of the 2007 Primary Assessment and question 25 of the 2010 Primary Assessment asks

students to find lines of symmetry and to "justify your answer." According to the scoring

guides, students who wrote the test in 2010 only had to show lines of symmetry while

students who wrote the test in 2007 were also required to give a definition of a line of

symmetry. This is problematic for teachers who look to scoring guides from past tests to

prepare their students for EQAO assessments.

Even students who had mastered the grade 6 curriculum would have a very

difficult time giving a correct answer to question 28 on the 2008 Junior Assessment (see

Figure A3.6) and, in fact, the example given in the Scoring Guide for demonstrating “a

complete solution process” is not complete. According to the scoring guide, a student

who wrote out the results of the rule for polygons with 3 to 8 sides with no justification



31

related to the geometry was given full credit while a student who gave it only for

quadrilaterals is not. Each of these solutions is equally incorrect mathematically; it seems

as though the test writers are willing to perpetuate the misunderstanding that showing

'enough' examples constitutes a proof.

There is a second difficulty with Question 28 of the 2008 Junior Assessment: it

assumes that patterns are determined by their first few terms. While this is true when you

restrict your attention to certain types of patterns, such as those formed by geometric or

arithmetic sequences, not all patterns are of this sort; in fact one could make many

reasonable patterns out of any finite number sequence. Several questions on the

assessments studied make the same incorrect assumption. For example, question 25 on

the 2011 Junior Assessment assumes that because two tokens are earned per task for up

to five tasks that two tokens will be earned per task for other numbers of tasks (see Figure

A3.7). It is clear, however, that many other situations are possible under the conditions

given: for example, Cole might earn a bonus for every eight tasks completed or he may

only be paid for the first 5 tasks that he completes, earning 10 tokens for completing 5 or

greater tasks. See Table 4 for references to other questions with similar problems.

Discussion

EQAO tests are designed to follow The Ontario Curriculum and it is clear that the

writers pay careful attention to making sure that all strands are represented on the tests.

However, a deeper analysis of alignment, attending to sub-strands and to specific

standards, reveals problems. Further, there are difficulties with the language or

mathematics on about 5 percent of assessment items studied.

The EQAO assessments evaluated are very well matched to the curriculum at a



32

strand level. As the comparison of textbook, curriculum, and assessment content showed,

the content of the assessments is much more closely aligned to curriculum than it is to the

textbooks. They test content from the curriculum at an appropriate grade level and do not

contain questions that are either too basic or too advanced for students of a particular age.

Further, assessments consistently met the categorical concurrence criterion at the strand

level, demonstrating that the EQAO assessments are able to measure students’ mastery of

each of the strands. In a similar vein, the EQAO assessments are well-balanced, in that

they do not significantly emphasize one expectation over another from year-to-year.

These observations indicate that the assessments are truly based on the Ontario

curriculum.

None of the assessments met the categorical concurrence criterion on the sub-

strand level or the range of knowledge criterion on either the strand or the sub-strand

level. It can be argued that this is primarily due to the length of the assessments: there

are thirteen sub-strands subdivided into 66 expectations in the grade 3 curriculum and 61

expectations in the grade 6 curriculum and only 36 assessment items on each test. A test

with 36 items based on The Ontario Curriculum would need to have more than two

expectations corresponding to every single item to have a shot at meeting both of these

alignment criteria. Even under these conditions, the objectives would have to be

perfectly balanced throughout the curriculum. Therefore, if the assessments are to truly

measure mastery of the breadth of the primary and junior curricula, it is necessary to

either create assessments with a greater number of questions or to design more items that

touch on several areas of the curricula.

The structure and the testing conditions of the EQAO assessments limit which



33

expectations and aspects of expectations can be authentically measured and limits the

ability for the tests to meet the range of knowledge criterion. A written assessment

completed individually in a classroom over a short amount of time simply cannot

measure the mastery of certain expectations, such as those involving actual collection of

data, ability to do mental arithmetic, or explorations through experiments. Detecting true

understanding of even seemingly straightforward expectations, such as understanding the

meaning of the 'whole' in a fraction, can be impossible without a more interactive form of

assessment (Ball & Peoples, 2007). Further, the fact that students have access to

calculators throughout the entire Junior Assessment means that none of the computational

fluency objectives at this level can be measured.

Attempting to address some of these difficult-to-measure expectations has

resulted in some awkward EQAO assessment items. A better solution would be to

modify the testing conditions by limiting calculator use on the grade 6 assessment,

including oral questions to test mental arithmetic, and adding an interview component to

more accurately assess conceptual understanding.10 Clearly, some of these changes are

more feasible than others; where changes are not possible, it should be acknowledged that

certain parts of the curriculum cannot be honestly assessed by the EQAO tests.

Five percent of the EQAO assessment items evaluated were not entirely

mathematically sound. Precision is one of the most important aspects of mathematics and

is a key characteristic that sets it apart from other disciplines; even if these errors do not

impact overall scores, allowing precision to slip on such a high-profile assessment is

unacceptable. Even the youngest students deserve an intellectually honest presentation of

10 Writing multiple tests to be given to different groups of students will not solve the alignment problem since individual progress of students is also tracked.



34

each subject that they study and the vague definitions, incorrect statements, and unclear

instructions that have appeared on the assessments have no place in mathematics. To

ensure that these issues do not continue to appear, trained mathematicians must be

meaningfully involved in every stage of the EQAO testing process, from drafting and

editing test questions to grading students' responses.11

Assessment design as much an art as it is a science and it is exceedingly difficult

to craft perfect assessments. However, given the increasing prominence and importance

of standards-based assessments, it is incumbent on organizations such as the EQAO to

write the best tests possible. This report has highlighted some aspects of Ontario's large-

scale elementary assessment that are commendable and others that can be improved with

changes to the structure of the tests, to the conditions under which the tests are taken, and

to the assessment writing and grading process. With these adjustments, the EQAO will

be able to provide more accurate information and send clearer messages to students,

teachers, and the public and will improve education accountability in Ontario.

11 I could not find any evidence that mathematicians are involved in the writing of the EQAO assessments.



35

References

Ball, D. L. & Peoples, B. (2007). Assessing a student’s mathematical knowledge by way

of interview. In A. Schoenfeld (Ed.), Assessing Mathematical Proficiency. New

York: Cambridge University Press.

Chandler, D. G., & Bronsan, P. A. (1995). A comparison between mathematics textbook

content and a statewide mathematics proficiency test. School Science and

Mathematics, 95(3), 118-123.

Education Quality and Accountability Office (2012). The Power of Ontario’s Provincial

Testing Program. Toronto: Queen’s Printer for Ontario.

Farenga, S. J., Joyce, B. A., & Ness, D. (2002). Reaching the zone of optimal learning:

The alignment of curriculum, instruction, and assessment. In R.W. Bybee (Ed.),

Learning science and the science of learning. Arlington, VA: NSTA Press.

Herman, J. L. (2004). The effects of testing on instruction. In S. Furhman & R. F.

Elmore (Eds.), Redesigning accountability systems for education (pp. 141-166).

New York: Teachers College Press.

Herman, J.L., Webb, N. L., Zuniga, S. A. (2007). Measurement issues in the alignment

of standards and assessments: A case study. Journal of Applied Measurement in

Education, 20(1), 101-126.

Kozlow, M. (2007). Model Selection for the Analysis of EQAO Assessment Data.

EQAO Research Bulletin #1. Retrieved from

http://www.eqao.com/Research/pdf/E/1_Research_Bulletin_1207_web.pdf.

La Marca, P. M. (2001). Alignment of standards and assessments as accountability

criterion. Practical Assessment, Research & Evaluation, 7(21).



36

Mohamud, A. & Fleck, D. (2010). Alignment of standards, assessment, and instruction:

Implications for English Language Learners in Ohio. Theory Into Practice, 49(2),

129-136.

Näsström, G. & Henriksson, W. (2008). Alignment of standards and assessments: a

theoretical and empirical study of methods for alignment. Electronic Journal for

Research in Educational Psychology, 16(6), 667-690.

Ontario Ministry of Education (2005). The Ontario Curriculum, Grades 1-8:

Mathematics. Toronto: Queen’s Printer for Ontario.

Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice

(AERA 2002 Presidential Address). Educational Researcher, 31(7), 3-14.

Roach, A. T., Elliott, S. N., & Webb, N. L. (2005). Alignment of an alternate assessment

with state academic standards: The validity of the Wisconsin Alternate Assessment.

Journal of Special Education, 38(4), 218-231.

Rothman, R. (2003). Imperfect matches: The alignment of standards and tests. Paper

commissioned by the Committee on Test Design for K-12 Science Achievement,

March 2003.

Webb, N.L. (1999). Alignment of science and mathematics standards and assessments in

four states (Research monograph, No. 18). Madison: National Institute for Science

Education.

Webb, N. L., 2005. Web Alignment Tool (WAT) Training Manual. Retrieved February 12,

2012. Available: http://www.wcer.wisc.edu/wat/index.aspx.



37

Wixson, K. K., Fisk, M. C., Dutro, E., & McDaniel, J. (2002). The alignment of state

standards and assessments in elementary reading. CIERA Report #3-024. Ann

Arbor: University of Michigan, School of Education, Center for the Improvement

of Early Reading Achievement.



38

Appendix 1

Detailed Results of Alignment Analyses

Table A1.1

Categorical Concurrence: Primary Assessment 2007

Strands and Sub-Strands Expect. # a

Hitsb Categorical Concurr.

Acceptablec

NSN: Number Sense and Numeration 19 18 ● NSN1: Quantity Relationships 10 3 ❍

NSN2: Counting 2 3 ❍ NSN3: Operational Sense 7d 12 ●

M: Measurement 16 11 ● M1: Attributes, Units, and Meas. Sense 10 6 ●

M2: Measurement Relationships 6 5 ❍ GSS: Geometry and Spatial Sense 13 5 ❍

GSS1: Geometric Properties 5 1 ❍ GSS2: Geometric Relationships 5 2 ❍ GSS3: Location and Movement 3 2 ❍

PA: Patterning and Algebra 10 6 ● PA1: Patterns and Relationships 6 3 ❍

PA2: Expressions and Equality 4 3 ❍ DMP: Data Management and Probability 8 6 ●

DMP1: Collection and Organization of Data 3 2 ❍ DMP2: Data Relationships 3 1 ❍

DMP3: Probability 2 3 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 3 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met. d Includes one additional expectation because coded items did not correspond to existing expectations (see the Methods section).



39

Table A1.2


Strands and Sub-Strands Expect.

#a Hitsb Categorical

Concurr. Acceptablec




M2: Measurement Relationships 6 3 ❍ GSS: Geometry and Spatial Sense 13 7 ●


PA: Patterning and Algebra 10 7 ● PA1: Patterns and Relationships 6 6 ●

PA2: Expressions and Equality 4 1 ❍ DMP: Data Management and Probability 8 5 ❍





40

Table A1.3







M: Measurement 16 8 ● M1: Attributes, Units, and Meas. Sense 10 5 ❍

M2: Measurement Relationships 6 3 ❍ GSS: Geometry and Spatial Sense 13 4 ❍








41

Table A1.4
















42

Table A1.5 Categorical Concurrence: Primary Assessment 2011










PA2: Expressions and Equality 4 1 ❍ DMP: Data Management and Probability 8 6 ●





43

Table A1.6 Categorical Concurrence: Junior Assessment 2007





NSN3: Operational Sense 8 2 ❍ NSN4: Proportional Relationships 3 4 ❍


M2: Measurement Relationships 10 8 ● GSS: Geometry and Spatial Sense 9 8 ●



PA2: Variables, Expressions, and Equations 4 5 ❍ DMP: Data Management and Probability 12 7 ●


DMP3: Probability 3 3 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 6 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met.



44

Table A1.7

Categorical Concurrence: Junior Assessment 2008















45

Table A1.8
















46

Table A1.9
















47

Table A1.10
















48

Table A1.11

Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2007

Strands and Sub-Strands Range of Objectives Range of

Know. Accept.c

Balance Indexd

Balance of Rep.

Accept.e Title Expect. # # Hita % of

Totalb

NSN1: Number Sense and Numeration 19 9 50 ● 0.83 ● NSN1: Quantity Relationships 10 2 20 ❍ 0.83 ●

NSN2: Counting 2 2 100 ● 0.83 ● NSN3: Operational Sense 7 5 71 ● 0.82 ●

M: Measurement 16 9 56 ● 0.94 ● M1: Attributes, Units, and Meas. Sense 10 4 40 ❍ 0.83 ●

M2: Measurement Relationships 6 5 83 ● 1 ● GSS: Geometry and Spatial Sense 13 4 31 ❍ 0.85 ●

GSS1: Geometric Properties 5 1 20 ❍ 1 ● GSS2: Geometric Relationships 5 1 20 ❍ 1 ● GSS3: Location and Movement 3 2 66 ● 1 ●

PA: Patterning and Algebra 10 5 50 ● 0.87 ● PA1: Patterns and Relationships 6 3 50 ● 1 ●

PA2: Expressions and Equality 4 2 50 ● 0.83 ● DMP: Data Management and Probability 8 4 50 ● 0.75 ● DMP1: Collection and Organization of Data 3 2 66 ● 1 ●

DMP2: Data Relationships 3 1 33 ❍ 1 ● DMP3: Probability 2 1 50 ● 1 ●

a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.



49

Table A1.12 Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2008


Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb



M: Measurement 16 7 44 ❍ 0.84 ● M1: Attributes, Units, and Meas. Sense 10 4 40 ❍ 0.83 ●

M2: Measurement Relationships 6 3 50 ● 1 ● GSS: Geometry and Spatial Sense 13 7 54 ● 1 ●

GSS1: Geometric Properties 5 4 80 ● 1 ● GSS2: Geometric Relationships 5 1 20 ❍ 1 ● GSS3: Location and Movement 3 2 66 ● 1 ●

PA: Patterning and Algebra 10 5 50 ● 0.83 ● PA1: Patterns and Relationships 6 4 66 ● 0.83 ●

PA2: Expressions and Equality 4 1 25 ❍ 1 ● DMP: Data Management and Probability 8 3 38 ❍ 0.87 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●





50



Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb




M2: Measurement Relationships 6 3 50 ● 1 ● GSS: Geometry and Spatial Sense 13 4 31 ❍ 1 ●

GSS1: Geometric Properties 5 1 20 ❍ 1 ● GSS2: Geometric Relationships 5 2 40 ❍ 1 ● GSS3: Location and Movement 3 1 33 ❍ 1 ●


PA2: Expressions and Equality 4 2 50 ● 0.83 ● DMP: Data Management and Probability 8 4 50 ● 0.87 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●

DMP2: Data Relationships 3 2 66 ● 1 ● DMP3: Probability 2 1 50 ● 1 ●




51

Table A1.14

Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2010


Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb


NSN2: Counting 2 1 50 ● 1 ● NSN3: Operational Sense 7 6 86 ● 0.8 ●

M: Measurement 16 7 44 ❍ 0.84 ● M1: Attributes, Units, and Meas. Sense 10 5 50 ● 1 ●

M2: Measurement Relationships 6 2 33 ❍ 1 ● GSS: Geometry and Spatial Sense 13 5 38 ❍ 0.83 ●


PA: Patterning and Algebra 10 4 40 ❍ 0.79 ● PA1: Patterns and Relationships 6 2 33 ❍ 0.9 ●

PA2: Expressions and Equality 4 2 50 ● 1 ● DMP: Data Management and Probability 8 3 38 ❍ 0.83 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●





52



Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb

NSN1: Number Sense and Numeration 19 7 37 ❍ 0.71 ● NSN1: Quantity Relationships 10 1 10 ❍ 1 ●

NSN2: Counting 2 1 50 ● 1 ● NSN3: Operational Sense 7 5 71 ● 0.79 ●


M2: Measurement Relationships 6 1 17 ❍ 1 ● GSS: Geometry and Spatial Sense 13 5 38 ❍ 0.85 ●



PA2: Expressions and Equality 4 1 25 ❍ 1 ● DMP: Data Management and Probability 8 5 63 ● 0.87 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●

DMP2: Data Relationships 3 2 66 ● 1 ● DMP3: Probability 2 2 100 ● 1 ●




53

Table A1.16 Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2007


Know. Accept.c

Balance Indexd

Balance of Rep.

Accept.e Title Expec. # # Hita % of

Totalb


NSN3: Operational Sense 8 2 25 ❍ 1 ● NSN4: Proportional Relationships 3 2 67 ● 1 ●

M: Measurement 12 9 75 ● 0.91 ● M1: Attributes, Units, and Meas. Sense 2 2 100 ● 1 ●

M2: Measurement Relationships 10 7 70 ● 0.89 ● GSS: Geometry and Spatial Sense 9 6 67 ● 0.88 ●

GSS1: Geometric Properties 4 2 50 ● 0.83 ● GSS2: Geometric Relationships 2 1 50 ● 1 ● GSS3: Location and Movement 3 3 100 ● 0.88 ●

PA: Patterning and Algebra 10 5 50 ● 0.77 ● PA1: Patterns and Relationships 6 2 33 ❍ 1 ●

PA2: Variables, Expressions, and Equations 4 3 75 ● 0.73 ● DMP: Data Management and Probability 12 5 42 ❍ 0.77 ● DMP1: Collection and Organization of Data 4 0 0 ❍ N/A ●

DMP2: Data Relationships 5 4 80 ● 1 ● DMP3: Probability 3 1 33 ❍ 1 ●




54


Strands and Sub-Strands Range of Objectives Range of Know.

Accept.c

Balance Indexd

Balance of Rep.


Totalb

NSN1: Number Sense and Numeration 18 8 44 ❍ 0.90 ● NSN1: Quantity Relationships 7 4 57 ● 1 ●

NSN3: Operational Sense 8 2 25 ❍ 1 ● NSN4: Proportional Relationships 3 2 66 ● 0.83 ●



GSS1: Geometric Properties 4 2 50 ● 1 ● GSS2: Geometric Relationships 2 1 50 ● 1 ● GSS3: Location and Movement 3 2 66 ● 0.75 ●


PA2: Variables, Expressions, and Equations 4 2 50 ● 0.83 ● DMP: Data Management and Probability 12 6 50 ● 0.88 ● DMP1: Collection and Organization of Data 4 1 25 ❍ 1 ●

DMP2: Data Relationships 5 2 40 ❍ 0.83 ● DMP3: Probability 3 3 100 ● 1 ●




55

Table A1.18

Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2009


Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb

NSN1: Number Sense and Numeration 18 7 39 ❍ 0.84 ● NSN1: Quantity Relationships 7 2 29 ❍ 0.83 ●

NSN3: Operational Sense 8 2 25 ❍ 0.83 ● NSN4: Proportional Relationships 3 3 100 ● 1 ●

M: Measurement 12 7 58 ● 0.81 ● M1: Attributes, Units, and Meas. Sense 2 0 0 ❍ N/A ●


GSS1: Geometric Properties 4 2 50 ● 1 ● GSS2: Geometric Relationships 2 0 0 ❍ N/A ● GSS3: Location and Movement 3 3 100 ● 0.73 ●


PA2: Variables, Expressions, and Equations 4 1 25 ❍ 1 ● DMP: Data Management and Probability 12 6 50 ● 0.83 ● DMP1: Collection and Organization of Data 4 2 50 ● 0.83 ●





56



Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb


NSN3: Operational Sense 8 2 25 ❍ 0.83 ● NSN4: Proportional Relationships 3 3 100 ● 0.83 ●





PA2: Variables, Expressions, and Equations 4 1 25 ❍ 1 ● DMP: Data Management and Probability 12 3 25 ❍ 0.83 ● DMP1: Collection and Organization of Data 4 0 0 ❍ N/A ●

DMP2: Data Relationships 5 2 40 ❍ 0.75 ● DMP3: Probability 3 1 33 ❍ 1 ●




57



Know. Accept.c

Balance Indexd

Balance of Rep.


Totalb


NSN3: Operational Sense 8 3 38 ❍ 0.87 ● NSN4: Proportional Relationships 3 2 66 ● 1 ●





PA2: Variables, Expressions, and Equations 4 3 75 ● 1 ● DMP: Data Management and Probability 12 6 50 ● 0.78 ● DMP1: Collection and Organization of Data 4 2 50 ● 1 ●

DMP2: Data Relationships 5 2 40 ❍ 0.83 ● DMP3: Probability 3 2 66 ● 0.75 ●




58

Appendix 2

Data comparing curricula defined by EQAO tests, The Ontario Curriculum, and

textbooks

Figure A2.1. Percentage of total hits on the 2007-2011 EQAO primary-level mathematics

assessments from each strand versus the percentage of grade 3 curriculum expectations

and textbook sections from each strand.

0

5

10

15

20

25

30

35

40

45

Number Sense

and Numeration

Measurement Patterning and

Algebra

Geometry and

Spatial Sense

Data Management

and Probability

Percent

EQAO Primary Assessments Curriculum Expectations Overall Textbook Content



59

Figure A2.2. Percentage of total hits on the 2007-2011 EQAO junior-level mathematics

assessments from each strand versus the percentage of grade 6 curriculum expectations

and textbook sections from each strand.

0

5

10

15

20

25

30

35

40

45

50

Number Sense

and Numeration

Measurement Patterning and

Algebra

Geometry and

Spatial Sense

Data

Management and

Probability

Percent

EQAO Junior Assessments Curriculum Expectations Overall Textbook Content



60

Table A2.1

Comparison between the content on the primary-level assessments by strands and the

content expected based upon the content of the curriculum

NSNa Mb GSSc PAd DMPe χ2 valuef

2007 +g 0h - 0 0 4.328 2008 0 0 0 0 0 1.939 2009 + 0 - 0 0 8.735 2010 + 0 0 0 0 2.307 2011 + 0 0 0 0 6.336 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.



61

Table A2.2

Comparison between the content on the primary-level assessments by strands and the

content expected based upon the content of textbooks


2007 0g 0 0 0 0 0.885 2008 0 -f 0 0 0 1.581 2009 0 - 0 0 0 2.339 2010 0 0 0 0 0 1.733 2011 0 - 0 0 0 2.191 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.



62

Table A2.3

Comparison between the content on the junior-level assessments by strands and the

content expected based upon the content of the curriculum


2007 -f 0 0 0 0 1.884 2008 0g 0 0 + 0 5.166 2009 0 0 0 0 + 2.381 2010 0 0 0 0 0 0.855 2011 0 0 0 0 + 4.636 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.



63

Table A2.4

Comparison between the content on the junior-level assessments by strands and the

content expected based upon the content of textbooks


2007 -f 0g 0 0 0 18.638 2008 - 0 0 0 0 27.125 2009 - 0 0 0 0 14.806 2010 - 0 0 0 0 15.023 2011 - 0 0 0 0 18.416 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.



64

Appendix 3

Notable Assessment Items

Figure A3.1: 2009 Primary Assessment, Question 29




65

Figure A3.3: 2008 Junior Assessment, Question 2




66




67





68

EQAO Alignment Analysis - University of Michiganmayess/Sarah_Mayes:_Teaching...The EQAO Primary and...

Documents

Transcript of EQAO Alignment Analysis - University of Michiganmayess/Sarah_Mayes:_Teaching...The EQAO Primary and...