PCAP 2010 - TECHNICAL REPORT PART I

Chapter 2

DEVELOPMENT OF THE ASSESSMENT INSTRUMENT

Table of Contents

SECTION I: Development of the assessment booklets................................................................31. Test format ..........................................................................................................................32. Task characteristics .............................................................................................................43. Item characteristics..............................................................................................................44. Administration time, documents, and specific considerations...............................................55. Mathematics assessment framework....................................................................................6

5.1. Subdomains of the mathematics assessment..................................................................65.2. Processes ......................................................................................................................95.3. Cognitive levels ..........................................................................................................10

6. Science framework............................................................................................................126.1. Competencies .............................................................................................................136.2. Subdomains ................................................................................................................13

7. Reading frameworks..........................................................................................................157.1. Text types and forms ..................................................................................................157.2. Subdomains of reading ...............................................................................................16

8. Linking..............................................................................................................................179. Working group ..................................................................................................................1710. Item-writing session ........................................................................................................1711. Item editing .....................................................................................................................18

11.1. Translation and comparison of items in English and French......................................1811.2. Editing for language and style...................................................................................1811.3. Scientific editing.......................................................................................................1911.4. Psychometric editing.................................................................................................19

12. Item approval by the jurisdictions....................................................................................20SECTION II: Questionnaire development..................................................................................21

1. Working group ..................................................................................................................212. Initial questionnaire framework and guiding principles......................................................213. Core questions...................................................................................................................22

3.1. Gender differences in mathematics .............................................................................223.2. Time allocation and use ..............................................................................................233.3. Special needs ..............................................................................................................233.4. Assessment.................................................................................................................233.5. Attitudes⁄motivations ..................................................................................................243.6. Student learning strategies ..........................................................................................243.7. Teaching strategies .....................................................................................................243.8. Opportunity to learn....................................................................................................25

4. Item types..........................................................................................................................25

2

5. Number of items................................................................................................................256. Item writing.......................................................................................................................26

6.1. Student Questionnaire.................................................................................................266.2. Teacher Questionnaire ................................................................................................266.3. School Questionnaire..................................................................................................27

3

DEVELOPMENT OF THE ASSESSMENT INSTRUMENT

SECTION I: Development of the assessment booklets

As of 2010, it was determined that PCAP would be administered to Grade 81 students. Development of the assessment instrument is an important stage in the project to ensure that the items in the assessment booklets correctly assess the skills of Grade 8 students across Canada. The following sections will present the various stages of development of the assessment booklets for the three subjects assessed by PCAP 2010: mathematics, science, and reading. They will therefore provide more detailed information on the test format, administration time, assessment frameworks, working group, and drafting and editing of items.

1. Test format

The PCAP assessment, a paper-and-pencil assessment, covers three assessment domains: mathematics, science, and reading. Mathemathics was the major domain, while science and reading were minor domains in the 2010 main administration. Just as with Programme for International Student Assessment (PISA), the focus changes with each assessment, with mathematics becoming a minor domain and science becoming the major domain in the PCAP main administration in 2013.

The PCAP 2010 assessment in mathematics was organized into eight groups, or clusters, with scenarios requiring the engagement of multiple mathematics strands and processes. The eight clusters were distributed within four booklets. Each booklet contained two clusters of mathematics items, one reading cluster, and one science cluster. The four booklets were distributed to students within a single class. Thus, every student completed two of the eight mathematics clusters of assessment items. In addition, all booklets contained a set of common items allowing for comparative measurement of student performance from one booklet to another. All the assessment booklets contained a Student Questionnaire at the end of the assessment booklet.

Each PCAP mathematics cluster was composed of three to four scenarios with items spanning four different subdomains (as described below in sub-section 5 “Mathematics assessment framework”). Each scenario comprised one to six items assessing the various concepts and skills that are taught in mathematics and focused on their relevance for the context of the assessment cluster. The clusters contained selected-response items and constructed-response items. The number of items per cluster varied slightly, depending on the distribution of item types in the cluster. No cluster contained only one type of item. Table 1 shows the distribution of the clusters, scenarios, and items for mathematics, science, and reading across four booklets.

1 Secondary Two in Quebec.

4

Table 1. Number of clusters, scenarios, and items by domain, by booklet

Mathematics Science ReadingClusters Scenarios Items Clusters Scenarios Items Clusters Scenarios Items

Booklet 1 2 8 25 1 3 11 1 2 8Booklet 2 2 8 25 1 2 13 1 2 9Booklet 3 2 6 25 1 1 6 1 2 9Booklet 4 2 7 25 1 2 18 1 1 4

2. Task characteristics

The scenarios were drawn from situations that were considered relevant, appropriate, and sensible for Canadian Grade 8 students. The presentation of a scenario included a brief narrative and could include tables, charts, graphs, or diagrams. The desired effect was that scenarios be relevant to students’ interests and lives, and sensitive to linguistic and culturaldifferences. Some scenarios were taken from students’ personal lives, from school/sports/leisure activities, or, on a larger scale, from the community/society. Most of the scenarios were meant to emulate the world outside of the classroom.

The assessment was designed at a reading level consistent with the literacy level expected of most Canadian Grade 8 students. Information in the items was presented in a variety of modes (e.g., graphically, in tables, symbolically). Because many jurisdictions in Canada assess the performance of both French- and English-language populations, English and French versions of the assessment were developed simultaneously and are considered to be equivalent.

3. Item characteristics

Selected-response (SR) itemsSelected-response (SR) items give the students specific choices from which they must select a response. Questions may appear in different formats: multiple‐choice, true or false, or yes or no. Multiple choice items generally include a stem statement with four choices, one of which is the correct answer, and three of which are carefully constructed distracters.

Constructed-response (CR) itemsConstructed-response (CR) items require responses ranging from single words or phrases to extended, constructed responses of two to three sentences. For mathematics, these responses can also include symbols, numbers, graphs, diagrams, and calculations. Generally, there were two forms of constructed-response items. The “show your work” type asked students to clearly demonstrate how he or she arrived at the final solution to a particular problem. The “explain your reasoning” type asked the student to provide a clear explanation of the processes used to arrive at the solution to the problem.

5

Table 2. Distribution of items by assessment domain and item type: selected response (SR) and constructed response (CR)

Domain Booklet 1 Booklet 2 Booklet 3 Booklet 4SR CR SR CR SR CR SR CR

Mathematics 8 17 7 18 8 17 6 19Science 8 3 11 2 0 6 14 4Reading 0 8 6 3 6 3 1 3

In mathematics, approximately 30 per cent of the questions were selected-response items and approximately 70 per cent of the questions were constructed-response items.

4. Administration time, documents, and specific considerations

Students were allotted 90 minutes to respond to the assessment items. They were entitled to an additional 10 to 15 minutes to complete the test, if necessary. After completing the items in the assessment booklet, students had 30 minutes to answer the Student Questionnaire.

Students were allowed to use the resources to which they normally have access in mathematics, science, and language classes. Guidelines for the use of calculators, computers, and manipulatives during the assessment are given below:

Use of calculatorsThis assessment did not focus on students’ ability to perform calculations but rather on their ability to choose the appropriate operation, to demonstrate their understanding, and to assess the relevance of their answer in a given situation. Consequently, all students were allowed to use a calculator, preferably the type they would normally use in their mathematics class.

Use of computersComputers were not permitted for this assessment. Although computers have become commonplace in all Canadian schools, the large disparity between the types of computers available, their use as a teaching tool, and the students’ familiarity with software could contribute to a biased administration of the assessment if computers were to be permitted.

Use of manipulativesThe use of manipulatives as teaching tools is encouraged by all jurisdictions, and they should be found in all schools. Manipulatives help and support students in developing a better understanding of concepts as they go from concrete to abstract representations. The assessment was designed so that the use of manipulatives was permitted if the student requested them.

6

5. Mathematics assessment framework

The PCAP mathematics assessment framework was developed by consultants, pan-Canadian assessment coordinators, educators who were experts in mathematics, and policy-makers in all jurisdictions. It is informed by the curriculum objectives, goals, and outcomes of the participating populations. As well, it reflects current research findings and best practices in the the learning of mathematics that are in alignment with international trends.

Like most human activities involving knowledge and skills, mathematics requires the integration of the many elements of the field of study when applied in the world at large. While the categorization and organization of mathematics into separate content strands and processes are needed to map the mathematical universe and develop curriculum, the learning and application of mathematics involve linking multiple strands and processes; for example, we use measurement with operations, geometry, and perhaps even algebra, whether we are building a bookshelf or designing a space-shuttle launch. The scope of this assessment was limited to those concepts and skills encountered and used in the courses of study of most Grade 8 students in Canada. Although based on the programs taught to Canadian Grade 8 students, this assessment was not a comprehensive assessment of all concepts and skills that a particular system expects Grade 8 students to master. The purpose of this assessment was to provide the jurisdictions with data to inform educational policy. It was not designed to identify the strengths or weaknesses of individual students, schools, districts, or regions.

Overall, for the purpose of the PCAP 2010 assessment, mathematics was broadly defined as a conceptual tool that students can use to increase their capacity to calculate, describe, and solve problems. The domain was divided into four strands or subdomains and five processes which are described in the following section.

5.1. Subdomains of the mathematics assessment

The PCAP mathematics domain was divided into four strands or subdomains: numbers and operations, geometry and measurement, patterns and relationships, and data management and probability.

Numbers and operations The student was asked to show evidence that he or she can: demonstrate an understanding of the inverse relationship between perfect squares and

square roots, multiplication and division, and addition and subtraction; find the exact square root of numbers that are perfect squares and the approximate square

root of numbers that are not perfect squares; demonstrate an understanding of and find factors for numbers less than 100; find prime factorization of composite numbers and use it to find least common multiples of

numbers less than 100; order and compare positive fractions and positive and negative decimals;

7

generate equivalent expressions for percentages, fractions, and decimals; represent rational numbers with diagrams and on a number line; explain and apply the order of operations for decimals, fractions, and integers; demonstrate an understanding of the four operations (+,-, ×, ÷) on positive fractions,negative

and positive decimals (× and ÷ of decimals limited to two-digit multipliers and one-digit divisors);

demonstrate an understanding of the four operations with integers; select appropriate operations to solve problems involving rational numbers (except negative

fractions) set in contextual situations; describe ways to estimate sums, differences, products, and quotients of positive fractions

and decimals; apply the commutative, associative, and distributive properties, and order of operations to

evaluate mathematical expressions; demonstrate an understanding of percentages greater than or equal to 0%; demonstrate an understanding of proportional relationships using per cent, ratio, and rate; use ratio and proportionality to solve problems involving percentages that arise from real-

life contexts, such as discount, interest, taxes, tips, and per cent increase and decrease; recognize a proportional relationship from context, table of values, and graph and use to

solve contextual problems; solve problems using proportional reasoning in the different subdomains, e.g., numbers and

operations, geometry, probability.

Geometry and measurementThe student was asked to show evidence that he or she can: compare and classify 2-D geometric polygons using appropriate geometric vocabulary and

properties, such as line symmetry, angles, and sides; apply the relationships for intersecting lines, parallel lines and transversals, and the sum of

the angles of a triangle to find the measures of missing angles and solve other problems; demonstrate an understanding of congruence of polygons; draw and describe the image of a combination of translations, rotations, and/or reflections

on a 2-D shape (not on a coordinate plane); identify and plot points in the four quadrants of a Cartesian plane using integral ordered

pairs; demonstrate an understanding of the relationships among radii, diameter, and

circumference of circles and use these relationships to solve problems; calculate the measures of the circumference and area of a circle and use the calculations to

solve contextual problems; calculate the perimeter and the area of triangles, rectangles, and parallelograms and use the

calculations to solve contextual problems; calculate the surface area of right prisms and pyramids and use the calculations to solve

contextual problems; identify, use, and convert among SI units to measure, estimate, and solve problems that

relate to length and area.

8

Patterns and relationshipsThe student was asked to show evidence that he or she can: represent linear patterns and relationships using words, drawings, tables, graphs, algebraic

expressions, and equations; make connections among various representations of linear relationships (words, drawings,

tables, graphs, algebraic expressions, and equations); use different representations of linear patterns and relationships to make generalizations,

predict unknown values, and solve problems; demonstrate an understanding of the different meanings and uses of variables as a place

holder, in rules, in formulae, as changing quantities, and as dependent and independent variables;

translate statements describing mathematical relationships into one or more algebraicexpressions or equations in a variety of contexts;

evaluate algebraic expressions given the value of the variable within the set of rationalnumbers (except negative fractions);

show that two or more expressions are equivalent by using properties such as commutative, associative, distributive, and order of operations;

show that two equations are equivalent by using properties of equality; order of operations; and commutative, associative, and distributive properties;

distinguish between algebraic expressions and algebraic equations; solve linear equations using the most appropriate method (concrete, inspection, trial and

error, and algebraic) involving a one-variable term for integral solutions and to verify solutions;

use linear equations to solve problems involving proportion and measurement problems (area, perimeter, unknown angles of polygons).

Data management and probabilityThe student was asked to show evidence that he or she can: collect data (i.e., formulate questions for investigation; select, justify, and use appropriate

methods of collecting data; evaluate issues such as sampling, biased and unbiased sampling, and the validity of an inference;

organize and display data (i.e., organize data into intervals; select, use, and justify an appropriate representation for displaying relationships among collected data, including circle, line, and bar graphs);

analyze data (i.e., make inferences and convincing arguments about a problem being investigated based on an interpretation and analysis of charts, tables, and graphs used to display given or collected data; evaluate data interpretations that are based on graphs, tables, and charts);

understand measures of central tendency (i.e., describe a set of data and solve problems using mean and range; compare different populations using mean and range; determine the effects of variation in data on measures of central tendency, including outliers, gaps, clusters);

9

understand probability concepts (i.e., identify all possible outcomes of two independent events using tree diagrams, area models, tables, or lists; determine probability of a single or two independent events, and describe using fractions, decimals, or percentages; use the probability of a single or two independent events to make predictions about a population;compare theoretical and experimental probabilities of a single and two independent events in appropriate contexts).

Each item targeted one of the four subdomains described above (i.e., numbers and operations, geometry and measurement, patterns and relationships, or data management and probability). Table 3 displays distribution of mathematics items across booklets according to these four subdomains.

Table 3. Number of mathematics items by subdomain, by booklet

Subdomain Booklet 1 Booklet 2 Booklet 3 Booklet 4Numbers and operations 10 9 12 11Geometry and measurement 5 5 7 7Patterns and relationships 5 6 6 7Data management and probability 5 5 0 0

5.2. Processes

Five processes have been identified in order to highlight ways of acquiring and using the content knowledge outlined above, notably: problem solving, communication, reasoning, representation, and connections. These five processes were interwoven throughout the subdomains of the mathematics assessment.

Problem solvingThe student needed to show evidence that he or she can: solve multi-step problems presented in context that require using and making connections

among mathematical concepts, procedures, and processes; solve multi-step problems presented in context that show evidence of understanding the

problem, making a plan, carrying out the plan, and evaluating the solution for reasonableness;

explain the process used to solve a problem and verify the reasonableness of solutions by using numbers, words, pictures/diagrams, symbols, and equations;

apply a variety of problem-solving strategies to solve problems, such as drawing a picture or diagram, looking for a pattern, using “guess and check,” making a table, working a simpler problem, or working backwards.

10

CommunicationThe student needed to show evidence that he or she can: communicate mathematical ideas and solutions clearly and precisely to others using

appropriate everyday and mathematical language, units of measurement, and a variety of representations (written, graphical, numerical, algebraic);

formulate clear and complete arguments using a variety of representations (written, graphical, numerical, and algebraic) to justify conjectures and solutions to problem situations;

use symbolic language of mathematics correctly.

ReasoningThe student needed to show evidence that he or she can: analyze a problem, make and assess conjectures, justify conclusions, and plan and construct

an organized mathematical argument by applying logical reasoning (inductive, deductive) and mathematical knowledge;

make and test generalizations from patterns and relationships using logical reasoning; use counter-examples to evaluate conjectures; evaluate mathematical arguments; select and use appropriately various types of reasoning (algebraic, geometric, proportional,

probabilistic, statistical, quantitative) to solve problems presented in context.

RepresentationThe student needs to show evidence that he or she can: create and use a variety of representations (written, graphical, numerical, and algebraic) to

organize, record, and communicate mathematical ideas; connect, compare, and translate among different mathematical representations; select and apply the appropriate representations to solve problems.

ConnectionsThe student needs to shows evidence that he or she can: recognize and connect mathematical concepts and procedures to contexts outside of

mathematics, such as other curricular areas, personal life, current events, sports, technology, arts and culture, media;

make connections between different representations (written, graphical, numerical, and algebraic) of mathematical ideas.

5.3. Cognitive levels

The cognitive demands were defined by the reasoning required by the student to correctly answer an item, thus referring to the complexity of mental processing that must occur to answer a question, perform a task, or generate a solution. The three categories of cognitive demands are identified as low, moderate, and high.

11

Cognitive Level I (low)The student can: recall information (facts, procedures, definitions); identify properties; recognize an equivalent representation; perform a specific or routine procedure; solve a one-step (word) problem; retrieve information from a table or graph; identify a simple number or geometric pattern; draw or measure simple geometric figures; recognize an example of a concept; compute a sum/difference/product/quotient; convert among different representations of a number (fraction, decimal, per cent).

Cognitive Level II (moderate)The student can: apply properties to evaluate an expression or find a measurement or solve a problem; represent a situation mathematically in more than one way; select, use, and interpret different representations depending on the situation; solve a contextual problem involving the use of more than one mathematical concept or

procedure; retrieve information from a graph or table or geometric figure and use this information to

solve a problem requiring multiple steps; extend a number or geometric pattern; formulate a routine problem given data and conditions; compare geometric figures or statements; compare two sets of data using the mean and range of each set; organize a set of data and construct an appropriate display; interpret a simple argument; justify a solution to a problem with one solution.

Cognitive Level III (high)The student can: analyze properties; describe how different representations can be used for different purposes; perform procedures having multiple steps and multiple decision points; solve an unfamiliar problem; generalize a pattern and write the rule algebraically; formulate an original problem given a situation; analyze a deductive argument; justify a solution to a problem with multiple solutions; analyze similarities and differences between procedures and concepts;

12

describe, compare, and contrast solution methods; interpret data from a series of data displays; formulate a mathematical model for a complex situation; analyze the assumptions made in a mathematical model.

Table 4. Distribution of items by cognitive demand

Level Categories of cognitive demand % DistributionI Low cognitive demand 20II Moderate cognitive demand 60III High cognitive demand 20

6. Science framework

As with mathematics, the science and reading frameworks were also developed by consultants, pan-Canadian assessment coordinators, and educators who were experts in their respective fields.

The PCAP 2010 science assessment is founded on a definition of scientific literacy that advocates that students’ evolving competencies in applying science-related attitudes, skills, and knowledge, as well as an understanding of the nature of science, enable them to conduct inquiries, solve problems, and make evidence-based decisions about science-related issues. Embedded in this definition of scientific literacy is the supposition that students have knowledge of the life sciences, physical sciences (chemistry and physics), and Earth and space sciences, as well as an understanding of the nature of science as a human endeavour.

The science assessment comprises items associated with the competencies and subdomains that provide opportunities for students to demonstrate their use of science-related attitudes, skills, and knowledge. The competencies and the combination of the five interrelatedsubdomains as defined by curricula across Canada, as well as the statements in CMEC’s Common Framework of Science Learning Outcomes K to 12 2 provided the foundation for the development of all test items.

Competencies: science inquiry, problem solving, and decision making.Subdomains: nature of science, nature of technology, knowledge of science, skills, and attitudes.

These competencies and subdomains are presented within a PCAP 2010 Science assessment unit that comprises an introductory scenario. Efforts were made to ensure that the contexts of the various scenarios were drawn from situations that were relevant, appropriate, and sensible

2 Com mon Fram ework of Science Learning Outcom es K to 12 (1997), Council of Ministers of Education, Canada, www.cm ec.ca/publications.

13

for Canadian students in Grade 8. Each assessment item represents one of the competencies of science inquiry, problem solving, or decision making, which are important to demonstrating scientific literacy, as well as one of the subdomains.

6.1. Competencies

An understanding of science is important for young people to be able to participate in and understand the science and technology that affect their lives both in the present and in the future. Scientific literacy is developed when students are engaged in demonstrating the competencies of science inquiry, problem solving, and decision making. PCAP-2010 Science places a priority on being able to assess these competencies.

Science inquiryScience inquiry requires students to address questions about the nature of things, involving broad explorations as well as focused investigations. This competency involves selecting alternative conclusions in relation to the evidence presented, providing reasons for conclusions based on the evidence provided, and identifying assumptions made in reaching the conclusion. Students demonstrate this competency by applying their knowledge of science, their skills, and their understanding of the nature of science to conduct science-related investigations.

Problem solvingProblem solving requires students to seek answers to practical problems requiring the application of their science knowledge in new ways. This competency involves selecting appropriate solutions in relation to an identified problem, providing reasons for the solution and how it meets the criteria to solve the problem, and identifying assumptions made in solving the problem. Students demonstrate this competency by applying their knowledge of science, their skills, and their understanding of the nature of science to solve science-related problems.

Decision makingDecision making requires students to identify questions or issues and pursue science knowledge that will inform the question or issue. This competency involves making an informed decision for a particular issue in relation to the evidence, providing reasons for the decision based on the evidence provided, and identifying assumptions and limitations of the chosen decision for that issue. Students demonstrate this competency by applying their knowledge of science, their skills, and their understanding of the nature of science to make informed decisions on science-related issues.

6.2. Subdomains

The domain of PCAP-2010 scientific literacy is divided into five subdomains: nature of science, nature of technology, knowledge of science, skills, and attitudes.

14

Nature of scienceThe nature of science involves an understanding of the nature of scientific knowledge and processes by which that knowledge develops. Science provides a way of thinking and learning about the biological and physical world based on observation, experimentation, and evidence. Science builds upon past discoveries. Theories and knowledge are continually tested, modified, and improved as new knowledge and theories supersede existing ones. Scientific debate on new observations and hypotheses is used to challenge, share, and evaluate data through peer interaction and dissemination of information through written publications and presentations.

Nature of technologyPCAP-2010 Science focuses also on the interrelationships between science and technology. Technology is mainly concerned with solving practical problems that arise from human needs and wants. Technology affects nearly every part of our lives and has strong links to science with important relationships and interdependencies. However, there are some fundamental differences that differentiate these two disciplines. Science focuses on the development and verification of knowledge, whereas technology focuses on trying to satisfy humans’ wants and needs by developing solutions and designing devices and/or systems that meet given criteria.

Knowledge of scienceKnowledge of science refers to the theories, models, concepts, and principles that are essential in the following strands of science: life sciences (biology), physical sciences (chemistry and physics), and Earth and space sciences. PCAP-2010 Science assesses students’ application of knowledge-related science inquiry, problem solving, and decision making in contexts relevant to Grade 8 students.

Knowledge chosen to be assessed should focus on enduring concepts that have lasting value beyond school learning. Knowledge of science that is assessed should not be simple recall of information. The majority of PCAP items were written for intended knowledge outcomes generally expected by the end of Grade 8 or in Quebec for students who are in the second year of secondary cycle one (Secondary Two).

SkillsThe competencies of science inquiry, problem solving, and decision making encompass ways of thinking that lead to or use conceptual knowledge of science. Students are expected to apply these competencies to real-life situations to solve problems and make informed decisions. This includes using evidence, validating evidence, and making judgments about the evidence in conjunction with considerations about social, environmental, health, and technological impacts and consequences.

Common to these competencies are skills related to being able to formulate a testable question or identify a problem, planning and carrying out a valid investigation, analyzing and interpreting the data to draw valid conclusions, and communicating the results. The subdomain of skills has been categorized into four strands: initiating and planning, performing and recording, analyzing and interpreting, and communication.

15

AttitudesStudents’ attitudes related to and about science are components in science curriculum documents across Canadian jurisdictions. The vast majority of jurisdictions include the development of positive attitudes as an important component to be embedded within science teaching and learning. These attitudes include interest in science, awareness of science-related issues, respect for the ideas of people with various backgrounds and views, support for scientific processes, collaboration with others, stewardship of the natural environment, and safety in science. PCAP-2010 Science categorizes attitudes into three strands: interest in and awareness of science-related issues, respect and support for evidence-based knowledge, and awareness of sustainable development and stewardship.

7. Reading frameworks

The reading framework statement for PCAP 2010 has not been altered from that used to define reading performance in the 2007 assessment in which reading was the major domain. This enables comparisons over time between the two cohorts (see Section 8, “Linking”). According to curricula across Canada, reading is a dynamic, interactive process whereby the reader constructs meaning from texts. The process of reading effectively involves the interaction of reader, text, purpose, and context before, during, and after reading.

7.1. Text types and forms

This assessment includes a variety of text types and levels of difficulty. These are broadly identified as fiction and non-fiction, recognizing that texts frequently mix forms or types for a variety of purposes. Texts selected are consistent with a broad range of student reading experiences and, in particular, those in the language arts classroom. The text types and forms included in the PCAP 2010 reading assessment are the following:

FictionFiction texts usually have a strong narrative aspect, including elements such as character, setting, conflict, plot, theme, and style. Most frequently, students are expected to engage with fiction texts primarily for literary, aesthetic purposes.

Non-fictionNon-fiction texts such as expository material (textbooks, essays, lab reports, and newspaper articles) generally have a different structure from fiction. For example, expository texts presentinformation, ideas, or a perspective through definitions, sequence, categorization, comparison, contrast, enumeration, process, problem/solution, description, or cause/effect. Some non-fiction texts, however, do include narrative elements. Non-fiction texts also include a wide assortment of informational texts. These texts may include a variety of forms or types, both continuous and non-continuous, which students read for practical or pragmatic purposes. For

16

example, students may read information texts for learning, for interest or recreational purposes, or for accomplishing a particular task. These texts may include articles, instructions, Web sites, and other media texts with graphics and other visuals. Non-fiction texts also include those written to argue a particular perspective or point of view and those written to persuade the reader to take some particular stand or action (persuasion/argument). These texts may include advertisements, editorials, letters to the editor, and speeches. Frequently, they also include visual components.

7.2. Subdomains of reading

In light of the interactive process linking the reader, text, purpose, and context, this assessment of reading considers the reader’s engagement with the text and his or her response to it. Language arts curricula across Canada identify comprehension, interpretation, and response and reflection as major organizing aspects of reading literacy. In this assessment, three subdomains of the integrated process of reading are assessed: comprehension, interpretation, and response to text (which includes response and reflection).

ComprehensionStudents understand the explicit and implicit information provided by the text. In particular, they understand the vocabulary, parts, elements, and events of the text.

InterpretationStudents make meaning by analyzing and synthesizing the parts/elements/events to develop a broader perspective and/or meaning for the text. They may identify theme/thesis and support that with references to details, events, symbols, patterns, and/or text features.

Response to textIn responding, the readers engage with the text in many ways: by making personal connections between aspects of the text and their own real/vicarious/prior experiences, knowledge, values and/or point of view; by responding emotionally to central ideas or aspects of the text; and/or by taking an evaluative stance about the quality or value of the text, possibly in relation to other texts and/or social or cultural factors.

Table 5 displays the distribution of reading items across booklets according to the threesubdomains.

Table 5. Number of reading items by subdomain, by booklet

Subdomain Booklet 1 Booklet 2 Booklet 3 Booklet 4Comprehension 3 3 3 1Interpretation 4 3 3 1Response to text 1 3 3 2

17

8. Linking

Common items (mathematics)A set of common scenarios and items in mathematics was used to verify and ensure that the four versions of the assessment booklets were parallel. There were three common mathematics scenarios appearing in all four booklets, with a total number of eight common items.

There were no common items appearing in all four booklets for science and reading.

Anchor items (2007/2010 comparison for reading)Because of subtle but substantial changes in the mathematics and science assessmentinstruments, it was possible to make the comparison only for the reading results. In 2007, reading was the main domain. To facilitate the comparison, the 2010 reading test was constructed from a subset of the 2007 items (no new items were added). These reading items, referred to as “anchor items,” are used to link the 2007 and the 2010 reading tests.

9. Working group

The working group consisted of experts in reading, mathematics, and science. They came from various jurisdictions, and several were bilingual. These experts were extensively involved in PCAP and took part in various stages of the project, such as development of the assessment framework, drafting of items, editing of items, and comparison of English and French items. Some also participated in the scoring sessions for field test items and main administration items.

10. Item-writing session

An item-writing session was held before the field-test administration of PCAP in mathematics, science, and reading. CMEC brought together a group of item writers who were content experts in the subjects being assessed.

However, CMEC also wanted to give each jurisdiction the opportunity to become involved in the item-writing process. Jurisdictions could send CMEC texts or scenarios for potential inclusion in the 2010 assessment. Working from texts or scenarios submitted by the jurisdictions, the experts conducted a pre-selection of those that could be used for the trial administration. The content experts then drafted items based on the selected texts or scenarios.

Once the items were drafted, other experts edited and selected items, keeping the best and rejecting those that were not as good. They also had to ensure that they had a range of items and that they selected items of various levels of difficulty in the PCAP assessment.

The experts participating in the drafting and editing of items or those sitting on the working group were not involved in all these stages. For example, an expert who drafted items might sit

18

on the working group but did not necessarily sit on the group of experts editing items. CMEC considered it important to have different groups of experts in each phase to draw on the expertise and different viewpoints of each.

11. Item editing

Before including items in an assessment, whether for the trial administration or the main PCAP administration, it was important that these items be edited from various perspectives by a group of experts. As much care as possible had to be taken to ensure that items were sound and would provide an accurate assessment of the skills of Grade 8 students across the country. This section will present the various steps taken by the group of experts when editing items:translation and comparison of items in English and French, editing for language and style, scientific editing, and psychometric editing.

11.1. Translation and comparison of items in English and French

As stated elsewhere, experts met to draft items for the three domains assessed by PCAP. Once all the items had been drafted, they had to be translated into both languages, English and French. CMEC translators translated the items, and then content experts revised the items to ensure that they were acceptable from an educational perspective.

In a broad-scale assessment such as PCAP, it is also important that the various versions of thetest be parallel in terms of language to avoid giving one group an advantage over another. Although an assessment may always include differences between items, it was necessary to ensure that the items in the English version and those in the French version were as equivalentas possible. The stated objective and the level of difficulty of items had to remain the same in both versions. A group of bilingual experts therefore met for about a week solely to compare the English and French items to ensure their equivalency.

11.2. Editing for language and style

An important step in the review of items is editing for language and style. The language editing had to cover such areas as grammar, syntax, spelling, and punctuation for each item, scenario,or graphic in each assessment booklet. The editors had to check such points as correct application of grammar rules to the language; syntax, for correct word order and sentence construction; correct spelling of words; punctuation such as commas, periods, apostrophes, and parentheses in the right places.

The stylistic editing then had to check spaces, fonts, number of lines, page composition, and the introduction of statements. Editors had to verify that font point size was the same for all items; spaces between lines in an item were the same throughout the booklets; page composition was consistent; each item began with a statement followed by a question; the number of lines for the student’s answer were appropriate for the length of the expected answer; and sources

19

were accurate, which means that when an item refers to a graphic on another page, the reference is in fact to the correct page.

11.3. Scientific editing

Scientific editing, another necessary step in the editing of items, checks and validates the correct answers, calculations, data, etc. In part, the four versions of the test contained several selected-response items with four possible answers. Editors had to ensure and verify that there was in fact only one correct answer and that the three other choices were logical decoys. In mathematics, the item generally required students to perform a calculation to obtain the correct answer. The calculation therefore had to be repeated to ensure that the final answer was one of the selected-response answers. Although there were no selected-response answers to check for the open-response items, the items (and sample answers) still had to be validated again to ensure that the correct descriptors were assigned and checked for accuracy, either by referring back to the text or performing the calculations.

Several mathematics and science questions or scenarios included tables, diagrams, and charts with data. Editors therefore had to verify and ensure the accuracy of the information. Students might also have to refer to a table or chart to obtain a correct answer. In the item, students were told on which page the table or chart in question could be found. Editors therefore had to ensure that the page number on which students could access the information was correct.

During item editing, it was important to verify that all components of a text or item were present so that students would be able to answer the question. If components were missing from the item, for example, students would be unable to answer the question correctly and these items would have to be excluded from the analyses because they could not accurately assess the students’ skills. Such situations must be avoided because it would be unfortunate to have to remove an item from the test, especially if that item could be useful in measuring students’ skills.

11.4. Psychometric editing

The experts in mathematics, science, and reading also had to conduct a psychometric edit of items. For selected-response items, one factor to be checked was the order of the possible answers. In reading, possible answers began with shortest and ended with the longest, thus from the shortest sentence or word to the longest. When the possible answers were numbers, they were placed in rising order, from the smallest to the largest. This approach to ordering possible answers thus placed the correct answer in random order. Each possible answer also had to be approximately the same length. If one were more detailed, students would be more inclined to opt for this choice and answer the item correctly, which had to be avoided in assessments. It was also important to check the accuracy of the correct answers to ensure that there was not a second answer that might also be correct, in order to avoid any ambiguity.

20

A scoring guide with descriptors was developed for constructed-response items by experts inmathematics, science, and reading. Various scores were assigned to students’ answers. For mathematics and science, scores could be 0 or 1, or 0, 1, and 2. In reading, the scores ranged from 0 to 3. Each score included a complete description as well as one or more examples taken from students’ answers. The experts therefore had to review all the scoring criteria and ensure that the scores established were clear and precise. This step was very important because in the item-scoring session for the three subjects, scorers received training on each item to be scored. They had to be able to properly distinguish each score so they could assign the one most consistent with the student’s answer.

The experts also had to review the table of specifications, which presents the master assessment plan. The experts had to validate the item types. For example, the assessment had to include a balanced mix of constructed-response items and selected-response items to make efficient use of the students’ assessment time while gathering critical and personal reactions in an open context.

12. Item approval by the jurisdictions

Before including items in the field test, it was important to obtain the approval of jurisdictions on the items selected. CMEC produced various assessment booklets for the three subjects assessed. They then sent these assessment booklets to the jurisdictions for their review of the scenarios and items. CMEC had to obtain the jurisdictions’ approval to include the scenarios and items in the field test.

21

SECTION II: Questionnaire development

Questionnaires were developed to obtain contextual data on mathematics, since it was the major domain in the PCAP-2010 assessment. The PCAP 2010 Grade 8 assessment included three questionnaires, one for participating students, one for teachers, and one for school principals. The overarching structure of the three questionnaires was derived from the Wang-Haertel-Walberg synthesis of research on factors associated with school learning. These questionnaires also focused on the particular need to capture factors associated with mathematics achievement. The questionnaires are intended to contextualize the assessment results. They include some core descriptive data useful for both policy and research; for example, student socioeconomic status (SES), school demographics, and teacher qualifications. Various topics also addressed policy-relevant issues. Some questions focused on the assessment’s major domain, mathematics, with the inclusion of questions about teaching and learning strategies and behaviours. Other questions were in areas that support the directions identified by ministries and departments of education, even if these do not have obvious links to achievement in the major domain. The intended purpose of this selection of topics was to provide information useful in research applicable to mathematics.

Approximately 32,000 students, 2,000 teachers, and 1,600 school principals responded to the questionnaires. The following sections will refer to the questionnaire development process, all three questionnaires administered, and revision of the questionnaire items.

1. Working group

The questionnaire 2010 working group included experts in measurement and evaluation and mathematics across Canada (in both English and French). The members had strong expertise in mathematics content, education research, statistics, and questionnaire item development. The 2007 versions of the questionnaires were used as a basis, but most of the items were substantially revised according to the new focus of assessment: mathematics. As a result, the questionnaire working group produced three questionnaires: Student Questionnaire, Teacher Questionnaire, and School Questionnaire. Mathematics experts also had a meeting to review the Student Questionnaire which had an emphasis on mathematics.

2. Initial questionnaire framework and guiding principles

At the time of the initial design in 2007, the working group reviewed models for questionnaire design found in three large-scale assessment programs: the School Achievement Indicators Program (SAIP), the Trends in International Mathematics and Science Study (TIMSS), and PISA. The working group took the position that any questionnaires designed for PCAP should be shorter and have a more explicit focus than those used in SAIP and PISA. In particular, it was argued that student time was at a premium and that student questionnaires should be significantly streamlined. It was also agreed that, in order to maximize research value, the

22

questionnaires should be designed around some specific research focus rather than taking the broad approach of earlier questionnaires.

The following principles were adopted when designing the questionnaires:

1. Include in the questionnaires some core descriptive data useful for both policy and research (e.g., student SES, school demographics, and teacher qualifications).

2. Other than core data, do not duplicate PISA.3. Attempt to probe fewer areas in greater depth.4. Identify policy-relevant issues.5. Exclude areas shown by SAIP and PISA to be non-productive.6. Focus on the major domain in developing questions around teaching and learning

strategies and behaviours.7. Identify a limited number of areas that support the directions identified by the Pan-

Canadian Education Research Agenda (PCERA).

In addition, the working group examined the limitation imposed by the short-term cross-sectional nature of the data on teaching and learning and agreed that an attempt might be made to ask questions designed to get at students’ longer-term schooling experience.

There was no clear sense that any of the existing frameworks were inherently better than others. In the same way that the PCAP assessment is neither explicitly curriculum-based nor literacy-based, a more eclectic approach to questionnaires is required, based on identified research priorities and the need to link the questionnaires to the major domain.

3. Core questions

The core 2010 section included a limited number of questions for descriptive purposes and for comparison or control variables in research models. Some of the topics addressed in the Student Questionnaire included student gender, student Aboriginal status, student home background, SES, immigration status, home language, and language of instruction. The Teacher Questionnaire included teacher demographics, teacher qualifications and assignments to mathematics, and teacher professional development in mathematics, while the School Questionnaire included school demographics and governance, community context, and composition of the student body. It was found that questions on home language used in PCAP 2007 were insufficient to pursue that area at the level of detail required for a special report on achievement of majority and minority official-language groups, so this area was considerably expanded for PCAP 2010.

3.1. Gender differences in mathematics

Differences in reading achievement favouring females have been a consistent feature of large-scale assessments. Differences in mathematics achievement tend to favour males but are much

23

smaller than the reading differences. The concern in the reading questionnaires was to uncover some potential explanations for this phenomenon by focusing explicitly on:

differential treatment of boys and girls in school; differential reading-related behaviours outside of school.

For mathematics, this issue is less strongly emphasized, but there remains an interest in following trends in gender differences over time.

3.2. Time allocation and use

Time has been a major feature of some other assessments. There is also a strong theoretical and empirical basis for time as a contributor to achievement. PCAP is trying to find ways to enhance the ability to measure time allocations and time loss by omitting previous variables that have little variance (e.g., length of school year) and by asking some more specific questions about engagement in school and in mathematics. These incude: time lost (days) time lost (class periods) time lost (within class sessions) time on subject areas length of class periods homework assignment and completion out-of-school time relevant to learning absenteeism exam times

3.3. Special needs

A set of questions addressed some of the research and policy issues surrounding how to treat students with learning disabilities or other difficulties that might inhibit their progress in school. The focus is on students with lower levels of achievement (i.e., Level 1) and especially those with identified disabilities requiring some form of special treatment in the school but who are not exempted from the PCAP assessment by virtue of these disabilities. The broad policy context around this area is the strong movement in most jurisdictions toward inclusion of these students in regular classes. Questions have been formulated in the following areas: accommodations for disabilities programming (modified programs) class composition

3.4. Assessment

Many jurisdictions have responded to concerns about the performance of students and schools by implementing jurisdictional assessment programs. These take different forms and are of

24

different degrees of maturity in different jurisdictions. Assuming that the underlying goal of this policy direction is to improve and not merely to describe achievement or entrench current levels, there is strong reason to examine assessment practices in the jurisdictions, and particularly the uses made of jurisdictional assessments. The intent here is to expand the scope of questions about assessment.Some areas for question development are: assessment practices teacher knowledge of assessment principles school and teacher use of external assessments student reaction to assessment (including attitudes to low-stake assessment) teaching to the test strategies to prepare students for assessment existence and use of external (e.g., district, jurisdictional) assessments

3.5. Attitudes⁄motivations

This area is examined in some detail in PISA. Questions and constructs in this area are consistently found to be related to achievement. However, it can be considerably streamlined in PCAP. This area can be adequately researched using PISA, and there is no need to duplicate what is found in PISA. The basic idea here is that PCAP should include only the minimal number of items needed to permit use of attitudes and motivations as control variables in research on teaching and learning strategies. Items were developed on: attitudes (general and subject-specific) interest self-concept

3.6. Student learning strategies

The study of student learning strategies is considered one of the core elements of PCAP. The questions in this key area linked to the mathematics assessment framework dealt with student cognitive and meta-cognitive strategies in mathematics, that is, the mathematics strategies that students use when confronting different tasks and at different levels of difficulty.

3.7. Teaching strategies

Both the SAIP and PISA questionnaires included lengthy lists of teaching strategies to which students (and teachers in SAIP) were asked to respond. These included generic questions about disciplinary climate, use of time, and student-teacher interactions, as well as more subject-specific questions. Typically, these questions were about the student’s or teacher’s experience in a particular class in the year of the survey. Because of this narrow scope, it seems likely that this results in systematic underestimation of the effects of teaching. Rather than simply duplicating the kinds of items found on the SAIP and PISA questionnaires, an attempt was made in designing the PCAP questionnaires to “reach back” to capture the student’s longer-term

25

classroom experience. While this will likely be difficult to do, it can, if successful, contribute to our understanding of students’ broader school experience and how this relates to achievement.

Thus, another small set of questions dealt with teaching perceptions purporting to contributeto mathematics achievement. Additional information about teaching strategies was gathered by asking students about their attendance at school and about their teacher’s classroom practices (subject-specific). Questions in this section include: teacher perceptions of what contributes to mathematics achievement student perceptions of their earlier school experiences with mathematics, and school questions on overall instructional philosophy and

approach to mathematics learning.

3.8. Opportunity to learn

Since opportunity to learn has often been considered one of the better predictors of achievement, a small set of questions were dedicated to the determination of the: student’s individual history of being taught mathematics parental activities related to opportunities to learn

One interesting feature of the PCAP 2010 Grade 8 assessment results is that the linkage of student performance to the three questionnaires will permit direct association of the output data (performance results) to the contextual elements for which information was gathered.

4. Item types

In the Student Questionnaire, School Questionnaire, and Teacher Questionnaire, most questions presented a range of answers, and participants generally could check only one of these answers. The questionnaires also included several opinion questions to measure attitudes and reactions. These questions were based on a Likert scale, to quantify their attitudes. A Likert scale is an ordinal scale on which the answers to a question are ranked in order. A series of statements was therefore presented to participants, and they had to indicate their level of agreement with each statement (e.g., “totally disagree” or “totally agree”). The questionnaires also included a few items in which participants had to write an answer, such as “number of hours or number of days on which…”

5. Number of items

The three questionnaires contained various questions related primarily to mathematics since that was the main domain assessed in 2010. The Student Questionnaire and School Questionnaire contained five sections, and the Teacher Questionnaire contained sevensections. All the questionnaires included a considerable number of items in each section. Participants could also respond to a number of statements for a single item. Participants had an allotted time to respond to all the items in the questionnaire. For example, students had about 30 minutes to fill out the Student Questionnaire.

26

6. Item writing

The initial questionnaire working group used model questionnaires from SAIP, TIMSS, and PISA. They therefore studied each item in these three large-scale assessments in order to draft questions for the three PCAP questionnaires: the Student Questionnaire, School Questionnaire,and Teacher Questionnaire. The working group started with the concept that the PCAP questionnaires had to be shorter and more targeted, so it had to draft and include questions that were likely to produce interesting data.

6.1. Student Questionnaire

Once students finished the assessment, they had 30 minutes to answer the questions in the Student Questionnaire. Most of these questions were related to mathematics since that was the main PCAP assessment domain. Approximately 32,000 students responded to the questionnaire.

The first section of the questionnaire included questions on the student’s personal information, either about the student’s parents or guardians or about himself or herself. These questions gathered demographic data about the student (e.g., gender, socioeconomic status, immigrantstatus). Section 2 of the questionnaire included questions to measure students’ attitudes and motivation, since these are generally related to performance. In part, some questions covered a student’s confidence in mathematics. The third section included questions referring to the breakdown of a student’s use of time. For example, some questions asked students how much time they spent on homework or other activities, and about absenteeism. The fourth section included a number of questions linked to classroom assessment (e.g., the assessment practices of teachers, use of scoring rubrics, etc). Section 5 of the Student Questionnaire surveyed teaching and learning strategies. For example, students had to indicate how often they engaged in classroom activities or assignments, and what kind of strategies they use to help withmathematics.

The data compiled from the Student Questionnaire will support a comparison and draw links between the variables studied as well as the student’s performance.

6.2. Teacher Questionnaire

It is equally relevant to gather information from teachers on the variables studied through the assessment. The Teacher Questionnaire was filled out by the mathematics teachers of Grade 8 students selected to take part in the PCAP-2010 assessment. They had to fill out the questionnaire at the time of the student assessment.

The Teacher Questionnaire contained seven sections with many questions. Section 1 included questions on personal information about the teachers, such as gender, their diploma or degree, and their education. The second section of the questionnaire referred to their professional

27

development, including questions about number of days and types of this development. The third section was related to the time management. In part, the questions covered how often they assigned homework or engaged in certain activities, the frequency of classroom incidents, number of hours lost, etc. The fourth section of the questionnaire covered the assessment methods used (e.g., frequency of use of assessment methods, item types on examinations, marking tables, training related to measurement and assessment). The questions in section 5focused on teaching strategies. For example, the questions refer to strategies used in mathematics instruction, such as re-teaching concepts and skills, adapting instructions and resources, or providing enrichment for advanced students. Other questions covered the frequency of classroom activities, or of use of calculators, software, and manipulatives. The theme of the sixth section of the Teacher Questionnaire was special-needs students. Questions were asked, for example, about the number of students with certain special needs and the presence of special-needs students in classes. The focus of the seventh and final section of the questionnaire was attitudes. The questions covered the teacher’s attitudes towardmathematics, including reasons why students do more or less well in mathematcs, teacher’s self-confidence, self-evaluation, and challenges.

The Teacher Questionnaire was linked to student results but used unique identifiers to preserve confidentiality.

6.3. School Questionnaire

The schools participating in this questionnaire were those selected for the main PCAP administration. The school’s principal was asked to answer the questions on the SchoolQuestionnaire. This provided information at the school level in relation to the subjects studied in the context of the PCAP-2010 mathematics assessment and allowed for analysis and linkagewith the students who completed the assessment.

The School Questionnaire contained many questions divided into five sections. The first section, as in all the other questionnaires administered, was reserved for general information. For example, questions covered the number of students enrolled in the school, the grades taught in the school, and the percentage of Aboriginal students. Time management was the theme covered in the second section of the questionnaire. Questions might cover the number of minutes in a learning or teaching period, by set period or week. The absenteeism rate was part of another question answered by the school’s principal. Section 3 of the questionnaire related to the assessment. In part, principals had to specify the number of external assessments for which results were or were not factored into marks. They also had to indicate their level of agreement with statements about the assessment results and the influence of these results on students’ learning (e.g., large-scale assessments, provincial assessments). The instructional climate was the subject of the fourth section of the School Questionnaire. This section contained only two questions. Respondents had to indicate their level of agreement in relation to the mathematics teaching atmosphere in their school and indicate the frequency of different events (e.g., professional development, parent nights, math days or contests). Section 5 of the questionnaire covered the context for instruction. The questions covered arrangements for

28

special-needs students, availability of Internet access, computers, and manipulatives (e.g., base-10 blocks, colour tiles, geometric solids).

The School Questionnaire was linked to student results but used unique identifiers to preserve confidentiality.