Australian Council for Educational Research Test for Schools Technical... · 3 Foreword This...

June 2012

PISA-Based Test for Schools

Technical report

Submitted to: Organisation for Economic Co-Operation and

Development (OECD)

Australian Council for Educational Research

Private Bag 55

Camberwell Vic 3124

Phone +61 3 9277 5555

Direct +61 3 9277 5511

ACN: 004 398 145; ABN: 19 004 398 145

3

Foreword

This report, produced by the Australian Council for Educational Research (ACER), describes

the procedures used for the development of assessment instruments for the PISA-Based Test for

Schools, a project of the Organisation for Economic Co-operation and Development (OECD).

In the report there are descriptions of the item development process, including how some items

are subsequently deleted, how items are modified before inclusion in trial testing and the results

of that testing; detailed descriptions of the resulting item characteristics are given. The report

describes how the analysis from trial testing and other information such as framework coverage

were considered to produce the pilot trial assessment for the project, which is being undertaken

in a number of countries in May-June 2012.

While the report describes the specific details of this project, the principles underlying its

implementation are similar to the approach used by ACER generally in its test development.

The report was produced under the guidance of the co-directors of the project, Juliette

Mendelovits and John Cresswell, with substantial contributions from Alla Berezner, Greg

Macaskill and Stephanie Templeton.

4

Table of Contents

Foreword .................................................................................................................................. 3

Chapter 1: Introduction ............................................................................................................. 6

PISA ................................................................................................................................................. 6 PISA-Based Test for Schools .............................................................................................................. 6 Phase 1 .............................................................................................................................................. 7 Phase 2 .............................................................................................................................................. 7 Organisation of this report ............................................................................................................... 7

Chapter 2: Test design and test development ............................................................................. 9

Test Design ........................................................................................................................................ 9 Test Development........................................................................................................................... 11 Field trial and equating study instrument characteristics .............................................................. 15

Chapter 3: Field Operations for the field trial and equating studies ........................................... 22

Recruitment of countries and locales for the field trial and equating studies ............................... 22 Field trial operations in participating countries and groups .......................................................... 23 Coding ............................................................................................................................................. 26

Chapter 4: Preparation of Instruments for the Pilot Trial of the PISA-Based Test for Schools ...... 30

Test Design for the Pilot Trial ......................................................................................................... 30 Selection of Items for the Pilot Trial Cognitive Instruments ........................................................... 31 Blueprints and actual distribution of items and score points by key PISA framework variables ... 31 National adaptation process for the pilot trial ............................................................................... 36 Pilot trial cognitive instrument construction .................................................................................. 36 Pilot trial questionnaire construction ............................................................................................. 37 Preparation of Manuals .................................................................................................................. 40

Chapter 5: Equating Methods and Results ................................................................................ 42

Differential Item Functioning by Gender ........................................................................................ 46 Item-by-country Interaction (Country DIF) ..................................................................................... 51 Test Targeting ................................................................................................................................. 52

Chapter 6: Proposal for replenishment of items ........................................................................ 60

Appendix 1: Audit of items developed for the Phase 1 and Phase 2 FT-ES .................................. 64

Mathematics ................................................................................................................................... 64 Reading ........................................................................................................................................... 74 Science ............................................................................................................................................ 91

Appendix 2: PBT Critical friends.............................................................................................. 110

Appendix 3: Item Characteristics ............................................................................................ 111

Mathematics ................................................................................................................................. 111 Reading ......................................................................................................................................... 125 Science .......................................................................................................................................... 140

Appendix 4: ICC gender performance ..................................................................................... 159

Mathematics ................................................................................................................................. 159 Reading ......................................................................................................................................... 169 Science .......................................................................................................................................... 181

Appendix 5: Pilot trial item pool characteristics ...................................................................... 195

Mathematics ................................................................................................................................. 195

5

Reading ......................................................................................................................................... 198 Science .......................................................................................................................................... 201

6

Chapter 1: Introduction

PISA

1. The OECD Programme for International Student Assessment (PISA) is a

collaborative effort among OECD member countries to measure how well 15-year-old students

approaching the end of compulsory schooling are prepared to meet the challenges of today’s

knowledge societies. The assessment is forward-looking: rather than focusing on the extent to

which these students have mastered a specific school curriculum: it looks at their ability to use

their knowledge and skills to meet real-life challenges. This orientation reflects a change in

curricular goals and objectives, which are increasingly concerned with what students can do

with what they learn at school.

2. PISA surveys take place every three years. The first survey took place in 2000

(followed by a further 11 countries in 2002), the second in 2003, the third in 2006, and the

fourth in 2009; the results of these surveys have been published in a series of reports (OECD,

2001, 2003, 2004, 2007, 2010) and a wide range of thematic and technical reports. Further

assessment is taking place in 2012. For each assessment, one of science, reading and

mathematics is chosen as the major domain and given greater emphasis. The remaining two

areas, the minor domains, are assessed less thoroughly.

3. PISA is an age-based survey, assessing 15-year-old students in school in grade

seven or higher. These students are approaching the end of compulsory schooling in most

participating countries, and school enrolment at this level is close to universal in almost all

OECD countries.

4. The PISA assessments take a literacy perspective, which focuses on the extent to

which students can apply the knowledge and skills they have learned and practised at school

when confronted with situations and challenges for which that knowledge may be relevant. That

is, PISA assesses the extent to which students can use their reading skills to understand and

interpret the various kinds of written material that they are likely to meet as they negotiate their

daily lives; the extent to which students can use their mathematical knowledge and skills to

solve various kinds of numerical and spatial challenges and problems; and the extent to which

students can use their scientific knowledge and skills to understand, interpret and resolve

various kinds of scientific situations and challenges.

5. PISA also uses student questionnaires to collect information from students on

various aspects of their home, family and school background, and school questionnaires to

collect information from schools about various aspects of organisation and educational

provision in schools.

PISA-Based Test for Schools

6. Since 2000, a growing number of countries around the world have taken advantage

of the opportunity to participate in PISA. There were 32 countries who participated in PISA

2000; the number has grown to 67 in PISA 2012. Generally, whole countries participate,

although there are some regions, such as Hong Kong, Macao and Shanghai from China, which

participate separately.

7. While this regular pattern of assessment has been taking place, there has been a

growing interest from entities smaller than countries to participate in a test that gives results

with a link to PISA and therefore allows them to make comparisons with PISA. These entities

might be regions, education systems or individual schools. There has also been an interest in

being able to do this assessment outside the normal three-yearly PISA schedule.

8. In response to this the Australian Council for Educational Research, which has been

responsible for the design and implementation of all PISA surveys up to 2012, was contracted to

produce a test that would be comparable to PISA. This test has become known as the PISA-

7

Based Test for Schools, to take account of its emphasis being focused on students in regions or

individual schools.

9. The assessment instrument was created in two phases, each of which will be

described in more detail in Chapter 2: Test design and test development.

Phase 1

10. The first phase was based on the original OECD request for a set of items that

would allow the three main areas of interest (domains) – reading, mathematics and science – to

be assessed in a single two-hour test. This required the creation of 40 minutes of testing time for

each domain.

11. To calculate the characteristics of the items they were placed into a trial program

which also equated the items to existing secure PISA items. This trial test and equating study

took place in four countries: Australia, Ireland, the USA and the UK.

Phase 2

12. The second phase involved developing sufficient additional items to create a test

with 90 minutes of testing time for each of the three domains of reading, mathematics and

science. This expanded item set was also placed into a field trial operation to test its

characteristics. At the same time the new items were equated with items that had been created

for Phase 1.

Organisation of this report

13. This technical report is designed to describe the technical aspects of the project at a

sufficient level of detail to give an understanding of the processes that were undertaken to arrive

at the assessment instrument which is being used in the Pilot Trial of the project.

14. There are six chapters in this report:

Chapter One—Introduction: Overall description of the project

Chapter Two— Test design and test development

a) Test designs of two field trial and equating studies with a few explanatory words

b) Test development

- Quality control measures (panelling, cog labs, critical friend reviews)

- Characteristics of the Phase 1 and Phase 2 Field Trial and Equating Study (FT-

ES) instruments.

Chapter Three—Field operations for the field trial and equating studies

- Country operations

- Coding

Chapter Four— Preparation of instruments for the pilot trial of the PISA-Based Test for

Schools

- Test design

- Selection of items

- Blueprints and achieved characteristics of the cognitive instruments

- National adaptations of the instruments

- Construction of the cognitive and questionnaire instruments

- Preparation of manuals

8

Chapter Five—Equating methods and results

- Differential item functioning by gender

- Item-by-country interaction (country DIF)

- Test targeting

- DPS from PISA showing item distributions for each domain

Chapter Six—recommendation for refreshing the item pool

Appendices: Detailed appendices of results pertaining to the chapters of the report are provided

including an audit of items developed for the Phase 1 and Phase 2 FT-ES (Appendix 1), item

analyses for all items in the pilot trial (Appendix 3), pilot trial item characteristic curves

showing all and gender performance (Appendix 4) and the pilot trial set of items with scale

scores and logit values (Appendix 5).

9

Chapter 2: Test design and test development

15. This chapter describes the test design and test development for Phase 1 and Phase 2

Field Trail and Equating Study (FT-ES) of the PISA-Based Test for Schools.

16. The main purposes of the field trial and equating study were to:

Select items for the PISA-Based Test for Schools from the newly developed item pool;

and

Place newly developed items on the existing PISA scale.

Test Design

Phase 1 FT-ES test design

17. The Phase 1 FT-ES item pool consisted of 194 items where 131 items were newly

developed items and 64 were link items selected from the main PISA assessment. The number

of the new and link items included in each of the domains is shown in Table 1.

Table 1 Number of new and link items in Phase 1 FT-ES

Domain

New

items

Link

items TOTAL

Mathematics 32 16 48

Reading 49 27 76

Science 50 20 70

TOTAL 131 63 194

18. Table 2 presents the Phase 1 FT-ES test design.

Table 2 Booklet design for Phase 1 FT-ES

Booklet 1 M1 M2

Booklet 2 M2 M3

Booklet 3 M3 M4

Booklet 4 M4 M1

Booklet 5 S1 S2

Booklet 6 S2 S3

Booklet 7 S3 S4

Booklet 8 S4 S1

Booklet 9 R1 R2

Booklet 10 R2 R3

Booklet 11 R3 R4

Booklet 12 R4 R5

Booklet 13 R5 R1

19. The Phase 1 FT-ES items were allocated to thirteen clusters (four mathematics

clusters, five reading clusters and four science clusters) with each cluster representing 30

10

minutes of test time. M1 to M4 denote the mathematics clusters, R1 to R5 denote the reading

clusters and S1 to S4 denote the science clusters.

20. Each cluster included a combination of new Phase 1 FT-ES items and link items

selected from the PISA main surveys’ item pool. The number of new items in the Phase 1 field

trial pools was twice as big as the number of link items. The number of items per cluster varies

according to domain, based on timing analysis conducted in PISA field trials and main surveys.

21. The items were presented to students in thirteen one-hour booklets, with each

booklet being composed of two clusters of the same domain. There are four each of

mathematics and science booklets and five reading booklets. Each cluster appears in both first

and second position in the booklets, to neutralise position effect in calibrating the items.

22. Another feature of the Phase 1 FT-ES test design is that each domain cluster

appears with two other clusters of the domain. For example cluster M1 appears with the clusters

M2 and M4. This allows establishing a linkage between single domain booklets for the item

calibration.

Phase 2 FT-ES test design

23. The Phase 2 FT-ES item pool consisted of 226 items where 159 items were newly

developed items and 64 were link items selected from the Phase 1 FT-ES. The number of new

and link items included in each of the domains is shown in Table 3.

Table 3 Number of new and link items in Phase 2 FT-ES

Domain

New

items

Link

items TOTAL


Reading 51 27 78

Science 64 22 86

TOTAL 159 67 226

24. Table 4 presents the Phase 2 FT-ES test design.

11

Table 4 Booklet design for Phase 2 FT-ES

Booklet 14 M5 M6

Booklet 15 M6 M7

Booklet 16 M7 M8

Booklet 17 M8 M9

Booklet 18 M9 M5

Booklet 19 R6 R7

Booklet 20 R7 R8

Booklet 21 R8 R9

Booklet 22 R9 R10

Booklet 23 R10 R6

Booklet 24 S5 S6

Booklet 25 S6 S7

Booklet 26 S7 S8

Booklet 27 S8 S9

Booklet 28 S9 S5

25. The Phase 2 FT-ES items were allocated to fifteen item clusters (five mathematics

clusters, five reading clusters and five science clusters) with each cluster representing 30

minutes of test time. M5 to M9 denote the mathematics clusters, R5 to R10 denote the reading

clusters, and S5 to S9 denote the science clusters.

26. Each clusters included a combination of new Phase 2 FT-ES items and link items

selected from the Phase 1 FT-ES new item pool.

27. The items were presented to students in fifteen one-hour booklets, with each

booklet being composed of two clusters of the same domain. Each cluster appears in both first

and second position in the booklets, to neutralise position effect in calibrating the items.

28. Similarly to the Phase 1 FT-ES test design, each Phase 2 FT-ES cluster appears

with two other clusters from the same domain. For example cluster M5 appears with the clusters

M6 and M9.

Test Development

The test development team

29. All item writers for the PISA-Based Test for Schools were drawn from the teams of

professional test developers in ACER’s Assessment and Reporting research programs. The item

writers formed into three smaller teams, one for each of the domains of mathematics, reading

and science. Each team included individuals who have been involved in test development for

the PISA international surveys. Each domain team was led by a coordinating test developer,

under the overall coordination of one of the project directors. Table 5 shows the number of test

developers for each domain.

12

Table 5 Number of test developers per domain

Domain Number of test

developers

Mathematics 5

Reading 8

Science 7

30. Test development for Phase 1 FT-ES was undertaken from August to December

2010. Test development for Phase 2 FT-ES was undertaken from July to October 2011.

Item development process

31. Items used in the field trial and equating studies, other than the PISA main survey

items used for linking in Phase 1 FT-ES, were for the most part developed specifically for the

PISA-Based Test for Schools. For reading and science, all items were developed exclusively for

the purpose. For mathematics, since there was an overlap in the item development periods for

PISA 2012 and Phase 1 of the PISA-Based Test for Schools project, and several of the ACER

mathematics test developers were working on both projects, some of the ACER-developed

items for PISA-Based Test for Schools mathematics began as part of the larger pool under

consideration for PISA 2012. Some of the ACER-developed items that were NOT selected for

the PISA 2012 Field Trial were reconsidered for the PISA-Based Test for Schools. Ultimately

26 such items were used as part of the PISA-Based Test for Schools.

32. Over the first five cycles of PISA, ACER has established a sound methodology for

developing high quality assessment instruments. Central to this methodology are various

cognitive laboratory methods designed to ensure that the intended cognitive outcomes are

reflected in the items, that the items developed have an appropriate distribution of difficulties,

and that the coding guides are adequate for consistent and accurate coding by expert coders. The

activities described below have been refined over the years since the inception of PISA and

were employed by the item development teams for PISA-Based Test for Schools.

Cognitive laboratory methods

33. Cognitive laboratory methods used for PISA include three types of activities:

cognitive walkthrough, cognitive interview and cognitive comparison studies.

Cognitive walkthrough: After initial development by individual item writers, all items

undergo cognitive walkthrough (sometimes referred to as item panelling, or item

shredding). At this stage, items are given the closest and most detailed possible scrutiny by

professional item writers to ensure that from every conceivable perspective, each item can

achieve its intended purpose. Items routinely undergo significant modification and

improvement following this step.

Cognitive interview: Items are used in small group cognitive interviews, with a test

developer working with four or five 14- to 16-year-olds. Each item is exposed to a small

group of students. “Think aloud” methods are used with individuals as they tackle items.

Cognitive group interviews are also used to further expose fundamental aspects of how

students perceive and react to items. The information gained through these cognitive

interviews provides hard evidence about the cognitive processes involved as students tackle

test items.

Cognitive comparison studies: This third type of cognitive laboratory involves piloting

sets of items with larger numbers of students in schools. Each unit is piloted with a

convenience sample of about 30 students reasonably representative of 15 year-olds in the

country concerned. This enables the relative difficulty of items to be compared (e.g., error

13

percentage, time to complete) and real student responses to be obtained to refine and

illustrate scoring guides.

34. The first two of these cognitive laboratory techniques were employed in the

development of all PISA-Based Test for Schools items. Cognitive comparison studies were used

only for the 26 mathematics items that began life as potential PISA 2012 items.

35. The items that survived the initial cognitive walkthrough and were submitted to

cognitive interview procedures are listed in Appendix 1. Appendix 1 shows the categorisation of

each item according to its major PISA framework variables. It also indicates the origin of each

item and its use in the two field trial and equating study administrations.

36. The number of items developed per domain in Phase 1 is shown in Table 6.

Table 6 New item development for Phase 1

Domain Developed to

cognitive interview

stage

Required for

Phase 1 FT-ES

Required for

two-hour PBT test:

40 minutes per

domain


Reading 113 47 20

Science 94 48 22

Total 270 127 58

37. For the Phase 1 development, the goal was a yield of sufficient items for a two-hour

assessment, with all students administered the same test material, comprising 40 minutes of

testing per domain.

38. For the Phase 2 development, the goal was to develop enough items to augment the

original yield so that the final total yield would be an item pool comprising 90 minutes of

testing per domain. Accordingly, test development for Phase 2 was targeted at developing an

additional 50 minutes of testing per domain.

39. The number of items developed per domain in Phase 2 is shown in Table 7.

Table 7 New item development for Phase 2

Domain Developed to

cognitive interview

stage

Required for

Phase 2 FT-ES

Required for

additional

50 minutes per

domain


Reading 112 50 25

Science 131 64 32

Total 308 154 77

40. It will be seen that the number of items developed to cognitive interview stage was

at least four times the number required for the final yield (a similar ratio to that applied for

PISA, from development to main survey). For both Phase 1 and Phase 2, the aim was to develop

enough items per domain to populate a given amount of testing time (40 minutes and 50

minutes respectively). Note that the calculation of the number of items required for the two-hour

14

PBT test, and hence the numbers of items developed at each preceding stage, was calculated

according to the time estimates for PISA main surveys, which vary according to domain.

Critical friend reviews

41. ACER sought feedback from external domain experts in order to check and refine

the quality of the items to be selected for PBT. Given the central aim of the PBT project to

emulate PISA quality in the test instruments, we approached members of past and current PISA

expert groups, and highly-regarded test developers from the PISA Consortia 2000-2012. All

experts who were approached (two mathematics, three reading and two science experts) agreed

to assist in this task. The critical friends are listed in Appendix 2.

42. Along with the items themselves, the experts were provided with the most recent

versions of the relevant PISA framework; an item feedback spreadsheet listing each item in the

relevant set and tabulating its main framework characteristics; and item feedback guidelines.

43. Modelled on the national centre reviews of items conducted for PISA, feedback was

requested on a number of criteria: curriculum relevance and interest level for 15-year-olds,

authenticity of context, and relevance to preparedness for life. Experts were also asked to

identify any cultural or gender concerns, potential translation issues, problems with the coding

guides or other issues such as the categorisation according to framework variables. Finally they

were asked to give an overall rating on priority for inclusion in the final instrument.

44. The items and supplementary material were uploaded to a designated secure area of

the MyPISA website (https://mypisa.acer.edu.au).

45. The critical review process was implemented in two batches. Items included in

Phase 1 FT-ES were dispatched to the experts in mid-September 2011 and their feedback was

returned by end October 2011. Items included in Phase 2 FT-ES were dispatched in early

January 2012 and the experts returned feedback by early February 2012.

46. While providing invaluable critical reviews, all experts commented on the “PISA-

likeness” and overall high quality of the material.

Item selection and preparation for the field trial and equating studies

Phase 1 item selection

47. The Phase 1 FT-ES instruments comprised, for each domain, sets of new items

developed for the project and a set of secure link items from previous PISA cycles (see Table 1

for the numbers of items in each category per domain).

48. In selecting link items, PISA domain leaders considered framework balance, range

of difficulty and a high level of reliability as prime criteria.

49. For the new items, the PBT test development coordinator for each domain

considered the item pool available and, drawing on evidence from cognitive laboratories, made

a selection for the Phase 1 Field Trial and Equating Study based on the following criteria:

fit to framework, such that the set reflected the target percentage ranges per category as

set out in the framework;

a range of (predicted) difficulty;

a fair and balanced representation of content, including concepts central to the domain;

interest level for 15-year-olds;

likely endurance (able to stand the test of time); and

lack of third party copyright obstacles.

Phase 2 item selection

50. The Phase 2 FT-ES included new items developed for Phase 2 and a set of linking

items that had been developed for and included in Phase 1 FT-ES (see Table 3 for the numbers

of items in each category).

15

51. In selecting the linking items from Phase 1, test developers considered:

the psychometric quality of the items, based on the empirical results of the Phase 1 FT-

ES in Australia, which had been administered, coded and analysed by end of

September 2011;

feedback on the Phase 1 items from critical friends; and

framework balance and range of difficulty.

52. To select the new items that had been developed for Phase 2, the same list of

criteria as that used to select the new items for Phase 1 FT-ES was employed: namely, fit to

framework, a range of difficulty, good representation of content, interest level for 15-year-olds,

likely endurance over time, and permission to use any third party material.

National adaptations

53. Source versions of each unit selected for each of the field trial and equating studies

were finalised at ACER and a Translation and Adaptation Spreadsheet (TAS) for the new items

was prepared for each phase, modelled on PISA TAS.

54. Coordinators at each FT-ES national location – consortium partners in Ireland, the

UK and USA – completed the TAS and returned it to ACER. ACER test development

coordinators for each domain considered each proposed adaptation and conducted an email

discussion with the national centre to resolve any issues. Finally, ACER test developers made a

decision about whether to accept suggested adaptations or not, guided by judgement of what

was necessary/important/allowable to adapt, and what should not be adapted (for example

because the proposed adaptation might make an item more difficult or easier); and guided also

by consistency in decisions across the participating countries. ACER then implemented the

agreed adaptations in national version units. The completed TAS, recording ACER’s decision

on each issue, was returned to the country/national centre for information. ACER test

developers completed the Australian versions.

55. The UK versions of the units were subsequently used as the source versions, in line

with the OECD policy of using British English for PISA source versions. The (UK) source

versions of the units were provided to the international schools that participated in Phase 2.

56. A booklet shell based on the booklet shell used for PISA main surveys was

designed for the PBT field trial and equating studies. Since no student questionnaire was

included in the FT-ES, a very short set of questions about student background was included at

the end of the test booklet, to collect information about the gender, age and grade level of each

participant. The student background page in the booklets for international schools also asked

about the language background of the student. The booklet shell was sent to the national centres

and groups for each Phase, for suggested adaptations. These were implemented by ACER in the

construction of the national materials for the field trial and equating studies.

57. The national versions of the units were assembled into national versions of clusters,

booklets and coding guides, and despatched via secure ftp sites to national centres for printing.

Field trial and equating study instrument characteristics

58. In this section we present the main framework characteristics of the item sets

fielded in each of the two field trial and equating studies. For each domain, tables set out the

number and percentage of items administered in Phase 1 FT-ES and Phase 2 FT-ES according

to variables defined in the relevant framework1. Target percentages from the framework are also

shown.

1 OECD. (2010). PISA 2009 Framework: Key Competencies in Reading, Mathematics and

Science. Paris: OECD.

16

Mathematics

Table 8 Mathematics instruments for field trial and equating studies: Process variable

Variable Phase 1: new items Phase 2: new items Frame-work

Target

N % N % %

Employing 19 59 24 56 55

Formulating 7 22 13 30 25

Interpreting 6 19 6 14 20

Total 32 100 43 100 100

Table 9 Mathematics instruments for field trial and equating studies: Content variable


Target

N % N % %

Change and relationships

9 28% 10 23% 32

Quantity 8 25% 11 26% 23

Space and shape 6 19% 12 28% 25

Uncertainty and data

9 28% 10 23% 20

Total 32 100 43 100 100

Table 10 Mathematics instruments for field trial and equating studies: Context variable


Target

N % N % %

Occupational 8 25% 9 21% 23

Personal 4 13% 11 26% 25

Scientific 11 34% 8 19% 27

Societal 9 28% 15 35% 25

Total 32 100 43 100 100

17

Table 11 Mathematics instruments for field trial and equating studies: Item format

variable


Target

N % N % %

Selected response 14 44% 15 35% 36

Constructed Response Expert

11 34% 9 21% 17

Constructed Response Manual

7 22% 19 44% 47

Total 32 100 43 100 100

18

Reading

Table 12 Reading instruments for field trial and equating studies: Text format variable


Target

N % N % %

Continuous 35 73% 31 61% 60

Non-continuous 13 27% 15 29% 30

Mixed and multiple

0 0% 5 10% 10

Total 48 100 51 100 100

Table 13 Reading instruments for field trial and equating studies: Text type variable


Target

N % N % %

Narration 5 10% 15 29% 15

Description 20 42% 10 20% Not specified

Exposition 13 27% 4 8% Not specified

Argumentation 10 21% 11 22% Not specified

Instruction 0 0% 11 22% Not specified

Total 48 100 51 100 100

Table 14 Reading instruments for field trial and equating studies: Situation variable

Variable Phase 1:

new items

Phase 2:

new items

Frame-work

Target

N % N % %

Personal 19 40% 27 53% 28

Public 11 23% 8 16% 28

Occupational 4 8% 5 10% 16

Educational 14 29% 11 22% 28

Total 48 100 51 100 100

19

Table 15 Reading instruments for field trial and equating studies: Aspect variable


Target

N % N % %

Access and retrieve

13 27% 12 24% 25

Integrate and interpret

22 46% 27 53% 50

Reflect and evaluate

13 27% 12 24% 25

Total 48 100 51 100 100

Table 16 Reading instruments for field trial and equating studies: Item format variable


Target

N % N % %

Selected response 24 50% 32 63% 55

Constructed response

24 50% 19 37% 45

Total 48 100 51 100 100

20

Science

Table 17 Science instruments for field trial and equating studies: Competency variable


Target

N % N % %

Explaining phenomena

26 51% 30 47% 41

Using evidence 15 29% 19 30% 36

Identifying issues 10 20% 15 23% 23

Total 51 100 64 100 100

Table 18 Science instruments for field trial and equating studies: Knowledge variable


Target

N % N % %

Of science 27 53% 32 50% 50

About science 24 47% 32 50% 50

Total 51 100 64 100 100

Table 19 Science instruments for field trial and equating studies: Knowledge of science

variable


Target

N % N % %

Physical systems 8 30% 9 28% 27

Earth and space systems

7 26% 9 28% 23

Technological systems

3 11% 5 16% 16

Life systems 9 33% 9 28% 34

Total 27 100 32 100 100

21

Table 20 Science instruments for field trial and equating studies: Knowledge about science

variable

Variable Phase 1: new items Phase 2: new items Phase 2: link items from

Phase 1

Frame-work

Target

N % N % N % %

Enquiry 11 46% 15 47% 6 50% 50

Explanations 13 54% 17 53% 6 50% 50

Total 24 100 32 100 12 100 100

Table 21 Science instruments for field trial and equating studies: Item format variable

Variable Phase 1: new items Phase 2: new items Phase 2: link items from

Phase 1

Frame-work

Target

N % N % N % %

Selected response 31 61% 44 69% 12 55% 64

Constructed response

20 39% 20 31% 10 45% 36

Total 51 100 64 100 22 100 100

22

Chapter 3: Field Operations for the field trial and equating studies

59. This chapter describes the conduct of the field trial and equating studies, and the

coding operation for constructed-response items.

Recruitment of countries and locales for the field trial and equating studies

60. The brief for the development of the PISA-Based Test for Schools instrument called

for the items to be fielded in four English-speaking countries. Attracting countries and schools

to take part in the study was a major challenge of this project, and is one of the main reasons for

the elongated timeline. ACER initially approached PISA national centres, partly because of our

confidence that they would ensure security of the PISA materials, and also because of their

familiarity with PISA administration standards such as those pertaining to quality assurance of

printing, distribution and test administration. ACER was only partly successful in recruiting

PISA national centres. One difficulty was that smaller English-testing countries were conscious

of the PISA main survey administration looming in 2012 and understandably did not want to

threaten their capacity to meet sampling requirements for PISA 2012 by over-burdening the

limited pool of schools with further requests for test administration.

61. Eventually, four country centres were recruited for the Phase 1 FT-ES, and three

country centres and a group of international schools (all with English as the language of

instruction) were recruited for the Phase 2 FT-ES.

62. Table 22 and Table 23 show the countries/centres for the two Phases, and the

achieved dates for testing, coding and initial data analysis.

Table 22 Phase 1 Field Trial and Equating Study

Country Responsible institution Testing

dates

Coding Initial data

analysis

Australia ACER May-June 2011

June-July 2011

September 2011

Ireland Educational Research Centre, St Patrick’s College, Dublin (PISA national centre)

November 2011

December 2011

January 2012

UK National Foundation for Educational Research (PISA national centre)

November-December 2011

January 2012 January 2012

USA Westat (PISA international consortium partner)

January 2012 February 2012

February–March 2012

23

Table 23 Phase 2 Field Trial and Equating Study

Country/Location Responsible institution Testing

dates

Coding Initial data

analysis

Australia ACER November-December 2011

January 2012 February–March 2012

UK National Foundation for Educational Research (PISA national centre)



USA Westat (PISA international consortium partner)



Europe and UAE (international schools)

ACER (staff based in Germany and ACER’s Dubai office)



Field trial operations in participating countries and groups

Quality of materials

63. Countries followed the PISA 2012 Field Trial quality assurance procedures for the

printing of test booklets. This included using the same paper and print quality that had been

agreed upon by the PISA Consortium in the 2012 Field Trial.

64. When producing the test materials, ACER ensured that the layout and pagination of

all national versions were equivalent to the source version.

Recruitment of schools

65. Table 24 shows the numbers of schools and students participating in each phase of

the field trial and equating studies, by country or group, as well as some information about the

gender and language background of the students. Participants were asked to complete a very

short background questionnaire at the end of the test booklet. For all administrations, students

were asked to record their gender. For the International Schools administrations only, students

were also asked to record whether the language spoken at home was English or a language other

than English. Discrepancies between the numbers for “All students” and for the gender and

language subgroups are accounted for by students having failed to complete this information.

24

Table 24 Numbers of schools and students by country/group and phase

Country/group Field trial and

equating

study

Number of

schools

participating

Number of students

confirmed by

schools

Number of students participating

All Boys Girls English Other Lang

Australia Phase 1 22 655 576 243 333 Not available

Not available

Australia Phase 2 29 897 749 392 357 Not available

Not available

England Phase 1 11 698 564 288 272 Not available

Not available

England Phase 2 11 829 674 363 311 Not available

Not available

International Schools – Dubai

Phase 2 14 969 816 422 394 184 616

International Schools – Europe

Phase 2 3 124 89 45 44 7 72

Ireland Phase 1 12 Not available 781 346 435 Not available

Not available

USA Phases 1 and 2

12 1859 1412 689 703 Not available

Not available

25

66. The following paragraphs provide more detail about each of the countries and

locales that participated in the Phase 1 and/or Phase 2 FT-ES.

Australia

67. Schools that ACER wished to approach were selected through the standard ACER

sampling process. The sample included schools from all Catholic, Independent and Government

jurisdictions. Permission to approach schools in Catholic and Government jurisdictions was

sought and approved before contacting the schools within these areas.

68. Although schools were to provide about 25 students (one class) to participate, actual

participation numbers were about 10-15% lower than the number of students confirmed from

the schools.

England

69. All the schools recruited for the equating study were from the Academies Enterprise

Trust. AET provided NFER with a list of their academies and these academies were invited to

take part in the study.

70. The response rate to the invitation letter was poor and reminder phone calls were

made in order to boost recruitment and meet the required participation rate. AET helped in the

recruitment of schools.

International schools

Dubai

71. For the Phase 2 FT-ES, only private schools following international curriculums,

with English as the language of instruction, were targeted. The sample from each school

included both genders. To limit the number of schools selected, only large schools were

sampled.

Europe

72. A small number of international schools in Europe participated in Phase 2 field trial

and equating study FT-ES. Only schools in which English is the language of instruction were

included.

Ireland

73. A convenience sample was used for the Phase 1 FT-ES. Schools were selected on

the basis that they were not sampled for the PISA 2012 main survey in Ireland.

74. By using the stratification variables used for the PISA 2012 main survey sample,

Ireland aimed to maintain a balance in terms of gender mix, SES composition and school type.

To limit the number of schools selected, only large schools were sampled.

United States

75. The United States equating study school sample was a self-selected group of

schools in central Iowa. The schools were in districts that formed a consortium organised by an

education advocacy business organisation, the Des Moines Business/Education Alliance.

76. The Business/Education Alliance gathered school participation and contact

information through the districts and passed the information to Westat. Westat then made

contact with the school coordinators at each school.

77. Phase 1 and Phase 2 field trial and equating studies were combined and run

simultaneously in participating schools in the USA.

26

Recruitment of test administrators

78. In Australia and the United Kingdom, test administrators for PISA-Based Test for

Schools had been previously trained for the PISA 2012 field trial. Australia hired qualified

teachers as test administrators and the United Kingdom hired former teachers. Similarly, in

Ireland test administrators were retired school principals or inspectors who had worked for

previous PISA administrations.

79. In the United States, test administrators were hired from the National Assessment of

Educational Progress (NAEP) and had wide experience in conducting student assessments. The

Field Manager for national PISA oversaw the staff.

80. For the international schools field operations, ACER staff located in Europe and

Dubai administered the tests, after training with colleagues from the PISA international or

national project teams.

Administration at the classroom level

81. All countries reported that the test administration went well. For a very small

number of schools in Ireland and Dubai, student discipline was an issue during the test session.

In the United States, some schools did not have adequate space to assess the large number of

students which meant that some sessions needed to be split into small groups or be conducted at

different times of the day.

82. The main issue that countries encountered however was absent students. In

England, some students refused to participate in the test or completed very little of the test

booklet. Test Administrators in Australia coordinated directly with schools to accommodate the

most suitable dates and times in order to minimise disruptions to the normal school timetables

and ensure that there were fewer absent students. In the United States, make-up sessions were

held to ensure that an adequate number of students was tested.

Security arrangements and confidentiality of the test material

83. In all countries, PISA security standards were applied to the PISA-Based Test for

Schools materials. The test administrators directly managed the distribution, administration and

collection of the materials at all times when in the schools and no copying of materials was

permitted. Test administrators ensured that all materials were returned promptly to the national

centres after testing. The national centres then securely delivered all the materials to ACER by

courier for coding.

Coding

Management and sequence of the coding operations

84. All coding of constructed-response items for the two field trial and equating studies

was conducted at ACER’s Melbourne offices. The overall manager of the coding process, and

the test developers who conducted training and quality control during the operation, were ACER

staff with experience in the conduct of coding operations in PISA international and PISA

national (Australia and Dubai /the United Arab Emirates), in 2009 and/or 2012.

85. Because of the concatenated implementation of the field trial and equating studies

across the participating countries and entities, coding was carried out in several stages for both

Phase 1 and Phase 2.

86. Table 25 shows the sequence of coding operations.

27

Table 25 Sequence of coding operations

Phase Participating country / entity Approximate Dates

Phase 1 Australia 16 June – 10 July 2011

Phase 1 Ireland 12 – 16 December 2011

Phase 1 United Kingdom 16 – 27 January 2012

Phase 1 United States 6 – 17 February 2012

Phase 2 Australia 16 – 27 January 2012

Phase 2 United Kingdom 6 – 17 February 2012

Phase 2 United States 6 – 17 February 2012

Phase 2 International Schools (Europe and UAE) 6 – 17 February 2012

Recruitment of coders

87. Coders were recruited from the pool of experienced coders who have worked in

PISA national coding teams for Australian and Dubai / UAE coding operations for each domain.

Many have been part of PISA coding teams since 2000. Coders were typically secondary school

teachers experienced in relevant disciplines who have retired or were on leave. As for PISA

main surveys, mathematics and science coding was undertaken by the same individuals (trained

and experienced mathematics and/or science teachers). Reading coding was undertaken by

secondary English or Humanities teachers.

88. There was a high degree of continuity in the coding teams across the separate

coding operations for Phase 1 and 2. The teams were expanded for the last phase of coding to

ensure rapid processing of the large amount of material. The total numbers of individuals

involved were seven reading coders and 14 mathematics and science coders.

89. Table 26 shows the number of coders per domain for each operation.

28

Table 26 Number of coders for each coding operation, by domain

Phase Participating country / entity Number of coders

Phase 1 Australia 4 reading, 4 mathematics,

4 science

Phase 1 Ireland 3 reading 6 mathematics/science

Phase 1 United Kingdom 6 reading, 8 mathematics/science

Phase 1 United States 6 reading, 8 mathematics/science

Phase 2 Australia 6 reading, 8 mathematics/science

Phase 2 United Kingdom 6 reading, 8 mathematics/science

Phase 2 United States 6 reading, 8 mathematics/science

Phase 2 International Schools (Europe and UAE) 6 reading, 8 mathematics/science

Preparation of coder training material

90. PISA main survey items were used for linking in Phase 1. For these items, coding

workshop material from the main surveys was used for training.

91. For the new material developed for PBT Phase 1 and Phase 2, test developers

selected student responses representing the various codes from the first batches of student

booklets that were returned to ACER from the FT-ES. These samples were reproduced in hard

copy and used in training along with the coding guides for each item, to promote consistency.

For both Phase 1 and Phase 2, initial sample responses used for training were selected from the

Australian student responses, as these were the first batches of student responses available.

Coder training

92. Coder training began with an orientation to the program and the assessment

constructs. Test development leaders for each domain then led the coders through each

constructed response item in a cluster, with discussion of coding guide descriptions and sample

responses. The coders had an opportunity to practise applying the coding guides on “live”

student responses. Test developers monitored consistency and accuracy on these sample

responses and remediated where necessary. In general, training was conducted on two clusters,

and then operational coding proceeded on those clusters.

93. Training was conducted at several points: for each new pool of items (Phase 1 and

Phase 2); whenever new coders were added to the team; and in the form of a short refresher if

there had been a long gap since the last operation.

Coding operation

94. Coders worked at the ACER coding centre in Melbourne, for a maximum of six

hours per day, with breaks every one and a half hours. Coding was conducted by item and by

cluster. Within each coding operation, all coders coded a proportion of the responses for every

constructed-response item, so that a maximum number of coding perspectives for each item was

collected.

29

95. Test development leaders were on call for discussing any substantive issues arising

during the coding operation – for example, unanticipated responses, or inadequate or ambiguous

wording in the coding guides (or in the items themselves). In the case of coding guides, these

were occasionally adjusted during the coding operation, and all previously coded responses

recoded.

Coding quality control

96. Since the field trial and the equating study were conducted as one operation, there

was no opportunity to review and adjust coding guides after the field trial that is available in

PISA itself. To compensate for this, it was decided to code every response twice, as a quality

control measure, with two independent codings. The first coder entered his or her code on a

coding sheet, which was removed before the booklet was given to the second coder. The second

coder entered his or code onto the booklet.

97. Data entry was conducted immediately following the coding, and any discrepancies

between the two codes for each response were highlighted. A member of the relevant coding

team then inspected the discrepancies against the student booklets. Where a straightforward

error was detected (such as the code for “incorrect” having been entered instead of the code for

“missing”), the coder entered the appropriate code as the third and finalised one. Where the

discrepancy indicated a difference of judgement, the response was referred to the lead test

developer, who made a decision as to the appropriate code, and entered it as the third and

finalised code.

98. In the course of the early stages of the coding operation for each phase, and as a

result of discrepancy coding, test developers made some adjustments to the coding guides. In

such cases, all student responses were recoded by the coding team to reflect the revised coding

guide.

99. In addition, lead coders for each domain kept a record of items that had caused

particular coding difficulties, such as frequent inconsistency in judgement. This information was

provided to the test developers, to feed into the subsequent selection of items for the Pilot Trial

Data entry and data cleaning

100. After initial data entry and entry of discrepancy codes by coders or test developers,

the data was delivered to the ACER psychometrics unit. Any invalid codes detected in the data

set were referred to the manager of coding operations for checking and correcting.

30

Chapter 4: Preparation of Instruments for the Pilot Trial of the PISA-Based Test for Schools

101. The aim of the field trial and equating studies described in the previous chapters

was to develop an assessment instrument for the PISA-Based Test for Schools, with similar

characteristics to the PISA main surveys that have been conducted in five administrations since

2000, comprising 90 minutes of testing time for each of the three domains of mathematics,

reading and science.

102. The OECD determined that an important step in the development and

implementation of the PISA-Based Test for Schools was to conduct a pilot trial with invited

schools and networks in May and June of 2012 using the developed assessment instrument.

103. The pilot round was planned with approximately 120–150 schools across three

English-speaking countries, with approximately 75–100 students participating in each school as

the target. In fact, schools in four countries were recruited by the OECD for the pilot trial:

Canada, Scotland, UK (excluding Scotland) and USA.

104. The objectives of the pilot trial were to test and validate elements of the PISA-

Based Test for Schools relating to (A) instrument design parameters; (B) accreditation process

of local test administration service providers and cost parameters; (C) quality assurance and

security procedures, and (D) reporting format of results to participating schools.

105. This chapter describes the test design, and test construction of the assessment

instruments, and preparation of materials including cognitive instruments, questionnaire

instruments and manuals for the pilot trial of the PISA-Based Test for Schools.

Test Design for the Pilot Trial

106. Table 27 sets out the test booklet design for the PISA-Based Test for Schools pilot

trial.

Table 27 Test booklet design

Book Block 1 Block 2 Block 3

1 R1 RSM M1

2 RSM M2 S2

3 M2 M1 R2

4 M1 S2 S1

5 S2 R2 R1

6 R2 S1 RSM

7 S1 R1 M2

107. The pool of items, selected from the new items created for this project and

administered in the field trial and equating studies conducted during 2011 and early 2012,

comprises about 93 minutes of testing time for each of the three domains of mathematics,

reading and science.

108. The design comprises two clusters of items from a single domain for each of

mathematics (M1 and M2), reading (R1 and R2) and science (S1 and S2), and one cluster

including items from all three domains (RSM). The key considerations in developing the test

design were as follows:

31

Each booklet is 120 minutes in length, to match the PISA main survey design and

therefore give PISA-Based Test for Schools students a “PISA-like” test experience.

The clusters in Booklets 1 to 7 are organised so that each cluster appears in three

separate booklets, in the first, second and third block (position). Furthermore, each pair

of clusters appears in one (and only one) booklet and each cluster appears with every

other cluster. This is to balance the effect of booklet position.

109. There was not enough material to create a fully-rotated design with 30-minute

clusters (which are used in the PISA main surveys). It was judged that full rotation of the

clusters and test length were priorities, as these factors will have more impact on test

performance than cluster length. It is anticipated that the mixing of domains in the hybrid cluster

will have little or no impact on test performance. The hybrid cluster comprises about 13–14

minutes of testing time for each domain. It may be noted that this amounts to slightly more test

material than specified in the agreed contract. An extra three to four minutes’ worth of items

was added for each domain to make up the number of minutes needed for the hybrid booklet.

110. The clusters were assembled so that each reflects, to the extent possible, the major

framework variables of the given domain, and so that each includes items of a range of

difficulties, based on estimations derived from analysis of data from the field trial and equating

studies.

Selection of Items for the Pilot Trial Cognitive Instruments

111. In selecting items for the Pilot Trial cognitive instruments, the test development

coordinator for each domain considered the item pool available from the two field trial and

equating studies and made a selection comprising approximately 93 minutes of testing time for

each of mathematics, reading and science.

112. In addition to the criteria that had governed the selection of items for the field trial

and equating studies (see paragraphs 47 to 52), selection for the pilot trial instruments had to

satisfy the following conditions:

The psychometric properties of all selected items had to be satisfactory (a description of

the psychometric analyses and outcomes is provided in Chapter 5: Equating Methods

and Results);

Items that generated coding problems had to be avoided unless those problems could be

properly addressed through modifications to the coding guides; and

Items given high priority ratings in the critical friend reviews were to be preferred, and

items with lower ratings were to be avoided.

113. In addition, the item set for each domain had to satisfy these conditions as far as

possible:

The major framework categories had to be populated as specified in the relevant

assessment framework; and

There had to be an appropriate distribution of item difficulties.

Blueprints and actual distribution of items and score points by key PISA framework variables

114. Table 28 to Table 41 show the targeting of the item set for each domain, and the

actual distribution achieved. In all cases the targets are based on the relevant targets in the PISA

main surveys (2009). The achieved distribution is reported in terms of both items and score

points; the number of score points is greater than the number of items because a small number

of items in each domain are polytomous, yielding two score points.

32

Mathematics

115. For mathematics the target number of items for 95 minutes of testing is about 40

items. Forty items were included in the pilot trial mathematics selection. Of these six items yield

two score points, so the total number of score points is 46.

Table 28 Mathematics instrument for pilot trial: Process variable

Target % Approximate

target no. of items

Actual no. of

items

Actual no. of

score points

Employing 55 22 20 22

Formulating 25 10 11 13

Interpreting 20 8 9 11

Total 100 40 40 46

Table 29 Mathematics instrument for pilot trial: Content variable


target no. of items

Actual no. of

items

Actual no. of

score points

Change and relationships 32 13 13 14

Quantity 23 9 9 10

Space and shape 25 10 10 13

Uncertainty and data 20 8 8 9

Total 100 40 40 46

Table 30 Mathematics instrument for pilot trial: Context variable


target no. of items

Actual no. of

items

Actual no. of

score points

Occupational 23 9 9 11

Personal 25 10 10 12

Scientific 27 11 11 12

Societal 25 10 10 11

Total 100 40 40 46

33

Table 31 Mathematics instrument for pilot trial: Item Type variable


target no. of items

Actual no. of

items

Actual no. of

score points


17 7 7 12


47 19 19 19

Complex Multiple Choice 8 3 3 4

Simple Multiple Choice 28 11 11 11

Total 100 40 40 46

Reading

116. For reading the target number of items for 95 minutes of testing was about 47 items.

There is one item that is not scored in the set (scoring of the response to this item,

PR6004Q05A, Museum Park, is combined with the score for PR6004Q05B). This item was

counted in determining the timing of the reading instrument, but is not counted in the following

tabulations; thus the number of items in totals shown below is 46. With three items yielding two

score points each, the total score points are 49.

Table 32 Reading instrument for pilot trial: Aspect variable


target no. of items

Actual no. of

items

Actual no. of

score points

Access and retrieve 25 11-13 17 18

Integrate and interpret 50 23-24 18 20

Reflect and evaluate 25 11-13 11 11

Total 100 47 46 49

Table 33 Reading instrument for pilot trial: Text format variable


target no. of items

Actual no. of

items

Actual no. of

score points

Continuous 60 28 30 33

Non-continuous 30 14 13 13

Mixed and multiple 10 5 3 3

Total 100 47 46 46

34

Table 34 Reading instrument for pilot trial: Text type variable


target no. of items

Actual no. of

items

Actual no. of

score points

Narration 15 7-8 8 9

Description (physical) Not specified

Not specified 15 16

Exposition (conceptual) Not specified

Not specified 8 8

Argumentation Not specified

Not specified 12 13

Instruction Not specified

Not specified 3 3

Total 100 47 46 49

Table 35 Reading instrument for pilot trial: Situation variable


target no. of items

Actual no. of

items

Actual no. of

score points

Personal 28 13 19 21

Public 28 13 8 8

Occupational 16 8 7 8

Educational 28 13 12 21

Total 100 47 46 49

Table 36 Reading instrument for pilot trial: Item format variable


target no. of items

Actual no. of

items

Actual no. of

score points


45 21

17 18


4 4

Complex Multiple Choice 55 26

7 9

Simple Multiple Choice 18 18

Total 100 47 46 49

Science

117. For science the target number of items for 95 minutes of testing was about 54 items.

This number of items was included in the pilot trial science selection, with five of the items

yielding two score points, for a total of 59 score points.

35

Table 37 Science instrument for pilot trial: Competency variable


target no. of items

Actual no. of

items

Actual no. of

score points

Explaining phenomena 41 22 22 24

Using evidence 36 20 20 23

Identifying issues 23 12 12 12

Total 100 54 54 59

Table 38 Science instrument for pilot trial: Knowledge variable


target no. of items

Actual no. of

items

Actual no. of

score points

Of science 50 27 25 28

About science 50 27 29 31

Total 100 54 54 59

Table 39 Science instrument for pilot trial: Knowledge of science variable


target no. of items

Actual no. of

items

Actual no. of

score points

Physical Systems 27 6-7 6 7

Earth and Space Systems 23 5-6 4 4

Technological Systems 16 4 5 7

Living Systems 34 8-9 10 10

Total 100 27 25 28

Table 40 Science instrument for pilot trial: Knowledge about science variable


target no. of items

Actual no. of

items

Actual no. of

score points

Enquiry 50 13-14 13 13

Explanations 50 13-14 16 18

Total 100 27 29 31

36

Table 41 Science instrument for pilot trial: Item format variable


target no. of items

Actual no. of

items

Actual no. of

score points


36 18-21

20 24


0 0

Complex Multiple Choice 64 32-36

16 17

Simple Multiple Choice 18 18

Total 100 54 54 59

National adaptation process for the pilot trial

Unit adaptation

United States and the United Kingdom (excluding Scotland)

118. As many source units were updated after the field trial and equating studies, it was

considered more reliable to adapt the pilot trial source units for national instruments rather than

bring the national versions of the FT-ES units up to date.

119. The FT-ES Test Administration Spreadsheets (TAS) were used to adapt the source

units. The FT-ES national versions of the units were also consulted. All national adaptations

listed in the TAS were verified by the test developers before finalising the United States and the

United Kingdom (excl. Scotland) national versions of the pilot trial units.

Scotland and Canada

120. Scotland and Canada had not participated in the field trial and equating studies. To

allow national adaptations for the pilot trial the centres coordinating were sent an updated TAS

for the pilot trial units and invited to make suggestions for adaptations using similar procedures

to those adopted for adaptations in the field trial and equating studies. Because of tight

schedules, however, there was less opportunity for negotiation. After agreed modifications had

been finalised by ACER test developers, adaptations were made directly to the source units.

121. All national versions of the units were checked by the instrument construction team

as well as the test developers to ensure that the adaptations were correctly implemented.

Booklet shell adaptations

122. The United Kingdom (excl. Scotland) and Scotland did not adapt the booklet shell

from the source version. The booklet shell adaptations that were agreed to for use by the United

States in the equating study were also used in the pilot trial. For Canada, the source booklet

shell was sent to the Canadian national centre for adaptation. No adaptation spreadsheet was

necessary and adaptation proposals from Canada were inserted by ACER directly into the

booklet shell.

Pilot trial cognitive instrument construction

Cluster construction

123. Source clusters were built from the finalised source units. Test developers reviewed

the source clusters to ensure that the pagination and item layout were correct. The national

versions of the clusters were then constructed using the national units and adjusted to match the

source clusters’ layout and pagination.

37

Booklet construction

124. Source booklet construction began after the source and national versions of the

clusters were finalised by the test developers. The source clusters were inserted into the booklet

shell and booklets were checked for pagination, layout, formatting and graphics before being

sent to test developers for the final checks. When the source booklets were finalised, national

versions of the booklets were created using the national clusters. All booklets were checked on-

screen as well as on the printed version. Experience has shown that problems with graphics in

particular, often become apparent only once they are printed.

125. After the Word versions of the booklets were finalised, the booklets were converted

to PDF. The PDF versions of the booklets were printed and compared to the Word versions to

ensure that no errors were introduced during the conversion process.

Additional adaptations to United States and Canadian booklets

126. Source booklets were constructed on A4 size paper. United States and Canadian

booklets needed to be adapted to be printable on Letter size paper. Margins, footers and headers

were adjusted to ensure that the units would fit onto Letter sized paper without affecting the unit

layout.

Pilot trial questionnaire construction

127. The student and school questionnaires for the pilot trial were constructed using the

past PISA questionnaires. The OECD specified the questions that should be included, drawing

on source versions from PISA 2003–2009. Items included in the pilot trial instruments are listed

below.

Source questionnaires

Student questionnaire

SECTION 1: ABOUT YOU

2009-ST01: Grade level

2009-ST02: Study programme

2009-ST03: Age

2009-ST04: Gender

2009-ST05: Pre-primary education attendance

2009-ST06: Starting age for primary education

2009-ST07: Grade repetition

SECTION 2: YOUR FAMILY AND HOME [WITH INTRO TEXT]

2009-ST08: Family structure

2009-ST09: Mother’s main job

2009-ST10: Mother’s education

2009-ST11: Mother’s qualifications

2009-ST12: Mother’s employment status

2009-ST13: Father’s main job

2009-ST14: Father’s education

2009-ST15: Father’s qualifications

2009-ST16: Father’s employment status

2009-ST17: Country of birth for student and parents

2009-ST18: Age at arrival in country of test

2009-ST19: Language spoken at home

2009-ST20: Home resources

2009-ST21: Family wealth

2009-ST22: Books in home

38

SECTION 3: YOUR READING ACTIVITIES [WITH INTRO TEXT]

2009-ST23: Time spent reading for enjoyment

2009-ST24: Enjoyment of reading

2009-ST25: Materials students read for enjoyment

SECTION 4: LEARNING TIME

2009-ST28: Minutes in a class period

2009-ST29: Number of class periods in each subject

2009-ST30: Number of class periods in total

2012-ST55: Hours typically spent out-of-school-time lessons

SECTION 5: YOUR SCHOOL

2012-ST88: Attitude towards school

2012-ST86: Student-teacher relations

SECTION 6: YOUR SCHOOL LESSONS AND SUBJECTS

2009-ST35: Number of students in test language class

2009-ST36: Disciplinary climate in test language lessons

2012-ST81: Disciplinary climate in mathematics lessons

2006-ST36: Importance of doing well in science, mathematics and test language subjects

SECTION 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS [WITH

INTRO TEXT]

2009-ST41: Metacognition strategies in reading: Understanding and remembering

2009-ST42: Metacognition strategies in reading: Summarising

SECTION 8: YOUR VIEWS ON <BROAD SCIENCE> [WITH INTRO TEXT]

2006-ST16: Enjoyment of science

2006-ST17: Self-efficacy in science

SECTION 9: CAREERS AND <BROAD SCIENCE> [WITH INTRO TEXT]

2006-ST27: School preparation for science-related careers

2006-ST28: Student information on science-related careers

2006-ST29: Future-oriented motivation to learn science

2006-ST35: Instrumental motivation to learn science

SECTION 10: YOUR MATHEMATICS EXPERIENCES

2003-ST30: Interest in, enjoyment of and instrumental motivation in mathematics

2003-ST31: Self-efficacy in mathematics

2003-ST32: Anxiety and self-concept in mathematics

2012-ST77: Teacher support in mathematics lessons

39

School questionnaire

SECTION A: THE STRUCTURE AND ORGANISATION OF THE SCHOOL

2009-SC02: Public or Private

2009-SC03: Funding

2009-SC04: Community

2009-SC05: Schooling

2012-SC23: Students leaving without certificate

2012-SC39: Quality assurance/improvement

SECTION B: THE STUDENT AND TEACHER BODY

2009-SC06: Enrolment

2009-SC07: Grade repetition

2009-SC08: English as second language

2009-SC09: Teaching staff

SECTION C: THE SCHOOL’S RESOURCES

2009-SC10Q01: Number students in grade

2009-SC10Q02: Computers available

2009-SC10Q03: Internet connected

2009-SC11: School capacity hindered

Remove 2009-SC11Q05 and SC11Q06 (library staff and support personal)

Include 2012-SC14Q11 to 2012-SC14Q13 (k to m on infrastructure)

SECTION D: SCHOOL INSTRUCTION, CURRICULUM AND ASSESSMENT

2009-SC12: Ability grouping

2009-SC13: Extracurricular activities

2009-SC14: Activities offered for English-language learners

2012-SC20: Extra mathematics lessons offered

2012-SC21: Purpose of additional mathematics

2009-SC15: Assessment methods

2009-SC16: Purposes of assessment

SECTION E: SCHOOL CLIMATE

2009-SC17: Teacher and student factors hindering school climate

2009-SC18: Parental expectations

SECTION F: SCHOOL POLICIES AND PRACTICES

2009-SC19: Admissions selection

2009-SC20: Transfer of students

2009-SC21: Accountability to parents

2009-SC22: Uses of achievement data

2009-SC23: Monitoring teacher practices

2009-SC24: School responsibilities

2009-SC26: School leadership

2012-SC35: Professional development in mathematics

40

SECTION G: PILOT ADMINISTRATION FEEDBACK

2012-SC50: Pilot Administration Feedback

National questionnaires

128. National versions of the questions were collected from the archived national

materials submitted in past PISA cycles. For questions from PISA 2012, adaptations were based

on the final versions of the PISA 2012 main survey Questionnaire Adaptation Spreadsheets

(QAS) or the PISA 2012 field trial materials in the case of the United Kingdom (excl. Scotland),

for which the main survey adaptations had not yet been finalised.

129. Not all questions were administered by the countries in past cycles. In the cases

where a country did not administer a question in PISA, that question was also removed from the

pilot trial questionnaires.

130. The United States added several national questions to the pilot trial school and

student questionnaires which are routinely added to PISA questionnaires:

USA-2009-ST05: Student is Hispanic or Latino

USA-2009-ST06: Student’s race

USA-2009-ST08: Enrolment in kindergarten

USA-2009-ST10 (sub-question): Grade repetition in kindergarten - sub-question in

international-2009-ST07

USA-2009-ST11: Highest level of school the student expects to complete

USA-2009-SC27: Percentage of students eligible for free- or reduced-price lunch.

Preparation of Manuals

131. Manuals were created to explain the correct procedures with a view to making the

implementation of the pilot study consistent with the implementation of assessments during the

normal PISA cycles. The following manuals were created:

Procedures Manual

School Coordinator’s Manual

Test Administrator’s Manual

132. These PISA-Based Test for Schools manuals were based on procedures being

undertaken for the PISA 2012 survey.

The Procedures Manual

133. The manual is aimed at the main organisers of the PISA-Based Test for Schools and

provides an overview of the procedures to be followed for the implementation of the pilot trial

assessment for the PISA-Based Test for Schools. It provides a brief overview of the rationale

and design of PISA and discusses the evolution of the PISA-Based Test for Schools project.

134. It also provides a description of the main resources needed to implement the test

such as test administrators, school coordinators, coders and a chronological guide as to when

certain activities should be undertaken.

The School Coordinator’s Manual

135. The manual is aimed at the coordinators in those schools that are taking part in the

pilot trial. The manual describes the tasks which should be undertaken by the School

Coordinators before the assessment, during the assessment and after the assessment. The School

Coordinator is the key person who will organise the time and place for the assessment and

notify students and teachers of what is happening. The School Coordinator is the liaison

between the organisers and the participating school.

41

The Test Administrator’s Manual

136. This manual is based very closely on the manual of the same name that was written

for the implementation of the PISA 2012 survey. This was done to ensure that the PISA-Based

Test for Schools is administered in exactly the same way to help maintain comparability

between the two assessments. The Test Administrator’s Manual describes in detail the tasks to

be done before the assessment and the procedures to be followed during the administration of

the assessment, including the exact script that must be read to the students before they start the

assessment. The manual stresses the need for consistent application of the existing set of

procedures used in PISA.

137. The manual also stresses the need for excellent communication between the Test

Administrator and the School Coordinator.

42

Chapter 5: Equating Methods and Results

138. The PISA Based Test for Schools (PBT) data consisted of two phases. The first

phase had PISA link items and new PBT items in an hour-long test of two clusters per booklet.

The test was balanced in that all items appeared in both the first and last half of the test. The

second phase included only PBT items, but some of these also occurred in the first Phase. It was

therefore possible to link together the data into one data matrix through these common items.

139. In this chapter, we report on the psychometric characteristics of the final selection

of items for the pilot trial, totalling 90 minutes of testing time for each domain. Some items

were discarded from the field trial and equating sets because of psychometric problems such as

excessive DIF or poor discrimination. Also some parts of complex multiple-choice items were

found to perform poorly and were omitted, and some partial credit items were collapsed if there

was insufficient differentiation between student abilities on some categories or when an

intermediate category was answered by very few students. To ensure sufficient coverage of

framework areas, some items with good psychometric properties were deleted in favour of items

with inferior psychometric properties, though this did not extend to retaining items with clearly

poor properties. Finally, some high-quality items were set aside because they were excess to

requirements.

140. The table below gives the original number and selected number of items. These

numbers exclude the PISA link items.

Table 42 Numbers of items in original FT-ES pool and in pilot trial selection

Domain Original Pool Selected

Mathematics 75 40

Reading 100 46

Science 114 54

141. The mathematics and science items were put on the current PISA scale (2003 and

2006 respectively) by anchoring the PISA link items at their 2003 and 2006 item parameters.

The reading items were mapped on to the 2000 scale by adjusting the PISA 2009 item

parameters by a linear transformation to the PISA 2000 scale. Details are given below.

142. It was noted that when the three domains were scaled without anchoring, the mean

difficulties of the PISA link items were lower for all three domains for the equating study than

the means for the original PISA items. It was suspected that at least some of this effect was due

to the PISA item parameters being estimated from a two-hour test, with the effect of low

response in particularly the fourth cluster (the last half hour) driving up the estimated

difficulties. When the PISA parameters were re-estimated over the first two clusters only,

effectively replicating the items’ behaviour in an hour-long test as was the Equating study, the

following mean decreases in item difficulties were observed:

Table 43 Reduction in mean item difficulty for one-hour PISA re-estimation

Domain Reduction

Mathematics 0.152

Reading 0.182

Science 0.108

43

143. These corrections were subtracted from the PISA mathematics 2003 and PISA

science 2006 difficulties to give the final link anchors. For reading, the PISA 2009 reading

items were transformed to the PISA 2000 logit scale using

PISA2000L=.883*(PISA2009L+0.85-0.12)+.0552

144. This is based on the transformation for missing gender for achievement scores.

145. The .182 correction was then subtracted from the transformed difficulties.

146. The transformations to put the PBT item parameters obtained from these equating

runs on the PISA point scales are:

Mathematics: PISA2003=100*(PISA2003L+0.1344)/1.2838+500

Reading: PISA2000=100*(PISA2000L-0.5076)/1.1002+500

Science: PISA2006=100*(PISA2006L-0.1797)/1.0724+500

147. Equating errors for the scales were calculated as in PISA 2009, using the link item

parameters from the scaling without anchoring compared to the PISA items. This gave equating

errors of 3.99 for mathematics, 5.94 for reading and 5.97 for science. These are higher in

general than the PISA equating errors, particularly for mathematics, and reflect the smaller data

set on which the equating is based.

148. Item statistics (Itanals) from the final equating run are provided for the selected

pilot trial set of items in Appendix 5, excluding the PISA link items. Table 44 gives the items

and the corresponding number in the files. The PISA link items are indicated with a yellow

background.

44

Table 44 Final Selected Items for Pilot Study

Mathematics Reading Science

1 PM033Q01 1 PR220Q01 1 PS256Q01

2 PM034Q01 2 PR220Q02B 2 PS408Q01

3 PM192Q01T 3 PR220Q04 3 PS408Q03

4 PM411Q01 4 PR404Q03 4 PS408Q04T

5 PM411Q02 5 PR404Q06 5 PS413Q04T

6 PM421Q01 6 PR404Q07T 6 PS413Q05

7 PM421Q02T 7 PR404Q10A 7 PS413Q06

8 PM421Q03 8 PR404Q10B 8 PS425Q02

9 PM462Q01D 9 PR412Q01 9 PS425Q03

10 PM571Q01 10 PR412Q05 10 PS425Q04

11 PM603Q01T 11 PR412Q06T 11 PS425Q05

12 PM603Q02 12 PR412Q08 12 PS428Q01

13 PM828Q01 13 PR424Q02T 13 PS428Q03

14 PM828Q02 14 PR424Q03 14 PS428Q05

15 PM828Q03 15 PR424Q07 15 PS498Q02T

16 PM833Q01T 16 PR437Q01 16 PS498Q03

17 PM5002Q01 17 PR437Q06 17 PS498Q04

18 PM5002Q02 18 PR437Q07 18 PS527Q01T

19 PM5008Q01 19 PR455Q02 19 PS527Q03T

20 PM5010Q01T 20 PR455Q03 20 PS527Q04T

21 PM5010Q02 21 PR455Q04 21 PS7000Q01

22 PM5013Q01 22 PR455Q05T 22 PS7000Q02T

23 PM5013Q03 23 PR456Q01 23 PS7002Q03

24 PM5015Q02 24 PR456Q02 24 PS7002Q04

25 PM5017Q02 25 PR456Q06 25 PS7012Q01

26 PM5017Q03 26 PR466Q02 26 PS7012Q03

27 PM5020Q02 27 PR466Q03T 27 PS7012Q04T

28 PM5021Q01T 28 PR466Q06 28 PS7013Q01

29 PM5021Q02 29 PR6002Q01 29 PS7013Q02

30 PM5026Q01 30 PR6002Q02T 30 PS7014Q01

31 PM5101Q01 31 PR6002Q04 31 PS7014Q03

32 PM5101Q02 32 PR6002Q05 32 PS7017Q01

33 PM5102Q01 33 PR6004Q01 33 PS7017Q04D

34 PM5104Q01 34 PR6004Q05B 34 PS7018Q03

35 PM5107Q01 35 PR6004Q06 35 PS7018Q04T

36 PM5107Q02 36 PR6008Q02 36 PS7020Q03

45


37 PM5109Q01 37 PR6008Q04 37 PS7020Q04

38 PM5109Q02 38 PR6008Q05 38 PS7021Q02T

39 PM5110Q01 39 PR6010Q01 39 PS7021Q04T

40 PM5110Q02 40 PR6010Q03 40 PS7201Q01

41 PM5124Q01 41 PR6010Q06 41 PS7201Q02

42 PM5124Q02 42 PR6010Q07 42 PS7201Q03

43 PM5124Q03 43 PR6011Q03 43 PS7205Q02

44 PM5142Q01 44 PR6011Q05 44 PS7208Q01

45 PM5142Q02 45 PR6011Q07 45 PS7208Q02

46 PM5145Q01 46 PR6013Q03 46 PS7211Q01

47 PM5163Q01 47 PR6013Q04 47 PS7211Q02T

48 PM5169Q01 48 PR6013Q06 48 PS7212Q01

49 PM5169Q02 49 PR6013Q08 49 PS7212Q02

50 PM5181Q02 50 PR6014Q03 50 PS7212Q04

51 PM5183Q01 51 PR6014Q06T 51 PS7215Q02

52 PM5183Q02 52 PR6014Q07 52 PS7215Q03

53 PM5185Q01 53 PR6015Q01 53 PS7216Q01

54 PM5185Q02T 54 PR6015Q03 54 PS7216Q03

55 PM5187Q01 55 PR6015Q04 55 PS7217Q01T

56 PM5188Q01 56 PR6015Q06 56 PS7217Q02

57 PR6017Q04 57 PS7220Q01

58 PR6017Q05T 58 PS7220Q02T

59 PR6017Q08 59 PS7220Q03

60 PR6017Q10 60 PS7221Q01

61 PR6017Q11 61 PS7221Q02

62 PR6018Q01T 62 PS7221Q03T

63 PR6018Q03 63 PS7221Q04

64 PR6018Q08 64 PS7222Q01

65 PR6018Q09 65 PS7222Q02T

66 PR6023Q01 66 PS7222Q03T

67 PR6023Q06 67 PS7224Q02T

68 PR6023Q07 68 PS7228Q02

69 PR6023Q08 69 PS7228Q03

70 PR6026Q02T 70 PS7228Q04

71 PR6026Q05T 71 PS7230Q02T

72 PR6027Q01T 72 PS7230Q03T

73 PR6027Q04 73 PS7231Q01T

46


74 PR6027Q10 74 PS7231Q04T

Differential Item Functioning by Gender

149. Within the context of Rasch modelling, an item was deemed to exhibit differential

item functioning (DIF) if the response probabilities for that item could not be fully explained by

the ability of the student and a fixed set of difficulty parameters for that item.

150. In DIF analysis the aim was not to find items that have a higher or a lower p-value

for one group than another – since this may result from genuine differences in the proficiency

levels of the two groups. Rather the aim was to identify items that appear to be too difficult or

too easy, after controlling for differences in the average proficiency in the two groups. In the

PBT equating study gender and country DIF were explored. Country DIF is described in the

next section, as the larger number of groups requires a different reporting method than gender

DIF. The DIF analysis was performed using the multi-facet model of ACER ConQuest2. For

gender DIF, an item-by-gender interaction term was added to the standard scaling model.

151. The index of gender DIF was the difference between the estimated item difficulties

by group of interest (gender) adjusted for differences in overall performance between the groups

on all the items. In other words, an item shows gender DIF if boys of the same ability as girls

have different probabilities in responding correctly to this item.

152. Table 45 to Table 47 show the tabular report for gender DIF of mathematics,

reading and science for all the new PBT items, including those not selected. This table presents

the DIF parameters for the girls. The values for the boys are -1 * the parameters for the girls.

Therefore, the difference between the estimates for the girls and the boys is the same as 2 * the

parameters for the girls. DIF values of > 0.20 or < -0.20 are taken to indicate the existence of

DIF and are in bold. A positive value indicates that the item was harder for girls than for boys

having the same ability, a negative value indicates that it was easier for girls than for boys of

matched ability. ICC graphs by Gender are provided in Appendix 4 for the selected items to

illustrate the differences. The item numbers in Appendix 4 match those in Table 44.

2 Adams, R. J., Wu, M. L., & Wilson, M. R. (2012). ACER ConQuest Version 3: Generalised

item response modelling software [computer program]. Camberwell: Australian Council for

Educational Research.

47

Table 45: Gender DIF table for Mathematics

Item ID DIF Item ID DIF Item ID DIF

PM5002Q01 -0.191 PM5021Q01T -0.221 PM5140Q02 0.104

PM5002Q02 -0.21 PM5021Q02 0.172 PM5140Q03 0.277

PM5002Q03D -0.055 PM5022Q01 -0.083 PM5142Q01 0.143

PM5002Q04D -0.046 PM5022Q02D 0.169 PM5142Q02 -0.019

PM5004Q01 -0.143 PM5023Q01 -0.077 PM5145Q01 -0.105

PM5004Q02D 0.112 PM5023Q02 -0.026 PM5145Q02 -0.115

PM5006Q01 -0.002 PM5026Q01 0.191 PM5146Q01 0.04

PM5006Q02D -0.052 PM5101Q01 0.034 PM5146Q02 0.132

PM5007Q01 -0.122 PM5101Q02 -0.207 PM5160Q01 -0.098

PM5007Q02D -0.031 PM5102Q01 0.174 PM5163Q01 -0.158

PM5008Q01 -0.03 PM5104Q01 -0.022 PM5169Q01 0.172

PM5008Q02D 0.034 PM5104Q02 -0.246 PM5169Q02 0.151

PM5010Q01T 0.015 PM5106Q01 0.222 PM5181Q01 0.033

PM5010Q02 0.14 PM5106Q02 0.189 PM5181Q02 -0.03

PM5013Q01 0.045 PM5107Q01 -0.183 PM5183Q01 -0.006

PM5013Q02 0.104 PM5107Q02 0.112 PM5183Q02 0.003

PM5013Q03 -0.184 PM5109Q01 -0.112 PM5185Q01 -0.059

PM5015Q01D 0.059 PM5109Q02 0.075 PM5185Q02T 0.031

PM5015Q02 -0.108 PM5110Q01 0.067 PM5186Q03T 0.314

PM5017Q01 0.057 PM5110Q02 -0.108 PM5187Q01 0.261

PM5017Q02 0.074 PM5121Q01 -0.053 PM5187Q02 0.096

PM5017Q03 0.111 PM5124Q01 -0.093 PM5188Q01 -0.254

PM5020Q01 0.117 PM5124Q02 0.208 PM5188Q02 -0.007

PM5020Q02 0.018 PM5124Q03 -0.014 PM5189Q01 -0.157

PM5020Q03 0.185 PM5140Q01 -0.505 PM5189Q02 0.117

48

Table 46: Gender DIF table for Reading


PR6002Q01 -0.229 PR6013Q04 -0.103 PR6022Q01 -0.051

PR6002Q02T -0.229 PR6013Q05 0.292 PR6022Q03 -0.095

PR6002Q04 -0.063 PR6013Q06 -0.108 PR6022Q07 -0.077

PR6002Q05 -0.086 PR6013Q08 0.164 PR6022Q08 -0.141

PR6003Q01 -0.162 PR6014Q01 0.067 PR6022Q09 0.068

PR6003Q02 0.153 PR6014Q03 -0.154 PR6023Q01 0.081

PR6003Q04 0.059 PR6014Q06T -0.098 PR6023Q03 -0.094

PR6003Q05 0.226 PR6014Q07 0.021 PR6023Q06 0.063

PR6004Q01 -0.187 PR6015Q01 -0.079 PR6023Q07 0.172

PR6004Q02B -0.096 PR6015Q03 -0.1 PR6023Q08 0.168

PR6004Q03 -0.161 PR6015Q04 -0.139 PR6024Q01 -0.003

PR6004Q05B 0.25 PR6015Q06 -0.407 PR6024Q02 0.064

PR6004Q06 0.084 PR6015Q07 -0.122 PR6024Q04 -0.244

PR6006Q01 0.044 PR6017Q04 0.148 PR6024Q05 -0.24

PR6006Q02 0.077 PR6017Q05T 0.004 PR6024Q06 0.112

PR6006Q05B -0.186 PR6017Q08 0.132 PR6024Q10 -0.066

PR6008Q02 0.19 PR6017Q10 -0.003 PR6025Q01 0.045

PR6008Q03 -0.2 PR6017Q11 -0.069 PR6025Q05 -0.137

PR6008Q04 0.086 PR6018Q01T 0.114 PR6025Q07 0.235

PR6008Q05 -0.067 PR6018Q03 -0.139 PR6025Q08 0.086

PR6008Q06 -0.037 PR6018Q04T -0.066 PR6025Q10 -0.125

PR6010Q01 0.048 PR6018Q06 0.079 PR6026Q02T -0.127

PR6010Q03 0.083 PR6018Q08 -0.041 PR6026Q03 0.246

PR6010Q04 -0.007 PR6018Q09 -0.066 PR6026Q05T 0.28

PR6010Q06 -0.157 PR6019Q02 -0.071 PR6026Q09 -0.123

PR6010Q07 0.248 PR6019Q05 0.275 PR6027Q01T -0.194

PR6011Q01 0.136 PR6019Q07 -0.139 PR6027Q03T -0.198

PR6011Q03 0.015 PR6019Q08 0.01 PR6027Q04 0.195

PR6011Q05 -0.042 PR6019Q09 -0.136 PR6027Q06 0.12

PR6011Q06 0.142 PR6020Q02T 0.028 PR6027Q07 0.118

PR6011Q07 0.277 PR6020Q04 -0.046 PR6027Q10 -0.014

49


PR6013Q01 -0.053 PR6020Q07 0.014

PR6013Q03 0.254 PR6020Q09 0.19

50

Table 47: Gender DIF table for Science


PS7000Q01 -0.098 PS7211Q02T -0.271 PS7015Q01T 0.277

PS7000Q02T 0.059 PS7211Q04 -0.162 PS7015Q04 -0.178

PS7000Q04 -0.073 PS7212Q01 -0.193 PS7017Q03 0.114

PS7000Q05 0.296 PS7212Q04 0.149 PS7017Q05T 0.086

PS7002Q03 -0.114 PS7215Q02 0.018 PS7019Q01 0.121

PS7002Q04 -0.047 PS7215Q03 -0.038 PS7019Q02 -0.041

PS7006Q01 -0.117 PS7216Q01 0.051 PS7022Q01T 0.272

PS7006Q02 -0.007 PS7216Q03 -0.112 PS7022Q02D 0.012

PS7008Q01 -0.164 PS7217Q01T 0.03 PS7023Q01T 0.197

PS7008Q02 -0.03 PS7217Q02 0.104 PS7023Q02T 0.091

PS7008Q03 -0.042 PS7217Q03T 0.215 PS7203Q01 -0.239

PS7008Q05 -0.166 PS7218Q02 -0.028 PS7203Q03T 0.019

PS7012Q01 -0.099 PS7218Q03 -0.151 PS7203Q05 0.045

PS7012Q02T 0.05 PS7220Q01 -0.091 PS7205Q03 0.052

PS7012Q03 0.099 PS7220Q02T 0.283 PS7209Q01 0.027

PS7012Q04T 0.251 PS7220Q03 0.101 PS7212Q02 0.055

PS7013Q01 0.073 PS7221Q01 0.217 PS7215Q01 0.15

PS7013Q02 0.066 PS7221Q02 0.027 PS7216Q02T 0.105

PS7014Q01 -0.091 PS7221Q03T 0.322 PS7218Q01T -0.069

PS7014Q03 0.032 PS7221Q04 -0.176 PS7219Q01 -0.053

PS7014Q04 0.172 PS7222Q01 -0.104 PS7219Q02T -0.026

PS7015Q02 -0.192 PS7222Q02T 0.276 PS7219Q03T -0.014

PS7015Q03T 0.057 PS7222Q03T 0.027 PS7219Q04 -0.099

PS7017Q01 0.112 PS7224Q01 -0.004 PS7222Q04T 0.212

PS7017Q04D -0.125 PS7224Q02T -0.309 PS7224Q03 -0.103

PS7018Q03 -0.133 PS7228Q02 -0.243 PS7224Q05 -0.047

PS7018Q04T -0.303 PS7228Q03 -0.281 PS7228Q01 -0.025

PS7020Q03 -0.017 PS7228Q04 0.07 PS7229Q01 -0.118

PS7020Q04 0.005 PS7230Q02T -0.037 PS7229Q02 0.086

PS7021Q02T -0.058 PS7230Q03T 0.05 PS7229Q03T 0.226

PS7021Q04T 0.11 PS7230Q04T 0.167

51


PS7201Q01 0.093 PS7231Q01T 0.142

PS7201Q02 -0.042 PS7231Q04T 0.167

PS7201Q03 0.019 PS7002Q02T 0.414

PS7205Q02 -0.067 PS7004Q01 0.234

PS7205Q04 -0.402 PS7006Q03 0.045

PS7208Q01 -0.025 PS7009Q01 -0.305

PS7208Q02 -0.082 PS7009Q02T -0.067

PS7208Q03 -0.044 PS7009Q03 0.034

PS7209Q02 -0.358 PS7009Q04 -0.211

PS7209Q03 -0.483 PS7010Q02T 0.05

PS7211Q01 0.264 PS7010Q04 0.076

Item-by-country Interaction (Country DIF)

153. After allowing for the effect of multiple comparisons for comparing country item

parameters against each other, using similar methods as for PISA, items were flagged for

Country DIF if the ratio of the item parameter to the standard error was greater than 5. We show

here only those items that were flagged for Country DIF under this criterion. The items where

DIF was flagged for a country are indicated in bold. Of these items, two were retained in the

pilot trial instrument, PS7012Q01 and PS7221Q02, coverage and otherwise good psychometric

properties being judged more important than large Country DIF.

Table 48: Items exhibiting Country DIF

Phase 1 Items

Item AUS GBR IRL USA

Dif Dif SE Dif Dif SE Dif Dif SE Dif Dif SE

PM5002Q04D 0.838 0.156 -0.415 0.159 -0.357 0.144 -0.065 0.15

PR6008Q03 -0.491 0.222 0.095 0.214 1.017 0.201 -0.62 0.207

PS7012Q01 0.213 0.211 0.996 0.258 -0.951 0.185 -0.258 0.192

Phase 2 Items

AUS GBR IRL USA

Dif Dif SE Dif Dif SE Dif Dif SE Dif Dif SE

PS7205Q04 -1.524 0.25 0.486 0.202 0.31 0.186 0.728 0.194

PS7211Q04 0.184 0.19 -0.445 0.196 -0.761 0.185 1.022 0.198

PS7221Q02 0.693 0.139 -0.701 0.139 0.172 0.126 -0.164 0.131

52

Test Targeting

154. Uni-dimensional scaling with ACER ConQuest was used to illustrate the

distribution of students and items for the equating runs. In Figure 1 to Figure 3 the student

ability distribution is displayed by ‘X’ on the left side of the picture and the item thresholds are

displayed by the item number on the right, with the upper and lower thresholds for 0,1,2 partial

credit items indicated by .1 and .2. The number of students represented by each ‘X’ is

indicated. The PISA link items are indicated in red. There is in general a fair spread of items,

though some parts of the range are a little under-represented, such as the range between -1

and .-2 in science.

53

Figure 1: Mathematics item difficulty and student achievement distributions

54

Figure 2: Reading item difficulty and student achievement distributions

55

Figure 3: Science item difficulty and student achievement distributions

56

155. Note that these maps are calculated using the convention that a student with an

ability at the level of the item would have a 50% chance of answering that question correctly.

For the definition of proficiency levels, it was decided in PISA 2000 that a student in the middle

of a level would correctly answer an item of average difficulty for that level with probability

62.5%, which would imply that the student would answer questions at the bottom of the level

about 80% percent of the time, and those at the top about 50% of the time. We adopt this policy

here as well, and the items’ positions on the PISA scales reflect this. A full discussion of this

issue is given in the PISA 2000 Technical Report.

156. Figure 4, Figure 5 and Figure 6 show the distribution of items in the pilot trial set

for each domain, mapped onto the described proficiency scales for mathematics, reading and

science respectively that are used for reporting PISA main surveys. In these figures, the scale

scores of the items are used, as provided in Appendix 5 in the column “Thresholds”.

57

Mathematics

Level Lower score limit

Number of score points at level

Characteristics of tasks

6 669 20

At Level 6 students can conceptualise, generalise and utilise information based on their investigations and modelling of complex problem situations. They can link different information sources and representations and flexibly translate between them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understanding along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings, interpretations, arguments, and the appropriateness of these to the original situations.

5 607 8

At Level 5 students can develop and work with models for complex situations, identifying constraints and specifying assumptions. They can select, compare, and evaluate appropriate problem-solving strategies for dealing with complex problems related to these models. Students at this level can work strategically using broad, well-developed thinking and reasoning skills, appropriately linked representations, symbolic and formal characterisations, and insight pertaining to these situations. They can reflect on their actions and formulate and communicate their interpretations and reasoning.

4 545 8

At Level 4 students can work effectively with explicit models for complex concrete situations that may involve constraints or call for making assumptions. They can select and integrate different representations, including symbolic representations, linking them directly to aspects of real-world situations. Students at this level can utilise well-developed skills and reason flexibly, with some insight, in these contexts. They can construct and communicate explanations and arguments based on their interpretations, arguments and actions.

3 482 7

At Level 3 students can execute clearly described procedures, including those that require sequential decisions. They can select and apply simple problem-solving strategies. Students at this level can interpret and use representations based on different information sources and reason directly from them. They can develop short communications reporting their interpretations, results and reasoning.

2 420 1

At Level 2 students can interpret and recognise situations in contexts that require no more than direct inference. They can extract relevant information from a single source and make use of a single representational mode. Students at this level can employ basic algorithms, formulae, procedures, or conventions. They are capable of direct reasoning and literal interpretations of the results.

1 358 1

At Level 1 students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are obvious and follow immediately from the given stimuli.

Below Level 1

<358

1

(PM5185Q01 threshold at 331.35)

Figure 4: Summary descriptions for the six levels of proficiency in maths and the number

of score points at each level in the pilot trial instrument

58

Reading




6 698 3

Tasks at this level typically require the reader to make multiple inferences, comparisons and contrasts that are both detailed and precise. They require demonstration of a full and detailed understanding of one or more texts and may involve integrating information from more than one text. Tasks may require the reader to deal with unfamiliar ideas, in the presence of prominent competing information, and to generate abstract categories for interpretations. Reflect and evaluate tasks may require the reader to hypothesise about or critically evaluate a complex text on an unfamiliar topic, taking into account multiple criteria or perspectives, and applying sophisticated understandings from beyond the text. A salient condition for access and retrieve tasks at this level is precision of analysis and fine attention to detail that is inconspicuous in the texts.

5 626 4

Tasks at this level that involve retrieving information require the reader to locate and organise several pieces of deeply embedded information, inferring which information in the text is relevant. Reflective tasks require critical evaluation or hypothesis, drawing on specialised knowledge. Both interpretative and reflective tasks require a full and detailed understanding of a text whose content or form is unfamiliar. For all aspects of reading, tasks at this level typically involve dealing with concepts that are contrary to expectations.

4 553 8

Tasks at this level that involve retrieving information require the reader to locate and organise several pieces of embedded information. Some tasks at this level require interpreting the meaning of nuances of language in a section of text by taking into account the text as a whole. Other interpretative tasks require understanding and applying categories in an unfamiliar context. Reflective tasks at this level require readers to use formal or public knowledge to hypothesise about or critically evaluate a text. Readers must demonstrate an accurate understanding of long or complex texts whose content or form may be unfamiliar.

3 480 14

Tasks at this level require the reader to locate, and in some cases recognise the relationship between, several pieces of information that must meet multiple conditions. Interpretative tasks at this level require the reader to integrate several parts of a text in order to identify a main idea, understand a relationship or construe the meaning of a word or phrase. They need to take into account many features in comparing, contrasting or categorising. Often the required information is not prominent or there is much competing information; or there are other obstacles in the text, such as ideas that are contrary to expectation or negatively worded. Reflective tasks at this level may require connections, comparisons, and explanations, or they may require the reader to evaluate a feature of the text. Some reflective tasks require readers to demonstrate a fine understanding of the text in relation to familiar, everyday knowledge. Other tasks do not require detailed text comprehension but require the reader to draw on less common knowledge.

2 407 10

Some tasks at this level require the reader to locate one or more pieces of information, which may need to be inferred and may need to meet several conditions. Others require recognising the main idea in a text, understanding relationships, or construing meaning within a limited part of the text when the information is not prominent and the reader must make low level inferences. Tasks at this level may involve comparisons or contrasts based on a single feature in the text. Typical reflective tasks at this level require readers to make a comparison or several connections between the text and outside knowledge, by drawing on personal experience and attitudes.

1a 335 8

Tasks at this level require the reader: to locate one or more independent pieces of explicitly stated information; to recognise the main theme or author’s purpose in a text about a familiar topic; or to make a simple connection between information in the text and common, everyday knowledge. Typically the required information in the text is prominent and there is little, if any, competing information. The reader is explicitly directed to consider relevant factors in the task and in the text.

1b 262 2

Tasks at this level require the reader to locate a single piece of explicitly stated information in a prominent position in a short, syntactically simple text with a familiar context and text type, such as a narrative or a simple list. The text typically provides support to the reader, such as repetition of information, pictures or familiar symbols. There is minimal competing information. In tasks requiring interpretation the reader may need to make simple connections between adjacent pieces of information.

Figure 5: Summary descriptions for the seven levels of proficiency in reading and the

number of score points at each level in the pilot trial instrument

59

Science




6 708 3

At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they demonstrate willingness to use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.

5 633 10

At Level 5, students can identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to situations. They can construct explanations based on evidence and arguments based on their critical analysis.

4 559 20

At Level 4, students can work effectively with situations and issues that may involve explicit phenomena requiring them to make inferences about the role of science or technology. They can select and integrate explanations from different disciplines of science or technology and link those explanations directly to aspects of life situations. Students at this level can reflect on their actions and they can communicate decisions using scientific knowledge and evidence.

3 484 14

At Level 3, students can identify clearly described scientific issues in a range of contexts. They can select facts and knowledge to explain phenomena and apply simple models or inquiry strategies. Students at this level can interpret and use scientific concepts from different disciplines and can apply them directly. They can develop short statements using facts and make decisions based on scientific knowledge.

2 409 9

At Level 2, students have adequate scientific knowledge to provide possible explanations in familiar contexts or draw conclusions based on simple investigations. They are capable of direct reasoning and making literal interpretations of the results of scientific inquiry or technological problem solving.

1 335 1 At Level 1, students have such a limited scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific explanations that are obvious and follow explicitly from given evidence.

Below Level 1

<335 2

(PS7221Q01 threshold at 246.19 and PS7221Q04 threshold at 296.92)

Figure 6: Summary descriptions for the six levels of proficiency in science and the number

of score points at each level in the pilot trial instrument

60

Chapter 6: Proposal for replenishment of items

157. The contract for the development of the Phase 2 field trial and equating study for

the PISA-Based Test for Schools (PBT) calls for the development of “a proposal for the

replenishment of approximately 50% of the items after the first round of PISA for Schools and

during the initial three-year period. The proposal should consider how the item replenishment

process would maintain adequate and balanced PISA frameworks coverage, item difficulty

distributions, and all trends analysis.” (Contract M11/45, page 12)

158. Replenishment of items is needed to maintain the security of the test, to ensure that

the items in the PISA Based Test are of the highest technical quality, to ensure that the item

pool maintains currency, and possibly to increase the number of items in the item pool. At the

same time, a sufficient number of items should be retained after any replenishment to ensure

robust trend analysis and reporting.

159. We present three different scenarios which describe the possible replenishment of

items within the first three years of implementation of the PISA-Based Test for Schools as a live

assessment – starting late in 2012 after the completion of the pilot trial currently underway.

Scenario 1 – Replenish the item pool incrementally to continue with 90 minutes of

assessment time in each domain

160. In the first scenario the PBT remains as a test which assesses students for a

maximum of 90 minutes in each domain. Table 49 shows a possible design for this to be

achieved over a three year period from end of 2012 to the end of 2015.

Table 49 Item replenishment to replace 50% of the item pool incrementally

Original

Reading

Items

left

Total

New

Reading

items

Total

Reading

items

Original

Maths

Items

left

Total

New

Maths

items

Total

Maths

items

Original

Science

Items

left

Total

New

Science

items

Total

Science

items

Into pilot trial

47

47 40

40 54

54

Mid-2014 35 12 47 30 10 40 41 13 54

End of 2015

23 24 47 20 20 40 27 27 54

161. By mid-2014 it is proposed that there be 12 new items in reading, 10 new items in

mathematics and 13 new items in science.

162. At the end of 2015 it is proposed that there be an additional 12 new items in

reading, 10 new items in mathematics and 14 new items in science.

163. The cluster design of approximately 40 minutes would be retained with this scheme,

with the new items being introduced as replacements within the existing clusters.

164. In this way half of the original pilot trial items will have been replaced by the end of

2015 while retaining sufficient items to allow for a measure of change if a school opts to do the

test at different times.

165. An advantage of this scenario is that the booklet design would remain stable within

the program. A further advantage is that the choice of which items to replace will be

unconstrained in comparison with the second scenario. A disadvantage is that items can be

affected by “the company they keep”, so that changing the composition of clusters by including

61

new items and removing old ones may change the performance of the remaining items, upon

which trend measures will depend.

Scenario 2 – Replenish the item pool by replacing clusters, to continue with 90

minutes of assessment time in each domain

166. In the second scenario the PBT remains as a test which assesses students for a

maximum of 90 minutes in each domain. Unlike in Scenario 1, intact clusters are replaced. In

the current pilot trial test design there are two clusters each of mathematics, reading and science

items, and one mixed cluster including items from all three domains. Therefore, either three-

sevenths of the items would be replaced – a little less than 50% (one of each of the pure

mathematics, reading and science clusters); or four-sevenths of the items will be replaced – a

little more than 50% (one each of mathematics, reading and science, and the mixed cluster). In

either case, the replacement items would be introduced in one change-over. This could be at the

end of 2014 or the end of 2015. Table 50 shows one of the various possibilities: a four cluster

replacement at the end of 2014.

Table 50 Item replenishment to replace at least 50% of the item pool, by cluster

Original

Reading

Items

left

Total

New

Reading

items

Total

Reading

items

Original

Maths

Items

left

Total

New

Maths

items

Total

Maths

items

Original

Science

Items

left

Total

New

Science

items

Total

Science

items

Into pilot trial

47

47 40

40 54

54

End of 2014

20 27 47 17 23 40 23 31 54

167. Under this variation, at the end of 2014 it is proposed that there be 27 new items in

reading, 23 new items in mathematics and 31 new items in science.

168. The cluster design of approximately 40 minutes would be retained with this scheme,

with the new items being introduced as intact replacement clusters.

169. In the version of Scenario 2 shown in Table 50 more than half of the original pilot

trial items will have been replaced by the end of 2014 while retaining sufficient items to allow

for a measure of change if a school opts to do the test at different times.

170. An advantage of this scenario is that the booklet design would remain stable within

the program. A further advantage, in comparison with Scenario 1, is that any effects of item

replenishment would be minimised, since retained items, used for trend measurement, would be

in the same company as in the previous administrations. Conversely, the choice of which items

to replace will be constrained in comparison with Scenario 1, since the decision for Scenario 2

must be made on the basis of whole clusters.

Scenario 3 – Increase the item pool to allow 120 minute testing in each domain

171. The test design in the pilot trial includes just over 90 minutes testing in each of the

three domains of reading, mathematics and science. It is possible, even likely, that some schools

may wish to test in only one domain in a 120-minute assessment – the normal PISA test length.

This would allow the schools to have a greater degree of detail than is possible in the pilot trial,

and could also allow for consideration of subscale reporting.

172. It is proposed in this third scenario, therefore, to increase the number of items so

that 120 minutes of testing per domain is included in the item pool. This scenario also allows for

the replacement of 50% of the original items. Table 51 shows a possible design for this to be

achieved over a three year period from the end of 2012 to the end of 2015.

62

Table 51 Item replenishment to increase the item pool

Original

Reading

Items

left

Total

New

Reading

items

Total

Reading

items

Original

Maths

Items

left

Total

New

Maths

items

Total

Maths

items

Original

Science

Items

left

Total

New

Science

items

Total

Science

items

Into pilot trial

47

47 40

40 54

54

Mid-2014 38 23 61 32 20 52 44 26 70

End of 2015

23 38 61 20 32 52 27 43 70

173. It will be seen that there is an increase to 61 items for reading, 52 for mathematics

and 70 for science. It is proposed that this increase in items occur by mid-2014 so that a 120

minute test is available to schools for the beginning of the Northern Hemisphere school year

2014-5. This would require the creation of 23 new items in reading made up of 9 items to

replace items from the pilot trial and 14 items to increase the time to 120 minutes for reading. It

would require the creation of 20 new items in mathematics made up of 8 items to replace

unsuitable items from the pilot trial and 12 items to increase the time to 120 minutes for

mathematics. It would require the creation of 26 new items in science made up of 12 items to

replace items from the pilot trial and 17 items to increase the time to 120 minutes in science.

174. By the end of 2015, it is proposed to introduce an additional set of replenishment

items, to replace one per domain of the 30 minute clusters assembled in mid-2014. This would

involve including an additional 15 new reading items, 12 new mathematics items and 17 new

science items. Half of the original items from the pilot trial would still be retained in each

domain, as shown in the last row of Table 51.

175. In this way half of the original pilot trial items have been replaced and the number

in the item pool has increased to allow for a 120 minute test in a single domain. This will be

done while retaining sufficient items to allow for a measure of change if a school opts to do the

test at different times.

176. An advantage of Scenario 3 is that it will give schools more flexibility in choosing

whether to administer one or all domains; it will also allow for the possibility of reporting on

subscales, which would enhance the diagnostic potential of the assessment at the school, district

or regional level. It will also bring the PISA-Based Test for Schools closer to the PISA study in

its design.

177. Disadvantages of Scenario 3 are that it is expensive relative to Scenarios 1 and 2

(requiring more item development), and that there may be a disruption of trend in mid-2014

because of the change in booklet design.

Procedure for item creation and equating

178. The procedure adopted for PBT item development, modelled on PISA item

development, will be followed for the creation of all new items for the PBT. This means that

working back from the number of new items desired, four times the number of items will be

created in the first instance, followed by a refinement process including cognitive laboratory

methods to select half of those items.

179. These items will be placed into a field trial in a number of countries, similar to the

process followed for the field trial and equating studies which were implemented for the

creation of the original pilot trial instrument. The possibility of including secure PISA items in

this field trial should also be considered to ensure maintenance of robust equating with PISA.

63

180. Following the field trial half of the items would be selected to be included in the

ongoing PISA-Based Test for Schools. The final selection of items will be made to ensure that

coverage of the PISA Assessment Frameworks and the range of difficulty of the assessment

items is maintained.

181. All three scenarios described offer the opportunity to replace 50% of the items in

the PBT while at the same time ensuring framework coverage.

64

Appendix 1: Audit of items developed for the Phase 1 and Phase 2 FT-ES

Mathematics

Unit Item

Code Unit Name Item Format 2012 Content Context Process Developed for

Equating

Study

PM5000Q01 Large Hadron Collider Simple Multiple Choice Space and shape Scientific Employing Phase 1 No

PM5000Q02 Large Hadron Collider Constructed Response Manual Space and shape Scientific Employing Phase 1 No

PM5000Q03 Large Hadron Collider Constructed Response Manual Quantity Scientific Employing Phase 1 No

PM5001Q01 Traffic Lights Constructed Response Manual Quantity Societal Employing Phase 1 No

PM5001Q02 Traffic Lights Constructed Response Expert Quantity Societal Interpreting Phase 1 No

PM5002Q01 Internet Use Simple Multiple Choice Uncertainty and data Societal Employing Phase 1 Phase 1 & 2

PM5002Q02 Internet Use Simple Multiple Choice Quantity Societal Employing Phase 1 Phase 1 & 2

PM5002Q03 Internet Use Constructed Response Expert Change and relationships Societal Employing Phase 1 Phase 1

PM5002Q04 Internet Use Constructed Response Expert Change and relationships Societal Formulating Phase 1 Phase 1

PM5003Q01 TV News Constructed Response Manual Quantity Occupational Employing Phase 1 No

PM5003Q02 TV News Simple Multiple Choice Quantity Occupational Employing Phase 1 No

PM5004Q01 Watering A Football Pitch

Constructed Response Manual Quantity Occupational Employing Phase 1 Phase 1

PM5004Q02 Watering A Football Pitch Constructed Response Expert Quantity Occupational Interpreting Phase 1 Phase 1

PM5005Q01 Doubling Rules Simple Multiple Choice Change and relationships Societal Employing Phase 1 No

65

Unit Item


Equating

Study

PM5005Q02 Doubling Rules Constructed Response Manual

Change and relationships Scientific Employing Phase 1 No

PM5006Q01 Cylinder Lock Simple Multiple Choice Uncertainty and data Societal Employing Phase 1 Phase 1

PM5006Q02 Cylinder Lock Constructed Response Expert Uncertainty and data Scientific Interpreting Phase 1 Phase 1

PM5007Q01 Sun Room Constructed Response Manual Space and shape Societal Employing Phase 1 Phase 1

PM5007Q02 Sun Room Constructed Response Expert Space and shape Societal Employing Phase 1 Phase 1

PM5008Q01 Tiling Pattern Constructed Response Manual Space and shape Societal Employing Phase 1 Phase 1 & 2

PM5008Q02 Tiling Pattern Constructed Response Expert Space and shape Societal Employing Phase 1 Phase 1 & 2

PM5009Q01 Dress Pattern Constructed Response Expert Space and shape Personal Employing Phase 1 No

PM5009Q02 Dress Pattern Constructed Response Expert Space and shape Personal Employing Phase 1 No

PM5010Q01 Climbing Frame Complex Multiple Choice Space and shape Personal Interpreting Phase 1 Phase 1 & 2

PM5010Q02 Climbing Frame Constructed Response Manual Space and shape Personal Formulating Phase 1 Phase 1 & 2

PM5011Q01 Pentagon Knot Constructed Response Manual Space and shape Personal Employing Phase 1 No

PM5011Q02 Pentagon Knot Simple Multiple Choice Space and shape Personal Interpreting Phase 1 No

PM5012Q01 Trucking Business Constructed Response Expert Uncertainty and data Occupational Employing Phase 1 No



PM5013Q01 Energy Use Simple Multiple Choice Uncertainty and data Scientific Employing Phase 1 Phase 1 & 2

PM5013Q02 Energy Use Constructed Response Manual Uncertainty and data Scientific Employing Phase 1 Phase 1

PM5013Q03 Energy Use Constructed Response Manual Uncertainty and data Scientific Interpreting Phase 1 Phase 1 & 2

66

Unit Item


Equating

Study

PM5014Q01 Truck Constructed Response Manual

Change and relationships Occupational Employing Phase 1 No

PM5014Q02 Truck Simple Multiple Choice Quantity Occupational Employing Phase 1 No

PM5015Q01 Aspect Ratio Constructed Response Expert Quantity Personal Employing originally developed for PISA 2012, revised for PCT Phase 1

PM5015Q02 Aspect Ratio Simple Multiple Choice Change and relationships Personal Employing

originally developed for PISA 2012, revised for PCT Phase 1 & 2

PM5016Q01 Download Simple Multiple Choice Quantity Scientific Employing originally developed for PISA 2012, revised for PCT No

PM5016Q02 Download Constructed Response Manual Quantity Personal Employing

originally developed for PISA 2012, revised for PCT No

PM5017Q01 Weather Forecast Simple Multiple Choice Uncertainty and data Scientific Employing originally developed for PISA 2012, revised for PCT Phase 1

PM5017Q02 Weather Forecast Simple Multiple Choice Uncertainty and data Scientific Interpreting originally developed for PISA 2012, revised for PCT Phase 1 & 2

PM5017Q03 Weather Forecast Constructed Response Expert Uncertainty and data Scientific Formulating originally developed for PISA 2012, revised for PCT Phase 1 & 2

PM5018Q01 Steam Car Record Simple Multiple Choice Quantity Scientific Formulating originally developed for PISA 2012, revised for PCT No

PM5018Q02 Steam Car Record Constructed Response Expert Change and relationships Scientific Formulating


PM5019Q01 Angkor Wat Simple Multiple Choice Space and shape Societal Interpreting originally developed for PISA 2012, revised for PCT No

PM5019Q02 Angkor Wat Simple Multiple Choice Space and shape Societal Employing originally developed for PISA 2012, revised for PCT No

PM5019Q03 Angkor Wat Constructed Response Expert Quantity Societal Interpreting originally developed for PISA 2012, revised for PCT No

PM5020Q01 Golden Rectangle Simple Multiple Choice Change and Scientific Employing originally developed for PISA Phase 1

67

Unit Item


Equating

Study

relationships 2012, revised for PCT

PM5020Q02 Golden Rectangle Constructed Response Expert Change and relationships Scientific Employing


PM5020Q03 Golden Rectangle Simple Multiple Choice Change and relationships Scientific Formulating


PM5021Q01 Freezer Temperatures Complex Multiple Choice Quantity Occupational Interpreting


PM5021Q02 Freezer Temperatures

Constructed Response Manual Quantity Occupational Employing


PM5022Q01 Which Formula? Simple Multiple Choice Change and relationships Occupational Formulating


PM5022Q02 Which Formula? Constructed Response Expert Change and relationships Occupational Formulating


PM5023Q01 Old Drills Simple Multiple Choice Quantity Occupational Employing originally developed for PISA 2012, revised for PCT Phase 1

PM5023Q02 Old Drills Simple Multiple Choice Quantity Occupational Employing originally developed for PISA 2012, revised for PCT Phase 1

PM5024Q01 Tower Simple Multiple Choice Change and relationships Scientific Employing Phase 1 No

PM5024Q02 Tower Simple Multiple Choice Change and relationships Scientific Employing Phase 1 No

PM5025Q01 Penrose Triangle Constructed Response Manual Space and shape Personal Employing


PM5025Q02 Penrose Triangle Constructed Response Manual Space and shape Personal Employing


PM5026Q01 Sculpture Constructed Response Expert Change and relationships Scientific Formulating


PM5026Q02 Sculpture Constructed Response Change and Scientific Employing originally developed for PISA No

68

Unit Item


Equating

Study

Manual relationships 2012, revised for PCT

PM5027Q01 Space Station Constructed Response Manual Quantity Scientific Employing


PM5027Q02 Space Station Constructed Response Manual

Change and relationships Scientific Formulating


PM5027Q03 Space Station Constructed Response Expert Change and relationships Scientific Formulating


PM5100Q01 Ladder Simple Multiple Choice Change and relationships Occupational Interpreting Phase 2 No

PM5100Q02 Ladder Constructed Response Manual Space and shape Occupational Employing Phase 2 No

PM5101Q01 Lighthouse Simple Multiple Choice Space and shape Scientific Formulating Phase 2 Phase 2

PM5101Q02 Lighthouse Constructed Response Manual Quantity Scientific Employing Phase 2 Phase 2

PM5102Q01 Tree Trunk Simple Multiple Choice Change and relationships Scientific Employing Phase 2 Phase 2

PM5103Q01 Basketball Squad Simple Multiple Choice Uncertainty and data Societal Employing Phase 2 No

PM5103Q02 Basketball Squad Constructed Response Manual Uncertainty and data Societal Interpreting Phase 2 No

PM5103Q03 Basketball Squad Simple Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

PM5104Q01 Baby Growth Simple Multiple Choice Uncertainty and data Personal Formulating Phase 2 Phase 2

PM5104Q02 Baby Growth Simple Multiple Choice Uncertainty and data Personal Employing Phase 2 Phase 2

PM5105Q01 Trout Farm Constructed Response Manual Uncertainty and data Occupational Employing Phase 2 No

PM5105Q02 Trout Farm Complex Multiple Choice Uncertainty and data Occupational Interpreting Phase 2 No

PM5106Q01 Free Tickets Simple Multiple Choice Uncertainty and data Societal Formulating Phase 2 Phase 2

69

Unit Item


Equating

Study

PM5106Q02 Free Tickets Constructed Response Expert Uncertainty and data Societal Formulating Phase 2 Phase 2

PM5107Q01 Taxi Fares Constructed Response Manual Quantity Societal Employing Phase 2 Phase 2

PM5107Q02 Taxi Fares Constructed Response Expert Quantity Societal Employing Phase 2 Phase 2

PM5108Q01 Tiles Constructed Response Manual Space and shape Occupational Formulating Phase 2 No

PM5108Q02 Tiles Constructed Response Manual Quantity Occupational Employing Phase 2 No

PM5109Q01 Arrowhead Kite Constructed Response Manual Space and shape Personal Employing Phase 2 Phase 2

PM5109Q02 Arrowhead Kite Constructed Response Expert Space and shape Personal Employing Phase 2 Phase 2

PM5110Q01 Fence Constructed Response Manual

Change and relationships Occupational Formulating Phase 2 Phase 2

PM5110Q02 Fence Constructed Response Expert Change and relationships Occupational Formulating Phase 2 Phase 2

PM5120Q01 Rolling Ball Game Constructed Response Manual Quantity Personal Employing Phase 2 No

PM5120Q02 Rolling Ball Game Complex Multiple Choice Change and relationships Personal Formulating Phase 2 No

PM5120Q03 Rolling Ball Game Constructed Response Manual Uncertainty and data Personal Formulating Phase 2 No

PM5121Q01 Leisure Time Simple Multiple Choice Uncertainty and data Societal Formulating Phase 2 Phase 2

PM5121Q02 Leisure Time Complex Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

PM5122Q01 Picture Frames Simple Multiple Choice Quantity Occupational Employing Phase 2 No

PM5122Q02 Picture Frames Constructed Response Manual Quantity Occupational Formulating Phase 2 No

70

Unit Item


Equating

Study

PM5122Q03 Picture Frames Constructed Response Manual


PM5123Q01 Olympic Gold Medals Complex Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

PM5123Q02 Olympic Gold Medals Simple Multiple Choice Uncertainty and data Societal Employing Phase 2 No

PM5124Q01 Concrete Constructed Response Manual






PM5140Q01 Botswana Simple Multiple Choice Space and shape Societal Interpreting Phase 2 Phase 2

PM5140Q02 Botswana Constructed Response Manual Space and shape Societal Formulating Phase 2 Phase 2

PM5140Q03 Botswana Simple Multiple Choice Quantity Societal Employing Phase 2 Phase 2

PM5141Q01 Contours Simple Multiple Choice Space and shape Scientific Interpreting Phase 2 No

PM5141Q02 Contours Constructed Response Manual Quantity Scientific Interpreting Phase 2 No

PM5141Q03 Contours Simple Multiple Choice Change and relationships Scientific Interpreting Phase 2 No

PM5142Q01 Machu Picchu Constructed Response Manual Quantity Societal Interpreting Phase 2 Phase 2

PM5142Q02 Machu Picchu Constructed Response Manual Quantity Societal Employing Phase 2 Phase 2

PM5142Q03 Machu Picchu Simple Multiple Choice Quantity Societal Formulating Phase 2 No

PM5143Q01 Titan Simple Multiple Choice Quantity Scientific Formulating Phase 2 No

PM5143Q02 Titan Simple Multiple Choice Quantity Scientific Employing Phase 2 No

71

Unit Item


Equating

Study

PM5143Q03 Titan Constructed Response Manual Quantity Scientific Employing Phase 2 No

PM5144Q01 Can Of Drink Simple Multiple Choice Quantity Societal Employing Phase 2 No

PM5145Q01 Decagon Tiles Constructed Response Manual Space and shape Scientific Employing Phase 2 Phase 2

PM5145Q02 Decagon Tiles Constructed Response Expert Space and shape Scientific Employing Phase 2 Phase 2

PM5146Q01 Iron-On Image Simple Multiple Choice Space and shape Occupational Employing Phase 2 Phase 2

PM5146Q02 Iron-On Image Constructed Response Manual Quantity Occupational Formulating Phase 2 Phase 2

PM5146Q03 Iron-On Image Simple Multiple Choice Quantity Occupational Employing Phase 2 No

PM5147Q01 Road Map Simple Multiple Choice Space and shape Personal Interpreting Phase 2 No

PM5147Q02 Road Map Constructed Response Manual Space and shape Personal Employing Phase 2 No

PM5148Q01 Hedge Simple Multiple Choice Space and shape Occupational Interpreting Phase 2 No

PM5149Q01 Race Track Simple Multiple Choice Space and shape Societal Interpreting Phase 2 No

PM5149Q02 Race Track Complex Multiple Choice Space and shape Societal Interpreting Phase 2 No

PM5150Q01 Eight Sided Dice Simple Multiple Choice Space and shape Occupational Interpreting Phase 2 No

PM5160Q01 Flatback Turtles Simple Multiple Choice Quantity Scientific Employing Phase 2 Phase 2

PM5161Q01 Aeroplane Landing Constructed Response Manual


PM5161Q02 Aeroplane Landing Constructed Response Expert Change and relationships Occupational Interpreting Phase 2 No

PM5162Q01 New Runway Constructed Response Manual Quantity Occupational Employing Phase 2 No

PM5163Q01 Hair Gel Constructed Response Manual Quantity Personal Employing Phase 2 Phase 2

72

Unit Item


Equating

Study

PM5164Q01 Gas Cylinders Constructed Response Manual Quantity Occupational Employing Phase 2 No

PM5165Q01 Internet Advertising Constructed Response Manual


PM5165Q02 Internet Advertising Constructed Response Manual

Change and relationships Occupational Formulating Phase 2 No

PM5166Q01 Silk Scarves Constructed Response Manual Space and shape Personal Employing Phase 2 No

PM5167Q01 Digital Music Constructed Response Manual Uncertainty and data Personal Employing Phase 2 No

PM5168Q01 Flag Of Zedland Constructed Response Manual Space and shape Societal Employing Phase 2 No

PM5168Q02 Flag Of Zedland Constructed Response Manual Quantity Societal Employing Phase 2 No

PM5168Q03 Flag Of Zedland Complex Multiple Choice Space and shape Societal Employing Phase 2 No

PM5169Q01 Shoe Sizes Constructed Response Manual

Change and relationships Personal Employing Phase 2 Phase 2

PM5169Q02 Shoe Sizes Constructed Response Manual

Change and relationships Personal Employing Phase 2 Phase 2

PM5181Q01 Pendulum Constructed Response Manual Uncertainty and data Scientific Employing Phase 2 Phase 2

PM5181Q02 Pendulum Constructed Response Manual

Change and relationships Scientific Employing Phase 2 Phase 2

PM5182Q01 Network Growth Simple Multiple Choice Quantity Societal Employing Phase 2 No

PM5182Q02 Network Growth Constructed Response Expert Quantity Societal Formulating Phase 2 No

PM5183Q01 House Frame Constructed Response Expert Space and shape Occupational Interpreting Phase 2 Phase 2

PM5183Q02 House Frame Simple Multiple Choice Space and shape Occupational Formulating Phase 2 Phase 2

73

Unit Item


Equating

Study

PM5184Q01 Networking Statistics Constructed Response Expert Uncertainty and data Societal Employing Phase 2 No

PM5184Q02 Networking Statistics Complex Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

PM5185Q01 Jeans Sizes Simple Multiple Choice Uncertainty and data Societal Interpreting Phase 2 Phase 2

PM5185Q02 Jeans Sizes Complex Multiple Choice Change and relationships Societal Interpreting Phase 2 Phase 2

PM5186Q01 Age Of Mothers Simple Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

PM5186Q02 Age Of Mothers Simple Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

PM5186Q03 Age Of Mothers Complex Multiple Choice Uncertainty and data Societal Employing Phase 2 Phase 2

PM5186Q04 Age Of Mothers Constructed Response Manual Uncertainty and data Societal Interpreting Phase 2 No

PM5187Q01 Jaden’s House Simple Multiple Choice Space and shape Personal Interpreting Phase 2 Phase 2

PM5187Q02 Jaden’s House Constructed Response Expert Space and shape Personal Employing Phase 2 Phase 2

PM5188Q01 Car Speed Constructed Response Manual Uncertainty and data Societal Employing Phase 2 Phase 2

PM5188Q02 Car Speed Constructed Response Manual Uncertainty and data Societal Employing Phase 2 Phase 2

PM5189Q01 Fill A Room Competition Constructed Response Expert Quantity Personal Employing Phase 2 Phase 2

PM5189Q02 Fill A Room Competition Constructed Response Expert Quantity Personal Employing Phase 2 Phase 2

PM5191Q01 Word Length Constructed Response Manual Uncertainty and data Societal Interpreting Phase 2 No

PM5191Q02 Word Length Complex Multiple Choice Uncertainty and data Societal Interpreting Phase 2 No

74

Reading

Unit Item

Code Unit Name Item Format 2012 Situation Text Format Text Type Aspect

Developed

for

Equating

Study

PR6000Q01 Honeybee Simple Multiple Choice Educational Mixed Description Access and retrieve Phase 1 No

PR6000Q02 Honeybee Complex Multiple Choice Educational Mixed Description Access and retrieve Phase 1 No

PR6000Q03A Honeybee Simple Multiple Choice Educational Mixed Description Integrate and interpret Phase 1 No

PR6000Q03B Honeybee Constructed Response Expert Educational Mixed Description Reflect and evaluate Phase 1 No

PR6000Q04 Honeybee Constructed Response Expert Educational Mixed Description Integrate and interpret Phase 1 No

PR6000Q05 Honeybee Constructed Response Expert Educational Mixed Description Reflect and evaluate Phase 1 No

PR6001Q01 Longer Life Simple Multiple Choice Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6001Q02 Longer Life Constructed Response Expert Personal Continuous Exposition Reflect and evaluate Phase 1 No

PR6001Q02X Longer Life Constructed Response Expert Personal Continuous Exposition Reflect and evaluate Phase 1 No

PR6001Q03 Longer Life Simple Multiple Choice Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6001Q04 Longer Life Constructed Response Expert Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6001Q05 Longer Life Constructed Response Expert Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6002Q01 Egyptian Mummies Simple Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 Phase 1

75

Unit Item


Developed

for

Equating

Study

PR6002Q02 Egyptian Mummies Complex Multiple Choice Educational Continuous Exposition Access and retrieve Phase 1 Phase 1

PR6002Q03 Egyptian Mummies Complex Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 No

PR6002Q04 Egyptian Mummies Constructed Response Expert Educational Continuous Exposition Access and retrieve Phase 1 Phase 1

PR6002Q05 Egyptian Mummies Simple Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 Phase 1

PR6002Q06 Egyptian Mummies Simple Multiple Choice Educational Continuous Exposition Reflect and evaluate Phase 1 No

PR6002Q07 Egyptian Mummies Constructed Response Expert Educational Continuous Exposition Reflect and evaluate Phase 1 No

PR6003Q01 Indian Mystic Simple Multiple Choice Personal Continuous Argumentation Integrate and interpret Phase 1 Phase 1

PR6003Q02 Indian Mystic Constructed Response Expert Personal Continuous Argumentation Integrate and interpret Phase 1 Phase 1 & 2

PR6003Q03 Indian Mystic Simple Multiple Choice Personal Continuous Argumentation Reflect and evaluate Phase 1 No

PR6003Q04 Indian Mystic Simple Multiple Choice Personal Continuous Argumentation Integrate and interpret Phase 1 Phase 1 & 2

PR6003Q05 Indian Mystic Constructed Response Expert Personal Continuous Argumentation Reflect and evaluate Phase 1 Phase 1 & 2

PR6003Q06 Indian Mystic Simple Multiple Choice Personal Continuous Argumentation Integrate and interpret Phase 1 No

PR6003Q07 Indian Mystic Simple Multiple Choice Personal Continuous Argumentation Reflect and evaluate Phase 1 No

PR6004Q01 Museum Park Constructed Response Manual Public Non-continuous Description

Access and retrieve Phase 1 Phase 1 & 2

76

Unit Item


Developed

for

Equating

Study

PR6004Q02A Museum Park Constructed Response Manual Public Non-continuous Description

Integrate and interpret Phase 1 Phase 1

PR6004Q02B Museum Park Constructed Response Expert Public Non-continuous Description Integrate and interpret Phase 1 Phase 1

PR6004Q03 Museum Park Simple Multiple Choice Public Non-continuous Description Access and retrieve Phase 1 Phase 1 & 2

PR6004Q04 Museum Park Simple Multiple Choice Public Non-continuous Description Access and retrieve Phase 1 No

PR6004Q05A Museum Park Not scored

Phase 1 Phase 1 & 2

PR6004Q05B Museum Park Constructed Response Expert Public Non-continuous Description Reflect and evaluate Phase 1 Phase 1 & 2

PR6004Q06 Museum Park Simple Multiple Choice Public Non-continuous Description Access and retrieve Phase 1 Phase 1 & 2

PR6005Q01 Eye Direction and Lying Simple Multiple Choice Personal Mixed Description

Integrate and interpret Phase 1 No


Access and retrieve Phase 1 No

PR6005Q03 Eye Direction and Lying

Constructed Response Manual Personal Mixed Description






PR6005Q06 Eye Direction and Lying Constructed Response Expert Personal Mixed Description

Reflect and evaluate Phase 1 No

PR6005Q07 Eye Direction and Lying Constructed Response Expert Personal Mixed Description


77

Unit Item


Developed

for

Equating

Study

PR6006Q01 Free Simple Multiple Choice Personal Continuous Exposition Integrate and interpret Phase 1 Phase 1

PR6006Q01X Free Simple Multiple Choice Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6006Q02 Free Constructed Response Expert Personal Continuous Exposition Reflect and evaluate Phase 1 Phase 1

PR6006Q04 Free Simple Multiple Choice Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6006Q05A Free Simple Multiple Choice Personal Continuous Exposition Reflect and evaluate Phase 1 Phase 1

PR6006Q05B Free Constructed Response Expert Personal Continuous Exposition Reflect and evaluate Phase 1 Phase 1

PR6006Q05X Free Constructed Response Expert Personal Continuous Exposition Integrate and interpret Phase 1 No

PR6006Q06 Free Constructed Response Expert Personal Continuous Exposition Reflect and evaluate Phase 1 No

PR6008Q01 Mobile Phones Simple Multiple Choice Public Non-continuous Description Integrate and interpret Phase 1 No

PR6008Q02 Mobile Phones Constructed Response Manual Public Non-continuous Description


PR6008Q03 Mobile Phones Simple Multiple Choice Public Non-continuous Description Integrate and interpret Phase 1 Phase 1 & 2

PR6008Q04 Mobile Phones Constructed Response Expert Public Non-continuous Description Reflect and evaluate Phase 1 Phase 1 & 2

PR6008Q05 Mobile Phones Constructed Response Manual Public Non-continuous Description


PR6008Q06 Mobile Phones Constructed Response Expert Public Non-continuous Description Integrate and interpret Phase 1 Phase 1 & 2

78

Unit Item


Developed

for

Equating

Study

PR6009Q01 Tennis Grunts Simple Multiple Choice Personal Continuous Exposition Integrate and interpret Phase 1 no



PR6009Q04 Tennis Grunts Simple Multiple Choice Personal Continuous Exposition Access and retrieve Phase 1 no


PR6009Q06 Tennis Grunts Constructed Response Expert Personal Continuous Exposition Reflect and evaluate Phase 1 no



PR6010Q01 Exploration Today Simple Multiple Choice Educational Continuous Exposition Access and retrieve Phase 1 Phase 1 & 2

PR6010Q02 Exploration Today Simple Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 no

PR6010Q03 Exploration Today Constructed Response Expert Educational Continuous Exposition Integrate and interpret Phase 1 Phase 1 & 2

PR6010Q04 Exploration Today Simple Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 Phase 1 & 2

PR6010Q05 Exploration Today Constructed Response Expert Educational Continuous Exposition Integrate and interpret Phase 1 no

PR6010Q06 Exploration Today Simple Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 Phase 1 & 2

79

Unit Item


Developed

for

Equating

Study

PR6010Q07 Exploration Today Constructed Response Expert Educational Continuous Exposition Reflect and evaluate Phase 1 Phase 1 & 2

PR6010Q07X Exploration Today Simple Multiple Choice Educational Continuous Exposition Integrate and interpret Phase 1 no

PR6011Q01 Returning Simple Multiple Choice Personal Continuous Narration Access and retrieve Phase 1 Phase 1

PR6011Q02 Returning Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 1 No

PR6011Q03 Returning Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 1 Phase 1

PR6011Q04 Returning Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 1 No

PR6011Q05 Returning Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 1 Phase 1



PR6012Q01 On The Island Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 1 No


PR6012Q03 On The Island Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 1 No



80

Unit Item


Developed

for

Equating

Study





PR6012Q10 On The Island Simple Multiple Choice Personal Continuous Narration Reflect and evaluate Phase 1 No

PR6012Q11 On The Island Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 1 No

PR6013Q01 About Cats Simple Multiple Choice Personal Continuous Argumentation Integrate and interpret Phase 1 Phase 1 & 2

PR6013Q02 About Cats Simple Multiple Choice Personal Continuous Argumentation Integrate and interpret Phase 1 No

PR6013Q03 About Cats Constructed Response Expert Personal Continuous Argumentation Integrate and interpret Phase 1 Phase 1 & 2

PR6013Q04 About Cats Simple Multiple Choice Personal Continuous Argumentation Access and retrieve Phase 1 Phase 1 & 2

PR6013Q05 About Cats Constructed Response Expert Personal Continuous Argumentation Integrate and interpret Phase 1 Phase 1 & 2

PR6013Q06 About Cats Simple Multiple Choice Personal Continuous Argumentation Reflect and evaluate Phase 1 Phase 1 & 2

PR6013Q07 About Cats Constructed Response Expert Personal Continuous Argumentation Reflect and evaluate Phase 1 No

PR6013Q08 About Cats Constructed Response Expert Personal Continuous Argumentation Reflect and evaluate Phase 1 Phase 1

81

Unit Item


Developed

for

Equating

Study



PR6014Q01 The Job of an Air Traffic Controller Simple Multiple Choice Occupational Continuous Description





Reflect and evaluate Phase 1 Phase 1



PR6014Q05 The Job of an Air Traffic Controller Constructed Response Expert Occupational Continuous Description


PR6014Q06 The Job of an Air Traffic Controller Complex Multiple Choice Occupational Continuous Description

Access and retrieve Phase 1 Phase 1

PR6014Q06X The Job of an Air Traffic Controller Constructed Response Expert Occupational Continuous Description


PR6014Q07 The Job of an Air Traffic Controller Constructed Response Expert Occupational Continuous Description


PR6015Q01 Kenya Tourism and Conservation Simple Multiple Choice Educational Continuous Description


PR6015Q02 Kenya Tourism and Conservation Simple Multiple Choice Educational Non-continuous Description


PR6015Q03 Kenya Tourism and Conservation Constructed Response Expert Educational Continuous Description

Integrate and interpret Phase 1 Phase 1 & 2

PR6015Q04 Kenya Tourism and Conservation

Constructed Response Manual Educational Continuous Description


82

Unit Item


Developed

for

Equating

Study

PR6015Q05 Kenya Tourism and Conservation

Constructed Response Manual Educational Non-continuous Description





Reflect and evaluate Phase 1 Phase 1 & 2

PR6015Q08 Kenya Tourism and Conservation Simple Multiple Choice Educational Mixed Description


PR6017Q01 The Two Campaigns Constructed Response Expert Personal Continuous Narration Access and retrieve Phase 2 No

PR6017Q02 The Two Campaigns Constructed Response Expert Personal Continuous Narration Reflect and evaluate Phase 2 No

PR6017Q03 The Two Campaigns Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 No

PR6017Q04 The Two Campaigns Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 Phase 2

PR6017Q05 The Two Campaigns Complex Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 Phase 2

PR6017Q06 The Two Campaigns Constructed Response Expert Personal Continuous Narration Access and retrieve Phase 2 No


PR6017Q08 The Two Campaigns Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 2 Phase 2


PR6017Q10 The Two Campaigns Simple Multiple Choice Personal Continuous Narration Access and retrieve Phase 2 Phase 2

83

Unit Item


Developed

for

Equating

Study

PR6017Q11 The Two Campaigns Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 Phase 2


PR6018Q01 Aski Gym Youth Programme Complex Multiple Choice Personal Continuous Argumentation


PR6018Q02 Aski Gym Youth Programme Simple Multiple Choice Personal Continuous Argumentation


PR6018Q03 Aski Gym Youth Programme Constructed Response Expert Personal Continuous Argumentation


PR6018Q04 Aski Gym Youth Programme Complex Multiple Choice Personal Continuous Argumentation




PR6018Q06 Aski Gym Youth Programme Constructed Response Expert Personal Mixed Argumentation






PR6018Q09 Aski Gym Youth Programme Constructed Response Expert Personal Mixed Argumentation


PR6019Q01 Writing a Travel Article Simple Multiple Choice Educational Continuous Instruction






84

Unit Item


Developed

for

Equating

Study

PR6019Q04 Writing a Travel Article Complex Multiple Choice Educational Continuous Instruction










PR6019Q09 Writing a Travel Article Constructed Response Expert Educational Continuous Instruction


PR6020Q01 Well-being in the United States Simple Multiple Choice Public Non-continuous Exposition


PR6020Q02 Well-being in the United States Complex Multiple Choice Public Non-continuous Exposition


PR6020Q03 Well-being in the United States Constructed Response Expert Public Non-continuous Exposition












85

Unit Item


Developed

for

Equating

Study





PR6021Q01 Greek and Chinese Science Simple Multiple Choice Educational Continuous Argumentation




PR6021Q03 Greek and Chinese Science Constructed Response Expert Educational Continuous Argumentation










PR6021Q08 Greek and Chinese Science Constructed Response Expert Educational Continuous Argumentation






PR6022Q01 Lost at Sea Simple Multiple Choice Personal Continuous Narration Access and retrieve Phase 2 Phase 2

PR6022Q02 Lost at Sea Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 No

86

Unit Item


Developed

for

Equating

Study

PR6022Q03 Lost at Sea Simple Multiple Choice Personal Continuous Narration Access and retrieve Phase 2 Phase 2

PR6022Q04 Lost at Sea Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 2 No



PR6022Q07 Lost at Sea Constructed Response Expert Personal Continuous Narration Reflect and evaluate Phase 2 Phase 2

PR6022Q08 Lost at Sea Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 Phase 2

PR6022Q09 Lost at Sea Simple Multiple Choice Personal Continuous Narration Reflect and evaluate Phase 2 Phase 2

PR6022Q10 Lost at Sea Simple Multiple Choice Personal Continuous Narration Access and retrieve Phase 2 No

PR6023Q01 Trial by Teen Simple Multiple Choice Occupational Mixed Argumentation Reflect and evaluate Phase 2 Phase 2

PR6023Q02 Trial by Teen Constructed Response Expert Occupational Mixed Argumentation Access and retrieve Phase 2 No

PR6023Q03 Trial by Teen Constructed Response Expert Occupational Mixed Argumentation Integrate and interpret Phase 2 Phase 2

PR6023Q04 Trial by Teen Simple Multiple Choice Occupational Mixed Argumentation Integrate and interpret Phase 2 No

PR6023Q05 Trial by Teen Constructed Response Expert Occupational Mixed Argumentation Access and retrieve Phase 2 No

PR6023Q06 Trial by Teen Simple Multiple Choice Occupational Continuous Argumentation Integrate and interpret Phase 2 Phase 2

87

Unit Item


Developed

for

Equating

Study

PR6023Q07 Trial by Teen Simple Multiple Choice Occupational Mixed Argumentation Integrate and interpret Phase 2 Phase 2

PR6023Q08 Trial by Teen Constructed Response Expert Occupational Non-continuous Argumentation Reflect and evaluate Phase 2 Phase 2



PR6024Q01 The Sculpture Simple Multiple Choice Educational Continuous Description Access and retrieve Phase 2 Phase 2

PR6024Q02 The Sculpture Simple Multiple Choice Educational Continuous Description Integrate and interpret Phase 2 Phase 2

PR6024Q03 The Sculpture Simple Multiple Choice Educational Continuous Description Access and retrieve Phase 2 No

PR6024Q04 The Sculpture Constructed Response Expert Educational Continuous Description Reflect and evaluate Phase 2 Phase 2

PR6024Q05 The Sculpture Constructed Response Expert Educational Continuous Description Reflect and evaluate Phase 2 Phase 2

PR6024Q06 The Sculpture Constructed Response Expert Educational Continuous Description Integrate and interpret Phase 2 Phase 2

PR6024Q07 The Sculpture Simple Multiple Choice Educational Continuous Description Integrate and interpret Phase 2 No

PR6024Q08 The Sculpture Simple Multiple Choice Educational Continuous Description Integrate and interpret Phase 2 No

PR6024Q09 The Sculpture Constructed Response Expert Educational Continuous Description Reflect and evaluate Phase 2 No

PR6024Q10 The Sculpture Simple Multiple Choice Educational Continuous Description Integrate and interpret Phase 2 Phase 2

88

Unit Item


Developed

for

Equating

Study

PR6025Q01 Welcome to America Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 Phase 2

PR6025Q02 Welcome to America Simple Multiple Choice Personal Continuous Narration Reflect and evaluate Phase 2 No

PR6025Q03 Welcome to America Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 2 No

PR6025Q04 Welcome to America Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 No

PR6025Q05 Welcome to America Simple Multiple Choice Personal Continuous Narration Integrate and interpret Phase 2 Phase 2


PR6025Q07 Welcome to America Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 2 Phase 2

PR6025Q08 Welcome to America Constructed Response Expert Personal Continuous Narration Integrate and interpret Phase 2 Phase 2


PR6025Q10 Welcome to America Simple Multiple Choice Personal Continuous Narration Reflect and evaluate Phase 2 Phase 2


PR6025Q12 Welcome to America Constructed Response Expert Personal Continuous Narration Reflect and evaluate Phase 2 No

PR6026Q01 Bus Timetable Simple Multiple Choice Public Non-continuous Description Access and retrieve Phase 2 No

PR6026Q02 Bus Timetable Complex Multiple Choice Public Non-continuous Description Access and retrieve Phase 2 Phase 2

89

Unit Item


Developed

for

Equating

Study

PR6026Q03 Bus Timetable Constructed Response Expert Public Non-continuous Description Access and retrieve Phase 2 Phase 2

PR6026Q04 Bus Timetable Complex Multiple Choice Public Non-continuous Description Access and retrieve Phase 2 No

PR6026Q05 Bus Timetable Complex Multiple Choice Public Non-continuous Description Access and retrieve Phase 2 Phase 2

PR6026Q06 Bus Timetable Constructed Response Expert Public Non-continuous Description Access and retrieve Phase 2 No

PR6026Q07 Bus Timetable Simple Multiple Choice Public Non-continuous Description Access and retrieve Phase 2 No

PR6026Q08 Bus Timetable Constructed Response Expert Public Non-continuous Description Integrate and interpret Phase 2 No

PR6026Q09 Bus Timetable Constructed Response Expert Public Non-continuous Description Reflect and evaluate Phase 2 Phase 2

PR6026Q10 Bus Timetable Constructed Response Expert Public Non-continuous Description Integrate and interpret Phase 2 No

PR6027Q01 Underground Cooking Complex Multiple Choice Personal Non-continuous Instruction


PR6027Q02 Underground Cooking Simple Multiple Choice Personal Non-continuous Instruction


PR6027Q03 Underground Cooking Complex Multiple Choice Personal Non-continuous Instruction




PR6027Q05 Underground Cooking Constructed Response Expert Personal Non-continuous Instruction




90

Unit Item


Developed

for

Equating

Study









91

Science

Unit Item

Code Unit Name Item Format 2012 Application Area

Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7000Q01 Oily birds Simple Multiple Choice Environment Social

Explaining phenomena scientifically

Physical systems N.A. Phase 1

Phase 1 & 2

PS7000Q02 Oily birds Complex Multiple Choice Environment Social

Using scientific evidence N.A.

Scientific enquiry Phase 1

Phase 1 & 2

PS7000Q03 Oily birds Simple Multiple Choice

Frontiers of science and technology Social


Physical systems N.A. Phase 1 No

PS7000Q04 Oily birds Constructed Response Expert Environment Social



Phase 1 & 2

PS7000Q05 Oily birds Constructed Response Expert Environment Social

Identifying scientific issues N.A.

Scientific enquiry Phase 1 Phase 1

PS7000Q06 Oily birds Simple Multiple Choice Environment Social


Scientific explanations Phase 1 No

PS7001Q01 Washing clothes Constructed Response Expert



Technological systems N.A. Phase 1 No

PS7001Q02 Washing clothes Constructed Response Expert




PS7001Q03 Washing clothes Simple Multiple Choice

Frontiers of science and technology Global



PS7002Q01 Cooking Simple Multiple Choice Natural resources Personal



PS7002Q02 Cooking Complex Multiple Choice Natural resources Global


Physical systems N.A. Phase 1 Phase 1

PS7002Q03 Cooking Constructed Response Expert Environment Global


Earth and Space systems N.A. Phase 1

Phase 1 & 2

92

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7002Q04 Cooking Simple Multiple Choice Natural resources Social


Scientific explanations Phase 1

Phase 1 & 2

PS7002Q05 Cooking Simple Multiple Choice Natural resources Social



PS7004Q01 Mortality rates Constructed Response Manual Health Global


Scientific explanations Phase 1 Phase 1

PS7004Q02 Mortality rates Simple Multiple Choice Health Social


Living systems N.A. Phase 1 No

PS7004Q03 Mortality rates Constructed Response Expert Health Social



PS7005Q01 Lungs over gills Complex Multiple Choice




PS7005Q02 Lungs over gills Simple Multiple Choice

Frontiers of science and technology Personal



PS7006Q01 Osteoporosis Constructed Response Expert Health Personal



Phase 1 & 2

PS7006Q02 Osteoporosis Constructed Response Expert Health Personal



Phase 1 & 2

PS7006Q03 Osteoporosis Simple Multiple Choice Health Global



PS7007Q01 Extreme survival Simple Multiple Choice Health Social



PS7007Q02 Extreme survival Simple Multiple Choice Health Social



PS7007Q03 Extreme survival Constructed Response Expert Health Global



93

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7007Q04 Extreme survival Simple Multiple Choice Natural resources Global



PS7007Q05 Extreme survival Simple Multiple Choice Hazard Global



PS7008Q01 Reverse osmosis Simple Multiple Choice








PS7008Q03 Reverse osmosis Constructed Response Expert








PS7008Q05 Reverse osmosis Constructed Response Expert environment Global


Technological systems N.A. Phase 1 Phase 1

PS7009Q01 Fish and tides Simple Multiple Choice



Earth and Space systems N.A. Phase 1 Phase 1

PS7009Q02 Fish and tides Complex Multiple Choice




PS7009Q03 Fish and tides Constructed Response Expert Natural resources Social



PS7009Q04 Fish and tides Constructed Response Expert Natural resources Social



PS7010Q01 Elephant species Simple Multiple Choice Environment Global



94

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7010Q02 Elephant species Complex Multiple Choice Environment Global



PS7010Q03 Elephant species Constructed Response Expert Environment Social


Scientific enquiry Phase 1 No

PS7010Q04(a) Elephant species Simple Multiple Choice Environment Global


Living systems N.A. Phase 1 Phase 1

PS7010Q04(b) Elephant species Constructed Response Expert Environment Global



PS7010Q05(a) Elephant species Constructed Response Expert Environment Global



PS7010Q05(b) Elephant species Constructed Response Expert Environment Global



PS7012Q01 First aid Simple Multiple Choice Health Personal


Living systems N.A. Phase 1

Phase 1 & 2

PS7012Q02 First aid Complex Multiple Choice Health Personal



Phase 1 & 2

PS7012Q03 First Aid Constructed Response Expert Health Personal



Phase 1 & 2

PS7012Q04 First aid Complex Multiple Choice Health Personal



PS7013Q01 Ocean currents Simple Multiple Choice




Phase 1 & 2

PS7013Q02 Ocean currents Constructed Response Expert Hazard Social


Earth and Space systems N.A. Phase 1

Phase 1 & 2

PS7013Q03 Ocean currents Complex Multiple Choice




95

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7013Q04 Ocean currents Simple Multiple Choice



Earth and Space systems N.A. Phase 1 No

PS7014Q01 Oil spills Constructed Response Expert Hazard Social

Using scientific evidence

Technological systems N.A. Phase 1

Phase 1 & 2

PS7014Q02 Oil spills Complex Multiple Choice Hazard Global



PS7014Q03 Oil spills Constructed Response Expert Health Social



Phase 1 & 2

PS7014Q04 Oil spills Constructed Response Expert Environment Personal



Phase 1 & 2

PS7014Q05 Oil spills Simple Multiple Choice


Identifying scientific issues


PS7014Q06 Oil spills Simple Multiple Choice Hazard Social



PS7015Q01 Cassowaries Complex Multiple Choice Environment Social



PS7015Q02 Cassowaries Constructed Response Expert Environment Social



PS7015Q03 Cassowaries Complex Multiple Choice Environment Social



PS7015Q04 Cassowaries Simple Multiple Choice Environment Social



PS7016Q01(a) Oil Leak Simple Multiple Choice Environment Social



96

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7016Q01(b) Oil Leak Constructed Response Expert Environment Social



PS7016Q02 Oil Leak Simple Multiple Choice Environment Social



PS7016Q03 Oil Leak Simple Multiple Choice Environment Social



PS7016Q04(a) Oil Leak Simple Multiple Choice Environment Social



PS7016Q04(b) Oil Leak Constructed Response Expert Environment Social



PS7017Q01 Galileo's clock Simple Multiple Choice












Phase 1 & 2

PS7017Q04 Galileo's clock Constructed Response Expert




Phase 1 & 2

PS7017Q05 Galileo's clock Complex Multiple Choice




PS7018Q01 Watering cattle Simple Multiple Choice












Phase 1 & 2

97

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7018Q04 Watering cattle Complex Multiple Choice




PS7019Q01 Blood fats Simple Multiple Choice Health Social



PS7019Q02 Blood fats Simple Multiple Choice Health Social



PS7019Q03 Blood fats Complex Multiple Choice Health Social



PS7019Q04 Blood fats Simple Multiple Choice Health Personal



PS7020Q01 Stone pots Complex Multiple Choice Natural resources Social



PS7020Q02 Stone pots Simple Multiple Choice Natural resources Social



PS7020Q03 Stone pots Simple Multiple Choice Natural resources Social



Phase 1 & 2

PS7020Q04 Stone pots Constructed Response Expert Natural resources Social



PS7021Q01 Sterilised milk Simple Multiple Choice Health Social



PS7021Q02 Sterilised milk Complex Multiple Choice Health Social



Phase 1 & 2

PS7021Q03 Sterilised milk Simple Multiple Choice Health Personal



PS7021Q04 Sterilised milk Complex Multiple Choice

Environment Social



Phase 1 & 2

98

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7022Q01 Flood hazard Complex Multiple Choice Hazard Global



PS7022Q02 Flood hazard Constructed Response Expert Hazard Global



PS7022Q03 Flood hazard Constructed Response Expert Hazard Social



PS7023Q01 Cholera Complex Multiple Choice Health Social



Phase 1 & 2

PS7023Q02 Cholera Complex Multiple Choice Health Personal



PS7023Q03 Cholera Simple Multiple Choice Health Social



PS7201Q01 Water Purification


Hazard Social Using scientific evidence N.A.



Simple Multiple Choice

Health Social Identifying scientific issues N.A.




Hazard Social Using scientific evidence N.A.


PS7202Q01 Crystal radios

Complex Multiple Choice Hazard Personal













99

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study


Complex Multiple Choice










Constructed Response Expert Hazard Personal



PS7203Q01 Living on Mars Constructed Response Expert

Frontiers of science and technology

Global Using scientific evidence N.A.


PS7203Q02 Living on Mars Complex Multiple Choice




PS7203Q03 Living on Mars



Global Explaining phenomena scientifically


PS7203Q04 Living on Mars Constructed Response Expert Health Personal



PS7203Q05 Living on Mars Simple Multiple Choice




PS7203Q06 Living on Mars Simple Multiple Choice




PS7204Q01 Mobile phones Constructed Response Expert Hazard Personal



PS7204Q02 Mobile phones Simple Multiple Choice Health Social



PS7204Q03 Mobile phones Simple Multiple Choice Health Social



PS7204Q04 Mobile phones Constructed Response Expert




100

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7204Q05 Mobile phones Complex Multiple Choice Hazard Social



PS7205Q01 Plate Tectonics







Hazard Global Using scientific evidence


PS7205Q03 Plate Tectonics Simple Multiple Choice


Global Identifying scientific issues N.A.




Hazard Global Explaining phenomena scientifically






PS7205Q06 Plate Tectonics Complex Multiple Choice




PS7206Q01 Nuclear power






Complex Multiple Choice Natural resources Social




Simple Multiple Choice Hazard Social







101

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7207Q01 Alkaline water

Complex Multiple Choice Health Personal








Simple Multiple Choice Health Personal








Constructed Response Expert Health Personal



PS7208Q01 Identifying Substances



Personal Using scientific evidence





Personal Identifying scientific issues N.A.





Personal Using scientific evidence N.A.







PS7209Q01 Floating Simple Multiple Choice


Social Explaining phenomena scientifically


PS7209Q02 Floating Constructed Response Manual




PS7209Q03 Floating Constructed Response Expert


Personal Explaining phenomena scientifically


PS7209Q04 Floating Constructed Response Expert




102

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7210Q01 MP3 Constructed Response Expert




PS7210Q02 MP3 Simple Multiple Choice




















PS7211Q01 Ceramics


Natural resources Global Explaining phenomena scientifically


PS7211Q02 Ceramics



Social Using scientific evidence N.A.


PS7211Q03 Ceramics





PS7211Q04 Ceramics



Social Identifying scientific issues N.A.


PS7212Q01 Nitrogen in Rocks




PS7212Q02

Nitrogen in Rocks





Simple Multiple Choice Natural resources Global



103

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study



Natural resources Global Identifying scientific issues N.A.


PS7213Q01 Fuel cells











Complex Multiple Choice Environment Global













PS7214Q01 Malaria

Simple Multiple Choice Health Social



PS7214Q02 Malaria

Complex Multiple Choice Health Social



PS7214Q03 Malaria

Constructed Response Expert Environment Global



PS7214Q04 Malaria

Constructed Response Expert Health Social



PS7215Q01 Blood Clotting


Health Personal Using scientific evidence N.A.




Health Personal Explaining phenomena scientifically




Health Social Explaining phenomena scientifically


104

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7216Q01 Fish Management










Natural resources Global Using scientific evidence N.A.


PS7217Q01 Ocean Debris


Environment Social Explaining phenomena scientifically




Environment Global Explaining phenomena scientifically










PS7218Q01 Underground Cooling


Hazard Social Explaining phenomena scientifically
















PS7219Q01 Hydroponics


Natural resources Social Explaining phenomena scientifically


105

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study



Natural resources Social Explaining phenomena scientifically




Natural resources Social Identifying scientific issues N.A.




Environment Social Explaining phenomena scientifically


PS7220Q01 Natural Gas






Environment Social Identifying scientific issues N.A.




Natural resources Social Identifying scientific issues N.A.


PS7221Q01 Clean Drinking Water















Health Social Explaining phenomena scientifically



Complex Multiple Choice Health Social

Identifying scientific issues


PS7222Q01 Texan Summer Simple Multiple Choice

Environment Social Identifying scientific issues N.A.


PS7222Q02 Texan Summer Complex Multiple Choice

Environment Global Using scientific evidence N.A.


106

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7222Q03 Texan Summer Complex Multiple Choice

Environment Global Using scientific evidence N.A.


PS7222Q04 Texan Summer




PS7223Q01 Car Tyres Simple Multiple Choice




PS7223Q02 Car Tyres Constructed Response Expert




PS7223Q03 Car Tyres Constructed Response Expert




PS7224Q01 Moon Rock Simple Multiple Choice




PS7224Q02 Moon Rock Complex Multiple Choice




PS7224Q03 Moon Rock




PS7224Q04 Moon Rock





PS7224Q05 Moon Rock Simple Multiple Choice



PS7225Q01 BRICKS Constructed Response Expert Natural resources Social



PS7225Q02 BRICKS Complex Multiple Choice Natural resources Social



107

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study

PS7225Q03 BRICKS Simple Multiple Choice




PS7228Q01 C-Dating Simple Multiple Choice


Social Identifying scientific issues N.A.


PS7228Q02 C-Dating Constructed Response Expert










Personal Using scientific evidence N.A.










PS7229Q01 Exploring the Ocean













PS7230Q01 Pesticides Complex Multiple

Choice Hazard Social Using scientific evidence N.A.



Choice Hazard Social



108

Unit Item


Context

setting

area Competency

Knowledge

Of Science

Knowledge

About

Science

Developed

for

Equating

Study


Choice Hazard Social




Choice Natural resources Social




Choice Natural resources Social Explaining phenomena scientifically


PS7231Q01 Pollen Allergy Complex Multiple Choice

Health Social Using scientific evidence N.A.


PS7231Q02 Pollen Allergy Simple Multiple Choice Health Social



PS7231Q03 Pollen Allergy Simple Multiple Choice Health Social



PS7231Q04 Pollen Allergy Complex Multiple Choice

Health Social Identifying scientific issues N.A.


PS7232Q01 Water-logged soil


Natural resources Personal Identifying scientific issues N.A.




Natural resources Social





Natural resources Social Explaining phenomena scientifically N.A.


PS7233Q01 Sunlight Simple Multiple Choice




PS7233Q02 Sunlight Complex Multiple Choice Hazard Social



PS7233Q03 Sunlight





110

Appendix 2: PBT Critical friends

Name Domain Role in PISA

Rodger Bybee Science Chair Science Expert Group, PISA 2006-9; Critical Friend for science, 2012

John Dossey Mathematics Mathematics Expert Group member, PISA 2000-2012, Problem Solving Expert Group member, PISA 2003-2012

Irwin Kirsch Reading Chair Reading Expert Group, PISA 2000-2009; Educational Testing Service (ETS) – PISA Consortium member 2000-2006, Critical Friend for reading, 2012

Stan Jones Reading Reading Expert Group member, PISA 2000; Financial Literacy Expert Group member, PISA 2012

Kees Lagerwaard Mathematics Mathematics test developer, PISA 2000-2012, CitoGroup – PISA Consortium member 2000-2006)

Rolf Olsen Science Science test development coordinator, Institutt for Lærerutdanning og Skoleutvikling (ILS) – PISA Consortium member 2000-2012, mathematics test developer, PISA 2012

Jean-Francois Rouet Reading Reading Expert Group member, PISA 2009; Critical Friend for reading, 2012.

111

Appendix 3: Item Characteristics

Mathematics

Item 17

item:17 (PM5002Q01 )

Cases for this item 843 Item-Rest Cor. 0.46 Item-Total Cor. 0.56

Item Threshold(s): -1.17 Weighted MNSQ 0.92

Item Delta(s): -1.17

Label Score Count % of tot Pt Bis t sig. p PV1Avg:1 PV1SD:1

1 0 220 26.1 -0.39 -12.37 0.000 -1.240 1.070

2 0 38 4.51 -0.11 -3.22 0.001 -1.080 0.820

3 1 549 65.12 0.46 14.98 0.000 0.150 1.090

4 0 32 3.8 -0.1 -2.85 0.004 -0.970 1.010

8 0 1 0.12 0.01 0.17 0.863 -0.820 0.000

9 0 1 0.12 -0.06 -1.76 0.079 -2.430 0.000

r 0 2 0.24 -0.06 -1.78 0.075 -1.460 0.780

Item 18

item:18 (PM5002Q02 )





1 0 84 9.96 -0.07 -1.96 0.050 -0.600 1.270

2 0 164 19.45 -0.06 -1.75 0.080 -0.560 1.020

3 1 371 44.01 0.4 12.65 0.000 0.310 1.140

4 0 175 20.76 -0.28 -8.5 0.000 -1.040 1.040

5 0 32 3.8 -0.15 -4.36 0.000 -1.100 0.890

8 0 2 0.24 -0.04 -1.02 0.308 -1.810 1.150

9 0 13 1.54 -0.06 -1.69 0.092 -1.000 1.330

r 0 2 0.24 -0.06 -1.86 0.063 -1.460 0.780

112

Item 19

item:19 (PM5008Q01 )


Item Threshold(s): 0.91 Weighted MNSQ 0.89

Item Delta(s): 0.91


0 0 562 66.43 -0.4 -12.5 0.000 -0.630 1.100

1 1 244 28.84 0.51 17.4 0.000 0.820 1.030

9 0 38 4.49 -0.21 -6.12 0.000 -1.360 1.200

r 0 2 0.24 -0.07 -2.01 0.045 -1.460 0.780

Item 20

item:20 (PM5010Q01T )


Item Threshold(s): -2.14 1.58 Weighted MNSQ 1.09

Item Delta(s): -2.11 1.56


0 0 19 2.29 -0.11 -3.22 0.001 -1.380 0.970

1 0 118 14.23 -0.19 -5.57 0.000 -1.210 1.150

2 1 530 63.93 -0.04 -1.26 0.210 -0.370 1.060

3 2 140 16.89 0.34 10.43 0.000 0.710 1.120

9 0 17 2.05 -0.13 -3.67 0.000 -1.650 1.260

r 0 5 0.6 -0.07 -2.11 0.035 -2.020 1.050

Item 21

item:21 (PM5010Q02 )



Item Delta(s): 0.58


0 0 495 59.71 -0.23 -6.71 0.000 -0.640 1.110

1 1 270 32.57 0.36 11.11 0.000 0.380 1.140

9 0 56 6.76 -0.19 -5.63 0.000 -1.280 1.060

r 0 8 0.97 -0.09 -2.7 0.007 -1.710 1.400

113

Item 22

item:22 (PM5013Q01 )





1 0 216 25.71 -0.22 -6.64 0.000 -0.930 1.050

2 1 475 56.55 0.41 13.08 0.000 0.120 1.140

3 0 64 7.62 -0.2 -6.02 0.000 -1.390 0.940

4 0 66 7.86 -0.12 -3.51 0.000 -0.980 1.060

8 0 1 0.12 -0.03 -0.96 0.338 -1.090 0.000

9 0 15 1.79 -0.1 -2.94 0.003 -1.220 0.680

r 0 3 0.36 -0.09 -2.68 0.008 -1.380 0.940

Item 23

item:23 (PM5013Q03 )



Item Delta(s): 0.51


0 0 484 57.62 -0.37 -11.46 0.000 -0.790 1.060

1 1 282 33.57 0.52 17.5 0.000 0.520 1.070

9 0 71 8.45 -0.2 -6.02 0.000 -1.120 0.960

r 0 3 0.36 -0.1 -2.8 0.005 -1.380 0.940

Item 24

item:24 (PM5015Q02 )



Item Delta(s): 0.30


1 0 95 11.36 -0.11 -3.32 0.001 -0.800 1.040

2 0 111 13.28 -0.08 -2.21 0.027 -0.550 1.180

3 0 268 32.06 -0.18 -5.43 0.000 -0.790 1.050

4 1 312 37.32 0.34 10.53 0.000 0.310 1.110

8 0 2 0.24 -0.02 -0.69 0.492 -0.810 0.290

9 0 33 3.95 -0.05 -1.31 0.191 -0.800 1.100

r 0 15 1.79 -0.05 -1.58 0.114 -1.180 0.980

114

Item 25

item:25 (PM5017Q02 )





1 0 23 2.69 -0.07 -2.06 0.040 -1.030 1.010

2 1 476 55.67 0.28 8.65 0.000 0.120 1.180

3 0 223 26.08 -0.09 -2.56 0.011 -0.590 1.200

4 0 120 14.04 -0.21 -6.35 0.000 -1.120 1.170

8 0 2 0.23 -0.02 -0.69 0.490 -1.400 0.560

9 0 5 0.58 -0.06 -1.64 0.102 -1.590 1.360

r 0 6 0.7 -0.15 -4.35 0.000 -2.080 0.840

Item 26

item:26 (PM5017Q03 )


Item Threshold(s): -1.95 -1.40 Weighted MNSQ 1.07

Item Delta(s): -1.10 -2.25


0 0 87 10.16 -0.34 -10.45 0.000 -1.790 0.800

1 1 100 11.68 -0.07 -1.97 0.049 -0.590 1.110

2 2 625 73.01 0.39 12.48 0.000 0.070 1.130

9 0 38 4.44 -0.19 -5.75 0.000 -1.820 0.940

r 0 6 0.7 -0.13 -3.93 0.000 -2.080 0.840

Item 27

item:27 (PM5020Q02 )



Item Delta(s): 1.33


0 0 460 55.22 -0.12 -3.55 0.000 -0.570 1.090

1 1 175 21.01 0.48 15.71 0.000 0.820 0.950

9 0 189 22.69 -0.29 -8.87 0.000 -1.020 1.030

r 0 9 1.08 -0.11 -3.08 0.002 -1.410 1.020

115

Item 28

item:28 (PM5021Q01T )





0 0 8 0.94 -0.07 -2.01 0.044 -1.100 1.300

1 0 13 1.52 -0.12 -3.54 0.000 -1.720 0.780

2 0 75 8.78 -0.23 -7.01 0.000 -1.380 1.150

3 0 83 9.72 -0.21 -6.15 0.000 -1.200 0.990

4 1 668 78.22 0.38 12.16 0.000 -0.010 1.190

9 0 3 0.35 -0.04 -1.09 0.278 -1.140 0.610

r 0 4 0.47 -0.12 -3.47 0.001 -2.160 0.790

Item 29

item:29 (PM5021Q02 )





0 0 412 48.13 -0.37 -11.82 0.000 -0.860 1.120

1 1 437 51.05 0.4 12.78 0.000 0.260 1.170

9 0 3 0.35 -0.08 -2.27 0.023 -1.970 1.690

r 0 4 0.47 -0.12 -3.64 0.000 -2.160 0.790

Item 30

item:30 (PM5026Q01 )



Item Delta(s): 2.37


0 0 688 82.3 -0.23 -6.76 0.000 -0.490 1.120

1 1 84 10.05 0.37 11.47 0.000 1.100 0.860

9 0 58 6.94 -0.07 -2.08 0.037 -0.760 1.220

r 0 6 0.72 -0.07 -1.98 0.048 -1.110 1.090

116

Item 31

item:31 (PM5101Q01 )



Item Delta(s): 0.61


1 0 51 12.14 -0.14 -2.95 0.003 -0.710 1.190

2 0 97 23.1 -0.1 -2 0.046 -0.460 1.110

3 0 99 23.57 -0.09 -1.94 0.053 -0.570 1.300

4 1 147 35 0.32 6.98 0.000 0.470 1.100

8 0 1 0.24 0.03 0.69 0.492 0.390 0.000

9 0 15 3.57 -0.07 -1.48 0.139 -0.780 1.260

r 0 10 2.38 -0.1 -1.95 0.051 -1.210 1.410

Item 32

item:32 (PM5101Q02 )



Item Delta(s): 0.98


0 0 257 61.34 -0.24 -5.02 0.000 -0.540 1.190

1 1 119 28.4 0.4 8.8 0.000 0.740 0.930

9 0 28 6.68 -0.19 -3.91 0.000 -1.040 1.080

r 0 15 3.58 -0.08 -1.67 0.095 -0.890 1.380

Item 33

item:33 (PM5102Q01 )



Item Delta(s): 2.74


1 0 194 45.86 -0.08 -1.65 0.099 -0.250 1.060

2 0 77 18.2 -0.06 -1.29 0.197 -0.330 1.210

3 0 105 24.82 0.05 1.02 0.307 -0.080 1.290

4 1 37 8.75 0.18 3.84 0.000 0.790 1.080

9 0 7 1.65 0.01 0.1 0.918 -0.090 1.550

r 0 3 0.71 -0.12 -2.4 0.017 -2.640 0.460

117

Item 34

item:34 (PM5104Q01 )





1 1 343 77.95 0.36 8.05 0.000 0.110 1.180

2 0 63 14.32 -0.28 -6.13 0.000 -1.240 1.040

3 0 16 3.64 -0.08 -1.62 0.106 -0.890 0.970

4 0 17 3.86 -0.17 -3.7 0.000 -1.410 1.140

9 0 1 0.23 -0.05 -1.04 0.297 -1.040 0.000

Item 35

item:35 (PM5107Q01 )



Item Delta(s): 0.39


0 0 246 57.61 -0.37 -8.17 0.000 -0.660 1.120

1 1 167 39.11 0.45 10.37 0.000 0.580 1.100

9 0 11 2.58 -0.16 -3.26 0.001 -1.180 0.920

r 0 3 0.7 -0.15 -3.12 0.002 -2.290 0.380

Item 36

item:36 (PM5107Q02 )


Item Threshold(s): 1.17 1.67 Weighted MNSQ 0.95

Item Delta(s): 2.10 0.75


0 0 260 60.89 -0.16 -3.43 0.001 -0.450 1.110

1 1 42 9.84 0.25 5.28 0.000 0.790 0.870

2 2 52 12.18 0.39 8.82 0.000 1.380 0.970

9 0 70 16.39 -0.3 -6.38 0.000 -0.970 0.900

r 0 3 0.7 -0.16 -3.24 0.001 -2.290 0.380

118

Item 37

item:37 (PM5109Q01 )



Item Delta(s): 0.48


0 0 252 57.27 -0.38 -8.47 0.000 -0.650 1.120

1 1 166 37.73 0.43 10 0.000 0.620 1.130

9 0 22 5 -0.11 -2.27 0.024 -0.890 0.770

Item 38

item:38 (PM5109Q02 )



Item Delta(s): 3.87 -0.19


0 0 261 59.32 -0.12 -2.58 0.010 -0.340 1.140

1 1 10 2.27 0.13 2.66 0.008 1.160 0.890

2 2 40 9.09 0.34 7.49 0.000 1.500 1.020

9 0 127 28.86 -0.11 -2.28 0.023 -0.460 1.180

r 0 2 0.45 -0.1 -2.03 0.043 -1.690 0.380

Item 39

item:39 (PM5110Q01 )





0 0 207 47.7 -0.34 -7.44 0.000 -0.700 1.090

1 1 221 50.92 0.37 8.23 0.000 0.450 1.140

9 0 3 0.69 -0.09 -1.84 0.066 -1.740 1.690

r 0 3 0.69 -0.1 -2.14 0.033 -1.500 0.550

119

Item 40

item:40 (PM5110Q02 )





0 0 284 65.44 -0.29 -6.39 0.000 -0.470 1.120

1 1 49 11.29 0.18 3.78 0.000 0.620 0.810

2 2 54 12.44 0.42 9.59 0.000 1.400 0.910

9 0 44 10.14 -0.15 -3.2 0.001 -0.560 1.200

r 0 3 0.69 -0.11 -2.32 0.021 -1.500 0.550

Item 41

item:41 (PM5124Q01 )





0 0 146 33.18 -0.45 -10.44 0.000 -1.020 1.060

1 1 286 65 0.48 11.4 0.000 0.290 1.120

9 0 5 1.14 -0.07 -1.57 0.117 -1.160 0.970

r 0 3 0.68 -0.12 -2.6 0.010 -2.230 0.980

Item 42

item:42 (PM5124Q02 )





0 0 182 41.36 -0.4 -9.07 0.000 -0.780 1.020

1 1 225 51.14 0.54 13.27 0.000 0.480 1.100

9 0 30 6.82 -0.24 -5.25 0.000 -1.310 1.080

r 0 3 0.68 -0.13 -2.67 0.008 -2.230 0.980

120

Item 43

item:43 (PM5124Q03 )



Item Delta(s): 0.12


0 0 217 49.32 -0.29 -6.4 0.000 -0.620 1.070

1 1 193 43.86 0.41 9.46 0.000 0.490 1.160

9 0 21 4.77 -0.17 -3.55 0.000 -1.070 1.260

r 0 9 2.05 -0.16 -3.39 0.001 -1.710 1.150

Item 44

item:44 (PM5142Q01 )





0 0 23 5.44 -0.24 -4.98 0.000 -1.430 0.890

1 1 396 93.62 0.26 5.5 0.000 -0.060 1.170

9 0 3 0.71 -0.08 -1.7 0.090 -1.170 1.440

r 0 1 0.24 -0.06 -1.2 0.232 -3.010 0.000

Item 45

item:45 (PM5142Q02 )





0 0 117 27.66 -0.36 -7.93 0.000 -1.040 1.150

1 1 302 71.39 0.38 8.45 0.000 0.220 1.010

9 0 3 0.71 -0.09 -1.92 0.056 -1.170 1.440

r 0 1 0.24 -0.06 -1.34 0.182 -3.010 0.000

121

Item 46

item:46 (PM5145Q01 )



Item Delta(s): 1.45


0 0 266 62.3 -0.24 -4.99 0.000 -0.480 1.130

1 1 93 21.78 0.46 10.74 0.000 1.020 1.040

9 0 65 15.22 -0.18 -3.74 0.000 -0.700 1.050

r 0 3 0.7 -0.15 -3.17 0.002 -2.290 0.380

Item 47

item:47 (PM5163Q01 )





0 0 162 36.82 -0.49 -11.79 0.000 -1.030 1.060

1 1 265 60.23 0.53 13.14 0.000 0.380 1.090

9 0 13 2.95 -0.14 -2.94 0.003 -1.120 0.560

Item 48

item:48 (PM5169Q01 )





0 0 108 25.71 -0.44 -10.08 0.000 -1.230 1.100

1 1 290 69.05 0.54 12.95 0.000 0.260 1.060

9 0 21 5 -0.22 -4.72 0.000 -1.500 0.900

r 0 1 0.24 -0.1 -2.16 0.032 -2.450 0.000

Item 49

item:49 (PM5169Q02 )





0 0 175 41.67 -0.39 -8.54 0.000 -0.870 1.070

122

1 1 206 49.05 0.55 13.53 0.000 0.530 1.030

9 0 38 9.05 -0.28 -5.98 0.000 -1.240 0.990

r 0 1 0.24 -0.11 -2.24 0.026 -2.450 0.000

Item 50

item:50 (PM5181Q02 )



Item Delta(s): 0.85


0 0 202 46.54 -0.27 -5.83 0.000 -0.570 1.060

1 1 137 31.57 0.57 14.23 0.000 0.990 0.930

9 0 93 21.43 -0.29 -6.39 0.000 -0.780 1.010

r 0 2 0.46 -0.11 -2.26 0.024 -1.460 0.780

Item 51

item:51 (PM5183Q01 )



Item Delta(s): -0.91 -1.55


0 0 49 11.48 -0.33 -7.1 0.000 -1.640 1.080

1 1 69 16.16 -0.2 -4.2 0.000 -0.760 1.000

2 2 274 64.17 0.5 12.03 0.000 0.330 1.030

9 0 32 7.49 -0.2 -4.24 0.000 -1.200 1.060

r 0 3 0.7 -0.14 -2.83 0.005 -2.290 0.380

Item 52

item:52 (PM5183Q02 )





1 0 80 18.74 -0.15 -3.18 0.002 -0.630 0.970

2 0 32 7.49 -0.13 -2.69 0.007 -0.920 1.040

3 0 81 18.97 -0.21 -4.37 0.000 -0.790 1.180

4 1 218 51.05 0.42 9.62 0.000 0.370 1.170

8 0 1 0.23 0.07 1.5 0.134 1.680 0.000

9 0 9 2.11 -0.15 -3.16 0.002 -1.410 1.210

123

r 0 6 1.41 -0.15 -3.2 0.001 -1.950 0.570

Item 53

item:53 (PM5185Q01 )





1 0 8 1.87 -0.12 -2.4 0.017 -1.600 1.470

2 0 5 1.17 -0.13 -2.77 0.006 -1.650 1.050

3 0 4 0.94 -0.04 -0.81 0.416 -0.600 1.730

4 1 409 95.78 0.19 4.04 0.000 -0.150 1.250

r 0 1 0.23 -0.1 -2.09 0.037 -2.450 0.000

Item 54

item:54 (PM5185Q02T )





0 0 2 0.47 -0.09 -1.77 0.077 -0.820 0.060

1 0 20 4.68 -0.16 -3.28 0.001 -1.230 1.110

2 0 106 24.82 -0.27 -5.84 0.000 -0.930 1.210

3 1 298 69.79 0.35 7.77 0.000 0.140 1.160

r 0 1 0.23 -0.11 -2.22 0.027 -2.450 0.000

Item 55

item:55 (PM5187Q01 )



Item Delta(s): 1.55


1 0 107 25.3 -0.03 -0.64 0.525 -0.280 1.180

2 1 88 20.8 0.16 3.4 0.001 0.470 1.050

3 0 148 34.99 0.05 1.05 0.293 -0.080 1.130

4 0 73 17.26 -0.14 -2.97 0.003 -0.650 1.180

9 0 3 0.71 -0.12 -2.55 0.011 -1.890 0.770

r 0 4 0.95 -0.13 -2.76 0.006 -2.200 0.940

124

Item 56

item:56 (PM5188Q01 )





0 0 111 26.24 -0.28 -6.08 0.000 -0.860 1.130

1 1 303 71.63 0.34 7.35 0.000 0.170 1.080

9 0 2 0.47 -0.05 -1.09 0.278 -1.420 0.530

r 0 7 1.65 -0.18 -3.82 0.000 -2.140 0.700

125

Reading

Item 29

item:29 (PR6002Q01 )



Item Delta(s): 0.84


1 0 136 35.32 -0.1 -2.04 0.042 0.060 1.080

2 1 160 41.56 0.24 4.93 0.000 0.860 1.060

3 0 41 10.65 -0.11 -2.2 0.029 0.030 1.020

4 0 40 10.39 -0.03 -0.67 0.503 0.150 1.290

9 0 8 2.08 -0.18 -3.62 0.000 -1.160 1.410

Item 30

item:30 (PR6002Q02T )



Item Delta(s): 1.49


0 0 5 1.3 -0.08 -1.62 0.106 -0.560 0.710

1 0 42 10.91 -0.28 -5.65 0.000 -0.470 1.090

2 0 65 16.88 -0.12 -2.39 0.017 0.070 1.190

3 0 156 40.52 0.05 0.97 0.331 0.340 1.130

4 1 114 29.61 0.27 5.54 0.000 0.970 0.990

9 0 3 0.78 -0.08 -1.63 0.103 -0.450 0.450

Item 31

item:31 (PR6002Q04 )





0 0 92 24.02 -0.31 -6.37 0.000 -0.220 1.050

1 1 241 62.92 0.57 13.57 0.000 0.850 0.930

9 0 50 13.05 -0.43 -9.17 0.000 -0.840 1.130

126

Item 32

item:32 (PR6002Q05 )



Item Delta(s): 0.23


1 0 43 11.2 -0.33 -6.74 0.000 -0.690 0.950

2 0 80 20.83 -0.03 -0.49 0.623 0.210 1.020

3 1 205 53.39 0.38 8.03 0.000 0.810 1.080

4 0 40 10.42 -0.15 -3.03 0.003 0.000 1.050

9 0 16 4.17 -0.15 -2.95 0.003 -0.540 1.230

Item 33

item:33 (PR6004Q01 )





0 0 135 17.62 -0.34 -9.94 0.000 -0.570 1.070

1 1 591 77.15 0.43 13.07 0.000 0.670 1.040

9 0 40 5.22 -0.23 -6.45 0.000 -0.850 1.100

Item 34

item:34 (PR6004Q05B )





0 0 109 14.44 -0.26 -7.32 0.000 -0.510 1.080

1 1 619 81.99 0.34 9.91 0.000 0.580 1.090

9 0 27 3.58 -0.21 -6.03 0.000 -0.760 1.320

127

Item 35

item:35 (PR6004Q06 )





1 0 9 1.17 -0.12 -3.27 0.001 -0.760 1.120

2 0 28 3.66 -0.27 -7.66 0.000 -1.340 0.860

3 0 30 3.92 -0.22 -6.26 0.000 -1.050 0.950

4 1 693 90.47 0.38 11.28 0.000 0.530 1.100

9 0 6 0.78 -0.06 -1.67 0.095 -0.490 0.590

Item 36

item:36 (PR6008Q02 )



Item Delta(s): 0.84


0 0 409 54.1 -0.26 -7.5 0.000 0.090 0.980

1 1 314 41.53 0.37 10.81 0.000 1.040 0.970

9 0 33 4.37 -0.24 -6.81 0.000 -0.620 1.230

Item 37

item:37 (PR6008Q04 )





0 0 75 9.95 -0.28 -7.91 0.000 -0.490 0.960

1 1 630 83.55 0.42 12.68 0.000 0.670 1.000

9 0 49 6.5 -0.29 -8.46 0.000 -0.830 1.060

128

Item 38

item:38 (PR6008Q05 )



Item Delta(s): 0.06


0 0 264 34.97 -0.2 -5.53 0.000 0.040 1.020

1 1 432 57.22 0.36 10.43 0.000 0.860 0.970

9 0 59 7.81 -0.3 -8.75 0.000 -0.630 1.120

Item 39

item:39 (PR6010Q01 )



Item Delta(s): 0.22


1 0 119 15.54 -0.22 -6.35 0.000 -0.240 0.970

2 1 419 54.7 0.33 9.74 0.000 0.900 1.060

3 0 135 17.62 -0.03 -0.79 0.431 0.170 1.060

4 0 80 10.44 -0.21 -5.85 0.000 -0.380 1.050

9 0 13 1.7 -0.08 -2.17 0.030 -0.360 1.370

Item 40

item:40 (PR6010Q03 )



Item Delta(s): 2.78


0 0 624 82.21 -0.19 -5.28 0.000 0.320 1.130

1 1 99 13.04 0.31 9.03 0.000 1.450 0.960

9 0 36 4.74 -0.16 -4.32 0.000 -0.320 1.010

129

Item 41

item:41 (PR6010Q06 )



Item Delta(s): 0.49


1 0 36 4.73 -0.19 -5.27 0.000 -0.630 1.190

2 0 195 25.62 -0 -0.01 0.991 0.330 1.020

3 1 374 49.15 0.36 10.68 0.000 0.940 1.080

4 0 143 18.79 -0.31 -9.04 0.000 -0.350 0.860

8 0 2 0.26 -0.04 -1.19 0.236 -0.550 1.600

9 0 11 1.45 -0.14 -3.89 0.000 -0.780 1.330

Item 42

item:42 (PR6010Q07 )



Item Delta(s): 1.50


0 0 462 61.19 -0.38 -11.11 0.000 0.060 1.060

1 1 228 30.2 0.49 15.38 0.000 1.310 0.900

9 0 65 8.61 -0.15 -4.11 0.000 -0.100 1.050

Item 43

item:43 (PR6011Q03 )



Item Delta(s): 0.23


1 0 94 25.27 -0.09 -1.73 0.085 0.120 0.990

2 0 49 13.17 -0.2 -3.89 0.000 -0.220 1.000

3 1 196 52.69 0.33 6.75 0.000 0.800 1.130

4 0 26 6.99 -0.19 -3.64 0.000 -0.460 1.090

9 0 7 1.88 -0.09 -1.71 0.088 -0.560 1.150

130

Item 44

item:44 (PR6011Q05 )



Item Delta(s): 2.30


0 0 276 75.82 -0.09 -1.76 0.080 0.260 1.130

1 1 62 17.03 0.24 4.79 0.000 1.120 1.090

9 0 26 7.14 -0.2 -3.95 0.000 -0.380 0.930

Item 45

item:45 (PR6011Q07 )





0 0 73 20.11 -0.36 -7.37 0.000 -0.510 0.930

1 1 274 75.48 0.44 9.32 0.000 0.680 1.070

9 0 16 4.41 -0.22 -4.21 0.000 -0.970 0.780

Item 46

item:46 (PR6013Q03 )



Item Delta(s): 1.13


0 0 384 51.89 -0.28 -7.94 0.000 -0.030 1.060

1 1 264 35.68 0.46 14.04 0.000 1.210 0.970

9 0 92 12.43 -0.24 -6.77 0.000 -0.380 0.970

Item 47

item:47 (PR6013Q04 )



Item Delta(s): 0.93


1 0 78 10.51 -0.14 -3.78 0.000 -0.130 1.290

2 0 217 29.25 -0.2 -5.5 0.000 -0.080 1.050

131

3 1 294 39.62 0.36 10.47 0.000 0.960 1.110

4 0 144 19.41 -0.09 -2.54 0.011 0.180 1.010

8 0 3 0.4 0.01 0.14 0.889 -0.400 0.690

9 0 6 0.81 -0.08 -2.12 0.034 -0.330 1.930

Item 48

item:48 (PR6013Q06 )





1 1 433 58.91 0.45 13.52 0.000 0.880 1.050

2 0 109 14.83 -0.11 -3.01 0.003 -0.110 1.050

3 0 112 15.24 -0.33 -9.5 0.000 -0.550 1.010

4 0 67 9.12 -0.18 -5.09 0.000 -0.320 0.840

8 0 4 0.54 0.01 0.37 0.710 0.270 1.370

9 0 10 1.36 -0.08 -2.21 0.028 -0.460 1.400

Item 49

item:49 (PR6013Q08 )



Item Delta(s): 0.63


0 0 157 44.48 -0.17 -3.19 0.002 0.080 1.090

1 1 160 45.33 0.33 6.59 0.000 0.950 1.040

9 0 36 10.2 -0.27 -5.26 0.000 -0.590 0.780

Item 50

item:50 (PR6014Q03 )





1 1 280 74.07 0.16 3.22 0.001 0.580 1.050

2 0 11 2.91 -0.15 -3.04 0.003 -0.550 1.280

3 0 56 14.81 0.01 0.2 0.840 0.200 1.250

4 0 27 7.14 -0.15 -2.91 0.004 -0.380 1.190

132

8 0 2 0.53 -0.04 -0.79 0.428 -0.880 0.810

9 0 2 0.53 -0.11 -2.24 0.025 -1.800 1.490

Item 51

item:51 (PR6014Q06T )





0 0 8 2.12 -0.2 -3.95 0.000 -1.260 1.100

1 0 14 3.7 -0.24 -4.84 0.000 -1.110 0.970

2 0 57 15.08 -0.33 -6.8 0.000 -0.460 1.040

3 0 86 22.75 -0.05 -0.96 0.337 0.250 0.990

4 1 141 37.3 0.2 3.86 0.000 0.640 0.890

5 2 71 18.78 0.31 6.39 0.000 1.320 0.920

9 0 1 0.26 -0.06 -1.15 0.250 -1.450 0.000

Item 52

item:52 (PR6014Q07 )



Item Delta(s): 0.26


0 0 159 42.4 -0.37 -7.67 0.000 -0.080 1.050

1 1 200 53.33 0.46 10.12 0.000 0.910 0.960

9 0 16 4.27 -0.24 -4.84 0.000 -1.080 1.120

Item 53

item:53 (PR6015Q01 )





1 0 10 2.65 -0.16 -3.08 0.002 -0.610 0.840

2 0 20 5.29 -0.26 -5.24 0.000 -0.890 1.000

3 1 252 66.67 0.32 6.59 0.000 0.710 1.060

4 0 91 24.07 -0.13 -2.59 0.010 0.050 1.080

8 0 1 0.26 -0.03 -0.49 0.622 -0.400 0.000

133

9 0 4 1.06 -0.1 -1.97 0.050 -0.550 0.400

Item 54

item:54 (PR6015Q03 )



Item Delta(s): 2.60


0 0 626 81.4 -0.1 -2.67 0.008 0.330 1.050

1 1 113 14.69 0.28 8.01 0.000 1.360 0.940

9 0 30 3.9 -0.31 -9.19 0.000 -1.120 1.010

Item 55

item:55 (PR6015Q04 )



Item Delta(s): 0.67


0 0 383 50 -0.4 -12.04 0.000 -0.060 0.960

1 1 348 45.43 0.51 16.6 0.000 1.090 0.900

9 0 35 4.57 -0.27 -7.8 0.000 -0.900 1.300

Item 56

item:56 (PR6015Q06 )





1 0 38 4.95 -0.19 -5.32 0.000 -0.590 1.200

2 0 49 6.38 -0.11 -3.11 0.002 -0.280 1.130

3 1 630 82.03 0.31 8.97 0.000 0.610 1.060

4 0 43 5.6 -0.19 -5.31 0.000 -0.430 1.020

8 0 1 0.13 -0.05 -1.3 0.193 -0.250 0.000

9 0 7 0.91 -0.05 -1.49 0.138 -0.460 0.710

134

Item 57

item:57 (PR6017Q04 )





1 0 40 10.58 -0.12 -2.3 0.022 0.080 0.910

2 1 286 75.66 0.35 7.2 0.000 0.790 1.000

3 0 17 4.5 -0.18 -3.5 0.001 -0.340 1.050

4 0 24 6.35 -0.23 -4.55 0.000 -0.380 0.940

8 0 1 0.26 -0.02 -0.45 0.653 -1.140 0.000

9 0 10 2.65 -0.12 -2.36 0.019 -0.180 1.160

Item 58

item:58 (PR6017Q05T )



Item Delta(s): -0.12 0.42


0 0 21 5.63 -0.3 -6.13 0.000 -0.640 0.920

1 0 78 20.91 -0.29 -5.82 0.000 -0.210 0.790

2 1 113 30.29 0.01 0.21 0.830 0.500 0.930

3 2 161 43.16 0.37 7.63 0.000 1.150 0.910

Item 59

item:59 (PR6017Q08 )





0 0 95 25.47 -0.3 -6 0.000 -0.100 0.930

1 1 251 67.29 0.41 8.55 0.000 0.910 0.960

9 0 27 7.24 -0.24 -4.66 0.000 -0.200 1.100

135

Item 60

item:60 (PR6017Q10 )



Item Delta(s): 0.01


1 0 38 10.27 -0.19 -3.62 0.000 -0.130 0.810

2 1 223 60.27 0.33 6.79 0.000 0.920 1.030

3 0 20 5.41 -0.25 -4.95 0.000 -0.440 0.970

4 0 86 23.24 -0.11 -2.09 0.037 0.270 0.910

9 0 3 0.81 -0.05 -0.98 0.326 -0.220 0.390

Item 61

item:61 (PR6017Q11 )



Item Delta(s): 0.56


1 0 94 25.41 -0.12 -2.41 0.016 0.260 0.980

2 1 183 49.46 0.2 3.99 0.000 0.920 1.030

3 0 67 18.11 -0.07 -1.39 0.166 0.300 1.020

4 0 24 6.49 -0.03 -0.63 0.529 0.190 0.980

9 0 2 0.54 -0.16 -3.08 0.002 -1.070 1.840

Item 62

item:62 (PR6018Q01T )





0 0 10 2.55 -0.15 -3.05 0.002 -0.760 0.870

1 0 9 2.3 -0.18 -3.53 0.000 -0.810 0.620

2 0 26 6.63 -0.32 -6.63 0.000 -0.960 1.090

3 0 46 11.73 -0.13 -2.54 0.011 -0.130 1.160

4 1 299 76.28 0.41 8.91 0.000 0.700 1.040

9 0 2 0.51 -0.06 -1.16 0.247 -1.510 2.160

136

Item 63

item:63 (PR6018Q03 )



Item Delta(s): 1.50


0 0 267 68.11 -0.33 -6.79 0.000 0.100 1.110

1 1 119 30.36 0.35 7.34 0.000 1.160 0.960

9 0 6 1.53 -0.07 -1.4 0.161 -0.440 1.780

Item 64

item:64 (PR6018Q08 )



Item Delta(s): -0.80 -0.47


0 0 53 13.52 -0.42 -9.14 0.000 -0.860 0.960

1 1 91 23.21 -0.16 -3.24 0.001 -0.030 0.960

2 2 237 60.46 0.51 11.72 0.000 0.940 0.930

9 0 11 2.81 -0.23 -4.62 0.000 -1.190 1.250

Item 65

item:65 (PR6018Q09 )



Item Delta(s): 0.13


0 0 156 39.8 -0.4 -8.52 0.000 -0.250 1.040

1 1 220 56.12 0.49 10.98 0.000 0.960 0.950

9 0 16 4.08 -0.24 -4.86 0.000 -0.680 1.360

Item 66

item:66 (PR6023Q01 )





1 0 36 9.23 -0.18 -3.54 0.000 -0.170 0.830

137

2 1 277 71.03 0.31 6.53 0.000 0.790 1.020

3 0 22 5.64 -0.19 -3.81 0.000 -0.290 0.830

4 0 49 12.56 -0.14 -2.76 0.006 0.000 1.080

8 0 1 0.26 0.03 0.55 0.580 0.710 0.000

9 0 5 1.28 -0.03 -0.56 0.573 0.020 1.300

Item 67

item:67 (PR6023Q06 )





1 0 7 1.8 -0.09 -1.7 0.090 -0.280 0.360

2 0 23 5.91 -0.25 -5.08 0.000 -0.650 0.900

3 1 318 81.75 0.35 7.4 0.000 0.750 1.000

4 0 33 8.48 -0.17 -3.33 0.001 -0.220 0.940

9 0 8 2.06 -0.13 -2.67 0.008 -0.510 1.060

Item 68

item:68 (PR6023Q07 )





1 0 9 2.31 -0.14 -2.82 0.005 -0.270 0.680

2 0 11 2.83 -0.15 -3.01 0.003 -0.640 0.780

3 0 43 11.05 -0.34 -7.02 0.000 -0.460 0.830

4 0 54 13.88 -0.24 -4.9 0.000 -0.120 0.800

5 1 264 67.87 0.53 12.22 0.000 0.940 0.940

8 0 1 0.26 0.01 0.27 0.786 -0.390 0.000

9 0 7 1.8 -0.09 -1.74 0.082 -0.250 1.250

138

Item 69

item:69 (PR6023Q08 )





0 0 78 20.05 -0.39 -8.23 0.000 -0.360 0.900

1 1 301 77.38 0.45 9.9 0.000 0.810 0.970

9 0 10 2.57 -0.21 -4.25 0.000 -0.710 0.850

Item 70

item:70 (PR6026Q02T )



Item Delta(s): 0.56


0 0 8 2.04 -0.04 -0.86 0.389 0.020 1.690

1 0 94 23.92 -0.16 -3.2 0.001 -0.010 0.930

2 0 91 23.16 -0.06 -1.1 0.274 0.170 1.040

3 1 187 47.58 0.26 5.32 0.000 0.860 1.100

8 0 1 0.25 -0.04 -0.84 0.400 -1.020 0.000

9 0 12 3.05 -0.17 -3.49 0.001 -0.580 0.830

Item 71

item:71 (PR6026Q05T )



Item Delta(s): 1.17


0 0 38 10.19 -0.21 -4.12 0.000 -0.180 1.230

1 0 57 15.28 -0.04 -0.81 0.419 0.210 0.960

2 0 136 36.46 -0.09 -1.67 0.096 0.170 1.040

3 1 135 36.19 0.25 4.98 0.000 1.000 1.020

9 0 7 1.88 -0 -0.07 0.944 0.430 0.740

139

Item 72

item:72 (PR6027Q01T )





0 0 8 2.07 -0.14 -2.75 0.006 -1.090 0.810

1 0 30 7.75 -0.1 -1.94 0.053 -0.280 1.180

2 0 108 27.91 -0.19 -3.72 0.000 -0.180 1.190

3 1 239 61.76 0.28 5.61 0.000 0.700 1.120

9 0 2 0.52 -0.06 -1.14 0.257 -0.360 1.070

Item 73

item:73 (PR6027Q04 )





1 1 332 85.79 0.41 8.79 0.000 0.570 1.110

2 0 19 4.91 -0.27 -5.53 0.000 -1.050 0.980

3 0 19 4.91 -0.18 -3.59 0.000 -0.780 1.040

4 0 15 3.88 -0.2 -4.09 0.000 -1.330 0.870

9 0 2 0.52 -0.08 -1.59 0.113 -1.630 1.540

Item 74

item:74 (PR6027Q10 )



Item Delta(s): 0.41


0 0 168 43.41 -0.22 -4.43 0.000 -0.070 1.160

1 1 191 49.35 0.32 6.62 0.000 0.830 1.120

9 0 28 7.24 -0.2 -3.9 0.000 -0.600 1.000

140

Science

Item 21

item:21 (PS7000Q01 )



Item Delta(s): 0.87


1 0 38 4.59 -0.14 -4 0.000 -0.330 0.820

2 1 279 33.74 0.24 7.1 0.000 0.430 0.860

3 0 207 25.03 -0.15 -4.25 0.000 -0.190 0.800

4 0 289 34.95 0 0.12 0.902 -0.040 0.800

8 0 3 0.36 -0.03 -0.75 0.453 -0.420 1.040

9 0 4 0.48 -0.07 -2.11 0.036 -1.280 1.760

r 0 7 0.85 -0.18 -5.27 0.000 -1.500 0.770

Item 22

item:22 (PS7000Q02T )





0 0 7 0.85 -0.07 -1.89 0.059 -0.650 1.390

1 0 41 4.95 -0.2 -5.97 0.000 -0.730 0.800

2 0 138 16.67 -0.17 -5.08 0.000 -0.430 0.820

3 1 630 76.09 0.32 9.62 0.000 0.240 0.800

9 0 5 0.6 -0.06 -1.75 0.081 -0.850 1.190

r 0 7 0.85 -0.17 -4.99 0.000 -1.500 0.770

Item 23

item:23 (PS7002Q03 )



Item Delta(s): 1.35


0 0 522 62.59 -0.09 -2.51 0.012 0.010 0.810

1 1 215 25.78 0.35 10.83 0.000 0.660 0.760

9 0 90 10.79 -0.31 -9.42 0.000 -0.670 0.790

r 0 7 0.84 -0.17 -4.95 0.000 -1.780 0.850

141

Item 24

item:24 (PS7002Q04 )





1 0 107 12.83 -0.23 -6.75 0.000 -0.580 0.790

2 0 33 3.96 -0.18 -5.13 0.000 -0.640 0.770

3 1 617 73.98 0.34 10.43 0.000 0.300 0.810

4 0 64 7.67 -0.06 -1.8 0.072 -0.210 0.950

8 0 3 0.36 -0.04 -1.03 0.301 -0.590 0.510

9 0 5 0.6 -0.12 -3.58 0.000 -1.290 0.690

r 0 5 0.6 -0.14 -4.01 0.000 -1.950 0.740

Item 25

item:25 (PS7012Q01 )



Item Delta(s): 0.77


1 0 109 13.49 -0.01 -0.23 0.818 -0.010 0.850

2 1 290 35.89 0.26 7.79 0.000 0.490 0.860

3 0 252 31.19 -0.18 -5.12 0.000 -0.200 0.850

4 0 144 17.82 -0.07 -2.07 0.039 -0.060 0.790

8 0 2 0.25 -0.02 -0.5 0.619 -0.380 0.110

9 0 7 0.87 -0.05 -1.55 0.121 -0.690 0.530

r 0 4 0.5 -0.12 -3.38 0.001 -2.320 0.620

Item 26

item:26 (PS7012Q03 )





0 0 268 33.17 -0.24 -7.12 0.000 -0.310 0.760

1 1 452 55.94 0.45 14.21 0.000 0.500 0.760

9 0 83 10.27 -0.32 -9.6 0.000 -0.770 0.760

r 0 5 0.62 -0.13 -3.85 0.000 -2.180 0.620

142

Item 27

item:27 (PS7012Q04T )





0 0 2 0.49 -0.1 -1.94 0.053 -1.770 0.650

1 0 21 5.17 -0.14 -2.94 0.003 -0.570 0.880

2 0 110 27.09 -0.09 -1.89 0.060 -0.250 0.840

3 1 270 66.5 0.2 4.04 0.000 0.190 0.920

9 0 1 0.25 -0.09 -1.73 0.084 -1.500 0.000

r 0 2 0.49 -0.12 -2.46 0.014 -2.620 0.050

Item 28

item:28 (PS7013Q01 )





1 0 112 13.53 -0.06 -1.82 0.070 -0.210 0.830

2 0 93 11.23 -0.24 -7.23 0.000 -0.630 0.750

3 0 62 7.49 -0.11 -3.27 0.001 -0.230 0.820

4 1 538 64.98 0.35 10.86 0.000 0.310 0.780

8 0 3 0.36 -0.04 -1.22 0.223 -0.260 0.500

9 0 12 1.45 -0.17 -4.9 0.000 -1.360 0.880

r 0 8 0.97 -0.18 -5.25 0.000 -1.440 0.740

Item 29

item:29 (PS7013Q02 )





0 0 203 24.52 -0.19 -5.41 0.000 -0.320 0.770

1 1 533 64.37 0.4 12.68 0.000 0.340 0.780

9 0 84 10.14 -0.32 -9.64 0.000 -0.770 0.800

r 0 8 0.97 -0.18 -5.28 0.000 -1.440 0.740

143

Item 30

item:30 (PS7014Q01 )



Item Delta(s): -0.57 -0.49


0 0 147 18.01 -0.27 -8.06 0.000 -0.590 0.770

1 1 215 26.35 -0.1 -2.84 0.005 -0.070 0.760

2 2 432 52.94 0.36 10.97 0.000 0.470 0.720

9 0 18 2.21 -0.17 -4.79 0.000 -0.930 0.630

r 0 4 0.49 -0.1 -2.72 0.007 -1.390 0.810

Item 31

item:31 (PS7014Q03 )



Item Delta(s): 0.87


0 0 451 55.27 -0.03 -0.76 0.448 0.030 0.800

1 1 279 34.19 0.24 7.14 0.000 0.450 0.770

9 0 82 10.05 -0.31 -9.46 0.000 -0.670 0.800

r 0 4 0.49 -0.11 -3.05 0.002 -1.390 0.810

Item 32

item:32 (PS7017Q01 )



Item Delta(s): 0.32


1 0 74 18.23 -0.11 -2.32 0.021 -0.250 0.760

2 0 70 17.24 -0.02 -0.36 0.722 -0.150 0.930

3 0 61 15.02 -0.14 -2.93 0.004 -0.400 0.790

4 1 178 43.84 0.32 6.88 0.000 0.430 0.860

8 0 5 1.23 -0.01 -0.15 0.881 -0.060 1.110

9 0 13 3.2 -0.21 -4.23 0.000 -0.950 0.670

r 0 5 1.23 -0.19 -3.91 0.000 -1.950 0.740

144

Item 33

item:33 (PS7017Q04D )



Item Delta(s): 1.74


9 0 136 16.98 -0.18 -5.1 0.000 -0.390 0.910

A 0 496 61.92 -0.07 -2.06 0.039 -0.020 0.810

G 1 131 16.35 0.28 8.21 0.000 0.750 0.800

H 1 24 3 0.15 4.42 0.000 1.110 0.600

r 0 14 1.75 -0.21 -6.07 0.000 -1.300 0.830

Item 34

item:34 (PS7018Q03 )



Item Delta(s): 1.16


1 0 148 18.45 0.01 0.15 0.882 -0.020 0.830

2 1 227 28.3 0.16 4.62 0.000 0.450 0.880

3 0 314 39.15 -0.05 -1.4 0.160 0.040 0.770

4 0 80 9.98 -0.05 -1.5 0.133 -0.120 0.910

8 0 5 0.62 0.02 0.51 0.612 -0.060 1.530

9 0 18 2.24 -0.15 -4.35 0.000 -0.660 0.880

r 0 10 1.25 -0.12 -3.44 0.001 -0.800 1.200

Item 35

item:35 (PS7018Q04T )



Item Delta(s): 2.04 -0.02


0 0 26 6.5 -0.06 -1.24 0.217 -0.170 0.600

1 0 81 20.25 -0.11 -2.22 0.027 -0.230 0.810

2 0 146 36.5 -0.07 -1.44 0.152 -0.110 0.820

3 1 78 19.5 0.13 2.54 0.011 0.310 0.890

4 2 54 13.5 0.23 4.66 0.000 0.660 0.850

9 0 5 1.25 -0.08 -1.54 0.124 -0.410 1.060

145

r 0 10 2.5 -0.16 -3.24 0.001 -0.480 0.970

Item 36

item:36 (PS7020Q03 )





1 0 18 2.27 -0.14 -4.08 0.000 -0.760 0.720

2 1 469 59.22 0.34 10.28 0.000 0.370 0.850

3 0 65 8.21 -0.21 -5.94 0.000 -0.550 0.610

4 0 215 27.15 -0.15 -4.12 0.000 -0.230 0.900

8 0 14 1.77 -0.04 -1.22 0.224 0.000 0.960

9 0 5 0.63 -0.09 -2.44 0.015 -0.920 0.950

r 0 6 0.76 -0.16 -4.45 0.000 -1.180 0.800

Item 37

item:37 (PS7020Q04 )



Item Delta(s): 0.69


0 0 212 53.4 -0.35 -7.31 0.000 -0.280 0.810

1 1 147 37.03 0.55 13.24 0.000 0.730 0.730

9 0 36 9.07 -0.3 -6.28 0.000 -0.810 0.790

r 0 2 0.5 -0.13 -2.52 0.012 -0.870 1.660

Item 38

item:38 (PS7021Q02T )



Item Delta(s): 0.89


0 0 30 3.62 -0.17 -4.83 0.000 -0.680 0.850

1 0 114 13.77 -0.24 -6.96 0.000 -0.360 0.780

2 0 394 47.58 0.02 0.5 0.619 0.060 0.820

3 1 283 34.18 0.23 6.93 0.000 0.450 0.820

9 0 6 0.72 -0.08 -2.19 0.029 -0.280 0.960

r 0 1 0.12 -0.04 -1.07 0.283 -1.510 0.000

146

Item 39

item:39 (PS7021Q04T )



Item Delta(s): 0.56


0 0 64 7.73 -0.19 -5.66 0.000 -0.420 0.880

1 0 166 20.05 -0.2 -5.92 0.000 -0.290 0.850

2 0 252 30.43 -0.02 -0.67 0.506 0.000 0.840

3 1 337 40.7 0.31 9.37 0.000 0.490 0.720

9 0 6 0.72 -0.06 -1.71 0.087 -0.230 0.890

r 0 3 0.36 -0.07 -2 0.046 -1.100 0.700

Item 40

item:40 (PS7201Q01 )



Item Delta(s): 0.15


0 0 153 38.06 -0.22 -4.52 0.000 -0.160 0.750

1 1 200 49.75 0.44 9.68 0.000 0.610 0.730

9 0 47 11.69 -0.32 -6.77 0.000 -0.570 0.650

r 0 2 0.5 -0.11 -2.2 0.029 -2.030 0.890

Item 41

item:41 (PS7201Q02 )





1 0 67 16.67 -0.21 -4.29 0.000 -0.180 0.780

2 0 29 7.21 -0.13 -2.65 0.008 -0.310 0.670

3 1 258 64.18 0.41 9.11 0.000 0.450 0.770

4 0 29 7.21 -0.15 -3.04 0.003 -0.430 0.680

8 0 3 0.75 -0.08 -1.64 0.103 -0.700 0.500

9 0 14 3.48 -0.18 -3.68 0.000 -0.800 0.770

r 0 2 0.5 -0.11 -2.13 0.034 -2.030 0.890

147

Item 42

item:42 (PS7201Q03 )



Item Delta(s): 2.40


0 0 290 72.14 0.01 0.27 0.787 0.160 0.840

1 1 48 11.94 0.28 5.82 0.000 0.880 0.690

9 0 63 15.67 -0.25 -5.23 0.000 -0.310 0.690

r 0 1 0.25 -0.1 -1.92 0.056 -2.660 0.000

Item 43

item:43 (PS7205Q02 )



Item Delta(s): 0.10


1 0 14 3.37 -0.07 -1.35 0.179 -0.090 0.750

2 0 27 6.49 -0.13 -2.67 0.008 -0.250 0.840

3 1 215 51.68 0.14 2.96 0.003 0.330 0.770

4 0 156 37.5 -0.01 -0.25 0.803 0.070 0.810

9 0 3 0.72 -0.18 -3.74 0.000 -1.900 1.080

r 0 1 0.24 -0.14 -2.83 0.005 -2.250 0.000

Item 44

item:44 (PS7208Q01 )





1 0 67 16.67 -0.1 -1.95 0.052 0.010 0.730

2 0 72 17.91 -0.24 -4.98 0.000 -0.350 0.720

3 1 222 55.22 0.35 7.52 0.000 0.470 0.820

4 0 32 7.96 -0.12 -2.52 0.012 -0.300 0.710

9 0 6 1.49 -0 -0.06 0.950 0.120 0.550

r 0 3 0.75 -0.14 -2.84 0.005 -1.890 0.670

148

Item 45

item:45 (PS7208Q02 )



Item Delta(s): 0.96


0 0 181 45.02 0.02 0.38 0.706 0.120 0.800

1 1 132 32.84 0.24 4.87 0.000 0.570 0.780

9 0 85 21.14 -0.26 -5.48 0.000 -0.300 0.780

r 0 4 1 -0.13 -2.57 0.011 -1.440 1.050

Item 46

item:46 (PS7211Q01 )



Item Delta(s): 1.43


0 0 287 68.5 -0.06 -1.19 0.233 0.030 0.820

1 1 103 24.58 0.28 5.91 0.000 0.580 0.700

9 0 21 5.01 -0.27 -5.83 0.000 -0.720 1.050

r 0 8 1.91 -0.24 -5.03 0.000 -1.300 0.680

Item 47

item:47 (PS7211Q02T )



Item Delta(s): 0.45


0 0 10 2.39 -0.13 -2.76 0.006 -0.690 0.910

1 0 59 14.08 -0.27 -5.7 0.000 -0.430 0.740

2 0 161 38.42 -0.03 -0.69 0.489 0.000 0.830

3 1 181 43.2 0.33 7.09 0.000 0.480 0.770

r 0 8 1.91 -0.24 -4.94 0.000 -1.300 0.680

149

Item 48

item:48 (PS7212Q01 )



Item Delta(s): 0.33


0 0 189 47.85 -0.07 -1.47 0.142 -0.040 0.860

1 1 179 45.32 0.24 4.91 0.000 0.420 0.810

9 0 20 5.06 -0.26 -5.28 0.000 -0.830 0.770

r 0 7 1.77 -0.2 -4.03 0.000 -0.810 0.710

Item 49

item:49 (PS7212Q02 )



Item Delta(s): 0.44


1 0 44 11.14 -0.2 -4.06 0.000 -0.400 0.720

2 1 170 43.04 0.19 3.76 0.000 0.380 0.850

3 0 51 12.91 -0.01 -0.13 0.900 0.190 0.900

4 0 116 29.37 0.04 0.73 0.465 0.020 0.800

8 0 1 0.25 -0.05 -1 0.319 -1.020 0.000

9 0 6 1.52 -0.12 -2.47 0.014 -1.370 1.240

r 0 7 1.77 -0.2 -4.02 0.000 -0.810 0.710

Item 50

item:50 (PS7212Q04 )



Item Delta(s): 2.89


0 0 251 63.54 0.16 3.24 0.001 0.160 0.810

1 1 31 7.85 0.25 5.07 0.000 1.070 0.660

9 0 103 26.08 -0.25 -5.19 0.000 -0.200 0.940

r 0 10 2.53 -0.21 -4.27 0.000 -0.700 0.630

150

Item 51

item:51 (PS7215Q02 )



Item Delta(s): 1.33


0 0 237 55.37 -0.14 -3.02 0.003 0.030 0.770

1 1 115 26.87 0.44 10.14 0.000 0.800 0.630

9 0 69 16.12 -0.26 -5.64 0.000 -0.310 0.860

r 0 7 1.64 -0.21 -4.41 0.000 -0.840 0.800

Item 52

item:52 (PS7215Q03 )



Item Delta(s): 0.94


0 0 244 57.01 0.01 0.15 0.884 0.100 0.800

1 1 142 33.18 0.2 4.26 0.000 0.480 0.780

9 0 17 3.97 -0.15 -3.24 0.001 -0.420 1.050

r 0 25 5.84 -0.29 -6.29 0.000 -0.610 0.760

Item 53

item:53 (PS7216Q01 )





1 0 16 3.82 -0.15 -3.2 0.001 -0.550 0.650

2 0 105 25.06 -0.13 -2.75 0.006 -0.160 0.850

3 0 50 11.93 -0.2 -4.17 0.000 -0.370 0.720

4 1 237 56.56 0.36 8 0.000 0.410 0.770

8 0 1 0.24 -0.03 -0.57 0.567 -0.120 0.000

9 0 4 0.95 -0 -0.08 0.937 -0.590 2.010

r 0 6 1.43 -0.23 -4.73 0.000 -1.390 0.780

151

Item 54

item:54 (PS7216Q03 )





0 0 216 51.55 -0.22 -4.57 0.000 -0.130 0.760

1 1 113 26.97 0.2 4.16 0.000 0.350 0.780

2 2 62 14.8 0.27 5.76 0.000 0.900 0.660

9 0 22 5.25 -0.22 -4.51 0.000 -0.680 0.920

r 0 6 1.43 -0.23 -4.91 0.000 -1.390 0.780

Item 55

item:55 (PS7217Q01T )





0 0 2 0.5 -0.1 -2 0.047 -1.000 0.530

1 0 17 4.23 -0.18 -3.72 0.000 -0.490 0.790

2 0 26 6.47 -0.3 -6.21 0.000 -0.830 0.680

3 0 19 4.73 -0.14 -2.92 0.004 -0.360 0.550

4 0 69 17.16 -0.08 -1.57 0.117 -0.110 0.740

5 1 266 66.17 0.4 8.65 0.000 0.440 0.770

9 0 2 0.5 -0.09 -1.8 0.073 -1.120 0.710

r 0 1 0.25 -0.09 -1.83 0.069 -2.660 0.000

Item 56

item:56 (PS7217Q02 )





0 0 97 24.13 -0.17 -3.42 0.001 -0.220 0.700

1 1 158 39.3 0.25 5.11 0.000 0.450 0.780

2 2 65 16.17 0.24 5.04 0.000 0.820 0.680

9 0 81 20.15 -0.33 -7.09 0.000 -0.420 0.680

r 0 1 0.25 -0.09 -1.88 0.061 -2.660 0.000

152

Item 57

item:57 (PS7220Q01 )





1 1 315 73.77 0.43 9.77 0.000 0.390 0.790

2 0 34 7.96 -0.2 -4.19 0.000 -0.320 0.590

3 0 35 8.2 -0.14 -3 0.003 -0.330 0.730

4 0 28 6.56 -0.18 -3.73 0.000 -0.440 0.650

8 0 2 0.47 -0.08 -1.75 0.080 -1.070 0.470

9 0 9 2.11 -0.23 -4.85 0.000 -1.190 0.690

r 0 4 0.94 -0.13 -2.64 0.009 -0.930 1.040

Item 58

item:58 (PS7220Q02T )





0 0 43 10.05 -0.22 -4.71 0.000 -0.430 0.760

1 0 111 25.93 -0.05 -0.99 0.322 -0.040 0.760

2 1 263 61.45 0.22 4.76 0.000 0.380 0.810

9 0 6 1.4 -0.04 -0.92 0.359 -0.300 1.420

r 0 5 1.17 -0.15 -3.13 0.002 -0.910 0.910

Item 59

item:59 (PS7220Q03 )



Item Delta(s): 0.26


1 0 77 17.99 -0.03 -0.72 0.475 0.030 0.900

2 0 80 18.69 -0.14 -2.89 0.004 -0.130 0.680

3 0 49 11.45 -0.19 -4.08 0.000 -0.270 0.750

4 1 207 48.36 0.33 7.09 0.000 0.490 0.770

8 0 5 1.17 0.01 0.28 0.783 0.080 0.890

9 0 5 1.17 -0.17 -3.5 0.001 -1.040 1.030

153

r 0 5 1.17 -0.16 -3.26 0.001 -0.910 0.910

Item 60

item:60 (PS7221Q01 )





1 0 4 0.96 -0.04 -0.84 0.404 -0.120 0.740

2 0 9 2.16 -0.12 -2.41 0.016 -0.500 0.940

3 1 394 94.71 0.23 4.7 0.000 0.210 0.800

4 0 5 1.2 -0.07 -1.45 0.149 -0.640 0.970

8 0 1 0.24 -0.12 -2.46 0.014 -2.050 0.000

9 0 2 0.48 -0.13 -2.74 0.006 -1.660 1.410

r 0 1 0.24 -0.13 -2.69 0.007 -2.250 0.000

Item 61

item:61 (PS7221Q02 )





0 0 151 36.3 -0.23 -4.8 0.000 -0.170 0.740

1 1 157 37.74 0.02 0.49 0.624 0.210 0.710

2 2 98 23.56 0.31 6.66 0.000 0.710 0.730

9 0 9 2.16 -0.18 -3.73 0.000 -1.020 1.280

r 0 1 0.24 -0.14 -2.93 0.004 -2.250 0.000

Item 62

item:62 (PS7221Q03T )



Item Delta(s): 0.01


0 0 3 0.72 -0.04 -0.79 0.432 0.300 1.490

1 0 44 10.58 -0.32 -6.97 0.000 -0.520 0.750

2 0 145 34.86 -0.1 -2.08 0.038 -0.040 0.840

3 1 223 53.61 0.32 6.81 0.000 0.430 0.720

r 0 1 0.24 -0.14 -2.89 0.004 -2.250 0.000

154

Item 63

item:63 (PS7221Q04 )





1 0 4 0.96 0.01 0.28 0.777 0.280 0.900

2 1 381 91.59 0.24 4.96 0.000 0.220 0.790

3 0 1 0.24 -0.09 -1.91 0.057 -1.740 0.000

4 0 24 5.77 -0.19 -3.85 0.000 -0.580 0.970

8 0 2 0.48 0.01 0.11 0.915 0.620 0.160

9 0 3 0.72 -0.15 -3.19 0.002 -1.090 1.140

r 0 1 0.24 -0.13 -2.72 0.007 -2.250 0.000

Item 64

item:64 (PS7222Q01 )





1 0 18 4.3 -0.12 -2.51 0.013 -0.570 0.770

2 0 9 2.15 -0.19 -3.92 0.000 -1.130 0.660

3 0 25 5.97 -0.2 -4.15 0.000 -0.700 0.730

4 1 358 85.44 0.35 7.75 0.000 0.250 0.810

8 0 2 0.48 0.01 0.18 0.857 0.100 0.780

9 0 3 0.72 -0.11 -2.16 0.031 -1.230 1.660

r 0 4 0.95 -0.18 -3.77 0.000 -1.230 0.560

Item 65

item:65 (PS7222Q02T )



Item Delta(s): 1.07


0 0 19 4.53 -0.16 -3.31 0.001 -0.490 0.670

1 0 78 18.62 -0.11 -2.33 0.020 -0.130 0.900

2 0 185 44.15 -0.01 -0.14 0.892 0.030 0.850

3 1 130 31.03 0.24 4.98 0.000 0.510 0.740

155

9 0 3 0.72 -0.12 -2.4 0.017 -1.060 1.380

r 0 4 0.95 -0.2 -4.07 0.000 -1.230 0.560

Item 66

item:66 (PS7222Q03T )





0 0 44 10.5 -0.2 -4.12 0.000 -0.360 0.750

1 0 109 26.01 -0.15 -3.11 0.002 -0.160 0.800

2 1 252 60.14 0.32 6.93 0.000 0.340 0.840

9 0 9 2.15 -0.05 -1.02 0.308 -0.300 0.730

r 0 5 1.19 -0.22 -4.52 0.000 -1.510 0.800

Item 67

item:67 (PS7224Q02T )



Item Delta(s): 1.91


0 0 44 10.95 -0.16 -3.23 0.001 -0.210 0.730

1 0 80 19.9 -0.16 -3.21 0.001 -0.070 0.810

2 0 201 50 0.06 1.1 0.271 0.170 0.790

3 1 70 17.41 0.29 5.98 0.000 0.780 0.830

9 0 3 0.75 -0.12 -2.35 0.019 -0.720 0.710

r 0 4 1 -0.13 -2.65 0.008 -1.440 1.050

Item 68

item:68 (PS7228Q02 )



Item Delta(s): 0.89


0 0 235 59.49 -0.2 -4.06 0.000 -0.070 0.870

1 1 135 34.18 0.37 7.91 0.000 0.590 0.750

9 0 19 4.81 -0.24 -4.84 0.000 -0.620 0.600

r 0 6 1.52 -0.22 -4.4 0.000 -0.980 0.600

156

Item 69

item:69 (PS7228Q03 )



Item Delta(s): 0.85


0 0 194 49.11 -0.2 -3.99 0.000 -0.090 0.800

1 1 138 34.94 0.49 11 0.000 0.670 0.730

9 0 57 14.43 -0.3 -6.26 0.000 -0.410 0.880

r 0 6 1.52 -0.22 -4.46 0.000 -0.980 0.600

Item 70

item:70 (PS7228Q04 )



Item Delta(s): 0.53


1 0 37 9.37 -0.15 -2.93 0.004 -0.230 0.590

2 1 163 41.27 0.32 6.62 0.000 0.520 0.820

3 0 127 32.15 0.01 0.17 0.864 -0.010 0.850

4 0 46 11.65 -0.2 -4.03 0.000 -0.320 0.770

8 0 2 0.51 0 0.05 0.960 0.240 0.980

9 0 14 3.54 -0.15 -2.97 0.003 -0.730 1.110

r 0 6 1.52 -0.21 -4.33 0.000 -0.980 0.600

Item 71

item:71 (PS7230Q02T )





0 0 31 7.45 -0.12 -2.48 0.013 -0.170 0.870

1 0 119 28.61 -0.22 -4.67 0.000 -0.200 0.750

2 1 263 63.22 0.31 6.64 0.000 0.380 0.770

r 0 3 0.72 -0.2 -4.09 0.000 -1.860 1.050

157

Item 72

item:72 (PS7230Q03T )



Item Delta(s): 1.41


0 0 1 0.24 -0.06 -1.25 0.213 -0.690 0.000

1 0 42 10.1 -0.17 -3.54 0.000 -0.220 0.870

2 0 110 26.44 -0.12 -2.39 0.017 -0.060 0.720

3 0 152 36.54 0.05 0.99 0.320 0.160 0.760

4 1 105 25.24 0.23 4.79 0.000 0.610 0.800

9 0 3 0.72 0 0.04 0.965 0.020 0.280

r 0 3 0.72 -0.2 -4.21 0.000 -1.860 1.050

Item 73

item:73 (PS7231Q01T )



Item Delta(s): 0.78


0 0 10 2.4 -0.09 -1.91 0.056 -0.270 0.400

1 0 52 12.5 -0.29 -6.23 0.000 -0.490 0.790

2 1 154 37.02 0.02 0.48 0.628 0.320 0.750

3 0 198 47.6 0.22 4.68 0.000 0.250 0.810

r 0 2 0.48 -0.18 -3.7 0.000 -2.450 0.290

Item 74

item:74 (PS7231Q04T )



Item Delta(s): 0.66


0 0 21 5.05 -0.19 -3.96 0.000 -0.490 0.900

1 0 70 16.83 -0.07 -1.34 0.182 0.010 0.690

2 0 157 37.74 -0.12 -2.44 0.015 -0.050 0.830

3 1 165 39.66 0.29 6.14 0.000 0.540 0.690

9 0 1 0.24 -0.09 -1.81 0.071 -0.670 0.000

r 0 2 0.48 -0.18 -3.81 0.000 -2.450 0.290

159

Appendix 4: ICC gender performance

Mathematics

169

Reading

181

Science

195

Appendix 5: Pilot trial item pool characteristics

Mathematics

Unit Item

Code Unit Name Content Context Process Cluster Delta tau1 tau2 Thresholds

PM5002Q01 Internet Use Uncertainty and data Societal Employing RSM -0.06

553.34

PM5002Q02 Internet Use Quantity Societal Employing RSM 0.63

641.52

PM5008Q01 Tiling Pattern Space and shape Societal Employing PM1 -0.90

715.52

PM5010Q01T Climbing Frame Space and shape Personal Interpreting RSM 1.07 -1.84 1.84 477.79 767.55

PM5010Q02 Climbing Frame Space and shape Personal Formulating RSM 0.26

689.81

PM5013Q01 Energy Use Uncertainty and data Scientific Employing PM2 -0.44

588.79

PM5013Q03 Energy Use Uncertainty and data Scientific Interpreting PM 2 2.60

684.44

PM5015Q02 Aspect Ratio Change and relationships Personal Employing PM 1 0.67

667.61

PM5017Q02 Weather Forecast Uncertainty and data Scientific Interpreting PM 1 -1.45

598.05

PM5017Q03 Weather Forecast Uncertainty and data Scientific Formulating PM 1 -0.88 0.57 -0.57 492.59 535.43

PM5020Q02 Golden Rectangle Change and relationships Scientific Employing RSM 0.15

747.69

PM5021Q01T Freezer Temperatures Quantity Occupational Interpreting PM 2 -0.37

493.13

PM5021Q02 Freezer Temperatures Quantity Occupational Employing PM 2 0.01

617.53

PM5026Q01 Sculpture Change and relationships Scientific Formulating PM 2 0.56

829.17

PM5101Q01 Lighthouse Space and shape Scientific Formulating PM 2 -1.05

691.84

PM5101Q02 Lighthouse Quantity Scientific Employing PM 2 1.50

720.58

PM5102Q01 Tree Trunk Change and relationships Scientific Employing PM 1 -0.64

858.06

PM5104Q01 Baby Growth Uncertainty and data Personal Formulating PM 2 0.13

504.74

196

Unit Item


PM5107Q01 Taxi Fares Quantity Societal Employing PM 2 -0.62

674.86

PM5107Q02 Taxi Fares Quantity Societal Employing PM 2 -1.34 0.67 -0.67 735.62 774.56

PM5109Q01 Arrowhead Kite Space and shape Personal Employing PM 1 -0.44

681.87

PM5109Q02 Arrowhead Kite Space and shape Personal Employing PM 1 -1.02 2.03 -2.03 783.13 793.26

PM5110Q01 Fence Change and relationships Occupational Formulating PM 1 0.56

627.26

PM5110Q02 Fence Change and relationships Occupational Formulating PM 1 1.17 0.51 -0.51 732.50 778.46

PM5124Q01 Concrete Change and relationships Occupational Formulating PM 2 -0.24

567.05

PM5124Q02 Concrete Change and relationships Occupational Formulating PM 2 -1.88

625.63

PM5124Q03 Concrete Change and relationships Occupational Formulating PM 2 0.41

653.98

PM5142Q01 Machu Picchu Quantity Societal Interpreting PM 1 -3.50

372.16

PM5142Q02 Machu Picchu Quantity Societal Employing PM 1 -1.33

540.72

PM5145Q01 Decagon Tiles Space and shape Scientific Employing PM 2 1.45

757.74

PM5163Q01 Hair Gel Quantity Personal Employing PM 1 -0.72

588.71

PM5169Q01 Shoe Sizes Change and relationships Personal Employing PM 1 -1.21

550.38

PM5169Q02 Shoe Sizes Change and relationships Personal Employing PM 1 -0.11

635.99

PM5181Q02 Pendulum Change and relationships Scientific Employing PM 2 0.85

710.61

PM5183Q01 House Frame Space and shape Occupational Interpreting PM 2 -1.23 0.32 -0.32 520.63 576.71

PM5183Q02 House Frame Space and shape Occupational Formulating PM 2 -0.26

624.54

PM5185Q01 Jeans Sizes Uncertainty and data Societal Interpreting PM 1 -4.02

331.35

PM5185Q02T Jeans Sizes Change and relationships Societal Interpreting PM 1 -1.28

545.16

PM5187Q01 Jaden’s House Space and shape Personal Interpreting PM 1 1.55

765.53

197

Unit Item


PM5188Q01 Car Speed Uncertainty and data Societal Employing RSM -1.39

536.28

198

Reading

Unit Item

Code Unit Name Aspect

Text

Format Situation Cluster Delta tau1 tau2 Thresholds

PR6002Q01 Egyptian Mummies Integrate and interpret Continuous Educational PR 2 0.84

576.19

PR6002Q02T Egyptian Mummies Access and retrieve Continuous Educational PR 2 1.49

635.36

PR6002Q04 Egyptian Mummies Access and retrieve Continuous Educational PR 2 -0.27

475.57

PR6002Q05 Egyptian Mummies Integrate and interpret Continuous Educational PR 2 0.23

521.47

PR6004Q01 Museum Park Access and retrieve

Non-continuous Public PR 1 -1.14

396.77

PR6004Q05B Museum Park Reflect and evaluate


363.59

PR6004Q06 Museum Park Access and retrieve


284.97

PR6008Q02 Mobile Phones Access and retrieve

Non-continuous Public PR 2 0.84

576.64

PR6008Q04 Mobile Phones Reflect and evaluate

Non-continuous Public PM 2 -1.56

358.14

PR6008Q05 Mobile Phones Access and retrieve


505.47

PR6010Q01 Exploration Today Access and retrieve Continuous Educational PR 2 0.22

519.93

PR6010Q03 Exploration Today Integrate and interpret Continuous Educational PR 2 2.78

752.61

PR6010Q06 Exploration Today Integrate and interpret Continuous Educational PR 2 0.50

545.29

PR6010Q07 Exploration Today Reflect and evaluate Continuous Educational PR 2 1.50

636.54

PR6011Q03 Returning Integrate and interpret Continuous Personal PR 2 0.23

520.74

PR6011Q05 Returning Integrate and interpret Continuous Personal PR 2 2.30

709.62

PR6011Q07 Returning Integrate and interpret Continuous Personal PR 2 -1.06

403.77

199

Unit Item


Text


PR6013Q03 About Cats Integrate and interpret Continuous Personal PR 1 1.13

603.00

PR6013Q04 About Cats Access and retrieve Continuous Personal PR 1 0.93

584.55

PR6013Q06 About Cats Reflect and evaluate Continuous Personal PR 1 -0.06

494.57

PR6013Q08 About Cats Reflect and evaluate Continuous Personal PR 1 0.63

557.74

PR6014Q03

The Job of an Air Traffic Controller Reflect and evaluate Continuous Occupational PR 2 -0.90

418.58

PR6014Q06T

The Job of an Air Traffic Controller Access and retrieve Continuous Occupational PR 2 1.07 -0.68 0.68 518.47 676.62

PR6014Q07 The Job of an Air Traffic Controller Reflect and evaluate Continuous Occupational PR 2 0.26

524.02

PR6015Q01

Kenya Tourism and Conservation Reflect and evaluate Continuous Educational RSM -0.44

459.94

PR6015Q03

Kenya Tourism and Conservation Integrate and interpret Continuous Educational RSM 2.60

736.25

PR6015Q04

Kenya Tourism and Conservation Access and retrieve Continuous Educational RSM 0.67

561.19

PR6015Q06

Kenya Tourism and Conservation Access and retrieve

Non-continuous Educational RSM -1.45

368.86

PR6017Q04 The Two Campaigns Integrate and interpret Continuous Personal PR 1 -0.88

420.49

PR6017Q05T The Two Campaigns Integrate and interpret Continuous Personal PR 1 0.15 -0.27 0.27 457.57 569.37

PR6017Q08 The Two Campaigns Integrate and interpret Continuous Personal PR 1 -0.37

466.84

PR6017Q10 The Two Campaigns Access and retrieve Continuous Personal PR 1 0.01

501.57

PR6017Q11 The Two Campaigns Integrate and interpret Continuous Personal PR 1 0.56

550.92

PR6018Q01T Aski Gym Youth Programme Access and retrieve Continuous Personal PR 1 -1.05

405.13

200

Unit Item


Text


PR6018Q03 Aski Gym Youth Programme Integrate and interpret Continuous Personal PR 1 1.50

636.81

PR6018Q08 Aski Gym Youth Programme Integrate and interpret Continuous Personal PR 1 -0.64 -0.16 0.16 391.22 493.02

PR6018Q09 Aski Gym Youth Programme Reflect and evaluate Mixed Personal PR 1 0.13

511.93

PR6023Q01 Trial by Teen Reflect and evaluate Mixed Occupational PR 2 -0.62

443.76

PR6023Q06 Trial by Teen Integrate and interpret Continuous Occupational PR 2 -1.34

378.50

PR6023Q07 Trial by Teen Integrate and interpret Mixed Occupational PR 2 -0.44

460.57

PR6023Q08 Trial by Teen Reflect and evaluate

Non-continuous Occupational PR 2 -1.02

407.49

PR6026Q02T Bus Timetable Access and retrieve


551.19

PR6026Q05T Bus Timetable Access and retrieve


606.27

PR6027Q01T Underground Cooking Access and retrieve

Non-continuous Personal RSM -0.24

478.21

PR6027Q04 Underground Cooking Access and retrieve

Non-continuous Personal RSM -1.88

329.23

PR6027Q10 Underground Cooking Integrate and interpret

Non-continuous Personal RSM 0.41

537.47

201

Science

Unit Item

Code Unit Name Competency Application Area

Context

setting area Cluster Delta tau1 tau2 Thresholds

PS7000Q01 Oily birds Explaining phenomena scientifically Environment Social RSM 0.87

611.91

PS7000Q02T Oily birds Using scientific evidence Environment Social RSM -1.27

412.36

PS7002Q03 Cooking

Explaining phenomena scientifically Environment Global RSM 1.35

656.30

PS7002Q04 Cooking Using scientific evidence Natural resources Social RSM -1.11

427.65

PS7012Q01 First aid

Explaining phenomena scientifically Health Personal PS 1 0.77

602.77

PS7012Q03 First Aid

Explaining phenomena scientifically Health Personal PS 1 -0.19

512.97

PS7012Q04T First aid

Explaining phenomena scientifically Health Personal PS 1 -0.77

458.70

PS7013Q01 Ocean currents Using scientific evidence

Frontiers of science and technology Global PS 1 -0.65

470.64

PS7013Q02 Ocean currents

Explaining phenomena scientifically Hazard Social PS 1 -0.62

473.53

PS7014Q01 Oil spills Using scientific evidence Hazard Social PS 1 -0.53 -0.04 0.04 434.83 528.08

PS7014Q03 Oil spills Explaining phenomena scientifically Health Social PS 1 0.87

611.63

PS7017Q01 Galileo's clock Identifying scientific issues

Frontiers of science and technology Personal PS 2 0.33

561.18

PS7017Q04D Galileo's clock Identifying scientific issues


692.66

PS7018Q03 Watering cattle Using scientific evidence


638.58

202

Unit Item


Context


PS7018Q04T Watering cattle Using scientific evidence Frontiers of science and technology Personal PS 1 1.01 1.03 -1.03 608.27 641.84

PS7020Q03 Stone pots Explaining phenomena scientifically Natural resources Social PS 2 -0.35

498.24

PS7020Q04 Stone pots

Explaining phenomena scientifically Natural resources Social PS 2 0.69

595.41

PS7021Q02T Sterilised milk Identifying scientific issues Health Social RSM 0.89

613.50

PS7021Q04T Sterilised milk Identifying scientific issues Environment Social RSM 0.56

583.28

PS7201Q01 Water Purification Using scientific evidence Hazard Social PS 1 0.15

545.14

PS7201Q02 Water Purification Identifying scientific issues Health Social PS 1 -0.54

480.80

PS7201Q03 Water Purification Using scientific evidence Hazard Social PS 1 2.40

754.30

PS7205Q02 Plate Tectonics Using scientific evidence Hazard Global PS 2 0.10

540.20

PS7208Q01

Identifying Substances Using scientific evidence Frontiers of science and technology

Personal PS 1 -0.11

521.09

PS7208Q02

Identifying Substances Identifying scientific issues Frontiers of science and technology

Personal PS 1 0.96

620.77

PS7211Q01

Ceramics Explaining phenomena scientifically

Natural resources Global PS 1 1.43

664.41

PS7211Q02T

Ceramics Using scientific evidence Frontiers of science and technology

Social PS 1 0.45

572.75

PS7212Q01

Nitrogen in Rocks Explaining phenomena scientifically


561.56

PS7212Q02

Nitrogen in Rocks Explaining phenomena scientifically


571.81

PS7212Q04 Nitrogen in Rocks Identifying scientific issues Natural resources Global PS 2 2.89

799.99

203

Unit Item


Context




Health Personal PS 2 1.33

654.99

PS7215Q03

Blood Clotting Explaining phenomena scientifically

Health Social PS 2 0.94

618.72

PS7216Q01

Fish Management Explaining phenomena scientifically

Natural resources Global RSM -0.18

514.09

PS7216Q03 Fish Management Using scientific evidence Natural resources Global RSM 1.09 -0.21 0.21 578.43 686.60

PS7217Q01T Ocean Debris


Environment Social PS 1 -0.63

471.94

PS7217Q02

Ocean Debris Explaining phenomena scientifically

Environment Global PS 1 0.81 -0.66 0.66 527.15 686.60



Hazard Social PS 1 -1.02

435.58

PS7220Q02T Natural Gas Identifying scientific issues Environment Social PS 1 -0.37

496.84

PS7220Q03 Natural Gas Identifying scientific issues Natural resources Social PS 1 0.26

554.84

PS7221Q01

Clean Drinking Water Explaining phenomena scientifically

Health Personal PS 2 -3.05

246.19

PS7221Q02



Social PS 2 0.50 -0.47 0.47 509.43 646.51

PS7221Q03T Clean Drinking Water


Health Personal PS 2 0.01

532.00

PS7221Q04


Health Social PS 2 -2.51

296.92

PS7222Q01 Texan Summer Identifying scientific issues Environment Social PS 2 -1.94

350.35

PS7222Q02T Texan Summer Using scientific evidence Environment Global PS 2 1.07

630.93

PS7222Q03T Texan Summer Using scientific evidence Environment Global PS 2 -0.35

498.33

204

Unit Item


Context


PS7224Q02T Moon Rock Using scientific evidence


Global PS 2 1.91

709.08

PS7228Q02 C-Dating

Using scientific evidence Frontiers of science and technology

Global PS 2 0.89

613.40

PS7228Q03 C-Dating


Global PS 2 0.85

609.77

PS7228Q04 C-Dating


Personal PS 2 0.53

580.30

PS7230Q02T Pesticides Using scientific evidence Hazard Social PS 1 -0.45

488.73

PS7230Q03T Pesticides Identifying scientific issues Hazard Social PS 1 1.41

661.98

PS7231Q01T Pollen Allergy Using scientific evidence Health Social PS 2 0.78

603.89

PS7231Q04T Pollen Allergy Identifying scientific issues Health Social PS 2 0.66

591.96

Australian Council for Educational Research Test for Schools Technical... · 3 Foreword This...

Documents

Transcript of Australian Council for Educational Research Test for Schools Technical... · 3 Foreword This...