PISA FOR DEVELOPMENT

Michael Ward

February, 2016

Benefits to countries of participating in PISA for Development

• International benchmarking – Education SDG

• Learning assessment results which better describe the country’s levels of proficiency

• Build capacity in assessment

• Join international community focused on learning outcomes with equity

• Gain access to the world’s leading assessment specialists and expertise

• Work with OECD to produce national PISA reports

115 countries have participated in many international large-scale assessments, including PISA

International assessments

LLECE

Uwezo

SACMEQ

PASEC ASER

PIRLS, PISA, TIMSS, STEP, LAMP, PIAAC, WEI-SPS

and EGRA have had global presence ...

SEA-PLM

... whereas others have had a

regional focus

5

Education SDG and Education 2030 Framework for Action

5 out of 7 Education SDG targets are focused on learning outcomes and skills – increased demand for measuring learning outcomes on a global scale and PISA is included as part of the SDG monitoring framework

Indicator 4.1.1 Means of Verification

Proportion of children/young people: (a) in Grade 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving at least a minimum proficiency level in (i) reading and (ii) mathematics, by sex

(c) PISA: 2015; 2018; 2021; 2024; 2027; and 2030

6

Education SDG: Proposed Global Indicator 4.1.1

PISA Cycle No. of low and middle income countries participating

2000 15

2003 14

2006 26

2009-10 34

2012 23

2015 26

2018 40*

7

PISA is already for Development

PISA and PISA-D

Outputs of PISA-D informing and

enhancing PISA from 2021 cycle onwards

What has been the experience of low- and middle-income countries in PISA?

• Participation built on prior experience with national, regional or international assessments

• Financial challenges of PISA participation greater than technical ones

• PISA results have informed some education policy

• PISA results highlight differences in education quality between high income and middle income countries

• Countries and OECD could do more to maximise participation by low and middle income countries

10

What do we need to do technically to enhance PISA for developing countries?

11

What are the needs and priorities of the participating countries?

What can we learn from other international large-scale assessments?

• Each of the assessments has been created to answer the needs of its own constituents.

• The assessments share more in common in the procedures they use than they do with content.

• The procedures which are common are those for creating frameworks, item development, sampling, scaling, analysis and reporting.

• The content most easily transferable is that contained in the contextual questionnaires – there are certain contextual characteristics needed across most developing countries.

• The least transferable content between assessments is the cognitive component – the test questions usually reflect very specific needs of the constituents, the curricula upon which they are based and the population at whom they are targeted.

Enhancing PISA to make it more relevant to developing countries

• Adjusting the PISA tests to better measure and discriminate at lower levels of proficiency

• Making the contextual instruments more relevant to lower-middle-income country contexts and policy issues

• Incorporating out-of-school youth in the assessment

• Helping countries to tackle financial and technical challenges through partnerships with donors and peers and through capacity building

• Ensuring the results of the assessment are used for policy dialogue and decision making through increased outreach to stakeholders, support for national analysis and reporting and sensitivity to local context

Proficiency level

Lowest score point in the level

Mathematics Reading Science

At or above the

baseline proficiency

Level 6 669 698 708

Level 5 607 626 633

Level 4 545 553 559

Level 3 482 480 484

Level 2 (baseline)

420 407 410

Below the baseline proficiency

Level 1 358

Level 1a 335

335

Level 1b 262

Below Level 1 . . .

Proficiency levels in mathematics, reading and science

In mathematics, below proficiency are those who score below 420

points

In reading, below proficiency are those who score below 407 points

In science, below proficiency are those who score below 410 points

Higher share of students from middle income countries scoring at Level 1 or below, compared with OECD average

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

Pe

rce

nt

Stu

de

nts

Sco

rin

g at

Le

vel 1

an

d B

elo

w, P

ISA

20

12

Mat

he

mat

ics

Level 1 Below Level 1

Proficiency level

Percentage of items at each level: PISA

Percentage of items at each level: PISAD

Level 6

78%

35%

Level 5

Level 4

Level 3

Level 2 (baseline)

22%

65%

Level 1

Below Level 1

16

Adjusting the PISA assessment to better measure and discriminate at lower levels of proficiency

Making the contextual instruments more relevant to developing country contexts and policy issues

18

Making the contextual instruments more relevant to developing country contexts and policy issues

PISA-D focuses on background, prosperity

outcomes and foundations for success

19

Making the contextual instruments more relevant to developing country contexts and policy issues

20

Incorporating out-of-school youth in the assessment

Strand A

Strand C

Incorporating out-of-school youth in the assessment – school and household survey

• A large enough sample size (2,700 completed cases) to test the validity of the questionnaire and the assessment items on the PISA scale and to provide a meaningful report for the countries

• To explore, test, validate and evaluate a methodology and an approach for identifying and assessing a nationally representative sample of 14-16 year-olds out-of-school or in –school in grades 6 and below in future cycles of PISA

• In-person interviews with 14-16 year-olds in grade 6 and below and in households

• Sampling plans that minimise costs to countries yet maximise results (probability-based sample and non-probability-based sample)

• Reading literacy and mathematics literacy tests (55minutes)

• Background questionnaire - respondent (35 minutes) and parent/guardian

• Test and questionnaire (respondent) delivered via tablet computer

22

Incorporating out-of-school youth in the assessment – school and household survey

Capacity development

Preparation and support

Technical assistance

Project implementation

plan

Capacity building plan

Capacity needs analysis

Making use of the results

Engagement & communication

Policy dialogue

Dissemination of results

Preparation of a country

report

Country-specific

data analysis

Preparation of in-country stakeholders

Project management structure

• PISA Governing Board, International Advisory Group and Technical Advisory Group

• Eight participating countries: Cambodia, Ecuador, Guatemala, Honduras, Panama, Paraguay, Senegal and Zambia

• 11 development partners

• 9 technical partners

• Five international contractors

• Associated experts

26

2015 2016 2017 2018 Develop

assessment frameworks, test design and master

instruments Develop survey

materials, including national assessment

instruments Carry out FT

activities, including coding and data entry

Analyze FT data and prepare for

MS

Carry out MS activities,

including coding and data entry

Analyze MS data

Report planning and dissemination

Phase 1

Phase 2

Phase 3

Phase 4

Project implementation

Education SDG 2015 - 2030

PISA Cycle No. of Participating Countries

2018 (&PISAD) 90

2021 110

2024 130

2027 150

2030 170

27

Post-project – use of PISA-D instruments in main PISA from 2021 onwards

OECD

Thank You

Michael Ward michael.ward@oecd.doc