Status of System-level data in PISA for Development participating countries

Said Voffal Friday 1 April 2016

UNESCO Institute for Statistics

v3

System level indicators in PISA

• Robust system-level indicators used to interpret policy context in relation to assessment results, used in global and country reports

• Data collection includes 9 tables in Excel workbook • Covering thematic areas:

– Education pathways/stratification – Assessments and public examinations in lower, upper

secondary, and entry exams to tertiary – Instruction time – Teacher salaries, training and recruitment – National accounts, Population and Educational

expenditures – Enrolment with coverage adjusted to education

expenditure

Worksheet 1: Education pathways/Stratification

• Age at which students experience the first selection in the education system

• List of type of schools or educational programmes availabe for 15-years old students

• Classification of programmes by ISCED 2011 levels

• Entrance ages and durations for pre-primary to secondary and programmes

Worksheet 2-1: Assessment and examination at secondary

• Existence and purpose or uses of national examinations and other standardised examinations.

The worksheet collects data on how exams are conceived and graded, as well as how results from these examinations are used and shared with other actors in the community

Worksheet 2-2: Tertiary education exams

• This sheet, similar to the previous one, collects data on examinations (not administered by upper secondary schools) to gain access to tertiary education. Additionally, it collects data on the influence of

factors other than examinations used by tertiary institutions to determine access to the first stage of tertiary education. Lastly, it collects data on alternative routes to access tertiary education.

Worksheet 3: Instruction time

• This questionnaire collects data on the amount of class sessions or teaching time dedicated to the intended curriculum (both compulsory and non-compulsory) in public institutions over a school year, for students aged 5 to 15, by ISCED 2011 level.

Worksheet 4: Annual statutory teacher’s salaries

• This worksheet collects data on annual statutory teachers' salaries by the teacher’s level of experience for ISCED levels 2 and 3 (lower and upper secondary) in public institutions.

Worksheet 5: Teachers’s pre-service training

• This worksheet collects data on the duration of teacher training programmes, the ISCED classification of the final qualification, and the percentage of current teacher stock with this type of qualification.

• It also collects data on competitive

examinations, teachers’ credentials, teaching practice, teacher registration and continuing education requirements for teachers.

Woksheet 6: National accounts and population data

• GDP

• PPP conversion factors

• Population data

Worksheet 7-1: Educational expenditure

• This worksheet collects data on total current and capital expenditure by type of institution and ISCED 2011 level .

Worksheet 7-2: Number of students adjsuted to expenditure data

• This worksheet collects data on the number of full-time equivalent students at each ISCED level, by type of institution, with the specification that data should be aligned to the same 12-month period as the financial year, as reported in worksheet 7-1.

How System-level data are gathered for OECD countries

• Most of the data are derived from the joint UNESCO-OECD-Eurostat (UOE) annual data collection

• Also from INES OECD network NESLI

Policy themes and UIS data collections

• Education pathways/stratification (ISCED mappings)

• Assessments and public examinations in lower secondary (OLO Catalogue)

• Instruction time (UIS questionnaire D)

• Teacher salaries and training (UIS questionnaire T)

• Educational expenditures (UIS questionnaire B)

• Enrolment (UIS questionnaires A and C)

Purpose of the assignment to UIS

• Apply system-level indicators framework in PISA for Development countries

• Consult national authorities in order to assess:

– Availability of system-level indicators

– Quality of system-level indicators

• Identify data gaps and make recommendations for improving data coverage and quality

• Most of the items are part of ongoing or new UIS data collection activities

Timetable for the assignment

• Start of the work: February 2015

• Countries covered by the study: Ecuador, Cambodia (desk research), Guatemala, Paraguay, Senegal and Zambia

• Site visits in countries from early March to end of May 2015

• Global report completion from mid-end 2015

Organization of the site visits

1. The National Project Manager(NPM) and the UIS agreed on a date for the site visit

2. Drawing upon documentation provided by the UIS, the NPM contacted national actors to schedule meetings for site visit

3. During the site visit, the NPM and the UIS met with different actors, collected data, assessed availability and quality for reporting

Using an assessment matrix

• Why? – To more systematically assess data coverage and

quality

– To target key improvements/interventions

– To enable use of existing international frameworks (e.g., DQAF, SABER)

• How? – The matrix is composed of broad categories and sub-

categories

– Assessment “rating” for each individual data source • latent, emerging, advanced

Evaluation matrix - quality

• 3 categories, 5 sub- categories:

– Coverage

• Statistical units covered

– Timeliness

• Periodicity of production

• Timeliness (gap between events and publication)

– Ownership of the information

• Legal framework

• Responsibility of teams

Matrix used to evaluate data quality

Coverage Time sensitiveness Ownership of the information

Statistical units covered Periodicity of production

Timeliness Framework for action

Responsibility of teams

Description

The scope of education statistics is broader than and not limited to a small number of indicators, nor to some sectors of the education system (e.g. general programmes, public sector, etc.)

The production of reports and other outputs from the data warehouse occur in accordance with cycles in the education system.

Final documents, statistics and financial data are disseminated in a timely manner.

Defining, collecting and managing information is an integral part of the educational system and the government.

Responsibility for collecting , processing, and disseminating educational information is given to a clearly designated institution

Latent

General programmes in the public sector are covered, for primary and secondary

The system produces data and statistics neither periodically.

The system does not produce information, data and statistics in a timely manner.

There is no framework in place

Specific teams are not identified

Emerging

Initial education system is covered, as well as the technical and professional programmes, in both public and private sector.

The system produces some data and statistics periodically.

The system produces some information, data and statistics in a timely manner.

Basic components of a framework or informal mechanisms are in place

Some specific teams are identified, but no platform for collaboration

Advanced

All the sectors of the education system are covered, including second chance programmes and literacy.

The system produces all data and statistics periodically.

The system produces all information, data and statistics in a timely manner.

Most elements of a framework are in place

Specific teams are identified, collaborative platforms are in place

Evaluation matrix – data availability

• 3 categories, 4 sub- categories:

– To the internal user / officer of the ministry

• Is the information available at the ministry?

• Is the ministry using the information?

– To the external user

• Can the information be found on the web or through public platform?

– To the international organisations

• Has information been transmitted officially to the UIS or other UN entity through its regular activities?

Matrix used to assess data availability

Inside officers Outside users International organisations

Description

Officers are aware of information available, and know how to access it (i.e. institutional/organizational structure).

Information- and Data-driven culture: information and data are disseminated and used for policy-making.

Users outside of the ministry have access to the information, either online, or through public-access platforms (phone, documentation centre, etc.).

Information has been shared with international organisations (e.g. the UIS) through its regular activities.

Latent

Officers are more or less aware of the information available, and rely on personal connections to find it.

There are no mechanisms to disseminate the documents or results, in order to improve the system.

Information is available through personal contact.

No data has recently been shared with the UIS/UN and to other international partners.

Emerging

There are some official channels through which national officers can access the information.

There are some mechanisms in place in order to make sure the documents or results are used in order to improve the system.

Some information is publicly available, either online, at documentation centre, or other ways.

Some information in some areas is sent to the UIS/UN, some gaps in data and timeline.

Advanced

Information is organized in a coherent and accessible manner, for example through a specific office or data warehouse.

There is a communication strategy to make sure the documents or results are disseminated and used in order to improve the system.

All the information is organized in a coherent manner and available in a single place (e.g. website, information centre, etc.).

Relevant information is regularly shared with the UIS/UN through its regular activities, and then relayed to other partner agencies.

Overall results of the assessment by theme

Table in questionnaire Latin America Sub-Saharan Africa Asia

Ecuador Guatemala Paraguay Senegal Zambia Cambodia

1 Education stratification

2-1 Assessments and

examinations

2-2 Tertiary entrance

examinations n/a n/a n/a

3 Number of class sessions per

year

4 Annual teacher salaries

5 Requirements for pre-service

teacher training

6 National accounts

7-1 Education expenditure

7-2 Number of students

Green=Advanced; Yellow=Emerging; Red=Latent ; White=partially or not evaluated; n/a=Not applicable

Results assessment for Cambodia

• Cambodia was not visited by UIS because its participation to the project has not yet been confirmed during the time of UIS visits to countries

• Only desk research has been conducted by the UIS. So country’ assessment should be considered partial.

Results assessment for Cambodia

• The institution in charge of implementing PISA-D is the Quality Assurance Department of the Ministry of Education, Youth and Sports.

• Based on data reported by Cambodia to UIS, the country is able to provide good quality data for worksheets 1, 2 and 3.

Challenges for Cambodia

• Education expenditure data: these are regularly reported to UIS through its education survey, but coverage is partial since they do not include household expenditure on education.

Results assessment for Ecuador

• Ecuador is in a very good position to respond to the system-level questionnaire. Each data table or worksheet in the questionnaire has an institution assigned to it to collect and/or manage the requested information.

• National Institute for Education Assessment (INEVAL)-PISA’ NPM- has a good level of institutional collaboration with national data producing institutions

Ecuador: challenges

• Educational expenditure:

Lack of data on household expenditure on

education

Lack of data on the number students aligned to expenditure data . But INEVAL and the Ministry of Education’s Co-ordination of Planning office have committed to work on this issue in order to provide the correct data in the system-level questionnaire

Assessment results for Guatemala

• Guatemala is in a good position to respond to the system-level questionnaire. Each data table in the questionnaire has an institution assigned to it to collect and/or manage the requested information.

• The General Directorate of Assessment and Education Research (DIGEDUCA), PISA’ NPM, is a public institution belonging to the Ministry of Education.

Challenges for Guatemala

• Instruction time: the legislation on duration of instruction sessions is not specific. An agreement within MoE offices is necessary to compile the data.

• Pre-primary education: inconsistencies in the entrance age and duration between ISCED 97 and ISCED 2011

• Expenditure data: lack of data on household expenditutre on education

Assessment results for Paraguay

• Paraguay is in very good condition to respond to the system-level questionnaire. Paraguay used to be part of the World Education Indicators (WEI) project.

• The General Directorate of Educational Planning, the PISA NPM, sits within MoE and is the institution responsible for Paraguay’s EMIS

Challenges for Paraguay

• Lack of data on household expenditure on education: the country needs to derive these data from existing household surveys

• Lack of centralized or unified entrance system to tertiary education

• Mismatch between enrolment and population data which results in inconsistent enrolment rates for pre-primary and primary education

Assessment results for Senegal

• Senegal is in a satisfactory condition to respond to the system-level questionnaire.

• The institution in charge of implementing PISA-D in Senegal is the National Study and Action Institute for Education Development (INEADE) within the MoE.

Challenges for Senegal

• The structure of Senegal’s education sector, and consequently its information systems, means that different ministries are responsible for different levels of education or particular orientations.

• Given this structure, it is clearly a challenge to ensure coordination among all the relevant institutions.

Challenges for Senegal(ctd)

• Timeliness issues in the availability of data on national assessments and examinations: due to delays in the publication of the results

• Instruction time: data are difficult to access and there is no proof that regulations are applied at school level

• Data on teachers’ salaries: difficult to identify the specific base salaries and to be sure of their consistency

Challenges for Senegal(ctd)

• The involvement of different institutions also affects the data on pre-service teacher training, entry into the teaching profession and professional development training (Worksheet 5)

• Education expenditure data: this is also a challenge for Senegal but there is currently a UNESCO (UIS and IIEP) project aiming at reinforcing the capacity of the country to produce, use and report expenditure data

Assesment results for Zambia

• Zambia has satisfactory capacity to produce the statistics requested through the system-level questionnaire.

• The country has established mechanisms for conducting high-stakes national examinations and national assessments, as well as implementing international student learning assessments.

Assesment results for Zambia

• Legislative instruments are in place to guide the education system’s implementation, to regulate its human resources, and to authorise the activities of various data-producing entities.

• The institution in charge of implementing PISA-D is the Examinations Council of Zambia, of the Ministry of Education, Science, Vocational Training and Early Education.

Challenges for Zambia

• Education expenditure: lack of data on expenditure in private institutions. Only data for insitutions managed by MoE

• Lack of timeliness in the publication of examination and assessment results on online government platforms, and of dissemination of annual education statistics bulletins in the public domain.

Challenges for Zambia(ctd)

• Another notable issue is a mismatch between primary education enrolment data and population data by age based on 2010 Population Census

Data producers should consider reviewing the quality of both sources of data to resolve the concerns around indicators that rely on age-specific data.

Conclusions and way forward

Overall remark:

• Despite the overall relatively good rating, not all the participating countries have the same ability to respond to the system-level questionnaire, just as they have different challenges in responding to the UIS annual education survey.

Conclusions and way forward

• Latin American countries taking part in PISA-D are in a strong position to provide the required data

• Based on data reported to UIS, Cambodia is in good position to provide System-level data

• Senegal and Zambia are also in good positions but face particular challenges in terms of the necessary sectorial co-ordination to provide the required data

Lessons learnt from PISA-D

• Most difficult common issue: provision of data on household expenditure on education

• Discrepancies between policies and their implementation: lack of control or disciplinary mechanisms in many countries to ensure that practice is kept up to national regulations (ex: regulations on instruction time in Senegal)

Lessons learnt from PISA-D

• Finally, and perhaps the most difficult challenge to tackle, is remodelling a country’s data infrastructure to make sure that data are available in one single hub

• Ensure inter-sectorial coordination between the national involved data producing institutions

Lessons learnt from PISA-D

• To avoid putting unnecessary burden on countries one should consider some modifications of the PISA-D questionnaire to avoid soliciting data and metadata that are already available at the international level, for example through the regular UIS activities.

Many thanks!

• For any question, please contact :

Said Voffal, from UIS at: s.voffal@unesco.org

LLECE and E2030:

an Instrument to Improve

Quality of Education

PISA for Development

International Advisory Group

3rd Meeting

Asuncion, Paraguay, 30 March - 1 April, 2016

• Coverage vs quality

• Quality is a multidimensional concept

• Learning is crucial

• Towards a more precise definition of education quality

• A dynamic concept that changes and evolves with time and changes

Education quality, the pending task in

the region

40.0

46.9

18.4

39.1

35.9

51.5

15.2

12.1

16.5

5.7

5.1

13.7

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Ciencias naturales

Matemática

Lectura

% of students in each performance level

Source: Full report: Learning achievements, TERCE (2015)

█ Level I

█ Level II

█ Level III

█ Level IV

The crisis of learning affects the most vulnerable

• Standardized tests on learning achievements

• PISA: Rate of students with low achievement in the different subject areas is

2 to 3 times higher than the OECD average

Reading

Mathematics

Natural

Sciences

• Goal 4: Ensure inclusive and equitable quality education and

promote life-long learning opportunities for all

Goal 4.1 Quality primary and secondary education leading to

relevant and effective learning outcomes

• LIMA Declaration: Evaluation of learning achievement for the

dimensions of quality

• Incheon Declaration: UNESCO will continue to lead and

coordinate SDG4 – Education 2030

• UNESCO General Conference: UNESCO’s role in the

implementation of the E2030 Agenda: “inclusion of the

regional mechanisms on assessment of learning, as part of

the E2030 follow-up instruments.”

• The learning process is a great protagonist in this new agenda

with teachers as its main driver

Education 2030, evaluation and learning

What is the LLECE?

• The Latin American Laboratory for Assessment of the Quality of Education (LLECE) was founded in 1994, in Mexico City

• This consists of a network of national-level Directors of Educational Assessment in LAC, coordinated by OREALC/UNESCO Santiago

• New approaches on education quality and evaluation.

• Information about learning achievements and associated factors.

• Towards a culture of evaluation: Strengthening of capacities of the local evaluation units.

• Promote change in education to improve the learning process in schools.

• Alliances and coalitions/South-South Cooperation

LLECE: instrument to improve education quality

LLECE participants

• 19 education systems:

18 countries, plus the

Mexican state Nuevo

León.

• 2015: Cuba, El Salvador,

Venezuela and Bolivia

• Caribbean countries: next

steps

LLECE’s studies: who and what is assessed?

Study Year Grades Maths Science Reading Writing

PERCE 1997 3 y 4 x x

SERCE 2006 3 y 6 x x option

al 6°

x x

TERCE 2013 3 y 6 x x obligatory 6°

x x

LLECE has conducted 3 studies:

• Maximize the use of TERCE's results to change

and improve education

• Understanding of education phenomena to

generate change in policies and practices.

• Less attention to rankings: indicators

interpreted outside their cultural and structural

context lead to misinterpretations and wrong

policies

TERCE: the region’s most representative

evaluation of learning outcomes in primary

education

TERCE Reports

Initial Background Information

Learning Achievements Executive report

Full report

Associated Factors Executive report

Full report

National reports: Contextualizing the results

• Analyze priority policies and identify intervention areas

• Promote the use of data in public and academic spheres

• Heterogeneity of results requires in-depth studies focusing on TERCE findings

Argentina Pending

Brazil Pending

Chile OK

Colombia Pending

Costa Rica OK

Ecuador* OK

Guatemala Pending

Honduras OK

Mexico Pending

Nicaragua* OK

Panama* OK

Paraguay OK

Peru Pending

Dominican Republic In process

Uruguay Pending

*only event/report pending

Thematic reports

Topic Technical/Financial

partner

ICTs,

Infrastructure

Inequality

IDB

Teachers

UNESCO

Strategy on

Teachers

Indigenous

students

UNESCO HQ

Gender UNESCO HQ

o School

environment

o Pre-primary

education

UNICEF TACRO

(TBC)

Contributions to teaching: Reading, Writing,

Mathematics and Natural Sciences

• Curricular Analysis of TERCE: domains and teaching approaches

• Teacher training programs

Countries

Country

total Percentage

USA 285 25,6%

Chile 186 16,7%

Great Britain 105 9,4%

Brazil 82 7,4%

Argentina 59 5,3%

Others 51 4,6%

Uruguay 49 4,4%

Paraguay 42 3,8%

Mexico 37 3,3%

Costa Rica 36 3,2%

Colombia 32 2,9%

Peru 29 2,6%

Guatemala 26 2,3%

Dominican

Republic 26 2,3%

Ecuador 25 2,2%

Panama 25 2,2%

Spain 17 1,5%

TOTAL 1112 100,0%

Downloads of TERCE BDD: International level

4% 24%

1% 11%

8%

52%

Consultants andresearchers

Public Institutions

InternationalAngencies

Mass media

Research Institutes

Universites

4% 12%

33% 21%

10%

7%

4% 9%

Multilevel analysis

National Analysis

Comparative analysis

Descriptive analysis

Associated Factors

Research

Journalism

Thesis

• Policy recommendations based on TERCE

• Recommendations of educational practices at

school level based on TERCE

• Course: Evaluation of education quality and

use of related data to influence decision-making in education polices and practices

• Evaluation units and Planning units of

Ministries of Education

Evaluation for planning purposes

SERCE

2,23 %

17,82 %

41,78 %

28,48 %

9,69 %

1,33%

13,48%

TERCE in sight

• Comunications product

directed at all agents involved

in education, particularly

decision-makers.

• Its aim is to deliver elements

for analysis about a specific

topic which is part of the

findings of the Third Regional

Comparative and Explanatory

Study (TERCE), and allow to

lead directions relating

education policies in the region.

Functioning structure of LLECE

Politically: at ministry level

Take use of existing regional mechanisms and bring

together decision-makers in education, such as:

OAS (Institutional agreement) MERCOSUR education sector (rotative)

CECC/SICA (rotative)

CARICOM (rotative)

Technically: at national coordinators

level

Offer more technical support to national coordinators.

Widen its field of action in the implementation of

the studies.

Include one meeting of national coordinators per

year.

LLECE’s studies: relation with

other international evaluations

PISA PISA-D PIACC

TERCE MIL

PIRLS TIMSS ICCS

AMÉRICA LATINA

• Inter-agency coordination for

the application of

evaluations.

• Review the study’s reach in

the region.

• Taking use of already

existing resorts.

• Complement the studies’

results.

COMMON DENOMINATOR

The future of LLECE: Agreements

2016 -2020

• Continue with the use and dissemination strategy

of TERCE results

• Design and develop LLECE’s Fourth Regional

Comparative and Explanatory Study (ERCE)

• Research for the Education 2030 Agenda:

education quality in an integral sense

• Capacity building and improvement trainings

that contribute to the strengthening of technical

measurement and education evaluation teams

CONCLUSIONS

• LLECE

A milestone in development cooperation in education

between the countries of the region.

High quality evaluation studies, which are

representative and adapted to the social, cultural

and economic context of the region.

The Laboratory is a key piece for monitoring E2030,

according to the Agenda’s Framework for Action.

• TERCE results must help influence decision-making, in

national contexts, concerning policies and practices in

education. Therefore, dissemination is a must.

• Participation of countries within LLECE is fundamental, in

order for E2030 to pervade all levels and reach all the agents

involved in education.

Thank you!

66

UNESCO

Regional Bureau of Education for Latin America and the Caribbean

National Office to Chile

Atilio Pizarro: a.pizarro@unesco.org

3rd meeting of the International Advisory Group Granados Park Hotel – Asunción, Paraguay

30 March – 1 April 2016

PISA for Development

ToR for an Independent Review

PISA-D Independent Review

Scope and purpose of the review

• Focused on the progress of the project in relation to its five main outputs and extent to which the delivery of these will achieve the project’s purpose

• Purpose of review is to help understand what has been achieved against the OECD’s original plans, how practicable those plans were as well as how relevant and valuable the project’s work is to developing countries’ and development partners' evolving education policies

Project’s outputs

• Contextual questionnaires and data-collection instruments enhanced;

• Descriptive power of cognitive assessments enhanced

• Analytical framework and methodological approach for including out-of-school 15-year-olds in assessments developed;

• Country capacity in assessment and analysis strengthened

• Engagement established with developing countries and partners for peer-to-peer analysis and learning opportunities to support the UN-led post-2015 process

Main issues to be reviewed

• Impact of the project

• Relevance of the project

• Sustainability of project achievements

• Management and partnership arrangements

Impact of the project

• Achievement of outputs and the purpose of the

project

• Expected impact on policy makers and the education systems in the participating countries

• Evidence that PISA results will be used in policy-making and/or sector work

• Dissemination and use of project deliverables

• Enhancement of main PISA

Relevance of the project

• Key lessons from the project to inform work

on improving education quality and improved student learning outcomes

• The role of the project in informing discussions of education quality and learning outcomes

• Promotion of evidence-based policy making

Sustainability of project achievements

• Achievement of capacity building outputs and

objectives

• Lessons from capacity building

• Success of peer-to-peer learning strategies

• Likely transition of countries from PISA for Development to main PISA

• Spill-over benefits of the project for student assessment as a whole

• Resource implications of implementation

Management and partnership arrangements

• Effectiveness of the governance and

management structures for the project

• Effectiveness of project management systems and processes

• Roles of PISA GB, the DAC and IAG and TAG

• Particular successes and challenges in implementing the project

Methodology

• Selection of experts

• Experts to propose a design, plan and methodology in accordance with ToR

• Collection and analysis of documents, data and information, interviews with stakeholders and review of documents produced by the project

• Possible use of surveys

Schedule

Inception Report by Early January 2019

Draft report by End February 2019

Presentation of final draft report to IAG by End March 2019

Final Report End April 2019

Deliverables

• Presentation to IAG in March 2019 of final draft report

• Final report of 50 pages

• IAG and TAG will have opportunity to comment on Inception report (10 pages) and Draft report (20 pages)

• Final report will inform OECD’s final report on the results of the project

Logistics and resources

• Review team will be responsible for own logistics (office space, administrative and secretarial support, telecommunications, printing of documentation, etc.) and for disseminating all methodological tools such as surveys, while the OECD team will facilitate this process to the extent possible by providing contact information

• An allocation of EUR 35 000 will be made for this activity within the project’s budget

IAG Decision

• The IAG is invited to re-affirm its approval of the ToR for an independent review of the project and to confirm that the revised timeline is acceptable.

3rd meeting of the International Advisory Group Granados Park Hotel – Asunción, Paraguay

30 March – 1 April 2016

PISA for Development

PISA Development Strategy

Three main issues driving change in PISA participation:

• OECD’s overall strategy on development – making OECD policy instruments more relevant to developing countries

• Growth in participation in PISA by middle income countries and demand for additional inputs and support.

• Likely inclusion of PISA in the Education SDG indicators framework that will be used to monitor progress towards a universal learning goal by 2030.

Pressure on OECD to respond quickly to these drivers, necessitates the development of a strategy

PISA Development Strategy - Rationale

Four main components:

• PISA for Development, designed to enhance PISA’s instruments to make these more relevant to a wider range of countries.

• Building on PISA for Development’s achievements, scaling-up activities and partnership options for participation in PISA by middle income and low income countries.

• Resource management arrangements for successfully implementing the strategy.

• Arrangements and activities for managing the risks associated with the strategy, in particular the impact of greatly increasing the number of participants in PISA on the PISA governance and management structures.

PISA Development Strategy

• Phase I “Design, planning and co-ordination” completed

• All participating countries prepared for implementation:

– Capacity Needs Analysis reports, Capacity Building Plans, and Project Implementation Plans completed or being completed

• Six expert papers published

• Phase II “Technical development, including review of instruments, design of enhancements, preparation of materials and planning for field trials” well under way and on track

• Confident that the project will be a success and field-trialled instruments will be available for inclusion in PISA 2021 cycle

Component 1: PISA for Development – on track

Component 2: building on PISA-D’s achievements

• PISA-D is already proving successful and looks likely to deliver an assessment that is more relevant to developing countries in time for including in PISA 2021 cycle

• Some other PISA-D outputs can be made available immediately to countries participating in PISA 2018 and PISA 2021

• PISA-D outputs and activities also provide vehicles for outreach to potential participating countries

Component 2: Three participation options

• Tier 1: Participation in future cycles of PISA;

• Tier 2: Participation in future cycles of PISA plus support for engagement and communication, analysis of results and dissemination of results and policy dialogue

• Tier 3: Participation in future cycles of PISA plus support for survey implementation, plus support for engagement and communication, analysis of results and dissemination of results and policy dialogue

Component 2: Tier Two

Participation in PISA, plus OECD support for:

Policy dialogue

Dissemination of results

Preparation of a country

report

Country-specific

data analysis

Preparation of in-country stakeholders

Component 2: Tier Three Participation in PISA, plus T2 plus OECD support for:

Technical assistance

Project implementation

plan

Capacity building plan

Capacity needs analysis

Component 2 – web-based tools for Tier Three

Web-based tool for country capacity needs analysis, capacity building planning and project implementation planning that PISA for Development has developed with the help of a consultant, Fernando Cartwright.

• URL: www.polymetrika.org/PISAD/

• Account name: PISASDG

• Password: 2isaprime#

Component 3: Resource management arrangements

• Tier 2: Fee for standard PISA partner participation fee plus EUR 250,000

• Tier 3: Fee for standard PISA partner participation fee plus EUR 400,000

• Need at least five countries to fund the requisite posts and inputs

• Adequate staff-to-country ratio would be in the region of one analyst for every 5-7 Tier Two and Tier Three countries

Component 4: managing the risks associated with the strategy

• Capacity challenges and the national and international costs involved will make it necessary for developing country participation in PISA to be supported by development partners, at least initially;

• Future PISA waves moving away from paper-and-pencil instruments – a challenge that seems to be manageable for many middle income and low income countries as digital technologies become widespread

Component 4: managing the risks associated with the strategy

• Managing over 100 countries participating in PISA – larger venues for meetings, more efficient meeting management, limiting the number of delegates attending PGB meetings

• May also require holding NPM meetings in regional batches – this would have implications for the transaction costs of the OECD and contractors, but is manageable with the additional resources that higher level of participation would bring

Component 4: Managing the risks associated with the strategy

• It will also be necessary to examine further any consequences of increased participation in PISA for the broader PISA system and the governance of PISA at policy and technical levels.

• But all of these risks are manageable, given an adequate level of resourcing based on increased participation

Education SDG 2015 - 2030

PISA Cycle No. of Participating Countries

2018 (&PISAD) 90

2021 110

2024 130

2027 150

2030 170

93

Post-project – use of PISA-D instruments in main PISA from 2021 onwards

Discussion • Are there any omissions in respect of the PISA

Development Strategy?

• How could the Secretariat’s strategy for outreach be improved?

• What should be the role of development partners and other partners in implementing the Strategy?

• How should countries already participating in PISA support new participants in the assessment?

Marien Peggy Martínez Stark Directora

Organización de Estados Iberoamericanos Oficina Paraguay

La Organización de Estados

Iberoamericanos para la Educación, la Ciencia y la Cultura (OEI)

es un organismo internacional de cooperación entre los países de la comunidad iberoamericana en los ámbitos de educación, cultura y

ciencia y tecnología.

Su labor se guía por las Metas Educativas Iberoamericanas 2021, cuyo fin es mejorar la calidad de la educación en la región para hacer frente a la pobreza y la desigualdad.

Ámbitos de acción:

Educación Ciencia Cultura

Países miembros:

• Conferencias Iberoamericanas de Ministros de Educación y Congresos Educativos

• Publicaciones

• Alianzas y asociaciones estratégicas

• Relaciones con otras regiones y países del mundo

Desarrollos Institucionales

Metas Educativas 2021

Origen y sentido: surge en el marco de la conmemoración de los Bicentenarios de las independencias

Aprobado el documento en la XX Conferencia de Ministros de Educación (Buenos Aires, 13 de septiembre, 2010); refrendado en la Cumbre de Jefes de Estado y de Gobierno (Mar del Plata, diciembre 2010).

Objetivos: mejorar la calidad y la equidad en la educación para

hacer frente a la pobreza y a la desigualdad y, de esta forma, favorecer la inclusión social.

Marco de las Metas Educativas 2021

Meta general quinta

Meta general novena

Meta general decimoprimera

Mejorar la calidad de la educación y el currículo escolar.

Ampliar el espacio iberoamericano del conocimiento y fortalecer la investigación científica.

Evaluar el funcionamiento de los sistemas educativos y del proyecto Metas Educativas 2021.

Relación “Educación 2030” y “Metas Educativas 2021”

Ambos documentos abordan con decisión retos aún no resueltos: analfabetismo, abandono escolar temprano, trabajo infantil, bajo rendimiento de los alumnos y escasa calidad de la oferta educativa pública. Además, enfrentándose a las demandas exigentes de la sociedad de la información y del conocimiento: incorporación de las TIC en la enseñanza y en el aprendizaje, apuesta por la innovación y la creatividad, desarrollo de la investigación y del progreso científico.

Área estratégica: Aseguramiento de la calidad de la Educación. Programa: Sistema de Evaluación para el aseguramiento de la calidad con base en una política pública de evaluación integral. Iniciativas Estratégicas: • Evaluación del desempeño de docentes, directores y supervisores. • Evaluación Institucional en establecimientos de EEB, EM e IFD. • Evaluación estandarizada de aprendizajes de estudiantes.

Agenda Educativa 2013-2018 del Ministerio de Educación y Cultura

Agenda Educativa 2013-2018 del Ministerio de Educación y Cultura

Evaluación del desempeño de docentes, directores y supervisores. Líneas de acción:

• Ajustes de los estándares e implementación del mecanismo de Evaluación de desempeño de los docentes y supervisores educativos.

• Definición de estándares, diseño e implementación del mecanismo de desempeño de directores.

Agenda Educativa 2013-2018 del Ministerio de Educación y Cultura

Evaluación Institucional en establecimientos de EEB, EM e IFD. Líneas de acción: • Aplicación del mecanismo de licenciamiento en IFD de

gestión oficial y privada. • Diseño e implementación del mecanismo de evaluación

institucional de establecimientos educativos de EEB y Media.

Agenda Educativa 2013-2018 del Ministerio de Educación y Cultura

Evaluación estandarizada de aprendizajes de estudiantes. Líneas de acción: • Definición de estándares para la evaluación. • Aplicación de evaluaciones censales de fin de ciclo en

Educación Escolar Básica y en Educación Media. • Divulgación de los resultados de la evaluación. • Fomento del uso y aprovechamiento de los resultados

evaluativos para la toma de decisiones educativas.

Cooperación de la OEI con el Ministerio de Educación y Cultura

Acciones preparatorias para la creación del Instituto Nacional de Evaluación Educativa: Diseño y desarrollo de una Maestría en Evaluación de la Calidad Educativa. Participan 20 funcionarios del MEC. De los cuales 6 han sido seleccionados para la acciones de PISA en las siguientes funciones : • Elaboración de cuestionarios de contexto. • Elaboración de items • Procesamiento de datos

Cooperación de la OEI con el Ministerio de Educación y Cultura

Acciones preparatorias para la creación del Instituto Nacional de Evaluación Educativa: Organización del Encuentro Internacional “Realidades y Proyecciones de los Institutos de Evaluación de la Calidad de la Educación de América Latina. Participaron técnicos y especialistas internacionales de Uruguay, México, Colombia, Chile, Cuba, Brasil y Paraguay.

Cooperación y soporte de la OEI al Ministerio de Educación y Cultura

en temas de Evaluación de la calidad Educativa

Gestión administrativa de fondos para la aplicación de las pruebas :TERCE y evaluación censal de los aprendizajes. Contratación de expertos internacionales para la elaboración del sistema nacional de evaluación de la carrera docente.

Cooperación de la OEI – PISA ¿Qué puede aportar la OEI a Pisa?

• Infraestructura para las capacitaciones. • Plataforma webex para la realización de:

seminarios cortos vía Web, • Reuniones técnicas vía Web entre países, • Consultas a expertos en evaluación e

intercambio de experiencias en el proceso de desarrollo de las fases en la implementación de PISA.

Cooperación de la OEI – PISA ¿Qué puede aportar la OEI a Pisa?

• Red de expertos y especialistas en Evaluación de la Calidad Educativa que son miembros del equipo de la Maestría.

• La Maestría de la Calidad Educativa que aporta formación en tres ejes: Evaluación de la calidad de los aprendizajes Evaluación Institucional Evaluación del ejercicio docente.

• Publicaciones • Alianzas y asociaciones estratégicas • Relaciones con otras regiones y países del mundo.

3rd meeting of the International Advisory Group Granados Park Hotel – Asunción, Paraguay

30 March – 1 April 2016

PISA for Development Conclusions & next steps

Strong representation at the meeting from:

• Countries signed up (Cambodia, Ecuador, Guatemala, Honduras, Paraguay, Senegal, Zambia) or committed (Panama) to participation in PISA for Development

• Development partners (France, Germany (BMZ/GIZ), IDB, Japan(JICA), Norway, UK (DFID), World Bank).

• International agencies (UNESCO, UIS, UNICEF, LLECE)

• International contractors.

Conclusions Day One

Conclusions Day One

• Two co-chairs elected:

– one representative of the participating countries: Rodolfo Elias (Paraguay)

– one representative of the development partners: Maria Soledad Bos (IDB)

• Annual report on project activities reviewed and approved with suggestions for improving presentation of project progress and highlighting risks (particularly related to risk 4 on project impact), and praised the substantial progress that has been made

• IAG members noted the new development partners and discussed the role of these within the IAG

• IAG members also noted the challenge of countries for financing their participation in international meetings and asked OECD to continue reporting on this

Conclusions Day One

• OECD confirmed that no more countries will join PISA-D

• IAG members noted that Strand C activities are on track despite there not yet being a signed contract with ETS and encouraged OECD to sign the contract as soon as possible

• Members noted the presentations by ETS on Strand A and commended the countries, contractors and Secretariat for the progress that has been made and the plans for 2016 and beyond

• IAG members discussed issues related to comparability in the context of PISA-D having no major domain, computer-based vs. paper-based, and trend links

Conclusions Day One • Members noted the presentations by TLB on Strand B and

commended the countries, contractors and Secretariat for the progress that has been made and the plans for 2016 and beyond

• There was a discussion of how the framework for the questionnaires are reflected in the items selected for the instruments

• Members noted the presentation by ETS on Strand C and commended the countries, contractors and Secretariat for the progress that has been made in preparing this component

• There was a discussion of the timeline for Strand C and countries noted the short window of time available for the field trial data collection

Conclusions Day Two: Capacity building

• The IAG noted the actions taken in support of capacity building, appreciated the presentations by the participating countries in this regard and noted the high value of this component of the PISA-D project

• Participating countries highlighted their interest in building capacity for analysing and reporting on results, suggested possible additional training on quality control and framework development, and indicated a need for funding for peer-to-peer learning activities

• IDB confirmed support for capacity building workshops in the Latin America region, which will be conducted in Spanish and cover item development, item response theory, communications, and secondary analysis. IDB also confirmed its support for peer-to-peer learning in Latin America.

Conclusions Day Two: Capacity building

• IAG noted the progress with establishing peer-to-peer learning partnerships and appreciated the commitment to the project by the peer countries from main PISA

• IAG noted the need to consider the software used for processing and analysing PISA-D data and to avoid the use of Proprietary Software wherever possible to avoid constraining countries’ options for analysis of assessment data

• IAG discussed how the OECD and the contractors are tracking the achievement of capacity building objectives and managing the risks associated with insufficient capacity for implementing PISA. IAG requested the OECD to continue to monitor this risk.

Conclusions Day Two: Engagement & communication

• IAG appreciated Kosovo’s presentation on its experience with engaging stakeholders and preparing these for PISA national reports and the dissemination of PISA results

• IAG also appreciated GIZ’s presentation of its capacity development work with Kosovo and noted the importance from this experience of targeting key stakeholders, such as school principals, teachers and education journalists, for both communication and capacity building

• IAG approved the revised version of the PISA-D engagement and communication strategy, noting the actions taken to strengthen the linkages between the project’s strategy and the strategies of the countries

Conclusions Day Two: Engagement & communication

• IAG commended participating countries on their progress with stakeholder engagement and communication activities since project launches in the countries

• IAG commented that the engagement and communication strategies were focused on PISA and should be balanced with a broader concern for education quality issues and PISA’s role in addressing these

• IAG suggested that the engagement and communication strategies and products of the countries should be targeted at the right level and at the right people in policy making positions to stimulate the kind of policy dialogue envisaged

Conclusions Day Two: Analysis & reporting

• The IAG welcomed the presentations by OECD and The Learning Bar of the draft analysis and reporting plan, including the structure and content for the country reports and the proposed reporting products

• The IAG reviewed and discussed the analysis and reporting plan and encouraged the development of further ideas for the series of PISA-D in-Focus publications

• Other comments on the plan included: sharpening the presentation of the educational prosperity model to make it clearer how this serves as an effective analytical framework for PISA-D; providing a stronger evidence base for the educational prosperity model; including an honest and realistic appraisal of the limitations of the PISA-D data set; setting out more clearly what the reports will be able to say about quality instruction and the benchmarks that will be used to assess this.

• The Secretariat outlined the timeline for finalising the analysis and reporting plan, including the deadline for receipt of written comments from the IAG (30th April 2016), the presentation of a second draft to the PISA-D TAG on 6 June; the finalisation of a third draft in Autumn 2016; and a final draft for the May 2017 IAG meeting, in light of field trial data analysis.

Conclusions Day Three

• IAG reviewed and discussed the UIS report on the status of system level data in the participating countries and noted the role that system level data plays in PISA and PISA-D

• Countries commented that the report is generally accurate but that there are some factual corrections to be made to some of the data included in the report – countries to send written feedback immediately following the IAG meeting to the Secretariat so that the report can be finalised

• UIS will finalise the report based on country feedback, and it will be published by mid-2016 and made available on the OECD website – the data will be transferred from UIS to OECD for use in the production of the national reports

Conclusions Day Three

• OECD confirmed that Panama will complete its own review of system-level data as part of its participation in PISA 2018

• OECD also confirmed that for Honduras, the analysis of this country’s system level data will likely be carried out by the expert currently working on their CNA/CBP/PIP. The OECD will also work with Cambodia to fill in the gaps in the analysis of their system level data.

• IAG appreciated and discussed UNESCO’s presentation on lessons from LLECE - TERCE related to engagement and communication with stakeholders and dissemination of results

Conclusions Day Three

• IAG reaffirmed its approval of the ToR for an independent review of the project subject to an increased budget for the review and including additional questions related to impact and sustainability

• IAG commented that the OECD should start planning for the data that the reviewers might like to draw on, such as data on the processes through which the project was implemented in the countries and how implementation played out.

• IAG confirmed that the revised timeline of conducting the review in 2019 is acceptable but suggested that the review begin in December 2018 as opposed to January 2019 so that the reviewers are on board for the launches of the national reports

Conclusions Day Three

• IAG discussed the OECD’s PISA Development Strategy and outreach to potential PISA participating countries and supported this, noting the following:

– Financing Tier 2 or 3 options will be a challenge for the countries that need this form of support most

– PISA-D countries will have valuable lessons to share with new countries that sign up to participate in future cycles of PISA, and it would be useful for them to form peer-to-peer partnerships with these

– OECD needs to continue engagement in the global Education SDG/2030 architecture to ensure complementarity with global monitoring priorities

– The PISA Development Strategy needs to maintain focus on country ownership and capacity building

– The Strategy also needs to take account of the changes in PISA (computer based assessment) and its implications for developing countries

Next steps

1. OECD to sign contract with international contractor (ETS) for Strand C by 30th April 2016

2. OECD to complete the report on system level data with UIS following receipt of feedback from the countries and to make this available on its website

3. OECD to sign participation agreement with Panama

4. OECD to complete the Capacity Needs Analysis and the design of Capacity Building Plans for Honduras and Panama

5. Honduras and Panama to finalise Project Implementation Plans

6. Participating countries to finalise outstanding agreements with development partners regarding contributions and support (e.g., international costs, in-country costs and activities)

Next steps (continued)

7. OECD to further develop and confirm peer-to-peer learning partnerships and implement the planned capacity development activities with the countries

8. OECD to finalise with development partners outstanding agreements for support to the project – general contributions and country-specific contributions

9. Third meeting of Technical Advisory Group on 6th June 2016

10. Project implementation: Strands A and B – Field Trial

– Third international/NPM meeting (4-8 April 2016)

– Fourth international/NPM meeting (4-8 July 2016)

11. Project implementation: Strand C – Technical Development and Field Trial

– Fifth international/NPM meeting (1-4 November 2016)

Next steps (continued)

12. IAG to send Secretariat comments on analysis and reporting plan by 30 April 2016; Secretariat to discuss second draft of plan with TAG on 6 June 2016; third draft for IAG review in Autumn 2016; final draft for IAG meeting in May 2017

13. OECD Secretariat with International Contractors to complete translation of key materials, such as manuals and frameworks, into French and Spanish

14. OECD to implement with participating countries the engagement and communication strategy for the project and country strategies

15. Fourth meeting of IAG from 17-19 May 2017 in Cambodia.