EDUCATION STAFF DEVELOPMENT PROGRAM

Assessing School Autonomy and Accountability (SAA) Policies

An Overview of SABER-SAA

Angela Demas Senior Education Specialist

The World Bank November 2014

Overview of presentation

1. Introduce the SABER-SAA Tool for analyzing

policy intent and the enabling environment

2. Methodology and data collection instrument

3. The SABER-SAA country report

4. Implementing SABER-SAA

3

SABER Domains or Policy Areas

Education inputs

Student enrollment

4

Expanding the Education Knowledge Base

to help countries assess, benchmark, and improve their education policies and governance

Quality of policies &

institutions Education

inputs

Quality of policy

implementation Learning for all

Quality and quantity of education delivered

5

SABER Policy Implementation

SABER Policy intent

School & household

surveys

EMIS EMIS, student assessments

Expanding the Education Knowledge Base

to help countries assess, benchmark, and improve their education policies and governance

Impact Evaluation

What is SABER-SAA?

SABER-School Autonomy and Accountability (SAA) analyzes school-based management policies in view of the level of managerial autonomy delegated to the school level, the existence and use of assessment, and the extent of accountability in the education system.

6

What does it do? • Classifies and analyzes education systems according to 5 policy

goals critical for enabling effective school-based management. What’s the goal? • To identify policy areas and actions that support better alignment

of managerial responsibilities at the school level, assessment of results, and use of assessments to promote accountability with an end view to increasing education quality and student learning.

What does it produce? • Country reports and other knowledge products that diagnose how well a country’s policies support school autonomy and accountability.

SABER-School Autonomy & Accountability

Motivation of SABER-SAA: How can we improve the performance of the education system?

More quantifiable issues: Access Internal efficiency Quality of learning Less quantifiable issues: Policies Institutional arrangements Processes Enabling environment

0%10%20%30%40%50%

8

Focus of SABER SAA

What matters for success in SAA?

Three Components of SBM: 1. Managerial autonomy, 2. The assessment of results, and 3. Use of assessment to promote accountability among stakeholders

When the 3 components of School-based Management are in balance with each other, the system performs better.

Teacher Quality EMIS

School Council Autonomy

Local decision-making based on evidence

Accountability Responsibility for school

and student performance

Assessment Measuring school and student performance

School Autonomy & Accountability The 3 A’s

SABER School Autonomy & Accountability: 5 Policy Goals

Policy Goals for SAA

1. Level of autonomy in planning and management of the school budget.

2. Level of autonomy in personnel management

3. Role of school council in school governance

4. School and student assessment

5. Accountability

5 Policy Goals for SAA

12 12

Personnel Autonomy

Student/School Assessment

Accountability

Participation

Budgetary Autonomy

1A Legal authority over the management of the operational budget

1B Legal authority over the management of the non-teaching staff salary

1C Legal authority over the management of the teacher salary

1D Legal authority to raise additional funds for the school

1E Collaborative budget planning

2A Autonomy in teacher appointment and deployment decision

2B Autonomy in non-teaching staff appointment and deployment decision

2C Autonomy in school principal appointment and deployment decision

3A Participation of the school council in budget preparation

3B Participation of the school council in financial oversight

3C Participation of the school council in personnel management

3D Participation of the school council in school activities

3E Participation of the school council in learning inputs

3F Transparency in community participation

4A Existence and Frequency of school assessments

4B Use of school assessments for making school adjustments

4C Existence and Frequency of standardized student assessments

4D Use of standardized student assessments for making school adjustments

4E Publication of student assessments

5A Guidelines for the use of results of student assessments

5B Analysis of school and student performance

5C Degree of Financial accountability at the central level, regional, municipal, local and school level

5D Degree of Accountability in school operations

5E Degree of learning accountability

Policy Action

SABER-School Autonomy & Accountability: How Data Are Collected

• Data collection instrument (questionnaire) organized around the 5 policy

goals and links to policy rubric.

• Collect documentation: Legislation, laws, policies • Conduct interviews: Policy-makers, officials

• Data and sources: researcher provides justification and reference materials to support answers to questionnaire.

Snapshot of Data Collection Instrument

Indicator 4. School and student assessment Sub indicator 4A. Existence and frequency of school assessments Justification References

Q1 Are schools assessed using the Ministry of Education criteria?

1. No

2. Yes

Q2 If yes, how often will it be assessed?

1.Not on a regular basis

2.Every few years

3.Every year

4.Every year including in-depth on a special topic (e.g. student poverty, school nutrition, migration)

Policy Goal 4: School and Student Assessment Policy Goal has 5 Policy Actions These two questions are for policy action 4A – Existence and

Frequency of School Assessments.

Producing Scores

Q1. --------? Q2. --------? Q3. --------? Q4. --------? Q5. --------?

Policy Action

Answers for each

Policy Action are analyzed

using Scoring Rules to produce a score from

1 to 4

Scoring Rules are defined by

the SABER-SAA Rubric, which describe a system’s level of development in relation to global good

practice on a scale from 1 to 4

Scoring Rules

Scores at the policy action level are then aggregated to the Policy Goal level.

SABER-SAA: Example of a Rubric Policy Goal 2: The level of autonomy in personnel management

Policy Action

Variable name Latent Emerging Established Advanced

2A Autonomy in teacher appointment and deployment decisions

Teachers must be appointed and deployed by the central government level under a union or civil service agreement.

Regional or municipal governments have legal authority to appoint teachers under union or civil service agreements. Appointments are subject to final review by central authorities.

Regional or municipal governments have legal authority to appoint and deploy teachers under union or civil service agreements without review by central authorities.

Schools (school principals, school council, parent association etc.) have legal authority to appoint teachers. Union and civil service agreement may or may not regulate the appointments.

2B Autonomy in non-teaching staff appointment and deployment decisions

Non-teaching staff must be appointed and deployed by central government level under civil service agreement.

Regional or municipal governments have legal authority to appoint non-teaching staff under civil service agreements.

Regional or municipal governments have legal authority to appoint and deploy non-teaching staff. Civil service agreement may or may not regulate the appointments.

Schools have legal authority to appoint non-teaching staff. Civil service agreement may or may not regulate the appointments.

2C Autonomy in school principal appointment and deployment decisions

Principals are to be appointed and deployed by the central level. Their performance is evaluated centrally and they can be transferred or fired by Central authorities.

Principals are to be appointed and deployed by the central level. Their performance is evaluated regionally or by municipal inspectors, which determines their tenure, transfer, or removal by Central authorities.

Principals are to be appointed and deployed by regional or municipal/local authorities, who are also responsible for their evaluation and have the authority for determining tenure, transfer, or removal.

Principals are to be appointed and deployed by municipal/local authorities in consultation with the School Council/ stakeholders at school level, or by the School Council alone. Municipal/local authorities are responsible for the principal's evaluation to determine tenure, transfer, or removal.

Classification of Scores

Latent

Emerging

Established

Advanced

Reflects policy not in place or limited engagement

Reflects some good practice; policy work still in progress

Reflects good practice, with some limitations

Reflects international best practice

The Country Report Shows where a country stands in relation to global good

practice as defined by the rubric

It serves as a platform to deepen dialogue and build consensus on system reform

It’s clear, concise, and neutral style conveys key messages to a broad range of stakeholders

The content is structured to ensure systematic coverage of all elements of the conceptual framework

Layout of SABER-SAA Country Report

Standard cover page that provides a brief overview of the results by policy goal. Inside the report (5-6 sections):

1. Introduction and country background information 2. The case for SAA – what matters (standard) 3. The country’s policy analysis results by policy goal 4. Policy recommendations for improvement 5. Optional – comparison of countries 6. Acknowledgements, Acronyms, References

Bulgaria Country Report

What did the policy analysis tell us about the state of SAA policies and their potential impact for improving learning?

Bulgaria’s Scores

21

1. School Autonomy in the planning and management of the school budget

Established

2. School Autonomy in Personnel

Management

Established

3. Role of the School Council in School Governance

Latent

4. School and student assessment Emerging

5. Accountability Emerging

SABER - School Autonomy & Accountability Assessment of Bulgaria

Score1 Autonomy in Budget Planning and Approval - Established

Municipalities receive funding for their schools through a per capita government transferSchool director in consultation with pedagogical staff has autonomy to prepare school budgetSchool director can set teacher salaries and bonuses following rules determined at central level. Board of Trustees, parents and community are not consulted nor have input to the budget. Schools have authority to raise additional funds from all sources - parents, NGOs, private sector, etc.

2 Autonomy in Personnel Management - EstablishedSchool director has full authority to appoint and deploy teachers using criteria determined at the school level and central level guidelines.School director has autonomy to hire and fire non-teaching staff following Bularian labor code. School Directors are hired and evalutated by the Regional Education Inspectorate and the municipality.

3 Participation of the School Council in School Governance - LatentBoard of Trustees (BOT) is not empowered to participate in budget preparation, school policies and decisionsBOT only has legal standing as a non-profit organization, no voice, and no oversight on budget issuesNo laws or systems to promote community participation in personnel matters, school activities and learning No provisions for the open election of BOT members and for calling general assemblies

4 Assessment of School and Student Performance - EmergingSchool inspections are not scheduled regularly and only assess for compliance, not for outcomesResults do not contribute to pedagogical, personnel or operational adjustmentsStudent assessments are conducted annually at grades 4, 7 and 12 There is no policy obligating schools to use data to make adjustments but schools do this in practice. Regions and schools receive data on student exam results, but not the public.

5 Accountability to Stakeholders - EmergingNo guidelines or mandates for use of or comparative analysis of student assessments results Rules and regulations guide financial management and transparency at all levels Some regulations for accountability in school operations exist but do not link rewards to performanceNo requirement to explain and simplify student assessment results to the public

Policy Goals

SABER-SAA application Other examples and how did it help?

• Informed Project Design • Nigeria – Disbursement Linked Indicators were developed

using information from the SABER assessments. Assisted project to craft relevant indicators for monitoring progress.

• Senegal – analysis used for new project design

• Informed policy for Education Sector Strategy • 3 Pacific countries – Samoa, Solomon Islands, Papua New

Guinea (SABER-SAA applied in partnership with AusAid)

• Deepened policy dialogue • Bulgaria and Kazakhstan – SABER-SAA combined with SABER

student assessment to understand better where the system could be improved for better learning outcomes.

Patterns Typical Approach to SBM

SBM is introduced through a pilot in a region(s) before going to scale

Often starts with policies that call for establishing School Councils.

Next step is to build capacity, some training on how to develop a School Improvement Plan (SIP)

Usually a school grant is involved – often for oversight of school maintenance or infrastructure

• Implementing autonomy or accountability alone has no additional effect on learning

• School autonomy must be complemented with school accountability to promote academic excellence

• Highest PISA scores come from countries where autonomy and accountability are implemented together

Autonomy and Accountability

Implementation Process

Getting Started

Step 1: Orientation and Training

Step 2: Develop Implementation Plan

Data Collection

Step 3: Carry out documentary evidence review and conduct interviews with credible informants to close knowledge gaps and corroborate evidence

Data Verification

Step 4: Validate data through a series of meetings

Step 5: Complete and submit DCI to the SABER-SAA Team for scoring

Report Preparation

Step 6: Prepare Country Report and obtain validation of information.

Key Players

• Data collection, analysis & reporting

• Technical support, scoring & feedback

• Validation, policy and its implementation

• Dialogue & operations

World Bank Country TTL

Government counterparts

Principal Investigator

World Bank SABER-SAA

team

EDUCATION STAFF DEVELOPMENT PROGRAM

THANK YOU

Contact World Bank Team SABER-SAA: TTL: Angela Demas (ademas@worldbank.org)