A tool for a snapshot assessment of a national statistical system
description
Transcript of A tool for a snapshot assessment of a national statistical system
A tool for a snapshot assessment of a national statistical system
Quality conference – 3-5 June 2014Vienna
Claudia Junker, Head of Unit "Statistical Cooperation", Eurostat
1
Content of the Presentation
1. Background2. Objectives3. Presentation of the tool4. Filling in the tool5. Using the tool6. Conclusions
2
Background
Original context is development cooperation, but the tool can be used also in a wider context
Results-based management and performance assessments are central in development
The monitoring of development policies and of aid delivery (and Budget support in particular) require the setting of benchmarks and the construction of systems of indicators
These benchmarks and systems of indicators are fed with public statistics which are provided by the countries themselves
Countries need baseline and benchmarks for assistance projects and their own development
=> What do we know about the quality of these statistics and how can we assess it in a user-friendly way?
3
Objectives:To design a user-friendly tool to help different stakeholders to assess: the quality of the data donors receive from the partner countries to monitor
the results achieved by the national policies and reform programmes, the performance of the support given by the EC to these policies and reform
programmes, The success of assistance projects visible in corresponding official statistical
data The state-of-play in the development of the NSS in a country, The efforts to improve the performance of the NSI/NSS => A natural vocation for supporting quality
4
Eurostat's commitment to quality statistics
European Statistics Code of Practice Regular self-assessments by the EU member states Peer reviews and Global assessments with partner countries (East and
South) Support to the African Charter on Statistics and similar regional initiatives
=> A natural vocation for supporting quality
5
Presentation of the tool
The tool consists of two modules:
Module 1: analyse and document the functioning of the national statistical system (NSS) as a whole, with a special focus on institutional, legal and organisational aspects.
Module 2: analyse and document the statistics in specific sectors. A wide range of sectors can be addressed (such as: education, health, agriculture, food security, private sector development, climate change and environment, economic growth, governance, infrastructure, external trade of goods, employment, living conditions, macro-economic stability, population).
6
Presentation of the toolModule 1: Overall National Statistical System (56 questions)
7
Themes Content Number of questions
Mainstreaming
Legal, institutional and strategic foundations (i.e. legal and institutional framework for the whole NSS, , integration and consistency of the statistical development strategy in the national development strategic framework)Organisation and coordination of the NSS
13
Organization
14
Resources
Adequacy of resources for statistics in the NSS (i.e. personnel in adequate quantity, quality of the personnel involved in statistics, equipment and infrastructure, financing)
12
Quality
Determinants of data quality (i.e. quality commitment, professional independence, impartiality, objectivity, methodology and appropriate statistical procedures)
13
Users Relations with users (i.e. relevance, accessibility) 4TOTAL 56
Presentation of the toolModule 1: National Statistical System (56 questions)
Examples of questions: Are statistics used in the development, implementation and monitoring of national development
policies? Are there statements made by high-level officials (president, ministers, parliament, etc.) on statistics
development? Is statistical independence a guiding principle for the functioning of the NSS? Does the NSS have mechanisms for assigning of responsibilities in the national statistical process? Is there a human resources policy (recruitment, career development, education and training) for the
NSI? Which proportion of the staff has a (permanent) internet connection? Are there internal handbooks / guidelines / recommendations for the statistical production processes? Are there formal processes in place to consult users about their statistical needs?
8
Presentation of the toolSector matrix
9
Development sectors / Issues Dimensions
Corresponding statistical sectors
Statistics producers
(NSS) Sources Main related
indicators
Health
Health statistics Ministry of Health DHS, MICS Administrative data from health structures
Child mortality Maternal mortality Main endemic diseases Budget share Other WHO and UNICEF indicators
Education
Education statistics Ministry of Education (Primary, Secondary, Tertiary)
Administrative data from education structures Population data
Literacy rates Enrolment rates Budget share Other UNESCO indicators
Agriculture
Agriculture (economic accounts, price, structures, production)
Ministry of Agriculture, Ministry of Rural Development, Ministry of environment
Agriculture census and surveys Farm register
Production Costs Prices Other FAO indicators
Employment
Offer and demand of work (balance between opportunities and capacities)
Labour market statistics Employment statistics
NSI Employment agency, Social security agency Ministry of Labor Ministry of Budget
Population census Household survey (Labour Force Survey) Administrative. data Public employment
Active population (by gender) Employment rates Unemployment Other ILO indicators
External trade of goods
External trade statistics
NSI Customs Partner countries
Customs records Imports and exports (quantities and values) Terms of trade indexes
Presentation of the toolModule 2: Sector statistics (between 59 and 61 questions)
10
Themes Content One data producer
Several data producers
Part 1 Sector information system assessment
Framework
Legal, institutional and strategic foundations (i.e. legal and institutional framework of the concerned sector information system, integration and consistency with the strategic framework supporting sector statistics development)
7 8
Resources
Adequacy of resources for statistics within the sectorinformation system (i.e. personnel in adequate quantity,quality of the personnel involved in statistics, equipmentand infrastructure, financing)
12 9
QualityDeterminants of data quality (i.e. quality commitment,professional independence, impartiality, objectivity,methodology and appropriate statistical procedures)
22 22
Users Relations with users (i.e. relevance, accessibility) 11 11
Part 2 Assessment of key indicators
Indicators Set of key criteria to roughly assess the quality of the indicator
9 9
Number of questions
Presentation of the toolModule 2: Sector statistics
Examples of questions: Is the development of sector statistics included in the Sector Development Strategies? Which proportion of the staff involved in producing sector statistics is equipped with a modern
computer? Is information on the methods and procedures used for the sector statistics published on the
NSI or Ministry website? Do all users get access to statistical releases at the same time? Are there partnerships or joint studies/works with the scientific community? Are their quality controls and checks on the reporting made by the concerned administrative
units (schools, health centres …)? How much of the statistical outputs in the sector are available via the internet? Is there a contact point to assist users?
11
Presentation of the tool
12
Far from complying with quality requirements
Not totally complying with quality requirements
Complying with quality requirements
Easy colour coding:
Objective criteria – Column in the matrix with management comments for filling in the tables
Presentation of the tool
13
Additional coding:
Recent progress
No changes
Deterioration
More subjective elements
Presentation of the tool
Detailed questions
Potential sources Situation Color
coding RemarksPerspec
tive coding
Identified sources
Initial assessment
Management of the table
Are statistics used in the development, implementation and monitoring of national development policies?
Some evidences found in the main policy papers. However, the reference to official statistics is not systematic and doesn't seem to be articulated around a clear diagnostic and a comprehensive data framework.
Partially ↑ ↑
Was statistical development addressed as part of these development policies?
No evidences found in the main policy papers. No additional information from the field mission
No
Are there statements made by high-level officials (president, ministers, parliament, etc.) on statistics development?
Several statements and articles mentionned during the field mission by the Vice Prime Minister and other high ranking officers from the government
Yes, but occasionaly
Statistics are included in the Government web site but under "Information". → →
Look in official documents, newspapers and other media for evidence of interest of decision makers in statistics.
Is the Statistical Day/Week celebrated?
Yes, statistical week in October 2010. A leaflet on the SDNSS has been prepared to this occasion.
Not regularly
No plan for further celebrations mentioned during the field mission. Additional clarification to put through the LSB or to find in the WB appraisal report
↓ ↓Regularly' means that the statistical day is celebrated each year or at least every two years.
During the latest celebrations, was the government represented?
The Vice Prime Minister opened and closed the ceremonies. This level is considered high in the lao context.
President or Prime Minister represented
It is not clear if other high level official participated in other events than the openning and closing sessions
→ →Yes' doesn't automatically mean green. Consideration must be given to the authority level of the Government representative. President or Prime Minister means green.
What is the degree of openess vis-a-vis external partners-donors or peers on statistical issues?
Assessment of the capacity to produce core economic indicators - Study made in July 2011 joiintly by the Lao Statistics bureau and the statistical Division of ESCAP (draft version)
Updated GDDS or recent review/assessment
Assessment by ESCAP July 2011. The obstacle to the adhesion to the GDDS must be looked for at the level of the line Ministries and not of the LSB
↑ ↑
Find reviews, studies, assessments of the statistical system that have been made with outside partners or relations established with international systems for assessing statistical data quality (GDDS/SDDS and peer reviews or global/sector assessments mainly).
LSB web-site: www.nsc.gov.laModule 1
Q1 : Statistical development mainstreamed as part of the national development policy
Prio
rity
Statistics should be an integral part of the national development policy, including for the design, monitoring and evaluation of Poverty Reduction Strategies, sector strategies, and other national development plans, as well as assessing progress made towards the MDGs. Most of the policies are linked to the achievements of the MDGS.
Oth
er
Official statements, Government decisions, National Budget, NSDS, National policy documents (SRPC), Reports in the media
Discussion with the LSB Deputy DG on November 13, 2012
Theme 1: Country strategy for statistics
Socio-economic Development Strategy 2020, 7th National Socio-economic Development Plan NSEDP 2011-2015, SDNSS 2010-2020
Look for evidence of whether statistics (i) have been used to design key country policy documents (diagnostics of the situation at macro and sector levels, indicators of achievement) and (ii) are addressed in these documents (attention given to the development and improvement of statistics systems at macro and sector levels). In some cases, the strategic matrix doesn't build on quantitative indicators. It may be necessary to go back to the preparatory studies.
14
Presentation of the toolPart 1 - Assessment at Sector/Ministry level EDUCATION
Detailed questions
Potential sources Situation Color
coding RemarksPerspec
tive coding
Identified sources
Initial assessment
Management of the table
Is there a law or an official agreement that organizes efficiently the production of sector statistics?
No specific law. But there is a practice withinthe Minsitry of Education that has been established during the last 20 years and that involves the schools, the districts and the provinces. There are also regular relations with the LSB (training, consultation on statistical operations)
The law addresses the rights and responsibilities of the line ministries for the production and the dissemination of statistics. A decree from the Ministry of Education established the Information Centre (role and responsibility) in 2006. There will be a new decree soon as ICT and statistics will be split in two different bodies.
→ Statistical law →
In most case, there will not be a specific law at the sector level. It is generally expected that the relations between the NSI and the other data producers in a country are driven by memorandum of understanding that sets the respective responsibilities, including data sets to be exchanged and calendar.
Does the National Statistical Office or other body responsible for statistics in the sector have access to the relevant administrative data?
Yes. No mandate is specifically written, but in practice, data are collected regularly from schools and villages.
Yes partialy
The system is managed by the Education Statistics and Information Technology Center of Education and Sport (SITCES). The system is improving year after year.
↑Interview with Deputy DG SITCES ↑
Oth
er
Is the actual work organization (mandate and attribution) in line with the law or the official agreement?
The LSB is responsible for the coherence of official statistics mainly through its responsibilities in the management and direction of the NSS. The Ministry of Education gather annual data from schools and villages.
Yes See Article 32 of the statistical law. → Statistical law →
To check if mechanisms (communication, dialogue, exchange of data …) that ensure coherence of methods and outputs between the different structures (NSI, sector Ministry, ) involved in the production of sector statistics
Module 2
Theme 1: Legal, institutional and strategic framework at the level of the sector (both the NSI and the sector Ministry)
Q1: Legal and institutional framework supporting the production of the sector statistics
Statistical laws, Government decisions, Memoranda of Understanding, Service Level Agreements, Questionnaire to DG
Prio
rity
15
Presentation of the tool
Part 2. Quality assessment – at the level of IndicatorsINDICATOR 1: MinJus - Number of judges
Detailed questions Potential sources Situation Colour coding Remarks Perspective
coding Identified sourcesInitial
assessment
Management of the table
Are there more than one national indicator on this issue?
No. Data produced by the MinJus No
Is the definition used in line with the international recommendations? Questionnaire to the
statistical service in charge, Compliance report
National methodology No
Is the methods used in line with the international recommendations from?
No
If the methodology has changed in the last years, are the changes documented and old estimates recalculated according to the new methodology?
or No change since 2009
No information available
Methodology changes are not a problem as far as the changes are documented, properly explained to the users, the revised methodology is applied to old series/data.
How long is the time series for this indicator Since 2009
LSMP 2009-2020 Check if it is a one shot production or a sustainable one well integrated in the national system.
Are the sources used and their coverage in line with the international recommendations?
Vertical informations system
No
Is the data frequency in line with the international recommendations?
Survey data (Demographic and Health Survey – DHS, Multiple Indicator Cluster Survey – MICS, Reproductive and Health survey - RHS)
Reports from Provinces on a quarterly basis
Is the indicator timely enough for the use made by the EC?
Data available for the annual Round Tables
Yes
Is the indicator publicly available and easy to find?
Certainly available for the Round Tables but not directly to the Public
No
16
Filling in the tool
What are the basic sources?Aspects linked to the legal and policy framework, to the organization and functioning of the system
Country macro-economic and social plan/strategy (Growth and poverty alleviation strategy or any similar policy document)
Generally available on the Government web site
National Strategy for the development of statistics (NSDS)
Information documented and regularly updated on the Paris 21 web site. Also generally available on the NSI web site
The statistical law and related application decrees
Generally available on the NSI web site
Other information on the NSS (coordination, exchanges among the data producers/users)
Sector strategy documents/plans Web sites of the related sector Ministries
Information on the Sector Information system
For Module 2 (mainly theme 1)
For Module 1 (mainly themes 1 and 2)
17
Filling in the tool
What are the basic sources?Aspects linked to practices: complements to the basic sources
Reports by the NSS (annual reports/plans, self assessments, studies, guides and methods…)
Not always available on the web (paper documents)
Reports by other donors Not always easy to identify and to gather particularly when there is no coordination mechanism for statistics
Contacts/discussions with the actors of the NSS
Direct knowledgeable source but sometimes
For all modules (mainly themes 3, 4 and 5 for Module 1 and themes 2, 3 and 4 for Module 2)
18
Filling in the tool Who will do it?
Staff from the EC Delegation or other donors alone
Even if only part of the question are answered, the tool can give useful indications, particularly if the objective is to support statistical improvement in areas most in needs
Experts statistician mobilized by the EC Delegation or other donors
This could help for the first exercise, repetition over the years being done then by the Delegation
EC Delegation with other donors
Filling in the tool could be the result of a coordination and information sharing exercise with the donors most interested and involved in the sector
EC Delegation or other donors and National Statistical System (NSS)
This would be the best way to initiate and feed a regular and articulated dialogue on the strengths and weaknesses of the NSS and to potentially open the way to concrete support
EC Delegation, other donors and NSS
Same as the above but involving all the interested donors. In some countries coordination for statistics already exist and could be fed with the results form the tool
NSS alone
This would be the result of a full appropriation of the tool by the NSS. Certainly the best solution in the long run.
19
In the framework of development cooperation In the framework of the NSS NSS alone
This would be the result of a full appropriation of the tool by the NSS. Certainly the best solution in the long run.
Experts statistician mobilized by the NSS
This could help for the first exercise, repetition over the years being done then by the NSS
Using the tool
Results Module 1: example
Theme 1: Stra
tegy for s
tatistics
Total 1
Theme 2: Organisa
tion of the N
SS
Total 2
Theme 3: Reso
urces
Total 3
Theme 4: Quality
determinants
Total 4
Theme 5: Relation w
ith users
Total 50
1
2
3
4
5
6
7
8
Module 1 - Global results
20
Using the toolResults Module 1: example
Thème 1: Stra
tegy for s
tatistics
Total 1
Thème 2: Organisa
tion of the N
SS
Total 2
Thème 3: Reso
urces
Total 3
Thème 4: Quality
determinants
Total 4
Thème 5: Relation w
ith users
Total 50
1
2
3
4
5
6
7
8
Module 1 - Global results No rigor in quality controls
Poor relevance of and accessibility to data
Strategy exists but not fully mainstreamed in policies
Legal framework in place but poor coordination
Gap in resources allocated to statistics
21
Using the tool
Results Module 1: example
Theme 1: Stra
tegy for s
tatistics
Q1: Mainstr
eamed statisti
cal d
evelopment
Q2: Status o
f statisti
cal s
trategy
Theme 2: Organisa
tion of the N
SS
Q1: Legal fr
amework
Q2: Organisa
tion and coordination
Theme 3: Reso
urces
Q1: Human re
source
s
Q2: Equipment
Q3: Financin
g
Theme 4: Quality
determinants
Q1: Quality
Commitment
Q2: Imparti
ality
Q3: Objecti
vity
Q4: Methodology r
esearch
Theme 5: Relation w
ith use
rs
Q1: Releva
nce
Q2: Acce
ssibilit
y
Module 1 - Detailed results
22
A snapshot of an NSS – using the tool
Results Module 1: example
Theme 1: Stra
tegy for s
tatistics
Q1: Mainstr
eamed statisti
cal d
evelopment
Q2: Status o
f statisti
cal s
trategy
Theme 2: Organisa
tion of the N
SS
Q1: Legal fr
amework
Q2: Organisa
tion and coordination
Theme 3: Reso
urces
Q1: Human re
source
s
Q2: Equipment
Q3: Financin
g
Theme 4: Quality
determinants
Q1: Quality
Commitment
Q2: Imparti
ality
Q3: Objecti
vity
Q4: Methodology r
esearch
Theme 5: Relation w
ith use
rs
Q1: Releva
nce
Q2: Acce
ssibilit
y
Module 1 - Detailed results
GDDS not joined, only a few statements by high-level officials on statistics
Participatory process for the design of the strategy but not translated into wok plans
The law is very comprehensive but the notion of confidentiality is not yet fully internalised
Share of responsibilities and coordination is established but poor execution in practice
Lack of trained and experienced personnel
Number of PC ok but poor internet access
Limited information available on financing
Sectors with no handbooks and no systematic quality controls
No rules for the correction of serious errors
No release calendar
No cooperation with academicsNo systematic consultation with the users
Poor usability of the web site for data dissemination
23
Using the tool
Results Module 2: specific indicators
22%
44%
33%
Number of judges
24
Using the tool
Results Module 2: specific indicators
22%
44%
33%
Number of judgesDefinitions and sources not in line with international recommendations
Only one producer and availability on time
Only short term series and data not easy to find for users
25
Using the tool
An instrument of statistical quality assessment for the EU Delegations
The tables could be filled and updated regularly as a monitoring instrument for the EU Delegations to assess the development of the statistical system of partner countries
26
Using the tool
An instrument of dialogue on statistics between donors (EU or others) and the partner countries
The tool could be embedded in annual reviews or monitoring exercises that could be done in the framework of a political dialogue between the donors and the country authorities,
The tool could ease identifying statistical areas in need to be supported by the donor community and help defining assistance programmes
27
Using the tool
An instrument of assessment of the progress made in the NSS
Filling the tables could be the result of a coordination work within the NSS. The results could be presented and discussed each year (regularly) at a meeting of the National Statistical Council or a similar entity. This would help adjusting work plans.
Examples and good practices could be drawn from exercises carried out in other countries.
28
Conclusions
What the Snapshot is:
In photography: a photograph that is "shot" spontaneously and quickly.The tool is a complex camera (comprehensive and articulated framework) that allows shooting spontaneously (the tool is available and can be used/understood by non-statisticians) and quickly (the tables can be filled in rapidly with available information)
In computing: the capture of an object's state at a specific moment in timeThe tool gives a picture of the statistical system of a country (or of sector statistics in a country) at the moment when the related tables are filled in. This picture can be taken at regular intervals (e.g. yearly) in order to monitor the progress made
29
Conclusions
What the Snapshot is not:
An instrument for rating and/or comparing countries The tool is “process-oriented” and aims at supporting a dialogue among the various actors involved
in the development of statistics in a country
A detailed analysis of the sector statistics There are more sophisticated and targeted frameworks for some statistical sectors (such as
Education/UNESCO) that detail the statistical processes leading to data production in more depth. The tool can be fed from these in-depth analyses and can also be an initial step before going deeper into the analysis.
30
Conclusions
Freely available from Eurostat upon request to send at: [email protected]
Available in three languages: French, English and Spanish Manual and a user-guide E-learning training course on how/why to use it
31