FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS 12 December 2007AFCAS meeting, Algiers...

12 December 2007

AFCAS meeting, Algiers

FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS

Improving the Data Quality monitoring framework (CCSA/Eurostat self assessment

initiative)

Case of FAO Producer Prices data (methodology and data quality self assessment):

by Carola FabiFAO Statistics Division

12 December 2007



Introduction

• The subject of data quality is consistently addressed in many international forums

• countries have always addressed data quality from both practical and theoretical perspectives

• growing consensus that policy and decisions are increasingly reliant on better data if they are to be effective

• convergence of data quality models

12 December 2007



Paper outline

• main differences between national and international criteria

• data quality dimensions of particular relevance to agricultural price data

• results of the self-assessment exercise on producer price data based on the CCSA checklist

• suggestions for improving data quality

12 December 2007



1. General concept of quality of official statistics

All agree that statistics data quality is multi-dimensional, but there are differences in the criteria identified and in the vocabulary used to define the main data quality axes.

12 December 2007




Eurostat’s 6 dimensions of data quality• Relevance: degree to which statistics meet users’

needs.• Accuracy: closeness of estimates or computations to

the exact or true value.• Timeliness and punctuality: time lag between the

reference period and the release data, time lag between the release date and the target date.

12 December 2007




Eurostat’s 6 dimensions of data quality

• Accessibility and clarity: physical conditions in which data can be obtained, data’s information environment, whether they are accompanied by appropriate metadata; extent to which additional assistance is provided.

• Comparability: measures of the impact of differences in concepts and procedures when statistics are compared between geographical area, non-geographical domains, or over time.

• Coherence: statistics’ adequacy to be reliably combined in different ways and for various uses.

12 December 2007




• Statistics Canada and OECD call Interpretability what is Clarity for Eurostat (in particular metadata)

• The IMF includes four dimensions under Serviceability and Accessibility and adds Pre-Requisites, Integrity, and Methodological Soundness (procedural dimensions for data quality).

• FAO Statistics Division is adhering to the Eurostat taxonomy

12 December 2007



2. Quality dimensions revised in an international context

• Assessing international databases quality is an important exercise

• Frameworks need to be adjusted – because global datasets cover heterogeneous

countries – international organisations have a limited normative

power on the methodologies implemented in the individual countries.

12 December 2007




1. Shared Fundamental principles

2. Production of international data:• raw data is the output of national statistical offices• transformation/production process aims at adding value

12 December 2007




3. Quality dimensions – At the national level, the debate on data quality is much more

advanced than at the international level. – Many quality dimensions developed for national data sets

carry over directly to international data sets. – There are at least two specific dimensions of quality, which

apply at the international level: • (i) coverage - for how many countries/regions are data available

and • (ii) comparability - to what extent is the information for different

countries/regions comparable?

12 December 2007




4. Degree of coordination/cooperation – harmonize practices– develop standards, (Importance of metadata and

information technology) – create integrated international statistical system, with

an appropriate division of labor

12 December 2007



3. Characteristics of price data • Price data are generally multi-dimensional • A substantial amount of work has been done for

collection of price data to compile CPI, PPI and fro calculating PPP’s – study poverty issues and international comparison of

purchasing power.

• These studies are oriented towards well organised consumer markets where agriculture is not clearly visible.

• Price data concepts depends on the objective of data collection, frequency depends on the economic conditions at large

12 December 2007



3. Characteristics of price data

Questions for price data collection• What type of price? Can we collect actual transaction

prices rather than list price?• Will we collect basic prices (excluding taxes, including

subsidies on products, and excluding transport costs)?• At what time?• How to take into account product variety, quality?

Frequency of price collection will depend on how frequently prices change.

12 December 2007




Guiding principles for data collection• How tightly defined (narrow) or general (broad)

the item specification should be an issue of great theoretical and practical importance.

• Selection of items should be based upon a complete list of important crops and livestock product being produced by the country.

• The rules for selection should also take account of the sampling methodology underlying the selection of markets/traders.

12 December 2007




Guiding principles for data collection• Special attention to seasonal items.• Considering data collection less often than monthly for

some items, thereby enlarging total sample. • Regular timing is particularly important when inflation is

rapid.• Preferably days of the week and times of the month

should be chosen taking into account socio-economic set-up.

12 December 2007



3. Characteristics of price data Collection of price data is being done by countries either using a

probability sampling techniques for getting unbiased estimates or using a purposive sampling selection method.

Reasons for adopting non-probability sampling methods are:• No sampling frame is available,• Bias resulting from non-probability sampling is negligible• Possibility that the sample can not be monitored for long-time• Availability of staff• Sample size is too small• To give freedom to price collector to choose locally popular

varieties. The representative item method is more suitable for homogeneous products.

12 December 2007



4. Quality criteria applied to price data and problems with price data received in the FAO

Relevance: • of FAOSTAT price data: the adopted definition stems

from the SNA93 where producer prices enter into the compilation of value of production and economic accounts.

• FAO Statistics division has not yet developed derived price-based indicators, whose relevance will have to be assessed in due course.

12 December 2007




Countries responses

Responses 2002 2003 2004 2006 2007 total n.

Africa 13 22 18 6 10 48

World 80 101 103 73 100 189

12 December 2007




ACCURACY: Countries response rate

Response Rate

2002 2003 2004 2006 2007

Africa 27% 46% 38% 13% 21%

World 42% 53% 54% 39% 53%

12 December 2007




ACCURACY: Ratio of official to estimated data

World Africa

Category 2003 2004 * 2005 * 2003 2004 * 2005 *

Estimate 53% 69% 70% 73% 91% 92%

Official 44% 28% 27% 25% 8% 7%

Semi-offic. 3% 4% 3% 1% 1% 1%

TOTAL 100% 100% 100% 100% 100% 100%

12 December 2007




Sources of inaccuracy:

• doubt on price concept adopted• use of non standard-units

12 December 2007



4. Quality criteria applied to FAO price data

TIMELINESS AND PUNTUALITY

IDEAL FRAME. DATA REFERENCE YEAR: 2006

2007 2008

May: data collection 01 Jan 2008: 2006 data available

July: replies from countries

December: validation and database up-date

maximum 2 years-lag

CURRENT FRAME. DATA REFERENCE YEAR: 2006

2007 2008

July: data collection 2006 data available

September: 60% of replies arrived at earliest on 31 January 2008:

Validation still on-going

12 December 2007




Time-lag all countries

months 1 23

45

67

89

1011

Lag2007

Lag2006

Lag2004

Lag20030

10

20

30

40

50

60

Months

Time lag - World

Lag2007

Lag2006

Lag2004

Lag2003

12 December 2007




Time-lag African countries

12

34

56

78

910

Sum of Lag2007

Sum of Lag2006

Sum of Lag2004

Sum of Lag20030

1

2

3

4

5

6

7

8

Time lag - Africa

Sum of Lag2007

Sum of Lag2006

Sum of Lag2004

Sum of Lag2003

month

Data

12 December 2007




Accessibility and Clarity:• Available on-line and for free• Complemented by metadata on concepts

and definitions, country notes, currencies

12 December 2007




Comparability and Coherence:• Comparability is impaired by the complexity of prices

that are a dynamic and multi-dimensional dataset• Countries change concept; six African countries

replaced consumer prices with consumer prices• Lack of metadata to distinguish between comparability

issues and revisions (accuracy)• Coherence will be strengthened in FAOSTAT with the

development of derived indicators (Indices, value of production)

12 December 2007



5. Self-assessment checklist on FAOSTAT’s price datasets

• tool for the systematic quality assessment of statistics compiled by international or supranational organisations.

• It is a work in progress based on tests and inputs from offices such as the FAO

• It has been developed within the Committee for Coordination of Statistical Activities (CCSA) project,

• on the use and convergence of international quality assurance frameworks

12 December 2007




• Generic checklist : it is designed to apply to statistics gathered by international organisations, irrespective of the subject matter area

• Modular approach: to account for differences in statistical systems and functions of international organisations and enable to tailor it to specific needs.

• Full exercise at intervals to identify strengths and weaknesses

• Annual partial exercise to monitor quality over time

12 December 2007




Chapters1. Background Information2. Conceptual Frameworks3. Users And Customers4. Data Providers5. Validation (country data)6. Validation (international aggregates)7. Data dissemination8. Statistical confidentiality9. It conditions10. Documentation11. Follow-up of the statistical production process

12 December 2007




SELECTED INDICATORS

3. Users And Customers3/6 Availability of information on user satisfaction3/8 Assess overall user satisfaction

5. Validation (country data)5/2 Degree of completeness of received country data5/3 Degree of completeness of received country metadata5/7 Extent of unit non-response in country surveys5/19 Distribution of countries by number of revisions5/20Average size of revisions by the countries

6. Validation (international aggregates)6/1 Necessity of editing received data6/18 Average size of revisions by FAO6/20 Overall assessment of disseminated data accuracy

12 December 2007




SELECTED INDICATORS

7. Data dissemination7/12 Timeliness of preliminary results7/13 Timeliness of final results7/15 Is punctuality kept?7/19 Coherence of data within the domain7/20 Coherence in areas where it is applicable7/23 Comparability over time7/24 Extent of use of standard concepts and methodologies by countries7/25 Asymmetries in mirror data

10.Documentation– 10/6 Assessment of the completeness of metadata

12 December 2007


FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONSAssessment diagram - Production (crop & livestock) data (reference

year 2006)

0

1

2

3

4

5Sec 3/6

Sec 3/8Sec 5/2

Sec 5/3

Sec 5/7

Sec 5/10

Sec 5/12

Sec 5/19

Sec 5/20

Sec 6/1Sec 6/18

Sec 6/20Sec 7/12

Sec 7/13

Sec 7/15

Sec 7/19

Sec 7/20

Sec 7/23

Sec 7/24

Sec 7/25

Sec 7/26Sec 10/6

Production

Prices

12 December 2007


FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONSAssessment diagram - Production (crop & livestock) data (reference

year 2006)

0

1

2

3

4

5Sec 3/6

Sec 3/8Sec 5/2

Sec 5/3

Sec 5/7

Sec 5/10

Sec 5/12

Sec 5/19

Sec 5/20

Sec 6/1Sec 6/18

Sec 6/20Sec 7/12

Sec 7/13

Sec 7/15

Sec 7/19

Sec 7/20

Sec 7/23

Sec 7/24

Sec 7/25

Sec 7/26Sec 10/6

Production

Prices

TIMELINESS

USERSMETADATA

COMPLETENES

VALIDATION– COUNTRY DATA

VALIDATION – INTERNATIONAL

DATA

12 December 2007



Summarizing quality – Assessment diagrams and FAO Data quality stamp

Production Prices Meta data International classifications Update schedule Global coverage Integrated - Integrated Up-to-date

12 December 2007



6. Suggestions for improvement• sound concepts and definitions followed by sufficient

explanations on country practices • Capacity building for not only producing primary statistics but

also secondary statistics to increase coherence of price data.• Establish a metadata system, which may provide details

about approach (sampling design, questionnaire used, concepts and definitions) adopted by the countries to collect data.

• Regular feedback between FAO and countries indicating inconsistency and discrepancy noted in the data.

• Review all national price data sources and centralisation in one office to progressively increase coordination, improve resource uses, data comparability and coherence

• Introduction in FAO’s annual questionnaire of check and pre-validation tools

FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS 12 December 2007AFCAS meeting, Algiers...

Documents

Transcript of FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS 12 December 2007AFCAS meeting, Algiers...