Q2010, Helsinki

Development and implementation of quality and performance

indicators for frame creation and imputation

Kornélia Mag

László Kajdi

Q2010, Helsinki

Q2010, Helsinki

Introduction, background

Eurostat financed grant (planned finish: August 2010.) The final report of the project will contain the following results regarding

to frame creation and imputation: Identification of the inputs, the result and the impacts of process

steps and the flow charts. Identification of the dimensions of process quality Detailed description of the specified standard variables like required

metadata, calculation and the possible variations of the variables, as well as recommendations for their application. The possible implementation of the indicators and the assessment of the measurement capability will be presented in the case of a concrete survey.

The development is based on the methods discussed in „Handbook on improving quality by analysis of process variables” (Eurostat [2003])”

Q2010, Helsinki

International experiences

Most often used and recommended indicators: For frame selection: coverage errors (over- and

undercoverage, misclassification and duplication) For imputation: imputation rate and ratio

International experiences Eurostat Standard Quality Indicators ONS indicators

Hungarian experiences

Process Quality Strategy Project: Process variables for the production process steps

Q2010, Helsinki

Frame creation

The fact that we can not observe the whole target population causes different errors on survey estimations.

Quality components Relevance: any differences in the definition of the target population

and the frame population or in the auxiliary information which are used in the survey can generate deficiencies on the relevance of the estimation.

Accuracy: any coverage problem that arises from not being able to sample from the whole of the target population can lead to the bias of the final estimation.

Timelines of the frames: the time lag between the reference date of the frame and the survey can cause coverage problems and consequently, it can have an effect on the final accuracy.

Q2010, Helsinki

Frame creation – process flowSurvey information:

Target population definition Auxiliary information Reference period

Frame description:Unit definitionAuxiliary information definitionReference periods of the units/items

Asessment procedures

Differences of the unit definitions

Differences of the auxiliary information definitions

Time lag between the reference periods

Deviation in the data

Errors

Missing units

Non population elements

Multiplicity elements

Incorrect or missing auxiliary information

Q2010, Helsinki

Process indicators Overcoverage I.: Number and percentage of exisiting units in the frame that are

not elements of the target population Overcoverage II.: Number and percentage of non-existing units in the frame Undercoverage.: Number and percentage of units which are not in the frame,

but are part of the second frame Misclassification: Number and percentage of units that are misclassified Number of multiplied units Number and percentage of the units excluded because of the time lag between

reference dates of the survey and the frameCalculation: Number of units in the frame constructed at a later stage, but excluded from the original frame / (Number of units in the frame used for the survey) * 100Metadata: Elements of the frame used for the survey; elements of the frame constructed at a later stageComment: This can be calculated if we use the quality check approach, which means portions of the frame are independently constructed at a time close to the survey.

Number and percentage of units with missing auxiliary information (by type)

Q2010, Helsinki

ImputationImputation is the method which serves the handling of problems concerning the missing, false or inconsistent responses identified during data-processing, editing.

Imputation takes place after the data collecting, data entry, editing, consistency checking, and outlier filtering phases. The quality of all these phases can affect all the forthcoming processing phases, including imputation.

Quality components affected by imputation

Accuracy: at origin imputation is used to improve the accuracy and completness of the data, but obviously, as a part of the processing it can cause errors. For example: Imputed data usually exhibit less variability than true data.

Timeliness: If we introduce an imputation method then the data capturing phase needs to be shorter. This ha an impact on the quality of the data capture phase. For example, if we plan to use the imputation we do not necessarily need to re-contact data suppliers for the missing or wrong data restoration.

Q2010, Helsinki

Imputation indicators Imputation rate, item level Imputation rate, unit level Rate of correction with previously imputed data

Calculation: number of cells corrected with previously imputed data / number of imputed cells.

Metadata: Data used for imputation with flags, imputed dataset with flags, description of the imputation method.

Comment: In case of using several data for the correction of a cell and one of them is corrected, than to imputation with previously imputed data must be considered.

Imputation ratio, item level Change in standard error due to imputation in case of main indicators

Q2010, Helsinki

The next steps of the project

Analysis and assessment of the measuring capability of the above identified indicators

Testing in the case of specific surveys EU-SILC Integrated business statistic survey

Q2010, Helsinki

Thank you for your attention!

Kornelia.Mag@ksh@hu

Laszlo.Kajdi@ksh.hu