Producing monthly estimates of labour Producing monthly estimates of labour market indicators exploiting the market indicators exploiting the longitudinal dimension of the LFS microdatalongitudinal dimension of the LFS microdata

R. Gatto, S. Loriga, A. SpizzichinoIstat, Italian Statistical InstituteLabour Force Survey unit

Silvia Lorigasilvia.loriga@istat.it

NTTS 2009 - Bruxelles 19-20 February 2009

NTTS 2009 2

LFS is a continuous survey; main results are produced on quarterly basis

Eurostat currently releases monthly unemployment estimates based on LFS data (national and EU level); methodologies are chosen by each NSI

Italy is still not producing monthly unemployment estimates (Italian figures are quarterly estimates)

From 2007 a study project is being conducted in Istat (supported by an Eurostat grant). The object is: to study a proper methodology to produce robust monthly

estimates based exclusively on LFS data (no administrative sources on unemployment are available in Italy)

to deal with the complexity of producing monthly estimates on regular basis, no later than 3 weeks after the end of each month (partial sample)

NTTS 2009 3

Methods used by Member StatesMonthly LFS estimates

Germany, Finland, Sweden

3-months moving averages LFS estimates

Netherlands, UK, Norway

Quarterly LFS estimates + monthly registered unemployment with Chow Lin model

Portugal

Linear extrapolation of registered unemployment + benchmark to LFS

Belgium, Bulgaria, Czech Republic, Denmark, Estonia, Ireland, Spain, France, Latvia, Lithuania, Hungary, Poland, Slovenia, Slovakia, Cyprus, Malta, Austria, Luxembourg, Croatia

NTTS 2009 4

Estimators comparison in the Italian studyCONTEXT External sources on monthly unemployment not

available (use of registered unemployed or Chow Lin)…

COMPARISON1. The direct monthly estimator from LFS (the LFS

sampling design has a monthly stratification)2. The regression composite monthly estimator from LFS3. The 3-months moving average estimator from LFS (only

if 1 and 2 are not satisfactory)

CRITERIA Sampling error, robustness, time series decomposition,

coherence with quarterly estimates

NTTS 2009 5

The Italian sample rotation scheme 2-(2)-2 households participate to the survey 4 times during a

15 months period 50% overlap between following quarters 50% overlap between a quarter and the same quarter of

the previous year The sample follows a monthly stratification:

50% overlap between months t and t-3 50% overlap between months t and t-12

Q1_Y1 Q2_Y1 Q3_Y1 Q4_Y1 Q1_Y2 Q2_Y2

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

NTTS 2009 6

The regression composite estimator (RCE) 1 It is a kind of composite estimator

Direct estimators: design based, only observed sample information

Composite estimators: model based, observed sample + additional information

It may be applied to repeated surveys with partially overlapping samples

The idea: as the employment status observed in a previous point in time is correlated with the current employment status, using it as auxiliary variable will improve the estimation

It is based on the regression of the usual cross-sectional estimator on a set of predictors computed on the overlapping sub-sample from previous time points

Level and changes estimates are improved

NTTS 2009 7

The regression composite estimator (RCE) 2 Singh et al. developed for the Canadian LFS a kind of

RCE in which micro-level past information are used as predictors, through calibration

Calibration estimator:

weights wk are computed in two steps: 1: initial weights dk are obtained as the inverse of the

inclusion probabilities 2: final weights wk are obtained solving the following

minimization problem under constraints:

solved through an iterative procedure

sk

kkwyY~

skkk

skkk

w

wddist

tx

),(min

NTTS 2009 8

The regression composite estimator (RCE) 3 The calibration estimator is the estimator currently used

to produce quarterly LFS estimates in Italy In the regression composite version of the calibration

estimator developed to produce monthly estimates for the Italian LFS

additional auxiliary variables have been introduced: the employment status observed at the individual level 3 months ago and 12 months ago (only for individuals in the overlapping sub-sample)

constraints are derived from the previous estimates referred to 3 months ago and 12 months ago obtained using this same estimator (X overlapping %)

No longitudinal auxiliary variables (no constraints) for the non overlapping sub-sample (no imputation as Canadian LFS estimator)

NTTS 2009 9

The regression composite estimator (RCE) 4 Monthly estimator similar methodology as the quarterly

estimator; good for coherence Based on weights computation; good to produce

consistent estimates of different variables Design based through the use of inclusion probabilities

in weights computation Micro-level model based through the use of individual

auxiliary variables It exploit the longitudinal dimension of the sample:

Considering 3 months ago and 12 months ago information we are adding information on both short and long term period

Estimation of changes at t-3 and t-12 improves More robust estimation of both trend and seasonality

NTTS 2009 10

Dealing with partial sample 1 According Reg. 577/98 interviews can be done until 5

weeks after each reference week Quarterly data have to be transmitted to Eurostat within

12 weeks of the end of the quarter Monthly estimates have to be produced and transmitted

no later than 3 weeks after the end of each month Monthly estimates have to be computed over a partial

sample Higher sampling errors Not at random: mode (capi/cati), reference week within the

month, household typologies, employment status Bias Variability over time Higher distance with final monthly and quarterly estimates

NTTS 2009 11

Dealing with partial sample 2Total interviews

0

2000

4000

6000

8000

10000

12000

14000

16000

1 2 3 4 5 6 7 8 9 10 11 12 13

not on time

on time

CAPI

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13

not on time

on time

CATI

0

1000

2000

3000

4000

5000

6000

7000

8000

1 2 3 4 5 6 7 8 9 10 11 12 13

not on time

on time

% not on time

0.0

10.0

20.0

30.0

40.0

50.0

60.0

1 2 3 4 5 6 7 8 9 10 11 12 13

tot

capi

cati

1stquarter2008weeklydata

NTTS 2009 12

Dealing with partial sample 3Total interviews

0

2000

4000

6000

8000

10000

12000

14000

16000

1 2 3 4 5 6 7 8 9 10 11 12 13

not on time

on time

CAPI

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13

not on time

on time

CATI

0

1000

2000

3000

4000

5000

6000

7000

8000

1 2 3 4 5 6 7 8 9 10 11 12 13

not on time

on time

% not on time

0.0

20.0

40.0

60.0

80.0

100.0

120.0

1 2 3 4 5 6 7 8 9 10 11 12 13

tot

capi

cati

2ndquarter2008weeklydata

NTTS 2009 13

Dealing with partial sample 4

% men

46.0

46.5

47.0

47.5

48.0

48.5

49.0

49.5

50.0

1 2 3 4 5 6 7 8 9 10 11 12

not on time on time

% population 15-64

58.0

60.0

62.0

64.0

66.0

68.0

70.0

1 2 3 4 5 6 7 8 9 10 11 12

not on time on time

% population 65 or more

15.0

17.0

19.0

21.0

23.0

25.0

27.0

29.0

1 2 3 4 5 6 7 8 9 10 11 12

not on time on time

From 4th quarter 2007to 3rd quarter 2008

monthly data

NTTS 2009 14

Dealing with partial sample 5

employment rate 15-64 (not weighted data)

54.0

55.0

56.0

57.0

58.0

59.0

60.0

61.0

1 2 3 4 5 6 7 8 9 10 11 12

not on time on time

unemployment rate (not weighted data)

5.0

5.5

6.0

6.5

7.0

7.5

8.0

1 2 3 4 5 6 7 8 9 10 11 12

not on time on time

From 4th quarter 2007to 3rd quarter 2008

monthly data

NTTS 2009 15

Dealing with partial sample 6 To correct for bias due to the partial sample, additional

constraints have been included in the calibration procedure:

Households by 4 rotation groups Households by mode (capi-cati) Households by reference week (4 or 5 in each month) Constraints are derived from the theoretical sample

By this way bias due to the partial sample is partially corrected and estimates over partial samples are closer to final estimates over the whole sample

NTTS 2009 16

Legenda

QCE(Q): Quarterly calibration estimator - per quarter (the usual quarterly estimator)

QCE(M): Quarterly calibration estimator - per month (the usual quarterly estimator applied to single months)

MCE(M): Monthly calibration estimator (a calibration estimator similar to the usual one, computed for each month)

MRCE(M): Monthly regression composite estimator (a calibration estimator similar to the usual one + constraints on condition at t-3 and t-12, computed for each month)

NTTS 2009 17

Employment estimates

21750000

21950000

22150000

22350000

22550000

22750000

22950000

23150000

23350000

23550000

23750000

01_2

004

02_2

004

03_2

004

04_2

004

05_2

004

06_2

004

07_2

004

08_2

004

09_2

004

10_2

004

11_2

004

12_2

004

01_2

005

02_2

005

03_2

005

04_2

005

05_2

005

06_2

005

07_2

005

08_2

005

09_2

005

10_2

005

11_2

005

12_2

005

01_2

006

02_2

006

03_2

006

04_2

006

05_2

006

06_2

006

07_2

006

08_2

006

09_2

006

10_2

006

11_2

006

12_2

006

01_2

007

02_2

007

03_2

007

04_2

007

05_2

007

06_2

007

QCE(Q)_emp

QCE(M)_emp

MCE(M)_emp

MRCE(M)_emp

NTTS 2009 18

Unemployment estimates

1200000

1300000

1400000

1500000

1600000

1700000

1800000

1900000

2000000

2100000

2200000

01_2

004

02_2

004

03_2

004

04_2

004

05_2

004

06_2

004

07_2

004

08_2

004

09_2

004

10_2

004

11_2

004

12_2

004

01_2

005

02_2

005

03_2

005

04_2

005

05_2

005

06_2

005

07_2

005

08_2

005

09_2

005

10_2

005

11_2

005

12_2

005

01_2

006

02_2

006

03_2

006

04_2

006

05_2

006

06_2

006

07_2

006

08_2

006

09_2

006

10_2

006

11_2

006

12_2

006

01_2

007

02_2

007

03_2

007

04_2

007

05_2

007

06_2

007

QCE(Q)_une

QCE(M)_une

MCE(M)_une

MRCE(M)_une

NTTS 2009 19

Erratic componentEmployment estimates

-500000

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

01-2

004

02-2

004

03-2

004

04-2

004

05-2

004

06-2

004

07-2

004

08-2

004

09-2

004

10-2

004

11-2

004

12-2

004

01-2

005

02-2

005

03-2

005

04-2

005

05-2

005

06-2

005

07-2

005

08-2

005

09-2

005

10-2

005

11-2

005

12-2

005

01-2

006

02-2

006

03-2

006

04-2

006

05-2

006

06-2

006

07-2

006

08-2

006

09-2

006

10-2

006

11-2

006

12-2

006

01-2

007

02-2

007

03-2

007

04-2

007

05-2

007

06-2

007

QCE(M)_emp

MCE(M)_emp

MRCE(M)_emp

NTTS 2009 20

Erratic componentUnemployment estimates

-150000

-100000

-50000

0

50000

100000

150000

01-2

004

02-2

004

03-2

004

04-2

004

05-2

004

06-2

004

07-2

004

08-2

004

09-2

004

10-2

004

11-2

004

12-2

004

01-2

005

02-2

005

03-2

005

04-2

005

05-2

005

06-2

005

07-2

005

08-2

005

09-2

005

10-2

005

11-2

005

12-2

005

01-2

006

02-2

006

03-2

006

04-2

006

05-2

006

06-2

006

07-2

006

08-2

006

09-2

006

10-2

006

11-2

006

12-2

006

01-2

007

02-2

007

03-2

007

04-2

007

05-2

007

06-2

007

QCE(M)_une

MCE(M)_une

MRCE(M)_une

NTTS 2009 21

Main references

As the regression composite estimatorSurvey Methodology – Volume 27, Number 1, June 2001Special section on Composite Estimation A.C. Singh, B. Kennedy and S. Wu “Regression Composite

Estimation for the Canadian Labour Force Survey with a Rotating Panel Design”and following papers by

W.A. Fuller and J.N.K. Rao P. Bell J. Gambino, B. Kennedy and M.P. Singh

As the calibration estimator J.C. Deville and C.E. Särndal (1992) “Calibration Estimators in

Survey Sampling” JASA, vol. 87

NTTS 2009 22

Annex 1Quarterly calibration constraints

X1-X28 X29-X138 X139-X168 X169-X170 X171-X172 X173-X176 X177-X182

domain 1domain 2domain 3domain 4domain 5domain 6domain 7domain 8domain 9domain 10domain 11domain 12domain 13domain 14domain 15domain 16domain 17domain 18domain 19domain 20domain 21

(domains are NUTS-2 regions)F

ore

igne

r po

pula

tion

EU

and

not

-EU

Hou

seho

lds

by 4

ro

tatio

n g

roup

s

Pop

ulat

ion

by

gen

der

fo

r e

ach

mo

nth

Pop

ulat

ion

by

gen

der

an

d 14

age

cla

sse

s

Pop

ulat

ion

in N

UT

S-3

do

mai

ns b

y g

end

er

and

5 a

ge

cla

sse

s

Pop

ulat

ion

in b

ig m

un

icip

alit

ies

by g

ende

r

and

5 a

ge

cla

sse

s

For

eig

ner

popu

latio

n b

y ge

nde

r

NTTS 2009 23

Monthly calibration constraints

X1-X28 X29-X108 X109-X188 X189-X198 X199-X206 X207-X238 X239-X254 X255-X294 X295-X310 X311-X326 X327-X342 X343-X358

rip 1

rip 2

rip 3

rip 4-5

(rip are NUTS-1 regions)

Po

pu

latio

n b

y g

en

de

r a

nd

14

a

ge

cla

sse

s

Po

pu

latio

n b

y N

UT

S-2

re

gio

ns,

g

en

de

r a

nd

5 a

ge

cla

sse

s

Po

pu

latio

n in

NU

TS

-3 d

om

ain

s b

y g

en

de

r

Po

pu

latio

n in

big

mu

nic

ipa

litie

s b

y g

en

de

r

Em

plo

yed

at

t-1

2 b

y N

UT

S-2

re

gio

ns

an

d g

en

de

r

Un

em

plo

yed

at

t-1

2 b

y N

UT

S-2

re

gio

ns

an

d g

en

de

r

Fo

reig

ne

r p

op

ula

tion

by

NU

TS

-2

reg

ion

s

Ho

use

ho

lds

by

NU

TS

-2 r

eg

ion

s a

nd

4 r

ota

tion

gro

up

s

Em

plo

yed

at

t-3

by

NU

TS

-2

reg

ion

s a

nd

ge

nd

er

Un

em

plo

yed

at

t-3

by

NU

TS

-2

reg

ion

s a

nd

ge

nd

er

Ho

use

ho

lds

by

NU

TS

-2 r

eg

ion

s a

nd

5 w

ee

ks

Ho

use

ho

lds

by

NU

TS

-2 r

eg

ion

s a

nd

mo

de

(ca

pi-c

ati)

NTTS 2009 24

Annex 2QCE(M)_emp

QCE(M)_emp

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

seasonal

erratic

NTTS 2009 25

MCE(M)_emp

MCE(M)_emp

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

seasonal

erratic

NTTS 2009 26

MRCE(M)_emp

MRCE(M)_emp

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

seasonal

erratic

NTTS 2009 27

QCE(M)_une

QCE(M)_une

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

seasonal

erratic

NTTS 2009 28

MCE(M)_une

MCE(M)_une

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

seasonal

erratic

NTTS 2009 29

MRCE(M)_une

MRCE(M)_une

-400000

-300000

-200000

-100000

0

100000

200000

300000

400000

seasonal

erratic