The use of administrative data for program evaluation

Emanuela GalassoDevelopment Research GroupThe World Bank

The role of Administrative records and complex surveys in the monitoring and evaluation of public policiesRio de Janeiro Nov 2014

Using administrative data: the question

• Administrative data: main objective is monitoring for decision making– can trace and describe trends of key inputs and outcomes of interest

• Question: how can it be used for impact evaluations?

• Complementary to survey data collection

• For either type of data sources, a pre-requisite for embarking in an impact evaluation is to have a clear identification strategy

• Call for expanding access to administrative data prominent US economists

Using administrative data for impact evaluations: the potential

• Size: Universe of the target population

• Best placed to assess programs targeted to very specific groups – to be oversampled in surveys (the extreme poor, migrants), or geographic areas

• By-product of the existing monitoring of the social policy: cost

• time: characterize dynamics of impact

• Flexibility: can potentially look at different target groups, assess new variants

Using administrative data for impact evaluations: the cons

• Data up-to-date? Differential capacity of municipalities or agencies in data collection

• Representativeness? ex.registries of the poor (as CadUnico) covers the entire target population?

• Data quality? Income (imputations), scope for over/under reporting (ex. Colombia PMT)

• Scope: limited set of variables. May be missing key outcomes or determinants of pathway of impact(ex. noncognitive skills, mental health, parenting practices)

Using administrative data for impact evaluation: the experience of Chile Solidario

• Non-experimental evaluation: method of regression discontinuity design

• Planned after the program had started

• Exploited the clear assignment :

• Eligibility based on proxy means score (CAS). Official cutoff by municipality

• Households invited from the bottom up of the CAS distribution in each municipality

• Program gradually rolled out, 4 cohorts 2002-2005, became law in 2004

Chile Solidario: objectives and approach

1. Social inclusion

• Targeting the extreme poor (and vulnerable): bottom 5% CAS. From explicit demand for services to identification and invitation

• Central role of psychosocial wellbeing in social policy: 2 years of psychosocial support, local social worker

Chile Solidario: objectives and approach

2. Social protection

• Short term cash transfer + guaranteed monetary transfers

• Preferential access for social services (demand)

• Making the supply side available (supply) and tailored to the needs of the poorest

• Beyond access to health and education: multiple dimensions and ‘minimum conditions’

Chile Solidario: objectives and approach

3. Promotion

•.Identification of key skills and endowments to sustain exit from poverty (demand)

• Human capital of young and current generation

• Psychosocial capital

• Making supply side available (supply) and tailored to the needs of the poorest

Data: Panel Survey Chile Solidario• CASEN 2003: identify a representative sample of ChS

participants and an observationally ‘comparable’ sample of non-participants • Rich set of outcomes (health, education, housing, income,

poverty/indigence) CASEN + added new modules: intergenerational mobility, psychosocial dimension (since 2006)• Representativeness? cohorts 2002-2003

• Mistake sampling scheme in 2004 sampled better off non-participants

• Followed sample 2004, 2006, 2007

• No baseline data

Data: administrative data

• Panel of the administrative registry: Ficha CAS

• Demographics, housing/durables, employment, income, monetary subsidies

• In 2007 FPS : expanded into attendance to preschool, school, health visits, more detailed employment module,

• Detailed geographical information: 13 regions, 346 municipalities, 7000 neighborhoods.

• Can distinguish families vs households

• Coverage: 1/3 in 2000/1, to 2/3 after 2007

• Ficha vigente – updated every 2 years. Substantial churning families in/out

The key role of the national ID

• Registro Unico Tributario RUT. use it to merge it to a large set of complementary data (SIIS presentation yesterday)

• Merged registry with:

• Administrative data from the program (identity participants)

• administrative data on social workers, caseloads (quality of implementation)

• participation on training/employment programs (unit level records available at the time of the evaluation)

Descriptives: target of the bottom 5% in each municipality led to under-coverage nationwide

Proportion of participant among eligible households:

Cumulative entry around 50-60% of the potential target population

Descriptives: social exclusion was higher among the poorest

Note: Only families eligible for SUF are considered (heads 20-50 years old).

CAS Distribution

SUF take-up rate

regression discontinuity design: ‘effective’ cutoffs

• Exploit gradual roll out of the program over time to estimate effective thresholds that vary by municipality, over time

• inference caveat: window CAS within municipality, larger support nationwide

Participation to the program by cohorts• Eligibility defined by the ‘effective’ cutoff

Main result: take-up 2 years

11% 1.2%

2.3% 4%

Main results: employment 2 years

Going beyond average impacts to understand mechanisms

• Program is tailored to the needs: it is important to account for the initial conditions of beneficiaries

• compare households who had vs had not satisfied outcomes before 2002)

• Program evolved over time (supply side):

• compare cohorts before/after 2004

• In theory with current Ficha Social+ expanded historical information on programs could look at 6-10 years impact

Supply side articulation: employmentPreferential access to:

• Self-employment programs (AAE)

• Employability (PNCL)

Exclusive access to:

• Job placement (wage subsidy SENCE, PROFOCAP)

• Self-employment programs (PAME)

• employability (habilitacion sociolaboral SENCE, competencia laboral mujeres PRODEMU, apoyo jovenes SENCE)

• Share programs exclusive to CS increased over time

• Lion share of programs for self-employment (75%)

Results by initial conditions: take-up• Impact driven mainly by those who were previously

disconnected from social services (SUF)

Results by initial conditions: take-up• Take-up Employment programs driven by those

inactive/unemployed before CHS

Final employment outcomes show the importance of the supply side

• Spouse: activation employment by 20% only for those cohorts exposed to the supply side expansion

To summarize• Social protection: activation of the demand for a large array of

social programs

• Important demand side constraints (information, transaction costs, psychic costs activation)

• Impact persists over time

• Social inclusion: activation demand for programs for those who were previously disconnected

• Take-up of monetary transfers

• Take-up of labor market programs

• Promotion: No. Activation demand does not translate into final outcomes if not met by supply side:

• Need comprehensive activation component: technical and soft skills training, internship, job intermediation and placement

Quality of the social worker associated with better take-up rates and employment

Quantile of SW quality

Average SUF take-up rate

10 0.3602

25 0.5692

50 0.6141

75 0.7019

90 0.8824

Quantile of SW quality

Avg. prop. Head emp.

10 0.5414

25 0.7059

50 0.7222

75 0.8700

90 0.9816

Conclusions• Social registries starting data source for impact evaluation:

move beyond monitoring

• Cross check with unit record data on access to services, use of services)

• Can go to fine level of disaggregation, exploit time dimension

• Might be missing key outcomes (income, psychosocial dimensions)

• Caveat representativeness, quality

• Even programs are already at scale, refine questions of impact: quality of implementation, complementarity of services to target groups, experimentation