MONITORING AND EVALUATION OF NATIONAL STRATEGIES FOR ROMA INCLUSION: WHAT DATA AND FOR WHAT PURPOSE?Andrey Ivanov, Senior Policy Advisor, Human Development and Roma Inclusion cluster, UNDP BRC

MONITORING AND EVALUATION 101

Basic typology Monitoring (the process) Evaluation (of the results)

Intermediary or final Using indicators

Input Output Outcome Impact

Applied at different levels Of the National strategy Of the Action Plans Of Individual interventions

Monitoring what determines the kind of data and the kind of indicators used

THE STRATEGIES: HOW TO GET RESULTS

Having a National Strategy drafted is the beginning, not the end. It needs to be matched by National Action Plans (usually covering 2 years periods and regularly

updated) Local action plans Sector specific and integrated projects

In the case of the strategy, for M&E we need Clear targets – numerical expression of the objectives Adequate indicators – the definition of the target (how do we

measure whether the objective was reached) Quantitative baseline – the starting point against which the

progress/regress is be quantified (the value of an indicator at to) Milestones – intermediary targets on the way to the general

target to keep track of progress (the value of the indicator at t2, t4, t6)

The lower you go, the higher the chances for real inclusion of Roma in the process

DIFFERENT RESULTS AT DIFFERENT LEVELS

National strategy Long-term change in the situation of the target

group Difficult to attribute results (but not impossible)

National action plan Closer link between inputs and outcomes Clear objectives (that are the strategy’s milestones

Local action plans Direct link to project outputs Clear territorial dimensions

Individual interventions Counterfactual possible although difficult

OPEN QUESTIONS

What targets for individual priority areas? Roma specific or general?

What baseline? 2004? 2011? 2013?

What source of data? Data availability determines the indicators

or the other way around? What milestones? The link to individual OPs

EXAMPLE: ENROLLMENT RATE

ANOTHER OPEN QUESTION: WHO’S ROMA

Politically sensitive (incl. misuse of data for political purposes)

Legal (data protection) or ethical considerations (privacy and fear of stigma) constrains

Insufficient attention to comparability across countries, sub-regions, ethnic groups

The crucial question: what to put in the denominator of an indicator?

The nightmare answer: whatever serves the purpose…

WHO IS ROMA? POSSIBLE OPTIONS

Self-identification Outside (‘imposed’) identification

By non-Roma By Roma

Combined (multi-stages) – used in the surveys of UNDP (2004 and 2011) and of FRA (2011)

Crucial decision to be made: are we addressing “all Roma” – or “Roma at risk of marginalization”? The answers is both politically and policy loaded.

POSSIBLE SOURCE OF DATA FOR M&E

The data set of Roma vulnerable to marginalization generated from the UNDP/WB regional survey that is part of EU Roma Pilot Project funded by DG REGIO and from FRA Roma Pilot Survey: Monitoring fundamental changes possible (but not short-term

fluctuations). Suitable for National Strategy evaluation Most indicators have a base-line populated by data from the

survey conducted in 2004 by UNDP The “best game in town” (because it’s the only one…)Caveats: Still a survey (a sample is always a sample) Expensive, provides data on “Roma vulnerable to

marginalization” – and not on “Roma in general”Other options Roma boosters in HBS Longitudinal surveys

SURVEY DATA IS… SURVEY DATA

GOING BEYOND ETHNIC IDENTITY

Be pragmatic - don’t be obsessed by (don’t ask) unanswerable questions like “Who’s Roma?” But don’t dilute the task of Roma inclusion either

Give priority to socio-economic status But still keep ethnic identity and specifics in sight

Stick to territorial characteristics Most of the vulnerable Roma live territorially in

separate (segregated) communities Territorial mapping of those communities is possible Once a detailed map of Roma-dominated communities

is available, it will be possible to correlate ethnic characteristics with territorial tags (individual’s address)

This will allow monitoring a standard set of indicators for a population living in an area with ***% of Roma

THE BENEFITS OF TERRITORIAL APPROACH Makes possible to identify the absolute number of the

population and not only a percentage It can be an option solving the problem of individual

respondents refusal to declare ethnicity in the census or to declare different one

Less susceptible to political fluctuations Is more comprehensive in terms of social inclusion

(targeting vulnerability per se) It grasps the marginalized, visibly excluded segment of

the Roma population Actually reflects the fundamental logic of inclusion (including

the excluded, not those included already) Is best for ensuring that control groups (non-Roma

living in the same area) are also included

AN OPTIMAL COMPROMISE

One approach cannot serve all purposes Apply different data sources for different

planning frameworks National Strategy – EU-wide survey

(representative of… - a matter of political compromise)

National Action Plans – territorially-focused mapping

Individual interventions – project outcome evaluation

Integration of the three levels requires clear milestones in strategies and action plans

CONCLUSIONS

Integrate the monitoring functions into the entire implementation chain of the strategy

Don’t rely on one source of data and give priority to territorial approaches

Include clear milestones in National Strategies that would serve as a link to the National action plans and OPs

Compete the entire vertical planning and M&E architecture (strategy plan call for proposals interventions)

Go beyond poetry in Operational Programs evaluation building the latter bottom up

Be aware: keeping evaluations vague means keeping them fake