Encuentro Hemisférico sobre Evaluación en la Primera ... Proyectos Actividad Documento...Dimensão...
Transcript of Encuentro Hemisférico sobre Evaluación en la Primera ... Proyectos Actividad Documento...Dimensão...
Encuentro Hemisférico sobre Evaluación
en la Primera Infancia
Asucion 2011
VisionDevelop child to develop society
MissionGenerate and disseminate
knowledge for integral early
childhood development
Multi-methods in monitoring and
impact evaluations of complex social
programs
The case of the Maria Cecilia S.
Vidigal Foundation’s Zero to There
Program in BrazilEduardo Marino, MSc
Thomaz Chianca, Phd
The case
• Zero to Three Program supported by the Maria Cecilia Souto Vidigal Foundation (FMCSV) in six municipalities in the state of São Paulo, Brazil
Botucatu
Itupeva
Penápolis
São Carlos
São José do Rio Pardo
Votuporanga
The case (cont.)
• Establishment of local partnerships among the local government, nonprofits and the private sector to promote ECD
• Initial intervention: capacity building in ECD (doctors, registered nurses, teachers, etc)
• Support from 2009-2013
The evaluation challenges
� Social programs are complex
� Involve several actors interacting in dynamic contexts
� Capturing changes attributable to a complex social
intervention require the combinations use of
traditional and non-linear methodologies
� Synthesizing evidence to reach evaluation conclusions
in real world evaluations is a difficult task
CONTEXT
CERTAINTY
Organazing Dynamics
(order, prediction,
control)
Self-organizing
Dynamics ( emerging patterns,
coherent but not
pretictable )
Unergarnezed
Dynamics(random,
unpatterned, caos)
Close to
Certanity
Clo
se t
o
Pla
ne
d
Far from
Certanity
Fo
r F
rom
Pla
ne
d
System Dynamics
Zimmerman, B and Colleagues. Humana System Dynamics (2001)
CONTEXT
CERTAINTY
Predictive
Design
Organic
Evaluation
Exploratory
Design
Close to
Certanity
Clo
se t
o
Pla
ne
d
Far from
Certanity
Fo
r F
rom
Pla
ne
d
Evaluation Design
Zimmerman, B and Colleagues. Humana System Dynamics (2001)
The evaluation scope
• Program baseline (2010)-
predictive design
• Monitoring system to
capture innovations and
provide timely feedback
for improvements-
organic design
• Impact evaluation(2014 -
2015) - predictive design
Dimensions – baseline and
impact
Children
+ + +
Dimensions, Criteria and
IndicatorsD1 - Health ,education and social professionals prepared to work effectively in early childhood
development (ECD)
Criteria 1
Indicators 7
D2 - Services to attend families in the areas of health, education and social services assimilate the
focus of ECD
Criteria 5
Indicators 46
D3 - Parents and caregivers prepared to help their children to fully develop
Criteria 1
Indicators 7
D4 - Changes in the services of the community and in public policies of ECD
Criteria 1
Indicators 5
Total criteria 8
Total indicators 65
D5 – Children’s readiness to learn (age 05 to 06)
Domains 5
Rubrics – example (qualitative)
• Indicator 1.1.1: Professionals stimulate the participation of fathers in prenatal consultancy appointments
• Rubric 1.1.1:
– 0=no effort or preoccupation in stimulating attendance of fathers in prenatal appointments;
– 1=some preoccupation;
– 2=clear preoccupation, but no effort;
– 3=clear preoccupation and some effort (not systematic);
– 4=systematic effort;
– 5=successful systematic efforts to stimulate the participation of fathers in prenatal consultancy appointments.
Rubrics – example (quantitative)
• Indicator 2.9.1: Adequate proportion of children per early childhood educator in all day care centers
• Rubric 2.9.1:
– 0=All day care centers have less educator than proposed as ideal by the Brazilian Camber for Basic Education
– 1=Less than 25% of day care centers have an adequate proportion children/educators
– 2=25% to 50% adequate
– 3=More than 50% to 75% adequate
– 4=More than 75% to 95% adequate
– 5=All day care centers with adequate proportion children -educators
Data Collection Methods (baseline)
Dimension Method
1. Professionals�Focus group facilitated by the evaluators (pediatricians, gynecologists,
nurses, daycare educators, pregnant women, mothers/fathers)
2. Services
�Focus group facilitated by the evaluators (pediatricians, gynecologists,
nurses, daycare educators, pregnant women, mothers/fathers)
�Questionnaire (executive secretaries of the projects)
�Analysis of secondary data (situational diagram, Datasus, IBGE, MEC)
3. Families
�Focus group facilitated by the evaluators (pregnant women and
mothers/fathers)
�Analysis of secondary data (situational diagram, Datasus, IBGE, MEC)
4. Community
�Analysis of secondary data (situational diagram, government plan,
CMDCA reports, program documents , clippings )
�Questionnaire (executive secretaries of the projects)
�Questionnaire (university representative)
5. Children�EDI – Early Development Instrument (Offord Centre for Child Studies –
McMaster University, Canada)
Spreadsheet for data entry and
analysesLinha de
Base
Atual Ajuste do
Peso
Ajustado pelo
peso
0 a 5 0 a 5 Linha de Base Atual
Dimensão 1: Profissionais de saúde e educação infantil preparados para atuar no DI Linha de Base Alcançado
21.5 32.6 11.1 38.9%
1. Profissionais estimulam a participação do pai nas consultas 2.0 3.5 2.0 3.5
2. Profissionais informam sobre sinais de perigo para a gravidez 1.0 4.0 1.0 4.0
3. Profissionais avaliam a aceitação da gravidez por parte dos pais 1.5 4.0 1.5 4.0
4. Profissionais avaliam a rede de suporte à mãe 2.0 3.0 2.0 3.0
5. Profissionais avaliam adesão da mãe (e o apoio do pai) às 1.0 4.0 0.0 0.0
6. Profissionais utilizam abordagem não prescritiva (clínica ampliada) nas 1.5 3.0 0.0 0.0
7. Profissionais orientam sobre a importância da interação dos pais com os 2.0 4.0 2.0 4.0
Subtotal do Critério 8.5 18.5 10.0 60.6%
Critérios
1.1. Profissionais de
saúde e de educação
infantil preparados
para atuar de
maneira eficaz no
desenvolvimento
infantil
Indicadores
Contribuição FMCSV
21.5 11.1 17.4
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Meeting Basic Needs
Linha de Base Contribuição FMCSV Necessidade Remanescente
Graph representation of situation
at baseline (4 dimensions)
PDI
Project 1
Project 2
Project 3
Project 4
Project 5
Project 6
Criteria: Mother and father are prepared to support the
development of their children
Sínteses
Indicators
Mother, father and caregivers use some moments of their routine as an
opportunity to stimulate the child's learning and strengthening of the bond
Mothers have enough support from peers, family members or friends
to routine activities in primary care with the child
Mother, father and caregivers adopt measures to prevent accidents at
home with their children
Pregnant women initiate prenatal care from the first trimester of pregnancy
Father or partner accompanies mother throughout the prenatal
Father or partner accompanies mother throughout child-birth labor
Pregnant women and fathers are well prepared for childbirth,
including prenatal and postpartum.
Mothers breastfeeding children exclusively up to 4 months of age
Critical aspects and best
practices – baseline
D5 - Children – EDI – Early
Development Instrument
• 120 items grouped into five domains
• Completed by teacher (early childhood educator)
• Context sections relevant to the local context
• Items adaptable to the local language/context
How the EDI works
The EDI assesses children’s readiness
to learn when they enter school by
looking at five key areas of child
development
Domains of child
development
• Physical health and well-being
• Social competence• Emotional maturity• Language and
cognitive development
• Communication skills and general knowledge
Key points about the EDI
• School entry age range (4-7)
• Teacher or parent can complete
• Minimal training involved
• Broad picture of developmental status
• Population- or group-level results
• Can be linked with other data sources (health,
sociodemographics, programs)
• Growing international database available for
comparison
Vulnerability in the domains
and overall by school schedule
Vulnerability in the domains
and overall by Bolsa Família
participation
Evaluation use
• Discussion of evaluation findings with each community project – baseline
• Innovations identified by the monitoring process are shared between the projects
• Evaluation findings being used by FMCSV and by community projects as the basis for planning for 2012
Some lessons learned
• Involving program staff in the development of dimensions, criteria, indicators and rubrics is important to ensure quality and buy-in
• Quantitative indicators from well-known databases are easier to analyze; however, they do not address most complex aspects of the evaluation criteria and dimensions
Some lessons learned (cont.)
• To become more grounded to reality,
indicators and rubrics need to be revised
after the first round of data collection
• The scoring should be discussed with
program staff and project representatives
to verify if the criteria and the findings
make sense
Some lessons learned
(cont.)
• Short reports (e.g., PPT
presentations) with main
findings are helpful to
engage stakeholders
• Final report should be
prepared after the discussion
of findings with stakeholders
to incorporate their
perspectives
• Timeframe between data collection/analyses and discussion of
findings with stakeholders should be kept short to keep the
momentum/interest