ABSTRACT Developing evaluation frameworks for large, multi-program projects: Building logic...

Developing evaluation Developing evaluation frameworks for large, frameworks for large,

multi-program multi-program projects:projects:

Building logic “meta-models” and Building logic “meta-models” and visual databasesvisual databases

Jason Newberry, Andrew Taylor, Robert Case, and Kristen Roderick Centre for Research and Education in Human Services

[email protected]

Overview of PresentationOverview of Presentation

How large government policy frameworks How large government policy frameworks create new challenges for health and create new challenges for health and social service evaluators social service evaluators

Ontario’s Early Childhood Development Ontario’s Early Childhood Development initiatives: A Work in Progressinitiatives: A Work in Progress

The development of an evaluation The development of an evaluation framework: logic “meta-models” and visual framework: logic “meta-models” and visual databasesdatabases

Please Note:Please Note:

Although this presentation draws on Although this presentation draws on examples from our ongoing work with examples from our ongoing work with programs run by the Ontario government, programs run by the Ontario government, the specific content of these examples is the specific content of these examples is intended only to illustrate the evaluation intended only to illustrate the evaluation process we have designed and does not process we have designed and does not reflect the official position of the Ontario reflect the official position of the Ontario Government regarding these programs.Government regarding these programs.

A New Reality for EvaluatorsA New Reality for Evaluators In the public sector, there is a greater need for evaluation In the public sector, there is a greater need for evaluation

because of:because of: A rebirth of interest in the social determinants of health and A rebirth of interest in the social determinants of health and

primary prevention.primary prevention. A new emphasis on results and evidence-based management.A new emphasis on results and evidence-based management. A new acknowledgement of the need to let different regions A new acknowledgement of the need to let different regions

develop unique approaches to addressing shared priorities.develop unique approaches to addressing shared priorities. Evaluators are now confronted with the task of Evaluators are now confronted with the task of

evaluating the impact of many diverse programs that fall evaluating the impact of many diverse programs that fall under very broad funding strategies. under very broad funding strategies.

This requires developing evaluation frameworks and This requires developing evaluation frameworks and strategies that are capable of evaluating very different strategies that are capable of evaluating very different programs that have been created to achieve similar programs that have been created to achieve similar goals.goals.

A Specific Example: A Specific Example: Early Childhood Development Early Childhood Development

Initiative in OntarioInitiative in Ontario Federal/Provincial funding of programs that broadly target early Federal/Provincial funding of programs that broadly target early

childhood development (0-6 years) and encourage different childhood development (0-6 years) and encourage different jurisdictions to develop their own approachesjurisdictions to develop their own approaches

In Ontario, 26 different provincial initiatives are fundedIn Ontario, 26 different provincial initiatives are funded Most initiatives have multiple service sites (e.g., one initiative has Most initiatives have multiple service sites (e.g., one initiative has

over 100 program sites)over 100 program sites) Within initiatives, each site may offer different services, or have a Within initiatives, each site may offer different services, or have a

different focusdifferent focus Types of interventions are very diverse. For example:Types of interventions are very diverse. For example:

Direct clinical treatment of childrenDirect clinical treatment of children Parent education and skill developmentParent education and skill development Pre- & post-natal servicesPre- & post-natal services Promotional campaigns, awareness-raisingPromotional campaigns, awareness-raising Social and practical supportSocial and practical support

A new emphasis on public sector A new emphasis on public sector accountability may lead to questions accountability may lead to questions like….like….

““Are funded Early Years programs in the Are funded Early Years programs in the province effective in improving early province effective in improving early childhood development?”childhood development?”

Beginning to answer “meta- questions”

New frameworks are needed that can lead to… ways of speaking to meta-level evaluation questions

like “to what degree did diverse, multi-site programs function as components in a larger coordinated plan?”

ways of comparing and consolidating the theories of change underlying diverse interventions with similar goals.

ways of comparing and consolidating the measurement approaches to diverse interventions.

What are the barriers?What are the barriers? Information overload and difficulty in organizing Information overload and difficulty in organizing

that information.that information. Information “underload” where evaluators are Information “underload” where evaluators are

twice and thrice removed from the field.twice and thrice removed from the field. Neither of the two “traditional responses” works:Neither of the two “traditional responses” works:

Large, centralized, standardized evaluation systems Large, centralized, standardized evaluation systems do not transfer well to the field; exceedingly difficult to do not transfer well to the field; exceedingly difficult to design with diverse programs.design with diverse programs.

Decentralized approaches which encourage each Decentralized approaches which encourage each program or community to do their own evaluation program or community to do their own evaluation often may promote poor quality evaluation designs; often may promote poor quality evaluation designs; makes very difficult to be summative across makes very difficult to be summative across programs.programs.

What are the solutions?What are the solutions?

Blended approaches, which emphasize Blended approaches, which emphasize shared theory of change and shared shared theory of change and shared definitions of key evaluation terms, but definitions of key evaluation terms, but allow for program-to-program and region-allow for program-to-program and region-to-region diversity in how measurement is to-region diversity in how measurement is carried out.carried out.

Using available technology to facilitate the Using available technology to facilitate the creation, understanding and ongoing creation, understanding and ongoing coherence of evaluation frameworks.coherence of evaluation frameworks.

Federal/Provincial/Territorial ECD Federal/Provincial/Territorial ECD AgreementAgreement

Four broad target areas:Four broad target areas: promote healthy pregnancy, birth and infancy;promote healthy pregnancy, birth and infancy; improve parenting and family supports;improve parenting and family supports; strengthen early childhood development, strengthen early childhood development,

learning and care; and learning and care; and strengthen community supports.strengthen community supports.

• Knowledge

• Skill

• Use of Services

• Parenting Behaviour

• Support

• Independence

• Use of Services

• Stress

• Early ID

• Behaviour

• Readiness

• Knowledge

• Collaboration

• Consistency

• Capacity

• Access

• Knowledge

• Awareness

• Commitment

• Support

Child Health

Family

Health

Community

Health

Service

System

• Parent Education

• Personal Support

• Assessment & Counselling

• Interactive Learning

• Clinical Care, Nutrition

• Screening & Assessment

• Referral

• Resources and Tools

• Networking

• Training

• Program Effectiveness

• Public Education

• Community Advocacy & Leadership

The Main Activities and Outcome Objectives of Early Childhood Development Initiatives

Activities

Outcome Objectives

Child Health

Family

Health

Community

Health

Service

System


• Interactive Learning

• Training• Public Education

• Knowledge• Skill• Parenting Behaviour• Support• Independence

• Use of Services• Behaviour• Readiness

• Knowledge• Collaboration• Consistency• Capacity• Access

• Knowledge• Awareness• Commit-

ment• Support

Examples of Preventive Activities and Associated Outcome Objectives in Childhood Development Initiatives

Child Health

Family

Health

Community

Health

Service

System

• Personal Support

• Assessment & Counselling

• Screening & Assessment

• Referral

• Use of Services


• Support

• Use of Services

• Early ID

• Behaviour

• Readiness

Example of Assessment and Treatment Activities and Associated Outcome Objectives in Early Childhood Development Initiatives

Child Health

Family

Health

Community

Health

Service

System


• Knowledge

• Skill

• Use of Services


• Stress

• Early ID

• Behaviour

• Readiness

Activities often have Outcomes at Multiple Levels

Implications for Practice Implications for Practice

Evidence Based

PlanningVision or Plan

Priority Goal Priority Goal

Outcome Evaluation

Activity or Program

Activity or Program

Activity or Program

Short-TermOutcome Objective

Short-Term Outcome Objective

Long-TermOutcome Objective



Beginning with Program MapsBeginning with Program Maps Initiative Focus Pregnancy, Birth and InfancyInitiative Components Direct Services

Activity Categories

Providing parent education & training (healthy pregnancy, prenatal nutrition, parenting transition, childbirth information, father involvement, etc.)

Providing screening & assessment Providing counseling Providing professionally led peer support groups

(e.g., prenatal topics) Providing personal & practical support (e.g,

lactation aids, baby loan depot) Providing referral information & hotline

Short-termOutcome Objectives

Improved knowledge of supports & services for healthy pregnancy & infant development

Increased use of other supports & services Improved parenting knowledge & practices Improved identification of at-risk parents, infants Increased feelings of family & social support

Long-Term Outcome Objectives

Increased parenting capacity Improved maternal health Improved infant health & well-being

Beginning with Program MapsBeginning with Program Maps

Promote children’s optimal development and readiness to learn, so that they may

reach their fullest potential

Improved child health &

physical well-being

AC

TIV

ITIE

S

SH

OR

T-

TE

RM

O

BJE

CT

IVE

S

GO

AL

S

LO

NG

-TE

RM

O

BJE

CT

IVE

S

Sample Program Logic Model

Program and System Supports

Developing & implementing

service coordination

policies, protocols

For all children to be healthy, safe, and secure within strong families and

communities

Pregnancy, Birth, and Infancy --- Parenting & Family Support

Increased accessibility of

pregnancy & child development intervention

programming

Improved school

readiness

Increased child

prosocial behaviours

Increased coordination & integration of services in community

Increased entry into appropriate

supports & services

Creating, disseminating

child development

resource materials

Developing networks of

service providers

Training of, consulting

with, service providers

Increased service provider

knowledge of pregnancy & child

development supports &

services/gaps

Increased service provider, parent, &

community awareness of the importance of the

early years

Increased parent &

community awareness of pregnancy &

child development

services

Promotional advocacy

campaigns

Ongoing parental

awareness media

campaigns

Organizing community

events

Improved knowledge of supports & services for

healthy pregnancy & infant/child

development

Providing referral info & hotline

Increased use of other supports &

services

Improved parenting

knowledge and practices (pre- & postnatal care, nutrition, child development,

parenting skills, health/safety)

Increased service provider

collaboration

Increased general

awareness of the importance

of parenting

Increased commitment,

support, programming

for EY services

Increased community

participation in EY &

prevention activities

Improved identification of at-risk parents,

infants and children (PPD,

P4, developmental

delay, etc).

Improved maternal

health

Improved infant health & well-being

Promote Healthy Pregnancy & the Birth of Healthy Babies

Provide small

learning groups for children

Providing parent

education & training

Providing professionally

led support groups for

parents

Providing child

screening & assessment

Providing counseling to parents

Providing personal, practical

support to parents

Evaluating provincial NP

services & community

impact

Increased capacity to

assess program

planning & effectiveness

Enhancing & expanding

core programming

Increased SP knowledge & skills

of pregnancy & child development best practices &

interventions

Increased child

stimulation &

interaction

Early Childhood Development,

Learning & Care

Providing parent

screening & assessment

Link to Map

Increased feelings of family & social

support

Increased parenting capacity

Improved emotional

functioning & well being of

children

Short-term Outcome Objectives

Improved knowledge of supports and services available

Improved knowledge of supports & services for healthy pregnancy & infant development

Increased use of supports & services Increased use of other supports & servicesIncreased parenting knowledge & practical skills Improved parenting knowledge & practicesIncreased early diagnosis & intervention Improved identification of at-risk parents, infantsIncreased social opportunities, networks & feelings

of support Increased feelings of family & social support

Coding Initiative MapsCoding Initiative Maps

Short-term Outcome Objectives

Improved knowledge of supports and services availableImproved knowledge of supports & services for healthy pregnancy & infant developmentIncreased use of supports & services Increased use of other supports & servicesIncreased parenting knowledge & practical skills Improved parenting knowledge & practicesIncreased early diagnosis & interventionImproved identification of at-risk parents, infants Increased social opportunities, networks & feelings of supportIncreased feelings of family & social support

Coding Initiative MapsCoding Initiative Maps

We can examine which initiatives share common activities and the different service approaches used...

ComponentComponent Early Childhood, Learning & CareEarly Childhood, Learning & CareDirect ServicesDirect Services

Child Child group group

programsprograms

Agency/Agency/centre/centre/

clinic-based clinic-based servicesservices

Drop-insDrop-ins Home visits Home visits & outreach& outreach

Activity Activity CategoriesCategories

PROVIDING INTERACTIVE LEARNING, ARTS, AND PROVIDING INTERACTIVE LEARNING, ARTS, AND PLAY OPPORTUNITIES PLAY OPPORTUNITIES Areas of focusAreas of focus: literacy, numeracy, homework clubs, arts & : literacy, numeracy, homework clubs, arts & crafts, play-based problem solving, empathy crafts, play-based problem solving, empathy

P1, P3, P1, P3, P4, P5, P4, P5, P8P8, , p10, p10, P6P6

P6P6, P3, , P3, P8P8

PROVIDING PHYSICAL ACTIVITY PROGRAMSPROVIDING PHYSICAL ACTIVITY PROGRAMS P6P6, P3, P3 P6P6

PROVIDING NUTRITION PROGRAMS PROVIDING NUTRITION PROGRAMS Areas of focusAreas of focus: nutrition awareness & education, meal & snack : nutrition awareness & education, meal & snack programsprograms

P1, P1, P6P6 P1, P1, P6P6

PROVIDING SCREENING, ASSESSMENT, & PROVIDING SCREENING, ASSESSMENT, & MONITORING MONITORING Areas of focusAreas of focus: physical health/nutrition, FAS/FAE, mental : physical health/nutrition, FAS/FAE, mental health & risk, development.health & risk, development.

P1, P3, P1, P3, P7P7, , P4, P9, p10, P4, P9, p10, P8,,

P9, P9, P11P11

PROVIDING COUNSELING, TREATMENT & CRISIS PROVIDING COUNSELING, TREATMENT & CRISIS INTERVENTIONINTERVENTIONAreas of focusAreas of focus: transition to school, FAS/FAE, mental health, : transition to school, FAS/FAE, mental health, immunization, well-child services, immunization, well-child services,

P2P2 P3, P3, P7P7, , P4, P4, P11P11, P10, P10

P2P2 P9 P9

CREATING LINKS TO OTHER RESOURCES, SERVICES & CREATING LINKS TO OTHER RESOURCES, SERVICES & PROGRAMSPROGRAMS

P2P2 P9 P9 P2P2 P9, P9, P11P11

AC

TIV

ITIE

S

SH

OR

T-

TE

RM

O

BJE

CT

IVE

SG

OA

LS

LO

NG

-TE

RM

O

BJE

CT

IVE

S

Overall Logic Model: Early Childhood Development, Learning & Care (column 3))

Decreased stress of children

Increased child awareness

(initiative specific – e.g., injury prevention)

GOALS

Increased early identification & diagnoses of

children

Miscellaneous initiative & system specific objectives

(see unique logic models)

Improved health &

physical well-being of children

Improved emotional functioning of children

Improved parent-child/fam

ily communication

Improved child

behaviour

Creating links to other resources,

services, & programs

Other direct

service columns

Providing physical activity programs

for children

Providing interactive

learning,arts, & play for children

Providing screening,

assessment, & monitoring for

children

Providing nutrition

programs to children

Providing counseling, treatment, &

crisis for children

Improved social skills,

communication & behaviour of

children

Improved cognitive ability of children

Increased use of other

supports & services

Go to Roll Up

Go to Parenting & Family Support Model

Go to Pregnancy, Birth, & Infancy Model

Go to Community, System, & Programming Supports Model

AC

TIV

ITIE

S

SH

OR

T-

TE

RM

O

BJE

CT

IVE

SG

OA

LS

LO

NG

-TE

RM

O

BJE

CT

IVE

S

Overall Logic Model: Early Childhood Development, Learning & Care (column 3))

Decreased stress of children

Increased child awareness

(initiative specific – e.g., injury prevention)

Increased early identification & diagnoses of

children

Miscellaneous initiative & system specific objectives

(see unique logic models)

Improved health &

physical well-being of children

Improved emotional functioning of children

Improved parent-child/fam

ily communication

Improved child

behaviour

Creating links to other resources,

services, & programs

Other direct

service columns

Providing physical activity programs

for children

Providing interactive

learning,arts, & play for children

Providing screening,

assessment, & monitoring for

children

Providing nutrition

programs to children

Providing counseling, treatment, &

crisis for children

Improved cognitive ability of children

Increased use of other

supports & services

Go to Roll Up

Go to Parenting & Family Support Model

Go to Pregnancy, Birth, & Infancy Model

Go to Community, System, & Programming Supports Model

Improved social skills,

communication & behaviour of

children

GOALS

Priority area: EARLY CHILDHOOD LEARNING & CARE

INITIATIVE

IMPROVED HEALTH & PHYSICAL WP21L-BEING OF CHILDREN

P1, P2, P12, P13, P6, P3, P4, P5, P9, P11, P19, P20, P21, P22, P8, P14, P9, P15, P16, P17, P18

IMPROVED EMOTIONAL FUNCTIONING & WP21L-BEING OF CHILDREN

P3, P7, P5, P4, P9, P8

IMPROVED COGNITIVE ABILITY

P2, P6, P3, P7, P21, P9, P11, P8, P9, P16, P17

IMPROVED SOCIAL SKILLS, COMMUNICATION, & BEHAVIOUR

P2, P6, P3, P7, P21, P9, P11, P8, P9, P16, P17

Long-term

outcome objectives

MISCP21LANEOUS INITIATIVE & SYSTEM SPECIFIC OBJECTIVES (e.g., increased early disclosure of sexual assault, greater immunization rates, decreases FAS incidence & infant morality, greater system independence)

P2, P7, P8, P15, P16, P17

We can examine which initiatives share common short-term and long-term objectives

IMPROVED HEALTH & PHYSICAL WELL-BEING OF CHILDREN

P1, P2, P12, P13, P6, P3, P4, P4, P9, P11, IP, P20, P21, P22, P8, P14, P10, P15, P16, P17, P18

IMPROVED EMOTIONAL FUNCTIONING & WELL-BEING OF CHILDREN

P3, P7, P4, P4, P9, P8

IMPROVED COGNITIVE ABILITY

P2, P6, P3, P7, P21, P9, P11, P8, P10, P16, P17

IMPROVED SOCIAL SKILLS, COMMUNICATION, & BEHAVIOUR

P2, P6, P3, P7, P21, P9, P11, P8, P10, P16, P17

Long-term outcome

objectives

MISCELLANEOUS INITIATIVE & SYSTEM SPECIFIC OBJECTIVES (e.g., increased early disclosure of sexual assault, greater immunization rates, decreases FAS incidence & infant morality, greater system independence)

P2, P7, P8, P15, P16, P17

We can move from the roll-up to the unique logic models of particular initiatives. These contain customized versions of shared objectives in the roll-up (in this example, P6)

Providing parenting services (information,

education and support)

Improved knowledge

of avail. supports& services

Increased parenting

knowledge and skill

Increased use of

services and supports


Providing a variety of targeted

programs for special

populations, including

teen parents

Providing ethnically and

culturally appropriate programs

Providing school

readiness programs

Providing programs which

support children’s healthy

development

Providing funding for community

based projects that meet identified

criteria

Providing ongoing support for

funded projects, and support for proposal development

Supporting local planning

and proposal review

Creating and running awareness campaigns

Creating community resources

Engaging local non-traditional

partners aschampions, funders

and committee members (e.g.,

business)

Improved child health and

physical well-being

Improved language and

cognitive skills of children

Improved social skills/competency

of children

Improved social maturity of

children

Improved communication

skills of children

Increased awareness of

avail community services

Increased community

participation in early years activities

Increased community collaboration/linkages

among service providers and other

supporters of parents

Increased new and locally-

created programs to fill gaps

Increased fit between (new and existing) services

and local priorities and

identified needs

Increased knowledge

among service partners about local gaps and

prioritiesIncreased

awareness in the community at large of the

importance of EY

Increased general awareness of the

importance of parenting

Improved sustainability of

services

Increased diversity of funding and

supports

Improved program quality

Decreased duplication of

services

More dense network of inter-

relationships among EY

services and other family supports

Increased commitment/supp

ort for EY services

Increased use of

services and

supports

Promote children’s optimal development

and readiness to learn, so that they may reach their fullest potential

Increased access to a more cohesive,

comprehensive range of EY programs,

services and parenting supports

Improved quality and capacity of existing children’s services

Stronger sense of shared philosophy for young children and

their families

Prenatal and Postnatal

(information, education and

support)

Increased opps. for children to participate in activities that

promote healthy development

Increased access to services for

targeted populations

Increased size and density of

families’ social

networks

Improved/smoother transitions to school

Increased linkages between service partners and alternative

resources

Reduced parent stress

Reduced child stress

Activities

Short term objectives

Long term

objectives

EY goals P6 goals

More dense network of inter-

relationships among EY

services and other family supports

Providing parenting services (information,

education and support)

Improved knowledge

of avail. supports& services

Increased parenting

knowledge and skill

Increased use of

services and supports


Providing a variety of targeted

programs for special

populations, including

teen parents

Providing ethnically and

culturally appropriate programs

Providing school

readiness programs

Providing programs which

support children’s healthy

development

Providing funding for community

based projects that meet identified

criteria

Providing ongoing support for

funded projects, and support for proposal development

Supporting local planning

and proposal review

Creating and running awareness campaigns

Creating community resources

Engaging local non-traditional

partners aschampions, funders

and committee members (e.g.,

business)

Improved child health and

physical well-being

Improved language and

cognitive skills of children

Improved social maturity

of children

Improved communication

skills of children

Increased awareness of

avail community services

Increased community

participation in early years activities

Increased community collaboration/linkages

among service providers and other

supporters of parents

Increased new and locally-

created programs to fill gaps

Increased fit between (new and existing) services

and local priorities and

identified needs

Increased knowledge

among service partners about local gaps and

prioritiesIncreased

awareness in the community at large of the

importance of EY

Increased general awareness of the

importance of parenting

Improved sustainability of

services

Increased diversity of funding and

supports

Improved program quality

Decreased duplication of

services

Increased commitment/supp

ort for EY services

Increased use of

services and

supports

Promote children’s optimal development

and readiness to learn, so that they may reach their fullest potential

Increased access to a more cohesive,

comprehensive range of EY programs,

services and parenting supports

Improved quality and capacity of existing children’s services

Stronger sense of shared philosophy for young children and

their families

Prenatal and Postnatal

(information, education and

support)

Increased opps. for children to participate in activities that

promote healthy development

Increased access to services for

targeted populations

Increased size and density of

families’ social

networks

Improved/smoother transitions to school

Increased linkages between service partners and alternative

resources

Reduced parent stress

Reduced child stress

Activities

Short term objectives

Long term

objectives

EY goals P6 goals

Improved social skills/competency

of children

has led to clearer and more accurate framework and terminology for summarizing early years activities and objectives.

has allowed us to describe activities in terms of both content and service delivery modality.

The ECD Visual Data BaseThe ECD Visual Data Base

allows us to talk about the focus of various types of work within the Early Years plan, not just presence and absence.

helps us to work towards cross-initiative consistency in understanding important outcome objectives and how they relate to one another.


allows ministry staff to quickly and easily understand individual initiatives and the connections between them.

contributes to a better understanding of how initiatives fit into the 4 priority areas the Federal/Provincial agreement.


Moving to measurementMoving to measurement

We have recently entered a phase that We have recently entered a phase that moves the framework towards moves the framework towards measurementmeasurement

We are repeating a process that is similar We are repeating a process that is similar to the generation of the program maps.to the generation of the program maps.

Each initiative is completing a table of Each initiative is completing a table of indicators, data items & sources, and indicators, data items & sources, and design/analysis recommendations, that design/analysis recommendations, that are linked to each objective are linked to each objective

Moving to measurementMoving to measurement

These tables will also be electronically These tables will also be electronically linked to the visual database (the roll-up linked to the visual database (the roll-up table and logic models)table and logic models)

ROLL UP OF ECDI INITIATIVES

Initiative Focus

Pregnancy, Birth, and Infancy

IMPROVED KNOWLEDGE OF SUPPORTS AND SERVICES AVAILABLE P9, P8, P3, P4, P14, P10, P18, P15, P16

INCREASED USE OF SUPPORTS AND SERVICES P1 P3, P4, P9, P14, P15, P10, P18, P16, P15 INCREASED PARENTING KNOWLEDGE AND PRACTICAL SKILLS:

Developmentally appropriate activities for children P23

pre/postnatal care P1 P13, P4, P15, P16, P18

parent & children nutritional needs, food preparation P1 P13, P8, P16, P18

ages and stages of child development P1 P13, P4, P8, P10, P23, P18

increased breastfeeding P1 P13, P10, P16, P18

child health and safety P1 P13, P4, P15, P16, P18

parent/child interaction P8, P10

behaviour management P8

language and cognitive development P8

child immunization P23

healthy lifestyle choices P3, P4

preventive dental health practices during pregnancy and early childhood P23

impact of addictions on pregnant women and their babies P4, P16

FAS/FAE (effects of alcohol/drugs, risk factors, and related issues) P4, P16 parental coping skills P10 INCREASED EARLY DIAGNOSIS AND INTERVENTION P3, P4, P10

Short-term

outcome objectives

INCREASED SOCIAL OPPORTUNITIES AND SOCIAL NETWORKS P16, P18

For a given initiative, we can move from a logic model to the roll-up, and from there to the indicator table to examine specific evaluation questions, methods, and measures. These may be transferable to initiatives sharing common objectives...

Outcome Objective Monitoring and Evaluation Questions

Performance indicator

Increased parenting knowledge & practical skills (pre- & postnatal care, infant/child health)

q1. What health & wellness behaviours were women more aware of and changes made post-partum re: safety nutrition, healthy growth, development? (H-N)

Women’s self-reports of their knowledge of healthy behaviours (H-N)Women’s self-reports of their health-related behaviours (H-N, PER)

Women’s self-reports of their knowledge & skills re: early child care & development(PER)

Clients’ self-reports of their knowledge of early childhood developmental milestonesorClients’ level of knowledge of early childhood developmental milestones(SUD)

Women’s self-reports of choosing healthy lifestyle behaviours during pregnancy (E-ST, H-N, SUD)

Table of Indicators, Measurement, and Design

CurrentDataitem(s)

Current data collection &managementmechanism(s)

Analysis plan for current data, gaps

Recommendations, feasibility, importance, gaps

Responses on “Client Behaviour Survey” (H-N)

Data gathered by NPs, in a pre-post design (H-N)

(no measure for PER)

Change may not be relatively uniform if services vary

Need details on ClientBehaviour Survey

Questionnaire responses

Ongoing data collection by NPs

Is this pre-post?

Two indicators provided here b/c it is unclear if the data item gathers self-reports of knowledge, or content (test-based) knowledge.

Table of Indicators, Measurement, and Design

The Indicator libraryThe Indicator library

This current phase of developing indicators, This current phase of developing indicators, examining current evaluation data, and examining current evaluation data, and proposing future evaluation activities has led to proposing future evaluation activities has led to the development of an “indicator library”.the development of an “indicator library”.

The indicator library is analogous to our The indicator library is analogous to our development of the roll up. While the roll-up development of the roll up. While the roll-up summarized ECD activities and objectives, the summarized ECD activities and objectives, the indicator library summarizes measurement and indicator library summarizes measurement and design across all the initiatives.design across all the initiatives.

facilitates cross-site collaboration within facilitates cross-site collaboration within initiatives but also collaboration across initiatives but also collaboration across initiatives.initiatives.

allows for the possibility of consistency of allows for the possibility of consistency of measurement across programs that share measurement across programs that share common objectives, programs models, common objectives, programs models, and populations.and populations.

fosters best practices in evaluation. fosters best practices in evaluation.

The Indicator libraryThe Indicator library

The lessons learnedThe lessons learned We began by building consistent and concrete We began by building consistent and concrete

theories of changetheories of change across diverse situations, across diverse situations, rather than trying to impose a consistent rather than trying to impose a consistent measurement system.measurement system.

We did this work with local programs and broad We did this work with local programs and broad province-wide systems province-wide systems at the same timeat the same time and and used each level to clarify the other.used each level to clarify the other.

This strategy has allowed us to use an This strategy has allowed us to use an incremental approachincremental approach to improving to improving measurement and reporting practices at many measurement and reporting practices at many levels and over time, which is cost-effective and levels and over time, which is cost-effective and builds ownership. builds ownership.

Overall meta logic models(1 for each priority area)

Initiative map

Initiative map

Initiative map

Initiative logic model



Initiative sites Initiative sites Initiative sites

Outcome evaluation frameworks & measurement

Local, flexible site level

evaluations

Provincial level evaluations of

initiatives

The Roll-up of activities and objectives

ABSTRACT Developing evaluation frameworks for large, multi-program projects: Building logic...

Documents

Transcript of ABSTRACT Developing evaluation frameworks for large, multi-program projects: Building logic...