11

Nemours Health Nemours Health Planning and Evaluation Planning and Evaluation

CollaborativeCollaborative

Learning Session I:Learning Session I:

Designing Effective Health Designing Effective Health ProgramsPrograms

22

Learning ObjectivesLearning Objectives

How to plan child health programs How to plan child health programs with outcomes in mindwith outcomes in mind

How to apply sound theory to the How to apply sound theory to the planning of child health programsplanning of child health programs

How to develop a program logic How to develop a program logic model that specifies program model that specifies program outcomesoutcomes

33

Planning with outcomes in Planning with outcomes in mindmind

44

What do we mean by What do we mean by outcomes?outcomes?

Outcomes are changes in the people, Outcomes are changes in the people, organizations, or systems that we organizations, or systems that we aim to serveaim to serve

55

What do we mean by What do we mean by outcomes?outcomes?

Changes in peopleChanges in people– KnowledgeKnowledge– Attitudes & beliefsAttitudes & beliefs– SkillsSkills– BehaviorBehavior– Health statusHealth status

66

What do we mean by What do we mean by outcomes?outcomes?

Changes in organizationsChanges in organizations– VisionVision– MissionMission– StrategiesStrategies– CapacityCapacity– CapabilitiesCapabilities– External RelationshipsExternal Relationships– Internal CultureInternal Culture

77

What do we mean by What do we mean by outcomes?outcomes?

Changes in systemsChanges in systems– System capacitySystem capacity– System costSystem cost– System organizationSystem organization– System coordinationSystem coordination– System qualitySystem quality– System accessibilitySystem accessibility

88

What does it mean to “plan What does it mean to “plan with outcomes in mind?”with outcomes in mind?”

Begin by specifying what outcomes Begin by specifying what outcomes we want to achieve for people, we want to achieve for people, organizations, or systemsorganizations, or systems

Work backward to design a Work backward to design a program that could achieve these program that could achieve these outcomesoutcomes

Make sure we are operating under Make sure we are operating under sound program theorysound program theory

99

Program theoryProgram theory

1010

What is “program theory?”What is “program theory?”

Program theory is the set of causal Program theory is the set of causal assumptions between your inputs, assumptions between your inputs, activities, outputs, and outcomesactivities, outputs, and outcomes

In other words:In other words:– What makes you so sure this will work?What makes you so sure this will work?

1111

Why is program theory Why is program theory important?important?

Every child health program in the Every child health program in the world has a program theoryworld has a program theory– Some are more explicit than othersSome are more explicit than others– Some are more sound than othersSome are more sound than others– Flawed program theory is one of four Flawed program theory is one of four

basic reasons why programs fail:basic reasons why programs fail: Flawed program theoryFlawed program theory Flawed program designFlawed program design Flawed program implementationFlawed program implementation Flawed program evaluationFlawed program evaluation

1212

Examples of program theoryExamples of program theory

Discuss whether the following statements Discuss whether the following statements are sound or unsoundare sound or unsound– If you immunize a child against MMR, then that If you immunize a child against MMR, then that

child will avoid MMRchild will avoid MMR– If you educate adolescents about the risks of If you educate adolescents about the risks of

tobacco, they will avoid smokingtobacco, they will avoid smoking– If you provide a family with Medicaid, they will If you provide a family with Medicaid, they will

take their children in for EPSDT visitstake their children in for EPSDT visits– If you teach a family how to make better food If you teach a family how to make better food

choices, their children will better control their choices, their children will better control their weightweight

1313

Logic modelsLogic models

1414

What is a logic model?What is a logic model?

In its simplest conception, it is simply In its simplest conception, it is simply a graphic or tabular description of a graphic or tabular description of your program. your program.

Looking deeper, the best logic Looking deeper, the best logic models illustrate the underlying models illustrate the underlying theory of the program, or "program theory of the program, or "program theory."  theory." 

1515

Acme Child Health Enrollment ProgramAcme Child Health Enrollment Program

AssumptionsAssumptions InputsInputs ActivitiesActivities OutputsOutputs OutcomesOutcomes

Theoretical Theoretical assumptions assumptions about why a about why a program will program will workwork

The resources The resources needed to needed to deliver the deliver the programprogram

Key actions of Key actions of program staff program staff and clientsand clients

Products of the Products of the programprogram

Changes in the Changes in the target target populationpopulation

Uninsured children Uninsured children who are eligible who are eligible but not enrolled but not enrolled will become will become enrolled ifenrolled if

their parents/ their parents/ caregivers:caregivers:Know they are Know they are eligibleeligibleValue coverageValue coverageKnow how to Know how to enrollenrollHave assistance Have assistance with enrollment with enrollment processprocessComplete Complete enrollment processenrollment process

10 CBOs10 CBOs20 outreach 20 outreach workersworkers2 training experts2 training expertsClient families Client families with childrenwith children$5,000 mini-$5,000 mini-grant to each grant to each organizationorganization

Train CBO staffTrain CBO staff

CBO staff:CBO staff:Interview Interview parents/ parents/ caregivers about caregivers about child health child health insurance statusinsurance statusEducate about Educate about Medicaid-S-CHIPMedicaid-S-CHIPOffer assistance Offer assistance with applicationwith applicationGuide through Guide through application application processprocessMonitor Monitor enrollmentenrollment

Parents/ Parents/ caregivers of 500 caregivers of 500 children children interviewedinterviewed100 educated 100 educated and offered and offered assistanceassistance70 accept 70 accept application & application & assistanceassistance50 are enrolled50 are enrolled

Families have Families have peace of mindpeace of mind

Children have Children have access to needed access to needed health serviceshealth services

Children have Children have better health better health statusstatus

Target Population: Uninsured children eligible for Medicaid or S-CHIP in Anytown, USA

1616

Acme Child Health Enrollment ProgramAcme Child Health Enrollment Program

AssumptionsUninsured children who are eligible but not enrolled will become enrolled iftheir parents/ caregivers:•Know they are eligible•Value coverage•Know how to enroll•Have assistance with enrollment process•Complete enrollment process

10 CBOs with total of 20 outreach workers

2 training experts

Inputs

$5,000 mini-grant to each organization

Client families with children

Target PopulationUninsured children eligible for Medicaid or S-CHIP

Interview parents/ caregivers

Educate eligible parents/

caregivers

Activities

Train outreach workers

Provide application assistance

500 interviewed

100 educated

Outputs

50 enrolled

70 accept assistance

Families have peace of mind

Children have access to needed services

Outcomes

Children have better health status

20 trained

Monitor enrollment

1717

Why are funders so Why are funders so interested?interested?

Logic models promote program improvementLogic models promote program improvement– The process of constructing a logic model forces the The process of constructing a logic model forces the

funded organization to develop more theoretically sound funded organization to develop more theoretically sound programsprograms

Logic models are efficient Logic models are efficient – When properly constructed, a logic model can convey When properly constructed, a logic model can convey

the basic blueprint of a program in one or two pages.  the basic blueprint of a program in one or two pages.  Logic models promote understanding and Logic models promote understanding and

communicationcommunication– Once funding organizations adopt the terminology of Once funding organizations adopt the terminology of

logic models -- terms like assumptions, inputs, activities, logic models -- terms like assumptions, inputs, activities, outputs, outcomes -- it gives them a common for outputs, outcomes -- it gives them a common for understanding and discussing the relative strengths and understanding and discussing the relative strengths and weaknesses of diverse programs.  weaknesses of diverse programs. 

1818

Why are funders so Why are funders so interested?interested?

Logic models provide the foundation for practical Logic models provide the foundation for practical program evaluation program evaluation – A properly constructed logic model helps the A properly constructed logic model helps the

funder identify the most important focal points for funder identify the most important focal points for process evaluation and outcome evaluation.  process evaluation and outcome evaluation. 

Logic models promote organizational learningLogic models promote organizational learning– As funders review and compare logic models, they learn As funders review and compare logic models, they learn

valuable information about the theory and practice of valuable information about the theory and practice of health and human service programming.  health and human service programming. 

– This helps the funder make better decisions and become This helps the funder make better decisions and become a more valuable partner in their ongoing relationship a more valuable partner in their ongoing relationship with community health and human service programs.with community health and human service programs.

1919

Why are grantees so Why are grantees so interested? interested?

Because funders are interested!Because funders are interested!– Programs with a well constructed logic model Programs with a well constructed logic model

may give themselves a competitive advantage in may give themselves a competitive advantage in fundraising.  fundraising. 

Because logic models improve program Because logic models improve program planningplanning– The process of constructing a logic model forces The process of constructing a logic model forces

programs to ask themselves a number of programs to ask themselves a number of questions about their target population, questions about their target population, assumptions, desired outcomes, etc.  To the assumptions, desired outcomes, etc.  To the extent that these questions are raised and extent that these questions are raised and answered during the planning phase, the chances answered during the planning phase, the chances of successful implementation are increased. of successful implementation are increased. 

2020

Why are grantees so Why are grantees so interested? interested?

Because logic models support program Because logic models support program implementation.  implementation.  – Many organizations that develop logic models for Many organizations that develop logic models for

funders also find that periodic review of the logic model funders also find that periodic review of the logic model in staff or board meetings can help the organization stay in staff or board meetings can help the organization stay focused on its mission assure that its resources are focused on its mission assure that its resources are being directed toward the most important outcomes.  being directed toward the most important outcomes. 

Because logic models support practical program Because logic models support practical program evaluationevaluation– By using the logic model as the basis for evaluation By using the logic model as the basis for evaluation

objectives, both the funder and the funded organization objectives, both the funder and the funded organization can assure themselves that the evaluation is focused on can assure themselves that the evaluation is focused on the most essential aspects of the program.the most essential aspects of the program.

2121

If you can answer these If you can answer these questions..questions..

– Who is your target population?  Who is your target population?  – What outcomes, or personal changes do you What outcomes, or personal changes do you

want to help this population achieve? want to help this population achieve? – What level of output do you hope to achieve, in What level of output do you hope to achieve, in

terms of people served, products produced, etc.  terms of people served, products produced, etc.  – What activities will be required to produce these What activities will be required to produce these

outputs?  outputs?  – And what inputs, or resources, will be required to And what inputs, or resources, will be required to

complete the necessary activities?  complete the necessary activities?  – What are the theoretical assumptions that lead What are the theoretical assumptions that lead

you to believe that people exposed to this you to believe that people exposed to this program will achieve the desired outcomes? program will achieve the desired outcomes? 

2222

..you can construct a logic ..you can construct a logic model!model!

A logic model is simply a way of A logic model is simply a way of illustrating what you should already illustrating what you should already know about your program or planknow about your program or plan

The most important purpose of the The most important purpose of the logic model is program improvement.logic model is program improvement.– If, in the course of constructing a logic If, in the course of constructing a logic

model, you identify gaps in your model, you identify gaps in your knowledge or planning, you simply knowledge or planning, you simply problem-solve until you can fill those gaps problem-solve until you can fill those gaps as best you canas best you can

2323

Constructing your own Constructing your own program theory and logic program theory and logic

modelsmodels(Transition to Logic Model Development Guide)(Transition to Logic Model Development Guide)