PROGRAM EVALUATION:KEEPING SCORE TO DEMONSTRATE

IMPACT

Molly Chamberlin, Ph.D. Indiana Youth Institute

9-28-2011

WHY IS EVALUATION IMPORTANT?

Accountability Progress and results More effective/efficient service Communication opportunities Focuses staff effort Engages employees throughout

organization Funding/resources

GOOD MEASURES PROVIDE:

Means to see if strategic plan is working

Focus attention on what matters most Common language within agency Measurement of outcomes, not just

outputs Systems for verifying results Support for funding requests

IMPACT CYCLE Mission >

Goals > Inputs >

Activities > Outputs >

Outcomes

PROGRAM LOGIC MODEL

KEYS TO SUCCESS

Evaluation / assessment is NOT a one-time event

Evaluation starts at the beginning of the project (not at the end)

Assignment of responsibility for process, particularly data analysis

Involve all levels of staff within your organization

ARE YOU READY?

Indicator Is there support from organizational leadership? Are there clearly written goals and objectives? Are the activities of the program clearly outlined? Is there a person or a group of people who are responsible

for leading evaluation efforts? Have financial and/or human resources been allocated for

evaluation efforts? Do all staff members agree upon the written goals and

objectives for the program? Do all staff members consider evaluation activities part of

their job requirements? Do other stakeholders (board, service recipients,

volunteers) value evaluation?

STEPS IN DESIGNING YOUR PLAN

Assign responsibility

Create a timetable

Communicate with staff

Use any and all available resources

REMEMBER AS YOU PLAN TO:

Be clear Be concise Remember that it all hasn’t happened

yet – adjustments to the process can (and should) be made as you learn from results

PROGRAM LOGIC MODEL AS PLAN FOR PROGRAM EVALUATION

Resources (Inputs)

Activities Outputs

Outcomes(immediate, short, long-

term)

WHAT IS YOUR PROGRAM GOAL?

GOAL: A big picture statement of what the program wants to accomplishGoal: Parents and caregivers possess the knowledge and skills to ensure safe and healthy development for infants and children

 Inputs

 

 Activities

 Outputs

 Outcomes

 

Success Indicators 

WHAT ARE YOUR PROGRAM OUTCOMES?

OUTCOMES: What does your program want to CHANGE?

 Inputs

 

 Activities

 Outputs

 Outcomes: Expected observable and

measurable results; usually a change in behavior, knowledge, attitude, or skill

 Increased parent/caregiver knowledge in specific areas Increased parent/caregiver practical skills in specific areas

Success Indicators 

WHAT ARE YOUR ACTIVITIES?

ACTIVITIES: What will the program DO in order to achieve the desired outcomes?

Inputs 

Activities: Actions or strategies employed to achieve desired

outcomes (if we do this, then the outcome)

Outputs 

Outcomes: Expected observable and measurable results;

usually a change in behavior, knowledge, attitude, or skill

 Survey parents and local providers to identify specific areas of need/knowledge and skill gaps

Provide online and local in-person training for parents and child care providers in specific areas of need Develop and administer grants for provider training (“train the trainer” models) in specific areas of need

 Increased parent/caregiver knowledge in specific areas Increased parent/caregiver practical skills in specific areas

Success Indicators

WHAT ARE YOUR OUTPUTS?

OUTPUTS: What will the activities produce or create? (What will our clients do as a result of our activities?

Inputs 

Activities: Actions or strategies employed to achieve desired outcomes (if we do this, then

the outcome)

Outputs: what will clients do as a result of our activities?

 Outcomes: Expected observable

and measurable results; usually a change in behavior, knowledge, attitude, or skill

 Survey parents and local providers to identify specific areas of need/knowledge and skill gaps

Provide online and local in-person training for parents and child care providers in specific areas of need Develop and administer grants for provider training (“train the trainer” models) in specific areas of need

Parents/caregivers participate in surveyParents/caregivers participate in online trainingsParents/caregivers participate in on-site trainingsProviders apply for training grantsProviders receive training grantsParents/providers are satisfied with training opportunities

 Increased parent/caregiver knowledge in specific areas Increased parent/caregiver practical skills in specific areas

Success Indicators

WHAT ARE YOUR RESOURCES (INPUTS)?

INPUTS: What resources will you use to conduct your activities?

Inputs: Resources (usually people/funds)

used in program development and implementation

 Activities: Actions or strategies employed to

achieve desired outcomes (if we do this, then the outcome)

Outputs: what activities will produce or create (quantity and/or quality of activities)

 Outcomes: Expected

observable and measurable results;

usually a change in behavior, knowledge, attitude, or skill

2 FTE Training Staff

.5 FTE Training Coordinator

1 FTE Grant Coordinator

Funding for trainings

Funding for training staff

Funding for grants

 Survey parents and local providers to identify specific areas of need/knowledge and skill gaps

Provide online and local in-person training for parents and child care providers in specific areas of need Develop and administer grants for provider training (“train the trainer” models) in specific areas of need

Parents/caregivers participate in surveyParents/caregivers participate in online trainingsParents/caregivers participate in on-site trainingsProviders apply for training grantsProviders receive training grantsParents/providers are satisfied with training opportunities

 Increased parent/caregiver knowledge in specific areas Increased parent/caregiver practical skills in specific areas

Success Indicators

HOW DO YOU QUANTIFY/MEASURE YOUR OUTCOMES?

SUCCESS INDICATORS: How do we know if we’re on track to achieve or achieved our outcomes?

Inputs: Resources (usually people/funds) used in

program development and implementation

 Activities: Actions or strategies

employed to achieve desired outcomes (if we do this, then the

outcome)

Outputs: what activities will produce or create (quantity and/or quality of activities)

 Outcomes: Expected observable

and measurable results; usually a change in behavior, knowledge, attitude, or skill

2 FTE Training Staff

.5 FTE Training Coordinator

1 FTE Grant Coordinator

Funding for trainings

Funding for training staff

Funding for grants

 Survey parents and local providers to identify specific areas of need/knowledge and skill gaps

Provide online and local in-person training for parents and child care providers in specific areas of need Develop and administer grants for provider training (“train the trainer” models) in specific areas of need

Parents/caregivers participate in surveyParents/caregivers participate in online trainingsParents/caregivers participate in on-site trainingsProviders apply for training grantsProviders receive training grantsParents/providers are satisfied with training opportunities

 Increased parent/caregiver knowledge in specific areas Increased parent/caregiver practical skills in specific areas

By 12/14, obtain at least $50,000 for trainings

By 12/14, obtain at least $10,000 for grants

By 10/14, hire or identify training staff and training coordinator

By 10/14, hire or identify grant coordinator

n/aBy 3/30, 40% of served parents and providers will participate in the survey (survey records)By 5/15, 20 online/5 onsite trainings will be offered (program records)By 5/15, 50% of served parents and local providers will participate in trainings (training logs)By 5/15, 20 applications will be submitted (program records)By 6/15, 17 applications will be approved (program records)By 6/15, 90% of participants will express satisfaction with trainings (paarticipant surveys)

By 4/30, 100% of trainings will meet established quality standards (expert content review) By 7/31, 85% of participants will pass a knowledge assessment in specified areas (knowledge assessments) By 7/31, 90% of participants will report increased knowledge and practical skills in specified areas (participant surveys)

WHAT’S NEXT?

Implement Repeat Communicate

Staff Board Funders Volunteers Community

MIS-STEPS TO AVOID

Collecting data before you have set goals and identified outcomes

Collecting too much data Using data collection tools or measurements that don’t

match the goals and outcome Attempting without leadership support Assigning to staff member(s) regardless of time and skills Using only goals required by funders Identifying outcomes that are not feasible or not connected

to activities and organizational/program mission Overcomplicating the evaluation

EVALUATION RESOURCES

On-line Resources Management Library http://managementhelp.org Kellogg Foundation http://www.wkkf.org University of Wisconsin Extension www.uwex.edu Children Youth and Families Education and

Research Network www.cyfernet.org Innovation Network www.innonet.org Capacity Building Resource Library

www.strengtheningnonprofits.org IYI Resources

IYI Kids Count Data Report www.iyi.org IYI Virginia Beall Ball Library www.iyi.org IYI Consulting Services and links www.iyi.org

THANK YOU!

Contact:Molly Chamberlinmchamberlin@iyi.org(317) 396-2718