Evelyn GonzalezProgram Evaluation

AR Cancer Coalition Summit XIVMarch 12, 2013

MAKING A DIFFERENCEEvaluating Programmatic Efforts

Overview of evaluationDefining SMART objectives for your

goalsKnow how to use different methods of

evaluation Be more willing to evaluate your

efforts

OBJECTIVES

…the systemati c collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program eff ectiveness, and/or inform decisions about future programming.

(Patt on, Uti lizati on Focused Evaluati on, 1997)

WHAT IS PROGRAM EVALUATION?

Did the program/intervention

work?

Was it worth it?

What worked; what didn’t?

Who did we reached?

Did we get our monies worth?

WHY EVALUATE

WHEN SHOULD WE BEGIN EVALUATION?

An evaluation plan is the “blueprint”

What will be evaluated What information will be

collected When will it be collected

What will be done with the results

EVALUATION PLAN

CDC FRAMEWORK FOR EVALUATION

• Collect

• Analyze Data

START

• Implementation of the program

• Gather data as you go

• Monitor

Planning

• Establish Goals & Objectives

• Establish baseline •Identify an Evidence- Base Program (EBP)

Evaluation

• As you implement

• End of program

Community/Audience

Stakeholders

Planning Phase

Implementation

Phase

Evaluation

Phase

Involve Stakeholders

Share results with Community & Stakeholders

The “grand reason” for engaging in your public health effort

Span 3 or more years

State the desired end result of the program.

GOALS: DEFINITION

More specific than goals.

They state how the goals will be achieved in a certain timeframe.

Well written objectives are SMART:

Specific

Measurable

Achievable

Realistic and Relevant

Time-framed

OBJECTIVES: DEFINITION

Specific

Who are you reaching (priority audience)?

What intervention will you use?

Where, setting

S.M.A.R.T.

Measurable

Dosing, how many times will you do the intervention

What is the expected outcomeIncrease of X% following the intervention

Decrease of smoking by X%

S.M.A.R.T.

AttainableIs your intervention feasible?

Realistic and RelevantDoes the objective match the goal?

Is it evidence-based program (EBP)?

S.M.A.R.T.

Time-framedBy when do you anticipate the change?End of the session

3,6,9 months

5 years

S.M.A.R.T.

You are working on an intervention that will increase awareness about breast cancer risk

Objective 1: Participants will be aware of the major risk factors for developing breast cancer.

How can this be re-written to be SMART?

SMART OBJECTIVE EXERCISE

Original:Participants will be aware of the major risk factors for developing breast cancer.

SMART Objective:Upon post test following the intervention, participants will be able to identify 3 major risk factors for developing breast cancer.

SMART OBJECTIVE EXERCISE

Original:This program will increase screening for colorectal cancer in Arkansas.

SMART:Colorectal screening will be increased by 5%, over the prior year for age appropriate males in Arkansas.

RE-WRITTEN:

Objective 1: Public Education for Breast Cancer Screening –

Increase knowledge and improve attitudes of all women withregards to the importance of breast cancer screening

Strategy 1 – Promote campaigns to educate the public about the importance of mammography.

Action 1 – Increase awareness among all women 40 and older of the importance of regular breast cancer screening

GOAL: PROMOTE AND INCREASE THE APPROPRIATE UTILIZATION

OF HIGH-QUALITY BREAST CANCER SCREENING

Planning—Develop the questions, consult with the program stakeholders or resources, make a timeline

Data Collection—Pilot testing. How will the questions be asked? Who will ask them?

Data Analysis—Who will analyze the data and how?

Reporting—Who will report and how? Who will receive the data and when? How will it affect the program

Application—How could your results be applied in other places?

THE EVALUATION PROCEDURE

Look at the evaluation methods used in the original EBP.

When discussing evaluation, think about these questions:What is important to know?What do you need to know versus what is nice to know?

What will be measured and how?How will this information be used?

PLANNING FOR EVALUATION

Indicators or measures are the observable and measurable data that are used to track a program’s progress in achieving its goals.

Monitoring (program or outcome monitoring, for example) refers to on-going measurement activity

SOME DEFINITIONS…

Process evaluation can find problems early on in the program. It includes an assessment of the staff, budget review, and how well the program is doing overall.

For this kind of evaluation, it may be useful to keep a log sheet to record each of your activities.

From Windsor et al., 1994

PROCESS EVALUATION

Impact evaluation can tell if the program has a short-term effect on the behavior, knowledge, and attitudes of your population.

It also measures the extent to which you have met your objectives.

From Green and Kreuter, 1991

IMPACT EVALUATION

Outcome evaluation looks to see if the long-term program goals were met.

These goals could be changes in rates of illness or death, as well as in the health status of your population.

From McKenzie & Smeltzer, 1997

OUTCOME EVALUATION

Identify Program Goals

For each goal: Identify Process Objectives

Identify Outcome Objectives

For each objective: Identify Indicators

Identify Data Source

Plan Data Collection

Plan Data Analysis

APPLICATION TO YOUR PROGRAM:

DATA COLLECTION METHODS

SurveysInterviewsFocus GroupsObservationDocument Review

You may develop a way to compare the baseline data from the needs assessment with the final outcome of your program.

Pre/Post survey in an education session.

This will let you see if you have achieved your objectives.

PRE- AND POST-EVALUATION

Primary sources Quantitative:

Surveys/questionnaires Qualitative: Focus groups, public

meetings, direct observation Qualitative: In-depth interviews

with community leaders, interviews with other program planners.

INFORMATION COLLECTION

Will depend on which EBP/Intervention selected

Answer these questions: What specific behaviors do I want my audience to acquire or enhance?

What information or skills do they need to learn to act in a new way?

What resources do I need to carry out the program?

What methods would best help me meet my objectives?

STRATEGIES

USING MIXED DATA SOURCES/METHODS

Involves using more than one data source and/or data collection method.

Your objectives should be measurable so that they can be evaluated.

The evaluation should be in line with your objectives.

Try not to make up new things to evaluate.

PROGRAM OBJECTIVES AND EVALUATION

You may want to do a pilot test in order to evaluate the effect of your program.

A pilot test is a practice run using a small group who are similar to your target audience.

PILOT TESTING

Evidence-based programs have already done some type of evaluation.

Look to see how the program was evaluated before. Try to use the same methods.

You do not have to evaluate everything!

REPLICATING THE EVALUATION

MONITORING PROGRESS

NOW THAT YOU’VE COLLECTED THE DATA, WHAT DO YOU DO WITH IT?

Analyzing data Who When How

Interpretation of results and sharing findings

Must be able to answer this!

Do not just look for the good outcomes

Learn from what didn’t work

Share both the positive and negative outcomes

SO WHAT?

DEVELOPING RECOMMENDATIONS

Your evaluation’s recommendations should be:

Linked with the original goals/SMART objectives.

Based on answers to your evaluation questions.

Should have stakeholder input

Tailored to the end users of the evaluation results to increase ownership and motivation to act.

SHARING RECOMMENDATIONS

CommunityExecutive SummaryFinal ReportNewsletter article(s)Website articleTown hall

meeting(s)Radio interviewsLocal newspapers

Institution & YourselfExecutive SummaryFinal ReportJournal articlesProfessional

conferencesPoster sessionsMeetings with

colleagues

TIPS & CONSIDERATIONS

Consult with partners with evaluation experience

Budget 10-15% for evaluation Staffing Build a database Analysis

Consider pilot testing your programPilot test your evaluation method &

tool(s)

Trust yourself. You know more than you think you

do!

Benjamin Spock