What Works in Multi-Site What Works in Multi-Site Evaluations of Nutrition Evaluations of Nutrition Education Interventions?Education Interventions?

What Works in Multi-Site What Works in Multi-Site Evaluations of Nutrition Evaluations of Nutrition Education Interventions?Education Interventions?

Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie KeihnerCalifornia Nutrition Network for Healthy Active Families

Research and Evaluation Unit

California Nutrition Network California Nutrition Network for Healthy, Active Familiesfor Healthy, Active Families

Targets low-income families

Conducts a social marketing campaign to increase: fruit and vegetable

consumption physical activity promote the Food Stamp

Program Contracts over 170

agencies Receives almost 106

million from USDA food stamp program

Strategies

Factors thatInfluence

FVC

Change inBehavior

Evaluation for contractors Evaluation for contractors receiving >$350kreceiving >$350k

Process

Short-term

ImpactLong-term

impact

Network Developed Network Developed MethodologyMethodology

1. SOW Impact evaluation objective2. Compendium of Surveys 3. Impact Evaluation Handbook 4. Trainings and workshops 5. Data entry templates 6. Evaluation report template7. Monitoring

Scope of Work Impact Scope of Work Impact Evaluation ObjectiveEvaluation Objective

Impact objective: By Sep. 30 20xx, a sample of 4th and 5th grade students at intervention sites will report an increase in fruit and vegetable consumption and an increase in one or more factors related to fruit and vegetable consumption, such as knowledge, preferences, outcome expectations and self-efficacy.

Impact Evaluation Impact Evaluation HandbookHandbook

Compendium of SurveysCompendium of Surveys

TrainingsTrainings

WorkshopsWorkshops

One-on-one TAOne-on-one TA

Data Entry TemplatesData Entry Templates

Created in MS ExcelProvides same results as SPSSCalculate pretest mean, posttest

mean, difference and p-value for a paired t-test

Customized for each survey

Data Entry TemplatesData Entry Templates

Data Entry TemplatesData Entry Templates

Evaluation Report TemplateEvaluation Report Template

18 questionsDesignMethodsResultsReflectionSuccesses

Report Template - Report Template - DesignDesign

1. Please type your impact objective. 2. What were the factors you measured?3. What intervention and strategies were implemented

to change the factors? 4. What survey tool did you use? 5. Were there questions on the survey tool that were

not addressed in the intervention? If so, why?6. Describe challenges faced during the intervention

implementation. 7. What type of design was used, e.g., pretest and

posttest, pre-posttest with a control, posttest only or other?

Report Template - Report Template - Evaluation Survey ImplementationEvaluation Survey Implementation

8. What type of design was used, e.g., pretest and posttest, pre-posttest with a control, posttest only or other?

9. Who were the survey respondents? (please give age range)

10. Where were they administered (name and type of locations, e.g., health department, community center, mobile home park, etc.)

11. When were the pretest and posttest surveys administered? (Please give dates.)

12. How many were administered? (If a different number of pretests and posttest please explain.)

13. How long did it take the respondents to answer the survey? (Please give range in minutes.)

Report Template - Report Template - Evaluation ResultsEvaluation Results

14. What were the results? How would you interpret these results? How might these results be used?

15. How would you interpret these results?

16. How might these results be used?

Report Template - Report Template - ReflectionReflection

17. What were the big challenges you encountered during the evaluation process?

18. Now, reflect on the successes you’ve had over the past year. Describe the best moment you had as a nutrition educator this year. Think about a moment when you knew your nutrition education had made a difference in someone’s life or give an example of a time when you were proud to be a nutrition educator.

19. Share a story about a one of the best moments you had while conducting the evaluation this year? Think about one of the highlights?

20. If Aladdin appeared with his magic lamp and offered you a wish to have more of those best moments or highlights what would you ask for?

Monitoring sheet (in MS Monitoring sheet (in MS Excel) includes columns for:Excel) includes columns for:

Contractor Name Contact Name Region Channel Federal Share Program Manager Required/Volunteer Contract External Evaluator Number of years in impact evaluation

Monitoring sheet Monitoring sheet (continued)(continued)

Evaluation Objective Factors (to be)/(that were) measured Design (posttest only, posttest with control, pre-

posttest, pre-post with control) Participant’s age (grade, adults/youth, in years) Survey Used Have a copy of survey Pretest date(s) Posttest date(s) Number of Surveys expected (collected)

Monitoring sheet Monitoring sheet (continued)(continued)

Need Data Entry Template (date) Sent data entry template (date) Received Report (in email from & date) Report Late/Received Draft Extension Granted (notes) Data Received Need Training

What were the indicators of What were the indicators of success?success?

Factor

# that measured

change in factor

03-04 04-05 05-06

Knowledge 7 7 19

Preferences 10 13 22

Self-efficacy 3 4 18

Outcome expectations

2 2

Consumption 1 7 22

PA preferences 0 1

Who has participated?Who has participated?

Channel03-04

n=12 (44%)

04-05

n=24 (67%)

05-06

n=45 (98%)

Schools 8 12 24

Colleges and universities

2 3 4

Health departments

1 6 11

County Offices of Ed.

2 6

Cooperative Extension

1 1 1

First 5 1

Funding Funding (% of total budget)(% of total budget)

Federal Funding of Contractors that Participated in Impact Evaluation

16

30

45

0

10

20

30

40

50

2003-2004 2004-2005 2005-2006

Year

Fu

ndin

g

(In

Million

s o

f D

ollars

)

(19%) (34%) (47%)

ResourcesResources

www.ca5aday.comResearch and Evaluation

See impact evaluation section at the bottom

www.harvestofthemonth.com

Hoorah! – More MattersHoorah! – More Matters