Slide 1

Stefanie Murray, MPHPatrick J. Lemmon Public Health Prevention Specialist, CDCCo-founder and former Office of Family Health, OHAExecutive Director Men Can Stop Rape Slide 2 Learning Objectives By the end of this session, participants will: Understand how to plan a simple program evaluation Feel confident that they can conduct a basic process evaluation of their programs Feel confident that they can conduct a basic outcome evaluation of their programs Slide 3 Text Chat Question When you think of doing evaluation, what words or feelings come to mind? Slide 4 What Is Evaluation? Evaluation is a systematic process to understand what a program does and how well the program does it. Slide 5 Types of Evaluation Formative (before or during activities) Process Evaluation Needs Assessment Summative (after activities are completed) Outcome Evaluation Impact Evaluation Slide 6 Todays Discussion Our mission is to: 1. help programs to develop indicators that will help them to determine whether their programs are working as they hope 2. share those results with other programs who are trying to do the same KEEP IT SIMPLE Slide 7 A Few Key Terms Triangulation: The idea that multiple indicators of success will be more instructive than only one Can look at specific program components Allows for easier mid-course corrections Less likely to show bias Each component is lower stakes Indicator: Measure of success Slide 8 Stages of Evaluation 1. Planning 2. Program Implementation 3. Data Collection 4. Analysis *Opportunities for mid-course correction to program 5. Reporting 6. Program Review and Revision Slide 9 Planning: Key Questions What are we hoping will change as a result of the program? How will we know that the change has happened? How can we measure the changes? What are the key components of implementation? Slide 10 A Note on Measures There are many tools you can use for this Do not rely only on surveys Think creatively, and incorporate into programming if you can Examples of tools: Interviews Pictures of activities Focus groups Facilitator notes Slide 11 Program Implementation: Key Questions Are we on schedule with our implementation? Any unanticipated barriers or changes? Slide 12 Data Collection: Key Questions How closely did our implementation follow the plan? Do we have all of the tools we need for the data collection? Slide 13 Analysis: Key Questions What changes have we seen as a result of our program? Is the change as large as we anticipated it would be? Slide 14 Reporting: Key Questions Who needs to know this, and what is the best way to share it with them? Newsletter Webinar Funder report Slide 15 Program Review and Revision: Key Questions How did our implementation look, compared to our plan? Do we need to modify the plan? Did we get the changes we wanted? If not, can we make tweaks to increase the likelihood that we will do better next time? Slide 16 Slide 17 Planning Our plan is to develop a coalition to convince a local school board to make changes to their policies related to sexual violence prevention programming At least 5 community partners Invited to speak before school board at least twice At least 3 of our proposed changes will be implemented Slide 18 Program Implementation Track attendance at the coalition meetings and school board hearings Note which organizations take on tasks, since this is an indicator of their buy-in Slide 19 Data Collection Participation rates at community meetings and school board meetings Have a year-end party for the participants at which we do an informal focus group Did you feel like part of a united effort? Did you feel you contributed actively? Did we make a difference? How? Note how often school board members reference members of coalition in their remarks Count our proposed changes that are implemented Slide 20 Analysis Did we get at least 5 active partners in the coalition? How long did that take? Did partners feel like part of an effort? Did we get invited to testify? How many times? How many of our proposals were implemented? Slide 21 Program Review and Revision What were the critiques we received from the partners in the focus group? How could they have felt more engaged? Can we make these partnerships last and take on additional efforts? Slide 22 Slide 23 Planning While developing curriculum, ask How will we know we got there? Example: Bystander intervention session Slide 24 Decide on tactics Well hand out 3x5 cards to participants and ask them to write how they would respond to this scenario: Youre with a group of your friends (all male), and one of them starts joking about a female classmate, who you know but not very well. He calls her an uptight bitch. A few of your friends laugh, but others are quiet and look a little uncomfortable. What do you do? Slide 25 Decide on criteria Cant just decide after-the-fact what the standards are. We are hoping for answers that are: Nonviolent Creative Recognize humanity of target and of person being intervened with Encouraging more pro-social behavior Slide 26 Decide what success looks like It is unlikely that we will get to 100% with our efforts, or else our test is probably not a very good one We will divide answers into GREAT, DECENT, and UNDESIRABLE We want at least 60% of our students to give at least a decent answer Slide 27 Program implementation Slide 28 Data collection and Analysis How would you rate the following responses? I wouldnt do anything. I support my friend. U Slide 29 How would you rate the following responses? Id tell my friend to show some respect. D Data collection and Analysis Slide 30 How would you rate the following responses? Id wait until later and then pull my friend aside and tell him that I wish he would be more respectful of other people, both the girl he was talking about and his friends who were uncomfortable with what he said. G Data collection and Analysis Slide 31 How would you rate the following responses? Id tell him to shut the hell up. D Data collection and Analysis Slide 32 How would you rate the following responses? I wouldnt laugh or anything, but I would not challenge my friend. U Data collection and Analysis Slide 33 How would you rate the following responses? Id ask him how he would feel if someone talked about his sister that way. G Data collection and Analysis Slide 34 How would you rate the following responses? Id remind him that he got into a fight last year when he heard someone saying something like that about his girlfriend. G Data collection and Analysis Slide 35 How would you rate the following responses? Id look at him so hed know I didnt like what I said, but I would not say anything. D Slide 36 How would you rate the following responses? Id laugh. U Data collection and Analysis Slide 37 How would you rate the following responses? Something almost like this happened to me last week. I told my friend that I didnt like what he said, and he kind of laughed it off, but then later he came up to me and apologized for offending me. GG Data collection and Analysis Slide 38 Sort results GREATDECENTUNDESIRABLE 34 3 Slide 39 Compare to goal GOALRESULTS 67 Slide 40 Reporting WE ARE AWESOME ! Slide 41 Questions? Slide 42 THANK YOU! Stefanie MurrayPatrick Lemmon stefanie.l.murray@state.or.uspatricklemmon@gmail.com