EVALUATION DESIGNevallab.unm.edu/.../day-2-materials/evaluation-design.pdf · 2020. 9. 15. ·...

EVALUATIONDESIGNUNMEvaluationLabSummerInstitute2018

[email protected] /evallab.unm.edu

Decisionsforyourevaluation

• Whoandhowmany?• Comparedtowhat?• Howwillyoucollectthedata?

7/31/18 UNMEvaluationLabSummerInstitute- EvaluationDesign 2

Who?

ParticipantsOtherstakeholders:

• staff• Boardmembers• partners• funders


Howmany?


Ifyouwanttomakeclaimsaboutoutcomes,dataneedtoberepresentative• EveryoneinyourprogramOR

• Arandomsampleofthoseinyourprogram(acliniccouldcollectcustomersatisfactiondatafromallvisitorsduringoneweekeveryquarter)

Bewareoflowresponserates• Youwantresponseratesof80%orhigher• Thosenotrespondingareprobablyyourlessenthusiasticparticipants


DepthUnderstandingdepthofexperienceisharderandmuchmoretimeconsuming

• Howdoparticipantsexperiencetheprogram?• Howdidtheprogramchangetheirlifestory(ifitdid)?

Hereyouarelookingforsimilaritiesanddifferencesandideallyyoukeepcollectinguntilyouhaveallthestories

• Inpracticethereislimitedtimeavailable,andyouneedtotradebreadthfordepth• In-depthexperiencesareincrediblyhelpfulforunderstandingyourprocesses• Justbeawarethattheymaynotberepresentative

Understandthatthosewhovolunteerforafocusgroupwilltypicallybeyourmostenthusiasticparticipants

• Trytofindwaystogetyourlesssatisfiedparticipantsthereaswell


HowMany?Breadthvs.Depth

DobothCombinerepresentativedataandin-depthdataprovidedbyahandfulofinformants.Buildthatintoyourevaluationdesign.


ComparedtoWhat?Keyquestionforevaluationdesign


Oneshot– nocomparison

Source:Russ-EftandPreskill,EvaluationinOrganizations

Posttestofwhatyouknow,yourattitudesorsituationatonepointintime.


Oneshot– nocomparison, continuedGood for

• establishingbaseline• processevaluations• learningaboutparticipants

Notgoodfor• outcomesevaluation

Foroutcomeevaluationyoualwaysneedacomparison

Unfortunately,youradministrativedataisoftenone-shot


RetrospectivePretest–individualcomparespreandpost



RetrospectivePretest–individualcomparespreandpost,continuedBenefit

• People’sstoriesofhowtheprogramchangedthemcanbeverypowerful

• Andifuniversalorrandomlyselected,thestoriesarecredible(andnotjustanecdotal)

• Youdon’tonlyhavetohaveyourevaluationacttogetherbytheendoftheprogram(notatthebeginning)

Caution• Thereissomeevidencethatpeoplelowertheirretrospectiveassessment ofthe presothattheycanreportanimprovement


One-GroupPretest-Posttest–comparisonofindividualattwopointsintime



One-GroupPretest-Posttest–comparisonofindividualattwopointsintimeFANTASTICfortrackingparticipants,especiallypreandpostassessmentsandmaterialconditionsBEWARE: withoutacontrolgroup,youdon’tknowthatprogramcausedchange...

• Yourparticipantscometoyouattheirlowestpoint,maybetheywouldhavedonebetterovertimeevenwithoutyourprogram

• Goodideatocollectadditionaldataabouthowparticipantsratetheprogramasafactorintheirimprovedcondition


One-GroupPretest-Posttest–comparisonofindividualattwopointsintimeChangeinattitudesor outlook

• ChildHopesurveypreandpost

Changeinknowledge• EnlaceRelaciones Saludables test(butthinkaboutwhetherknowledgetranslatestobehavior)

• Supportedbyqualitativedata– letter,focusgroup

Changeinfamilyfunctioning• PB&JNCFAS,assessedbystaff (butthinkabout “gradeinflation”)• PFS (butthinkaboutlessself-knowledgeinthebeginning)• EnlacePSI

Changeinparticipationingovernmentsupportprograms• ParticipationMedicaid,SNAP,TANF



Appendix: Child Hope Scale (modified)

21


Figure5.PercentofChildrenWhoAgreedwithEachStatementatBeginningandEndofTwo-WeekSummerProgram

Note:15childrencompletedtheBeginningsurveyand14completedtheEndsurvey.Source:Binder,Melissa.2014.EvaluationoftheTaosHealthSystemsPeñascoCommunityHealth�KidsFIRSTInitiative.

93%

79%

64%57%

79%

64%

53%

80%

40%

60%67%

33%

I think I am doing pretty good.

I know what to do when I need

something (hungry, cold, need a hug).

I feel I'm doing just as good as my

classmates.

When I have a problem I can think of things to do to take care of the

problem.

I think the things I do now will help me

later when I am older.

Even when others want to quit, I know that I can find ways

to solve the problem.

Beginning End


5

Figure 1. Percent Scoring Correctly on Pre-and Post Questions, by Theme

Note: Data collected through pre- and post- tests for 139 Relaciones Saludables students for nine semesters from year to year.

The letters highlight different ways the program has affected the lives of the participants, mainly referring to the enhanced self-knowledge and self-awareness they have experienced. One participant wrote:

A mi en lo personal me ha servido como terapia porque me ha ayudado a conocerme, amarme, valorarme como ser humano y a entender a mi propia familia y a otros a adquirir conocimientos personales para mi vida diaria como el trato a mi pareja y a los que conviven a diario conmigo y que ahora veo la vida diferente. Un cambio total en mi vida. (To me personally, it [Relaciones Saludables] has been like therapy because it’s helped me to know myself, love myself, value myself as a human being and understand my own family and others to acquire personal knowledge for my daily life like how my partner, I, and those that live with me daily, treat each other and now I see life differently. A complete change in my life.) [emphasis added, translated by evaluation team].

65%

50%

39%

49%

29%

84%75% 74% 75%

64%

Self-esteem Communication Conflict Resolution Sexuality IntimateRelationships

Pre Post

“… [Relaciones Saludables] has

been like therapy because it’s

helped me to know myself,

love myself, value myself as a

human being and understand

my own family and others…”


NMEvaluationLab2016

5

Figure3.FamiliesratedatbaselineorbetterontheNCFASatintakeanddischarge

Note:DatarepresentallclientswithmatchedNCFASassessmentsbetweenJanuary2010andJanuary2016,about900clientsintotal.

NCFASdomainsareratedonaseven-pointscale.Onaverage,familiesinthegeneralprogramimprovebyabouthalfapointontheseven-pointscale.FamiliesintheTime-LimitedReunificationprogramimproveby.6and.7pointsintheareasofCaregiver/ChildAmbivalenceandinReadinessforReunification.

Effectsizesrangefrom.38to.71.Effectsizesshowthechangeinscoreasaproportionofthestandarddeviationofscoresinthefirstassessment.Oncontinuousmeasures,effectsizesforevidence-basedprogramsaretypicallyinthe.3range,sotheseeffectsizesareveryencouraging.Ofcourse,sincewedonothaveacontrolgroup,wecannotsaywithcertaintyhowfamilieswouldfarewithoutPB&Jprogramming.ButwecansaythatfamiliesimprovesignificantlyduringthetimetheyworkwithPB&J.Effectsizesforeachdomainareprovidedintable2intheappendix.

49.3%

51.1%

46.0%

51.9%

52.2%

51.9%

43.2%

51.7%

62.8%

54.5%

34.9%

30.8%

27.6%

33.4%

35.7%

31.7%

23.0%

33.0%

38.6%

36.4%

Environment

ParentCapabilities

FamilyInteractions

FamilySafety

ChildWell-Being

Social/CommunityLife

Self-Sufficiency

FamilyHealth

Caregiver/ChildAmbivalence

ReadinessforReunification

FirstNCFAS LastNCFAS


NMEvaluationLab2016

7

ProtectiveFactorsSurvey(PFS)

ForthePFS,parentspickafrequencyordegreeofagreementonaseven-pointscale.Examples

offrequencyandagreementquestionsareprovidedinfigure5.

Figure5.SamplequestionsfromtheProtectiveFactorsSurvey

Never

Very Rarely

Rarely

About Half the Time

Frequently

Very Frequently

Always

1. In my family, we talk about problems. 1 2 3 4 5 6 7

Strongly disagree

Mostly Disagree

Slightly Disagree Neutral Slightly

Agree Mostly Agree

Strongly Agree

12. There are many times when I don’t know what to do as a parent.

1 2 3 4 5 6 7

PB&Jfamiliesshowedimprovementineachofthesubscales.Parentsreportedabouta.5point

increaseinFamilyFunctioningandResiliency,a.4pointincreaseinSocialandEmotional

Supportanda.3pointincreaseinConcreteSupport.Alloftheseimprovementsarestatistically

significant.ThechangeinNurturingandAttachmentismuchsmaller;parentsranked

themselvesveryhighinthisareatobeginwith.(Seefigure6belowandtable4inappendix.)

Figure6.Protectivefactorsatintakeanddischarge

*Differencesaresignificantata5%levelusingatwo-tailedt-test.

FivequestionsonthePFSrelatetoparentingknowledgeandstyle.Parents’responsesshow

improvementson3oftheseitems.Thelargestimprovementisinresponsetothestatement:

“TherearemanytimeswhenIdon’tknowwhattodoasaparent.”Thisitemwasreverse-

codedsothatalargerscoremeansthatparentsbecamemoreconfidentabouttheirparenting

knowledge.Theotherimprovementsweregreaterunderstandingthatchildrendonot

misbehavetopurposelyupsettheirparents,andinnotlosingcontrolwhendisciplining.(See

figure7belowandtable5inappendix.)

4.6

5.1 5.0

6.0

5.15.6

5.3

6.1

FamilyFunctioning/

Resiliency*

SocialEmotional

Support*

ConcreteSupport* Nurturingand

Attachment

IntakeScore DischargeScore


Enlace Comunitario Evaluation Report

10

Figure 3. Parental Stress Index Surveys Show Parents’ Stress Level Percentiles Declined Over Time in the Family Program

Note: Data represent 120 survey responses for 30 unique participants. Data are from mid-2010’s to 2017.

B. Pre- and Post-Program Assessment for Crianza con Cariño. The evaluation team analyzed the 39 matched pre- and post- assessments from the Crianza class, and the findings started a discussion around the questionnaire itself. After review, the Enlace staff realized that this instrument had been created prior to the revision of the curriculum and no longer captures the courses’ content and emphasis. We thus relegated the analysis to the appendix. (See Appendix D.)

C. Post-program evaluation survey for Crianza con Cariño

In the third instrument, the post-program evaluation for the parenting course Crianza con Cariño, the 31 total responses from the last 2 years show participants were overall very grateful and thankful for participating in the course in response to the open-ended questions that asked what the best part of the program was and if they would recommend the program to a friend or relative. 100% of respondents said “Yes” that they would recommend the program to a friend or relative. Additionally, questions 5, 6 and 7 were helpful to this evaluation. In question 5, participants rank on a scale from 1 to 5 (1= “Not at all” to 5= “Very Much”) whether “Before the Crianza program, you used a.) emotional or verbal punishment or b.) physical punishment like spanking.” 40% of respondents reported using emotional or verbal punishment and 43% of respondents reported using physical punishment before the Crianza program (between 3 and 5 on the scale).

39

56

64

70

Final

8-week

4-week

Intake



Controlgrouponemeasure

Randomlyassignedtoprogram

Randomlyassignedtocontrolgroup

Controlgrouponemeasure

Beopentopossibilityofidentifyingacontrolgroup• Evenifnotrandomassignment,isthereagroupyoucan’tservebecauseofgeographyorsomeothercharacteristicnotrelatedtoneedforprogram?

CouldworkforpeopleonwaitlistYouneedlotsofpeopletomake thiswork


CrystalBridgesMuseumstudy

• Walton Family Foundation established the museum in Arkansas in 2011

• Free museum tours to schools. • Demand for tours far exceeded capacity and researchers paired

similar classrooms and randomly chose the one to receive the tour. The other classroom, which was offered a tour in the following semester, served as the control.


Greene, Jay, Brian Kisida and Daniel Bowen. 2014. “The Educational Value of Field Trips.” Education Next (Winter): 79-86.

CrystalBridgesMuseumstudy,continued

• The tour employed student-driven discussions of five paintings. Guides provided artist name and the title of the painting, facilitated the discussion, and gave additional information only in response to student questions. This approach aims to increase student engagement and learning.

• Several weeks after the tour, both the “treated” and control students, more than 10,000 of them, completed a survey.




Controlgrouppre-post

Randomlyassignedtoprogram

Randomlyassignedtocontrolgroup

Controlpre-post

Beopentopossibilityofidentifyingacontrolgroup• Evenifnotrandomassignment,isthereagroupyoucan’tservebecauseofgeographyorsomeothercharacteristicnotrelatedtoneedforprogram?

CouldworkforpeopleonwaitlistYoudon’tneedasmanypeopletomake thiswork- Buthardtofollowupwithcomparisongroup


CaseStudies

In-depthexplorationstolearnhowprogramimpactedindividualparticipants


Ideas?Questions?


EVALUATION DESIGNevallab.unm.edu/.../day-2-materials/evaluation-design.pdf · 2020. 9. 15. ·...

Documents

Transcript of EVALUATION DESIGNevallab.unm.edu/.../day-2-materials/evaluation-design.pdf · 2020. 9. 15. ·...