DIY Evalua*on AnnualEvalua)onLabWorkshop
MelissaBinder,Director,NMEvalua)onLab@UNMApril21,2017
2of26
You may already have a really great evalua*on system in place
4principles
8bestprac)ces
3of26
Principle #1: Evalua*on should serve the interests of your organiza*on
• Addresswhatismostimportanttoorganiza)on:yourcorevalues,yourguidingmission,whatyoureallycareabout
• Maybedifferentfromtheinterestsofyourfunders
4of26
BEST PRACTICE: measure what you care about
Goalsandobjec)vesfrommissionstatements,logicmodels,andtheoriesofchange
5of26
Principle #2: Evalua*on should be integrated into your day-to-day opera*ons
• Datacollec)onshouldbepartofyourrou)neopera)ons–aSendance,clientsa)sfac)on,dischargesummaries• Withanevalua)onsprocessinplace,youwon’tbegoingonadatahuntevery)meyouneedtowriteagrantreport
6of26
BEST PRACTICE: data collec*on should not be a burden
• Yourdatacollec)onshouldn’tposeanundueburdenonstafforpar)cipants• Youreallydon’tneedmorethantwopointsin)meforeachpar)cipant.IntermediatedataisoXenTMI.• Ifyourweeklypar)cipantshavetofilloutthesamesa)sfac)onsurveyeveryweek,theyaregoingtoprovidelessthough[ulfeedback
7of26
Principle #3: Have a feedback loop
Collectdata
Reviewdata
Adjustprogram
Ifyouarecollec)ngdata,reviewingitandadjus)ngyourprogram,youaredoingevalua)onright.
8of26
BEST PRACTICE: integrate data into decision making
• Reportandreviewdataonaregularcycle• Staffmee)ngs• Annualreport• BoardofDirectors
9of26
Principle #4: Evalua*on should feel sa*sfying and valida*ng Andmaybeevenfun.
10of26
BEST PRACTICE: evalua*on data is useful.
Evalua)onprovidessuppor)ngdocumenta)onforwhatyouknowyouaredoingright...
...andilluminatesthewayforward
DATA COLLECTION & ANALYSIS
12of26
Measure what you care about
13of26
Mission statement
ArtWorksseekstofulfillthefollowingoutcomes:
• Teachersbecomeengagedwiththeartsandlearntointegrateartintoclassroomlessonsandpedagogy.
• Usingencounterswiththearts,students:o Prac)ceanalysis,discussionandapprecia)onforawidevarietyofartformsandculturaltradi)ons,
o Expandtheirvocabularyandprac)celiteracyskillssuchasself-expressionbothverballyandinwri)ng,
o Developnewskillsinreasoning,boosttheirconfidenceands)mulatetheirimagina)on.
14of26
Logic model
Inputs
HumanCapitalPB&JFamilies
LeadershipteamDirectservicestaff
SupportstaffBoardofDirectorsVolunteers&Interns
Professionaldevelopment
FinancialCapitalBraided&leveragedapproachtofinancial
sourcesPrivatedonations
Government&privategrants
SocialCapitalReputationincommunity,
Communitypartners&funders,Evaluation
Accreditation
PhysicalCapitalPhysicalsitesembeddedincommunity,Furniture,
Medicalsupplies,Communicationequipment
Transportation,FacilitiesandMaintenance
Interactionswithothersaregenuinelyhopeful,welcoming,respectful,
nonjudgmental,positiveandvalidating
Responsiveness&adaptabilityallowsustomeetpeoplewheretheyare,helpthem
buildontheirstrengthsandsupportthe
capacitytocopeintheirjourneytohealpast
trauma
Heart-CenteredApproach
Program
ParentandchildrelationshipChilddevelopment
assessmentsChilddevelopment
curriculumEarlyinterventionservices
Pre-kindergartenHomevisiting
CasemanagementSocialworkCounseling
Speech,occupational,andphysicaltherapy
NursingTransportation
Meals
Activities
CoalitionbuildingPubliceducationPoliticalaction
Advocacy
IntermediateBenchmarks
MutualtrustPrograminvestment
Participation Family
INTACTHealthy
attachmentNurturing&SAFEENVIRONMENT
Hassocialnetwork
FamilyresilienceSocialnetwork
Accesstoconcretesupports(food,housingemployment,childcare,
healthcare&education)
ParentPositiveparenting
skills
Socially&emotionallycompetent
Advocatesforself,childandfamily
Understandschilddevelopment
Outcomes
ChildOPTIMAL
DEVELOPMENTHealthy,resilient,&
socially&emotionallycompetent
Self-advocacyREADYFORSCHOOL
Policy
Statewidechildabusepreventionprogram
Publichealthapproachtoinfantandchildhoodhealth
Adequatesocialsupportforbasicfood,housing,
employment,healthcareandeducationneeds
Transparentandaccessibledata
Assumptions: Regardlessofbackgroundorhistoryeveryfamilyisworthyanddeservessupport,andthepreservationoffamilyserveschildrenandcommunity.Everyfamilyandculturehasstrengths,identifyingthemthroughearlysupportandcollaborativeeffortsmakesallthedifferenceandcreatesthegreatestchangeandimpact.Weallcanchange,everyindividualisworthwhileandeverychilddeservessupportwithoutlabelingorstigma,ourjusticecouldberestorativeandchildabusecanbeprevented.
Assets: Diversity,culture,climate,naturalbeauty,intergenerationalfamilies,creativeandprogressivecommunity,UNM,Medicaid.
External Factors: Poverty,intergenerationalviolence,inadequatefundingforsocialservices,isolation,inadequatetransportation,perceptionsofsingleparentfamilies,nocomprehensivestatewidechildabusepreventionplan,fragmentedhealthmentalhealthsystem,lackofaddictionservices,lackofaffordablehousing,oppressiontowardsmarginalizedgroupsbecauseofsystemicracism.Noaccesstostatedata.
Staffwithfamilies
Parentswithchildren
Allwithcommunity
ReflectiveParallelProcess
Perceptions
Publicawarenessabouttrauma,resilienceand
protectivefactorsandtheimportance
ofprevention
Growth
AwarenessofstrengthsFeelingprotected&
supportedIncreasedcapacityto
copeOwnershipforactions,
accountabilityReducedImpulsivity
Parent&childresilience
IdentifyingrealisticgoalsandstepstoreachthemConnectiontosupportsCreatingowncommunity
SafetyPlanningProvidingprotection&supportforchildrenFewertriggers,more
optionsImprovednurturingandsignsofattachment
Action
PB&JLOGICMODEL
StrategyStartearly
MultigenerationalParentingskills&
knowledgeConcretesupportReferraltoservices
Engagesocialsupportnetwork
Children grow to full potential in
nurturing families and a supportive
community
15of26
Another logic model
16of26
BEST PRACTICE: Collect both quan*ta*ve & qualita*ve data ArtWorksClassroomTeacherSurvey,administeredviaSurveyMonkey
ü Askdirectly(canbequan)fied)ü Includeopen-endedques)ons,whichelicitpar)cipantexperience(getusefulqualita)vedata)
EnlaceComunitario:knowledgequiz&par)cipantleSers
17of26
Direct ques*ons – easy to quan*fy
18of26
BEST PRACTICE: less is more for quan*ta*ve data
74% 70% 72%77%
84%
Enhancedabilitytoar)culatetheirfeelingsandopinions
Improvedvocabulary
Improvedlisteningskills
Increasedabilityandwillingness
toreflect
Greateruseofcolorful
languageandimagery
PercentofClassroomTeachersWhoReportedThatStudentsGained“ALot”or“AGreatDeal”ofLanguageArtsSkills
Principle#1:EvaluaEonshouldservetheinterestsofyourorganizaEon
19of26
BEST PRACTICE: more is less for quan*ta*ve data
PercentofClassroomTeachersWhoReportedThatStudentsGained“Limited,”“Some,”“ALot”or“AGreatDeal”ofLanguageArtsSkills
2% 3% 3% 2% 4%
24%27% 25%
21%
12%
30%
36% 35%
46% 44%44%
34%37%
31%
40%
Enhancedabilitytoar)culatefeelingsandopinions
Improvedvocabulary
Improvedlisteningskills
Increasedabilityandwillingnesstoreflect
Greateruseofcolorfullanguageandimagery
Limited Some ALot AGreatDeal
Thisistoomuchinfo!
20of26
Combining quan*ta*ve and qualita*ve ques*ons will make both more credible
Quan)fiable
Wasitreallyintegra)on?
21of26
BEST PRACTICE: iden*fy themes and provide examples for qualita*ve data
OneofArtWorks’goalsistobuildconfidence.TeachersoXenseethisinstudentswhoarequiet,shyorstruggling.Afewexamples:
ReachingQuiet,ShyandStrugglingStudents Grade Ihaveastudentwhoisatleast1gradelevelbehindclassinreadingandcanfeelfarbehindothers.ButaOerthepost-performanceworkshopwithGregory,hewasSOproudofhisartworkandsawhowhecouldkeepupwithhispeers.
1
OneofmystudentswhostruggleswithreadingwasabletomakewonderfulobservaEonsaboutthepainEngsandbackthemupwithreasonsandevidence,truly"reading"thepainEng.
3
IsabelaisashystudentwhoreallylovedtheicebreakeracEvityofthrowinganimaginarystonetoothers.Atfirstshedidn'twanttogetthestonethrowntoher,butwhenshedid,itmadeherfeeloktobeafoolonstage.Shesaidit"changedher".
6
22of26
Measuring what you care about can be tricky
Manyoutcomesaremuchhardertomeasure,andrarelydoyouhaveclassroomteacherswhocanreportonyourclients!Whatareyoualreadycollec)ng?
• Assignments• Screenings• Needsassessments• Intakeanddischargeforms• Par)cipantsa)sfac)onforms
23of26
Useful comparisons
• Pre-Post• Comparisongroup(butbewareofsampleselec)on*)• Par)cipant’sownassessmentofchangeortransforma)on(evenmoreconvincingwhencoupledwithquan)ta)veoutcomes)
*Sampleselec)oniswhenpar)cipantsaredifferentfromnon-par)cipants.Par)cipantsmaybemoremo)vatedormoredesperatethannon-par)cipants.Inthefirstcase,youwilloverstateyourprogrameffects.Inthesecondcase,youwillunderstateyourprogrameffects
24of26
PB&J families compared at intake and discharge
25of26
BEST PRACTICE: collect representa*ve data
• Doyouhavedataforalloralmostallpar)cipants?- 80%ormoreisgood
• Doyouhavedatafromarandomsample?- Youdon’tneedallpar)cipantsifyouarecollectarepresenta)vesample.Youcouldsurveyallpar)cipantstwoweeksperyear.
26of26
Great evalua*on at Enlace Comunitario
Top Related