What Do Programs Want?
description
Transcript of What Do Programs Want?
![Page 1: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/1.jpg)
What Do Programs Want?What Do Programs Want?
Jeanette Treiber, PhDProject DirectorCenter for Program Evaluation and ResearchUC Davis School of Medicine
![Page 2: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/2.jpg)
Objective
Determine local agencies’ evaluation capacity building needs
![Page 3: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/3.jpg)
Background California Department of Public Health
(CDPH)’s Tobacco Control Program is carried out by
61 Public Health Departments Approximately 40 Competetive Grantees
![Page 4: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/4.jpg)
More Background CDPH Strategy
Norm Change Community base Community
Assessments 3yr work plans Evaluation
![Page 5: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/5.jpg)
Evaluation ComponentsLocal Program Evaluation requirements
Comprehensive Evaluation Plan Process and Outcome Evaluation Internal or external evaluators Progress Reports Final Evaluation Reports
![Page 6: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/6.jpg)
Tobacco Control Evaluation Center (TCEC)
TCEC provides Evaluation Capacity Building services to California TC programs through:
TATrainingToolsFER scoring and feedback
![Page 7: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/7.jpg)
Rationale for TCEC Services“…pressures have been accompanied by
increasing awareness (and often frustration) on the part of funders as they confront the limited ability of the organizations they fund to meet the challenge of evaluating programs in ways that are likely to yield either genuine outcome-oriented accountability or useful information for formative improvement programs” Stevenson et.al. 2002
![Page 8: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/8.jpg)
Evaluation Capacity Building Observed need and response in past
decade Various approaches to CB Various definitions Various approaches to measuring effect of CB Still little research
![Page 9: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/9.jpg)
Method
1. A 21 item electronic survey (using the online service surveymonkey.com) to a list of 170 project directors and evaluators.
2. Analysis of access database for technical assistance from 2004 to 2009 (approx. 4.5 yrs)
![Page 10: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/10.jpg)
Results (survey)Need for training or materials High or
Moderate Need
Using culturally competent evaluation methods ……………….79.4 %
Writing evaluation plans ……………………………………….59.9 %
Adapting/developing data collection instruments ……………...57.1 %
Interpreting evaluation results…………………………………..54.1 %
As a reference during evaluation activities…………...................51.5 %
Writing evaluation reports……………………………………….50 %
Collecting evaluation data……………………………………….42.9 %
![Page 11: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/11.jpg)
Projected Use of TCEC (survey)
Projected use Response Percent
Finding, developing, or adapting data collection Instruments………………………………………..68.6
Writing an evaluation plan ……………………….57.1
As a reference during evaluation activities ………………………………48.6
To build overall evaluation capacity of the project …………………………….42.9
![Page 12: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/12.jpg)
Preferred Format (survey)
Webinar 68 %
“How-to” evaluation guides on website
57.1%
Individual consultation by phone or e-mail
45.7%
Website strategy exchange 42.9%
![Page 13: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/13.jpg)
Requests for Assistance (TA Log)
Total number of TA requests between 11/2/04 and 7/16/09
512Average: 11 requests/month
Up to 27 requests/agency total
![Page 14: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/14.jpg)
Main Request types (TA Log)
Type of requests
0
50
100
150
200
250
300
Plans tools Analysis Other
![Page 15: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/15.jpg)
Type of Data Collection Instrument
020406080
100120140160180200
Key informantInterviews
Surveys Observation tools
39
189
44
![Page 16: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/16.jpg)
TA DemandNumber of projects that have requested TA
0
10
20
30
40
50
60
70
LLA Compet
Total number ofprojects
Total number ofprojectsrequesting TA
(79%)(71%)
![Page 17: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/17.jpg)
FER Score and TA requestsFinal Evaluation report scores for Local Lead
Agencies were correlated with frequency of TA request.
No statistically significant correlation was found.
![Page 18: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/18.jpg)
Summary/Conclusion Majority of local health departments AND
community organizations in CA TC require evaluation assistance
Greatest need is with DCIs, developing Eval plans, and help conducting evaluation with diverse populations.
Most DCI dev. need is with surveys Webinars are the most desired mode of
training delivery Individual assistance requests vary greatly
in frequency, scope, and content
![Page 19: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/19.jpg)
Limitations Response rate to survey was only 20% TA assistance was not consistently entered
into log in the same manner Requests for TA are somewhat dependent
on marketing of TA TCEC team has changed personnel
throughout the 4.5 years
![Page 20: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/20.jpg)
What do Programs Want?
•Flexible, Varied Evaluation Assistance;
•Convenient Delivery
![Page 21: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/21.jpg)
ReferencesCalifornia Department of Public Health Services,
Tobacco Control Program. http://www.cdph.ca.gov/programs/tobacco/Pages/default.aspx
Innovation Network – Transforming Evaluation for Social Change. http://www.innonet.org/
Preskill, Hallie. (2005). Building Evaluation Capacity. 72 Activities for Teaching and Training. Thousand Oaks (Sage).
![Page 22: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/22.jpg)
More ResourcesStevenson, John, Paul Florin, Dana Scott Mills, and
Marco Andrade. (2002). Building Evaluation Capacity in Human Service Organizations: A Case Study. Evaluation and Program Planning 25, 233-243.
Tobacco Control Evaluation Center (TCEC). Center for Program Evaluation and Research, UC Davis. http://programeval.ucdavis.edu
University of Wisconsin Extension. Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/index.html
![Page 23: What Do Programs Want?](https://reader035.fdocuments.in/reader035/viewer/2022070419/56815d6e550346895dcb763f/html5/thumbnails/23.jpg)
ContactJeanette Treiber, PhDProject DirectorCenter for Program Evaluation and ResearchTobacco Control Evaluation CenterUC Davis School of Medicine
[email protected]://programeval.ucdavis.edu1616 Da Vinci CourtDavis, CA 95618