Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M....
-
Upload
gerald-carter -
Category
Documents
-
view
213 -
download
0
Transcript of Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M....
![Page 1: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/1.jpg)
Perspectives on Impact Evaluation
Cairo, Egypt March 29 – April 2, 2009
Presented by:
Wayne M. Harding. Ed.M., Ph.D. , Director of Projects, Social Science Research & Evaluation, Inc., Burlington, MA USA [email protected]
Cheryl Vince-Whitman, M.B.A., Ed.M., Senior Vice President, Education Development Center, Newton, MA USA [email protected]
A Model for Using TechnicalA Model for Using TechnicalAssistance to Improve the Evaluation Assistance to Improve the Evaluation
Capacity of Local ProgramsCapacity of Local Programs
![Page 2: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/2.jpg)
2
Agenda
• Brief Overview of Service to Science (STS).
• Evaluation Findings About Service to Science
• Key Characteristics of The Technical Assistance Provided By Service To Science
![Page 3: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/3.jpg)
Overview of Overview of Service to Service to ScienceScienceInitiativeInitiative
![Page 4: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/4.jpg)
4
The Need for Service-to-Science
• Federal agencies and others in the USA have promoted the use of evidence-based programs to prevent substance abuse.
• The supply of “proven” programs is too limited to meet many local conditions.
• There are many other programs that may work, but lack evidence of their effectiveness
![Page 5: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/5.jpg)
5
Service-to-Science Goals
• To help innovative prevention interventions that address substance abuse (or related issues) develop, improve, and document evidence of their effectiveness.
• To increase the pool of effective and appropriate interventions.
![Page 6: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/6.jpg)
6
Organization of Service-to-Science
• Funded by Center for Substance Abuse Prevention (CSAP) in 2004.
• Implemented through 5 regional prevention centers (Centers for the Application of Prevention Technologies – CAPTs).
• In 2007, CSAP funded a related initiative for programs designed for Native Americans. Implemented by the Native American Center for Excellence (NACE).
![Page 7: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/7.jpg)
7
Service-to-Science Activities
• CAPTs solicit prevention program nominations from State prevention officials.
• CAPTs assess the readiness of candidate programs.
• CAPTs conduct regional Service to Science Academies.
• CAPTs provide follow-up technical assistance.
![Page 8: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/8.jpg)
8
Service-to-Science Activities (Continued)
• Service to Science Academy participants – eligible to compete for “subcontract awards” to further enhance evaluation capacity.
• CAPTs provide limited technical assistance post subcontract award.
![Page 9: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/9.jpg)
9
Number of Programs Served Through Service to Science
• As of July 2009, the CAPTs had provided 294 prevention programs with technical assistance.
• Since 2006, 110 programs have received the subcontracts of up to $30,000.
![Page 10: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/10.jpg)
Selected Selected Evaluation Evaluation Findings Findings
![Page 11: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/11.jpg)
11
Methods• Pilot study (2006) interviews and/or online survey
with program Directors. Target sample of 79 programs; 58 (73%) responded.
• Six case studies of 2005 STS participants and extended follow-up case studies on 5 that received mini-subcontracts. Six case studies of 2006 participants.
• 2007 online survey targeted 142 programs that had been participants for at least 15 months. 93 responded (a 77% rate after removing 21 programs for which respondents were no longer available).
![Page 12: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/12.jpg)
Participant Participant ProgramProgram
Characteristics Characteristics
![Page 13: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/13.jpg)
13
Primary Behaviors Targeted for Change
Percent (a)
Alcohol Use/Abuse 80%
Illicit Drug Use/Abuse 60%
Tobacco Use 58%
Violence (including bullying) 37%
HIV/AIDS 8%
(a) Multiple responses allowed. N=93.
![Page 14: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/14.jpg)
14
Ages of Population(s) Served
Percent (a)
Adolescents (ages 13-17) 71%
Childhood (ages 6-12) 51%
Young Adults (ages 18-25) 19%
Early Childhood (ages 0-5) 10%
Adults (ages 26-55) 10%
Older Adults (ages >55) 7%
No Distinction by Age 3%
(a) Multiple responses allowed. N=93.
![Page 15: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/15.jpg)
15
Race/Ethnicity of Population(s) Served
Percent (a)
No Distinction 85%
Hispanic or Latino 9%
American Indian or Native Alaskan 5%
Black or African American 4%
White 4%
Asian American 3% Native Hawaiian or Other Pacific Islander 3%
(a) Multiple responses allowed. N=93.
![Page 16: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/16.jpg)
16
Strategies Used
Percent (a) Percent
Improve Knowledge / Awareness
90% Mobilize Community 27%
Improve Skills 85% Improve Identification and Referral
25%
Healthy Alternatives 79% Improve Policies 17%
Improve Norms 59% Improve Laws 12%
Improve Mental / Physical Health
54%Improve Access to and Quality of Care
11%
Build Collaboration 36% Improve Enforcement 8%
(a) Multiple responses allowed. N=93.
![Page 17: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/17.jpg)
Service To Service To ScienceScience
Outcomes Outcomes
![Page 18: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/18.jpg)
18
Value of STS ServicesPercent (a)
Satisfied with CAPT TA(Very Satisfied or Somewhat Satisfied vs. Somewhat Dissatisfied or Very Dissatisfied)
72%
CAPT TA Useful (A Little, Somewhat, or Very Useful vs. Not at All Useful)
91%
Would Recommend STS to Similar Programs (Definitely Yes or Probably Yes vs. Probably No or Definitely No)
91%
(a) N=93.
![Page 19: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/19.jpg)
19
Factors That Facilitated Progress
When asked to discuss factors that facilitated progress on their evaluation the largest percentage of clients (79%) identified TA from the CAPTs.
(a) From the 2006 pilot study. N=58.
![Page 20: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/20.jpg)
20
TA Contributed to Program Capacity(a) in 5 Most Common Issues
Percent (b)
Improving understanding of program rating systems such as NREPP & NASADAD
95%
Improving understanding of evaluation 95%
Developing and/or improving a logic model 92%
Developing and/or implementing a more rigorous evaluation design
95%
Improving organizational commitment to evaluation
84%
(a) Contributed “A Little,” “Somewhat,” or “A Great Deal” vs. “Did Not Contribute at All.” (b) N=93.
![Page 21: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/21.jpg)
21
Seeking Recognition and/or Funding
Percent (a)
Plan to seek or are seeking recognition from NREPP or NASADAD
57%
Sought recognition from local, regional or other national organizations
13%
Prepared a journal article 13%
Presented evaluation findings at professional conference or meeting
13%
Used evaluation findings to help secure funding
23%
(a) N=93.
![Page 22: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/22.jpg)
Defining Defining Features Of Features Of Service To Service To Science TAScience TA
![Page 23: Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e845503460f94b85c4d/html5/thumbnails/23.jpg)
23
Defining Features of Service to Science TA
• Client-centered
• Customized
• Long-term
• Relationship-based
• Flexible