Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006...
-
Upload
estella-robinson -
Category
Documents
-
view
213 -
download
0
Transcript of Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006...
![Page 1: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/1.jpg)
Final Summary Evaluation: State Projects Serving Individuals
with Deaf-Blindness
October 23, 2006
NCDB Sponsored Webinar
Presented by Richard Zeller
![Page 2: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/2.jpg)
Presentation Overview
Self-assessment and verification review purpose and design
Summary of Year One (37) and Year Two (11) project self-assessments
Summary of on-site reviews
Feedback from projects and review teams
Evaluator Recommendations
Discussion
![Page 3: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/3.jpg)
Evaluation Requirement
During year 2, each project must...
"conduct a comprehensive self-evaluation. The evaluation must include a review of the degree to which the project is meeting proposed objectives and goals and an evaluation of outcome data... In addition, the Department of Education intends to conduct a limited number of on-site evaluations based on a stratified randomized sample of sites.”
(RFP, page C-4)
![Page 4: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/4.jpg)
Evaluation Purposes
Summative project evaluation questions: Are projects goals and objectives being achieved? Do projects address each RFP priority? Do projects have appropriate outcome data?
Formative project evaluation: Provide a continuous improvement process for individual projects to use.
National report - both summative and formative: provide information OSEP can use to guide needed system improvements.
![Page 5: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/5.jpg)
Evaluation Constraints
48 projects - single and multi-state Staffing from partial to several FTE Common design to allow summaries Resources come primarily from the projects Assess whether projects are addressing RFP
priorities
![Page 6: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/6.jpg)
Evaluation Design “Work scope” standards: Priorities (a) - (i) and
General Requirements (a) - (c) Priority questions:
What types of strategies are used? Is work being completed in a timely fashion? Are intended results being achieved? Are outcome data available (efforts and effects)? Are improvement plans in effect?
“General Requirements:” Are these requirements being appropriately addressed?
![Page 7: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/7.jpg)
Evaluation Design (continued)
Self-assessments parallel MSIP’s Continuous Improvement and Focused Monitoring System (now the SPP & APR).
Verification Reviews (site visits) during this evaluation, reviews became a check on the self-evaluation process and a way to provide TA to the project.
Adjustments made in both self-assessment and review designs during implementation.
![Page 8: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/8.jpg)
Self-Assessment Summary
Priorities (a) - (e): Strategies Timeliness Results Data - Effort and Effect Adjustments/Future Plans
Priorities (g) - (h) and General (a) - (c): Are the priorities addressed (Yes/No) Are there standards that apply?
![Page 9: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/9.jpg)
Strategies Described
While the relative application of “ongoing” strategies was higher in year two, they
were also more distributed across projects than in year one.
Year One Year Two
Linear 1% 1%
Cyclical 15% 17%
Ongoing 36% 45%
Combined 48% 38%
![Page 10: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/10.jpg)
For all timeliness item ratings
Year One
Year Two
active/behind schedule 15% 19%
not implemented/on schedule
1.4% 3%
active/on schedule 83% 78%
![Page 11: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/11.jpg)
For all result item ratings:
Year One
Year Two
exceeding expectations 13% 13%
meeting expectations 72% 65%
below & approaching expectations
12% 20%
well below expectations 1% 1%
cannot rate 3% 1%
![Page 12: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/12.jpg)
For all effort item ratings:
Ratings cluster: 3 projects in year one and 2 in year two rated “extensive” data for more than 6 items
Year One
Year Two
no data 2% 2%
some level of effort data 13% 33%
some quality of effort data 6% 7%
some level & quality of effort data 62% 45%
extensive level & quality of effort data
18% 13%
![Page 13: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/13.jpg)
For all effect item ratings:Year One
Year Two
no data 12% 20%
outcome data 71% 68%
impact data 16% 12%
Clarification:• Outcomes are the immediate results of your
assistance (e.g., teacher skills gained in training)• Impacts are results your clients have when they
apply what you have taught them (e.g., they teach & children learn communications skills)
![Page 14: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/14.jpg)
Overall Item Ranking (hi to lo)Year Two
(a)(3) R-B practices
(a)(1) State capacity
(d) Collaboration
(a)(4) Provider skills
(b)(1) Census
(a)(5) Address child/family needs
(e) Disseminate
(b)(2) Assess critical child needs
(a)(2) Systemic change
(c)(1) Evaluate effectiveness
(b)(3) Assess state needs
(c)(3) Advisory evaluation design
(c)(2) Measure child outcomes
Year One
(c)(3) Advisory evaluation design
(d) Collaboration
(b)(1) Census
(a)(1) State capacity
(a)(3) R-B practices
(a)(4) Provider skills
(b)(3) Assess state needs
(a)(5) Address family/child needs
(e) Disseminate
(a)(2) Systemic change
(c)(1) Evaluate effectiveness
(b)(2) Assess critical child needs
(c)(2) Measure child outcomes
![Page 15: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/15.jpg)
Areas Needing More Attention?
(c)(2) Measure child outcomes (a)(2) Systemic change(b)(3) Assess state needs(b)(2) Assess child needs(c)(1) Evaluate effectiveness
(c)(3) Advisory evaluation design?
![Page 16: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/16.jpg)
Adjustments/Future Plans
All projects/all strategies: about 81% (Year One) v. 63% (Year Two) of strategies are to “continue as proposed”
In Year One, 8 projects accounted for 55% of planned changes
In Year Two, all projects plan some changes in strategy, with 7 adopting new strategies
The most common areas of adjustment across both years were priorities (c)(1), evaluation and (c)(2), measurement of child outcomes
![Page 17: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/17.jpg)
Priorities (g) & (h)
Affirmative Response:Year One
Year Two
(g) OSEP Directed TA 27% 0%
(g) Web-based TA 92% 82%
(g) Community of Practice 92% 82%
(h) Advisory Standards 97% 100%
(h) Act on Advisory Recs 100% 100%
(h) Advisory Change 38% 36%
![Page 18: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/18.jpg)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
![Page 19: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/19.jpg)
General Priorities (a) - (c)
General Priority AreaYear One
Year Two
(a) Employ people with disabilities? 59% 73%
(a) Try to employ people? 68% 73%
(a) Advance people with disabilities? 62% 36%
(a) Change employment practices? 35% 36%
(b) Involve people with disabilities? 100% 100%
(c) Does project have a website? 86% 91%
(c) Is web-site accessible? 81% 82%
(c) Planning website improvement? 95% 45%
![Page 20: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/20.jpg)
How Projects Relate Priorities to Work Design (Part 1, Year 2)
Objs or Goals Priority Cites Priorities / Obj Objs / Priority
33 41 1.2 1.7
20 38 1.9 1.1
23 88 3.8 1.2
34 164 4.8 1.8
20 120 6.0 1.1
6 42 7.0 0.3
26 182 7.0 1.4
14 116 8.3 0.7
3 27 9.0 0.2
14 133 9.5 0.7
10 111 11.1 0.5
![Page 21: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/21.jpg)
Total Priority Cites by 11 Projects (Year 2, Part 1)(a)(1) State capacity 117
(a)(2) Systemic change
76
(a)(3) R-B practices 96
(a)(4) Provider skills 87
(a)(5) Assess family & child needs
64
(a)(6) Other 17
(b)(1) Census 34
(b)(2) Assess critical child needs
60
(b)(3) Assess state needs
54
(c)(1) Evaluate effectiveness
84
(c)(2) Measure child outcomes
49
(c)(3) Advisory evaluation 42
(d) Collaboration 70
(e) Disseminate 80
(g) OSEP specified TA 26
(h) Maintain Advisory 23
(a) Employ Individuals with disabilities
7
(b) Involve Individuals 43
(c) Web site accessibility 33
![Page 22: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/22.jpg)
Verification Visit Summary
Sites Visited (in order): Year One: IN, FL, NJ, NY, WA, MO, CO, MT, CA Year Two: KY, MI , NC, TN
Process: Team of 3 reviewers each rated their agreement with the Project’s ratings for each priority and offered comments on each priority.
Revisions to the process and report form were made during the first year (simplifications) and again before year two (in response to suggestions).
![Page 23: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/23.jpg)
Site Review Participation# Staff # Stakeholders
IN 4 20FL 5 12NJ 5 8NY 5 19WA 4 14CO 4 9MT 4 10MO 5 7CA 8 12KY 6 5MI 5 13NC 2 12TN 3 5
![Page 24: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/24.jpg)
Who were the reviewers and how many sites did they visit?
Reviewer Name Year One Year TwoZambone 1Sharpton 1 1Bove 2McLetchie 2 2Rafalowski Welch 2Syler 2 2Fankhauser 3 1Dalke 4 3Rachal 4 2Steveley 6 1
![Page 25: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/25.jpg)
Did site reviews tend to validate project self-assessments?
Agreement with project: “any reviewer’s rating of each project item self assessment rating”
Agreed with Project Year One Year Two Strongly Agree 85% 60% Mostly Agree 10% 25% Somewhat Agree 3% 2% Somewhat Disagree 2% 1% Strongly Disagree <1% 0%
![Page 26: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/26.jpg)
Were reviewer ratings (after discussion) reliable?
Agreement here was defined as “all reviewers rated the way on a given item”
Year One: 32 actual disagreements, or 96.9% complete agreement on all items
Year Two: 1 disagreement, or 99%+ agreement on all items
Only 5 sites where disagreements among reviewers occurred; most of them in one site
![Page 27: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/27.jpg)
How did projects and reviewers view the value of these two processes?
Self-assessment and improvement planning is a necessary function for the system of state projects.
The past and current processes and forms are complex and redundant, given the way the work is organized.
Both processes (self-assessments and review visits) have value, but both need substantial redesign.
![Page 28: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/28.jpg)
Projects’ View of the Value of the Self-Assessments
Project Ratings (# reporting): Year 1 Year
2
High value 9 3
Moderate value 18 7
Some value 9 1
More trouble than it was worth 1
![Page 29: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/29.jpg)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.Year Two
![Page 30: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/30.jpg)
What Some Projects Liked
Prompted communication with state program sites Forced staff to consider value of work The forms forced project to focus and limit narrative Improvement over earlier self-evaluation processes Aligned proposal to RFP, so not hard to use Format was easy and more logical [than year 1] Separate narrative allowed project to show how
priorities were woven into goals & objectives
![Page 31: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/31.jpg)
What Projects Didn’t Like Priorities, criteria & proposed work not aligned Evaluation rules were not in the RFP Accessibility problems with form Redundancy (e.g., attachments & narratives) Word functions don’t work in the template form Too many reports for one year Too much time - takes away from TA Form accessibility (couldn’t enlarge print?) Narrative, priorities & ratings in three documents Format - impossible to match priorities to activities
![Page 32: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/32.jpg)
Review Team Recommendations to Sites:
Expand partnerships (B, C, 619, others) - others must do the work of system change
Family networking/support (parent-to-parent) Define/structure TA and intent - child change,
local capacity building, systems change Systematize data collection (census, needs,
efforts and effects on individuals/systems) Use evaluation for program improvement
![Page 33: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/33.jpg)
Review Team Suggestions
Better self-assessment instructions Consolidate Progress Report & Project Evaluation Clarify evaluation standards in the RFP Cluster priorities (eliminate redundancies) Value of the review process is the TA provided Effort & effect need better definition Change forms: Neither Year 1 or 2 worked for all Align Priorities and evaluation model
In future evaluation & review processes:
![Page 34: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/34.jpg)
Evaluator Recommendations
The next RFP should have 5 program priorities (e.g., skill development, system capacity/change, child census/performance, family services, dissemination R-B practices)
Combine self-assessment and reporting in a single system with prescribed indicator measures for each priority for all projects
For larger projects (>$500K) adopt standard 3+2 procedures
![Page 35: Final Summary Evaluation: State Projects Serving Individuals with Deaf-Blindness October 23, 2006 NCDB Sponsored Webinar Presented by Richard Zeller.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e265503460f94b15d0d/html5/thumbnails/35.jpg)
Discussion:Were Evaluation Purposes Achieved?
Summative project evaluation questions: Are projects goals and objectives being achieved? Do projects address each RFP priority? Do projects have appropriate outcome data?
Formative project evaluation: Provide a continuous improvement process for individual projects to use.
National report - both summative and formative: provide information OSEP can use to guide needed system improvements.