Program Evaluation Follow-up...
Transcript of Program Evaluation Follow-up...
![Page 1: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/1.jpg)
Program Evaluation Follow-up Webinar
Lee R. Gaseoma, Ed.D.
Suzanne Malson, M.S.
![Page 2: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/2.jpg)
American Indian
Vocational Rehabilitation
Training and
Technical Assistance Center
The content of this presentation was developed under a grant from the U.S. Department of Education (CFDA 84.250Z; PR/Award Number
H250Z150002). However, this content does not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal
Government.
![Page 3: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/3.jpg)
Do you have questions or comments to share on the first webinar regarding program evaluation or the second webinar regarding the community of practice (CoP)?
![Page 4: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/4.jpg)
Learning Objectives
AIVRS STAFF WILL…
LO1 …understand the purpose of program evaluation
LO2 …make distinctions in the required AIVRS performance reports
LO3 …be able to work through the program evaluation process, methods, and analysis
LO4 …identify qualitative and quantitative data methods
LO5 …understand formative and summative evaluation methods
LO6 …utilize basic descriptive statistics to enhance the AIVRS annualreport, the final 5-year AIVRS report and Tribal reports
LO7 …become familiar with sources of data for program evaluation
![Page 5: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/5.jpg)
Topics of Interest from CoP 1. Purpose of program evaluation and CoP
2. Example of techniques used in program evaluation
3. Implement program evaluation described in the AIVRS grant proposal
4. Data gathering, management, and analysis
5. Use of data to prepare annual AIVRS performance report
6. Program evaluation provides performance feedback
![Page 6: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/6.jpg)
Purpose of program
evaluation
![Page 7: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/7.jpg)
Purpose of Program Evaluation
•Demonstrate AIVRS program effectiveness to funders
• Improve the implementation and effectiveness of programs
• The better management of limited resources
•Document program accomplishments
![Page 8: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/8.jpg)
Purpose of Program Evaluation
• Justify current AIVRS program funding
•Support the need for increased levels of funding
•Document AIVRS program development and activities to help ensure successful replication
![Page 9: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/9.jpg)
Purpose of CoP
![Page 10: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/10.jpg)
Purpose of Community of Practice (CoP)
•CoP is a group of people who work together to solve a problem or improve practice in their area of professional work
•Deepen participants’ knowledge and expertise by interacting on an ongoing basis
![Page 11: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/11.jpg)
Purpose of Community of Practice (CoP)
•CoPs may be large in scale and deal with complex problems
•CoPs may be small in scale that focus on a problem at a specific level
![Page 12: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/12.jpg)
Example of Techniques used
in Program Evaluation
![Page 13: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/13.jpg)
Quantitative and Qualitative Techniques
Quantitative Techniques Qualitative Techniques
Surveys/Questionnaires Observations
Pre/post Tests Interviews
Existing Databases Focus Groups
Statistical Analysis Non-statistical (methods vary)
![Page 14: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/14.jpg)
Implement Program Evaluation Described in the AIVRS Grant
Proposal
![Page 15: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/15.jpg)
AIVRS Grant Proposal
•Describes the quality of Program Evaluation
•The evaluation methods must be thorough, feasible and appropriate to measure the program goals, objectives, and outcomes
![Page 16: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/16.jpg)
AIVRS Grant Proposal•Evaluation methods should include the use of objective performance measures clearly related to the intended program outcomes
•This means the use of an unbiased evaluation tool that will measure your outcomes
![Page 17: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/17.jpg)
AIVRS Grant Proposal
•Two evaluation criteria to consider is the validity and reliability of your evaluation tool
•Evaluation methods will provide quantitative and qualitative data
![Page 18: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/18.jpg)
AIVRS Grant ProposalA well-designed program evaluation will provide accurate performance feedback and periodic assessment of progress toward achieving intended outcomes
![Page 19: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/19.jpg)
AIVRS Grant Proposal•Evaluation strategies describe how appropriate and reliable data will be obtained and evaluated
•It is important to use external evaluators who have the experience and knowledge of AIVRS programs
![Page 20: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/20.jpg)
AIVRS Grant Proposal•Program evaluation procedures should be phased in with other project tasks. For example the use of consumer satisfaction surveys
•Evaluation plans provide performance feedback to influence program operations
![Page 21: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/21.jpg)
Data Gathering,
Management, and Analysis
![Page 22: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/22.jpg)
What methods are you using to
gather, manage and analyze
program data?
![Page 23: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/23.jpg)
Program Evaluation Data
Quantitative data•Consist of numbers or quantities•Closed-ended questions•Limited choices in answers
![Page 24: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/24.jpg)
Data Source
Case Management produces quantitative data•Number of Applicants•Number of Individual Plan of Employment, (IPE)•Successful Employment Outcomes
![Page 25: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/25.jpg)
Program Evaluation Data•Qualitative•Looks at quality•Open-ended questions•Unlimited choices in answers that is obtained in focus groups •This approach is time consuming but worth the effort
![Page 26: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/26.jpg)
Data Source
Program Outreach Activities•Disability Awareness Day•Cultural Awareness Day
![Page 27: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/27.jpg)
Data SourceCommunity Education•Presentations on Program at Tribal and Local Functions •Meetings with Local Primary Employers
![Page 28: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/28.jpg)
Data SourceCommunity Education •Partner with State Independent Living Council (SILC) and State Rehabilitation Council (SRC)
•Implement co-operative agreement with State VR Agencies
![Page 29: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/29.jpg)
Use of Data to Prepare Annual
AIVRS Performance
Report
![Page 30: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/30.jpg)
AIVRS Annual Performance Report Includes 9-Sections
1. Program budget
2. Program goals and objectives
3. Number of consumers served
4. Types of VR services provided
5. Educational goals and employment outcomes
![Page 31: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/31.jpg)
AIVRS Annual Performance Report
6. Job Training Common Measures
7. Interaction with State VR agency
8. Program self-evaluation
9. Consumer satisfaction
![Page 32: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/32.jpg)
Program Evaluation Provides
Performance Feedback
![Page 33: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/33.jpg)
The program evaluation cycle provides AIVRS
programs with performance feedback while the program is in progress, as well as at
the end of the program period
![Page 34: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/34.jpg)
![Page 35: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/35.jpg)
AIVRS Purpose Statement
![Page 36: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/36.jpg)
AIVRS Purpose Statement
![Page 37: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/37.jpg)
AIVRS Purpose Statement
![Page 38: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/38.jpg)
Thank you!
![Page 39: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/39.jpg)
Contact Information• Lee R. Gaseoma, Ed.D., Director
• Email: Lee Roy Gaseoma [[email protected]]
• PH: 928-523-6829
• Suzanne Malson, M.S.
• American Indian VR Training & Technical Assistant Specialist
• Email: [email protected]
• Phone: 928-523-8602
![Page 40: Program Evaluation Follow-up Webinaraivrttac.org/wp-content/uploads/2016/04/052716_Follow-Up_CoP.pdfAIVRS Grant Proposal •Evaluation methods should include the use of objective performance](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f3dfce4a6fc6e42292b5f16/html5/thumbnails/40.jpg)
Physical NAU Location
•American Indian VR Training & TA Center
•PO Box 5630
• (912 Riordan Ranch Rd.)
•Flagstaff, AZ 86011-5630
•FAX: 928-523-9127
•TTY: 928-523-1695