Evaluating Open Source Software for Use in Library Initiatives: A
Evaluating Statewide Quality Initiatives
Transcript of Evaluating Statewide Quality Initiatives
![Page 1: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/1.jpg)
Evaluating Statewide Quality Initiatives
Kathryn Tout Child Trends
![Page 2: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/2.jpg)
2
• This session and the Quality Initiatives Research and Evaluation Consortium (INQUIRE) are supported by the Office of Planning, Research and Evaluation in the Administration for Children and Families through the CCEEPRA contract with Child Trends
Acknowledgements
![Page 3: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/3.jpg)
![Page 4: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/4.jpg)
4
• Review the importance of evaluating quality initiatives
• Identify how evaluation can be initiated and incorporated into ongoing activities
• Review an evaluation planning template • Ask practical questions about evaluation
Session Overview
![Page 5: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/5.jpg)
5
Have you…. • Reviewed implementation data in a recent
meeting? • Reviewed evaluation results for a recent
project in your state/territory? • Written a Request for Proposals (RFP)? • Managed an evaluation contract?
Experience with Evaluation
![Page 6: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/6.jpg)
6
Which topics are you evaluating? • QRIS? • Quality improvement initiative? • Professional development? • Resource and referral services? • Health and safety activities? • Standards?
Experience with Evaluation
![Page 7: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/7.jpg)
Section Header Why invest in evaluation of quality initiatives?
![Page 8: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/8.jpg)
8
• Identify implementation successes and challenges
• Examine the effectiveness of new initiatives
• Document outcomes for stakeholders • Support short- and long-term planning
Evaluation can…
![Page 9: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/9.jpg)
9
Performance Monitoring vs. Evaluation: Both are needed!
Performance Monitoring Evaluation
How many programs enrolled in the QRIS? Were targets met?
Was QRIS enrollment adequate to meet the needs of families? Why do some geographic areas have lower QRIS enrollment?
How many hours of TA were delivered to programs?
How is TA being delivered? To what extent are coaches following a recommended model?
Are programs increasing their star level?
Which programs are improving their quality? Do changes in star rating align with changes in observed practice?
![Page 10: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/10.jpg)
10
What are your questions?
![Page 11: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/11.jpg)
Section Header How should we get started with evaluation?
![Page 12: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/12.jpg)
12
• Where are the largest investments in the system?
• Where are your concerns? • Is evaluation required by a funder?
Identify high priority activities and research questions
![Page 13: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/13.jpg)
13
Match evaluation focus to the initiatives’ stage of implementation
• Outreach • Recruitment • Staffing of the initiative (hiring, training, supervision)
Launch of initiative
• Awareness of the purpose of the initiative • Participants’ and stakeholders’ perceptions of
implementation and initial improvements • Reach of the initiative
Early implementation
• Changes in practice • Changes in outcomes for children and families
Full implementation
• Use evaluation findings to document impact and/or propose changes to design of the initiative
Mature implementation
![Page 14: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/14.jpg)
14
Example: QRIS Quality Coaching
• What are the characteristics of the coaches? • Does the training for coaches meet standards for
effective professional development? Launch of initiative
• How are coaches spending their time with programs? • How do programs perceive the coaching? • What dosage and content are delivered, and how do
they vary by program?
Early implementation (1-2 Years)
• Do teachers change their practices? • Does program quality improve?
Full implementation (2-5 years)
• Are changes in quality sustained? • What are the successes and challenges of coaching? • What revisions to the coaching should be considered?
Mature implementation
![Page 15: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/15.jpg)
15
• Work with an evaluator when developing or revising a data system
• Ensure rigid procedures for data entry across all who interact with the system so that data are reliable and useful for evaluation
• Save and make historical data accessible for evaluation
Strengthen existing data systems
![Page 16: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/16.jpg)
16
What are your questions?
![Page 17: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/17.jpg)
Section Header What steps can we take when planning an evaluation?
![Page 18: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/18.jpg)
18
• Step 1. Develop a logic model for the quality initiative or a system of initiatives – Specify outcomes – Link expected outcomes with activities – Identify outputs that can be tracked
Evaluation Planning Template
![Page 19: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/19.jpg)
19
• Step 2. Create or connect with an Evaluation Advisory Committee – Engage key stakeholders – Identify evaluation consultants – Create a dissemination plan
• Step 3. Identify research questions that align with the theory of change and stage of implementation
Evaluation Planning Template
![Page 20: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/20.jpg)
20
• Step 4. Identify resources for evaluation – Map existing data sources – Examine existing performance management
data – Identify staff with expertise to support
evaluation or hire a consultant – Identify funding sources (including public and
private funders) for evaluation – Develop evaluation partnerships if possible
Evaluation Planning Template
![Page 21: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/21.jpg)
21
• Step 5. Align resources with evaluation strategy – Lower cost – analysis of existing data,
surveys and interviews, focus groups – Higher cost – observations of quality,
collecting child development data
Evaluation Planning Template
![Page 22: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/22.jpg)
22
• Step 6. Create a project through an RFP or other mechanism
• Step 7. Monitor research process and emerging findings
• Step 8. Disseminate findings – Tailor presentations depending on audience
Evaluation Planning Template
![Page 23: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/23.jpg)
23
What are your questions?
![Page 24: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/24.jpg)
Section Header Resources
![Page 25: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/25.jpg)
25
TA Resources
• ADD QUALITY CENTER RESOURCES ON EVALUATION
![Page 26: Evaluating Statewide Quality Initiatives](https://reader031.fdocuments.in/reader031/viewer/2022012018/616869fdd394e9041f6f6d0c/html5/thumbnails/26.jpg)
26
QRIS Evaluation Resources