1

Resources for State Oral Health Program Evaluators

January 30, 2013

Cassandra Martin Frazier, MPH, CHESCDC, bkx9@cdc.gov

B. J. Tatro, PhDASTDD, bj@bjtatro.com

2

General Reminders

• This webinar will be recorded and archived on the ASTDD website

• Questions will be addressed after the speakers are finished. Please type your question into the “chatbox” that will appear at the end of the webinar and then click on the bubble to the right of where you type your question to send it to the moderator

• Please respond to the polling questions at the conclusion of the webinar

3

• This presentation was supported by Cooperative Agreement 5U58DP001695 from CDC, Division of Oral Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC

4

Desired Results

• Increase awareness of the importance of evaluation and evaluation capacity within state oral health programs

• Increase familiarity with the scope of public oral health evaluation topics

• Increase familiarity with and access to a variety of practical and useful evaluation resources relevant to public oral health

5

PART 1Importance of Evaluation and Evaluation Capacity in Public Oral Health

6

Evaluation: A Key Component of State Oral Health Program Infrastructure

Resources

Surveillance Capacity

Planning & Evaluation Capacity

Policy and Programs

Leadership & Partnerships Coalitions & Community Advocates

Oral Health for All

7

Status of Evaluation in State Oral Health Programs• In 2012, ASTDD completed a report on the status and

factors impacting state oral health infrastructure. The capacity to conduct evaluation was one of these. Findings included the following:– According to CDC, the number of funded states reporting using

evaluation to improve programs had increased from 3 to 11 of the 12 by the end of their second state grant funding cycle

– In a 2009 ASTDD survey, most states indicated that they had access to evaluation expertise, but 13 did not

– Currently there is no way to assess and track if and how non-CDC funded states are evaluating their programs, partnerships, or policies

8

Key Factors Associated with Program Success/Improvements in Oral Health“Evaluation helps build infrastructure and enhance sustainability when results are used to improve programs, increase program visibility and demonstrate program achievements. Strong programs have evidence-based goals, conduct routine evaluation and alter their programs based on evaluation results.”(ASTDD, State Oral Health Infrastructure and Capacity: Reflecting on Progress and Charting the Future, 2012)

9

Evaluation Capacity

Evaluation helps build infrastructure but the capacity to do evaluation must be present

Evaluation requires:• Sufficient and appropriate staffing• Ongoing training• Access to data• Access to decision makers• Financial resources

10

Importance of Evaluation in Public Oral HealthIn the ASTDD Guidelines for State and Territorial Oral Health Programs, there are 10 Essential Public Health Services to Promote Oral Health in the US, one of which is:

Evaluate effectiveness, accessibility, and quality of personal and population-based oral health promotion activities and oral health services

11

State Roles in Public Oral Health Evaluation• Create a comprehensive evaluation plan

• Evaluate availability, utilization, appropriateness, and effectiveness of activities and services, with partners

• Provide evaluation consultation to partners

• Validate and share results with community members, stakeholders, and other partners

• Use results in oral health improvement and planning

• Leverage and allocate sufficient resources for evaluation

12

Essential Public Health Services to Promote Oral Health in the U.S.

Assessmen

t• Surveillance• Analyze

determinants and respond

• Inform, educate, empower

Policy

Develop

men

t

• Community partnerships, leverage resources, advocate

• Policies and plans

Assurance

• Laws and regulations

• Services• Workforce• Evaluation• Research

13

Evaluation of Other Essential Public Oral Health Services

Virtually all of the other nine Essential Services have an evaluation component!

14

Evaluation of Other Essential Public Oral Health ServicesEssential Service Evaluation-related State Role/Activities

Surveillance Review effectiveness of surveillance systems to track oral health status and disparities

Inform, educate, empower Evaluate effectiveness and quality of communication and promotion programs

Community partnerships, leverage resources, advocate

Evaluate partnerships and their effectiveness in creating oral health improvements

Policies and plans Determine impact of oral health policies

Laws and regulations Evaluate impact of laws and regulations

Services Evaluate impact of interventions and advocacy on access to services

Workforce Evaluate professional development and dental public health workforce for their effect on oral health outcomes

Research Evaluate results of implementing the research agenda

15

Funder Expectations for Evaluation• Federal Government: “A central pillar of good government is a culture

where answering such [outcome and impact] questions is a fundamental part of program design and where agencies have the capacity to use evidence to invest more in what works and less in what does not.” (Program Evaluation, in Budget of the United States Government, Fiscal Year 2011: Analytical Perspectives, 2010, 90–92. Washington, DC: U.S. Government Printing Office)

• Foundations: “The field of philanthropy is undergoing a fundamental transition toward more performance-centered and forward-looking evaluation approaches that provide foundations and their grantees with timely information and actionable insights.” (Mark Kramer, From Insight to Action: New Directions in Foundation Evaluation, April 2007)

16

CDC Structure for Evaluation• Describe the program

– Develop logic models– Develop a comprehensive evaluation plan and objectives

• Document what happens in the program– Maintain accurate and organized records on programs– Designate a person responsible for evaluation– Create success stories

• Improve the program– Use evaluation findings to plan program improvements

• CDC’s Framework for Program Evaluation in Public Health

17

CDC Structure for Evaluation• Incorporate evaluation into the following

required activities for CDC-funded states:– Leadership– The surveillance system– The state oral health plan– Partnerships and state oral health coalitions– Sealant programs– Policy development

18

Competencies for Public Oral Health EvaluatorsCDC has identified competencies for State oral health evaluators

Conducting Yourself as an Evaluator

1. Interpersonal skills2. Professional Practice

Conducting an Evaluation3. Evaluation Approach4. Program Description5. Evaluation Plan6. Program Monitoring and Tracking7. Collect, Analyze, and Interpret Data8. Evaluation Reporting/Communication9. Evaluation Use

19

Competencies for Public Oral Health EvaluatorsCompetencies based on: • Guiding Principles for Evaluators, American Evaluation

Association• CDC, Framework for Program Evaluation• Stevahn L, King J, Ghere G, Minnema J.

Establishing Essential Competencies for Program Evaluators. American Journal of Evaluation, 2005; 26 (1) (43-59)

• Yarbrough, D, Shulha, L, Hopson, R, and Caruthers, F (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA: Sage

• ASTDD Competencies for State Oral Health Programs

20

PART 2Resources for Public Oral Health Evaluators

21

Evaluation Planning and ManagementOverview of EvaluationResource: Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide (updated 2011) Description: Comprehensive “how to” guide for planning and implementing evaluation activities based on CDC’s Framework for Program Evaluation in Public Health. Designed to assist evaluators in planning and implementing practical evaluations. Includes real-world examples, tips, and worksheets. Excellent tool for beginner evaluators.

Resource: W.K. Kellogg Foundation Evaluation Handbook (1998)

Description: A handbook that provides a broad overview of Kellogg’s philosophy and expectations for evaluation and a framework for conducting evaluation projects.

22

Evaluation Planning and ManagementEvaluation Project ManagementResource: Managing and Implementing an Evaluation, Guidelines and Tools for Evaluation Managers (2008, Catholic Relief Services) Description: A guidance document on evaluation management, including management skills, communications, and dealing with the unexpected.

Resource: Evaluation Plans and Operations Checklist (1999, Daniel Stufflebeam)

Description: A checklist for conducting preliminary meta-evaluations of the evaluation plan or process in order to strengthen the evaluation operations.

Resource: Evaluation Budget Development Checklist (2001, Jerry Horn)

Description: A checklist designed to assist evaluators and others to think through the development of an evaluation budget. This instrument is divided into several categories of typical cost items for conducting evaluations.

23

Evaluation Planning and ManagementEvaluation Planning

Resource: Community-Based Oral Health Programs: A Need and Plan for Evaluation (2005, W.K. Kellogg Foundation)Description: A report describing the major components that should be included in evaluations of community-based oral health programs, along with suggestions for specific measures that could be considered.

Resource: Developing an Effective Evaluation Plan (2011, CDC)

Description: A workbook to help public health program managers, administrators, and evaluators develop a joint understanding of what constitutes an evaluation plan, why it is important, and how to develop an effective evaluation plan in the context of the planning process.

24

Evaluation of Prevention and Promotion ProgramsSchool-based Sealant ProgramsResource: Seal America: The Prevention Invention, Step 10, Program Evaluation (Updated 2011)Description: This manual is designed to assist health professionals in initiating, implementing, and evaluating a school-based dental sealant program to help prevent dental caries in children and adolescents. It includes practical suggestions for evaluating program effectiveness and efficiency. Resource: Sealant Efficiency Assessment for Locals and States (SEALS) (Updated 2009, CDC)Description: An Excel-based program that captures, stores, and analyzes data from school-based sealant programs. SEALS reports generated atthe school, local, and state levels contain a variety of performance measures that can be used to assess program effectiveness and economic impact.

25

Evaluation of Prevention and Promotion ProgramsFluoridation Program ManagementResource: Community Water Fluoridation Program Review (2004, CDC)Description: A questionnaire to assist state programs in conducting a self-assessment of their community water fluoridation program. Based on the Engineering and Administrative Recommendations for Water Fluoridation guidelines and the ASTDD Best Practices guidelines on fluoridation. To request a copy of the questionnaire, please contact Cassandra Frazier or Kip Duchon.Resource: Oral Health Maps (CDC)Description: An interactive mapping application that presents indicators of oral health and preventive interventions for oral health. Can view data by state. Resource: Fluoride Legislative User Information Database (FLUID)Description: A comprehensive database of federal and state policies regarding community water fluoridation. Enables users to access information based on legal fact and compare their current or proposed policies with others from across the country.

26

Evaluation of Prevention and Promotion ProgramsCommunications

Resource: Communications Plan Template for State Oral Health Programs (2010, ASTDD)Description: This tool is designed to help stakeholders plan and evaluate health promotion and other state oral health communications.

27

Evaluation of Policy WorkPolicy WorkResource: What Gets Measured, Gets Changed: Evaluating Law and Policy for Maximum Impact. Jamie F. Chriqui, Jean O’Connor, and Frank J. Chaloupka, in Using Law, Policy, and Research to Improve the Public’s Health, Spring 2011Description: An article that describes the need for and relevance of conducting systematic policy evaluation, factors to consider when evaluating laws and policies, and examples of how policy impact studies have informed policy discussions and decision making. Provides samples of evaluation questions. Resource: The Evaluation Exchange, Vol. XIII, No. 1 (Spring 2007, Harvard Family Research Project)Description: This volume is dedicated to evaluation of advocacy and policy change and has several relevant articles. Resource: The Policy Tool Guidebook (CDHP and CDC, 2009)

The Guidebook includes a section on evaluation, including an assessmentof baseline capacity for use by state oral health directors or other leaders and a participant survey for those engaged in the policy process.

28

State Plan EvaluationState Plans

Resource: CDC State Oral Health Plan Evaluation Workbook (~2013, CDC)

Description: This is a comprehensive and practical guide to evaluating the implementation and impact of state oral health plans. Provides at-a-glance summaries, tips, and strategies for keeping stakeholders engaged, evaluation approaches and tools, resources, and action steps. Release expected in 2013.

Resource: Oral Health State Plan Review Index (2003, CDC)

Description: A checklist to help plan and develop the state oral health plan. Can be used to conduct a review of the existing oral health plan.

Resource: State Oral Health Plan Comparison Tool (2012, CDHP)

Description: A database that provides an overview of the current state ofstate oral health plans and facilitates cross-state comparisons.

29

Evaluation of SurveillanceSurveillance

Resource: Updated Guidelines for Evaluating Public Health Surveillance SystemsMMWR, July 27, 2001; 50(RR13) (1-35)Description: This report provides guidelines for evaluating surveillance systems based on CDC’s Framework for Program Evaluation in Public Health, research and discussion of concerns related to public health surveillance systems, and public comments.

30

Evaluation of Leadership and Leadership DevelopmentLeadership Development

Resource: ASTDD Competencies for State Oral Health Programs (2009, ASTDD)

Description: Lists competencies in seven domains, including Domain 2, Plan and Evaluate Programs. For each competency, provides four levels from basic to advanced. There are links on the ASTDD website to associated tools and information about the availability of technical assistance.

Resource: The National Public Health Leadership Institute: Evaluation of a Team-Based Approach to Developing Collaborative Public Health Leaders, American Journal of Public Health, April 2005, Vol. 95, No. 4 (641-644)Description: Describes an evaluation of the National Public Health Leadership Institute, which aims to develop collaborative leaders and strengthen networks of leaders who share knowledge and jointly address public health problems. Results are based on analysis of interviews conducted 12-18 months after the program end and project final reports.

31

Evaluation of Partnerships and CollaborationPartnerships and Collaborations

Resource: Handbook on Planning, Evaluating and Improving Interagency Collaboration in State Oral Health Programs (ASTDD, 2012)Description: The handbook and companion workbook provide a framework for evaluating collaboration between state oral health programs and other entities. A variety of tools, as well as oral health examples, are provided. Technical assistance is available through ASTDD.

Resource: Evaluating Partnerships to Prevent and Manage Chronic Disease , FD Butterfoss, Preventing Chronic Disease, April 2009, Vol. 6, No. 2 Description: This article describes how to develop a comprehensive evaluation strategy based on partnership theory, select outcomes indicators, choose methods and tools, and use results for accountability and improving effectiveness.

32

Where to Find New Resources• ASTDD website (slides from this webinar will also be posted on the ASTDD

website)• CDC evaluation website• Join the American Evaluation Association (AEA) or a local affiliate• AEA Linked-In• University of Wisconsin Extension Center evaluation website• Western Michigan University, the Evaluation Center• Online Evaluation Resource Library – National Science Foundation website• Bureau of Justice Assistance evaluation website• Conferences, including the AEA Annual Conference,

National Oral Health Conference and preconference workshops, and CDC/AEA Summer Institute

• Online and live trainings, including AEA, CDC, and colleges/universities

33

Next Steps• Are you interested in:– Future evaluation webinars?– NOHC evaluation roundtables?– NOHC evaluation presentations?– NOHC evaluation pre-sessions?– A evaluation community of practice?– An evaluation blog?

• Are you willing to post evaluation articles, data sources, tools, or other resources to a shared site, if it were available?