1

National Indicators and National Indicators and Qualitative Studies to Measure the Qualitative Studies to Measure the Activities and Outcomes of CDC’s Activities and Outcomes of CDC’s

PRC ProgramPRC ProgramDemia L. Sundra, MPH

Prevention Research Centers Program

AEA 2006, Portland, OR

The findings of this presentation are the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention

PRC National Evaluation Contributors PRC National Evaluation Contributors 2004-Present2004-Present

Collaborative Evaluation Design Team (CEDT)Geri Dino (co-chair), West Virginia University

Cynthia Boddie-Willis, MA Dept. of HealthJosefina Dhungana, UCLA community representative

Katie Barnes, UNC Chapel Hill community representativeKenneth McLeroy, Texas A & M

Thomas Reischl, Univ. of MichiganDelores Pluto, Univ. of S. CarolinaMichelle Kegler, Emory University

David Collins, Morehouse School of Medicine

PRC Program Office at CDCEduardo Simoes (co-chair), Demia Sundra, Jo Anne Grunbaum,

Margaret Kaniewski

ORC MacroNicola Dawkins, Doryn Chervin, Stephanie Kamin, Amee Bhalakia,

Gayle Payne and Jesse Gerwig-Moore

Outline of PresentationOutline of Presentation

Overview of National PRC Program Evaluation Purpose Indicators Qualitative studies Activities supplementing evaluation

Lessons Learned

Levels of PRC EvaluationLevels of PRC Evaluation

Evaluation of PRC researchVs.

Local evaluation of single research center Vs.

Evaluation of national program/network of 33 centers

Project DEFINE Project DEFINE 2004-Present2004-Present

DDelivering elivering EEvidence and vidence and FFindings: indings: IImplementing the mplementing the NNational ational EEvaluationvaluation

Developing a systematic approach that will allow the PRC program to describe and assess program inputs, activities, outputs and outcomes over time

Guided by CDC Framework for Program Evaluation in Public Health1

1Centers for Disease Control and Prevention.  Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11). 03/05 from: http://www.cdc.gov/eval/framework.htm

Purposes of the PRC National Purposes of the PRC National EvaluationEvaluation

The national evaluation will accomplish the following purposes:

Primary purpose: To demonstrate national program accountability

Secondary purpose: To facilitate program improvement

Evaluation Approach Evaluation Approach

Quantitative Indicators33 PRCs Qualitative Studies

Document Review – 33 PRCsInterviews – Sample of PRCs

NATIONAL EVALUATION ACTIVITIES

Supplements Helps Identify

Case Studies & Stories from single PRCs

COMMUNICATION/DISSEMINATION ACTIVITIES

Program IndicatorsProgram Indicators

Quantitative Indicators33 PRCs

NATIONAL EVALUATION ACTIVITIES

Program IndicatorsProgram Indicators Addressing national accountability

Looking for the overarching critical components of PRCs that are feasible and meaningful to count and summarize

Balancing burden and highlighting variety

Collecting data through PRC Information System

Development of IndicatorsDevelopment of Indicators CEDT rated indicators: Useful, feasible,

meaningful, relevant Ongoing work with CEDT to refine

Developed corresponding questions for PRC Information System

Sent out for feedback from all PRCs Finalizing changes/beginning Information

System development

Current Status of IndicatorsCurrent Status of Indicators

23 program indicators cover inputs, activities, outputs, and outcomes of a national prevention research program

Research Training Partnerships Dissemination

Reflect variety of program perspectives

Information System is in development

Examples of Research Examples of Research Indicator ConceptsIndicator Concepts

ACTIVITIES Research

projects

INPUTS Project funding Academic institution

support

OUTCOMES Distribution of products Dissemination and

adoption of interventions Policy and environmental

changes

OUTPUTS Effectiveness of

interventions Publications

Sample Reporting – Fictitious Examples of Sample Reporting – Fictitious Examples of Research Indicator DataResearch Indicator Data

Across 33 PRCs, there are 287 projects, totaling $147 million in funding

Funding: 32% CDC, 15% NIH, 52% Other 82% of projects are research

In the past 5 years of research at 33 PRCs, 29 interventions were funded out of core PRC Program dollars:

10 interventions found promising 4 found effective 1 found adoptable

Examples of TrainingExamples of TrainingIndicator ConceptsIndicator Concepts

INPUTS Project funding

OUTPUTS Number of people trained Number of students

working with PRCs

ACTIVITIES Trainings, by

topic, audience, format, duration

Sample Reporting – Fictitious Examples of Sample Reporting – Fictitious Examples of Training Indicator DataTraining Indicator Data

Nationally, 1,484 individuals attended training programs sponsored by PRCs in 2006. The numbers and types of individuals trained include the following:

105 Policy Makers 541 Students 287 Public Health and Health Care Practitioners 311 Community Members 151 Researchers 89 Project Staff

Evaluation Approach Evaluation Approach

Qualitative StudiesDocument Review – 33 PRCsInterviews – Sample of PRCs

NATIONAL EVALUATION ACTIVITIES

Qualitative Study DevelopmentQualitative Study Development Used for concepts

critical to program priority for some stakeholders suitable for qualitative work

Reserved resources from beginning

Four Descriptive Studies Four Descriptive Studies Interaction between PRC researchers and

their communities around core prevention research project

PRCs’ characteristics related to organizational and partnership structures

Variety, goals, and contextual factors of PRCs’ core prevention research

Diversity of PRC training, technical assistance, and mentoring with communities and partners

Data Collection MethodsData Collection Methods Document Review – example topics

Staffing counts and structures Community Committee Guidelines

Interviews – example topics Participation of communities in research Benefits of being in the PRC network

Supplementing Evaluation Data Supplementing Evaluation Data

Supplements Helps Identify

Case Studies & Stories from single PRCs

COMMUNICATION/DISSEMINATION ACTIVITIES

Case Studies and StoriesCase Studies and Storieswww.cdc.gov/prcwww.cdc.gov/prc

Case Study – WVU/Not-On-TobaccoCase Study – WVU/Not-On-Tobacco

Story Story – WVU/Not-On-Tobacco– WVU/Not-On-Tobacco

Evaluation Approach Evaluation Approach

Quantitative Indicators33 PRCs Qualitative Studies

Document Review – 33 PRCsInterviews – Sample of PRCs

NATIONAL EVALUATION ACTIVITIES

Supplements Helps Identify

Case Studies & Stories from single PRCs

COMMUNICATION/DISSEMINATION ACTIVITIES

Lessons LearnedLessons Learned

Expect to use mixed methods

Local feedback was critically important to national evaluation strategy

Advisory group (CEDT) feedback Broader PRC Program feedback

Keep your eye on the prize Accountability of national program helps everyone

www.cdc.gov/prc

Demia L. Sundra, MPHHealth Education Specialist, PRC Program,

Division of Adult and Community Health,National Center for Chronic Disease Prevention and Health Promotion

770-488-5506dsundra@cdc.gov