Evaluation of Large Initiatives of Scientific Research at the National Institutes of Health
-
Upload
santiago-martin -
Category
Documents
-
view
27 -
download
0
description
Transcript of Evaluation of Large Initiatives of Scientific Research at the National Institutes of Health
Evaluation of Large Initiatives of Scientific Research at the National Institutes of Health
Mary KaneConcept Systems Inc.
William M. TrochimCornell University
American Evaluation AssociationNovember 4, 2006
The Context
Changing nature of science Interdisciplinary, collaborative Large initiatives for complex problems Expansion of use of large center grants as research
funding mechanism Similar issues reported in the European Union (EU) in
connection with the evaluation of Science, Technology, and Innovation (STI) policies
Government wide accountability expectations GPRA PART ExpectMore.gov
Good science requires good management
Evaluation of Large Initiatives
National Cancer Institute Transdisciplinary Tobacco Use Research
Centers (TTURCs), (2001 – 2003) Centers for Disease Control
Prevention Research Centers Network, 2003-2005
National Institute of Allergies and Infectious Diseases AIDS Clinical Trials Network, Division of AIDS,
National Institutes of Health (2005 – present)
Evaluation Approach Culture change Collaboration and involvement of
researchers, funders, consultants Understand initiative life-cycle Develop initiative logic model Link comprehensive measures and tools to
model Keep costs and respondent burden low Assure scientific objectivity and credibility Address multiple purposes and audiences Design for re-use where possible Report and utilize results Provide an opportunity for reflection and
learning
Initiative Life Cycle Model
Conceptual Model
Measures
Questions
Stakeholders
Context• Motivation• Capacity• Structure• Expertise• Support
The context includes the organizational structures and organizational constraints that delimit evaluation activities. Issues include:
At each stage a wide variety of stakeholders need to be involved both in helping determine what questions should be addressed in evaluation and in providing their assessments of initiative performance and outcomes.
At each stage there are a variety of evaluation questions with more prospective questions earlier in the life-cycle and more retrospective ones later. Processes are needed for for prioritizing which questions will be addressed at each stage.
Evaluation is an empirical activity. Consequently, measures that are related to the constructs in the conceptual model needed at every stage.
Structured Conceptualization
Evaluation Methods
Needs Assessment
Evaluability Assessment
ImplementationEvaluation
ProcessEvaluation
OutcomeEvaluation
ImpactEvaluation
Cost-Effectiveness & Cost Benefit Evaluation
Secondary Analysis
Meta-Evaluation
Conceptual
Model
Formative/Ex Ante
Summative/Ex PostMethods
PolicyContext
New Initiatives
Strategic Impact
Policy Implications
StrategicGoals
Plan DisseminateImplementDevelop
Plan
DevelopImplement
Disseminate
The TTURC Case Study
Transdisciplinary Tobacco Use Research Centers
History RFA released 12/98 Grants reviewed 7/99 First award 9/99 Reissuance 9/04
Approximately $75 million in first phase
TTURC Life Cycle Model
Model Development
Concept Map
Engage the Community
Diversity & SensitivityRelationships & Recognition
Active Dissemination
Technical Assistance
Training
Research Methods
Research Agenda
Core Expertise & Resources
Evaluation System Plan
1234567…….
1234567…….
1234567…….
Logic Model
Inputs Outputs OutcomesActivities
Active Dissemination
TrainingEngage the Community
Technical Assistance
Core Expertise & Resources
Research Agenda
Community Health change
AnalysesMeasures
Measures & AnalysesConceptual Map & Logic ModelConceptual Map & Logic Model
FinancialReport (SF259a)
FinancialReport (SF259a)
Budget & JustificationBudget & Justification
BibliometricsBibliometrics
ProgressReport Summary
ProgressReport Summary
PublicationsPublications
Expenditures& CarryoverExpenditures& Carryover
ResearcherFormResearcherForm
Peer EvaluationPeer Evaluation
Peer EvaluationPeer Evaluation
Financial AnalysisFinancial Analysis
PersonnelReportPersonnelReport Personnel AnalysisPersonnel Analysis
EvaluationAnalysis
EvaluationAnalysis
Content AnalysisContent Analysis
Survey AnalysisSurvey Analysis
ProgressReport (PHS2590)
ProgressReport (PHS2590)
1. How well is the collaborative transdisciplinary work of the centers (including training) accomplished?
2. Does the collaborative transdisciplinary research of the centers lead to the development of new or improved research methods?
3. Does the collaborative transdisciplinary research of the centers lead to the development of new or improved scientific models and theories?
4. Does TTURC research result in scientific publications that are recognized as high-quality?
5. Does TTURC research get communicated effectively?6. Are models and methods translated into improved
interventions?7. Does TTURC research influence health practice?8. Does TTURC research influence health policy?9. Does TTURC research influence health outcomes?
Evaluation Questions
Evaluation Questions
1. How well is the collaborative transdisciplinary work of the centers accomplished?
• What are TTURC researcher attitudes about collaboration and transdisciplinary research?
• How do researchers assess performance of their centers on collaboration, transdisciplinary research, training, institutional support and center management?
• What are examples of collaboration, transdisciplinary and training activities of the centers?
• What is the quality and impact of the collaboration, transdisciplinary and training activities of the centers?
• Do TTURC research publications provide evidence of collaboration and transdisciplinary research, and how do they compare with “traditional” research?
• How effective and efficient is the management of the TTURCs?
Subquestions:
Evaluation Questions
1. How well is the collaborative transdisciplinary work of the centers accomplished?
Data Sources:• Researcher Form
• Attitudes about Transdisciplinary Research Scale (15 items)• Center Collaboration Scale (15 items)• Attitudes about Collaboration in Research Scale (8 items)• Institutional Support Index (12 items)• Overall Ratings of collaboration, transdisciplinary integration, training,
institutional support• Content Analysis of annual progress reports for activities, results and barriers
(code on collaboration, transdisciplinary integration, training, institutional support)
• Peer evaluation• Annual progress reports• Publications
• Bibliometric analysis of publications• Collaboration within and across institutions and centers• Numbers of fields represented by publications, cited and citing articles,
weighted by impact of journals• Management analysis
• Personnel• Budget and Financial
Researcher Form Each center responsible for
generating measures for 3-4 clusters on the map (at least two centers reviewed each cluster)
Compiled into measure development database, draft measure produced
25 closed-ended questions each with multiple subquestions
Overall performance ratings by outcome area
Open-ended Comments
244 specific measurement items proposed across the 13 content clusters
Scales and Indexes Attitudes about Transdisciplinary Research Scale (15
items) Center Collaboration Scale (15 items) Attitudes about Collaboration in Research Scale (8 items) Institutional Support Index (12 items) Methods Progress Scale (7 items) Science and Models Scale (17 items) Barriers to Communications Scale (8 items) Center-to-Researcher Communications (5 items) Center External Communications (2 items) Progress on Development of Interventions Index (12 items) Policy Impact Index (4 items) Translation to Practice Index(9 items) Health Outcome Impact Scale (6 items)
Researcher Survey8. Collaboration within the center
o.
n.
m.
l.
k.
j.
i.
h.
g.
f.
e.
d.
c.
b.
a.
95% CI
4.64.44.24.03.83.63.4a. Support staffing for the collaboration.b. Physical environment support (e.g., meeting space)
for collaboration.c. Acceptance of new ideas.d. Communication among collaborators.e. Ability to capitalize on the strengths of different
researchers.f. Organization or structure of collaborative teams.g. Resolution of conflicts among collaborators.h. Ability to accommodate different working styles of
collaborators.i. Integration of research methods from different fields.j. Integration of theories and models from different
fields.k. Involvement of collaborators from outside the center.l. Involvement of collaborators from diverse disciplines.m.Productivity of collaboration meetings.n. Productivity in developing new products (e.g., papers,
proposals, courses).o. Overall productivity of collaboration.
Content Analysis
Code approximately 80-90 project reports per year by the 13 outcome clusters
Did three rounds of reliability testing and refinement of coding definitions
Final reliability > .9
Progress Report Content Analysis – Years 1-3
Collaboration
Transdisciplinary Integration
External Recognition and Support
Science & Models
Publications
Interventions
Communication
Policy Implications
Translation to Practice
Health Outcomes
Training
Internal Recognition And Support
Methods
(data from Content Analysis of Annual Progress Report Form PHS2590)
Peer Evaluation – Years 1-3
Training
Collaboration
Transdisciplinary Integration
Internal Recognition And Support
External Recognition and Support
Methods
Science & Models
Publications
Interventions
Communication
Policy Implications
Translation to Practice
Health Outcomes
Bibliometric Analysis
What is a TTURC publication? Results from TTURC research Cites TTURC Grant Number Independent peer evaluation would
identify the influence Components of bibliometric analysis
Publications, citations, cited (references)
Journals of publication, citing, cited Field (Current Contents) Year
Bibliometric Analysis Indicators
Journal Impact Factor (JIF) – average number of citations of a journal of all articles published in previous two years
Journal Performance Indicator (JPI) – average number of publications to date for all publications in a journal in a particular year
Field Journal Performance Indicator – JPI for all journals in a field
Adjusted Journal Performance Indicator (Expected Citations) – JPI for a specific type of publication
5-year Impact – Average number of citations to publications over a five year period
Bibliometrics
Paired Samples Test
.2007 1.97978 .25348 -.2228 .6241 .792 60
.6367 2.66952 .34180 .0657 1.2077 1.863 60
.5961 2.58795 .33135 .0426 1.1497 1.799 60
Citations -Expected Citations
Pair1
Citations - JournalPerformanceIndicator (JPI)
Pair2
Citations - FieldPerformanceIndicator (FPI)
Pair3
Mean Std. DeviationStd. Error
Mean Lower Upper
90% ConfidenceInterval of the
Difference
Paired Differences
t df
On average, there were .64 more citations of TTURC publications than for other publications in the same journal.
On average, there were .6 more citations of TTURC publications than for other publications in the same field.
Citation of TTURC publications is significantly higher than for journal and field comparison groups.
Bibliometrics
Descriptives
Difference (Citations - Expected Citations)
6 -1.9633 .53114 .21684 -2.5207 -1.4059 -2.43 -1.2317 1.0835 3.15161 .76438 -.5369 2.7039 -1.74 11.0023 .2887 3.02642 .63105 -1.0200 1.5974 -2.43 11.00
20002001Total
N MeanStd.
Deviation Std. Error Lower Bound Upper Bound
95% Confidence Interval forMean
Minimum Maximum
ANOVA
Difference (Citations - Expected Citations)
41.170 1 41.170 5.392 .030160.333 21 7.635201.503 22
Between GroupsWithin GroupsTotal
Sum ofSquares df Mean Square F Sig.
Only the two complete years were used in this analysis.
Citations lower than expected in year 1, higher in year 2.
Citation of TTURC research publications is significantly increasing over time relative to expectation.
Financial Analysis
(data from Financial Status Reports of grantees)
Cumulative Percent of Federal Funds Spent by Grantee
80%
62%
99%
78%
72%
46%
77%
91%
74%
68%
57%
93%
82%
94%
87%
66%
40%
50%
60%
70%
80%
90%
100%
Period 1 Period 2
Center 1
Center 2
Center 3
Center 4
Center 5
Center 6
Center 7
Total
Carryover
(data from Budget Justification, Annual Progress Report Form PHS2590)
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Year 1 Year 2 Year 3
% s
ub
pro
ject
s
1
2
3
4
5
6
7
Mean
Percent of subprojects by center and year that reported a carryover.
Delay of projectstart
Unanticipatedobstacles
Changes inprocess -practical
Other-Specify Not stated
Reasons for Carryover
Reasons for Carryover
0
0.1
0.2
0.3
0.4
0.5
0.6
Causes of Delay or Unanticipated Obstacles
Sta
ffing
issu
eIm
plem
enta
tion
or lo
gist
ical
issu
eR
esea
rch/
met
hods
issu
eG
rant
ing
agen
cy is
sue
Infra
stru
ctur
e is
sue
Oth
er-S
peci
fy
Not
sta
ted
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
(data from Budget Justification, Annual Progress Report Form PHS2590)
What Worked
Less Promising Researcher Survey – one wave Content Analysis – costly, time
consuming Peer Evaluation of publications
More Promising Researcher Survey scales Peer evaluation of progress reports Financial Analysis Bibliometrics
Conclusions Sustainability Challenges
Funding challenges Researcher motivation
Methodological Challenges Peer review Bibliometrics Integrating results
Organizational Challenges Agency resources Grantee resources External contractors
Utilization Challenges Building over multiple time points Building over multiple initiatives