Evaluating research consortium
-
Upload
mark-david-lim -
Category
Health & Medicine
-
view
111 -
download
0
description
Transcript of Evaluating research consortium
Evaluating the value of
research-by-consortium
Mark David Lim, PhD
September 4, 2014
actionFasterCures is an “action tank” driven by a singular
goal – to save lives by speeding up and
improving the medical research system.
A center of the Milken Institute, we are a nonprofit
and nonpartisan organization that works with all the
sectors of the medical research and development
ecosystem.
bringing a new discovery from lab to market is a
long, expensive and risky road
innovationMilken Institute partnered with
Sanofi to host an Innovation
Retreat in 2011
Meeting yielded 40+ policy and
R&D recommendations including:
• Open innovation and
cooperation among competitors
• Collaborating in the
precompetitive space
• Defining metrics of success
Shared scientific challenge
Widely-usable tool
Virtual team to create / qualify
research-by-consortium
Academia /
Clinical
IndustryGovernment
Patient
groups
research-by-consortium
Temporary association
of researchers that share
resources and effort for
a common objective.
Consortia integrate
multiple types of
knowledge, data from
multiple sources, and
align different interests.
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
Operational Framework Landscape
consortiapedia.fastercures.org
• Mission/governance
• Financing
• Data-sharing
• Intellectual property
• and others…
Database
369 consortia
• Disease focus
• Types of tools
• Where and who
• Why
Planned release:
end of 2014
• Consortium-provided
content
• Cross-comparison of
consortia
• Point-of-contact
objectives• Share findings from analysis of
the consortia landscape
• Propose a new framework for
measuring the value of
research-by-consortia efforts
• Have an open dialogue around
the utility and feasibility of
measuring consortia value
Metrics
- collaboration and partnerships
- framework of consortia
What is important to you?
- output, efficiency
Who and what
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
More than half focused on disease/condition
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
Sharing comparator arm
data from clinical trials
Research assays,
animal models
Genomic/clinical
databaseT2D patients
AgedBrainSYSBIO
Age-associated
pathways
Breadth-of-scale: Innovative Medicines Initiative
€1 952 573 292
€ 756 906 619
Infectious diseases - 39%
€ 213 636 872
Drug discovery - 11%
€ 186 102 324
Brain disorders - 10 %
€ 118 189 462
Metabolic disorders - 6%
€ 116 287 312
Drug safety - 6%
€ 76 872 548
Stem cells - 4%
€ 74 004 854
Cancer € 74 345 401
Data management 4%
€ 68 069 432
Inflammatory disorders
€ 55 930 954
Biologicals
€ 49 310 000
Geriatrics
€ 39 901 138
Lung diseases
€ 38 994 284
Education and training
€ 30 531 192
Sustainable chemistry
€ 20 462 255
Drug delivery
€ 18 118 249
Drug kinetics
€ 14 910 397
Relative effectiveness
IMI
fundingCorporate
contribution
IMI Report: May 2014 Highlights
Consortium lifespan: 5 - 6 years
Inception
Ramp up
Mid-stream
Wind down
Closure
| 1 year | 2 - 3 years | 1 year |
Scientific challenge
Sponsor engagement
Governance
Agreements
Tool concept
Engaging tool-builders
Project plan
Project launch
Team culture
Infrastructure
Project execution
MilestonesDeliverables
Licensing/IP
Dissemination Data management
Licensing/IP
Dissemination
Royalties
Evaluations should be simple
Hub-and-spoke – central source of information
Innovative Medicines Initiative, Critical Path Initiative,
Foundation for the National Institutes of Health,
Health & Environmental Sciences Institute
Formalized agreements and governance
transparency
Established timelines and milestones
project management
Evaluation = Support
Inception
Ramp up
Mid-stream
Wind down
Closure
| 1 year | 2 - 3 years | 1 year |
Financial and in-kind commitment
Monitoring & EvaluationSteering
Committee
Board of
Directors
Many formal evaluationsSteering Committee Board of Directors
Sponsors Consortium Staff
Research Team
Many informal evaluationsSteering Committee Board of Directors
Sponsors Consortium Staff
Research Team
How do you evaluate
consortia?
What do you value?
Efficiency
- Convening
- Executing
- Managing
- Concluding
Output
- Level of adoption
- Business strategy alignment
- Government roles
- Creating opportunities
- R&D cost/time/efficiency
Output: eye of the beholder
Government • public health
• regulatory science
• de-risk innovation
• economic growth
• state-of-science research guidance documents
Industry • accelerate pipeline
• new therapeutic area
• access resources
• de-risk innovation
• access intellectual
capital
Academia • access resources
• opportunities for
publications
• training opportunities
• identify collaborators
Patient
organizations
• accelerate pipelines
• advance basic
research
• de-risk medical
product development
Consortium
researchers
• simplify day jobs
• access resources
• networking
• training / education
Bibliometrics
• By the end of 2013, IMI projects had
delivered over 600 scientific
publications in over 300 journals
• The citation index of papers from IMI
projects is twice the world average,
and higher than the EU average..
Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013
Bibliometrics and collaboration
Pre IMI funding award Post IMI funding award
Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013
Collaborations – who / what
Co-authorship – 69%Cross-sector collaboration – 42%Cross-project collaboration – 37%Cross-disease collaboration – 31%
IMI researcher networks by sector
Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013
Value of consortiaHow will the output be used? Is consortium on-track?
Therapeutic area core strategy vs opportunistic
Platform methods / tools clinical trials, personalized medicine,
data standards / exchange, assays
Others?
Project Name Outcome Output Area
IMIDIA Smaller clinical trials and Personalized
medicine; Faster development times,
Reduced attrition, and Predictive
models
Biomarkers and
personalized medicine;
Efficacy
Diabetes
COMPACT Faster development times, Reduced
attrition, and Predictive models
Efficacy Biologicals
Safe-T Smaller clinical trials and Personalized
medicine; Faster development times,
Reduced attrition, and Predictive
models
Biomarkers and
personalized medicine
Drug Safety
Examples of IMI consortia
Not all consortium outputs are publishable – licenses, databases
Publications are retrospective, rarely primary/secondary deliverable
Different stakeholders = different expectations on output
Bias: "Sexiness" of the science
Virtual collaborations - no dedicated laboratory/workspace
Semi-committed teams - not their day jobs
Human capital - turnover, advancement
Numerous consortia, different operational models
- cross comparison?
Complexities for evaluation by output
#
What do you value?
Efficiency
- Convening
- Executing
- Managing
- Concluding
Output
- Level of adoption
- Business strategy alignment
- Government roles
- Creating opportunities
- R&D cost/time/efficiency
Evaluating efficiency
Tracking progress - convene to perform
Coordinating virtual teams• Within work streams
• Across work streams
• With governing bodies
Resolving bottlenecks• Maintaining scope
• Appropriate expertise / resources
• Communications
• Conflicts / adaptability
• Team member turnover
Dynamics of teamwork
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Wooten, U. Houston, Science of Team Science conference
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Development- goals, mission
Conceptualization- research question, framework
Implementation- launch, conduct
Translation- application
Wooten, U. Houston, Science of Team Science conference
Hall et al, Trans Behavioral Med (2012)
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Assess situation- recognition
Plan formulation- goal setting, expectations
Plan execution
- monitoring, communication, coordination
Team learning- lessons learned
Wooten, U. Houston, Science of Team Science conference
Burke et al., J. Applied Psychology (2006)
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Wooten, U. Houston, Science of Team Science conference
Tuckman & Jensen, Group and Organizational Studies (1977)
Forming- tasks, strategy, team
Storming- roles and interactions
Norming- rules, roles, expectations
Performing- tasks, implementation
Adjourning- finalizing
Tracking consortium progress via metrics
Inception
Ramp up
Mid-stream
Wind down
Closure
Collective orientation
Interpersonal relations
Goal setting
Teamwork concept
Knowledge
consideration
Role clarification
Team subgroups
Cohesion / collective efficacy
Evolved interpersonal
relations
Maintaining shared vision
Problem solving / adaptability
Knowledge accommodation
Evolved role clarification
Autonomy & interdependence
Collective knowledge
transformation
Evolved interpersonal relations
Defining accomplishments
Problem solving
Mediated information
exchange
Autonomy & interdependence
Convene Perform Transition
Collective orientation
Interpersonal relations
Goal setting
Teamwork concept
Knowledge
consideration
Role clarification
Team subgroups
Cohesion / collective efficacy
Evolved interpersonal
relations
Maintaining shared vision
Problem solving / adaptability
Knowledge accommodation
Evolved role clarification
Autonomy & interdependence
Collective knowledge
transformation
Evolved interpersonal relations
Defining accomplishments
Problem solving
Mediated information
exchange
Autonomy & interdependence
Which phase?
Convene Integrate Implement
Collective orientation
Interpersonal relations
Goal setting
Teamwork concept
Knowledge
consideration
Role clarification
Team subgroups
Cohesion / collective efficacy
Evolved interpersonal
relations
Maintaining shared vision
Problem solving / adaptability
Knowledge accommodation
Evolved role clarification
Autonomy & interdependence
Collective knowledge
transformation
Evolved interpersonal relations
Defining accomplishments
Problem solving
Mediated information
exchange
Autonomy & interdependence
Leveraging human capital
Periodic survey of team dynamics
Steering Committee Board of Directors Research Team
Consortium Staff
62%
coherence in mission
35%
contribution
Correctional action:
- Increased face-to-face interaction
- Document-sharing technology
- Conflict resolution
Framework for reports
Sponsors
Operational
efficiency
Alignment to
strategy
Consortium Staff
Metrics = better communication?
• Output - Did the team deliver?
• Technical milestones – binary
• Team dynamics - Could the team have done better?
• Leverage resources and expertise
• Adaptability
Mid-term report overview
Technical progress: 4 / 5 milestones accomplished
Stage of team: Perform
Team integration across disciplines
Document sharing / development
Researcher engagement
Steering committee alignment
Interdependency defined
Conflicts resolved
How was output used?
IMI Executive Office and other consortia
Business strategy
• Open new therapeutic approaches and research avenues
• Reduce R&D costs, time to market and development risk
• Increase the efficacy and/or safety of existing drugs
Indirect benefits
• Education and training
• Spin off companies
• New partners (patients, foundations, academic, SME)
• Increased interest in geographic investments
• Implementation of standards / best practices / tools into strategy
• Informing regulatory science (policy / guidelines)
• Publication output and extent of collaboration
• Intellectual Property metrics
Others
Design• Consortium management
• Consortium participants
• Sponsor/stakeholder
Refine• Other consortia – managers/participants
• Other sponsor within same sector
Pilot• Several consortia
Optimize• Analyze / Evaluate
• Optimize survey vehicles
• Re-pilot
?
Utility• Need for these evaluations?
• Inform best practices?
• Other non-consortium partnerships?
Approach• Right approach?
• Aligning consortia with business strategy necessary
after concept development?
• Generalizable?
• Other key elements to measure?
• Indirect effects?
• Who/how to pilot?
Implementation• How to measure (surveys, etc)?
• Who measures?