Designing, Developing, and Evaluating Team-Science Support in a

Research Development Office

Overview

• Introductions• Overview of supporting TS and how RD professionals can help (HFK)• Building a RD office focused on TS support (BR)• Hiring and integrating a TS expert/facilitator into an existing RD office (DW)

Tools available to help (HFK)

#NORDP2019 Providence, RI April 29 – May 1, 2019

Betsy Rolland, PhD MLIS MPH

Director, Research Development & Team Science, Carbone Cancer Center and Institute for Clinical

and Translational ResearchUniversity of Wisconsin-Madison

David Widmer, PhDManager, Scientific

DevelopmentMemorial Sloan Kettering Cancer

Center

Holly J. Falk-Krzesinski, PhD

Vice President, Research Intelligence

Global Strategic Networks Elsevier

#NORDP2019 Providence, RI April 29 – May 1, 2019

5

What is Team Science?!

Cross-disciplinary Research

Collaboration

Team Science

"The universal increase of team science, its increasing impact, and boundary-spanning creative capacity all point to team science as central to the future of scientific and technological

advancement.” Professor Benjamin Jones, Northwestern University, SciTS Conf 2013

6

Research Development for Team Science § What

§ Navigation and support for collaboration, cross-disciplinary research, research teams, and grantsmanship for collaborative opportunities

§ How§ Focusing on connecting the science of team science (empirical

research on scientific teams) and the praxis of team science (the practical aspects of conducting science in teams)

§ Who§ Translate empirical research findings about team science into

evidence-based effective practices for scientific teams, team leaders, and institutional leadership

7

Research Development for Team Science § Collaboration Facilitation

§ Collaborator identification and referral/facilitation § Catalytic research development events (e.g., interdisciplinary meetings, research

symposia)§ Proposal Development

§ Collaborative research & funding opportunity identification§ Grantsmanship & proposal development support for collaborative grant

opportunities§ Team Science Training

§ Team science grantsmanship training§ TeamScience.net online tool§ Leadership training§ Collaborative Communication Workshops

§ Policy§ Appointment, Promotion & Tenure guidelines

Needs assessment at UW: What is the problem we are trying to solve?

• Low levels of federal funding vs comparable large R1 institutions• High levels of institutional funding (pilots, internal grant programs) – reduced

motivation to submit large grants• Need to turn UWCCC and ICTR pilots into federal funding• Low levels of large, team-science projects (U, P, multi-PI, etc.) – organizational

barriers and lack of incentives to submit• Support for team science is minimal, scattered, not discoverable, and not widely

available• Objective: to increase education about what’s available and what’s necessary

• Overarching Goal: • To slowly, iteratively build a suite of evidence-based services available to large

teams

#NORDP2019 Providence, RI April 29 – May 1, 2019

Building a New RD office focused on TS Support

• New position: Director of Research Development and Team Science• 25% Carbone Cancer Center (UWCCC)• 25% Institute for Clinical and Translational Science (ICTR, our

UW CTSA, as Director of the Team Science core)• 25% School of Medicine and Public Health (SMPH)

• Areas of Focus: • Providing RD and TS support for team-based proposals• TS education for individuals and teams• TS interventions for new teams or struggling teams• Science of Team Science research (ICTR)

#NORDP2019 Providence, RI April 29 – May 1, 2019

Team Science Services

#NORDP2019 Providence, RI April 29 – May 1, 2019

Research Program Development

Funding and Collaborator Identification

Proposal Development

Collaboration Start-up

Project Evaluation and Reporting

• TS meeting facilitation

• Guidance on research integration

• Monitoring coming RFAs

• Convening teams for new announcements

• Matchmaking, espfor methodologists

• Governance strategy

• Facilitate discussions around TS elements of proposals, help with writing, esphow to respond to TS-specific review criteria

• Collaboration and team management plans

• Development of Manual of Operations (MoO)

• Implementation of governance strategy

• Communications strategy and systems development (portal, listservs, social media)

• TS and communications training for team

• Meeting facilitation, espkick-off and full-team F2F meetings

• Set-up of process measures for collaboration

• Conduct qualitative assessments of collaboration

• Interventions, if warranted, to increase team functioning

Intake Process

• We can’t help everyone (hopefully)• Started by prototyping processes with individuals or very small teams• As we build to larger teams, need to create criteria for who gets our limited

resources. Possible criteria:• Institutional priority areas• Funding agency priority areas (esp for UWCCC and ICTR center grants)• Complexity of proposal’s team-science or interdisciplinarity requirements• Size of team • Amount of funding• Maturity of the team

#NORDP2019 Providence, RI April 29 – May 1, 2019

Metrics (under development)

• Goal: assess the team-science services and their impact• How are the supported teams doing?

• Measures of collaboration (HFK to discuss tool)• Long-running teams (teams work together long-term: via grants and publications)

• How are we doing with the services we provide?• Strong demand for team-science services• Satisfaction surveys: were we helpful, did the teams feel they got the help needed?• Resources available for teams and usage of those resources (e.g., governance plans,

authorship policies, multi-PI plan samples)• # teams submitted grants with TS sections or review criteria that we helped• # new teams convened for coming RFAs or high-priority topics• # grant kick-off meetings convened and facilitated• # team-science interventions delivered to both nascent and established teams• # evaluation plans developed and supported• # individuals and teams trained in TS best practices • $ submitted and secured

#NORDP2019 Providence, RI April 29 – May 1, 2019

Science of Team Science

• Develop evidence-base and training for support of Team Science Facilitators

• Goal of TSFs: To offload the “collaboration work” of Team Science onto a trained facilitator with deep scientific knowledge

• Potential activities: • Strategic planning• Collaboration development• Facilitating difficult and contentious discussions (as

well as warm and cuddly ones)• Leading frequent, iterative, and corrective

program evaluations• Developing and implementing consortium-level

metrics of success• Planning next steps, particularly scientific and

policy translation• Identifying synergies across intraconsortium

projects, as well as making connections with related projects elsewhere

#NORDP2019 Providence, RI April 29 – May 1, 2019

November 1 2017 (26) (11) 1581-1582; DOI: 10.1158/1055-9965.EPI-17-0471

Goal: To craft an intervention and test across the CTSAs

Integrating Team Science into an Existing RD Office

MSK in a Nutshell

#NORDP2019 Providence, RI April 29 – May 1, 2019

Memorial Sloan Kettering Cancer Center(incorporated 1960)

Memorial Hospital for Cancer and Allied Diseases

(founded 1884)

Sloan Kettering Institute for Cancer Research

(founded 1945)

14 Departments 9 Programs

22 Collaborative Research Centers

1,300 Faculty 200 Research Labs 1,000 Fellows 1,133 Clinical Trials

RA at MSK in a Nutshell

#NORDP2019 Providence, RI April 29 – May 1, 2019

Office of Research and Project Administration ORPA

Grants & Contracts (G&C)

Grants Management

Specialists (GMSs)

Research Funding

Development Managers

(RFDMs)

Contracts Team

Research Financial

Management (RFM)

Research Fund Managers

(RFMs)

Effort Compliance

Manager (ECM)

Financial Reporting

Compliance Manager

Research Financial Planning

(RFP)

Operations(Ops)

Clinical Research

Finance (CRF)

Budget Analyst

Clinical Trial Analyst

Revenue Management

Specialist

AR/AP Analyst

Funding Development Team

G&C FDT in a Nutshell

#NORDP2019 Providence, RI April 29 – May 1, 2019

Grants & Contracts’ Funding Development Team (G&C FDT) Responsibilities:

Offers proposal review assistance,

utilizing grants expertise and knowledge of

proposal "marketing skills"

to assist in development of

winning applications

Identifies funding sources,

Engages in proposal development activities, and Interprets and

explains funding agency

requirements.

Provides outreach and training to

faculty and staff on funding search databases and strategies and resources for proposal and application

development.

Additionally, support needed to:• Stimulate collaborative research development• Provide hands on support for complex proposals, particularly for basic and

translational sciences• Expand scientific knowledge on the team to allow for more FOA/PI matching;

redundancy for team lead Adding an incremental position focused on complex grants and team science:

• Fostering collaborative team science and the early stage development of complex grants

• Oversight of the advanced stages of proposal development (editing, data analysis, scientific review, cohesiveness)

The Funding Development Team 2.0

MSK Lead Applications 2015 2016 2017 2018 Total

Total Complex Grants Submitted (Ps, Us) 10 12 12 10 44

# of Complex Grants submitted where basic science faculty is Project Leader

4 3 4 7 18

# of Complex Grants submitted where basic science faculty is PI

1 2 1 0 4

Complex Grant Submissions 2015-18

#NORDP2019 Providence, RI April 29 – May 1, 2019

Justification of Role

Garnering support for an incremental position.

Main focus of role:1. Stimulate collaborative research development and partnership and foster innovative,

collaborative team science among cross-disciplinary faculty and researchers2. Catalyze with faculty members new programs, research initiatives, and opportunities

by identifying common and complementary research interests3. Partner with department and Center leadership to foster team science 4. Provide direct and coordinated support for proposal development and the

production of high value/impact complex grant proposals 5. Utilize scientific knowledge and expertise to enhance the Funding Development

Team’s capacity and effectiveness in identifying funding opportunities

#NORDP2019 Providence, RI April 29 – May 1, 2019

Strategic Value to the Institution• Soliciting collaborators through

understanding of PI’s research projects and gaps/needs

• Maintaining/managing Faculty Research Project Database

• Speed Dating on Scientific Techniques to encourage organic team development

• In-depth Complex Grant Development

• Scientific support for funding development

#NORDP2019 Providence, RI April 29 – May 1, 2019

Building a Job DescriptionEnvironmental Scan – Holly’s JD library

Qualifications• Advanced science degree (PhD or MD)• 4-7 years’ experience in research development in a biomedical science field• Scientific writing and editing experience• Demonstrated effective leadership ability to develop strategies & outreach efforts to maximize research funding• Understanding of grant proposal strategy and work experience structuring projects to increase competitiveness • Extensive knowledge of NIH grant funding mechanisms, policies and procedures• Experience supporting multi-disciplinary, multi-investigator complex grants development

Main responsibilities of role:• Scientific Grant Team Development• Funding Development• Proposal Development

#NORDP2019 Providence, RI April 29 – May 1, 2019

Integration into FDT

Before and after – Complementary & Additive

• Faculty Engagement

• Proposal Development

• Funding Development

• Complex Grants Coordination

#NORDP2019 Providence, RI April 29 – May 1, 2019

Who Gets Support?

• Criteria for Support:• Why say “Yes” to some rather

than How to say “No” to others

• Prioritization: If everyone is a priority than no one is

#NORDP2019 Providence, RI April 29 – May 1, 2019

Who Supports Us?

• Tapping Institutional and Sponsor Knowledge and Expertise

• MSK Leadership• Society Members/Officers• Program Personnel

#NORDP2019 Providence, RI April 29 – May 1, 2019

Hiring Process: Scientific Development Manager

• Challenges

• Clarification of responsibilities when responsibilities are untried• Use others’ experiences as a guide to best practices• Ground floor – try new things• New mechanism: embrace ambiguity

• Succinctly show integration

#NORDP2019 Providence, RI April 29 – May 1, 2019

27

TOOLS YOU CAN USE

28

SciTS Group on Mendeley

https://www.mendeley.com/community/science-of-team-science-(scits)/

29

P&T Policy Recommendations

Klein, J. T. & Falk-Krzesinski, H. J. Interdisciplinary and collaborative work: Framing promotion and tenure practices and policies. Res. Policy, In Press (2017).

30

A Field Guide/Partner Agreement§ Overall Goals & Vision§ Who Will Do What§ Sharing/Storing

Reagents & Data§ Authorship, Credit§ Contingencies &

Communicating§ Conflict of Interest

See also Bennett, L.M., and Gadlin, H. (2012). Collaboration and Team Science: From Theory to Practice. J Invest Med 60, 768-775

https://www.cancer.gov/about-nci/organization/crs/research-initiatives/team-science-field-guide

31

A Toolbox Dialogue Initiative (TDI, formerly Toolbox Project) ‘Collaborative Communication Workshop’ provides a philosophical yet practical enhancement to cross-disciplinary, collaborative science. Rooted in philosophical analysis, the Toolbox workshop enables investigators, research development professionals, project managers, and collaborators to engage in a structured dialogue about their research assumptions and cross-disciplinary collaboration. This yields both self-awareness and mutual understanding, supplying individuals with the robust foundation needed for effective collaborative research. Led by Toolbox Project Facilitators, Workshop participants will engage in small group discussion and share respective views in response to a number of probing statements about science motivation, methodology, confirmation, objectivity, values, and reductionism.

Toolbox Dialogue Initiative

http://tdi.msu.edu/

32

Individual Collaboration Readiness Tool

§ The Motivation Assessment for Team Readiness Integration and Collaboration (MATRICx) is a psychometric instrument that measures motivations and threats to collaboration in knowledge producing teams (KPTs) of biomedical and health professionals. It is calibrated using Rasch analysis and provides users with individual, team, and composite profiles of collaborative and cooperative strength.

http://matricx.net/

33

Collaboration Success Wizard§ On-line diagnostic survey for geographically

distributed collaborations. The survey probes factors that may strengthen or weaken the collaboration. The Wizard provides both personal and project-level reports to help build successful and productive collaborative projects.

http://hana.ics.uci.edu/wizard/

34

Mendeley to Suppport CollaborationCollaborate with colleagues worldwideCreate a new group dedicated to your topic and invite colleagues from all over the world to join. You can also create Private Groups that are only visible to invited members, letting you share information securely. Groups make it easier to discover ideas and inspire new ones.

Curate and share reading listsYou can browse other members' public reading lists to discover relevant content. You can also review articles with your collaborators. When a group member adds a note/highlight/summary to a document, the edit is visible to all members of the group

Engage in interesting discussionsConnect with like-minded researchers in institutions around the world so that you can combine your knowledge and find new avenues for research.

35

Levels of Collaboration Survey

Frey, B.B., Lohmeier, J.H., Lee, S.W., and Tollefson, N. (2006). Measuring collaboration among grant partners. American Journal of Evaluation 27, 383-392.

36

Case Studies: Portable Team Science Training

37

TeamScience.net

References: Hesse, B.W. (2011). COALESCE (CTSA Online Assistance for Leveraging the Science of Collaborative Effort). JAMA: The Journal of the American Medical Association 306, 1925-1926.

www.teamscience.net

• Learn to perform trans-disciplinary, team-based translational research

38

Collaboration Plan Development

https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceBiblio.aspx?tid=3&rid=3119http://www.teamsciencetoolkit.cancer.gov/public/TSResourceBiblio.aspx?tid=3&rid=3261

39

NIH§ Scored (Core) Review

Criteria§ An existing standard Scored

Review Criterion§ Additional team science-specific

review elements associated with one or more of the five standard Scored (Core) Review Criteria

§ A new, additional sixth Scored Review Criterion

§ Additional Review Criteria§ Extra Review Elements

NSF§ Additional Solicitation

Specific Review Criteria§ Systems Approach§ Interdisciplinary Integration§ Network Structure§ Management, Organization and

Evaluation§ Transdisciplinarity/Synergy§ Quality and Value of

Collaboration§ Others

Team Science-Specific Review Criteria

Source: Holly Falk-Krzesinski, initially presented at NORDP 2018

40

Team Science Toolkit

www.teamsciencetoolkit.cancer.gov

41

SciTS Listserv§ The Science of Team Science (SciTS) listserv facilitates conversation

among individuals who are engaged in, studying, or managing team science, in the US and internationally. The listserv is maintained collaboratively by the SciTS Team at the National Cancer Institute, Division of Cancer Control and Population Sciences, Behavioral Research Program (http://cancercontrol.cancer.gov/brp/scienceteam) at the NIH.§ TO SUBSCRIBE: Send an email with a blank subject line to: listserv@list.nih.gov. The message body

should read: subscribe SciTSlist [your full name]. Please do not include the brackets. For example, for Robin Smith to subscribe, the message would read: subscribe SciTSlist Robin Smith. You will receive a confirmation email.

§ TO POST TO THE LISTSERV: Send an email to SciTSlist@list.nih.gov. Any subscriber may post to the list.

§ TO VIEW THE ARCHIVES: To view the archives of all previous postings, go to: http://list.nih.gov/archives/SciTSlist.html

§ TO RECEIVE MESSAGES IN A DAILY DIGEST: The default setting sends you each message as it is posted to the listserv. To receive one daily digest, instead, go to: http://list.nih.gov/cgi-bin/wa.exe?SUBED1=SciTSlist&A=1 and select “digest” as your subscription type.

§ TECHNICAL PROBLEMS WITH YOUR SUBSCRIPTION? Contact the list administrator, Judy Kuan, at: kuanj@mail.nih.gov. Please be sure to state that your email is in reference to the SciTS listserv.

42http://www.scienceofteamscience.org

Science of Team Science Conference

43

Professional Society for the Science of Team Science

§ The problems impacting our world are increasingly complex, yet we are able to address them through scientific pursuit now more than ever. Addressing many complex problems depends on cross-disciplinary collaboration, a key component of team science.

§ INSciTS serves as a conduit to ensure that empirical findings about team science are accessible and useful for effective collaborative practice and policy, bridging both science and praxis.

§ INSciTS membership draws together researchers, scholars, practitioners, funders, thought leaders, tool developers, policy makers, and institutional leaders from all disciplines and types of organizations.

§ Together, we are dedicated to enhancing fundamental understanding of how to collaborate and to best support and foster team science.

§ Research generated by INSciTS’ broad and diverse community can enhance the effectiveness of science teams, as well as diverse research collaborations across social sciences, humanities, and professions.

§ We strive as an organization to translate evidence-based findings to improve the global scientific enterprise and best serve society’s needs. Furthermore, INSciTS is focused on preparing the next generation of experts, equipped with the skills and perspectives necessary to lead, train, and participate in team science.

44

National Academies Consensus Report

http://sites.nationalacademies.org/dbasse/bbcss/currentprojects/dbasse_080231

45

Academy of Medical Sciences Report

http://www.acmedsci.ac.uk/policy/policy-projects/team-science/

46

Canadian Academy of Health Sciences Report

http://www.cahs-acss.ca/academic-recognition-of-team-science-how-to-optimize-the-canadian-academic-system/

47

Fostering Collaboration

https://p.widencdn.net/fi0lp1/ACAD_RL_RI_BRO_FosteringCollaboration_EN_WEB

48

Connect with Me

Holly J. Falk-Krzesinski, PhD

Vice President, Research Intelligence Global Strategic Networks ¨ Elsevier

Chicago, IL, USA

h.falk-krzesinski@elsevier.comhttp://www.linkedin.com/in/hollyfk

@hfalk14+1 847-848-2953

Questions?

Holly Falk-Krzesinski: h.falk-krzesinski@elsevier.com

Betsy Rolland: brolland@wisc.edu

David Widmer: widmerd@MSKCC.ORG