Using Metrics to inform the Return on Investment (ROI): A Public Funder’s Perspective

Sean Coady, Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute

NIH Virtual Workshop on Data Metrics February 19, 2020

2

NHLBI Data/Materials Sharing

database of Genotypes and Phenotypes (dbGaP), GWAS, microRNA, gene expression profiling

Trans-Omics for Precision Medicine (TOPMed); WGS, metabolic profiles, protein and RNA expression, DNA methylation

BioData Catalyst Storage, Toolspace, Access and analytics for big data Empowerment

BioLogic specimen & data repositories INformation Coordinating Center (BioLINCC); biospecimens & phenotypic data

Clinical TrialsObservational Studies

3

Background – Data Repository

1999 NHLBI IRB approves data repository protocol. Website opens in 2000 (NIH IRB # 12-H-N198)

Purpose:In order to take full advantage of NHLBI supported clinical trials and epidemiologic studies and maximize their research value, data should be made available, under appropriate terms and conditions, to the largest possible number of qualified investigators in a timely manner.

Background BioLINCC

BioLINCC established in 2009 to coordinate activities of Data Repository and Biorepository

4

5

Current controlled access portfolio: 50 Observational studies (778,732 participants), 157 trials (460,816 participants)

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28

Cardiovascular CTTransplant/Transfusion CT

Primary/Sec. Prevention CTAsthma CT

Other CTOther OS

Heart Failure CTDisorder Population OS

Other Lung CTEmergency Medicine CT

General Population OSRespiratory Distress CT

Race/Ethnic Specific OSSickle Cell CT

Number of Studies

Data Repository Metrics

Utilization Metrics

Outcome Metrics

Effort Metrics Number of interactions and time to complete data access request Support encounters Monitoring

6

Utilization Metrics: Completed data access requests

0

50

100

150

200

250

300

350

400

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Fulfi

lled

Dat

a Ac

cess

Req

uest

s

7

Utilization Metrics: Cumulative data access requests for trial and observational study data (thru, 2019)

8

Utilization Metrics: Observed / Expected Utilization by study type

9

0 1 2 3 4 5 6 7 8 9 10

General Population OSRace/Ethnic Specific OSRespiratory Distress CT

Primary/Sec. Prevention CTHeart Failure CT

Disorder Population OSCardiovascular CT

Other OSOther CT

Other Lung CTEmergency Medicine CT

Sickle Cell CTAsthma CT

Transplant/Transfusion CT

Observed Data Requests (% of Total)/ Expected Data Requests (% of Portfolio)

10

Utilization Metrics: Trends in Utilization by Study Type

0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0 24.0

General Population OSRespiratory Distress CT

Heart Failure CTRace or Ethnicity Population OS

Primary/Sec. Prevention CTCardiovascular CT

Other OSDisorder Population OS

Emergency Medicine CTOther CT

Sickle Cell CTOther Lung CT

Transplant/Transfusion CTAsthma CT

Average number of data access requests per study per year

2013-2015 2016-2019

Utilization Metrics: Primary reason for data access request

11

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

1. New Question 2. Meta Analysis 3. Statistical Methods

4. Clinical Trial Methods 5. Comparison Group 6. Clinical Prediction, Risk Prediction

7. Other (Pilot, Simulation)

Utilization Metrics: Time from availability of study to first access request

CT: 60.8%OS: 74.8%

CT: 83.5%OS: 91.6%

CT:. Median 270 days.OS: Median 205 daysLog-rank p=0.296

12

Outcome Metrics: Publications

0

20

40

60

80

100

120

140

160

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Cou

nt o

f pub

licat

ions

Number of publications per calendar year

13

Outcome Metrics: Time to “Incident” publication by type of study (through 2019)

14

Log-rank p=0.858

1 YearCT: 7.3%OS: 2.1%

3 YearCT: 22.0%OS: 25.4%

5 YearCT: 41.5%OS: 46.1%

1179 publications596 Included data from 1 or more trials

69 Trials 1+ publications

506 investigators have published at least once. Mean number of publications per investigator publishing=3.0

15

Outcome Metrics: Citation Percentile (Top N%) publications using repository trial data, observational data and 10% random sample of all NHLBI supported articles (Articles thru May 2015)

Kruskal-Wallis Test p=0.333

Outcome Metrics: Workforce training

23% of completed data access requests indicate primary user has 0-5 years of research experience

QVR search of NIH grant applications found only two training applications (1 K01, 1 K99) mentioned BioLINCC

16

Messages from the metrics

Demand for secondary use of data from clinical strudiescontinues to steadily increase

Growing demand for data from clinical trials Metrics can suggest gap areas Data from repositories fulfill a range of purposes Unclear if publication rates are low Citation metrics suggest quality of publications utilizing repository

data are similar to publications supported directly by the Institute Need to better assess role of repositories in training new

investigators

17