June 12, 2018

Setting the Standard:NPCR and SEER Join Forces to

Establish Data Quality BenchmarksSerban Negoita, Clara Lam, Rebecca Ehrenkranz, Amy Solis,

Reda Wilson, Manxia Wu, Vicki Benard

!2

NCI SRP and CDC CSB: Joint Project on Data Quality

Objectives:

▪ Minimize work duplication, optimize the use of resources

▪ Develop benchmarks to be applied consistently independent of the funding agency

▪ Create reference points to assess the effects of 2018 NAACCR standards implementation

▪ Develop infrastructure for real-time monitoring of data quality

!3

Benchmarks - Definition

▪ Benchmark: A slang or jargon term, usually meaning a measurement or point of reference taken at the beginning of a survey or project, used for comparison with subsequent measurements of the same variable; sometimes it means the best or most desirable value of the variable. Alternatively, an acceptable standard in evaluation (e.g., of air quality, of performance).

Dictionary of Epidemiology, 2014 edition

▪ ‘point of reference’ ‘most desirable’ ‘acceptable standard’

!4

Benchmark Development: Grade (Differentiation) and Special Grade Systems

▪ Grade data element suggested by the CDC NPCR based on the results of the NPCR quality audits

▪ Data element frequently used in research

▪ Reference points for grade necessary: significant changes in 2018 for Grade data collection standards ▪ See NAACCR Grade Coding Instructions and Tables Manual

!5

Phase I: Planning/ Protocol Development

▪ Select period of observation, tumor behavior, histology criteria, etc.

▪ Select a two-grade system tumor site: urothelial cancers

▪ Select a three-grade system tumor site: stomach

▪ Select a four-grade system tumor site: colon and rectum

▪ Select cancer type-specific grade variables: ▪ Gleason Score for prostate cancer (SSF 6,8,10)

▪ Bloom - Richardson Score for breast cancer (SSF 7)

!6

Pre-analytic: SEER Program Coding and Staging Manual - Grade History Timeline

SEER Program Coding and Staging Manual: Grade History Timeline: Gleason Score, Prostate

2001 2002 2003200

4 2005200

6200

7 2008200

9201

0201

1201

2201

3 2014 2015 2016

Gleason's Score Grading Gleason's Score GradingSEER Value

Gleason's Score Grading

SEER Value

2--4I, well

differentiated 2--4 I, well differentiated 1 2--6 I, well differentiated 1

5--7II, moderately differentiated 5,6

II, moderately differentiated 2 7 II, moderately

differentiated 2

8--10III, poorly

differentiated 7--10 III, poorly differentiated 3 8--10 III, poorly differentiated 3

!7

Pre-analytic Phase: Collaborative Stage - Grade History Timeline

Collaborative Stage: Grade History Timeline: Gleason Score, Prostate

2001 2002 2003200

4 2005200

6200

7 2008200

9201

0201

1201

2201

3 2014 2015 2016

Collaborative Stage did not exist

SSF 6: Gleason's Score SSF 8: Gleason's Score on Needle Core Biopsy/TURPCode Description Code Description

0 Test not done 998No needle core biopsy/TURP

performed2 -- 10 Gleason's score 2--10 Gleason's score

988 - 999 NA/Unk 988, 999 NA/UnkSSF 10: Gleason's Score on Prostatectomy/Autopsy

Code Description998 No prostatectomy/autopsy performed

2--10 Gleason's score988,999 NA/Unk

!8

Pre-analytic Phase: AJCC - Grade History Timeline

AJCC: Grade History Timeline: Gleason Score, Prostate

2001-2002 2003200

4 2005200

6200

7 2008200

9201

0201

1201

2201

3 2014 2015 2016AJCC 5 AJCC 6 AJCC 7Gleason's Score (Description) Gleason's Score Description Gleason's Score Description2--4 (well differentiated) 2--4

Well differentiated/slight anaplasia

1 -- 6

Well differentiated/slight anaplasia5,6 (moderately differentiated) 5--6

Moderately differentiated/moderate anaplasia

7 (moderately poorly differentiated) 7--10

Poorly differentiated/undifferentiated/marked

anaplasia7

Moderately differentiated/moderate anaplasia

8--10 (poorly differentiated) 8--10

Poorly differentiated/undifferentiated/marked anaplasia

X Grade cannot be assessed X Gleason score cannot be processed

!9

Phase II: Pre-analytic – search for grade collection data quality standards

▪ Verify if benchmarks for grade collection available: ▪ NAACCR members

▪ IARC

▪ European cancer registries

▪ Lit research grade distribution reported by clinical trials and observational studies ▪ Gleason Score ranges across all studies in lit review: n>1,000

▪ GS 2- 6: 29 - 63%

▪ GS 3+4 = 7: 10 - 56%

▪ GS 4+3 = 7: 3 - 25%

▪ GS 8 -10: 1 - 20%

!10

Phase III: Analytic – Outliers Detection for Gleason Score

▪ Selection ▪ Analytic dataset provided by CDC team ▪ Includes SEER and NPCR funded registries, 2001-2016

▪ Gleason score analysis restricted to dx. years 2010-2016

▪ Selection criteria: prostate cancer, positive histologic confirmation

▪ Gleason score post-prostatectomy: surgery code 30-80 (confirming prostatectomy)

▪ Indicators (proportions) of interest: ▪ % cases Gleason Score Unknown (999)

▪ % cases Low Gleason Score 2-6 (prognostic group 1, least aggressive)

▪ % cases High Gleason Score 9-10 (prognostic group 5, most aggressive)

!11

Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry

SSF 10, Gleason Score = 999, Dx. Years 2010-2015

0%

3%

6%

9%

12%

T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A

!12

Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry (n = 36)

SSF 10, Gleason Score = 999, Dx. Years 2010-2015

0%

3%

6%

9%

12%

T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A

Q1: 25th

percentile

Q3: 75th

percentile

!13

Benchmarking Method: IQR Rules for Outliers

▪ Benchmarking method: Fences at Q1-1.5*IQR and Q3+1.5*IQR

% Gleason Score Unknown (SSF 10 = 999)

Q1 (25th percentil

e)Q3 (75th

percentile) IQRLower Fence Upper Fence

1.3% 2.6% 1.4% -0.8% 4.7%

% Low Gleason Score (SSF 10 = 2-6)25

Percentile

75 Percentile IQR

Lower Fence Upper Fence

21.4% 29.2% 7.8% 9.6% 41.0%

% High Gleason Score (SSF 10 = 9-10)25

Percentile

75 Percentile IQR

Lower Fence Upper Fence

5.9% 7.9% 2.0% 2.9% 10.9%

!14

Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry

SSF 10, Gleason Score = 999, Dx. Years 2010-2015

0%

3%

6%

9%

12%

T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A

!15

Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year

SSF 10, Gleason Score = 999, Dx. Years 2010-2015

0%

3%

6%

9%

12%

T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A

!16

Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry

SSF 10, Gleason Score 002-006, Dx. Years 2010 - 2015

0%

11%

23%

34%

45%

K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C

!17

Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year

SSF 10, Gleason Score 002-006, Dx. Years 2010 - 2015

0%

11%

23%

34%

45%

K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C

!18

Proportion High Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry

SSF 10, Gleason Score 009-010, Dx. Years 2010 - 2015

0%

5%

10%

15%

20%

C S EE DD Q J HH X W G AA N I GG Z U M F II D P B L R JJ CC Y T K FF E H BB O A V

!19

Proportion High Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year

SSF 10, Gleason Score 009-010, Dx. Years 2010 - 2015

0%

5%

10%

15%

20%

C S EE DD Q J HH X W G AA N I GG Z U M F II D P B L R JJ CC Y T K FF E H BB O A V

SSF 10, Gleason Score 009-010, Dx. Year 2016

0%

5%

10%

15%

20%

!20

Conclusions

▪ Large number of registries is needed to develop valid benchmarks

▪ Pre-analytic phase is important: ▪ Subject mater expertise necessary to select the right targets (indicators)

▪ Lit. research, research of international standards – limited return

▪ Numerous statistical methods available – additional research necessary to select the optimal method ▪ IQR-based fence method – relatively easy to implement

▪ Immediate applicability – it identified outliers in the distribution of Gleason Score for cases diagnosed in 2016

▪ Collaboration really helped, minimized effort to obtain datasets and history timelines

▪ Feasible, inexpensive method that resulted in reference points for assessing data completeness and frequency distribution Gleason Score

!21

Many Thanks to:

NCI Division of Cancer Control and Population Sciences Leadership

CDC Division of Cancer Control Leadership

Analysts at CDC and IMS:

Jessica King (CDC)

Denise Duran(CDC)

MaryBeth Freeman(CDC)

Jennifer Stevens (IMS) Registrars and all those

contributing to cancer surveillance

www.cancer.gov www.cancer.gov/espanol

!23

Benchmarks in Cancer Surveillance

Cancer system outcomes = cancer data sets Measurable characteristics of the cancer data sets Missing data (% unknown) Precision (% NOS) Validity (trueness) : re-abstraction/ re-coding : observed vs expected distribution

!24

Quality Indicators

▪ Current data quality indicators evaluate: ▪ Processes ▪ Case ascertainment - completeness

▪ Death clearance - DCO rate

▪ De-duplication - duplicate rate

▪ Data editing - edits-failing cases

▪ Quality of output (data sets, data elements) ▪ Sex, age, race, county at dx – all about demographics

▪ Both important, process indicators difficult to interpret and use ▪ No quality indicators for clinically-relevant data elements!

!25

Population-Based High-Quality Clinically-Relevant Data

▪ Reference points (benchmarks) ▪ define quality

▪ Methods to compare against reference points (benchmarking) ▪ Measure and track improvement

▪ Infrastructure/ institutional support (standards setters) ▪ Select the reference point and measurement methods

▪ Integrate tracking/monitoring in registry operations

▪ Use benchmarks to promote population-based cancer data sets

!26

Institutional support needed to develop benchmarks for cancer surveillance

!27

Providing High-Quality Cancer Data

▪ How does the cancer surveillance community: ▪ Defines high-quality?

▪ Measures improvement toward high-quality?

▪ Makes optimal resource allocation decisions to achieve high-quality?

▪ How to inform cancer data users on the high-quality of population-based surveillance data ▪ Differentiate our data products from other data sources

▪ Substantiate the need for additional resources

▪ Accomplish cancer control and research mission

!28

Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry

SSF 10, Gleason Score 002-006, Dx. Years 2010 - 2015

0%

11%

23%

34%

45%

K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C

Lowest % based on lit. research

!29

Complexity for Benchmark Selection

▪ Select reference points and measurement methods

▪ Prioritization decisions on data items that need benchmarks ▪ Treatment, stage, prognostic factors, cancer identification?

▪ Requires research, analysis, interpretation of existing data and publications

▪ Are there any existing benchmarks use by registries or oncology organizations, domestically or internationally?

▪ Statistical methods:

▪ Outlier detection

▪ Trend-based forecasting

▪ Measures of dispersion

▪ Observed versus expected categorical tests

▪ Other?

!30

Methods for Benchmark Selection

▪ Use of external datasets to select benchmarks: ▪ Administrative/Claims data for treatments

▪ Patterns of care: treatment, stage, prognostic factors

▪ Re-abstraction/recoding studies: ▪ Manual re-abstraction/re-coding ▪ Large sample size

▪ Multiple registrars

▪ Adjudications of best answers

▪ Natural Language Processing used for re-abstractions, automated re-coding/re-consolidation ▪ NLP algorithms need validation

!31

Project Timeline

▪ NCI-CDC team meetings started December 2017

▪ Multiphase project ▪ Planning/ Protocol development

▪ Pre-Analytic Assessment

▪ Analysis – evaluation of statistical methods ▪ Detection of outliers

▪ Trend-based forecasting

▪ Interpretation and recommendations

▪ Implementation in program activities ▪ Implementation in registry systems dashboards