June 2012 UpdateJune 14, 2012

Andrew J. Buckler, MSPrincipal Investigator,

QI-Bench

WITH FUNDING SUPPORT

PROVIDED BY NATIONAL

INSTITUTE OF STANDARDS AND

TECHNOLOGY

Agenda for Today• Progress on folder organization, naming

conventions, and ISA roll-ups (Gary)• Demonstration of multiple time point batch

analysis capability (Mike)• Progress on automated statistical analysis (Kjell)

22222222

First Generation: 3A Challenge• Created newly named data set upon

request so that all dicom files had an extension of dcm.

• These files follow our 1st generation standard for naming file layout and results.

• Meeting program needs; but from system point of view, essentially no automation.

33

Infrastructure progress for next generation:

• Created naming conventions and file formats for roll up files.

• Created scripts for generating roll up files for location and change data.

44

Example: Reference Data Sets for QIBA compliance

• Our most fully 2nd generation compliant reference data sets.

• Totality of data from DICOM to analysis is included.

• Study data, in ISA-TAB (like) structure is available for analysis.

55

MULTI-TIMEPOINT BATCH ANALYSIS DEMO

6

Demonstration of automated analysis using R controlled by Iterate.

7

Iterate workflow for running an R script

777777

Test/Re-test Study• Test/Re-test is one of the building-block study designs for

understanding a measurement system• Helps us to understand the variation we see in the system

under the “no change” condition

• Fundamental components of the analysis– Scatterplot of test values versus retest values– Mean versus difference of test/re-test values (i.e. Bland-Altman)– Intraclass correlation coefficient (ICC):

• variation due to the subjects (class) relative to the unexplainable variation.– Minimum detectable change (MDC):

• minimum change considered to be statistically significant

888888

Test/Re-test Example• CT Volumetry data

– 32 subjects in “coffee break” design• 2 subjects excluded as uncomputable

– Analysis of volume calculations

• Results can be generated for any study that follows the Test/Re-test format

999999

101010101010

Volume Difference

value1 value22 7.554111 8.0970284 771.499954 654.2557966 22099.090961 22461.1688568 45558.404866 45411.06484310 8.713604 7.79649012 7824.517733 8166.32472514 31671.844931 29196.96870316 12.363709 13.83378518 19.949875 19.91985420 11.560880 6.16771422 37485.766057 32506.89271224 2.451236 2.45123926 33261.732159 12771.46856028 7720.802020 11.45661330 18012.668569 18377.25751232 8725.320453 8880.53900034 7.780063 8.64938936 16.691373 56622.55576738 8.241816 6070.07127240 16.342152 10.60224442 13.167784 12.48320944 6593.866812 6193.35183746 6.043523 6724.99050748 5.826424 4.81851450 6683.346163 7479.94868052 15.816401 18.15936354 7432.544980 7393.72354356 6.786277 6.78628158 26144.753408 31253.57102360 717.220963 745.647667

111111111111

121212121212

131313131313

Volume Summary

Volume Calculation N Mean Standard Deviation Min Median Max1 ProportionalChange 30 3.51 38.66 -99.70 0 99.942 VolumeDifference 30 1339.28 11401.51 -20490.26 0 56605.86

141414141414

Intraclass Correlation

Intraclass Correlation for Test-retest: 0.68 Minimum Detectable Change : 20835.38

Other Building-Block Studies(In Process)

• Inter- and intra-reader variability

• Studies of bias and variation when truth is known via phantoms

• Studies for understanding the variation due to standard, uncontrolled factors (e.g. site, machine, etc.)

151515151515

Formal Deployment: QIBA/RIC instance• We are well on the way to having a

“third party” installation of Execute.• These reference data are largely in

our 2nd generation format for naming and layout.

• Whereas CT volumetry has been (and continues to be) the focus of the test bed, the first use of the RSNA instance is DCE-MRI.

1616

1717

Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing

standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them

without huge costs– Public wants to trust the decisions that they contribute to

• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data

• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders

1818

Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased

statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of

generalizability. • We make formal specification accessible to diverse groups of experts that are

not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into

representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or

requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of

collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.

1919

QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)

• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik)

• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)

• Collaborators / Colleagues / Idea Contributors– Georgetown (Baris Suzek)– FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)– Northwestern (Pat Mongkolwat)– UCLA (Grace Kim)– VUmc (Otto Hoekstra)

• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …

• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)

2020