Brussels, 20 March 2013Bart Vannieuwenhuyse 1

Topic 2-

Quality Metrics

Bart VannieuwenhuyseSenior Director Health Information SciencesJanssen R&D

Brussels, 20 March 2013Bart Vannieuwenhuyse 2

Scope of the project –

Purpose – “improving”

“what gets measured, gets done”

What are “Quality Metrics”?

A “metric” is a measure. “Quality” is something a “customer” defines. A “Quality Metric”, therefore, is a measure of quality as defined by the

customer.

NOTE 1: A “customer” might be defined as anybody with an expectation of receiving something of value in exchange for something else of value, either external to or internal to an organization.NOTE 2: Not all “Metrics” are “Quality Metrics”

Topic 2 – Quality Metrics

http://www.capatrak.com/Files/PresH%20-%20Metrics.pdf

Brussels, 20 March 2013Bart Vannieuwenhuyse 3

Contributing projects

Topic 2 – Quality Metrics

Brussels, 20 March 2013Bart Vannieuwenhuyse 4

Convergence challenges

Define scope – agree on areas with highest need

“Internal” vs “External” application of metrics

Potential opportunities to leverage (tbd)

Improving efficiency of collaboration in projectProcess to improve project deliverables

Measuring quality of (external) dataIdentifying quality of (sub)contractors

Topic 2 – Quality Metrics

Brussels, 20 March 2013Bart Vannieuwenhuyse 5

Topic 2 – report back

• Quality Metrics – domains:

– Project quality• Quality of deliverables – internal “peer review” generally adopted• Project management – “on time – on budget” generally adopted

– Project impact• Uptake of solutions – need for further development of metrics

(e.g. Service registry using text mining in BioMedBridges)• Scientific impact – publications, possibility to further improve on

speed and breadth of sharing results• Societal / health care impact – need for further development of

more standardized approaches

– Data quality …

Brussels, 20 March 2013Bart Vannieuwenhuyse 6

Data Quality

“Data quality is the end product of a whole process”

QType of Use

(Care – Research)

Context of

creation

Quality of Solution

Quality of Usage

Metrics 1 Metrics 2

“All elements need to be of the right quality”

A Rolls Royce with 3 wheels is a crappy car

Brussels, 20 March 2013Bart Vannieuwenhuyse 7

Data quality - process

• Context of data creation – meta-data– Should be made explicit– Provenance must be clear

• “medical context” - clarity on reimbursement and “medical practice”

• Clarity on who created the data

– Mapping to common ontologies

• Type of use drives selection of data– Data should be fit for intended use – Care vs Research– Options to select data sources on available meta data

Brussels, 20 March 2013Bart Vannieuwenhuyse 8

Data quality - metrics

• Quality of solution – metrics 1– Adopt existing standards e.g. ISO 25000

• SDLC like approach (engineering)• Functional suitability - Reliability• Performance efficiency - Security• Compatibility -

Maintainability• Usability - Portability

– STEEEP – Safe Timely Efficient Effective Equitable Patient-centered (IOM – US)

• Quality of usage – metrics 2• Effectiveness - Freedom of

risk• Efficiency - Context

coverage• User satisfaction

Brussels, 20 March 2013Bart Vannieuwenhuyse 9

Data quality - dimensions

• Accuracy • Quantitative vs Qualitative data (origin of data)• Benchmarking to check accuracy (TransForm, OMOP, EU-ADR)

• Completeness• Needed granularity – data available? (TransForm selection tool)• “Longitudinality” – length of available Hx

• Timeliness• Data “freshness” – latest update

• Reliability• Who created the data – who is responsible • Trustworthiness – traceability (versioning, time-stamping)

• Structured - Unstructured

Brussels, 20 March 2013Bart Vannieuwenhuyse 10

Next steps

• Data Quality Metrics community– Convene individuals from all EU projects dealing with re-use of existing data– Consolidate existing approaches across EU projects – share current solution– Classifications of data quality metrics – check availability of ISO standards for

eHealth data – if not, consider developing one? (ISO 8000 general data quality)– Consolidate available quality standards of solutions (e.g. ISO 25000)– Recommendation for projects to focus on data quality even before projects starts– Develop common approaches to evaluating data quality – “benchmarking”

analogy of computer chips // radar-graph– Have guidelines on Data quality – e.g. when creating new data / attention to

meta-data (training)– Develop and share analytical methods that deal with “imperfect data”

Data quality is a journeyAnd even the longest journey starts with the

first step