Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor...
Transcript of Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor...
![Page 1: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/1.jpg)
@Lauren_Houston
Lauren Houston¹, Dr Yasmine Probst¹, Dr Allison Humphries¹
¹School of Medicine, Faculty of Science, Medicine and Health,
University of Wollongong
Measuring data quality
through a source data
verification audit
![Page 2: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/2.jpg)
@Lauren_Houston
Health data has long been
scrutinised1,2
A large proportion of errors
are from transcribing data3,4
No “gold standard” method exists
to measure data quality error rates.
Background
1. Y.W. Lee et al. (2006) 3. M. Mealer et al. (2013)
2. M.N. Zozus et al. (2014) 4. M.L. Nahm et al. (2008)
![Page 3: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/3.jpg)
@Lauren_Houston
What is source data
verification?
Source data
Electronic
record
Case report
form
![Page 4: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/4.jpg)
@Lauren_Houston
What are the gaps in the knowledge? ICH GCP guidelines are non specific to amount, timing
and frequency of monitoring5
Cost-effectiveness of SDV6
No single definition to define data quality or universally
accepted method to measure error rates7,8
Audits may be published but not for public viewing8
5. ICH GCP (1996) 7. R.Rostami et al. (2009)
6. C. Baigent et al. (2008) 8. R.V. Gómez-Rioja et al. (2013)
![Page 5: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/5.jpg)
@Lauren_Houston
Aim
To monitor data quality
by developing and conducting
source data verification
audits to achieve
quality assurance.
![Page 6: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/6.jpg)
@Lauren_Houston
Study Background
UOW and IHMRI
Healthy-lifestyle blinded-RCT
12 months
5 clinical Accredited Practising Dietitians
To limit bias the audit was blinded
![Page 7: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/7.jpg)
@Lauren_Houston
100% SDV on the 10% random sample
Quality assurance rule developed whereby if,
>5% of data variables were incorrect a second
10% random sample was extracted
Manual verification checks conducted
Calculation of error rate
Method
![Page 8: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/8.jpg)
@Lauren_Houston
Participants
n=210
Random sample
n=21
Total of 820 data points
from 21 participants
If >41 (5%) data points
are incorrect
Second 10% random
sample of original
participants
Procedure of data audit
Participants
n=210
Random sample
n=21
Total of 685 data points
from 21 participants
If >34 (5%) data points
are incorrect
100% SDV of all data
points
10%
random
sample
100%
SDV
![Page 9: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/9.jpg)
@Lauren_Houston
Audit 1
Source data verification
Anthropometric
Physiological
Medications
Electronic
spreadsheet
record
Coded
electronic
spreadsheet
record
Source
documents
![Page 10: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/10.jpg)
@Lauren_Houston
Audit 2
Source data verification
Anthropometric
Medications
Electronic
spreadsheet
record
Coded
electronic
spreadsheet
record
Source
documents
![Page 11: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/11.jpg)
@Lauren_Houston
Coded electronic
spreadsheet
![Page 12: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/12.jpg)
@Lauren_Houston
Statistical Analysis
Total error = (code 2+3+4) / (code 1+2+3+4)
Data “not entered” (code 5) – excluded
Chi square, p<0.05
Post-hoc; adjusted standardised
residuals and z test of column
proportions
![Page 13: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/13.jpg)
@Lauren_Houston
0
0.5
1
1.5
2
2.5
3
Anthropometric Physiological Medications
%Minor error
Audit 1 Audit 2
0
2
4
6
8
10
12
14
16
Anthropometric Physiological Medication
%
Major error
Audit 1 Audit 2
Results
![Page 14: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/14.jpg)
@Lauren_Houston
0
10
20
30
40
50
60
Anthropometric Physiological Medication
%Not recorded data
Audit 1 Audit 2
0
5
10
15
20
25
30
Anthropometric Physiological Medication
%Not entered data
Audit 1 Audit 2
Results
![Page 15: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/15.jpg)
@Lauren_Houston
0
10
20
30
40
50
60
70
80
90
100
Anthropometric audit 1
Anthropometricaudit 2
Medicationaudit 1
Medicationaudit 2
Totalaudit 1
Totalaudit 2
% Correct Incorrect
Correct and Incorrect
![Page 16: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/16.jpg)
@Lauren_Houston
ResultsChi square had a significant difference;
χ2 (4, 1293) = 672.405, p = 0.000
Adjusted standardised residuals determined audit
sections were significantly different
From the z test of column proportions
anthropometric audit 1 and 2; medications audit 1
and 2 do not differ.
All other sections differed from each other.
![Page 17: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/17.jpg)
@Lauren_Houston
SummaryAudit 1 physiological section <5% error
Average total error anthropometric (9%), medications
(76%) and overall (34.5%)
Proportion of error trended upward as length of study
increased
“Not recorded” (code 4) data had the greatest
contribution to total error
![Page 18: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/18.jpg)
@Lauren_Houston
DiscussionImportance of identifying errors and determining
solutions
If >10% of a clinical dataset is erroneous the data
may be considered unreliable
Developed a 5% quality assurance rule
Data quality variations
![Page 19: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/19.jpg)
@Lauren_Houston
Source documents considered the “gold standard”
Source document-to-electronic spreadsheet
Audits cannot guarantee 100% free from error
Clinical research setting and trial design
Did not determine the impact of audit findings
Limitations
![Page 20: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/20.jpg)
@Lauren_Houston
Examine the 10% snapshot model with a 5% quality
assurance error rate
Standardise a SDV audit process
Assess the frequency and
cost-effectiveness
Overcome barriers and increase awareness
Recommendations
![Page 21: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/21.jpg)
@Lauren_Houston
How many
errors are too
many?
![Page 22: Measuring data quality through a source data verification ... · @Lauren_Houston Aim To monitor data quality by developing and conducting source data verification audits to achieve](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5df1e4e0d9f1732175a42/html5/thumbnails/22.jpg)
@Lauren_Houston
AcknowledgementsDr. Yasmine Probst
Dr. Allison Humphries
Sr/Prof. Linda Tapsell
A/Prof. Marijka Batterham
Illawarra Health and Medical
Research Institute
Smart Foods Centre
All participants and staff
involved