Post on 28-Jan-2016
description
Validation in Statistics Canada Health Surveys
Presentation to RRFSS Workshop
June 20, 2007Vincent Dale
Outline
Statistics Canada quality assurance framework
Ensuring data accuracy Past validation projects Future projects Future directions
Quality Assurance Framework
Relevance Degree to which information meets the real needs of data users
Timeliness Delay between the reference period and the date upon which the information becomes available
Accessibility Ease with which information can be obtained from the Agency
Accuracy Degree to which the information correctly describes the phenomena it was designed to measure
Interpretability Availability of supplementary information and metadata necessary to interpret and utilize information appropriately
Coherence Degree to which information can be successfully brought together with other statistical information within a broad analytic framework and over time
Statistics CanadaQuality Assurance Framework
Trade-offs between aspects of quality These are actively managed through a
variety of processes, including:– User and stakeholder feedback mechanisms– Program review– Data analysis and dissemination– Standards and documentation (concepts,
variables, classifications)
Ensuring Data Accuracy
Questionnaire development– Wherever possible, validated questionnaire modules are
used Sometimes modified for use in population-based survey Sometimes not as valid as advertised
Questionnaire testing– STC policy requires testing of all new questionnaires
Cognitive interviews and focus groups
Coherence versus accuracy– Sometimes better to keep measure stable even if imperfect
Ensuring Data Accuracy
Sampling error – error attributed to studying a fraction of a population rather
than carrying out a census
Non-sampling error – coverage errors– response errors– non-response errors– processing errors– estimation errors– analysis errors
Ensuring Data Accuracy
Explosion of health survey data– More data, more often for smaller levels of
geography
Increasing attention paid to validity– Health measures– Administrative data– Complimentary surveys
What is validity?
Face validity Internal validity
– construct validity
External validity– Criterion– Sensitivity, specificity, predictive value
Past CCHS Validation Projects
Health Care Utilisation
– Data linkage of CCHS responses with BC administrative health records
– Supplemented with analysis of: Respondent interpretation and formulation of responses Interviewer behaviour and training Patterns in response changes, edits and timing of
response entry
Contacts with Health Professionals
Results of linkage:– Compared to provincial health records:
Most CCHS respondents (58%) reported fewer primary care physician contacts
On average, CCHS respondents reported 1.7 fewer primary care physician contacts
Older CCHS respondents and respondents with better self-perceived health tended to report fewer contacts
Younger respondents and respondents with poorer self-perceived health tended to report more contacts
Contacts with Health Professionals
Recommendations from study:– Revise wording of specific questions to minimize
misinterpretation– Facilitate consistent interviewer probing
techniques– Improved edits and CAPI/CATI application
navigation for interviewers to facilitate changes to previously-answered questions
Evaluation of coverage of linked CCHS and hospital inpatient records
Probabilistic linkage used to identify CCHS 1.1 respondents (excluding Québec) hospitalized over a 14-month period– Health person-oriented information database
(HPOI) is a virtual census of hospital admissions and used as the standard
Survey weights applied to the 8230 CCHS records which were found in the HPOI database
Evaluation of coverage of linked CCHS and hospital inpatient records
Number hospitalized in acute-care hospitals, Sept. 1, 2000 – Nov. 3, 2001, aged 12+, Canada excluding Québec
CCHS HPOI
Unweighted
(n)
Weighted
(N)
Count
(N)
Coverage rate
TOTAL 8,230 1,334,909 1,612,269 82.8%
Evaluation of coverage of linked CCHS and hospital inpatient records
Under-reporting rates similar between women and men
– Lower among Manitoba residents (69.2%)– Higher among individuals aged 12-74 (86.1%) than those
aged 75+ (70.3%)
Under-reporting is an essential prerequisite to further analyses based on the CCHS – HPOI linked data
– Use of the linked file could lead to bias due depending on province/territory of residence and age
CCHS Measured Height & Weight
In 2005, height / weight were measured for a sub sample of CCHS Cycle 3.1 participants (n=4567)– Weight: mean difference between measured and
self-reported weight of 2.1 kg (2.5 kg for women)– Height: mean difference between measured and
self-reported height of -0.7 cm (-1.0 cm for men)– BMI: mean difference between measured and
self-reported BMI was 1.1
CCHS Measured Height & Weight
Overweight Obese Class II, III
(25.0 to 29.9)
Total % Total % Total % Total % Total %
('000) ('000) ('000) ('000) ('000)
Self-reported BMI category
Both sexes
Underweight (<18.5) 271 67 308 3 1 0 1 0 0 0Normal weight (18.5 to 24.9) 131 33 10,163 94 2,651 30 120 3 4 0Overweight (25.0 to 29.9) 0 0 388 4 5,851 67 1,894 44 134 9Obese Class I (30.0 to 34.9) 0 0 0 0 244 3 2,247 52 603 39Obese Class II, III (=35) 0 0 0 0 0 0 22 1 822 53
Total 402 100 10,859 100 8,746 100 4,288 100 1,562 100
(<18.5) (18.5 to 24.9) (30.0 to 34.9) (=35)
Measured BMI category
Underweight Normal weight Obese Class I
CCHS Mode Effect Study
Potential differences associated with two methods of collection used in CCHS
– CAPI: computer assisted personal interview– CATI: computer assisted telephone interview
Used a split-panel design with a unique sample frame
– secondary sampling units randomly assigned to CAPI or CATI.
– Fully integrated as part of CCHS cycle 2.1– 11 sites selected to provide a good representation of each
region in Canada
CCHS Mode Effect Study
Important differences observed for obesity rates – CAPI = 17.9%; CATI = 13.2%
Physical activity index – inactive persons– CAPI = 42.3%; CATI = 34.4%
Statistically significant differences for contact with medical doctors and unmet health care needs
No significant differences observed in the vast majority of health indicators
CCHS Mode Effect Study
Overall results show that cycles 1.1 and 2.1 are largely comparable despite an increase in CATI collection for Cycle 2.1 (2003)
Results led to a decision to measure exact height and weight for a sub-sample of respondents in cycle 3.1 (2005)
Led to improved standardization of interviewer procedures across the two collection modes
Future Validation Projects
Scale Reliability - Factor Analysis
Construct validity / scale reliability:– Cronbach’s Alpha calculated for scales used in
CCHS questionnaire– Results could be published in user guide
What are standards? Some researchers feel that scores should be above 0.8
CCHS Depression Module
Currently, CIDI Short form for Major Depression (CIDI-SF) is used in CCHS
– Also used in NPHS and several regional and provincial surveys
Some problems with its use in CCHS– Has not been validated against International Classification of
Disease (ICD)– Evaluates 12-month prevalence, not necessarily current
treatment need– Does not evaluate some items related to clinical significance
Patient Health Questionnaire (PHQ) identified as potential CIDI-SF replacement
CCHS Depression Module
Primary goals of potential validation study:– Determine the validity of the CIDI-SF and PHQ in
relation to a gold standard diagnostic interview (SCAN – Schedules for Clinical Assessment in Neuropsychiatry)
– Identify optimal scoring procedures for the PHQ in Canadian population-based studies
CCHS Depression Module
Samples of n=200 subjects to be drawn in two sites (English and French)
– Supplemented with n=100 subjects selected from psychiatric outpatient settings to increase the number of positive cases of major depression
Each participant to be administered:– 1) Standard demographic module– 2) PHQ-9– 3) Module to distinguish between clinical depression and
bereavement– 4) SIDI-SF– 5) Set of modules to assess consequences of construct in
terms of quality of life
CCHS Depression Module
Sensitivity and specificity of the CIDI-SF and PHQ to be measured using the SCAN as a gold standard
Ordinal CIDI-SF ratings to be correlated with PHQ ordinal ratings using Spearman correlation coefficient
Test of construct validity of PHQ to be performed using exploratory factor analysis
Internal consistency of scales and subscales to be assessed using Cronbach’s alpha
Test-retest reliability of PHQ and CIDI-SF and inter-rater reliability of the SCAN will be evaluated for 50 respondents
CCHS Depression Module
The estimated cost for the project exceeded $200,000
Due to our inability to secure external funding and the lack of available budget and personnel internally, there are no concrete plans to proceed with study
Directions Forward
Focus on accuracy, interpretability and coherence Trade-offs between aspects of data quality
Improved timeliness, accessibility and relevance
How good is “good enough”? Partnerships
– Are there areas where CCHS, RRFSS and others can collaborate ?
Contact Information
Vincent DaleSurvey Manager, Canadian Community Health Survey
613-951-4265
Sylvain TremblayContent Manager, Canadian Community Health Survey
613-951-2528