Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of...
-
Upload
phillip-kennedy -
Category
Documents
-
view
214 -
download
0
Transcript of Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of...
![Page 1: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/1.jpg)
Q-BANK: Development of a Tested- Question Database
Questionnaire Design Research Lab
Office of Research and MethodologyNational Center for Health Statistics
![Page 2: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/2.jpg)
Cognitive Interviewing
at NCHS
A Brief Overview
![Page 3: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/3.jpg)
Questionnaire design problems
Some possible threats to accuracy derived from questionnaires: Questions not understood as intended Don’t adequately capture respondent
experience Pose an overly challenging response task Don’t provide a legitimate option for an answer
![Page 4: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/4.jpg)
Questionnaire design problems (2)
Errors can be systematic (bias) or random (noise; high variance)
Problems may not be visible in the actual data, inviting delusions about the quality of self-reports that have been collected
![Page 5: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/5.jpg)
“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward
![Page 6: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/6.jpg)
“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward But suppose we ask:
What do people think of as their “abdomen”?
![Page 7: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/7.jpg)
What, to you, is your abdomen?
![Page 8: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/8.jpg)
“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward But suppose we ask:
What do people think of as their “abdomen”? What sort of threshold do people use for
answering “yes”?
![Page 9: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/9.jpg)
“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward But suppose we ask:
What do people think of as their “abdomen”? What sort of threshold do people use for
answering “yes”? What period of time are people thinking about
while answering?
![Page 10: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/10.jpg)
“In the last year, have you been bothered by pain in the abdomen?” Possible revisions:
Show shaded picture of abdomen Drop “bothered” Use “In the past 12 months”
“In the past 12 months, have you had pain in the abdomen? By abdomen, we mean the shaded area in this picture.”
Clear alternatives address these problems, with no apparent drawbacks
![Page 11: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/11.jpg)
Shaded picture of the abdomen
![Page 12: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/12.jpg)
What is cognitive interviewing? 1) An administration of a draft
questionnaire…2) With additional verbal information
collected about the responses…3) And used to either:
Evaluate the quality of responses Or help to determine whether the question is
generating the sort of information its author intends.
Within these general parameters, there are a number of variants:
![Page 13: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/13.jpg)
Practices of cognitive interviewing The “purist” version– encourage
participants to think out loud while answering questions; interviewer “prompts” participants without much other activity.
![Page 14: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/14.jpg)
Practices of cognitive interviewing (2) Newer versions rely much more heavily on
direct probing. But even so, there can be much variation: Probes scripted or based on
interview/participant particulars (more “active” role for the interviewer)
Probes that are “cognitive” or more narrative in nature
A relatively realistic administration of the questionnaire (with debriefing at the end) or intensive discussion throughout
![Page 15: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/15.jpg)
Some general practices We generally opt for:
Designing a semi-scripted interviewing protocol. It asks some probes in anticipation of potential problems, but allows improvisation.
Selecting a group of interviewees who would be typical recipients of the questionnaire and collectively cover a range of situations. Get a demographic/educational mix as much as possible.
Considering the interviewers to be investigators and not simply data collectors (methodologists with subject matter knowledge).
![Page 16: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/16.jpg)
What sorts of probes should you ask?Consider this simple model of the response
process as a starting point:
Comprehension: Respondent interprets the question
Retrieval: Respondent searches memory for relevant information
Estimation/Judgment: Respondent evaluates and/or estimates response
Response: Respondent provides information in the format requested
![Page 17: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/17.jpg)
Probes in cognitive interviews
Traditionally the majority of attention focuses on comprehension
Direct comprehension of terms: What does 'dental sealant' mean to you?"
But we also find that the most efficient way to learn about frame of reference and interpretation is to ask for narrative about the basis for the respondent’s answer Why did you answer that way? You said ___; tell me more about that.
Or, ask for paraphrase of the question.
![Page 18: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/18.jpg)
Probes in cognitive interviews (2)To get at other aspects of the response process: Recall: How did you figure your answer to that? How
do you remember that? Confidence: How certain are you about that? (can be
used to identify judgment issues/uncertainty) Response categories: How did you pick an answer to
that? Was it difficult to find a response category? You can’t go wrong with:
“Tell me more about that?” “How so?”
![Page 19: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/19.jpg)
The end result Qualitative analysis follows one or more rounds of
interviews Project reports include rich data on
Participant experiences and how these line up with question wording
Participant interpretations of the questions Problems participants had answering Potential ambiguities, trade-offs involved in asking
question a certain way Reports provided to sponsor, including
recommendations for changes when appropriate Then what?
![Page 20: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/20.jpg)
Q-BANK Origination 2001 QUEST conference
Losing knowledge Repeat testing
![Page 21: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/21.jpg)
Lack of “place” Implications:
Personal Administrative Survey Research
![Page 22: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/22.jpg)
Impact on Question Evaluation Inability to compare studies
Hinders methodological conclusions Prevents insight Stagnant method
Limited use to survey research
![Page 23: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/23.jpg)
Database Structure National Center for Health Statistics
original developers1. Access to past evaluation projects2. Methodological research in question design
![Page 24: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/24.jpg)
Q-BANK houses: Individual questions Test reports
Test findings Methods used
Question characteristics
![Page 25: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/25.jpg)
Question Subject Matter Numerous, open categories Accessed by drop-down list Examples:
Demographic: Income, Race, Gender, NationalityHealth: Insurance, Chronic conditions, SmokingEmployment: Status, Occupation, Job change,
Salary, Benefits
![Page 26: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/26.jpg)
Question Information Type1. Events, Actions and Behaviors2. Objective Characteristics3. Subjective Characteristics4. Speculations5. Attitudes/Opinions6. Knowledge Tests7. Other
![Page 27: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/27.jpg)
Response Category Type
1. Yes/no2. Select one textual3. Select one numeric4. Mark all that apply5. Allocation6. Open-ended textual: Delineated & Non-
delineated7. Open-ended textual: Delineated & Non-
delineated8. Other
![Page 28: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/28.jpg)
Response Error Indicator
1. Interviewer difficulty2. Visual design problem3. Missed instructions4. Unknown terms5. Ambiguous concepts 6. Overly complex7. Assumption8. Double barrel9. Questionnaire effects10. Recall & estimation error11. Inadequate response options12. Problematic answer space13. None as tested
![Page 29: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/29.jpg)
Value of Q-BANKSurvey Research Tool for:
Questionnaire Designers Subject Analysts Survey Methodologists
![Page 30: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/30.jpg)
Interagency Effort Co-Sponsors:
Office of Behavioral and Social Science Research,
Bureau of the Census, National Science Foundation, National Cancer Institute, Bureau of Labor Statistics
NCHS provides internet access
![Page 31: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/31.jpg)
Current Status On-line to steering committee members Testing and refinement Broader access possibly by end of year Looking for sponsors and partners
![Page 32: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/32.jpg)
ChallengesPast Integrating interagency needs
Additional fields for establishment surveys Additional fields for self-administered surveys Reworking existing categories: response error
typologies, question type typologies, question topic categories
Establishing procedures Methodological requirements of evaluation
reports
![Page 33: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/33.jpg)
ChallengesCurrent Coding consistency Populating the database Staffing limitations Funding limitations
![Page 34: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/34.jpg)
ChallengesFuture Maintaining a methodological research
agenda Continual eye toward development
![Page 35: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/35.jpg)
Searching in Q-BANK- Agency Name- Survey Title- Response Error- Advanced Searches
![Page 36: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/36.jpg)
Q-BANK Search Fields
![Page 37: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/37.jpg)
Agency Name Search
![Page 38: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/38.jpg)
Agency Name Search
![Page 39: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/39.jpg)
Agency Name Search
![Page 40: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/40.jpg)
Q-BANK Reports
![Page 41: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/41.jpg)
Searching on Survey Title
![Page 42: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/42.jpg)
Searching on Response Error
![Page 43: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/43.jpg)
Advanced Searches
![Page 44: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/44.jpg)
Advanced Searches
![Page 45: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/45.jpg)
Cognitive Testing Final Reports Don’t get lost in the weeds Important information about a tested
question is lost without referencing the final report
Conclusions about specific questions can be distorted without the final report
![Page 46: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/46.jpg)
Contents of a Typical Report Introduction Method Procedures Summary of findings Question-by-question analysis
![Page 47: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/47.jpg)
Why refer to the final report?
Search categories may not be straightforward Example: “At this time, is (CHILD) covered by
health insurance that is provided through an employer or union or obtained directly from an insurance company?” Type of response error = double barreled question
![Page 48: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/48.jpg)
Why refer to the final report?
Search categories may not be straightforward Example: “At this time, is (CHILD) covered by
health insurance that is provided through an employer or union or obtained directly from an insurance company?” Type of response error = double barreled question
The interpretive process and underlying construct can’t be gleaned from the search category Example: “Have you ever taken any herbal
supplements for yourself?” Type of response error = ambiguous concept
![Page 49: Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6a5503460f94b66f5d/html5/thumbnails/49.jpg)
Final Reports Q-Bank is a good tool for quickly finding
relevant reports of different cognitive testing projects
Searches not meant to replace final reports