Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L....
-
Upload
carlos-richardson -
Category
Documents
-
view
214 -
download
1
Transcript of Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L....
![Page 1: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/1.jpg)
Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau
Rebecca L. Morrison
![Page 2: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/2.jpg)
2
Outline
Introduction & Background
Development of the Guidelines
Preliminary Guidelines
Application of Guidelines
Conclusion & Next Steps
![Page 3: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/3.jpg)
Introduction & Background
![Page 4: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/4.jpg)
4
Introduction & Background
U.S. Census Bureau:– “…leading source of quality data…nation’s
people and economy”– Tries to minimize burden of data collection
Burden = time, level of effort
Reduce cognitive burden through visual design, structure of data request
![Page 5: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/5.jpg)
5
Introduction & Background (cont’d)
Paper does not focus on specific question wording and order issues.
Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.
![Page 6: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/6.jpg)
6
Development of the Guidelines
![Page 7: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/7.jpg)
7
Development of the Guidelines: Census Bureau
Agency standards for various survey quality issues
No standard for design of questionnaires
Economic surveys moving toward consistent “look and feel”
![Page 8: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/8.jpg)
8
Development of the Guidelines: ESMS
Establishment Survey Methods Staff– Group of survey methodologists– In-house consultants to economic
programs– Noticed differences across questionnaires– Survey programs work with ESMS on
questionnaire design and pretesting– Catalyst towards consistency
![Page 9: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/9.jpg)
9
Development of the Guidelines: Methods
Questionnaire design principles from standard texts
Special attention to literature on visual design
Based on pretesting a variety of questionnaires with respondents
![Page 10: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/10.jpg)
10
Development of the Guidelines: Considerations
List of design elements to consider
Used to assess tradeoffs within specific survey conditions
Not a cookbook
![Page 11: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/11.jpg)
Preliminary Guidelines
![Page 12: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/12.jpg)
12
Guideline A: Text styles
Be aware of text styles, and how they are used for emphasis within a survey instrument.
![Page 13: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/13.jpg)
13
Guideline A: Example
2004 Annual Survey of Local Government Finances
![Page 14: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/14.jpg)
14
Guideline A: Example
![Page 15: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/15.jpg)
15
Guideline A: Text styles
These are favored by respondents:– Print item numbers in reverse-print
bubbles, e.g., , , – Print questions in bold black text.– Print instructions in plain text or italics.– Use an 8-point font or larger.
![Page 16: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/16.jpg)
16
Guideline A: Text styles
Answer spaces:– Open, non-delineated vs. delineated
– Respondents do not seem to have a strong preference
– Be consistent
![Page 17: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/17.jpg)
17
Guideline A: Text styles
Key codes / Punch codes:– De-emphasize processing codes for
respondents– Print in a darker shade of background
color– Example:
![Page 18: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/18.jpg)
18
Guideline B: Reduce visual clutter. Use natural navigational paths.
Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.
![Page 19: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/19.jpg)
19
Guideline B: Reduce visual clutter. Use natural navigational paths.
![Page 20: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/20.jpg)
20
Guideline B: Reduce visual clutter. Use natural navigational paths.Format in 1997 Economic Census
confusing:– Lines were “speedbumps” to navigation– Not clear how respondent was to navigate
two adjoined columns
Format changed for 2002– Single column of questions– Generally, a single column of response
options
![Page 21: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/21.jpg)
21
Guideline B: Reduce visual clutter. Use natural navigational paths.
Two columns of questions– Occasionally done when questions:
• Are shorter• Do not involve extensive instructions• Do not ask for numerical information
– Survey of Business Owners (SBO)• Information collected is categorical or ordinal,
not interval or ratio
![Page 22: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/22.jpg)
22
Guideline B: Reduce visual clutter. Use natural navigational paths.
Spread out the questions across more pages
Use bulleted lists
More open space = more “user-friendly”
![Page 23: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/23.jpg)
23
Guideline C: Instructions
Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.
![Page 24: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/24.jpg)
24
Guideline C: Instructions
Instructions convey specifications, intent of question
Respondents tend not to pay attention to instructions, or only look when they think they need them
![Page 25: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/25.jpg)
25
Guideline C: Instructions
2 Goals for instructions:– Eliminate, or reduce, amount of
instructions located separate from question
– Place instructions/information where it is most needed
![Page 26: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/26.jpg)
26
Guideline C: Instructions
Convert instructions into questions so respondents attend to them– When content critical to correct
interpretation of later questions– When it helps clarify/correct reported data
Example (2004 Medical Expenditure Panel Survey):
![Page 27: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/27.jpg)
27
Guideline D: Avoid sentence fragments and key words
Phrase data requests as questions or imperative statements, not as sentence fragments or key words.
![Page 28: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/28.jpg)
28
Guideline D: Avoid sentence fragments and key words
Types of data requests– Question (question word, question mark)– Imperative statement (report, enter, add)– Sentence fragment (key words, no verb)
![Page 29: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/29.jpg)
29
Guideline D: Avoid sentence fragments and key words
Example, 2002 Economic Census:– Is this establishment physically located
inside the legal boundaries of the city, town, village, etc.?
Versus:– Type of municipality where this
establishment is physically located
![Page 30: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/30.jpg)
30
Guideline D: Avoid sentence fragments and key words
Cognitive interview study: respondents prefer questions over sentence fragments
Survey methodology grad students: questions more effective, imply respondent has to do something
![Page 31: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/31.jpg)
31
Guideline E: Simple vs. complicated questions
Ask additional, simple questions, rather than fewer, more complicated ones.
![Page 32: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/32.jpg)
32
Guideline E: Simple vs. complicated questions
Length and complexity of question affects how long it takes for respondent to understand it
More complicated questions might also be more likely to be double-barreled
May be easier for respondents to answer series of shorter, simpler questions
![Page 33: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/33.jpg)
33
Guideline E: Simple vs. complicated questions
Example: 2002 Industrial Research & Development Survey
![Page 34: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/34.jpg)
34
Guideline E: Simple vs. complicated questions
2006 R&D Survey
![Page 35: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/35.jpg)
35
Guideline F: Use matrices judiciously
Use matrices judiciously, and consider the likely respondents’ background when deciding whether or not to use them.
![Page 36: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/36.jpg)
36
Guideline G: Be consistent!
Items should be numbered in the same way
Bold and italicized print should always mean the same thing
Navigation path should remain constant
Data requests should be in the same form
Variation can be confusing for respondents
![Page 37: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/37.jpg)
37
Application of Guidelines
![Page 38: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/38.jpg)
38
Guideline A: Text Styles
Work in progress among estab surveys at Census Bureau
Economic Census automation required consistent design features– Plain text: questions– Italics: instructions, definitions,
include/exclude lists– Bold: Separating items into categories, sums
of added lines, emphasis within questions
![Page 39: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/39.jpg)
39
Guideline A: Text Styles
Survey of Business Owners:
– Questions: plain– Instructions: bold and/or italics– Emphasis withinquestions: bold and/or italics
![Page 40: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/40.jpg)
40
Guideline B: Reduce visual clutter. Use natural navigational paths.
Survey of Residential Alterations and Repairs (SORAR)– Two columns to one column– Removed or lightened lines– Use of bulleted lists– More open space
![Page 41: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/41.jpg)
41
Guideline B: Reduce visual clutter. Use natural navigational paths.
Old
SORAR
![Page 42: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/42.jpg)
42
Guideline B: Reduce visual clutter. Use natural navigational paths.New SORAR
![Page 43: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/43.jpg)
43
Guideline B: Reduce visual clutter. Use natural navigational paths.Two columns:
Survey of
Business
Owners
![Page 44: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/44.jpg)
44
Guideline B: Reduce visual clutter. Use natural navigational paths.BEA quarterly foreign direct investment (FDI)
Complex navigational path
Crowded text
![Page 45: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/45.jpg)
45
BEA form
revised
Guideline B: Reduce visual clutter. Use natural navigational paths.
![Page 46: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/46.jpg)
46
Guideline C: Instructions
Economic census continues to use separate instruction sheets and booklets
Two questionnaires moved many instructions to questionnaire:– 2007 Commodity Flow Survey– BEA quarterly FDI form
![Page 47: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/47.jpg)
47
Guideline C: Instructions
![Page 48: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/48.jpg)
48
Guideline D: Avoid sentence fragments and key words
Mixed success in application across surveys
![Page 49: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/49.jpg)
49
Guideline D: Avoid sentence fragments and key words
Commodity Flow Survey uses mix of questions and imperative statements
![Page 50: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/50.jpg)
50
Guideline E: Simple vs. complicated questions
Example:
2002 Survey
of Business
Owners
![Page 51: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/51.jpg)
51
Guideline E: Simple vs. complicated questions
Example: 2007
Survey of Business
Owners
(still in cognitive
pretesting)
![Page 52: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/52.jpg)
52
Guideline F: Use matrices judiciously
Example:
BEA
quarterly
FDI
![Page 53: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/53.jpg)
53
Guideline F: Use matrices judiciously
![Page 54: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/54.jpg)
54
Conclusion & Next Steps
![Page 55: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/55.jpg)
55
Conclusion & Next Steps
Guidelines are preliminary, based on principles of questionnaire design
Further refinement is necessary before they can be applied to establishment surveys at Census Bureau
![Page 56: Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.](https://reader035.fdocuments.in/reader035/viewer/2022062618/5514bcfc550346b0338b46bb/html5/thumbnails/56.jpg)
56
Conclusion & Next Steps
Other agencies/organizations may develop guidelines that work better for their environment
There is no cookbook for questionnaire design