1

Cognitive Aspects Associated with Sample Selection Conducted by Respondents in Establishment Surveys

La Toya Barnett Thomas

Rebecca L. Morrison

Grace E. O’Neill

U. S. Census Bureau

2

Outline

Background MethodologyCognitive requirements of selecting a

sampleResultsConclusionFuture Research

3

Background: CFS Information

CFS = Commodity Flow Survey.Mandatory and collected every 5 years. Last survey covered 2002, next survey to cover

2007.Collects information on shipment of goods across

the country for a 1 week reference period.

Unique Feature: respondents required to select their own sample to complete form.

4

Background: The Issue

Asking non-statisticians to perform a major statistical task – systematic sampling.

Not a normal survey task in general.Respondents drawing their own sample was an

issue in 2002 survey.Sponsors suggested cognitive testing in

preparation for 2007.Sample selection covered in 3 sections (Items

D, E, and F) of the survey.

5

Background: The Issue (cont’d)

Item D –total number of outbound shipments during reference period.

Item E – guides R’s through process of selecting their sample, based on their response in Item D.

Item F –information about shipments from their sample (dates, type of shipment, destination, how transported).

6

Methodology

Three rounds of cognitive interviews (October 2005 – August 2006)

Nationwide

64 interviews total

Companies were a mix of industries, sizes, products, and shipping arrangements.

7

Methodology (cont’d)

Tourangeau’s (1984) Four Step Cognitive Response Model– Comprehension– Retrieval– Judgment– Communication

8

Cognitive Requirements of Selecting a Sample: Comprehension

Understanding navigation and meaning of words and/or diagrams.

Getting lost in navigation causes difficulty in completing task.

Basically, if Rs lack comprehension, the task of selecting their own sample will be difficult.

9

Cognitive Requirements of Selecting a Sample: Retrieval

Accessing several types of records:– Fleet of vehicles and/or trucks– Parcel delivery services– Customer pick-ups

Records could be on paper, in electronic form, or both.

Records in multiple places – need to know that they exist and how to access them.

10

Cognitive Requirements of Selecting a Sample: Judgment

Which records Rs will select to report in survey.

How to apply instructions to their records.– How many records will be in my sample?– How many do I need to report for?– What if I cannot access all necessary

records?

11

Results: Round 1

The 2002 form was used in this round of testing to identify shortcomings.

12

2002 CFS Questionnaire, Item E

13

2002 CFS Questionnaire, Item E

  

14

2002 CFS Questionnaire, Item E

 

  

15

Sample Selection vs. “Audit”

Some Rs thought we were asking for an audit of shipping records and suggested calling Item E an “Audit of Shipping Records”

“Audit” can have a negative connotation

Item E could be similar to an audit, but is in fact systematic sampling.

16

Results: Round 2

Purpose of this round was to resolve issues from Round 1

Item E revised from Round 1:– Placed on one page– Eliminated diagrams and split table into

two pieces– Instructions turned into questions

17

Item E, Round 2

18

Results: Round 2 (cont’d)

Having Item E on one page was helpful.

The revised Item E did not adequately connect Items D and F. – Asks Rs to re-report their response to Item D– Later asked how many shipments they would

report in Item F– Were able to complete Item E, but did not know

why

19

Results: Round 2 (cont’d)

Language continued to be an issue– “Selection rate” and “selection method”– Difficult for people with non-statistical

backgrounds to comprehend

Rs who reported <40 outbound shipments found Item E difficult to complete.– No sample needs to be drawn, but it was difficult

to convey that info

20

Results: Round 3

Purpose was to ensure changes made were working to assist Rs in sample selection process.

Several changes made from Round 2: – Added skip instruction to Item D, so Rs with <40

shipments would go straight to Item F (and skip sampling process)

– Added purpose statement, so Rs knew why they were drawing a sample

– Refined wording so it was less statistical

21

Results: Round 3 (cont’d)

More changes from Round 2:– Process and instructions were numbered

to assist navigation– An example was added, but not

emphasized (indented and placed at bottom of the page), so Rs would not focus on it exclusively

22

Final Version of Item E for 2007 CFS Questionnaire

23

Final Version of Item E

Instructions contained less text and were numbered for easier navigation.

Only one table, where Rs could mark the row indicating which records they would select (“every nth outbound shipment”)

Sample selection process explained in 7 short steps.

Example was de-emphasized in order to act as verification that Rs did everything correctly.

24

Conclusion

Very difficult for non-statisticians to do a statistical task, as in Item E. Language is key.

Conveying relationship between Items D, E, and F was challenging.

Most significant and helpful change: putting all of Item E on one page and removing the diagrams.

25

Conclusion (cont’d)

Diagrams did not work because Rs did not read accompanying text.

However, there are cases where diagrams can be helpful in surveys, especially when accompanying text can be read and easily understood.

26

Conclusion (cont’d)

CFS is currently in the field and is performing very well, overall.

Incoming calls:– Clarify sampling process– Express curiosity on why sampling is necessary

This supports the fact that it is very difficult for non-statisticians to perform a statistical task.

27

Future Research

Explore other surveys that require Rs to select their own samples.

Perhaps topic of other surveys could explain the difficulty or ease of Rs completing task without much confusion

Answer could lie in the amount of statistical knowledge Rs have, the field they work in, and how the survey explains the sample selection process.

28

Thank you!

For questions, please contact:

La Toya Barnett Thomas

(301) 763-9588

La.Toya.Barnett@census.gov