The Annual Structural Business Survey Developing and Testing an Electronic Form
description
Transcript of The Annual Structural Business Survey Developing and Testing an Electronic Form
![Page 1: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/1.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 1
The Annual Structural Business SurveyDeveloping and Testing
an Electronic Form
Ger SnijkersEvrim Onat
Rachel Visschers
Statistics NetherlandsDivision of Business Statistics
![Page 2: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/2.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 2
1. Dutch Annual StructuralBusiness Survey
Annual survey of economic activity Mandatory 75.000 business each year:
• Sample of small firms, bigger firms each year Until now: paper questionnaire
• length may differ, 20 pages is typical Three parts:
• revenues and costs• summary of business accounts:
profits and losses• industry specific specifications
![Page 3: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/3.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 3
2. The paper questionnaireCharacteristics
A4 booklet• Right page: items• Left page: help texts - long and voluminous
Items are grouped in sections• Long sections
• Completion process: complicated and hard• Large amount of detailed information• Broad range of business information:
several departments, several respondents• No match with definitions used, but same label:
CBS – business definitions• Lay-out: misinterpratations and errors
![Page 4: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/4.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 4
Vernieuwde huisstijl:2. Onderzoek naar de PS
Q oud
![Page 5: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/5.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 5
3. The web questionnaireProject goal
Develop a web questionnaire• same contents• mixed-mode design: paper and web• support completion process:
- motivate respondents to use this mode• into the field: March 2006
Start of project: June 2004
In the field: Spring 2006
![Page 6: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/6.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 6
3. The web questionnaire Developing and testing
In five stages:
1. Testing the prototype (31-1-’05)• pre-tests to test usability: 3 waves
2. Revision of questionnaire (1-9-’05)• expert reviews
3. Testing of revised questionnaire (1-1-’06)• additional usability tests
4. Implementation of field pilot (1-3-’06)
5. Implementation of survey (1-3-’07)
![Page 7: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/7.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 7
3.1. The web Q: The prototype
![Page 8: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/8.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 8
3.1. The prototype: 3 test waves
Test 1 Test 2 Test 3
Period Aug. 2004 Oct. 2004 Nov./Dec. 2004
Form Blaise IS Blaise -EDR
Blaise -CBSquest
On/off-line On-line Off-line Off-line
Mode Browser CD-rom Downloadablevia internet
Number of interviews
15 CBS employees
37 businesses
6 businesses
Tested by CBS employees,external designer
6 business interviewers
2 cogn. lab interviewer + 2 business interviewer
Results Experiences from CBS employees
Experiences from interviewers
Experiences with respondents
![Page 9: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/9.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 9
3.1. The prototypeResearch issues
1. How does the e-form work in practice?• Completing the questionnaire• Question-and-answer process
2. What features should be included to make it easy to use?• Respondent friendly: ‘Computer-assisted’ tools• User wishes
3. How should the web Q be designed in relation to the paper Q?• The same or a different design• ‘look-and-feel’ of paper and e-form
![Page 10: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/10.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 10
3.1. The prototypeResearch issues
1. How does the Q work?
Laborious and complex process• Long, complex questionnaire (≥ 25 items)• Complex completion process:
- several sessions, several informants- kick-and-rush behaviour
• Imagine ... a respondent sitting behind his/her computer ...
Respondent got lost in the questionnaire
![Page 11: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/11.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 11
2. Features to make it easy to use?• What am I supposed to do (next)?
• Easy to download, install, complete, send data back• It is one process: downloading – sending data back, • Clear instructions and explanations (but not read)
• How is the questionnaire built up? • Show how the questionnaire is structured: overview• Help to find the way in the questionnaire• No hidden rules, no unexpected functionalities
• Where am I? What did I do so far?• Provide overview of the completion process• Clear navigation, no scrolling• Printing function
3.1. The prototypeResearch issues
![Page 12: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/12.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 12
2. Features to make it easy to use? Con’d• Printing
• Passing on sections of Q to other departments• Checking the data before transmitting• Getting authorisation to release the data
• Calculations• Well accepted, … even expected
• Entry-search• ‘Google’-like search: “Where to put these data?”
• Navigation and overview• Choose a setup people are familiar with:
- setup of tax office, windows explorer
3.1. The prototypeResearch issues
![Page 13: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/13.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 13
3. Design of paper and web Q?• The computer is different than paper
• The web Q reacts to the respondent• Reading from the screen is different• Navigating and getting an overview
works differently• Kick-and-rush behaviour,
even stronger than on paper• The use of computer-assisted functionalities;
the respondent expects the computer to react
3.1. The prototypeResearch issues
![Page 14: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/14.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 14
3.1. The prototypeConclusions of pre-test waves
• Visual design Clear and logical Simple, transparent, consistent No hidden and unexpected functionalities
• Support the completion process Other mode, other features, other visual design
Different than paper form, same ‘look-and-feel’
• Tailor to kick-and-rush behaviour Small sections, small tasks Short and clear explanations
![Page 15: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/15.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 15
3.2. The revised questionnaire
Based on:• Pre-test results• Expert reviews
• Iterative process with- Professional designer- Questionnaire designers- Methodologists
• A user friendly design was put first, not the IT tool
• New prototypes designed in Power Point
![Page 16: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/16.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 16
![Page 17: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/17.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 17
3.3. The revised questionnaireConclusions of add. pre-tests
• 10 concurrent in-depth interviews• Usability and user friendliness has been
improved respondents enjoyed working with the
questionnaire they could handle the task even though … the task had not changed
Web questionnaire design is communication design
![Page 18: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/18.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 18
3.4. Field pilot
Set-up:
March-July 2006 7200 businesses, 5 industries Advance letter with
• internet address: www.cbs.nl/productiestatistiek• user name and password• leaflet to introduce web questionnaire
and explain why the survey is conducted• paper questionnaire not mentioned
![Page 19: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/19.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 19
Leaflet
![Page 20: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/20.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 20
3.4. Field pilot
Goals: Implementation of web questionnaire
• paper and web flows
Test the whole process• downloading – completing – sending-in data• investigate completion process in the field:
- debriefing interviews with respondents- audit trails
Response rates Data quality
• data editing, mode effects
![Page 21: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/21.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 21
3.5. Survey
Now running About 75.000 business received this
questionnaire
![Page 22: The Annual Structural Business Survey Developing and Testing an Electronic Form](https://reader033.fdocuments.in/reader033/viewer/2022051316/56814796550346895db4ca62/html5/thumbnails/22.jpg)
Tuesday, June 19, 2007 ICES3, June 18-21, 2007, Montreal 22
Web questionnaire design is communication design