Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a...
Transcript of Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a...
![Page 1: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/1.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Informatics:The Future of Survey Methodologyand Survey Statistics Training in
the Academy?
Survey Informatics:The Future of Survey Methodologyand Survey Statistics Training in
the Academy?
Allan L. McCutcheon
AAPOR Annual Conference
16 May 2015
11
![Page 2: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/2.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Acknowledgement and Disclaimer
• This material is based upon work supported by theNational Science Foundation under grant No. SES1132015.
• Any opinions, findings, and conclusions orrecommendations expressed in thismaterial/presentation are those of the author and do notnecessarily reflect the views of the National ScienceFoundation.
![Page 3: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/3.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
With Special Thanks to
• Mark Schulman (AbtSRBI)
• LeenKiat Soh (Co-PI)
• Adam Eck (Post-doctoral Fellow)
33
![Page 4: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/4.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Avoiding Obsolescence!Can We Adapt and Evolve?
• Our fate? (Thank you to Mark Schulman)
![Page 5: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/5.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
What is Survey Informatics?
55
ComputerScience
Mathematics& Statistics
Cognitive & SocialPsychology
SurveyInformatics
![Page 6: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/6.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
The Emergence of Web Surveysand Online Panels
• Challenges of existing (i.e., phone) technology– Caller id (Callegaro, McCutcheon & Ludwig 2009)– Falling response rates (Pew Research Center 2012)
• Resistance to online surveys has been similar to theemergence of telephone surveying– Skepticism on the part of professional practitioners– Resistance by many with vested interest in older technologies
• Still, the profession eventually adopts the newtechnology– AAPOR Report on Online Panels (POQ, Winter, 2010)
66
![Page 7: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/7.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
RISE OF TECHNOLOGY INSURVEY METHODOLOGY
77
• Survey methodologists are turning to the use oftechnology to– gather higher quality data– increase throughput and overall quantity in data collection– improve analysis efficiency and effectiveness
![Page 8: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/8.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Four Types of Data for Online Surveys
• Substantive data (survey responses)• Paradata (process data)
– Time of day survey taken– Data device (operating system, version)– Response latency– Back-ups, changes– Complexity (question, page)
• Metadata– Codebook– Data collection agency/firm– Field dates
• Auxiliary data– From external (e.g., commercial data bases)– E.g., neighborhood characteristics
88
![Page 9: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/9.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
These Lead to Complex Data Structures
99
AE4
![Page 10: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/10.jpg)
Slide 9
AE4 Do we need to redact any of this? Maybe as Dr. McCutcheon?Adam Eck, 3/19/2014
![Page 11: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/11.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
RISE OF TECHNOLOGY INSURVEY METHODOLOGY
1010
GovernmentAgencies
andBureaus
GovernmentAgencies
andBureaus
CommercialCompanies;
SurveyResearch
Companies
CommercialCompanies;
SurveyResearch
Companies
Academic andResearchInstitutions
Academic andResearchInstitutions
InformationCommunication
Web 2.0
Mobile
Big Data
Digital Sensing
Cloud ComputingCrowd Sourcing
Social Network
SURVEYMETHODOLOGY
SURVEYMETHODOLOGY
![Page 12: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/12.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
RISE OF TECHNOLOGY INSURVEY METHODOLOGY
1111
• Survey research companies have increased use ofcomputer-based (including web-based) response optionsto– improve responsiveness and flexibility
• Administration, configurability, adaptivity– better collect, organize, and store respondent data– better track sessions to generate paradata– reach more respondents (increase response rates)
AE1
![Page 13: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/13.jpg)
Slide 11
AE1 I changed this from "increase coverage" to "increase response rates" becasue it could actually be hurting coverage -- you lose all peoplewithout technology (an ever decreasing subset of the population)Adam Eck, 3/19/2014
![Page 14: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/14.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
RISE OF TECHNOLOGY INSURVEY METHODOLOGY
1212
• Survey methodology researchers are also exploring theuse of advanced Web 2.0 and information-communication technologies to– better elicit information from respondents
• smartphones for data collection (in self-administered and interview-based surveys)
• interactive online surveys– more effectively and efficiently visualize respondent data– model and directly track user behaviors
• Such as to survey daily activities by keeping track of steps taken byrespondents
AE2
![Page 15: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/15.jpg)
Slide 12
AE2 Do you have a reference for this last bullet point? Is this done by survey methodologists or CS people? And is this for monitoring(unobtrusive surveys?). That is my impression from the bullet point, but I'm not sure what you were going for. If so, I like the passivemonitoring aspect.Adam Eck, 3/19/2014
![Page 16: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/16.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
A Case for a Survey Methodology that IncludesComputer Science and Statistical Modeling
1313
• Survey methodologists are embracing technology offersopportunities to– Expand the reach of products from computer science (CS) to
improve how we perceive and act in the world around us– Develop a synergistic relationship with CS to jointly improve our
understanding of both fields and their practices• That is, as survey methodologists employ technological
advances in their operations, new problems will arisethat will also be of interest to CS theoretically andpractically to a wide range of domains
![Page 17: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/17.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
IMPROVING SURVEY METHODOLOGY THROUGHCOMPUTER SCIENCE
1414
http://www.ypte.org.uk/environmental/meat-free-mondays/209http://prblog.typepad.com/strategic_public_relation/2007/06/top-10-reasons-.html
Data AnalysisRespondentSupport &Adaptation
DataAugmentation
SystemDesign
Enhancement
AE3
![Page 18: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/18.jpg)
Slide 14
AE3 I like this slide a lotAdam Eck, 3/19/2014
![Page 19: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/19.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
IMPROVING SURVEY METHODOLOGY THROUGHCOMPUTER SCIENCE
1515
• Data Analysis: Use CS techniques to handle moreeffectively and efficiently large scale of data (includeautomation, meta-tagging, and management)– Pattern recognition, machine learning, and data mining
![Page 20: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/20.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Future of Survey Research
• Declining response rates– CATI response rates at 10%
• Caller ID• Declining landline• Increasing refusal• Cell phone limitations (manual dialing)
– Web surveys have low response rates• Difficult to get reasonable sampling frames online
• Promise of panels– Must still deal with respondent engagement– Ease of response– Mobility and timing
• “Personal assistants” (Avatars)– Test-to-speach– Natural language recognition (Siri, Cortana, Simon)– Digitized assistants (Nvidia Face Works Tech Demo)
1616
![Page 21: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/21.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
IMPROVING SURVEY METHODOLOGY THROUGHCOMPUTER SCIENCE
1717
• Example: how to store multiple levels of survey data?– Flat files lack flexibility (large redundancy)– Paradata: variable length, sequential data
• Different per respondent
• Approach: relational database (MySQL)– Easier to link relationships– More compact representation overall– Faster querying to retrieve data– Easier to maintain (add, remove, modify)– Allow for more versatile data analyses
![Page 22: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/22.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Challenges Within the Academy
• Traditional departments are relatively siloed– Often resistant to cooperation without additional resources– Interest in new technology most typical among younger faculty
• i.e., least powerful faculty
• New ‘kids on the block’ are likely to be resource deprived– Need for visionary and entrepreneurial administrators
• Likely will require private-sector partnering
• In the midst of ‘paradigmatic shift’– Will require ability to change at least part of training relatively
quickly
• Persisting need for substantial portion of current surveymethodology curricula
• Students will need to learn more than their predecessors
1818
![Page 23: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/23.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Conclusions
• Technology will play an increasing role in surveyresearch
• Statistics will play an increasing role in survey research– Analysis of data and paradata– Predictive modeling (CASD, predictive analytics)– Linkage algorithms for administrative, auxiliary, geo-data
• New forms of data and paradata, such as social media,will need to be explored
• There will be obstacles to overcome within the academy
• The challenges will make for an exciting future– All of us will have lots to learn in the years ahead
1919
![Page 24: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/24.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Thank youThank you
Allan L. [email protected]
2020
![Page 25: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/25.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Something New: Smartphones Take Over
2121
![Page 26: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/26.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Device Type Trends in Gallup Panel
2222
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
Apr
-12
May
-12
Jun-
12
Jul-1
2
Aug
-12
Sep
-12
Oct
-12
Nov
-12
Dec
-12
Jan-
13
Feb-
13M
ar-1
3
Apr
-13
May
-13
Jun-
13
Jul-1
3
Aug
-13
Sep
-13
Oct
-13
Nov
-13
Dec
-13
Jan-
14
Feb-
14M
ar-1
4
Apr
-14
May
-14
Jun-
14
Jul-1
4
Aug
-14
Phone
Tablet
![Page 27: Survey Informatics: The Future of Survey Methodology and ...€¦ · Slide 12 AE2 Do you have a reference for this last bullet point? ... – Test-to-speach – Natural language recognition](https://reader034.fdocuments.in/reader034/viewer/2022051909/5ffd24adc23f0530ca02172d/html5/thumbnails/27.jpg)
Survey Research and MethodologyUNL-GALLUP Research Center
Survey Research and MethodologyUNL-GALLUP Research Center
Passive Data Collection
2323