The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S....
-
Upload
austin-harrison -
Category
Documents
-
view
219 -
download
0
Transcript of The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S....
![Page 1: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/1.jpg)
The Unrecognized Interviewer
Studying Respondent Behavior in an
Establishment Survey of U.S. Academic
Institutions
Presented at the Third International
Conference on Establishment
Surveys (ICES-III)
June 20, 2007
Scott D. Crawford Survey Sciences
Group, LLC
Emilda B. RiversNational Science
Foundation
![Page 2: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/2.jpg)
2
Our Establishment Survey
• Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS)
• Surveys U.S. academic institutions granting graduate degrees in science, engineering (S&E), and selected health-related fields
• Jointly sponsored by NSF, the National Institutes of Health and the Department of Energy
![Page 3: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/3.jpg)
3
Brief History of the GSS
![Page 4: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/4.jpg)
4
What is collected in the GSS?
• GSS produces national estimates for S&E and selected health-related fields on: fall graduate enrollment counts
by demographic categories by main mechanisms and sources of financial support
postdoctoral (postdoc) appointment counts by demographic categories by main sources of support by first professional degrees in medical and related fields
non faculty research staff counts by demographic categories by first professional degrees in medical and related fields
![Page 5: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/5.jpg)
5
How is the GSS data collected?
![Page 6: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/6.jpg)
6
What potential errors exist in this effort?
• Errors of NonobservationSamplingCoverageNonresponse
• Observational ErrorsRespondent InstrumentMode
Very little is known!
![Page 7: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/7.jpg)
7
The GSS Respondent: Very Little is Known
• Are they knowledgeable about both postdocs and graduate students? If so, are they the best person to respond
to these requests?• What resources do the respondents rely upon
to provide a response?• Do they have direct access to the source of
these data?• How are the data organized?• How do respondents translate definitional
issues in their response?
![Page 8: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/8.jpg)
8
The Establishment Response Process
![Page 9: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/9.jpg)
9
The Establishment Response Process
Response Behavior Survey
![Page 10: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/10.jpg)
10
The Response Behavior Survey (RBS)
• Unique characteristics that make this approach feasibleHigh response rates to GSS
Demonstrates strong motivation to complete
Institutional nature of the data collection Many respondents have it as part of their job
responsibilities to respond
Survey design unchanged in years may have provided a culture where it is easy for respondents to critique
Comfort with the web in the GSS Allows for a rapid (web) follow-up
![Page 11: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/11.jpg)
11
RBS Sample
• Frame GSS department contact person list
• Things considered Consistency with GSS design Large enough to identify trends and comparison
cells in specific response characteristics (quantitative baseline study)
Small enough to minimize burden on respondents in an ongoing study
Some GSS departments do not have postdocs Given the respondent-department link, we needed
to consider the fact that some respondents had multiple departments
![Page 12: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/12.jpg)
12
RBS Questionnaire Content
• Respondent characteristics• Involvement in the GSS• Institutional characteristics• Identifying resources for the GSS
response• Data sources• Barriers to access of data sources• Assessment of data quality• Survey topic definitions – their use and
understanding
![Page 13: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/13.jpg)
13
GSS / RBS Survey Flow
![Page 14: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/14.jpg)
14
RBS Contact Strategy
• Data and mode of each contact effort for the early and late sample• Approximately 45 days of data collection required, most of it was
during the primary study (GSS) data collection period.
Contact Mode Days After Prev. Contact
1st Contact Heads up email from NSF NA
2nd Contact Invitation letter mailed 14 days
3rd Contact Invitation email sent 7 days
4th Contact 1st email reminder sent 6 days
5th Contact Letter reminder mailed 3 days
6th Contact 2nd email reminder sent 5-7 days
7th Contact 3rd email reminder sent 4 days
8th Contact Telephone calls All conducted in late Sept. and early Oct.
![Page 15: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/15.jpg)
15
RBS Response & Completion Rates
Contact Mode Cumulative Response Rate (RR2)
Completion Rates
1st Contact Heads up email from NSF
0.0% NA
2nd Contact Invitation letter 1.8% NA
3rd Contact Invitation email 24.9% NA
4th Contact 1st email reminder 38.8% NA
5th Contact Letter reminder 45.9% NA
6th Contact 2nd email reminder 54.6% NA
7th Contact 3rd email reminder 59.4% NA
8th Contact Telephone calls 72.1% NA
OVERALL 72.1% 84.5%
EARLY SAMPLE 74.7% 85.3%
LATE SAMPLE 61.6% 80.7%
![Page 16: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/16.jpg)
16
The “It’s Not Me” Phenomenon
• Right from the start the RBS uncovered something amiss in the response behavior of the departmental person “of record” 6.5% of the sample reported that they were not
involved in the 2005 GSS
• Potential causes (unknown) Sample frame problem – ineligible department Institutional responder using departmental responder
information for something other than originally intended – potentially indicates a need for a 2nd layer RBS
![Page 17: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/17.jpg)
17
Who are our GSS respondents?
• DemographicsGender, 26.2% are maleEducation
29.0% have less than a BA/BS degree 25.4% only a BA/BS
• Position (top 4 categories)27.8% administrative24.2% administrative support10.9% department chair9.5% faculty
![Page 18: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/18.jpg)
18
Who are our GSS respondents?
• Involvement in maintaining records56.6% maintain graduate school records18.5% maintain postdoc records Percent for maintaining postdoc records
only increases to 37.8% when looking at those in postdoc only departments
![Page 19: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/19.jpg)
19
Who are our GSS respondents?
• Are professional survey respondentsHave been involved in the GSS for a mean
of 6.5 years29.5% respond to other NSF surveys, 52.3%
respond to other surveys in generalHowever, only 7.1% have been involved
with IPEDS• May not be the most knowledgeable
Only 58.9% believe they are the most knowledgeable person to answer questions about postdocs
• 19.9% are users of GSS data
![Page 20: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/20.jpg)
20
What do respondent characteristics tell us?
Education• While the survey is meant to collect
data on grad students and postdocs, the responder is not likely to be of same educational status
Assumptions on definitions may be incorrect
Interpretations that are effecting counts may not be appropriate
![Page 21: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/21.jpg)
21
What do respondent characteristics tell us?
Professional respondents • These are likely to be people with job
responsibilities to complete surveys
• With 52.3% of the GSS responders also responding to other general surveys, the potential exists for a more streamlined format that may rely on these professionals’ experience and resources
![Page 22: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/22.jpg)
22
What do respondent characteristics tell us?
Most knowledgeable • With nearly half not the most
knowledgeable respondent, this raises questions about potential problems in:
The departmental respondent selection process
How the survey is structured
![Page 23: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/23.jpg)
23
What did the GSS respondents report on their own?
• When asked about reporting all of the data for specific data types
The majority reported all of the data on graduate students
However, most were not able to report all of the data on postdocs
Relied on assistance and other resources to report
They reported that they had less knowledge of the computer systems that manage such data
![Page 24: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/24.jpg)
24
RBS Measures of Data Quality
• Actual measures (break-off rates, item missing data in the GSS) were not sufficient to be used as a measure of data quality (few beak-offs and few item missing data)
• Self-reported estimate of measure quality was used – respondent provided their thoughts on the quality of the data they were providing.
Some internal validity was found as our expectations of correlations with self-assessed data quality panned out through further analyses.
![Page 25: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/25.jpg)
25
Response Behavior Effects on Data Quality
Impact on Perception of Data Quality
Positive No Impact NegativeWhen there is a person who always responds to requests for student data (Postdoc counts andfinancial support data only)
Whether the respondent is usually selected by position or on an individual basis.
Whether the respondent is one of the people always responsible for responding to student data requests or the GSS
Whether respondents have been involved in providing data for other surveys
![Page 26: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/26.jpg)
26
Response Behavior Effects on Data Quality
Impact on Perception of Data Quality
Positive No Impact NegativeData sources is easily accessible by respondents
With how the primary data source is maintained or stored
When GSS respondents do not know whether the department has an official postdoc definition
Data source is considered accurate by the respondent
The format of the primary data source (being aggregate or individual)
Respondents who consider their data source as “complex”
Respondent is familiar with how the data is entered in the primary data source
![Page 27: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/27.jpg)
27
Response Behavior Effects on Data Quality
• It is clear that the current GSS departmental contact is not the right one to provide postdoc data in most cases. Future evaluations of postdoc data collection should focus on the development of protocols for the identification of the correct person.
Postdoc offices appear to be a key starting point for this exploration
Knowledge of postdoc policies, as well as knowledge and accessibility of data sources, should be explored as a way to quickly identify the correct postdoc reporting individual
![Page 28: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/28.jpg)
28
Instrument Interaction in the Response Behavior Context
• A little more than 1/2 of GSS respondents indicated that they had read the instructions for the criteria of what constitutes a postdoc
Their perceived data quality was much higher than for those who had not read the criteria.
![Page 29: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/29.jpg)
29
Instrument Interaction in the Response Behavior Context
• Less than half of departments’ postdoc definitions are consistent with the GSS’ definition.
• However, there is no significant difference in self-assessed data quality between departments whose definition is consistent with the GSS and those whose is not.
• While departments/institutions have different definitions of postdocs, if the respondent read the GSS definition, they were able to provide good quality data (perceived). Quality problems emerged when the definition was not used.
![Page 30: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/30.jpg)
30
Instrument Interaction in the Response Behavior Context
RECOMMENDATIONS
• Definitions and their usability must be improved
• Efforts to improve postdoc data should examine why foreign counts are the easiest data
element to provide – something about that data makes it easier to report
![Page 31: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/31.jpg)
31
The Practice of Responding
• Does time of response (early, middle or late) correlate with perceived postdoc data quality?
• What specific recommendations does the respondent have about improving data quality?
![Page 32: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/32.jpg)
32
9 out 10 Reported that Current Data Collection Timing was Adequate
Those who didn’t, reported that these months would be better
0
5
10
15
20
25
30
35
40
Better Month 2.4 1.4 2.2 33.6 39.3 35.7 36.9 19.4 8.7 6.6 3.7 4.5 3.3 2.7 3.1 28.6
Jul-04
Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Jun-05
Jul-05
Aug-05
Sep-05
Don't
Kno
![Page 33: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/33.jpg)
33
How did timing of response play a role in data quality?
• Middle responders provided lower perceived quality postdoc data than early and late responders.
• There was no difference between self-assessed data quality between early responders and late responders.
![Page 34: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/34.jpg)
34
Postdoc Counts Varied by Response Timing to the GSS
• Does lower perceived data quality (middle responders) translate into meaningful or significant survey response differences?
Mean Count of Postdocs Reported per Department
3.01
4.03
4.74
00.5
11.5
22.5
33.5
44.5
5
EarlyResponder
MiddleResponder
LateResponder
![Page 35: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/35.jpg)
35
Responses:The Web Mode
• Most GSS respondents used the web survey as their primary mode of providing data
• The majority of respondents reported a preference for the web mode
![Page 36: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/36.jpg)
36
Responses:The Paper Mode
• The paper survey was used more as a worksheet to collect responses prior to submission.
Approx. one-third of the respondents rely on the paper survey to use as a worksheet for completing the Web later.
![Page 37: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/37.jpg)
37
Top Responses for Improving Response Time & Data Quality
Regarding GSS submission Providing more time
– Earlier distribution of the GSS– Extending the deadline
Improving data contacts (frequency and the right person)
Regarding their own institutions Improving their databases Improving support, reducing workload, expanding
personnel
Regarding the format of the GSS Make the format more user-friendly, simplify the form Clarify wording and definitions Designate a contact person for support
![Page 38: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/38.jpg)
38
Study Recommendations
• Consider additional efforts to support institutions and responders in their process of providing GSS and postdoc data
• Given differences in response patterns uncovered in the RBS, a more extensive nonresponse analysis of GSS data is recommended.
• Research the design of paper worksheet versions of the GSS rather than a form intended to be submitted.
![Page 39: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/39.jpg)
39
Is the RBS an effective tool?
• Potential strengthsAssisted in the identification of potential
frame issues (“it’s not me”)Effectively described the demographic
characteristics of the individual responderProvided some understanding of how the
respondent – establishment relationship can effect data quality
Identified at least one area (timing of GSS) where conflicting results show that the answer for a redesign is not so simple
![Page 40: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/40.jpg)
40
Is the RBS an effective tool?
• Potential weaknessesDid not capture much information about
other “people” who were involved in the response process
Self reported measure of quality has little hard evidence so far – should be validated
![Page 41: The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649dd25503460f94ac8f9c/html5/thumbnails/41.jpg)
41
Is the RBS an effective tool?
• Overall, the answer is yes.
• Further research is required to further develop models of the response process when an individual is responding for and within the context of an establishment.