SURVEY RESEARCH

KRISTOPHER, KIM, SHELLEY AND JILLIAN

Getting Started…

“A telephone survey says that 51 percent of college students drink until they pass out at least once a month. The other 49

percent didn't answer the phone. “-Craig Kilborn

The Following Topics will be Discussed:

Theoretical Tradition and Epistemological Paradigm

Most Appropriate Types of Research Questions Major Features of the Method Major Criteria for Rigour Strengths and Weaknesses Designing a Project using the Method

Theoretical and Epistemological Background

In general, a survey involves the collection of information from a large group of people or a population, commonly via:Opinion SurveysPolitical PollsTV Viewing PollsHowever…

Theoretical and Epistemological Background

…our focus is on survey research, which is conducted to advance scientific knowledge or develop theory.

Survey research is frequently utilized within the Social Sciences, including disciplines like Psychology, Marketing, and Organizational Behaviour.

Positivism

Positivism has become a dominant institutional form in social research.

However, during the 1970's and 1980's prominent concerns were raised about the limits of quantitative data and methods often associated with positivism; including survey research designs.Eg: Concerns explained.

Positively More Positivism

Assumption: Objective world which science can “mirror” with privileged knowledge.

Key Focus: Search for contextual and organizational variables which cause organizational actions.

Key Theories in Paradigm: Contingency theory, systems theory, population ecology, transaction cost economics of organizing, dustbowl empiricism, etc.

Goal of Paradigm: Uncover truth and facts as quantitatively specified relations among variables.

Positively More Positivism II

Criteria for Assessing Research: Prediction=Explanation, Rigour; internal/external

validity, reliability. Unit of Analysis: The variable. Research Methods and Types of Analysis:

Experiments, questionnaires, secondary data analysis, quantitatively coded documents.

Quantitative: Regression, Likert scaling, structural equation modelling.

Qualitative: Grounded theory testing.

Major Features of the Method

Variables are often operationalized when researchers ask people questions as a way of getting data for analysis and interpretation (the questions are either asked by an interviewer, or written down and given to respondents for completion)

2 Forms of surveys: Questionnaire: document containing questions and others

types of items designed to solicit information appropriate for analysis

In dept Interviewing: ask people questions in order to gather data

Survey research is especially appropriate for making descriptive studies of large populations

Data collected may be used for explanatory purposes

The Beauty of Questionnaires

It provides a method of collecting Data by either asking people questions or asking them to agree or disagree with statements representing different points of view

Used primarily in survey research, but also in experiments, field research and other modes of observations

Questions can be open ended (respondents supply their own answer-similar to a short answer question on an exam), answers can be in the form of writing on paper or verbally reporting answers to an interviewer

Example: what is your opinion on abortion? It can also be closed ended (select from a list of answers provided)

This type of questionnaire is more common than open-ended ones because they provide greater uniformity of responses, and are more easily processed

Example: From a range of 1-5, how would you rate this hotel? With 1 being poor, and 5 as excellent

A Closer Look at Closed Ended Questions

Two structural requirements for closed ended questions: 1. Response categories provided should be exhaustive (should

include all possible responses that might be expected-even if answers may be obscure to you and me). Usually researchers try to ensure this by adding a category such as “other” and may follow “please specify”

2. The answer category must be mutually exclusive: respondent should not be compelled to select more than one to ensure this, we must carefully consider each combination of categories, could the respondent reasonably choose more than one answer?

If our survey is constructed carefully, we usually do not need to add an instruction to tell respondents to select the “best answer”.

Things to Note for whenAsking Questions

Make Items Clear: questionnaire items should be precise so that the respondent knows exactly what the researcher is asking

Avoid Double-Barrelled Questions: Researchers asking respondents for a single answer to question that actually has multiple parts

Respondents must be competent to answer: When asking respondents to provide information, must keep in mind whether they can do so reliably

Respondents must be willing to answer: Often, we would like to learn things from people that they are unwilling to share with us

Continued…

Questions should be relevant: Questionnaire should be relevant to most respondents

Shorts Items are the best: respondents are often unwilling to study an item in order to understand it, respondent should be able to read something quickly, understand the main point, and select an answer without difficulty, answers should not be misinterpreted

Avoid Negative Items/”Negative” words: appearance of a negation in a questionnaire items paves the way for easy misinterpretation

Avoid Biased Items and Terms: Meaning of someone’s response to a question depends in large part on its wording, some questions seem to encourage particular response more than do other questions

Three Main Methods for Administering Survey Questionnaires

Self administered questionnaires: respondents are asked to complete the questionnaire themselves (example: Mail survey), administer questionnaire to a group of respondents gathered at the same place at the same time.

Surveys administered by interviewers in face-to-face encounters

Surveys conducted by the telephone

Mail Distribution and Return

Research worker can either hand deliver questionnaires, requesting that respondents mail the completed questionnaires to the research office or after questionnaires are mailed, researcher visits homes to pick them up and check for completeness

Completion rate seems to be higher when research worker delivers the questionnaire, picks it up or both.

Basic method of collecting data through mail is to send questionnaire, with a letter of explanation and a self addressed, stamped envelope for retuning (for example, like the surveys we receive in our mailboxes, many often have the “postage paid” indicated in the corner of the envelope)

The Main reason for not returning questionnaires is that it’s TOO MUCH TROUBLE!

Acceptable Response Rates

Response rates: the number of people participating in a survey divided by the number selected in the sample (percentage form)

Inferential statistics used with survey analysis assumes that all members of the initial sample complete and return their questionnaires

Because that is very rare to happen, response bias is a concernresearchers often hopes for the possibility that the respondents look essentially like a random sample of the initial sample, and a somewhat smaller random sample of the total population

Continued…

Overall response rate is one guide to the representativeness of the sample reponsdentshigh response rates means less chance of significant response bias, with low response rates , correspondents are likely to differ from the respondents in ways other than their willingness to participate in the survey

Rough guides (with no statistical basis) to what is a good response rate: 50% is adequate for analysis and reporting, 60% is good and 70% is very good

Interview Surveys

Interviewers ask questions orally and record respondents answers Usually done in a face-to-face encounter, but telephone interviewing

follow similar guidelines Most interviews require more than one interviewers, but small scale

interviews can be done by one person Researchers must assume that a questionnaire item will mean the

same thing to every respondent and every given response must mean the same when given by different respondents

The interview’s presence should not affect a respondent's perception of a question or the answer given

To save time and money, a given interviewer is typically assigned to complete all the interviews in a particular geographical area

General Guidelines for Survey Interviewing

Appearance and Demeanor: interviewers should dress in a fashion similar to that of the people they’ll be interviewing (cleanliness and neatness in modest apparel). Dress and grooming are typically regarded as signs of a person’s attitudes and orientations. Interviewers should be pleasant, must communicate interest in getting to know the respondent without appearing to spy (never too casual or clingy). Interview will be more successful if the interviewer can become the kind of person the respondent is comfortable with, respondents deserve the most enjoyable experience the researcher can provide

Continued… Familiarity with Questionnaire: must be able to read questionnaire items to

respondents without error and stumbling over words, lines must be read as though they are part of a natural conversation

Interviewer must be familiar with the specifications prepared in conjunction with the questionnaire

Some questions will not fit a given respondent's situation, interviewer must determine how the question should be interpreted in that situation

It would be better for the interviewer to leave a question unanswered than to spend a period of time searching though the specifications for clarifications or trying to interpret the relevant instructions

Following Question Wording Exactly Recording Responses Exactly Probing for Responses: Sometimes respondents will give an

inappropriate/incomplete answer, request for an elaboration can be useful (Probe). Probes must be completely neutral, must not affect the nature of subsequent response

Advantages of Doing an Interview Survey

They typically attain high response rates (a properly designed and executed interview survey ought to achieve a completion rate of at least 80-85%)

Presence of and interviewer decreases the number of “don’t knows:” and “no answers”, since clarification can easily be done on the spot

Interviewers can clarify matters if respondent clearly misunderstands the intent of the question

Can observe respondents as well as ask questions

Telephone Surveys:Positive Factors (+)

Saves both money and time May dress anyway you please without affecting the

answers respondents give Respondents may be more honest in giving socially

disapproved answers (no eye to eye contact) Interviewers can communicate a lot about themselves

over the phone, even though they can’t be seen Allow greater control over data collection if several

interviewers are engaged in the project

Telephone Surveys:Negative Factors (-)

Some phone numbers are unlisted (but this has been erased through a technique called random digit dialling)

Bogus surveys-ones that are actually sales campaigns disguised as research

The ease in which people can hang up Answering machines (using machines to screen

calls)however research has showed that this had not yet had a significant effect on the ability of telephone researchers to contact prospective respondents

Comparing theThree Different Methods

Self Administered Questionnaires

Interview Surveys Telephone Surveys

Cheaper and quicker Produce fewer incomplete questionnaires

Cheaper and can be mounted and excited quickly

Costs no more to conduct a national survey than a local one of the same sample size

More effective for complicated surveys

Safer when interviewing high crime areas

Require a small staff Conduct survey based on sample of addresses or phone numbers rather on names

Impact of interviewers on responses is somewhat lessened when they can’t be seen

Computer Assisted Telephone Interviewing (CATI)

Increasingly used by academic, government and commercial survey researchers

Central computer is programmed to select a telephone number at random and dials it

Interviewer introduces study and asks the question displayed on the screen and then types that answer into the computer terminal (depending on the question-open/close ended)

Computer automatically prepares the data for analysis, or researcher can begin analysing data before the interview is complete (gaining and advanced view of how the analysis will turn out)

New Technologies andSurvey Research

CAPI (Computer assisted personal interviewing): face to face interviews rather than over the phone

CASI (Computer assisted self interviewing): respondent reads questions on the computer screen and enters his/her own answers

CSAQ (Computerized self administered questionnaire): respondent receives the questionnaire on a floppy disk, bulletin board, or other means and completes the questions, software then accepts the answers. Respondent then returns the data file

TDE (Touchtone data entry): initiates process by calling a number at the research organization, prompts a series of computerized questions, respondent answers by pressing the keys on the telephone pad

VR (Voice recognition): same as TDE, but the system accepts spoken responses

Internet and world wide web

Do’s and Don’ts for Conducting Online Surveys

DO Consistent wording between the

invitation and the survey Use plain simple language Offer to share selected results from

others who also have completed the survey

Plan the time of day and day of week to mail (when will respondents most likely be reading mail at home?)

Be aware of technical limitations (Will respondents have programs needed to access)

Test incentives, rewards and prize drawings to determine the optimal response

Limit studies to >15 minutes

DO NOT Use terms such as

unique ID number in the invitation, then ask respondents to type “password” when they get to the survey

Force the respondent to scroll down the screen for the URL for the study location

Secondary Analysis

Survey research involves 3 steps: questionnaire construction, sample selection, and data collection (interviewing/self administered questionnaires)

This is a form of research in which the data is collected and processed by one researcher are reanalyzed- often for a different purpose, by another

Example: General Social Survey (GSS) Advantage: cheaper and faster than doing original surveys, benefit

from the work of top flight professionals, enhance possibility of meta analysis (researcher brings together a body of past research on a topic)

Disadvantage: question of validity (have no assurance that data collected will be appropriate for you research interests)

Rigour in Survey Research

1. Internal Validity To be internally valid, the conclusions of the

research must be supported by the data. Internal validity is judged according to the

accuracy with which a description of particular events represents the data.

The essence of internal validity for survey research is complete confidence that your conclusions come from the data.

Reliability

2. Reliability Reliability is generally concerned with

replication: an account is considered to be reliable if the data are reproducible.

If the analytic strategy were repeated by the same or different investigator, then the results should be the same.

Another Type of Validity

3. External Validity External validity or generalizability for survey

research is captured by the question, “How can one determine the extent to which the findings of a particular inquiry have applicability in other contexts or with other subjects?”

Essentially, the extent to which the effect can be generalized to populations, settings, treatment variables, and measurement variables.

External Validity Continued…

This can also refer to “fit” or the degree to which the audience or reader of the report is able to transfer the research findings to contexts outside of the study situation to other settings.

The researcher must supply a substantial amount of clear and detailed information or thick description about the issue/phenomenon studied and the setting in which that issue/phenomenon was found.

The degree of transferability is a direct function of the similarity or “fittingness” between the two contexts.

Strategies for Ensuring Rigour

1. Verification Strategies The most important way to ensure that

research is rigourous is to focus on verification during the study.

Verification is the process of checking, confirming, making sure, being certain.

Doing this as the study is conducted is key as the researcher then can identify and correct threats to reliability and validity as they surface.

Initial Strategies Address:

Investigator Responsiveness Methodological Coherence Sampling Data Analysis Thinking Theoretically

Additional Strategies

There are other strategies that can be used during the research to contribute to rigour, namely:

Prolonged Engagement Participant Checks Journal Writing Peer Review Audit Trail.

Weaknesses for Survey Research

Lacks the context of social life Appear superficial on coverage of complex

topics Subject to artificiality Surveys can also be inflexible because the

initial study design must remain the same Therefore, surveys are weak on validity

Strengths for Survey Research

Describes the characteristics of a large population Allows a large sample of respondents Surveys are flexible, allows flexibility in your analyses Surveys have a strength in measurement Helps determine unemployment rates, and voting

intensions Helps examine official documents such as marriage,

birth, or death records Therefore, surveys are strong on reliability

Survey Design

1. Purpose 2. Respondent Group 3. Questionnaire Construction 4. Administration of Questionnaire 5. Analysis Example: Sample Questionnaire (Handout)Student Opinion Survey on Group Assignments in

University Courses

Format Guidelines

Introduction Spacing Explanations Order of items Pre-testing

Additional Resources

Presentation Website:http://members.shaw.ca/kristopher.skinner/soc315

Group Project E-mail: soc315project@hotmail.com