Foundations of Research Survey Research This is a PowerPoint Show Open it as a show by going to slide show. Click through it by pressing any key. Focus & think about each point; do not just passively click. This is a PowerPoint Show Open it as a show by going to slide show. Click through it by pressing any key. Focus & think about each point; do not just passively click. Dr. David J. McKirnan, 2014 The University of Illinois Chicago Do not use or reproduce without permission Part 1 Center for Epidemiologic Studies National Institute of Mental Health Foundations of Research 2 Topic areas & formats Testing Hypotheses with surveys General issues in Survey research Sources of bias (or fraud) in survey research Examples of surveys We will address five topics in the Survey modules: Part 1 will address the first four. 13. Survey Research Part 1 Foundations of Research 3 Survey Research Topic areas & formats General issues in Survey research Sources of bias (or fraud) in survey research Examples of surveys Foundations of Research 4 What do surveys measure? Knowledge Information re: current events, political or consumer choices Awareness: e.g., of Public health resources, government decisions... Attitudes and Beliefs Preferences or evaluations: e.g., attitudes toward racial or ethnic groups, consumer preferences... Beliefs about political or social events: which party provides the strongest security for the U.S.? Feelings or moods: quality of life, depression / anxiety, marital satisfaction... Behavior Behavioral intentions; Intent to vote, financial plans, exercise goals. Self-reports of previous or on-going behavior; voting in the last election, alcohol and drug use, exercise patterns. Foundations of Research 5 Descriptive research Testing hypotheses; Testing the generalizability of experimental results; Predicting an event or outcome; Pragmatic / applied questions. Survey research; General uses of surveys Survey methods have a wide range of applications, from single-item consumer satisfaction (How useful did you find this web site) to full-fledged, theory-driven behavioral research. For convenience we will consider 5 categories: Foundations of Research 6 Uses of surveys; descriptive research Descriptive research Epidemiology is the study of how behaviors, disease states, or similar issues are distributed across the population. Epidemiology uses many methods, such as standard crime or disease reporting. Even Google search data are used to track flu spread.track flu spread Many epidemiological studies use direct survey methods, such as phone or face-to-face survey interviews. Knowledge of, e.g., how to access health care Feelings or moods, such as the rate and distribution of depression Behavioral patterns, such as alcohol or drug use or gun ownership Foundations of Research 7 The origins of epidemiology Dr. John Snows Cholera map & the closing of the Broad St. pump. In London of 1854 a cholera outbreak raged through several poor neighborhoods of London. Sewage and other effluvia that ran through gutters created a dreadful smell (a miasma) that was blamed for the outbreak. The concept of infectious disease transmission through water supplies was not well understood. Dr. John Snow, one of the physicians charged with stopping the epidemic, noted a particularly fetid cesspool in front of 40 Broad St., proximal to a water pump used by the neighborhood. He decided to empirically map the cholera cases in the area. He proposed that water from the pump, not the miasma, was the cause of the outbreak. He was was generally disbelieved, but convinced the town governors by his evidence. Only much later would tracking of disease outbreak be labeled epidemiology. Click for Wikipedia article. Image: html Foundations of Research 8 "I had an interview with the Board of Guardians of St. James's parish, on the evening of Thursday, 7th September, and represented the above circumstances to them. In consequence of what I said, the handle of the pump was removed on the following day. The origins of epidemiology Dr. John Snows Cholera map & the closing of the Broad St. pump. Snows map showed the bulk of cases to be near the pump at 40 Broad St. and to radiate out from there. As he noted in his 1855 book:book By carefully describing the distribution of cases and the circumstances around the pump, Snow was able to empirically demonstrate a likely cause. His hypothesis was supported by the epidemic quickly subsiding once the pump handle was removed. Click for Ted talk by Steven Johnson. S. Johnson, The Ghost Map (2007), Riverhead Books. Foundations of Research 9 Descriptive research Epidemiology Political / social description is what we often think of as surveys. Opinion polls about society, the government, or current events, e.g. Gallup Polls, or systemic studies by Pew Memorial Trust.Gallup PollsPew Memorial Trust. The Consumer Confidence Index is a highly standardized poll that is used for basic economic decision making.Consumer Confidence Index The Census, of course, is our national information source.Census Uses of surveys; descriptive research Foundations of Research 10 Descriptive research Epidemiology Political / social description. Testing hypothesis Assessing blocking variables We often assess blocking variables to test how a given attitude or behavior varies across important social groups. Correlational studies A key form of analysis is examining the association among different variables Uses of surveys; descriptive research e.g., gender, age group, ethnicity, geographic location e.g., what are the correlates of dieting Foundations of Research 11 Descriptive research Epidemiology Political / social description. Testing hypothesis Assessing blocking variables Correlational studies Examine generalizability of experimental results E.g., The Consumer Reports survey on therapy we discussed in quasi- experiments Predict event or outcome; E.g., election polling Pragmatic / applied, E.g., marketing, or consumer surveys. Uses of surveys; descriptive research Foundations of Research 12 Surveys; populations Who do we want to generalize to? Our sampling frame is based on our hypothesis or empirical question. Sampling: breadth internal validity tradeoff Key dimensions: Demographic ethnic / age / gender groups, all Americans Behavioral likely voters, alcohol users, home buyers... Self-identification Republicans / Democrats, students See Design and sampling overview See also: diminishing validity of political polling.Design and sampling overviewdiminishing validity of political polling Foundations of Research 13 Research methods is a wonderful course Does not strongly agree at all agree How many times this semester have you skipped class?______ How many hours per day do you spend reading the material?______ Question Formats: Closed-ended items Specific rating scale or highly structured prompts Most reliable for concrete behaviors An attitude can be assessed in several ways: Researchers typically use the M of several related items to create a more reliable measure of a variable. Behavioral (content valid) indictors Direct (face valid) assessment Foundations of Research 14 Closed-ended items, cont. Moods & Feelings Below is a list of different feelings. Circle the number that shows how many days you felt each of these over the PAST WEEK. Rarely or A Little A moderateMost or all of none of of the Timeamount of the time the time the time ( less than 1 day)(1 or 2 days)(3 - 4 days)(5 - 7 days) I was bothered by things that usually do 0123 not bother me. I felt I could not shake off the blues even 0123 with help from my friends or family. I had trouble keeping my mind on what 0123 I was doing. I felt depressed.0123 I felt that everything I did was an effort.0123 My sleep was restless.0123 I was happy.0123 I enjoyed life.0123 I felt sad.0123 Example: The CES-D Depression inventory: We may use the M score of these 9 items as our depression index Sum of item ratings / 9 These items are reversed in the final score Foundations of Research 15 Moods & Feelings Below is a list of different feelings. Circle the number that shows how many days you felt each of these over the PAST WEEK. Rarely or A Little A moderateMost or all of none of of the Timeamount of the time the time the time ( less than 1 day)(1 or 2 days)(3 - 4 days)(5 - 7 days) I was bothered by things that usually do 0123 not bother me. I felt I could not shake off the blues even 0123 with help from my friends or family. I had trouble keeping my mind on what 0123 I was doing. I felt depressed.0123 I felt that everything I did was an effort.0123 My sleep was restless.0123 I was happy.0123 I enjoyed life.0123 I felt sad.0123 Closed-ended items, cont. # of symptoms: items rated 2 or 3 Or we may count the number of symptoms We may have a cut point: e.g., moderate depression is defined as 4+ symptoms Center for Epidemiologic Studies National Institute of Mental Health Foundations of Research 16 Closed-ended item s, summary Chief virtue : clear operationalization Specific & concrete; we know exactly what the participant is responding to Easy to quantify & use statistically Can be tested for reliability Chief liability : potential insensitivity Brief & simply worded; potentially superficial Top down; issues are imposed on the participant Discrimination studies: no option for has no attitude Attitudes / moods: not sensitive to participants personal perspectives (?). Foundations of Research 17 Survey formats; Open-ended items General textual / qualitative response; More sensitive to the respondent How have you enjoyed your methods class so far? Please list the three things that first come to mind when you think of The Foundations of Research. More difficult to interpret Can be analyzed as qualitative data (see discussion in Descriptive data.)Descriptive data. Can be quantified; frequency counts of citations or statements linkages analysis of co-occurring statements Often presented as textual portrayal plus minor quantitative analysis. Foundations of Research 18 Mixed survey formats Personal Safer Sex Guidelines How strict are your personal guidelines or rules for safer sex (e.g., condom use, safe relationships, etc.)? Not at allSomewhatVery Extremely Strict StrictStrictStrict What are your rules for safer sex? Have you ever refused to have sex 0123 with someone to stay safe? Neveronce or a fewmany twice timestimes Personal Safer Sex Guidelines How strict are your personal guidelines or rules for safer sex (e.g., condom use, safe relationships, etc.)? Not at allSomewhatVery Extremely Strict StrictStrictStrict What are your rules for safer sex? Have you ever refused to have sex 0123 with someone to stay safe? Neveronce or a fewmany twice timestimes Example of a mixed question format from a survey of womens sexual practices. Closed-ended attitude scale Open-ended description Simple behavioral index. Foundations of Research 19 Survey topics & item types Surveys assess: Knowledge Attitudes or preferences Ongoing or intended behavior Closed-ended formats Highly structured, easy to analyze Potentially insensitive Open-ended formats More sensitive to the participant Potentially ambiguous or difficult to analyze Surveys typically Use multiple items Employ several formats. S U M M A R Y Psychology 242 is a wonderful course list the three things that first come to mind Foundations of Research 20 Survey Research Topic areas & formats General issues in Survey research Sources of bias (or fraud) in survey research Examples of surveys Foundations of Research 21 Forms of survey administration Self-report questionnaire Paper and pencil or internet-based ; Primarily closed-ended, structured questions Limited open-ended items Assume at least moderate reading level Cheap & easy to administer Internet: Representativeness very dubious Face-to-face interview Door step, formal research center, or telephone Allows in-depth qualitative questions Many studies combine questionnaire & interview formats (Telephone version becoming obsolete) All data collection increasingly computer-based Foundations of Research 22 Cost / population access Different methods are more / less likely to reach certain populations, e.g.: Disfranchised / poor populations often not reached by internet or telephone Cell phones & avoidance of telemarketers less availability for telephone surveys Stigmatized populations less available for face-to-face interviews, more available via internet. Participant sophistication Participants may not be able to accurately report certain topics Attitudes toward stem cell research from readings. What factors are most important to your choice of political candidate.... Describe the amounts and types of proteins you eat during a typical week... Rationality bias; many questions (incorrectly?) assume a rational reason for behavior: Why do you have unsafe sex... What is your chief reason for using alcohol each night General issues in surveys Foundations of Research 23 Clear face-valid items addressing embarrassing topics yield less valid responses How often are you dishonest with your friends? Have you ever cheated on an exam....? High social desirability wording elicits inaccurate responses Do you support protecting our Nations forests for future generations ? (Does yes mean you an environmentalist?). Do you feel there are ways your husband could be closer...? (Does yes make you are unhappy in your marriage?). Populations differ in social desirability responding; may be a confound in studying group differences Women report more suicidal thoughts, but may be more willing to disclose, creating a possible confound Desirability can be minimized by: Anonymous surveys Assurances of confidentiality Computer administration (no personal interaction) Careful wording / pilot testing of items Social Desirability Responding Foundations of Research 24 Click for article from phys.org Social desirability responding Do people lie on surveys? Men routinely report more sex partners than do women. If the sample is unbiased by gender, number of partners should balance for men & women. Social desirability hypothesis: Women underestimate partners Men overstate partners Much of the difference due to: A high proportion of women who report 1 partner A few men who report many partners. Possible sample bias (confound?) in who responds to such surveys? Click image for NY Times article Foundations of Research 25 General issues in surveys: Time Frames Rare(er) events require a long time frame to assess When was your last doctors visit These questions asses the last time you left a romantic relationship Longer term recall can be surprisingly unreliable Recall of last doctor visit highly unreliable when checked against medical records Shorter time frame yields more reliable responding Memory is better for more recent effects Exit interviews from medical visits far more reliable than even 2- week retrospective measures. Current, concrete behaviors are more accurately reported than behavioral trends. In general, how often do you miss a dose of your medication Lets go over each of the past 7 days and tell me if you took or missed your medication dose. Less reliable than Foundations of Research 26 General issues in surveys: Question Order Questions trigger participants memory or attention, and can bias questions that follow, e.g.: Do you think Social Security & Medicare payments have kept up with inflation.. Do you favor or oppose Democratic efforts to expand Medicare payments... Bias can be limited by counterbalancing questions. Using different question orders in different versions of the survey. then Foundations of Research 27 General Issues Summary Survey administration Internet increasingly important as self-report method Face-to-face interviews more common in clinical research Time frames & question order can influence responses Population access & sophistication Some groups are difficult to reach Creates threat to External validity Assumption that participants understand survey materials often questionable. Social desirability responding Inhibited responding threatens Internal Validity May represent a confound if groups differ in desirability set. Summary Foundations of Research 28 Survey Research Topic areas & formats General issues in Survey research Sources of bias (or fraud) Examples of surveys Foundations of Research 29 Bias / Fraud in survey research Social research is increasingly important to political & cultural debates. Effects of gay marriage Political approval ratings Scientific consensus on global warming Research on working mothers Pressure for confirmatory results encourages bias or outright fraud (see this weeks article on Opinion Polls).Opinion Polls In the study structure Items used Sample In the interpretation of results Cherry picking Simple distortion Foundations of Research 30 Example of fraudulent survey use Example Opposition to gay marriage Judges have struck down religious doctrine as a basis for disallowing gay marriage April DeBoer and Jayne Rowse after closing arguments in their challenge to Michigans marriage restriction. Click image for coverage. Credit Mandi Wright/Detroit Free Press, via Associated Press Opponents have sought evidence of civil harm to justify discrimination Harm to children has emerged as key issue in the debate APA; multiple studies show no harm. Mark Regenerus Survey: Children in households with gay/lesbian parent fare worse.Survey Funded & cited widely by gay marriage opponents Study sample and interpretation of results wildly biased Has been disavowed by Renenerus Academic Department and American Sociological Assn. Continues to be central evidence in court cases Foundations of Research 31 Bias in survey research: Leading or biased items Sources of survey fraud; question wording can elicit a response desired by the researcher; How much do you support the administrations actions to protect you and your children from terrorists Wording can normalize a response, e.g., When do you feel that it is O.K. to cheat on an exam?..when I really do not know the material.. when others are doing it.. when I think the exam is unfair Vague wording can be interpreted in a biased fashion Is there anything your husband could do to be more intimate with you? Push polls: a survey can be used to actually create an attitude. Foundations of Research 32 An item traps the participant into endorsing a specific view. When the data are released the biased wording is ignored. Publicity about the findings (e.g., by politically biased news organizations) are used to further create or change attitudes. Many political & social organizations use this strategy to Ostensibly measure attitudes objectively Use the results to influence popular opinion. Example of a highly biased survey: The Republican National Committee Future of American Health Care survey. The survey was distributed in several counties as part of a fund raising letter. It is clearly a push poll designed to create fear of health care reform. It got limited distribution, but is a great example of a Push Poll Push Polls: Foundations of Research 33 GOP health survey (2009 2010) Some of these items are simple lies or manipulations based on lies designed to induce anti-health care attitudes Other are powerful (and dishonest) emotional manipulations Some of these items are simple lies or manipulations based on lies designed to induce anti-health care attitudes Other are powerful (and dishonest) emotional manipulations Foundations of Research 34 Forms of survey bias: Questions that, if you accept their assumptions, can only be reasonably answered one way Provide leading or emotionally manipulative information to induce an attitude rather than simply measure it, to provide politically useful data.. Foundations of Research 35 Biased surveys: Democratic example An example from the Democrats, that is also used for fund-raising. Foundations of Research 36 Democratic biased survey (2007) Manipulative presentation of questionable information Simple emotional manipulation Distorted description that may be changed in presentation of findings Who could disagree item. Foundations of Research 37 Summary: Manipulating attitudes by surveys 1. Ask manipulative or highly leading questions 2. Find high levels of agreement (and potentially change participants attitudes). 3. Publicize and often distort or overstate the findings via highly biased news sources 4. News reports themselves lead to attitude change among people who are uncertain or uninformed. Summary Foundations of Research 38 Survey Research Topic areas & formats General issues in Survey research Sources of bias (or fraud) Examples of surveys Foundations of Research 39 Examples of surveys & data, 1 Consumer reports survey of mental health care question Satisfaction with therapy. Differences between types of therapy. population Self-Identified group: U.S. mental health care users sample Self-selected convenience sample: Readers who got therapy & returned the survey, n=4000 data Attitudes & behavior Self-report questionnaire, cross-sectional findings Descriptive & hypothesis tests High satisfaction for most treatments Foundations of Research 40 Examples of surveys & data, 2 Monitoring the future youth studies question Social behavior Academics Alcohol & drug use Health. population Demographic group: All U.S. youth, 15 -> 21 years old. sample Random sample: Sample of High School health classes, n=3000 -> data Knowledge, attitudes and behavior Face--to--face interviews & questionnaires, longitudinal (bi-yearly) findings Mostly descriptive Assess yearly trends/shifts in drugs, grades, emotional well being Foundations of Research 41 Examples of surveys & data, 3 Gallup, Time/CNN, other polls question Political opinions, Lifestyle information Social attitudes, e.g., managed care populations Demographic: -Eligible voters, -Target age groups Self-identified: Democrats Behavioral: Voters, ACA users General: - U.S. adult population samples National, random Digit dial telephone, n=150 to >500 data Knowledge, Attitudes, Behavior Brief interview, cross-sectional findings Descriptive Ratings of politicians, Consumer preferences Approach to Affordable Care Act Foundations of Research 42 Examples of surveys & data, 4 Exit polls question Election outcome, possibly stratified by state / region population U.S. electorate National and/or local electoral district sample Probability Stratified random sample of electoral districts. data Self-reported behavior Self-report interview, cross-sectional findings Descriptive/ predictive Increasingly inaccurate predictions See reading on shifts in use of polling data in U.S. politicsreading Foundations of Research 43 Examples of surveys & data, 5 Social Issues Survey of Chicago gay / lesbian community question - Stress & coping - Alcohol & drug use - Responses to HIV / AIDS population Self-identified Self-identified gay, lesbian, & bisexual adults in Chicago. sample Targeted multi- frame community newspapers, organizations, & mailing lists, n=3500 data Attitudes & behavior Self-report questionnaire, cross-sectional findings Descriptive & hypotheses - High experience of discrimination - Less stress & alcohol-drug use than expected Foundations of Research 44 Examples of surveys & data, 6 National Institute on Drug Abuse Household survey of Alcohol and Drug use question Alcohol-drug use and problems, treatment use, health effects. population National All U.S. adults sample Random Multi- stage: 1.Census tract 2.Household, 3. Any adult member n>4000 data Knowledge, attitudes & behavior Face - to - face Interview, successive cross-sectional (each 5 years) findings Typically descriptive Age & regional differences in substance use, trends over time in use & problems Data often used for hypothesis-oriented secondary analyses (i.e., as archival data). Foundations of Research 45 Summary: Testing Hypotheses Surveys typically use multiple items to measure each hypothetical construct Correlations among items tell us if they are reliable in measuring the same construct. We use Mediating Analyses to Test hypotheses about correlations between constructs Build or test theory Cross-sectional analyses are difficult to interpret Causal direction? 3 rd variable problem Longitudinal analyses help us determine causal direction Summary