Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey...

23
Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30 SSB 447 SOCI 60: Lecture 18 Benjamin Graham Wednesday, April 3, 13

Transcript of Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey...

Page 1: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Survey Research (3): Survey Experiments & Survey Examples

Benjamin GrahamOffice Hours: M 11:30-12:30, W 10:30-12:30

SSB 447

SOCI 60: Lecture 18 Benjamin Graham

Wednesday, April 3, 13

Page 2: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Housekeeping

• No class on Thursday• Homework 4 is due a week from Thursday

• I will e-mail it out this afternoon

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 3: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Experiments (Review Question)

• What are the three components of a true experiment?• A. At least 2 groups; treatment precedes the post-test, random

assignment• B. At least 2 groups, treatment precedes the post-test, non-random

assignment• C. At least 2 groups, pre-test precedes the treatment, random assignment• D. None of the above

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 4: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Survey Experiments

• The “treatment” is the version of the survey you are given.• Generally we vary the information you are provided

• The dependent variable is respondents’ answer to some question on the survey• Comes after the treatment• Same question for everyone

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 5: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Survey Experiments: so so good, so so bad

• What is totally awesome about survey experiments?• Ethically and logistically much easier than “real” experiments• Random assignment is super easy

• What are some of the drawbacks?• Does our “treatment” actually look like the concept we’re interested in?

• i.e. Do people respond to stories the way they respond to real events?

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 6: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Example

• If you were born in an odd numbered month, you are group A• If you were born in an even numbered month, your are group B

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 7: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Group A ONLY

• A terrorist group has taken 100 US citizens hostage and threatens to kill them if the US does not release several of their members currently being held at Guantanamo bay. • What should the President do?

A. Agree to the dealB. Refuse to negotiateC. Launch a military assault against the Iranian government

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 8: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Group B ONLY

• A terrorist group has taken 2 US citizens hostage and threatens to kill them if the US does not release a Yemeni cleric currently being held at Guantanamo bay. • What should the President do?

A. Agree to the dealB. Refuse to negotiateC. Launch a military assault against the Iranian government

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 9: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Survey Experiments

• First we take our sample (randomized, if possible)• Then divide it into a treatment group and one or more comparison groups

(definitely randomized)

• The treatment is some form of vignette (story). We describe a situation that we can’t actually create.

• Varying the details of a story is much easier than varying actual events.

• Hypothesis: Americans are more willing to support negotiating with terrorists when the risk of US casualties (from non-negotiation) is large.

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 10: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Another Example

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Firm # ______

6""

________________________________________________________________________________________________________________________________________________________________________________"

![SURVEY!EXPERIMENT]![Respondents!are!randomly!assigned!to!either!scenario!1!or!scenario!2]!C8.2."For"this"next"question,"we’ll"ask"you"to"think"of"some"of"the"entry[level"professional"jobs"at"your"firm."I"am" going" to" read" a" description" of" a" candidate," and" based" on" this" description," I" will" ask" you" to" rate" the"candidate"on"a"scale"of"1"to"10,"where"1"indicates"a"very"low"quality"candidate,"and"10"is"the"highest"quality"candidate."After"that,"I"will"also"ask"you"to"rate"on"a"scale"of"1"to"7"on"how"likely"you"would"be"to"hire"this"candidate."""

On"a"scale"of"1"to"10,"where"1"is"poor"quality"and"10"is"highest"quality,"how"would"you"rate"this"candidate?"[C821]"

How"likely"would"your"firm"be"to"hire"this"candidate?"[C822]"

Scenario!1:"Candidate"A"is"a"recent"graduate"from"a"reputable"academic"institution"in"the"Philippines"and"studied"business."He"has"very"good"academic"credentials,"and"graduated"in"the"top"10%"of"his"class."He"has"heard"a"lot"about"your"firm,"and"he"seems"enthusiastic"about"the"work."

" "

C822:""""1["Extremely"unlikely"" 2["Unlikely"" 3["Somewhat"unlikely"" 4["Neither"likely"nor"unlikely"" 5[Somewhat"Likely"" 6["Likely" "7[Extremely"Likely"

On"a"scale"of"1"to"10,"where"1"is"poor"quality"and"10"is"highest"quality,"how"would"you"rate"this"candidate?"[C823]"

How"likely"would"your"firm"be"to"hire"this"candidate?"[C824]"

Scenario!2:"Candidate"A"is"a"recent"graduate"from"a"reputable"academic"institution"in"the"Philippines"and"studied"business."He"has"very"good"academic"credentials,"and"graduated"in"the"top"10%"of"his"class."He"has"heard"a"lot"about"your"firm"because!his!uncle!is!one!of!the!managers,"and"he"seems"enthusiastic"about"the"work.""

" "

C824:""""1["Extremely"unlikely"" 2["Unlikely"" 3["Somewhat"unlikely"" 4["Neither"likely"nor"unlikely"" 5[Somewhat"Likely"" 6["Likely" "7[Extremely"Likely"

!! !Section!D:!Political!Risk!and!Dispute!Resolution(Renumbering"Necessary)"Now"I"have"a"few"questions"about"the"business"and"regulatory"environment"in"the"Philippines.""We"are"interested"in"understanding"the"business"climate"your"firm"faces.""D7.5."Does"your"firm"ever"take"steps"intended"to"affect"government"policy"in"the"Philippines"at"the"national"level?"_____"[D75]"" " 1"–"Yes""" 2"–"No""

If"answer"to"question"D7.5"is"yes:"D7.5A."Compared"to"other"firms"in"your"sector,"is"your"firm"more"or"less"effective"in"influencing"government"policy"at"the"national"level?"______[D75A]""" 1"–"Less"Effective"" 2"–"Equally"Effective"" 3"–"More"Effective""

"D7.5B."What"makes"your"firm"more"or"less"effective"than"other"firms"in"your"sector"at"influencing"government"policy"at"the"national"level?"[D75B]"

________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________"

Ben Graham� 4/2/13 10:36 AMFormatted Table

Wednesday, April 3, 13

Page 11: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Combining Survey Experiments and Shielded Response

• I want to know if people under-report a particular behavior in surveys• Group A: Ask the question directly• Group B: Use shielded response

• If people are under-reporting, what should be true?

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 12: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Example

Listed  below  are  the  various  things  that  businesses  can  do  to  improve  rela6ons  and  influence  government  policy.  Please  indicate  which  of  these  strategies  your  firm  has  used  in  the  past.  (Put  1  to  all  that  apply)

_______  D8.1  Contribute  to  governmental  funding  for  public  works  and  services  

  _______  D8.2  Invite  officials  for  dinner  and  entertainment  in  order  to  get  more  “face  6me”     _______  D8.3  Privately  increase  individual  officials’  “gray  income.”  (lagay)  

_______  D8.4  Contribute  to  governmental  capacity:  e.g.  giN  a  new  wing  to  the  government  building,  giN  office  equipment  etc.  

_______  D8.5  Invite  officials  to  visit/inspect  your  business  to  highlight  your  contribu6on  to  the  local  economy  and  community.  

_______  D8.6  AUend  business  forums  and  func6ons  organized  by  the  government.  

Group  A:  Ask  each  item  individually.Group  B:  Asked  how  many  items  on  the  list  they  par6cipated  in.

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 13: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Checking Understanding

•We add up the total number of items that each firm in Group A admits to doing, and compare that to the number of items that firms in Group B admit to doing. What should be true about Group A (aswered each item individually) and Group B (told us the total number of things they did) if no one is underreporting?•A. Group A reports a higher average number of items•B. Group B reports a higher average number of items•C. Both groups report EXACTLY the same number of items•D. Both groups report the same number of items, plus or

minus random sampling error.

IR 211: Lecture 19 -- Survey Research (3) Benjamin Graham

Wednesday, April 3, 13

Page 14: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

 The  Survey

1.First  survey  to  compare  diaspora-­‐owned  firms  to  other  foreign  firms.  

2.Random  sample  of  foreign  firms  in  Georgia.• Snowball  sample  of  7  addi6onal  diaspora-­‐owned  firms    

3.Final  sample:  161  firms  from  38  countries,  primarily  registered  2004-­‐2007.• 30  diaspora-­‐owned  firms• 121  non-­‐diaspora-­‐owned  foreign  firms  (10  unknown)

4.Respondents  were:• 75%  firm  managers,  25%  owners• 91%  ethnic  Georgians

Political Risk, Social Networks, and Social Responsibility Benjamin A.T. Graham

Wednesday, April 3, 13

Page 15: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Tes6ng  the  Diaspora  Difference

1. Similarity  in  terms  of  firm  demographics.

2. Social  networks  are  important  to  firm  loca6on  decisions.    

3. Diaspora-­‐owned  firms  use  social  networks  to  gain  compe66ve  advantages.

4. Diaspora-­‐owned  firms  do  not  engage  in  more    pro-­‐development  or  socially  responsible  behavior.

Political Risk, Social Networks, and Social Responsibility Benjamin A.T. Graham

Wednesday, April 3, 13

Page 16: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Wednesday, April 3, 13

Page 17: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Using  Social  Networks  to  Gain  Compe66ve  Advantage

1. How  important  are  your  owners’  and  managers’  family  rela*onships  to  increasing  the  profitability  of  your  firm?

•    Also  asked  about  friendships.2. Has  your  firm  ever  rented  or  purchased  real  estate  

with  the  help  of  a  family  member  of  one  of  your  firm’s  owners  or  managers?

• Also  asked  about  friends.

Political Risk, Social Networks, and Social Responsibility Benjamin A.T. Graham

Wednesday, April 3, 13

Page 18: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Use  of  Social  Networks

-1.5 -0.5 0.5 1.5

Friendships for Real Estate

Family for Real Estate

Friendships and Profitability

Family and Profitability

Social Networks and Firm Entry

Expectations of My TheoryDiaspora Mean - Non-Diaspora Mean

Mean = 0, Standard Deviation = 1

Wednesday, April 3, 13

Page 19: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Use  of  Social  Networks

-1.5 -0.5 0.5 1.5

Friendships for Real Estate

Family for Real Estate

Friendships and Profitability

Family and Profitability

Social Networks and Firm Entry

Actual Findings:Difference in Means

Diaspora Mean - Non-Diaspora Mean

Standard Deviations Normalized to 1

Wednesday, April 3, 13

Page 20: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Pro-­‐Development  and  Socially  Responsible  Behavior

Ques6ons  include:

1. “Compared  to  employees  at  other  firms  in  this  sector,  employees  at  this  firm  are  able  to  provide  a  higher  quality  of  life  for  their  families.”

2. “This  firm  contributes  to  economic  development  in  Georgia  more  than  other  firms  in  this  sector.”

Political Risk, Social Networks, and Social Responsibility Benjamin A.T. Graham

Wednesday, April 3, 13

Page 21: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Political Risk, Social Networks, and Social Responsibility Benjamin A.T. Graham

Wednesday, April 3, 13

Page 22: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Pro-­‐Development  and  Socially  Responsible  Behavior

-1.5 -0.5 0.5 1.5

Contribute to Charity

Contribute to Development

Prioritize Local Labor

Pay Higher Salaries

Higher Quality of Life

Professional Development

*Minimize Total Labor Costs

Expectations of My TheoryDiaspora Mean - Non-Diaspora Mean

Mean = 0, Standard Deviation = 1

Wednesday, April 3, 13

Page 23: Survey Research (3): Survey Experiments & Survey Examples · Survey Research (3): Survey Experiments & Survey Examples Benjamin Graham Office Hours: M 11:30-12:30, W 10:30-12:30

Pro-­‐Development  and  Socially  Responsible  Behavior

-1.5 -0.5 0.5 1.5

Contribute to Charity

Contribute to Development

Prioritize Local Labor

Pay Higher Salaries

Higher Quality of Life

Professional Development

*Minimize Total Labor Costs

Actual Findings:Difference in Means

Diaspora Mean - Non-Diaspora Mean

Standard Deviations Normalized to 1

Wednesday, April 3, 13