Post on 16-Oct-2020
Richard Johnston University of British Columbia (ex University of Pennsylvania
& National Annenberg Election Study)
Presented to ELECDEM Conference University of Exeter 19 November 2009
Challenges in Survey-Based Electoral Research
Representation Challenges
Persons
Opinions
Processes
… tradeoffs among these
Modes of survey data collection Mail Face-to-Face Telephone
Web Comparisons combine considerations that are: intrinsic to mode as such
very critical to representing opinion
byproducts of how mode constrains how samples are drawn very critical to representing persons quite critical to representing processes
Mode and Sampling I: Unit Nonresponse
Coverage
Mail F TF Phone > Web (what is the frame?)
Contact
Mail (but within-HH issues) Web (but so what?) Phone > FTF
Compliance
FTF > Phone ? Web > Mail
Nonresponse and bias
FTF > all others
Phone & Web > Mail
Phone vs Web? -- debate over mix of equal-probability of selection and self-recruitment
€
≈
€
≥
€
≈
€
≥
Representing persons
Mode and Sampling II: Power per respondent -- Clustering
FTF is expensive
samples are clustered spatially to minimize cost per interview
Representing persons
Mode and Sampling II: Power per respondent -- Clustering
Effective N of observations =
where: J = number of clusters
n = observations per cluster
= intraclass correlation =
= between-cluster variance
= within-cluster variance
€
Jn1+ (n −1)ρ
€
ρ
€
τ 2
τ 2 +σ 2
€
τ 2
€
σ 2
Representing persons
Mode and Sampling II: Power per respondent -- Clustering
Effective N of observations =
where: J = number of clusters
n = observations per cluster
= intraclass correlation =
= between-cluster variance
= within-cluster variance
€
Jn1+ (n −1)ρ
€
ρ
€
τ 2
τ 2 +σ 2
€
τ 2
€
σ 2
Representing persons
Mode and Sampling II: Power per respondent -- Clustering
Effective N of observations =
where: J = number of clusters
n = observations per cluster
= intraclass correlation =
= between-cluster variance
= within-cluster variance
€
Jn1+ (n −1)ρ
€
ρ
€
τ 2
τ 2 +σ 2
€
τ 2
€
σ 2
Representing persons
Mode and Q&A Factors intrinsic to mode: Presence or absence of an interviewer
mail and web: no interviewer FTF and phone: interviewer
but cf CAPI – can make some response private but cf automated phone surveys – ditto
Visual vs oral/aural
Representing opinions
Interviewer effects Interviewers can:
assist respondent “model professionalism” but force the pace artificially bend interpretation of items produce respondent/interviewer interactions – e.g. race,gender inhibit socially undesirable response
Web and mail facilitate socially disfavoured opinions e.g. sexuality, opinion on sexuality, opinion on race
Representing opinions
Visual vs Oral/Aural minimum of interpretation vs premium on literacy
web: forget about political knowledge indicators
Rank order effects: visual first item oral/aural last item
Optimizing vs satisficing – measurement error
non-differentiation
acquiescence
non-response (not distributionally neutral, cf Berinsky)
web response seems to be more optimized
Mode in the CSES Module 2
MODE OF INTERVIEW | Freq. Percent Cum.
----------------------------------------+-----------------------------------
1. FACE-TO-FACE | 46,172 71.86 71.86
2. TELEPHONE | 5,658 8.81 80.66
3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72
4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53
5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00
----------------------------------------+-----------------------------------
Total | 64,256 100.00
(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total
----------------------+-------------------------------------------------------+----------
00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116
03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769
05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225
07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514
10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482
12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674
15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200
15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022
15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823
20302002. CZECH REPUB | 948 0 0 0 0 | 948
20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026
24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196
25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000
27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000
27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023
34402004. HONG KONG ( | 582 0 0 0 0 | 582
34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200
35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446
37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367
37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212
38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439
39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977
41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500
41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000
48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991
52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574
55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741
57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052
60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032
60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200
61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794
62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303
62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801
64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913
64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496
70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002
72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212
75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060
75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418
82602005. GREAT BRITA | 0 0 0 0 860 | 860
84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066
----------------------+-------------------------------------------------------+----------
Total | 46,172 5,658 4,533 3,092 4,801 | 64,256
Mode in the CSES Module 2
MODE OF INTERVIEW | Freq. Percent Cum.
----------------------------------------+-----------------------------------
1. FACE-TO-FACE | 46,172 71.86 71.86
2. TELEPHONE | 5,658 8.81 80.66
3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72
4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53
5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00
----------------------------------------+-----------------------------------
Total | 64,256 100.00
(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total
----------------------+-------------------------------------------------------+----------
00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116
03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769
05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225
07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514
10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482
12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674
15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200
15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022
15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823
20302002. CZECH REPUB | 948 0 0 0 0 | 948
20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026
24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196
25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000
27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000
27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023
34402004. HONG KONG ( | 582 0 0 0 0 | 582
34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200
35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446
37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367
37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212
38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439
39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977
41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500
41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000
48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991
52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574
55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741
57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052
60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032
60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200
61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794
62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303
62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801
64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913
64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496
70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002
72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212
75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060
75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418
82602005. GREAT BRITA | 0 0 0 0 860 | 860
84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066
----------------------+-------------------------------------------------------+----------
Total | 46,172 5,658 4,533 3,092 4,801 | 64,256
Mode in the CSES Module 2
MODE OF INTERVIEW | Freq. Percent Cum.
----------------------------------------+-----------------------------------
1. FACE-TO-FACE | 46,172 71.86 71.86
2. TELEPHONE | 5,658 8.81 80.66
3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72
4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53
5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00
----------------------------------------+-----------------------------------
Total | 64,256 100.00
(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total
----------------------+-------------------------------------------------------+----------
00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116
03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769
05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225
07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514
10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482
12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674
15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200
15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022
15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823
20302002. CZECH REPUB | 948 0 0 0 0 | 948
20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026
24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196
25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000
27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000
27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023
34402004. HONG KONG ( | 582 0 0 0 0 | 582
34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200
35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446
37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367
37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212
38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439
39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977
41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500
41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000
48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991
52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574
55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741
57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052
60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032
60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200
61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794
62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303
62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801
64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913
64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496
70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002
72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212
75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060
75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418
82602005. GREAT BRITA | 0 0 0 0 860 | 860
84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066
----------------------+-------------------------------------------------------+----------
Total | 46,172 5,658 4,533 3,092 4,801 | 64,256
Mode in the CSES Module 2
MODE OF INTERVIEW | Freq. Percent Cum.
----------------------------------------+-----------------------------------
1. FACE-TO-FACE | 46,172 71.86 71.86
2. TELEPHONE | 5,658 8.81 80.66
3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72
4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53
5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00
----------------------------------------+-----------------------------------
Total | 64,256 100.00
(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total
----------------------+-------------------------------------------------------+----------
00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116
03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769
05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225
07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514
10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482
12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674
15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200
15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022
15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823
20302002. CZECH REPUB | 948 0 0 0 0 | 948
20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026
24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196
25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000
27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000
27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023
34402004. HONG KONG ( | 582 0 0 0 0 | 582
34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200
35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446
37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367
37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212
38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439
39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977
41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500
41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000
48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991
52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574
55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741
57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052
60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032
60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200
61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794
62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303
62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801
64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913
64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496
70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002
72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212
75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060
75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418
82602005. GREAT BRITA | 0 0 0 0 860 | 860
84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066
----------------------+-------------------------------------------------------+----------
Total | 46,172 5,658 4,533 3,092 4,801 | 64,256
Satisfaction with democracy N = 48662 SD = 0.27
Mode (base is FTF) No controls + demographics + country fe
Phone 0.01 (0.004) -0.00 (0.004) 0.09 (0.016)
Self - mail 0.08 (0.006) 0.02 (0.006) 0.17 (0.009)
Self + phone 0.12 (0.005) 0.10 (0.005) 0.21 (0.010)
Self + FTF 0.12 (0.004) 0.11 (0.004) 0.15 (0.011)
Entries are coefficients from OLS. Standard errors in parentheses. Source: CSES
Persuade others N = 52751 pr(1) = 0.21
Mode (base is FTF) No controls + demographics + country fe
Phone 1.75 (0.06) 1.62(0.06) 2.87 (0.47)
Self - mail 16.06 (0.78) 18.89 (0.95) 15.20 (1.64)
Self + phone 3.33 (0.13) 3.67 (0.15) 0.36 (0.05)
Self + FTF 0.71 (0.03) 0.79 (0.04) 0.28 (0.04)
Entries are odds ratios from logistic regression. Standard errors in parentheses. Source: CSES
A Plea for Information “Administrative” in general, including for nonresponding
units facilitation of estimating unit selection effects
Care in coding of mode what are the critical features?
intrinsic sampling
primary sampling unit what is the true power of the sample?