Surveying Rare Subpopulations:

A Comparison of Census-Based Frame, Location and Snowball Sampling

Johan A. MistiaenThe World Bank

(DECDG)

Probability Sampling: every element in the study population has a known, nonzero probability of inclusion in the sample. This enable use to compute linear unbiased estimates of population parameters from the sample and standard errors (if second-order inclusion probabilities are known).

Sampling frame: a list from which the probability of selection of every element in the study population can be computed.

What if a sampling frame is not available (or too costly to construct)?

Recap context

Case Study: The Brazil Nikkei Household Survey

MOTIVATION

Increasing attention by researcher and policy makers to migration/remittances

But few detailed and representative surveys of households of migrants exist

Many specialized surveys of migrants are non-probability samples, making it hard to generalize from them

=> We conducted an experiment in Brazil to compare three different survey methods

Context

World Bank survey of Brazilians of Japanese descent (Nikkei)

Approximately 1.2-1.9 million of them among Brazil’s 170 million population

Japanese law allows third-generation and below unrestricted access to labor market

Estimated 265,000 Brazilian migrants in Japan, sending approximately $US2billion in remittances annually.

Survey methods used in our study are equally applicable to attempts to survey migrants in destination countries.

Surveying Methods

In most countries, surveying migrants is survey of rare element

Three approaches used in practice:– Stratified sampling using population census

– Chain-referral methods such as snowball sampling

– Sampling at locations where immigrants tend to cluster. Time and space/intercept sampling.

=> Question as to how well other methods perform compared to random sample.

Our experiment

Compare three different methods in Brazil

Use the same questions and same survey firm

Wish to compare characteristics of households with and without migrants, estimate proportion of households receiving remittances and with migrants in Japan

Context: population predominately urban; crime a concern

=> Communications with community organizations, use where possible of Nikkei interviewers.

Method 1: Stratified random sample

Restrict to Sao Paulo and Parana provinces where 80% estimated to live

(still 47 million people)

Use 2000 Brazilian Census to classify households as Nikkei or not (approximately)

Census doesn’t ask ethnicity, instead asks: race (yellow), place of birth, whether individual has lived elsewhere in last 10 years

Second complication: only 10% long form of census asks this.

Method 1: Stratified random sample

Steps:

– Select 75 census tracts, with stratified random sampling

– Carry out door-to-door listing of households in these census tracts

– Return and administer questionnaire to households identified as Nikkei

– Follow-up round to re-try and interview those who refused on first set of interviews, use shorter interview if refuse longer one.

Map of Nikkei Target Clusters in the State of Sao Paulo

Listing

Letters sent out to 150 Nikkei associations with bases in the areas chosen, to explain survey

Census tract averages 301 houses

42 interviewers in Sao Paulo and 24 in Parana used for 50 and 25 tracts.

Three attempts made to interview household; proxy-reporting used if no one at home.

Summary of listing

Listed 22, 539 dwellings Among these, detected 839 Nikkei households –

528 interviewed in person, 311 by proxy-reporting Initial phase of interviewing interviewed 247 Nikkei

households Returned and interviewed another 156 (45 long

interview, 111 short interview) Found some of those identified as Nikkei by proxy

were not Estimate in the end:

– Parana: 76% interviewed, 12% refuse, 10% absent– Sao Paulo: 45% interviewed, 31% refuse, 17% absent

Comparing households in 2 phases

Main difference is that households that required the additional effort to survey are much more likely to refuse to say how much they receive in remittances.

Look similar in many other respects to those who answered first time.

Snowball survey

Contacted 25 Nikkei organizations and asked each to provide 3 names as seeds.

Targeted sample of 300 households

Received 67 seed names

Two main problems:– Some of seed households refused

– Many households refused to provide referrals

In the end have sample of 100 households, 60 seed households, 40 referrals.

Intercept survey

Consulted with local researchers, Nikkei organizations, and Sudameris officers to select broad range of locations which Nikkei community frequents

Chose 9 fixed points, and 6 events– Sports club, metro station in Liberdade neighborhood, two

Feiras (Sunday marketplaces), hospital, grocery stores, language school, outside branch of bank.

– Japanese film event, large cultural festivals, Japanese food festival, Japanese art exposition, Christmas concert and music festival.

Short questionnaire – 62 questions, 7 minutes.

Intercept survey

At each location, 2 interviewers used, one to count number passing through location, one to interview

Interviewers there for 129 total hours over 2 weeks (~8.5 hours/location).

Each person asked how often in past 2 weeks had frequented any of the other locations – used to reweight answers.

Only 19% had visited only one location – on average individuals had visited 3.18 out of 15 locations during the two week period.

Weight:

15

1

Pr,1minj

i jatsampledsindividualofFractionjvisitsip

Table 4: Sao Paulo Intercept Survey: December 9-20, 2006

Number of Number of Refusal Time spent Approximate

Intercept Point interviews refusals rate (%) in location number in location

Fixed point locations

Coopercotia Atlético Clube 34 23 40.4 8.5 hours 368

Estação Metrô São Joaquim 49 37 43.0 14 hours 1436

Feira da Liberdade 34 3 8.1 5 hours 1282

Feira Livre da Rua Carneiro 34 3 8.1 7 hours 1635

Hospital Santa Cruz 42 12 22.2 8 hours 374

Mercearia Marukai 54 76 58.5 13 hours 2583

Mercearia Satsuyama 36 69 65.7 11 hours 1922

Sociedad Brasileira de Cultura Japonesa-Bunkyo 34 25 42.4 9 hours 311

Agencia Sudameris 34 24 41.4 8 hours 186

Events

Cinema Bunkyo 34 19 35.8 4 hours 97

ACAL Toyo Matsuri - Festival Oriental 30 22 42.3 9 hours 824

ACESA Motitsuki Matsuri (Festival Gastronômico) 29 1 3.3 8 hours 424

Fundação Mokiti Okada -Exposição de Obras de Arte 4 2 33.3 9 hours 67

Coral Bunkyo - Concerto de Natal 34 3 8.1 4 hours 704

Grupo the Friends-Koohaku Utagassen 2006 (Festival musical) 34 6 15.0 11 hours 1731

516 325 38.6 128.5 hours 13944

Fourth method?

Use phone book to select individuals with minority names? (Osili with Nigerians in Chicago)

90% have landline phone (most of rest have cellphones)

With intermarriage, 74.8% have recognizably Japanese surname

Only 24% appear in the phone book Only 20% in phone book with Japanese

name!

Results: Connection to Japan

Table 11: Comparison of Characteristics of Nikkei Across Different Sampling Methods

Stratified Unweighted Weighted

Survey Intercept Intercept Snowball

Household Characteristics

Household Size 3.25 3.29 3.45 3.66*

Percentage of households with member who:

Reads Japanese/Nikkei newspapers 13 39*** 28*** 45***, +++

Watches Japanese/Nikkei TV programs 23 43*** 36** 54***, +++

Reads Japanese/Nikkei books/magazines 15 42*** 30*** 52***, +++

Reads newspapers from Nikkei associations 12 39*** 28*** 63***, +++

Checks Japanese/Nikkei websites on the internet 5 24*** 18*** 25***

Sample Size (households) 204 516 516 100

Sao Paulo

Results: Migration and Remittances

Table 12: Comparison of Migration and Remittances of Nikkei Across Different Sampling Methods

Stratified Unweighted Weighted

Survey Intercept Intercept Snowball

% of households which:

Migration

Have a Member currently in Japan 19 35*** 33** 30

Have a Member who has returned

from work/study in Japan 41 65*** 59*** 51

Remittances

Receive remittances from Japan 14 10 10 10

Refuse to say if they receive remittances 3 2 3 7

Refuse to say how much they receive if receiving 31 73*** 82*** 50+

Sample Size 204 516 516 100

Sao Paulo

Remittances

Proportion refusing to report how much they receive is much higher (82%) in intercept survey than in household survey (31%).

Fear of crime likely to lead to more reticence in public places.

Comparison of costs

Total cost per household interviewed:– Stratified survey: US$212 ($2 per dwelling listed;

$80 per interview)– Snowball survey: US$100– Intercept survey: US$30

Recall survey lengths differ:– Stratified and Snowball: 36-page questionnaire,

1 hour to complete– Intercept: 3-page questionnaire, 7 minutes to

complete

Conclusions

Snowball and Intercept methods oversample individuals more closely connected to community

Intercept method does seem to give results closer to random sample when reweight data – suggests multiple location sampling important

Snowball, often seen as cheapest and easiest, may be just as difficult to carry out in some applications.

Conclusions

Choice of survey method does make large and significant differences to estimates of means of interest, and to regression analysis.

Cost considerations may suggest:

– Intercept survey for exploratory analysis, or where target population attends community locations

– Stratified survey for representative analysis.