Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim

43
Using Probing Techniques to Assess Intercultural Validity The Case of Attitudes Towards Immigrants Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim

description

Using Probing Techniques to Assess Intercultural Validity The Case of Attitudes Towards Immigrants. Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim. Overview. Establishing comparability across countries Conventional use of cognitive techniques - PowerPoint PPT Presentation

Transcript of Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim

Using Probing Techniques to Assess Intercultural Validity

The Case of Attitudes Towards Immigrants

Michael Braun & Dorothée Behr

GESIS – Leibniz Institute for the Social Sciences,

Mannheim

Overview

Establishing comparability across countries Conventional use of cognitive techniques Problems and alternatives Methods and data Results Conclusions

Establishing comparability across countries

Establishing comparability across countries

• In intercultural research, measurement invariance has to be established across countries

• Application of data-analytic procedures does not get – or at least not always – at causes of incomparability (possible exception: ML-SEM)

• Possible solution: cognitive techniques (category-selection probing, comprehension probing, specific probes)

Conventional use of cognitive techniques

Conventional use of cognitive techniques

• as a pretesting device to detect bad items and improve a questionnaire

• mainly in a cognitive laboratory, i.e. not under regular field conditions

• respondents are briefed about the setting as being a pretest and their task being to help find (comprehension) problems

• often “professional” pretest respondents are sought who might develop lay theories

Conventional use of cognitive techniques

• normally conducted by interviewers proactively searching for hidden comprehension problems

• mostly applied to behavioral questions

• traditionally based on small quota samples (often not more than 20 interviews)

• low number of cases does not permit analysis of diverging argumentation patterns, shared only by a small proportion in the population, and quantification of results is impossible

Conventional use in intercultural studies

• mainly aiming at the improvement of translations

• often application to ethnic groups in only one country

• cognitive interviewing across countries very difficult to organize (availability of cognitive laboratories; standardization of procedures)

• different house styles in recruiting respondents and guidelines specifying interviewer behavior

Problems and alternatives

Problems and an alternative

• personal interviews even counterproductive (standardization of interviewer qualification and behavior hard to achieve)

• time requirements of implementation by face-to-face interviews in the laboratory

• no compelling reason (other than costs under traditional implementation) for restrictions

• application of cognitive interviewing in the field as independent part of the research process

Advantages of using internet surveys

• allow for a higher degree of anonymity, improving data quality

• reduction of respondent burden

• online access panels as cost-efficient means to increase sample size and to investigate small subgroups with diverging response behavior

Our project

“Enhancing the Validity of Intercultural Comparative Surveys. The Use of Supplemental

Probing Techniques in Internet Surveys”

Funded by the German Science Foundation (DFG)applicants: Michael Braun, Wolfgang Bandilla &

Lars Kaczmirek

Project goal

Detection of interpretation differences in international surveys by cognitive methods (category-selection, comprehension and specific probing).

Demonstrating the feasibility to implement cognitive methods in non-probability online access panels.

Testing the approach with substantive survey topics, in the present case immigration items from the International Social Survey Program (ISSP) 2003 National Identity module.

Methods and data

There are different opinions about immigrants from other countries living in [country]. (By “immigrants” we mean people who come to settle in [country]) How much do you agree or disagree with each of the following statements?

a. (crime) Immigrants increase crime rates.

b. (economy) Immigrants are generally good for [country’s] economy.

c. (jobs) Immigrants take jobs away from people who were born in [country].

d. (culture) Immigrants improve society by bringing in new ideas and cultures.

[5-point Likert scale from agree strongly to disagree strongly; crime and jobs reverse coded, such that high values = negative attitudes.]

Study design and implementation

Non-probability online access panels in 7 countries/regions (ca. 500 cases each).

Canada, Denmark, United States, (western and eastern) Germany, Hungary, Spain.

Target population defined as country nationals aged btw. 18-65.

Quotas set according to gender, education, age. Data collection period January 2011.

Study design and implementation

Specific probing: “Which type of immigrants were you thinking of when you answered the question? The previous statement was: #.”

4 immigrant items were rotated using 4 splits, only the first item was probed

Challenges

Can we reproduce the answer patterns found in the ISSP data for the different countries?

Do answers to the specific probes reveal different patterns across countries?

Results

Attitudes towards immigrants

Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM

Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6

Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7

United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7

W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6

E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7

Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8

Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7

Attitudes towards immigrants

Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM

Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6

Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7

United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7

W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6

E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7

Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8

Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7

Attitudes towards immigrants

Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM

Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6

Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7

United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7

W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6

E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7

Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8

Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7

Attitudes towards immigrants

Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM

Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6

Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7

United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7

W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6

E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7

Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8

Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7

Attitudes towards immigrants

Country Crime Economy Jobs CultureISSP CICOM ISSP CICOM ISSP CICOM ISSP CICOM

Canada 2.8 3.0 2.4 2.6 2.7 3.1 2.3 2.6

Denmark 3.9 3.8 3.3 3.1 2.2 2.5 2.6 2.7

United States 2.8 3.1 2.8 2.9 3.1 3.4 2.5 2.7

W. Germany 3.6 3.5 3.1 2.9 3.1 2.8 2.6 2.6

E. Germany 3.7 3.5 3.3 3.0 3.5 3.0 2.8 2.7

Hungary 3.8 2.7 3.5 3.2 3.6 3.0 3.2 2.8

Spain 3.4 3.6 2.6 3.0 3.0 3.2 2.5 2.7

CICOM compared to ISSP

ISSP data are neatly reproduced in the CICOM data in most cases.

As for deviations, it is not clear whether our sample or social change is the reason (ISSP data were collected in 2003).

It is particularly the Hungarians which seem to be less xenophobic in CICOM than ISSP

Answer patterns across countries

Probing question not answered Answers to probing not pertinent to question, not

categorizable answers (“other” category) Reference to immigrants in general, no specific group Reference to specific ethnicities, and to which Reference to immigrant groups defined by either the

positive or negative behavior referred to in the item Reference to the legal/illegal immigrant distinction

Results from specific probing: no answer and “other”

No answer Other

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

11.213.811.311.815.512.59.7

23.328.116.020.920.532.712.1

“other“: partly interpretation as category-selection probing

Results from specific probing:Percentages for major answer types

Immigrants in general

Specific ethnicities

Defined by pos/neg pole

Legal-illegal distinction

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

32.629.621.026.227.917.233.1

18.218.825.228.623.732.338.3

6.4 5.64.1 5.61.3 1.37.9 4.55.6 6.52.1 2.61.7 2.4

2.70.0

23.90.00.40.82.8

Results from specific probing:percentages of major specific ethnicities

Islamic countries

Eastern Europe

Asia Latin America

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

5.214.02.3

20.917.08.4

17.7

0.04.50.4

13.813.014.012.8

14.92.24.63.83.4

19.43.4

3.50.2

23.10.00.00.2

20.6

Probing and reality

This reflects nicely immigration reality in most of these countries:

Canada: really multi-ethnic, many Asians … Denmark: who knows this country??? United States: though also multi-ethnic, Latino migration

dominant West (and East) Germany: Turks first, Russians (but

perhaps partly ethnic Germans referred to as Russians), Latin Americans unfortunately missing

Probing and reality

Spain: Romanians as biggest single migrant group, followed by Marroccans; Latin Americans lumped together is biggest group (but in many cases not distinguishable from natives – or even originally Spanish remigrants from southern America, e.g. from Argentina)

But: What the hell have the Chinese to do with Hungary? Where are they?

Magyar Chinatown in Budapest

Further results

Immigrants from EU-15 and from Subsaharan Africa are also mentioned, but not as frequently.

Sinti and Roma are rarely mentioned, and mostly restricted to Hungary.

Other groups are still less often mentioned and are put in the “other“ category in our coding schema.

Further results

Very small country differences in xenophobia. After inclusion of the probing codes no country

differences left. Unsurprisingly, thinking of immigrants defined by

the positive pole reduces and by the negative pole increases xenophobia.

Results from specific probing:general attitudes towards immigrants dependent on

thinking of the major specific ethnicities

Overall mean

No spec. group

Islamic countries

Eastern Europe

Asia Latin America

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

11.212.112.211.712.211.712.3

10.610.911.510.611.711.512.4

13.613.913.813.113.711.113.2

a13.2a

13.513.812.212.8

12.211.611.112.910.711.011.3

11.5a

13.0aaa

12.3

Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

Results from specific probing:general attitudes towards immigrants dependent on

thinking of the major specific ethnicities

Overall mean

No spec. group

Islamic countries

Eastern Europe

Asia Latin America

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

11.212.112.211.712.211.712.3

10.610.911.510.611.711.512.4

13.613.913.813.113.711.113.2

a13.2a

13.513.812.212.8

12.211.611.112.910.711.011.3

11.5a

13.0aaa

12.3

Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

Results from specific probing:general attitudes towards immigrants dependent on

thinking of the major specific ethnicities

Overall mean

No spec. group

Islamic countries

Eastern Europe

Asia Latin America

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

11.212.112.211.712.211.712.3

10.610.911.510.611.711.512.4

13.613.913.813.113.711.113.2

a13.2a

13.513.812.212.8

12.211.611.112.910.711.011.3

11.5a

13.0aaa

12.3

Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

Results from specific probing:general attitudes towards immigrants dependent on

thinking of the major specific ethnicities

Overall mean

No spec. group

Islamic countries

Eastern Europe

Asia Latin America

CanadaDenmarkUnited StatesW. GermanyE. GermanyHungarySpain

11.212.112.211.712.211.712.3

10.610.911.510.611.711.512.4

13.613.913.813.113.711.113.2

a13.2a

13.513.812.212.8

12.211.611.112.910.711.011.3

11.5a

13.0aaa

12.3

Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

Further results

In general: Thinking of immigrants in general is connected to less

xenophobia. Thinking of immigrants from Islamic and Eastern

European countries is connected to higher xenophobia. Thinking of immigrants from Asia is related to less

xenophobia. Thinking of Latin Americans does not show a clear

tendency (in the US: negative).

Further results

Differences between issues and specific immigrant groups mentioned are not really dramatic

Islamic countries are mentioned most frequently with crime item (15.4%) and least with economy item (9%)

The same for eastern Europeans: 14.3% in connection with crime, 4.8% in connection with the economy

There are no relevant issue-specific differences for Asians and Latin Americans

Conclusions

Some Conclusions

The reasoning in the seven countries/regions is highly similar on an abstract level, though the concrete groups of immigrants mentioned differ.

Respondents with positive beliefs tend to use inclusive statements (e.g. they refer to all immigrants), respondents with negative focus more on selective groups.

In the American context, the legal status of immigrants is mentioned more often than in the European context.

Latin Americans are clearly not equivalent in the Spanish and US-American contexts.

Thank you!