Asking Survey Questionsin Many Tongues

Paper presented at theESS Launch Conference

Brussels, November 25-26, 2003

Janet A Harkness, ZUMAGermany

Overview

• English as survey lingua franca• The ESS translation context• ESS requirements and procedures• Common survey practice

and where the problems lie• The ESS and lessons learned

English as survey lingua franca

• We can all speak English• Source questionnaires often in English• Texts anchored to language• Language-driven challenges (two way)

The ESS Context: Facts and Figures• Round One fielded in 22 countries• Fielded in 20-21 languages, depending on how you count• All languages used as first by 5% and

more

• 5 countries fielded in more than one language (12 languages)

• Most often shared languages (to date): French Italian German.

• 9 countries shared one or more languages

The WP Context: Resources & Researchers

• ESF• EC and Partners• CCT • Scientific Advisory

Board• Participating

countries• Translation

Workpackage

Translation Expert Panel Dr. Hans Hönig,

Dr. Paul Kussmaul,

Beth-Ellen Pennell, MA,

Dr. Alisú Schoua-Glusberg

Key Workpackage Component

TRAPD

A framework for ESS translation and assessment to deal with the complex requirements

Translation framework TRAPD

Translation, Review, Adjudication, Pre-testing and Documentation

5 interrelated processes with protocols involved in producing a final translated version

TRAPD combines translation with assessment

Key Activities for Round One

• Develop framework for ESS translation and assessment: TRAPD

• Overview report on TRAPD

• Guidelines for NCs on translation and assessment procedures, team members ‘ roles.

• Strategies for languages shared across countries

• Documentation framework and guidelines

Additional practical measures

• Basic guidelines on organising translation and assessment, selecting & training translators

• Documentation template for problems, divergences, sharing harmonisation

• Source questionnaire annotated to help translators

• Consultation hotline

Ongoing work--Round One into Round Two

• Probe interviews with (external) translators on the source questionnaire

• Analysis of (external) translators‘ appraisals of selected questions from ESS translations

• Comparisons of selected translations for shared languages

In Sum

Funds, Guidelines, Specifications, Requirements, Illustrations and Examples Templates, Annotations, Hot-line, Presentations and Reports

Ongoing: cognitive-based work on the source questionnaires and translations

Isn’t this an unusual amount of effort?

Aren’t questionnaires easy to translate?

Questionnaires at first glance

• Easy questions, simple wording• Response categories often one or two

words• Simple and repeated instructions• Repetition of lexical and structural

elements throughout • Similar or identical questions in other

surveysAnd the work team often knows English

Questionnaires at second glance

• Questionnaires are complex texts•They look easier than they are• Good questions not easy to construct• Measurement-in-text poses special problems• Cross-disciplinary requirements (e.g., measurement, translation)• Interdisciplinary expertise neededA key problem: resulting challenges under-estimated

A Complex Text Type

• Text-with-blanks genre• Often largely generic • Texts within the text• Low Context- jump from one topic to

another• Cross-national aims encourage general

formulations• Computer applications can technically

complex source formats

One example of not-so-easy features

• Measurement properties lead to ‘surveyspeak’ and ‘scalespeak’

Surveyspeak and Scalespeak are textual traces of measurement(Harkness 1996)

Surveyspeak

Q: All things considered, how satisfied are you with your life as a whole nowadays ? (ESS, B13)

Q: How‘s life?

Q: When did you last visit a dentist?

Would you say that you are //very certain, somewhat certain, neither certain nor uncertain, somewhat uncertain or very uncertain// about the date you just gave?

A:

Conversational Follow-up: Uh-HuhReally ?Are you sure?

Scalespeak -- response categories

How important or unimportant is ----

Extremely ImportantSomewhat importantNeither important/or unimportantSomewhat unimportantExtremely unimportant

Scalespeak

Extremely unimportant (ESS E13)

Somewhat importantSomewhat unimportant

Somewhat difficult

Somewhat ugly

Response scales are difficult to translate

Differences across languages/ cultures in• quantification structures & potential negation

• saliency of concept (satisfaction, justness) • disclosure norms• response styles• numeracy and related notions• scaling norms (1-6; 1-10?)

Known Measurement Effects

vs. Allow/Not allow

Allow/ Forbid

So what about differences such as:

vs.Agree/

Not agree

Agree/ Disagree

Agree/Reject

or

Translations may not be able to match English scale properties

German: ‘Agree’ / ‘Not agree’/ ‘Reject’

English : Agree/ Not agree / Disagree

Despite these complexitiesviews on translationtranslatorsassessment and testingare often pre-theoreticaland procedures followed not grounded in research

Not by chance that the ESS work package focused on Survey translation process procedures

Specifications, briefing, & training materials for translators and for those hiring translators

Survey translation evaluation procedures

Providing access to know-how to implement translation components

These are what the survey community in general still lacks

Thus the ESS effort and activities meet real needs

It is a necessary amount of effort

In addition

• No design costs, only translation and assessment • Core questionnaire designed for replication

• Even ESS translations (in a deluxe version) are a very small part of total national survey costs.

Lessons learned from:

Contact with national teams

Documentation from national teams

Discussions within the Expert Panel and CCT

Cognitive interviews with external translators

Related work in other projects(eg SHARE, EUYOUPART, ISSP)

Some lessons learned

1. Procedures and guidelines crucial Team work Iterative translation and review Sharing and harmonisation

2. More hands-on input would help more On the job training Training opportunities and materials

Lessons learned continued

3. Pretesting –TRApD –more needed on source questionnaires

4. Version documentation is essential – but the Ugly Duckling of the work package. Need to help it have its proper place in a user-friendly form

5. Version harmonization raises scheduling issues. No harmonization discussion raises comparability issues

6. More intensive cross-cultural input neededQuestionnaire contentQuestionnaire formulations

[Raises planning, timing and procedural questions]

Thank you very much for your time and attention