THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT) TBLT Honolulu – September 2007 Katholieke...
18
THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT) TBLT Honolulu – September 2007 Katholieke Universiteit Leuven Belgium Universite it van Amsterdam The Netherland s [email protected] Language testing and culture
-
Upload
gabriel-brooks -
Category
Documents
-
view
214 -
download
1
Transcript of THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT) TBLT Honolulu – September 2007 Katholieke...
THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT)
TBLT Honolulu – September 2007
Katholieke Universiteit LeuvenBelgium
Universiteit van Amsterdam
The Netherlands
Language testing and culture
Do you congratulate someone
with or on their birthday?
Question …Question …
Neither! It’s a trick question. We don’t use ‘to congratulate’ or ‘congratulations’ in connection with birthdays in English, except in very special cases (18, 21, 100, etc.): we normally just say ‘Happy Birthday’ or ‘Many happy returns’, instead. And we don’t shake hands as we do so.
However, we can congratulate people on getting a job, winning an award, or getting married, for instance. And then a handshake is often part of the ritual. ‘With’ is never used with ‘to congratulate’ or ‘congratulations’ – it’s always ‘to congratulate (someone) on’… and ‘congratulations on’ ..
Answer …Answer …
The Certificate of Dutch as a The Certificate of Dutch as a Foreign LanguageForeign Language
- CNaVT (Certificaat Nederlands als Vreemde Taal)
- administration in more than 40 countries around the world: from Madrid to Jakarta and from Warsaw to Rio de Janeiro
- 2500 candidates each year
Assess as direct as possibleAssess as direct as possible
- construct on the basis of needs analysis
- authenticity: 6 exams represent 6 different TLU domains
- task-based: exam tasks represent real life language use situations
Valid assessmentValid assessment
Uiterwijk & Vallen (Language Testing 2005, 22)
(…) the question is: does the test measure a particular construct for the different groups in a valid way? In other words, does it measure, for various subgroups, what is claims to measure?
The effect of cultureThe effect of culture
Marquardt & Gillam (Language Testing 1999, 16)
The most frequent problem related to misinterpretation of assessment results is related to cultural (…) differences between test items, examiners and their (…) candidates.
Fairness to candidatesFairness to candidates
McNamara & Roever (Language Testing. The Social Dimension)
An interpretative argument is only as strong as its weakest link. It is the social responsibility of testers to report efforts to estimate the impact of these factors and to control them.
Item bias analysisItem bias analysis
- statistical procedures to find out if an item is shows DIF
- research the factors responsible for the DIF
- determine if the cause of DIF is construct (ir)relevant, leading to bias
CNaVT item bias researchCNaVT item bias research
- PTIT exam 2007
- 130 items, spread over 2 versions, 9 tasks per version
- Facet-analysis: facet ‘country’
- Mantel-Haentzel analysis
CNaVT item bias researchCNaVT item bias research
Facets
- empirically 1324 bias terms item * country
- 68 interactions showed DIF
- 56 disadvantageous to subgroup
- 12 advantageous to subgroup
CNaVT item bias researchCNaVT item bias research
CNaVT item bias researchCNaVT item bias research
Example 1
Country: Indonesia
Item: Question in form
‘What sort of object have you lost?’
CNaVT item bias researchCNaVT item bias research
Example 2
Country: Russia
Item: Question from friend in letter
(…) ‘But I often think back with nostalgia about my holiday in your country. (…) And you? What did you do during your holidays? Did you travel? Where to? Or did you stay home and what did you do?’ (…)
Our issues concerning DIF analysisOur issues concerning DIF analysis
- Categorisation: on the basis of nationality, region? No strict boundaries. Or: what is ‘culture’?
- Determination that culture is the causing factor of variance: so many intervening factors (for example: socio-economical status)
- Determination that DIF is part of the construct > tension
What we doWhat we do
- awareness of our construct: what (concerning culture) is relevant, what is irrelevant
- generate input on possible cultural bias from different sources in different stadia
- In case of suspicion of irrelevance: DIF analysis
To concludeTo conclude
More research
Further define our construct
Communication to stakeholders
Fairness to candidates!!!
THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT)
TBLT Honolulu – September 2007
Katholieke Universiteit Leuven
Universiteit van Amsterdam
www.cnavt.org
