This article was downloaded by: [University of Birmingham]On: 04 October 2014, At: 07:35Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

Studies in Higher EducationPublication details, including instructions for authorsand subscription information:http://www.tandfonline.com/loi/cshe20

Cultural Equivalence inthe Assessment of Homeand International BusinessManagement Students: A UKexploratory studyGlauco De VitaPublished online: 25 Aug 2010.

To cite this article: Glauco De Vita (2002) Cultural Equivalence in the Assessment ofHome and International Business Management Students: A UK exploratory study, Studiesin Higher Education, 27:2, 221-231, DOI: 10.1080/03075070220120038

To link to this article: http://dx.doi.org/10.1080/03075070220120038

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, orsuitability for any purpose of the Content. Any opinions and views expressedin this publication are the opinions and views of the authors, and are not theviews of or endorsed by Taylor & Francis. The accuracy of the Content shouldnot be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions,claims, proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly in connectionwith, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly

forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

Studies in Higher Education Volume 27, No. 2, 2002

Cultural Equivalence in theAssessment of Home andInternational Business ManagementStudents: a UK exploratory studyGLAUCO DE VITAOxford Brookes University, UK

ABSTRACT This article explores the cultural equivalence properties of commonly adopted assessmentmethods by � rst subjecting multiple-choice tests, coursework assignments and essay response examina-tions to critical scrutiny for evidence of bias. Then, using data from a large cohort enrolled on a� rst-year business studies programme in a UK university, a comparative analysis of the academicperformance of home and international students reveals considerable differences, with the marks of thelatter being substantially lower than those of the former. In order to establish whether a particularassessment method is culturally biased compared to others, following ordinary least squares estimationof regressions on each assessment method for the home and international student subsamples of thecohort, a Chow test is employed. The main � nding indicates that assessment by examination penalisesinternational students beyond differences in ability levels, as measured by multiple-choice test andcoursework assignment scores. On the basis of this evidence, it is suggested that, in culturally mixedclasses, the exclusive adoption of the end-of-course examination be avoided.

Introduction

With the internationalisation of higher education, UK universities have become increasinglydiverse in their student populations. Figures collected by the Higher Education StatisticsAgency (HESA, 1996, 2000) show that, over the 1994–95/1998–99 period, the internationalstudent population in higher education in the UK grew by 34%. This growth has beenparticularly pronounced in business and management subjects (the context of this study)where, in the year ending 31 July 1999, over 17% of the 137,727 students enrolled (HESA,2000) were ‘non-UK’ (i.e. students from other European Union [EU] countries or fromoverseas).

The increased presence of international students has raised new pedagogic questionssuch as whether the way we currently teach can effectively cater for the learning needs of bothhome and international students, and whether special skills in teaching across cultures needto be developed. While many gaps in this literature are quickly being � lled in, with muchwork emerging on how institutions can manage this diversity and ensure quality learning forall students (Tompson & Tompson, 1996; Ladd & Ruby Jr, 1999; Campbell, 2000; De Vita,2000), little research has been undertaken in the area of assessment of learning in multi-cultural educational settings and, to the author’s knowledge, none has been done in relation

ISSN 0307-5079 print; ISSN 1470-174X online/02/020221-11 Ó 2002 Society for Research into Higher EducationDOI: 10.1080/030750702201200 38

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

222 G. De Vita

to the comparative performance of home and international students across the variousassessment techniques commonly employed in business and management degrees. This isstriking in that such information may serve as a useful indicator to monitor the appropriate-ness of current entry requirements for international students, and to make inferences on theextent to which adopted assessment methods are culturally fair indicators of ability acrossdiverse student groups. Yet, although cultural fairness constitutes a primary conceptualconsideration in assessment, the extent to which the task and the assessment situation aredesigned to afford all students the opportunity to demonstrate what they know, regardless ofdifferences in cultural, ethnic or linguistic backgrounds (cultural equivalence), continues tobe, with few exceptions (e.g. Lauvas et al., 2000), a neglected issue in the literature on theselection of assessment methods.

This article represents an initial attempt to address these issues, within the context ofbusiness management education. It begins by subjecting multiple-choice tests (MC), course-work assignments (CW) and end-of-course examinations (EX) to critical scrutiny for evi-dence of cultural bias. This section will make the most of the fairly scarce relevant literatureand existing anecdotal evidence, and offer new insights stemming from the critical re� ectionsof the author. Here, any conclusions reached will be judicious in nature, and should beviewed cautiously since, ultimately, the measurement of cultural fairness in assessment is anempirical issue.

Following some methodological considerations, the relationship between home andinternational students’ performance (measured by the actual marks awarded to students) willbe inspected, using data from a large � rst-year cohort enrolled in a business studies degreeprogramme of a UK university.

A � nal empirical contribution will speci� cally test for differences in regressions on MC,CW and EX across the home and international student subsamples of the cohort to establishwhether each of these assessment methods is a culturally fair indicator of ‘ability’ (in the senseof achievement of learning outcomes), relative to the others, across the two groups.

A Critical Review of the Three Assessment Methods

Under ever greater budget constraints, it should come as no surprise that the multiple-choicetest, by virtue of ease and accuracy of computer marking, is seen as the most ef� cient meansfor assessing large classes in business and management degrees. Though often criticised formeasuring merely recognition skills, for turning courses into ‘memory Olympics’, and thus,for punishing � rst-class minds, it can be argued that, with a bit of effort, more sophisticatedmultiple-choice questions can be constructed to elicit particular higher-order skills. Ascontended by Wood (1993), for example, when students are required to work out an answerand then search among alternatives for it, cognitive processes that go beyond mere recogni-tion and recall are likely to be activated. Brown et al. (1994) make this point too, and advisethe use of multiple-choice tests even in courses where learning outcomes are associated withhigh levels of analytical ability. It has also been suggested that, being a good predictor ofstudents’ performance in written assignments and closed-book � nal examinations (Au,1997), multiple-choice testing is ‘as … fair as other traditional assessment techniques inexamining students’ (p. 248). Au’s work, however, neglects the critical factor of culturalfairness, that is, the extent to which the information yielded by this particular assessmentmethod may be biased or misleading (compared to others) in favour of particular groups ofstudents.

The very concept of the ‘multiple-choice test’, that only one answer can be right, is nota cross-cultural universal. As reported by Estrin & Nelson-Barber (1995), for example, for

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

Cultural Equivalence in Student Assessment 223

Alaska native students, the idea of choosing only one answer over all others is antithetical totheir way of thinking. And so it should be. After all, in real life, there are seldom questionsfor which there is only one correct answer. The most important questions lend themselves tomany different answers, and for some questions there is no right answer at all. This is alsotrue in application to the business and management discipline, a subject area which hingeson the evaluation of alternative perspectives, the questioning of theory and what is taken forgranted. When viewed as a cultural product standardised on a majority culture, the premiseof ‘one right answer’ is also against the spirit of a multicultural curriculum, one which givesvoice to unconventional or silenced perspectives, and which values the variety of views ofwhat constitutes truth and knowledge.

Greater concerns regarding the administration of multiple-choice tests in culturallydiverse classes emerge when, by locating the discussion in the context of the validitydimensions of assessment (Messick, 1989), we probe the ways in which different individualsrespond to the items of a multiple-choice test. To date, the empirical literature has onlyfocused on gender effects. Factors such as risk-taking behaviour and tolerance for ambiguity,however, being clearly related to key variables in cross-cultural research, are likely toin� uence the performance of culturally diverse students in different ways. As reported byRyan (2000), for example, Chinese students can be disadvantaged by multiple-choice tests‘as they view it improper to guess an answer’ (p. 49). These factors make the still irresoluteissues, concerning scoring systems and instructions to deal with guessing, the selection andordering of questions, number of alternatives, omitted possibilities, alternate reasoning andwording nuances, even more controversial, as they may introduce severe cultural bias. Whilstitem response theory methods (Baker, 1985; Hambleton, et al., 1991), such as differentialitem functioning (Holland & Wainer, 1993), allow for the identi� cation of differences inscores across groups of test takers that are attributable to a variable other than ability (gender,race, ethnicity, etc.), little evidence exists in the literature to suggest that such techniques areregularly employed in business and management courses to detect culturally unfair items, andhence eradicate cultural bias from multiple-choice tests.

Another important consideration related to cultural fairness concerns the issue of timelimits. Some international students may, in fact, be at a relative disadvantage under thisassessment method, as they may need more time to � nish a multiple-choice test than homestudents. Some supporting evidence for this comes from the study with ethnic minoritygroups conducted by Wood (1990), and from the Civil Service Commission (CSC) study byBethell-Fox (1988), who found that ethnic minority candidates experienced increasingdif� culty as the multiple-choice test progressed. Whether such dif� culties stem from less thanadequate reading and comprehension skills in English, from problems in coping with a novelassessment technique, or simply from poor time management is not known. But in acomparative study of Hispanic and Anglo college students, Llabre & Froman (1987) foundthat while Hispanic test takers consistently spent more time than Anglo examinees on the testitems, if they were allowed extra time to complete the test, they would do as well. This raisesthe question of what is being assessed by putting students under time limits. If they achievethe learning outcomes, does it matter that they take a bit longer?

Whether time extensions would actually compensate for limited vocabulary and less thanadequate reading skills, however, remains subject to debate. It is also questionable whetherthe answer lies in treating international students as one of those special groups that, likestudents with disabilities, need more time. The issue of fairness concerning time limitsshould, by de� nition, relate to all students in the class population, and be treated at its roots.The standard operational benchmark for multiple-choice testing is that at least 75% ofstudents who take the test should be able to � nish 95% or more of the questions on the test

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

224 G. De Vita

(Wood, 1993). The problem here is that multiple-choice papers are rarely, if ever, piloted onrepresentative control groups which also include students from non-English speaking back-grounds.

Even greater concerns regarding cultural fairness on assessment arise when looking atclosed-book � nal examinations (either in the form of short-answer or open-ended questions).By requiring students to organise, synthesise and express knowledge at a speed (the studentis typically given a maximum of 2–3 hours to answer 3–4 essay questions), this method maywell be measuring writing skills in English as much as, or even more than, academicknowledge—a distinction that, by introducing the possibility that the content of an examin-ation may be measuring two different constructs (critical thinking and writing) across diversegroups of the population, questions the cross-cultural validity of the construct that theexamination purports to measure.

In comparison with multiple-choice tests, it has also been argued that while students whoare not pro� cient in writing skills (which is the case for many international students) might,with alternative answer options to choose from, be encouraged to at least engage in recallactivities, open-ended questions, with the extra demand of having to put often complex ideasinto writing, may actively discourage a response. This is what Harris (1995) calls ‘intellectualself-censorship of second language students: if one cannot express a complex idea, the ideawill not appear’ (p. 89). For those students who are not discouraged, there is always the issueof awareness of the local conventions for presenting and structuring written material since thelogic of the student’s own culture and language is likely to in� uence the style and structureof written work in ways which may not meet local tutors’ expectations. Although it is dif� cultto quantify the impact of unconventional discourse patterns on grading (Cortazzi & Jin,1997), enough anecdotal evidence exists (Zaharna, 1996; De Vita, 2000; Ryan, 2000) toconsider that the in� uence of culture on writing style is a critical variable on students’performance under such an assessment method.

The most destructive connotation of the closed-book � nal examination, however, wouldseem to be the timed nature of this assessment method. This is a feature which, by callingfor a swift regurgitation of ideas, often impedes examinees from demonstrating the achieve-ment of the very learning outcomes that the examination is supposed to assess, whilepenalising students from cultures which value re� ection over a quick response.

Last, but not least, the end-of-course examination, being primarily implemented as aform of summative assessment (i.e. for the purpose of grading rather than feedback), makesit dif� cult for lecturers to check and guide students’ progress, thus depriving students of theessential formative element of the learning process.

Although, as Mullins & Roberts (1996) pointed out, ‘it is still the case in most [UK]Universities that scores from formal, time-constraint examinations compose the bulk ofstudents’ marks’ (p. 14), in recent years, assessment by coursework (in the form of case-studyquestions or business reports) has become more common in business and managementdegree courses. The main merit of coursework assignments is that they allow much moretime for completion (anything between 2 and 6 weeks), thus offering students time to exploreand crystallise ideas, and check the grammar and syntax of drafts (an attribute much valuedby students, especially those for whom English is not their � rst language). By providing morefreedom for students to express themselves (Becker & Johnston, 1999), and the opportunityto display originality, in-depth understanding of the topic, and higher-order skills of theBloom taxonomy type (i.e. analysis and evaluation), the coursework assignment can beconsidered a more ‘authentic’ approach to assessment (Torrance, 1995), which, as noted byYorke et al. (2000), ‘offers more potential as a learning experience for students than does theend-of-course examination’ (p. 9). The potential of this learning experience could be

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

Cultural Equivalence in Student Assessment 225

maximised by asking students to submit an early draft of their assignment, and then attendformal one-to-one ‘progress meetings’, during which critical issues can be raised to and by thetutor. Such an approach would also render assessment by coursework less vulnerable toplagiarism, and, in turn, increase the con� dence that can be placed in the work beinggenuinely that of the student. As with the end-of-course examination, the main disadvantagesof this assessment method lie in the subjectivity and questionable reliability of the markingprocess, problems that are only partially mitigated by the use of clear assessment criteria,marking schemes and moderation.

Methodological Considerations and Model Speci� cation

The considerations made in the previous section should be viewed cautiously since, as notedin the introduction, the measurement of cultural fairness in assessment is an empirical issue.Unfortunately, however, it is not easy to measure whether a method of assessment doesafford all students the opportunity to demonstrate their competence, regardless of differencesin cultural, ethnic or linguistic backgrounds.

Ideally, one would wish to be able to obtain data on speci� c sources of cultural bias forany given assessment method, explicitly model these sources of bias as independent variablesin a regression on assessment scores and, following estimation, test the statistical signi� canceof such determinants. The problem, of course, is that since these variables are non-quantitative in nature, such data do not exist. It follows that any arti� cially constructedstatistical index of cultural bias would, at best, be a poor proxy for the construct that isintended to be measured.

A more feasible approach is to run a regression on the following analysis-of-variancemodel (i.e. a regression model which contains explanatory variables that are exclusively‘dummy’ or qualitative in nature), for each assessment method:

ASi 5 a 1 bDi i 5 1, 2, … n (1)

where AS 5 marks of students under a speci� c assessment method; Di 5 1 if internationalstudent, 5 0 otherwise (i.e. home student).

Model (1) may enable us to � nd out whether, ceteris paribus, nationality (as a proxy forculture), makes any difference in a student’s mark. Assuming that the usual ordinary leastsquares assumptions hold, from (1) we obtain:

a 5 mean mark of home students 5 expected value of (ASi u Di 5 0)a 1 b 5 mean mark of international students 5 expected value of (ASi u Di 5 1)

That is, the intercept term a gives the mean mark of home students and the slope coef� cientb tells by how much the mean mark of international students differs from the mean mark ofhome students, a 1 b, re� ecting the mean mark of international students. A test of the nullhypothesis that there is no difference in the marks of home and international students can beeasily carried out by running regression (1) and � nding out whether, on the basis of a t test,the estimated b coef� cient is statistically signi� cant.

The problem with this approach is that, though informative with respect to the compar-ative performance of home and international students, since differences in performanceacross the two groups may be simply due to actual differences in ability level, it precludes usfrom making any conclusive inferences on the cultural equivalence properties of the assess-ment methods employed, even when equation (1) is estimated with respect to each method.

In this context, therefore, it is important to know if the assessment methods employedare culturally fair indicators of ability across the two groups, not just whether home andinternational students perform differently under any given type of assessment. In order to

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

226 G. De Vita

establish whether a particular assessment method is culturally biased or misleading (com-pared to others), with respect to the academic performance of international students,regardless of differences in actual (and unobservable) ability levels, a two-step approach isemployed in this study.

The � rst step entails identifying the relationship between different assessment methods(i.e. the extent to which scores achieved under a particular assessment technique are usefulpredictors of performance under other types of assessment), using a sample comprising homestudents only. Since an assumption of unidirectional causality is untenable, this involvesrunning three regressions: one where the multiple-choice test (MC) is assumed to be afunction of the coursework mark (CW) and examination mark (EX), one where CW isassumed to be a function of MC and EX, and one where EX is supposedly determined byMC and CW.

Irrespective of the type of relationship found to exist among the variables (i.e. the threeassessment methods in question), since there is no reason to assume that under culturalequivalence such relationships should not be the same across student groups, the second stepentails re-estimating the same set of equations using a sample consisting of internationalstudents only, and checking whether (for each equation) the same relationships governing thevarious assessment methods hold. If they do not, that is, if inconsistencies emerge in therelationships among assessment methods across the two samples, then these can be solelyattributed to intrinsic cultural biases.

What we need, therefore, is a statistical procedure that allows us to test for differencesin regressions on MC, CW and EX, across the home and international student groups of thecohort. One of the most popular methods of testing for differences between two regressionsis the Chow test (Chow, 1983). Application of this test entails the following steps.

(i) First, we run the three ‘pooled’ regressions (comprising both home and internationalstudents) on MC, CW, and EX. From these equations we obtain the residual sum ofsquares (RSS), say, S1, with degrees of freedom (df) 5 n 2 k, where n 5 total number ofobservations, and k 5 total number of parameters estimated.

(ii) After having split the sample into two subsamples (one of home students and one ofinternational students), we run each pair of individual regressions (on MC, CW andEX) and obtain their RSS, say, S2 and S3, with df 5 n2 2 k, and n3 2 k, respectively,where n2 is the number of observations in the home student subsample, and n3 is thenumber of observations in the international student subsample. We then add these twoRSS, to derive S4 5 S2 1 S3, with df 5 n2 1 n3 2 2k.

(iii) We then obtain S5 5 S1 2 S4.(iv) Finally, we apply the F test as follows: [S5/k]/[S4(n2 1 n3 2 2k)] with df 5 k,

n2 1 n3 2 2k. If the computed F exceeds the critical Fk, n2 1 n3 2 2k, we reject thehypothesis that the two regressions are the same.

A potential complication in this analysis lies in the role that gender effects may play in theperformance of students across assessment methods. Recent evidence by Walstad & Robson(1997) and Becker & Johnston (1999), for example, suggests that women do not do as wellas men when assessed by time-constrained multiple-choice tests. As a preliminary investiga-tion of the student performance by nationality and gender, we shall, therefore, start byrunning equation (1), and the following equation (2), for each assessment method:

ASi 5 a1 1 a2 D2i 1 a3 D3i 1 a4 (D2iD3i) i 5 1, 2, … n (2)

where AS 5 marks of students under a speci� c assessment method, a1 5 mean mark of (male)home students, D2 5 1 if female (0 if male), D3 5 1 if international student (0 if home

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

Cultural Equivalence in Student Assessment 227

student), a2 5 differential effect (on a1) of being a female, a3 5 differential effect (on a1) ofbeing an international student, a4 5 differential effect (on a3) of being a female internationalstudent. Of particular importance here is parameter a4, which allows us to test whether thedifferential effect of the gender dummy D3 (i.e. being female) is constant across home andinternational students. In other words, there may be interaction between the two variables D2and D3 and, therefore, their effect on mean AS may not be simply additive as in a2 and a3but multiplicative as in a4.

The Study

The raw data sample consists of 327 business and management undergraduates enrolled inthe � rst-year module ‘Introduction to Business and its Environment’ at Oxford BrookesUniversity (academic year 1999/2000). The module had a 16-week duration. In addition togroup work involving poster presentations, the � nal grade of students on the module wasdetermined by three components for the assessment of individual abilities: a 2000 wordwritten case-study report on a company’s competitive strategy (to be submitted in week 8),an end-of-term multiple-choice test (scored using a zero-based negative marking systemwhereby ‘incorrect guesses’ carried a penalty of 2 0.33%), and a closed-book � nal examin-ation comprising a mixture of short answer, essay-type and case-study related questions. Foreach assessment task, the marking criteria adopted were detailed in the course guide, andclearly reiterated to students during teaching sessions. In addition, students were given priorfamiliarisation with the various assessment formats. To increase ‘test-wiseness’, students werealso given examples of multiple-choice and examination questions beforehand.

The data set was then scrutinised so as to remove from the sample those students whohad not completed all of the assessment components. Given the comparative purpose of thisstudy, since ‘cultural hybrids’ (binational and/or bilingual students) would have introducedsevere bias into the subsamples, following a cross-check with each students’ registrationrecords, those subjects who, in spite of being born abroad, had lived and/or studied in the UK(and vice/versa) were also removed from the sample. The remaining data set (comprising 304subjects) was then split into two subsamples: one consisting of 195 home (UK only) students,and the other made up of 109 international students (within the latter, 24 different national-ities were represented). In both subsamples, women outnumbered men (118:77 and 66:43,respectively).

Results and Discussion

Using the data described above, estimation of equation (1) on MC, CW, and EX gave thefollowing results (the � gures in parentheses are the computed t ratios):

MCi 5 67.41 2 12.17 Di i 5 1, 2, … 304 (3a)(70.20) ( 2 7.59)

CWi 5 59.19 2 6.24 Di i 5 1, 2, … 304 (3b)(87.75) ( 2 5.54)

EXi 5 61.96 2 20.46 Di i 5 1, 2, … 304 (3c)(57.04) ( 2 11.28)

In all cases (3a, 3b and 3c), the estimated b is statistically signi� cant at the 5% level ofsigni� cance, and the results indicate that the mean mark recorded by home students undereach assessment method is higher than that recorded by international students. Speci� cally,

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

228 G. De Vita

the estimated mean score of home students in the multiple-choice test is just over 67%( 5 estimated a), and that of international students is about 55% ( 5 estimated a 2 estimatedb). In terms of coursework assignments, while the mean mark of home students is about59%, the international students’ mean mark is approximately 53%. The biggest gap, how-ever, occurs with assessment by examination. Here, while the mean mark of home studentsis almost 62%, the mean mark for international students’ scores is approximately 41%, justslightly above the pass mark of 40%. Contrary to expectations (see Yorke et al., 2000), theaverage coursework mark recorded by home students is slightly lower than the average markthe same students recorded in examinations. Whilst the similarity in the coursework andexamination scores recorded by home students may be attributable to a context-speci� csimilarity of the type of demand made by both types of assessment, such an explanationmakes the severe under performance of international students in the examination (comparedto coursework) even more controversial, suggesting the presence of the ‘differential constructvalidity’ issue discussed earlier. The point is that if CW and EX are measuring similarconstructs, then this should be true for all students. Taylor & Bobbitt-Nolen (1996) makethis a critical condition of what makes assessment valid, by arguing that evidence for thevalidity of an assessment is provided if ‘students perform consistently across differentmeasures of the same construct’ (p. 6).

Following the estimation of the more sophisticated model speci� cation given in equation(2), which also takes account of the potential gender effects on home and internationalstudents’ marks across assessment modes, we obtained:

MCi 5 67.11 1 0.49 D2i 2 14.16 D3i 1 3.28 (D2iD3i) i 5 1, 2, … 304 (4a)(43.92) (0.25) ( 2 5.54) (1.00)

CWi 5 58.90 1 0.47 D2i 2 5.56 D3i 2 1.13 (D2iD3i) i 5 1, 2, … 304 (4b)(57.71) (0.34) ( 2 3.09) ( 2 0.49)

EXi 5 60.75 1 2.00 D2i 2 21.68 D3i 1 2.01 (D2iD3i) i 5 1, 2, … 304 (4c)(35.18) (0.90) ( 2 7.51) (0.54)

The results show that for all three regressions (on MC, CW, and EX), only the estimatedcoef� cients a1 (mean score of male home students) and a3 (mean score of male internationalstudents) are statistically signi� cant at the 5% level. Hence, gender, regardless of nationality,would appear to have no signi� cant differential effect on multiple-choice test, courseworkassignment or examination marks.

Finally, by applying the steps for implementation of the Chow test outlined earlier, thefollowing results were obtained:Pooled OLS Regressions

MCi 5 19.76 1 0.36 CWi 1 0.41 EXi RSS 5 S1 5 37541.3 (5a)(5.20) (5.14) (10.69)

CWi 5 37.19 1 0.22 MCi 1 0.10 EXi RSS 5 S1 5 22971.2 (5b)(16.61) (5.14) (3.00)

EXi 5 2 3.17 1 0.66 MCi 1 0.27 CWi RSS 5 S1 5 60571.2 (5c)(5.20) (10.69) (3.00)

For each regression, n 5 304, k 5 3, and df 5 301.

OLS Regressions on Home Students’ Results

MCi 5 23.86 1 0.28 CWi 1 0.43 EXi RSS 5 S2 5 22994.3 (5a.h)(4.21) (3.12) (7.12)

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

Cultural Equivalence in Student Assessment 229

CWi 5 41.15 1 0.17 MCi 1 0.10 EXi RSS 5 S2 5 14081.6 (5b.h)(11.56) (3.12) (1.95)

EXi 5 18.43 1 0.48 MCi 1 0.18 CWi RSS 5 S2 5 25452.7 (5c.h)(3.02) (7.12) (1.95)

For each regression, n 5 195, k 5 3, and df 5 192.

OLS Regressions on International Students’ Results

MCi 5 17.21 1 0.45 CWi 1 0.33 EXi RSS 5 S3 5 14062.5 (5a.i)(2.84) (3.94) (5.13)

CWi 5 35.43 1 0.27 MCi 1 0.05 EXi RSS 5 S3 5 8582.7 (5b.i)(10.16) (3.94) (0.88)

EXi 5 0.57 1 0.59 MCi 1 0.14 CWi RSS 5 S3 5 25351.7 (5c.i)(0.06) (5.13) (0.88)

For each regression, n 5 109, k 5 3, and df 5 106.

With respect to the regressions on MC, since S4 5 (S2 1 S3) 5 37057, and S5 5 (S1–S4) 5 484.3, the computed F statistic is 1.2981 { 5 [(484.3/3)/(37057/298)]}. The F3,298

critical value at the 5% signi� cance level is 2.60. Since the computed F of 1.2981 is lowerthan 2.60, we can conclude that the two regressions on MC are not statistically different.

In terms of the regressions on CW, given S4 5 22664.3 and S5 5 306.9, the computedF statistic is 1.3450. Since the computed F does not exceed the critical F, in this case alsowe cannot reject the hypothesis that the two regressions are the same.

Finally, we compare the regressions on EX. Since S4 5 50804.4 and S5 5 9766.78, thecomputed F of 19.0961 is signi� cant at the 5% level. We can therefore conclude that the tworegressions are different.

Whilst the Chow test does not allow us to pinpoint the source of the difference, � ndingthat, unlike the regressions on MC and CW, the two regressions on the examination resultsof home and international students are statistically different, con� rms that assessment byexamination penalises international students beyond their differences in actual ability levels,as measured by multiple-choice and coursework assessments. It would appear that, perhapsdue to the timed nature of this assessment method, the examination is more susceptible(compared to other assessment techniques) to capturing irrelevant constructs related tocultural or linguistic characteristics that affect student performance differently across groups,and hence invalidate score inferences about target constructs.

One important implication of this result is that the exclusive adoption of the timedexamination ought to be avoided in culturally mixed classes of business and managementstudents since, compared to other assessment techniques, it places international students ata particular disadvantage with respect to their UK counterparts. Alternatively, studentsshould be informed and counselled on their assessment strengths and weaknesses, and thenallowed � exibility over which assessment paths they select.

Conclusions

This article set out to explore the cultural equivalence properties of three of the mostcommon assessment methods in business and management degree courses. The results of theempirical study indicate that there are considerable differences in the academic results ofhome and international students, with the marks of the latter being substantially lower than

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

230 G. De Vita

those of the former. Although this pattern applies to all the assessment techniques reviewedin this study, application of the Chow test has shown that assessment by examination placesinternational students at a particular disadvantage (compared to other assessment methods),and may, therefore, be considered not as culturally fair as coursework assignments andmultiple-choice tests in assessing culturally mixed classes. The study also � nds no evidencein support of the generally assumed gender in� uence on student performance. Being a manor a woman, regardless of nationality, would appear to have no signi� cant differential effecton multiple-choice test, coursework assignment and examination marks.

In spite of a sample size large enough to ensure meaningful inference, such � ndingsought to be considered as preliminary. Whilst the empirical analysis reported in this articleincorporated all the explanatory variables on which data were available, many other factorsnot explicitly modelled (e.g. age) can be hypothesised to in� uence student performance.Furthermore, students’ marks across assessment methods may re� ect attributes of theclassroom environment, such as the emphasis placed on related skills (e.g. memorisation,good sentence construction). Since all the students in the sample used in this study weremembers of the same class, potential classroom-speci� c effects on students’ performancecould not be controlled for. Most importantly, the general distinction between ‘home’ and‘international’ students, only allows us to scratch the surface of the issues raised by this studysince both these categories are far from homogeneous.

In view of these limitations, the comparative performance of students from differentcultural or ethnic origins, as well as the cultural equivalence properties of different assessmentmethods, are worthy of further empirical investigation using large undergraduate samplesfrom a broad range of business and management classes, across teaching institutions.Additionally, given the UK standpoint of this study (which has treated non-UK students asinternational), parallel analyses reporting on multicultural assessment experiences in othercountries would add to the cross-cultural dimension of this line of inquiry. Since culturalfairness in assessment is a cross-disciplinary concern, the extension of this work acrossacademic disciplines would also be valuable.

Acknowledgements

Financial support for this work was given by Oxford Brookes University in the form of a1-year academic fellowship in teaching and learning. I am also grateful to John Prior, forsupplying me with the raw data set used in this study, Professor Alan Jenkins and Chris Rust,for their encouragement, and two anonymous referees for their insightful comments.

Correspondence: Glauco De Vita, Brookes Business School, Oxford Brookes University,Oxford OX33 1HX, UK; e-mail: gde-vita@brookes.ac.uk

REFERENCES

AU, A.K-M. (1997) Cognitive style as a factor in� uencing performance of business students across variousassessment techniques, Journal of Managerial Psychology, 12, pp. 243–250.

BAKER, F.B. (1985) The Basics of Item Response Theory (Portsmouth, NW, Heinemann).BECKER, W.E. & JOHNSTON, C. (1999) The relationship between multiple-choice and essay response questions

in assessing economics understanding, Economic Record, 75, pp. 348–357.BETHELL-FOX, C.E. (1988) Fairness in Selection: adverse impact and the Civil Service Commission’s Executive

Of� cer Qualifying Test. Part I: Evidence of bias (London, Recruitment Research Unit Report No. 35).

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4

Cultural Equivalence in Student Assessment 231

BROWN, S., RUST, C. & GIBBS, G. (1994) Strategies for Diversifying Assessment in Higher Education (Oxford,Oxford Centre for Staff and Learning Development).

CAMPBELL, A. (2000) Cultural diversity: practising what we preach in higher education, Teaching in HigherEducation, 5, pp. 373–384.

CHOW, G.C. (1983) Econometric Methods (New York, McGraw-Hill).CORTAZZI, M. & JIN, L. (1997) Communication for learning across cultures, in: D. MCNAMARA & R. HARRIS

(Eds) Overseas Students in Higher Education (London, Routledge).DE VITA, G. (2000) Inclusive approaches to effective communication and active participation in the multi-

cultural classroom: an international business management context, Active Learning in Higher Education, 1,pp. 168–180.

ESTRIN, E.T. & NELSON-BARBER, S. (1995) Issues in cross-cultural assessment: American Indian and Alaska nativestudents (San Francisco, CA, Far West Laboratory).

HAMBLETON, R.K., SWAMINATHAN, H. & ROGERS, J. (1991) Fundamentals of Item Response Theory (NewburyPark, CA, Sage).

HARRIS, R. (1995) Overseas students in the United Kingdom university system, Higher Education, 29,pp. 77–92.

HIGHER EDUCATION STATISTICS AGENCY (HESA) (1996) Students in Higher Education Institutions 1994/95(Cheltenham, HESA).

HIGHER EDUCATION STATISTICS AGENCY (HESA) (2000) Students in Higher Education Institutions 1998/99(Cheltenham, HESA).

HOLLAND, P.W. & WAINER, H. (1993) Differential Item Functioning (Hillsdale, NJ, Lawrence ErlbaumAssociates).

LADD, P. & RUBY, R. JR (1999) Learning styles and adjustment issues of international students, Journal ofEducation for Business, 74, pp. 363–367.

LAUVAS, P., HAVNES, A. & RAAHEIM, A. (2000) Why this inertia in the development of better assessmentmethods? Quality in Higher Education, 6, pp. 91–100.

LLABRE, M.M. & FROMAN, T.W. (1987) Allocation of time to test items: a study of ethnic difference, Journalof Experimental Education, 55, pp. 137–140.

MESSICK, S. (1989) Validity, in: R. LINN (Ed) Educational Measurement (Washington, DC, American Councilon Education).

MULLINS, L. & ROBERTS, M. (1996) Assessment strategies: some comparisons between the UK and the USsystems of higher education, International Journal of Educational Management, 10, pp. 11–15.

RYAN, J. (2000) A Guide to Teaching International Students (Oxford, Oxford Centre for Staff and LearningDevelopment).

TAYLOR, C.S. & BOBBITT-NOLEN, S. (1996) What does the psychometrician’s classroom look like? Reframingassessment concepts in the context of learning. Education Policy Analysis Archives, 4, pp. 1–38.

TOMPSON, B.H. & TOMPSON, G.H. (1996) Confronting diversity issues in the classroom with strategiesto improve satisfaction and retention of international students, Journal of Education for Business, 72,pp. 53–57.

TORRANCE, H. (1995) Evaluating Authentic Assessment (Buckingham, Open University Press).WALSTAD, W.B. & ROBSON, D. (1997) Differential item functioning and male–female differences on multiple-

choice in economics, Journal of Economic Education, 28, pp. 155–172.WOOD, R. (1990) Ethnic Minorities and Ability Tests (London, Commission for Racial Equality).WOOD, R. (1993) Assessment and Testing: a survey of research (Cambridge, Cambridge University Press).YORKE, M., BRIDGES, P., WOOLF, H., COLLYMORE-TAYLOR, D., COOPER, A. & FITZPATRICK, V. ET AL. (2000)

Mark distributions and marking practices in UK higher education: some challenging issues, ActiveLearning in Higher Education, 1, pp. 7–27.

ZAHARNA, R.S. (1996) Managing cross-cultural challenges: a pre-K lesson for training in the Gaza Strip,Journal of Management Development, 15, pp. 75–87.

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 0

7:35

04

Oct

ober

201

4