Evaluation of Validity and Reliability of Diversity...

Evaluation of Validity and Reliability ofDiversity Icebreaker Questionnaire

Tetyana Sydorenko

Ladislaus von Bortkiewicz Chair of StatisticsC.A.S.E. – Center for Applied Statisticsand EconomicsHumboldt–Universität zu Berlinhttp://lvb.wiwi.hu-berlin.dehttp://www.case.hu-berlin.de

http://lvb.wiwi.hu-berlin.de

http://www.case.hu-berlin.de

Motivation 1-1

Diversity Icebreaker (DI)

� Questionnaire developed by HumanFactors AS, Norway

� Measurement of preferences forcommunication and cooperation

� One page questionnaire available in 19languages

� Easy to understand, takes 10 minutes toanswer

Evaluation of Diversity Icebreaker Questionnaire

http://www.human-factors.no

http://www.human-factors.no

Motivation 1-2

Application of Diversity Icebreaker

Main goals:� Self assessment� Teamwork

Fields for Application:� Communication and conflict

management� Diversity Management workshops� Leadership skills development

workshops� Intercultural training


Motivation 1-3

Research question

Assessment of DI quality using appropriate statistical methods withrespect to:Validity� degree to which evidence and theory support the

interpretations of test (questionnaire) scores� results of the test (instrument) should fit the construct

Reliability� consistency of the measurement instrument (questionnaire)� each item of the instrument should measure the underlying

construct


Outline

1. Motivation X

2. Description of Questionnaire3. Research questions4. Methods and Results5. Conclusions


Description of Questionnaire 2-1

Concepts of DI

BLUE� Practical, logical, goal oriented� Focus on detail� Use facts, examples, numbers,

calculations� Structured and well prepared



Concepts of DI

RED� Personal and enthusiastic� Care for others, like to be in a

group� Focus on social consequences and

community spirit� Show trust and consideration



Concepts of DI

GREEN� Focus on higher ideas, the

larger picture andconnections

� Positive to change, visionary� Open to creative and

innovative ideas� Set high goals� Unstructured and intuitive


Research questions 3-1

Importance of personality and abilitymeasurement

� Assessment of skills and knowledgeI selection of candidates for human resourcesI evaluation of learning process (testing)

� The great need for valid and reliable instruments� Commercializing of instruments e.g. TOEFL


Research questions 3-2

Evaluation of DI

� Ipsative formatI variables are not independentI correlations ρ = − 1

k−1 , where k is number of variablesI application of Factor Analysis inplausible

� unclear if the questionnaire is valid and reliable withoutipsative format

� validity of DI model for other populations

SolutionProve with appropriate statistical methods that the DI modelfits data in a non-ipsative format in different samples


Methods and Results 4-1

Data

� Transformation of questionnaire� Three data sets in new format

Language Norwegian German English

Sample size n 127 117 59

Female in % 40 50 36

Male in % 59 48 64

Average Age na 22.90 24.50

St. Deviation of Age na 3.23 3.99

Table 1: Overview data-sets



Reliability Analysis: Cronbach’s α

� consistency of latent construct Y =∑m

i=1 Xi , Xicorresponding items

� α proves whether the variance of the latent construct S2Y is

greater than∑m

j=1 S2Xj

α =m

m − 1

S2

Y −m∑

j=1S2

Xj

S2Y

αst =

m · r̄1 + (m − 1) · r̄



Cronbach’s α Results

Dimension Norwegian German English

BLUE 0.84 0.79 0.82RED 0.85 0.84 0.83GREEN 0.78 0.66 0.79

Table 2: Cronbach’s αst in data sets

� RED items: increase in Cronbach’s α if item v9 was droppedin Norwegian and German sample and v12 in English sample

� GREEN items: increase in Cronbach’s α if item v27 wasdropped in Norwegian sample, v5 in German and English, v7 inGerman data set



Explanatory Factor Analysis (EFA) forBLUE dimension

Norwegian German English

Horn 2 2 3Kaiser 3 5 5

Table 3: Number of factors for BLUE dimension

Common factors in all data sets:� Preference for working with numbers v28, v31,� Preference for precise communication and decisions v8, v15,

v18, v35� Preference for being practical minded v20, v37

Problematic items: v1, v4, v11, v23, v25, v40



EFA for RED dimension



Table 4: Number of factors for RED dimension

Common factors in all data sets:� Preference to be in a group: v2, v22, v42� Extraversion: v6, v26, v38� Consideration of other’s feelings: v14, v30, v32, v34

Unstable factor:� Preference for personal communication v9, v12, v16



EFA for GREEN dimension



Table 5: Number of factors for GREEN dimension

Common factor in all data sets:� Creative ideas and solutions v33, v36, v39� Broader perspectives v7, v10, v17

Problematic items: v3, v5, v13, v21, v24, v27, v41, v49



EFA for the whole item pool



Table 6: Number of factors for the whole item’s pool

� RED items loaded on one/two factors in all data sets� mixed factors containing BLUE and GREEN items



Two parameter Item Response Model (2PL)

P(Yij = 1 | θi ) =exp [αj(θi − δj)]

1 + exp [αj(θi − δj)]

� i = 1, . . . , n observations, j = 1, . . . , p items� θi latent trait of person i� δj difficulty of item j ,� αj discrimination power of item j

Goals:� Evaluation of item’s difficulty parameter and separation power

in BLUE/RED/GREEN sets� Estimation of δj , αi and detection of items which function

badlyEvaluation of Diversity Icebreaker Questionnaire


2PL for BLUE dimension (German data)

Figure 1: ICC curves for BLUE items in German sample



2PL for BLUE dimension (English data)

Figure 2: ICC curves for BLUE items in English sample



2PL for RED dimension (German data)

Figure 3: ICC curves for RED items in German sample



2PL for RED dimension (English data)

Figure 4: ICC curves for RED items in English sample



2PL for GREEN dimension (German data)

Figure 5: ICC curves for GREEN items in German sample



2PL for GREEN dimension (English data)

Figure 6: ICC curves for GREEN items in English sample



Multiple Indicator Multiple Cause Model(MIMIC)

Is there any significant native speaker effect obBLUE/RED/GREEN dimensions (estimation of betaj)

P(Yij = 1) = G

{δj + αj

(∑k

βkxik + θi

)}

� xik covariate (e.g. gender, native speaker)� G link function (e.g. logistic function)� At the beginning: Confirmatory Factor Analysis (CFA)� Incorporation of covariate and comparison of model fit



CFA BLUE dimension

Model RMSEA CFI TLI WRMR

one factor

without v23 0.08 0.58 0.50 1.77without v1, v8, v11,v23 v37 0.07 0.88 0.84 0.94without v1, v8, v11,v23, v35, v37 0.09 0.86 0.80 1.06

two factors

F-Numbers and F-Blue 0.05 0.85 0.83 0.90

Table 7: Model fit of BLUE latent trait in German sample



CFA RED dimension

RMSEA CFI TLI WRMR

all RED items 0.05 0.94 0.92 0.83without v12 0.05 0.94 0.92 0.83with covariate native speaker 0.05 0.92 0.90 0.89

Table 8: Model fit of RED latent trait in German sample



CFA GREEN dimension

RMSEA CFI TLI WRMR

all GREEN items without v5 0.04 0.85 0.82 0.88without v3,v5, v7, v10, v27 0.09 0.80 0.72 0.99

two factors 0.07 0.80 0.73 0.92

Table 9: Model fit of GREEN latent trait in German sample



CFA for whole item pool

RMSEA CFI TLI WRMR

all items 0.05 0.63 0.60 1.16without v3, v5, v6, v8, v27 v35 0.03 0.83 0.82 1.01GREEN by v12 0.03 0.84 0.83 1.01

Table 10: Model fit for three factor model in German sample


Conclusions 5-1

Conclusions I

Reliability analysis� the GREEN dimension has the lowest value of Cronbach’s α

EFA for single dimension� Factor structure of RED items could be replicated in different

data sets� Different factor models for BLUE and GREEN dimensions in

different data setsEFA for whole item pool� RED items build single factor/factors� Mixed factors with BLUE and GREEN items in all datasets


Conclusions 5-2

Conclusions II

2PL Model� RED items had on general higher separation power� More BLUE and GREEN items with low discrimination

coefficientsCFA and MIMIC for single dimension� Accaptable model fit only for the RED dimension� Covariate native speaker does not have significant influence on

RED latent traitCFA and MIMIC for whole item pool� Three factor model does not hold� Especially BLUE and GREEN items should be elaborated


Evaluation of Validity and Reliability ofDiversity Icebreaker Questionnaire

Tetyana Sydorenko

Ladislaus von Bortkiewicz Chair of StatisticsC.A.S.E. – Center for Applied Statisticsand EconomicsHumboldt–Universität zu Berlinhttp://lvb.wiwi.hu-berlin.dehttp://www.case.hu-berlin.de

http://lvb.wiwi.hu-berlin.de

http://www.case.hu-berlin.de

References 6-1

References

William O. Dunlap and John M. CornwellFactor analysis of ipsative measuresMultivariate Behavioural Research 29(1), 1994

Bjørn EkelundThe application of a model which integrates marketsegmentation and psychological theories to change energyconsumption in householdsHenley Management College, 1997

Bjørn Ekelund and Eva LangvikDiversity Icebreaker. How to Manage Diversity ProcessesHuman Factors Publishing, 2008


Evaluation of Validity and Reliability of Diversity...

Documents

Transcript of Evaluation of Validity and Reliability of Diversity...