Multivariate Models in Questionnaire Development

PRINCIPLES OF QUESTIONNAIRE DEVELOPMENT: MULTIVARIATE APPROACH

Dr. D. Dutta Roy, Ph.D.Psychology Research Unit

INDIAN STATISTICAL INSTITUTE

203, B.T. Road, Kolkata – 700 108E-mail: [email protected]

http://www.isical.ac.in/~ddroyVenue:P.G. Department of Psychology, Dr. Babasaheb

Ambedkar Marathwada University, Aurangabad (MS) 431004, India

mailto:[email protected]




http://www.isical.ac.in/~ddroy

Dr. D. Dutta Roy, Indian Statistical Institute

Questionnaire development is an Art. Develop Multivariate Temperament


Items are flying, feel,catch and paint them with your inner voice.


Observation

• Observe everywhere, and ask question:– Why does it happen?– How can I control it ?

• Observe carefully, you may find light in the shadow behind light.


Exploration

Do extensive literature survey; Organize them systematically and explore. like a child.


Dreaming

• Spend time alone and dream.

• Dream life style of your population for research.

• Relate each property or dimension of your research construct to their life style.

• You may find multi-dimensions.


Experience your dream


Respect Classics

• Review Pioneering articles;

• Study their perspectives;

• Do trend analysis of construct development;


Perspective

Study same object of research from multiple perspective.


Painting & Writing your feelings

Your writing will help you to develop items/statements of questionnaire


Questionnaire Development is a process of structuring and restructuring items


Statistical Modeling (Hypothetical item linkage)

D1 D2

D3 D4

I2I1

I4

I5 I6

I10

I11I7

I8

I9

I3

D: Dimensions; I: Item

•Classify items by content analysis;

•Write Dimension and hypothesize item linkages across different quadrants.

•Each dimension represents specific trait of research construct.


Statistical Modeling(Hypothetical Regression)

D1= i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+i11w11

D2=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10 +i11w11

D3=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10+ +i11w11

D4=i1w1+ i2w2+ i3w3+ i4w4+ i5w5+ i6w6+ i7w7+ i8w8+ i9w9+ i10w10 +i11w11

D:Dimension; I:Item; w=Beta weight


Find out variables for Host’s response to items

Agent

Host

Environment


Dynamics of variables

Initial

E

X

P

R

E

S

S

I

O

n

Middle Terminal

A EH

H

H


Make sampling in such a fashion so that role of intervening variables on item response can be controlled

Can we assess their perception, beliefs and attitudes ?


Observe and interview sample to find out underlying reasons behind item-response variation


MULTIVARIATE ANALYSIS OF QUESTIONNAIRE

RESPONSE


Myths about Multivariate Statistical Models


MYTHS• Statistical treatment of more than 2 variables is

multivariate statistics;– No, when more than 2 variables are interrelated with

each other, we can use multivariate statistics.In questionnaire, multiple questions measuring same issue are used. So MVS assumptions are more effective in questionnaire development.


Myth 2• Purpose of multivariate statistics

is to establish correlation among sets of variables.– True. But it’s purpose is not

limited in determining relation among sets of variables. It tends to partial out the effect of some intervening variables on relationship among sets of variables. Response to items varies with other than the construct measured by questionnaire, therefore, control of intervening variable on item response is necessary.

IVINTR_V

DV

It measures intervening vars and gives insight to rectify


Myth 3

• Loss of original score – Accepted, if analysis

extracts more latent properties within the variable.


What is Multivariate Statistics ?• MVS refers to the set of statistical tools that

simultaneously analyze multiple measurements on each individual or object under investigation.

• It is the linear combination of variables with empirically determined weights.

– Variate value= w1X1+ w2X2+ w3X3 – wnXn

• W=weight determined by the multivariate technique; X=observed variable

• MVS is the extension of univariate (central tendency, SD, variance) and bivariate analysis (cross tabulation, correlation, ANOVA, simple regression).

• In MVS, all the variables must be random and interrelated in such ways that their different effects can not meaningfully be interpreted separately.


Assumptions for Multi-variate Regression Models

• linearity of relationships, • homoscedasticity (same level of relationship for

the full range of the data), • interval or near-interval data, • untruncated variables, proper specification of the

model,• lack of high multicollinearity, and • multivariate normality for purposes of hypothesis

testing.


Multivariate model for item analysis

• Multivariate model helps researcher to develop insight about possible impact of rejecting individual item on the set of items in which specific item belongs to. This is specially true when researcher assumes interdependence among sets of items.

• When questionnaire assesses multi-traits, multivariate analysis helps to understand the latent structure or inherent relations among the different traits. So, it represents psychological map of the respondents.


Reliability AnalysisReliability refers to the consistency of

scores.Types : Time and Internal consistency;

Time Consistency

Test- Retest reliability

Test-Retest multi-item response consistencyTest-Retest multi-trait consistency

Internal consistency

Split-half reliability

Split-half Canonical correlation

Others

Rational Equivalence (non-metric)

Item-Item Correspondence

Cronbach's alpha (metric data)

Correpondence map of traits


Test-Retest Multi-item response

• All items do not behave in same fashion always.

• Identify inconsistent items in the set across periods..

Last supper: Leonardo Da Vinci


Test-Retest Multi-item response Consistency (8 months interval)

Tree Diagram for 6 Variables

Complete Linkage (after 8 months)

Euclidean distances of Items

Reading for application

Linkage Distance

APX

APAB

APP

APN

APM

APH

2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0


Complete Linkage

Euclidean distances

Linkage Distance

APAB

APP

APN

APM

APX

APH

2.5 3.0 3.5 4.0 4.5 5.0 5.5


Test-Retest Multi-Trait Consistency (8 months interval)


Complete Linkage

Euclidean distances

Linkage Distance

HATOT

RCTOT

AETOT

AFFTOT

ACHTOT

KNTOT

APPTOT

5 10 15 20 25 30 35 40

Euclidean distances (test retest reliability.sta)

APPTOT KNTOT AFFTOT RCTOT AETOT HATOT ACHTOTAPPTOT 0.00 13.34 27.96 22.07 25.34 36.22 16.25KNTOT 13.34 0.00 24.45 19.21 21.86 34.50 12.33AFFTOT 27.96 24.45 0.00 13.23 12.88 15.75 18.00RCTOT 22.07 19.21 13.23 0.00 13.15 20.66 13.89AETOT 25.34 21.86 12.88 13.15 0.00 18.00 15.17HATOT 36.22 34.50 15.75 20.66 18.00 0.00 26.91ACHTOT 16.25 12.33 18.00 13.89 15.17 26.91 0.00


Complete Linkage

Euclidean distances

Linkage Distance

HATOT

AETOT

RCTOT

AFFTOT

ACHTOT

KNTOT

APPTOT

5 10 15 20 25 30 35 40

Euclidean distances (test retest reliability.sta)

APPTOT KNTOT AFFTOT RCTOT AETOT HATOT ACHTOTAPPTOT 0.00 11.66 29.15 26.17 22.98 37.27 15.26KNTOT 11.66 0.00 26.65 23.81 19.34 34.91 14.18AFFTOT 29.15 26.65 0.00 14.11 16.06 16.03 23.73RCTOT 26.17 23.81 14.11 0.00 14.25 18.11 21.73AETOT 22.98 19.34 16.06 14.25 0.00 20.47 18.89HATOT 37.27 34.91 16.03 18.11 20.47 0.00 31.72ACHTOT 15.26 14.18 23.73 21.73 18.89 31.72 0.00

After 8 mothsTool: Reading motivation questionnaire (Dutta Roy, 2002); N=72 students of same school


Split-half

• Upper and lower part of the questionnaire sometimes differ in item content.

• All items do not reflect same content always.


Split-half Canonical correlation

Split-half Canonical correlation provides knowledge about the percent of variance in the one set explained by the other set of variables along a given dimension .


Study: Split-half Canonical correlation

• 12-item Likert type 5 point scale assessing attitude towards workers education was administered to 1600 rural workers of WB.

Min-Max25%-75%Median value

Box & Whisker Plot

-1

1

3

5

7

9

11

REV_1 REV_2 REV_3 REV_4 REV_5 REV_6

Min-Max25%-75%Median value

Box & Whisker Plot

0

2

4

6

8

10

REV_7 REV_8 REV_9 REV_10 REV_11 REV_12

Split-half rtt=0.85; Cronbach’s alpha = 0.87•Canonical correlation coefficient between the sets (first 6 and last 6 items) = 0.78, Chisq(36)=1558.3, p<0.0000.

REV_7 REV_8 REV_9 REV_10 REV_11 REV_12REV_1 0.46 0.39 0.39 0.33 0.31 0.22REV_2 0.37 0.34 0.38 0.36 0.39 0.28REV_3 0.35 0.32 0.33 0.42 0.45 0.39REV_4 0.34 0.31 0.32 0.29 0.32 0.24REV_5 0.48 0.43 0.41 0.39 0.36 0.30REV_6 0.57 0.48 0.48 0.37 0.36 0.32

Plot of Canonical Correlations

Number of Canonical Roots

Valu

e

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 2 3 4 5 6


Internal Consistency• It measures whether several

items that propose to measure the same general construct produce similar scores. For example, if a respondent expressed agreement with the statements "I like to ride bicycles" and "I've enjoyed riding bicycles in the past", and disagreement with the statement "I hate bicycles", this would be indicative of good internal consistency of the test.


Item-Item correspondence: Internal consistency among 42 items of Reading Motivation questionnaire

2D Plot of Column Coordinates; Dimension: 1 x 2

Input Table (Rows x Columns): 72 cases x 42 items

Standardization: Row and column profiles

Reading Motivation questionnaire

Dimension 1; Eigenvalue: .16550 (16.55% of Inertia)

Dim

ensi

on 2

; Eig

enva

lue:

.098

53 (9

.853

% o

f Ine

rtia)

APHAPM

APNAPP

APXAPAB

KNH

KNIKNK

KNUKNWKNZ

AFI

AFT

AFV

AFY

AFAA

AFAB

RCJ

RCL

RCP

RCSRCY

RCZ

AEK

AEM

AEO

AEQ

AES

AET

HAL

HAN

HAO

HARHAU

HAV

ACHJ

ACHQ ACHR

ACHW

ACHX

ACHAA

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

-3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0

Correspondence Map shows cluster of intrinsic reading motivation items and extrinsic motivation items are scattered widely.


Correspondence map of traits

2D Plot of Column Coordinates; Dimension: 1 x 2

Input Table (Rows x Columns): 14 x 6


Dimension 1; Eigenvalue: .24804 (70.26% of Inertia) Dim

ensi

on 2

; Eig

enva

lue:

.075

17 (2

1.29

% o

f Ine

rtia)

C1

C2

C3C4

C5

C6

-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2Row.CoordsCol.Coords

2D Plot of Row and Column Coordinates; Dimension: 1 x 2Input Table (Rows x Columns): 14 items x 6 response categories



Dim

ensi

on 2

; Eig

enva

lue:

.075

17 (2

1.29

% o

f Ine

rtia)

Row1

Row2

Row3

Row4

Row5

Row6

Row7

Row8

Row9

Row10

Row11

Row12

Row13

Row14C1

C2

C3C4

C5

C6

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4

2D Plot of Row Coordinates; Dimensions: 1 x 2Input Table (Rows x Columns): 14 x 6



Dim

ensi

on 2

; Eig

enva

lue:

.075

17 (2

1.29

% o

f Ine

rtia)

Row1

Row2

Row3

Row4

Row5

Row6

Row7

Row8

Row9

Row10

Row11

Row12

Row13

Row14

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4


Validity

• Can questionnaire predict change in criterion ?

• Can questionnaire make difference ?


Types of Validity

Expert JudgmentContent

Item-Total correlation

Construct

Factorial

Criterion

Concurrent

Predictive Convergent Discriminative


Validity Validity denotes the extent to which an instrument is measuring what it is supposed to

measure It indicates extent of relationship between a scale and the measure of independent criterion variable. It is assumed that criterion variable is reliable and valid.

Bi-variate techniques:Content Validity (It is concerned with the relevance of contents of items, individually and as a whole):

Correlating experts’ judgement; Item-item or item-total correlation

Criterion - related Validity (Correlating questionnaire scores with criterion variable): Correlating the test with criteria during data collection (Concurrent validity);before or after (keeping time gap) (Predictive validity).

Construct Validity : It is concerned with the extent to which questionnaire measures a theoretical construct or trait. Construct is a sort of concept, which is formally proposed with definition and is related to empirical data.

Techniques:Factor analysis (Factorial validity);Correlating with other theoretical measure with which the developing instrument should correlate (convergent

validity)Correlating with other theoretical measure with which the developing instrument should not correlate

(Discriminant validity)


Content validity

Colors of all items are not same Some item behaves differently

Content Validity (It is concerned with the relevance of contents of items, individually and as a whole): . •Correlating experts’ judgement; •Item-item or item-total correlation


Predictive validity

• Questionnaires measuring same variable provide different results.

• Which questionnaire predicts success ?


Concurrent validity

• Collect data simultaneously with two instruments measuring similar construct.

• Use canonical correlation if product moment correlation is significant.


Construct validity

• Questionnaire should measure the underlying theories of research construct.


Factorial validity

• Items can be classified by respondent’s latent phenomena.


Factorial validity

• Two questionnaires can be mixed with respondent’s frame of reference.

• Principal component analysis can be used to find common properties.


Studies on Validity of questionnaire


Studies

CONTENT Item-Rating correspondence of questionnaire measuring computer programming task taxonomy.

CRITERION RELATED

Step-wise Multiple regression in predicting academic achievement through reading motivation.

Predictive

CONSTRUCT Correspondence analysis to explore latent structure of RMQ Factorial

Discriminative Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ)


Content validity: Item-Rating Correspondence

Response categoriesItems R1 R2 R3 R4 R5 R6I1 3 1 11 11 46 122I2 13 9 18 39 65 57I3 14 29 36 43 30 49I4 12 9 17 30 43 90I5 13 13 54 36 52 33I6 4 3 6 31 56 101I7 3 9 20 50 58 61I8 59 48 33 29 16 16I9 11 24 33 53 46 34I10 36 10 35 41 43 36I11 19 29 38 55 42 18I12 50 38 28 26 30 39I13 11 37 38 41 42 32I14 88 43 30 19 10 11

Input data for CA

Plot of Eigenvalues


Total Inertia=.35305 Chi²=994.55 df=65 p=0.0000

Number of Dimensions

Eig

enva

lue

0.00

0.05

0.10

0.15

0.20

0.25

0.30

1 2 3 4 5

Eigenvalues and Inertia for all DimensionsInput Table (Rows x Columns): 14 x 6Total Inertia=.35305 Chi²=994.55 df=65 p=0.0000

Singular Eigen- Perc. of Cumulatv ChiValues Values Inertia Percent Squares

1 0.50 0.25 70.26 70.26 698.732 0.27 0.08 21.29 91.55 211.763 0.13 0.02 4.59 96.14 45.684 0.10 0.01 2.93 99.07 29.175 0.06 0.00 0.93 100.00 9.21

Dutta Roy,D. (2002). Correspondence between item and rating on the checklist of relative importance of computer programming tasks., Journal of Psychometry, 16,2,67-76.


Criterion related validity:Stepwise Multiple regression

1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00Dependent Variables 1.Bengali 1.002.Arithmetic 0.79** 1.003.English 0.22** 0.28** 1.004.Total 0.83** 0.88** 0.65** 1.00Independent Variables 5.Application 0.32** 0.34** -0.13 0.22** 1.006.Knowledge 0.22** 0.11 0.03 0.15* 0.13 1.007.Affiliation -0.15* -0.13 0.06 -0.09 -0.27** -0.36** 1.008.Recognition -0.27** -0.23** 0.15* -0.14 -0.42** -0.28** 0.22** 1.009.Achievement 0.19** 0.19** 0.03 0.17* 0.01 -0.05 -0.34** -0.27** 1.0010.Aesthetic -0.04 -0.03 0.03 -0.02 -0.25** -0.07 -0.21** -0.01 -0.29** 1.0011.Harm avoidance

-0.34** -0.32** -0.07 -0.30** -0.29** -0.35** 0.04 0.12 -0.21** -0.11 1.00

Inter correlation matrix of reading motivation variables (n=200)

Step No.

Reading Motivation variables

R R2 R2 F df

1 Harm avoidance 0.34 0.12 0.12 25.96 1,1982 Application 0.41 0.17 0.05 12.94 2,1973 Recognition 0.44 0.19 0.02 5.25 3,1964 Achievement 0.45 0.2 0.01 2.2 4,1955 Knowledge 0.46 0.21 0.01 2.12 5,194

1 Application 0.34 0.11 0.11 25.582 Harm avoidance 0.41 0.17 0.05 133 Achievement 0.43 0.19 0.02 4.74 Aesthetic 0.44 0.19 0 1.05

1 Recognition 0.15 0.02 0.02 4.682 Harm avoidance 0.17 0.03 0.01 1.483 Application 0.2 0.04 0.01 2.24

1 Harm avoidance 0.3 0.09 0.09 20.132 Application 0.33 0.11 0.02 3.973 Achievement 0.35 0.12 0.01 3.03

P

Prediction to First language achievement0.000.000.020.140.15

Prediction to Arithmetic achievement1,198 0.002,197 0.003,196 0.034,195 ns

Prediction to English score achievement1,198 0.032,197 ns3,196 ns

Prediction to Total Score1,198 0.002,197 0.053,196 0.08

Data collected from Bengali medium schools


Correspondence analysis to explore latent structure of RMQ

Totalf % f % f % f % f % f % f %

Achievement 6 3.68 31 3.22 115 8.85 236 17 265 21.6 201 21.1 48 15 902Application 1 0.61 20 2.08 50 3.85 126 9.07 274 22.3 296 31.1 135 42.2 902Knowledge 3 1.84 22 2.28 68 5.23 156 11.2 239 19.5 304 31.9 110 34.4 902Aesthetic 29 17.8 146 15.2 249 19.2 270 19.4 155 12.6 43 4.52 10 3.13 902Affiliation 29 17.8 219 22.7 320 24.6 202 14.6 109 8.88 23 2.42 0 0 902Harm avoidance 85 52.2 417 43.3 239 18.4 100 7.2 45 3.66 13 1.37 3 0.94 902Recognition 10 6.13 108 11.2 258 19.9 299 21.5 141 11.5 72 7.59 14 4.38 902Total 163 100 963 100 1299 100 1389 100 1228 100 952 100 320 100 6314

Scoring Categories0 1 2 3 4 5 6

Row.Coords

Col.Coords

Correspondence between Reading motivation variables and scoring categories




Dim

ensi

on 2

; Eig

enva

lue:

.080

61 (1

6.74

% o

f Ine

rtia)

Achievement

Application

Knowledge

Aesthetic

Affiliation

Harmavoidan

Recognition

SCORE_0

SCORE_1

SCORE_2

SCORE_3

SCORE_4

SCORE_5

SCORE_6

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5

Chi-square (36) = 3041.4, p<0.00001


Discriminative validity of Attitude towards School Infrastructure questionnaire (ASIQ)

ASIQ would be able to differentiate between good and poor infrastructure schools. Methods: ASIQ was administered to the students of primary schools with good and poor

infrastructures. Good and poor criteria was determined by the 8 indices available in District Information System of Education (DISE). Based on Median rank, schools were classified into high and low infrastructure categories.

Results: stepwise discriminant function analysis ([Wilks’ Lambda=0.77, Rao’s R (9,153)=5.13, p<0.00] ) extracted five most important attitudes namely, reliability, equal opportunity, comfort, safety, cleanliness in predicting differences between good and poor infrastructure schools. The overall classification accuracy is 82.8% . This suggests high predictive capacity of the Discriminant function obtained.

Difference between schools with Good and Poorinfrastructure

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Cleanliness

Safety

Com

fortability

Adequacy

Exploring

Reliability

Easiness

Equal

Opportunity

Willingness

to Participate

Attitudinal Dimensions Good InfrastructurePoor InfrastructureCleanliness 5.05 -0.46Safety -1.27 -0.45Comfort 4.93 3.76Reliability 6.2 4.51Equal Opportunity 2.58 2.03Constant -26.27 -15.57

Fisher’s Linear Discriminant Functions for differentiating Schools with Good and Poor Infrastructure

Multivariate model is the Journey to Harmony

THANK YOU

Multivariate Models in Questionnaire Development

Education

Transcript of Multivariate Models in Questionnaire Development