Jiang et al./Measuring IS Service Quality

The application of the measure to the IS field hasgamered a great deal of recent debate (Kettingerand Lee 1997; Pitt et al. 1997; Van Dyke et al.1997). There is a psychometric concern ofoperationalizing a single concept as the differenceof two separate elicitations and also empiricalambiguity of the construct structure. The use ofthe difference scores presents a number ofpotential flaws, including reduced reliabiiity, poorconvergent validity, and unstable dimensionality(Van Dyke et ai. 1997). SERVQUAL as adoptedfor information systems has been inconsistent inlerms of dimensional structure, reliability, andvalidity (Cronin and Taylor 1992; Kettinger andLee 1997; Kettinger et al. 1995; Parasuraman etai. 1994). The question is whether the effects ofthese issues are serious enough to exclude theuse of SERVOUAL in the IS setting.

Using an IS professional sample populationmatched to a sample of IS users, we re-examineSERVOUAL issues from the IS professional side:(1) the dimensionality of the instrument, (2) theconvergent validity, and (3) the reliability mea-sures of the difference scores. We then examineIhe expectation gap between the IS user and ISprofessional according to the same criteria. Sinceexpectation gaps are expected to impact per-ceptions (Ginzberg 1981 ), we compare the resultsof the expectation gap to the dimensions of themore common user satisfaction scale (Baroudiand Orlikowski 1988).

Ci Empirical Support I

. To addressthe difference score concerns involved-.: in SERVQUAL, empirical analysis is necessary.{::: Pitt et al. (1997), based upon user samples,

calculated the reliability adjusted for differencesiH and demonstrated no reliability problem asso-

ciated with the SERVQUAL. Kettinger and Lee(1997 addressed the dimensionality problemusing student samples across different campusesand found consistent dimensions existed in the IS-

adapted SERVOUAL. Qthers found a differenti; number of dimensions depending on the popula-

tion involved (Cronin and Taylor, 1992: Kettingeret al. 1995; Parasuraman et al. 1994; Pitt et al.1995). Further studies of user populations areclearly needed as are studies examining theappropriateness of using SERVQUAL from theperspective of IS professionals to analyze gapsbetween providers and customers.

Sampie

To obtain a sample of IS professionals andmatched IS users, the SERVOUAL and the usersatisfaction (UIS) questionnaires were mailed to200 managers in different organizations in theU.S. The 200 managers selected were those whoagreed to participate from 612 contacts made withdifferent organizations. The list of organizationsand managers for contact was extracted from amore comprehensive listing of organizations main-tained by an economic development center at aMidwestern university.

These managers were first contacted directly bythe authors or graduate assistants. Each mana-ger was asked to secure a response from an ISprofessional for the SERVQUAL instrument(Appendix A). The manager was also asked tosecure a response from an IS user for theSERVOUAL instrument (Appendix 8) and for theUIS instrument (Appendix 0). Managers whoreturned both the IS professional version and theuser versions were considered to have returnedmatched sets. All of the respondents wereassured that their responses would be keptconfidential. A total of 186 questionnaires werereturned, which included 168 matched sets. Thedemographic information of these respondents isshown in Table 1.

Before any analysis was conducted on thedimensionality or scales, the data was examinedfor potential biases. An ANOVA was conductedby using service quality (as the dependentvariable) against each demographic categoryshown in Table 1 (independent variable). Resultsdid not indicate any significant relationships. Non-response bias was examined by comparing our

MIS Quarterly Vot. 26 No. 2/June 2002

Jiang et al /Measuring IS Service Quality

Table 1. Demographics

1. GenderMaleFemaleNo Response

2. AgeUnder 2525 to 3435 to 4445 and overNo response

3. Work ExperienceUnder 5 years5 to 9 years10 to 14 years15 to 19 years20 years or more

4. Experience in Different Applications1 to 3 areas4 to 6 areasMore than 6 areasNo response

5. Total Number of Employees in OrganizationLess than 50 people50 to 99 people100 to 249 people250 to 499 people500 to 999 people1,000 to 2,499 people2,500 people or moreNo response

IS Professionals

111662

226346363

39464114272

5158627

4332262210121310

IS Users

661011

437134191

66442317272

expectation measures on the SERVQUAL scalesand the UIS scales to previous studies (BaroudiandOriikowski 1988, Pitt et al. 1998). Chi-squaretests found no difference between the means ofour sample to those in the other studies oncenormalized tc a five-point scale. Additionally, thesample was split into early and late respondentsand t-tests found no difference in the means ofany SERVQUAL dimension. Non-response biasdid not arise as an issue based on these tests.

Dimensionality of SERVQUALfrom the Other Side

If the measurement model provides a reasonablygood approximation to reality, confirmatory factoranalysis (CFA) accounts for observed relation-ships in a data set. The chi-square test providesa statistical test of the null hypothesis that themodel fits the data. In addition, other fit indicesare typically used to identify overall goodness of

148 M/s Quarterly Vol. 26 No, 2/June 2002

Jiang et al./Measuring IS Service Quality

Table 2. Confir

Fit Index

RMRChi-square

d.f.

Chi-square/d.f.

CFINNFI

GFIAGFI

matory Factor Analysis fo

Threshold(s .10)

( i 5.0)( .90)U .90)( i .90)(> .80)

Model 1.057

196.7965

3.030.880.850.840.78

r SERVQUAL ModeModel 2

.048106.6

641.670.950.940.910.87

S

Model 3.041

125.1684

1.490.950.940.910.87

Mod9l 4.041

86.8459

1.470.960,940.920.88

Notes:(1) RMR = Root Mean Square Residual(2) CFI = Comparative Fit Index(3) NNFI ^ Bagozzi (1980) Non-normed Index(4) GFI = Goodness of Fit Index(5) AGFI = GFI Adjusted for Degrees of Freedom(6) Model 1 = (Responsiveness, Assurance, Empathy, and Reliability) as one dimension(7) Model 2 = (Responsiveness, Assurance, and Empathy) and Reliability as two dimensions(S) Model 3 = Responsiveness, Assurance, Empathy, and Reliability as four dimensions, 16 tem

as in Parasuraman et al. (1994)(9) Model 4 = Responsiveness, Assurance, Empathy, and Reliability as four dimensions, 13 item

as in Kettinger and Lee (1994) as shown in Figure 1

fit. Previous studies of rigor have found theSERVQUAL tangibles dimension to be weak(Cronin and Taylor1992,1994; Kettinger and Lee1994,1997; Parasuraman etal, 1991). We beginouranaiysis with the four dimensional model usedin other studies of IS service quality because of

^ the recency of the results and the IS orientation ofthe instrument (Kettinger and Lee 1994, 1997).We test one-, two-, and four-dimensional models

I found in other recent studies, including a newermodel proposed by the developers of SERVQUAL(Parasuraman et al. 1994). The model we carry

' fonward in the analysis (model 4) compares;. favorably to the remaining models. The preferred levels of each index for the CFA and the results of

the models are shown in Table 2, Analysis wasconducted with LISREL 8.51 using maximum

r; likelihood estimation on the covariance matrix.

The correlations and descriptive statistics forthese dimensions appear in Tables 3 and 4.Patterns of mean, median, skewness, and kurtosisin Table 4 were examined according to convention(Ghisellietal. 1981). The responses had reason-able, skewness (less than 2), and kurtosis (lessthan 5). This indicates a lack of bias in thesample in the measured variables. The four-dimensional model is highlighted in Figure 1.

SERVQUAL Validity fromthe Other Side

Convergent validity and discriminant validity wereexamined. Empirically, convergent validity can beassessed by reviewing the t-tests for the factor

MIS Quarterly Vot. 26 No. 2/June 2002

Jiang et al./Measunng IS Service Ouaiity

Tabie 3. Corr

Reliability

elation of Dimensions m Model 4

Reliability

1.00

Responsiveness 0,82Std, Errort-value

AssuranceStd, Errort-vaiue

EmpathyStd, Errort-value

(0.05)17.34

0.81(0.06)14.12

0.65(0.07)9.96

Responsiveness

1.00

0.95(0.05)17.55

0.84(0.06)15.28

Assurance

1.00

0.91(0.06)15.20

Empathy

1,00

iptive Statistics of tiie 4D SERVQUAL Model (Model 4)

MeanVarianceMedianSkewnessKurtosis

Reliability

.64

.91

.33

.721.22

Responsiveness

.46

.69

.33

.69

.88

Assurance

.36

.50

.33

.39

.65

Empathy

.23

.39

.00

.911.52

loadings. If all factor loadings for the indicatorsmeasuring the same construct are statisticallysignificant (greater than twice their standarderror), this can be viewed as evidence supportingthe convergent validity of those indicators(Andersen and Gerbing 1988), All t-tests Vi^ eresignificant (Table 5) showing that ali indicators areeffectiveiy measuring the same construct, or highconvergent vaiidity.

Empirically, discriminant validity is achieved whenthe correlations betv i^een any two dimensions aresignificantly different from unity (Bagozzi andPhillips 1982), Evidence regarding discriminantvalidity can be obtained by using the chi-square

difference test. The chi-square difference testcompares an unconstrained model that estimatesthe correlation between a pair of constructs and aconstrained modei which fixes the value of theconstruct correlation to unity. The difference inchi-square between these models is a chi-squarevariate with degrees of freedom equal to one. Asignificant chi-square difference implies that theunconstrained model is a better fit for the data,thereby supporting the existence of discriminantvalidity (BagozziandPhillips,1982), The results ofthe chi-square difference tests generally supportthe discriminant validity of the scales: howeverthe ASSURANCE scale exhibits some historicaldiscriminant validity problems (see Table 6),

150 MIS Quarteriy Voi. 26 No. 2/June 2002

Jiang et ai./Measuring IS Service Quaiity

jrement ModelFour Factors

' SERVQUAL Reliability fromthe Other Side

Reliability refers to consistency of measurement.A construct is reliable if, for example, it providesessentially the same set of scores for a group ofsubjects upon repeated testing. There are anumber of different ways that reliability can beexamined. In the present study, the compositefeliability, variance extracted estimates, andCronbach alpha values were examined.

Composite reliability reflects the internal con-sistency of the indicators measuring a given factor(Fornel! and Larcker 1981). The compositereliability can be computed by taking the square ofthe sum of standardized factor loadings for thatfactor divided by the sum of the error varianceassociated with the individual indicator variablesand the square of the sum of the standardizedfactor loadings (Forneli and Larcker 1981). Thecomposite reiiabilities for each SERVQUALdimension are shown in Table 7. Results indicate

M/S Ouarterly Vol. 26 No. 2/June 2002 151

Jiang e at./Measuring IS Ser

Table 5. Convergent Validity of Model 4 |Constructs and Indicators

ReliabilityDRELl (item 5)DREL3(item7)DREL4 (item 8)

ResponsivenessDRESP2 (item 11)DRESP3 (item 12)DRESP4 (item 13)

AssuranceDASSl (item 14)DASS3(item16)DASS4 (item 17)

EmpathyDEMP1 [item 18)DEMP3(item20)DEMP4(item21)DEMP5 (item 22)

Standardized Loadings

0.810.800.89

0.740.730.69

0.710.470.65

0.650,610,620.68

adings significant at p < .01 level.

Table 6. Discriminant Validity of Model 4

Construct PairREL-RESPREL-ASSREL-EMP

RESP-ASSRESP-EMPASS-EMP

AChi-Square22.43

14.84

77.81

0.08

10.24

2.75

A Degrees ofFreedorr

1

1

1

1

1

1

Discriminant Validity

Yes-

Yes*

Yes*

No

Yes*

No

'Indicates significant at p ^ .01 li

152 MIS Quarterly Vot. 26 No. 2/June 2002

Jiang e a/,/Measuring IS Service Ouatity

ConstructIndicators

Fieliability

Responsiveness

Assurance

Empathy

CoR

Service

mposJteliability

.87

.76

.64

.73

Gap of IS ProfessionalsVariance Extracted

Estimate

.70

.52

.38

.41

CronbachAlpha

.87

.76

.65

.74

Adjusted(Johns 1981)

,84

.67

.64

.67

an acceptable level of reliability, although theassurance scale is lower than desired for empi-rical analysis (Carmines and Zeiler 1988). In

__ addition, the traditional Cronbach alpha values foreach ofthe SERVQUAL dimension are shown forcomparison. The Johns (1981) adjusted formulator difference score alpha value was also applied.

Variance extracted estimates, as discussed by_ Fornel and Larcker, assess the amount of

variance that is captured by an underlying factorin relation to the amount of variance due tomeasurement error. Fornel and Larcker suggest

Ilhat it is desirable a construct exhibit estimates of.50 or larger, because estimates less than .50indicate that variance due to measurement erroris larger than the variance captured by the factor.

'^ However, this test is quite conservative. Very;. often, variance extracted estimates will be below-^ .50, even when reliabilities are acceptable. Thei^ variance extracted estimates for each dimension,, of SERVQUAL are also shown in Table 7.

Expectation Gap and Validation

. One premise of the SERVQUAL model is that thegaps are produced by a series of prior gapsIZeithamI et al. 1990). One of these is a gapbehween the expectation of the user and the abilityotthe service provider to understand their desires.Pitt et al. (1998) found the gaps to be present andmeaningful in the interpretation of the service gap.Ginzberg (1981) presented a similar concept thatproposes that a gap in expectations between IS

professionals and IS users will lead to a lack ofsatisfaction on the part of the user, a predictiveform of final perceptions. We explore this premiseby examining the relationship between the expec-tation gap and a common measure of user satis-faction, the UIS (Baroudi and Qriikowski 1988).

First, the gap scores are taken for the expec-tations of the users and the IS professionals forthe SERVQUAL instrument items. We restrictourselves to the items in the four dimensionslocated in the previous analysis. The results forthe expectation gap measures are tested to thesame rigor as the service gap measures. Table 8shows the CFA fit results to the four-dimensionalmodel. Figure 2 shows the model as fit by theCFA, Table 9 presents the correlations of the fourdimensions and Table 10 has the descriptivestatistics. Table 11 has the results of the conver-gent validity tests and Table 12 shows the resultsof the tests for discriminant validity. Reliabilityfigures are in Table 13.

To examinetherelationshipofthe expectation gapto UIS, the UIS instrument was first validatedaccording to the same rigor as the SERVQUALinstrument. UIS is a more widely accepted instru-ment and results from this data followed expecta-tions (Baroudi and Orlikowski 1988), The struc-ture found by Baroudi and Oriikowski held in thissample, with the three dimensions of informationproduct, staff and services, and knowledge/involvement present. Due to the acquisition oftheexpected structure, the CFA, reliability, andvalidity results of the UIS are not presented herefor the sake of brevity.

MIS Quarterly Vol 26 No. 2/June 2002

Jiang et al./Measuring IS Service Quality

Fit Index

RMR

Chi-square

d.f.

Chi-square/d.f.

CFI

NNFI

GFi

AGFI

Threshold

(. .10)

(: 5.0)(.. .90)

(.. .90)(.: .90)

( .80)

Modei 4

0.54

120.30

59

2.03

0.92

0.89

0.90

0.85

Figure 2. Expectation Gaps Measur

154 M/S Quarterly Voi. 26 No. 2/June 2002

Jiang et al./Measuring IS Sen/ice Quality

FAFF

1

i

iII

UISK

NEm

path

y

11I

ura

nce

If,

enes

s

1

Resp

abi

lityRe

l

_

1,00

Rel

iabi

litySt

d, Er

ror

t-valu

e

o

-0,

079,

42

Resp

onsiv

enes

sSt

d, Er

ror

t-valu

e

oq

001-

,00-

:0

0,90

-0.

0713

.85

Assu

ranc

eSt

d. Er

ror

t-valu

e

1,00

0,84

-0.

0613

,43

0.6'

-0-

010

,5

0,68

-0,

079,

42

Empa

thy

Std,

Erro

rt-v

alue

.00

-0.1

5-0,

09-1.

58

-0,

19-0,

10-1.

89

'--' 9 T

0,01

-0,

100,

14

UISK

NOW

Std.

Erro

r

S

' o CM

-0,

06-0.

10-0,

63

-0.

16-0,

10-1,

51

CO o -O-

d d cri

0,07

-0,

10 0,73

UISI

Pst

d. Er

ror

t-vai

ue1,

00

10,

580,

078,

55

-0.

19-0,

09-2.

04

-0.

31-0,

10-3,

27

-0,

1

-1,

[

9 9 9

UIST

AFF

Std.

Erro

rt-v

alue

M/s Ouanerly Vol. 26 No. Z-June 2002 155

Jiang et al./Measuring IS Ser

Fit Index

RMR

Chi-squared.f.

Chi-square/d.f.CFI

NNFI

GFI

AGFI

Threshold

(= .10)

(; 5.0)(. .90){, .90)(i .90)(> .80)

Model 4 ^

0.54 i f120.30

59

2.03

0.92

0.89

0.90

0.85

Figure 2. Expectation Gaps Measurement Model

154 MIS Quarterly Vol. 26 No 2/June 2002

Jiang et ai.measuring K

liabi

lity

Std.

Erro

rv

alue

00

o CTJ

sp

onsiv

enes

sSt

d. Er

ror

t-valu

e

1.00

r2 o 5o o o

ssur

ance

Std.

Erro

r