Development of a Measure to Assess the Quality of User-Developed Applications

8/17/2019 Development of a Measure to Assess the Quality of User-Developed Applications

http://slidepdf.com/reader/full/development-of-a-measure-to-assess-the-quality-of-user-developed-applications 1/10

Development of a Measure to Assess the Quality of User-Developed Applications

Suzanne Rivard Guylaine Poirier&ole desHEC M i n i s t i x eJustice

Montrt?al, Que. Montrt?al, Qut?.

Abstract

Building on work in so fi a re engineering and n end-usercomputing, his study developed nd assessed a measureouser-developedapplications quality. he quality constructcomprises eight dimensions: reliability, effectiveness,portability, economy, user-fiiendliness.understandability,verifiability. nd maintaina bility. In turn, each qualitydimension is com posedof a set of criteria. Finally, eachcriterion is measured by a seriesof items.The nstrumentwas tested by meansof a SUN^ involving I I O end-users.Confirm atory fac tor analysis, using the partial leastsquares technique,was wnducted. he results indicate thatthe S6-items instrument i s reliable nd valid. nd that itmight be a useful tool for researchers and or practitionersalike.

Introduction

Today’s organizations have less financial resources todevote to information technology th n before. While theannual increase in Information systems (IS) budgets was14.6 during the 1975-1985 period, it slowed down to alow 2.2 in 1993 [18, 411. As well, the number ofworkstations in the 100 computer premier organizations,that is, the organizations that make the most efficient useof their computing resources, decreased by 20.6 between1990 and 1992, from a median of 55 workstations per 100employees in 1990 to a median of 44 workstations per100 employees in 1992 [13,14,15.16.17]. This reducedavailability of slack resources for data-processing has eadorganizations to exert more control on their spendings oninformation systems activities, including end-usercomputing. For this purpose, robust evaluation tools areneeded to improve the management of end-user computing,and to make sure that human, financial, and materialresources devoted to it are used efficiently.

Management of end-user computing is an important issuein the public and the private sectors, that requires validand reliable measures [12]. In a survey of 13 1 information

systems directors of Canadian organizations conducted byBergeron, Rivard and Raymond [7]. as much as 96 ofthe respondents indicated that they would use aninstrument for evaluating the quality of end-user

1060-3425194 3.00 1994 EEE

Louis Raymond F m p i s BergeronUQTR Universitk Laval

Trois-Rivii?res, Qu6. Q u b ,Qut?.

computing activities if such an instrument was available.Only four percent did not show interest in such aninstrument. Indeed, the development, and validation ofvarious tools and techniques for IS measurement is one ofthe key issues for IS mauagement in the 1990s [30].

End-users are employees, outside the information systemsdepartment, who use computer applications toaccomplish their organizational tasks. Among them areuser-developers who develop applications for themselvesor for their coworkers [27. 351. While the technicalabilities of userdevelopers may vary considerably, they are

basically able to analyze, design and implement computerapplications [a] he purpose of the present study,which follows the preliminary work by Rivard, Lebrunand Talbot [37], was he development of an instrument tomeasure the quality of userdeveloped applications UDA).

The following section provides an overview of the conceptof software quality, first from a software engineeringperspective, and then from an end-user computingstandpoint. The definition of software quality used in thestudy is then presented, followed by the methodologyadopted to assess it. Finally, the results of the dataanalyses conducted to assess the measurement properties ofthe nstrument are presented and discussed.

Software quality

For several years now, the definition and measurement ofsoftware quality has been a concern for researchers andpractitioners in the area of software engineering [8,9, 11,

20, 24. 33, 381. The early efforts ([24], for instance)resulted in the identification of several dimensions ofsoftware quality: reliability, maintainability, availability,precision, tolerance to errors, accuracy, and efficiency.However, it was soon found that such a list of qualityattributes was not readily usable, since different attributesappeared o share the same meaning, and that a given termcould bear d i f f m t meanings.

Following these early developments, which provided anon-organized view of software quality, a consensus started todevelop around the notion that quality was a multi-dimensional construct, and that its understanding required a

522 b e d i n g s of the Twenty-SeventhAnnual HawaiiInternational Conferenceon System Sciences,1994



hierarchical definition [8,9]. A conceptual model for sucha definition was first proposed by Cavan0 and McCall[9]who define quality as a four layer construct. At the toplevel of the hierarchy is the construct of quality which isconstituted by second level quality dimensions. In turn,each quality dimension is composed of third level criteriaFinally, each criterion is measured by a set of metria oritems, located at the bottom level of the hierarchy.

From this gener ic model of software quality, severalhierarchical definitions emerged [2,20.32]. and effortswere made to operationalize them I33.381. A furtherexamination of these operationalizations reveals that mostdimensions of software quality are measured throughmetria that are quite close to system code. For instance,Petrova and Veevers [33] review 26 studies proposingmeasures of software reliability. Most of the studiesoperationalized software reliability through code-relatedmetrics such as number of knots and number of binarydecisions. Similarly, Robillard, Coallier, and Coupal[38], propose metria for operationalizing the testabilityand readability of a system. These include number ofindependent paths, number of loops, mean nesting level,and comments’ volume ratio. As demonstrated byRobillard et al., such an approach is most appropriatefrom a software engineering perspective. However, whilethe conceptual definitions of software quality areimmediately relevant to the measurement of userdevelopedapplications quality. the highly technical stance adoptedfor their operationalization is not readily usable in thecontext of end user computing, where the user perspectivehas to be taken into account [34]. This aspect isemphasized by Amoroso and Cheney [11 who define user-developed applications quality as “the degree to which anapplication of high grade accomplishes or attains its goal

from the perspective of the end user”(p.2).

Consequently, while the software metrics literature ismost useful in providing a generic definition forapplication quality, efforts remain to be made in order tooperationalize it for the end-user computing context. Tworecent studies attempted to address this issue. Relying onthe software metrics literature, Rivard, Lebrun and Talbot[37] proposed such a measure which included ten qualitydimensions: reliability, user-friendliness, integrity,extendibility, correctness, understandability, portability,economy, efficiency, and verifiability. The ten dimensionsincluded a total of 24 criteria, measured via 93 items. Theresults of the pilot study undertaken to assess theirinstrument were encouraging. However, since theirsample was quite small (28 respondents), these authorscalled for care in the interpretation of the results and formore assessment efforts.

Relying on system success literature, Amoroso andCheney [ 11 focussed on end-user information satisfactionand on application utilization as components ofapplication quality. As a result of their study, theypropose two instruments which measure intended andactual utilization patterns, as well as user informationsatisfac tion. The instruments were found to havereasonably high levels of reliability and validity.

Keeping in mind the perspective adopted by these twostudies, that is, that the end-user perspective is ofparamount importance in measuring application quality,this study pursued further the efforts made by Rivard et al.Following Churchill’s [lo]“menda t i ons on constructdevelopment and validation, the instrument proposed byRivard et al. was reexamined in light of the results of thepilot test they had conducted as well as of the softwaremetr ia and the IS literature. During this exercise, adocument recently published by the InternationalOrganization for Staudardization QSO) with the objectiveof providing a standardized ramework for the examinationof software quality, was found to be most useful [25].This examination lead to a refinement of the measurewhich is presented in Figure 1. The proposed view ofquality includes eight dimensions: reliability,effectiveness, portability , economy, user-friendliness,understandability, verifiabili ty, and maintainabili ty. Agiven dimension is measured via a number of criterionvariables which, in turn, are measured through a series ofitems. Each dimension, along with the criteria thatcompose it, is presented below; the Figure also indicatesthe number of items used for assessing each criterion.

Reliability. Software reliability relates to the ability ofan application to perform its intended functions with

appropriate precision [ 2. 9,321. According to the ISO,reliability refers to “a set of attributes that be r on thecapability of software to maintain its level of performanceunder stated conditions for a stated period of time” [25].From the literature, five criteria were found to contributeto the reliability of an application; they are: security,integrity, coherence, functionality, and absence of errors.

Effic iency. Efficiency is appropriately defined by theI S 0 as “a set of attributes that be r on the relationshipbetween the level of performance of the software and theamount of resources used, under stated conditions” (p.4).This is consistent with the view shared by several authors[9.24]. The IS0 further breaks down efficiency into twocomponents: time behavior and resource behavior. Timebehavior is related to response time and throughput rates;resource behavior rders to the amount of resources requiredto perform the function.

523



Figure 1User-developed application quality

Portability. While some authors refer to the degree ofportability of an application as its ability to run ondifferent computers [ll] . others talk about the effortrequired to trausform a program from a given configurationto another [9]. In a more general manner, the I S 0 defiesportability as the set of attributes “that be r on he abilityof software to be transferred from one environment toanother” (p.4). Ge ne dh bi li ty and adaptability are twokey criteria for assessing portability [25].

Economy. The cost-benefit aspects of an application areparticularly relevant to the context of end-user computing.While it is not often cited in the context of large softwareprograms [39], the economy dimension is central tomanagerial concerns in the case of user-developed

applications [7. 351. Following Rivard, Lxbrun andTalbot, it is defined here around a single criterion, that ofprofitability

User-friendliness. User-friendliness is critical in thecontext of end-user computing. It is defined here as theease of learning how to use a system, how to operate it,how to prepare the input data, to interpret the results, andto recover from errors [2,3,28,39.46]. Three criteriaemerge from the literature; they are: accessibility, helpfeatures. and ease of use.

Understandability. Understandability is the extent towhich one can understand what an application does, itsstructure, and its modules [37]. Understandability can be

524



assessed through criteria such as modularity, conciseness.clarity, uniformity, structuredness. and infonnativeness.

Verifiability. Verifiability is defined as the ease oftesting the application, to ensure it performs ts intendedfunction [2,8,9,20.32]. The main criterion to assess

verifiability is that of testability.

M a i n t a i n a b i l i t y. Maintainability is “a set ofattributes that bear on the effort needed to make specifiedmodifications” (p.4). The dimension of maintainabilitycomprises three main criteria: modifiability, flexibility,and compatibility [2,3,8].

Methodology

Using as a firs t pool of items those found in Rivard etal.’s questionnaire, an nstrument was developed. While afurther examination of the literature suggested theinclusion of new items, it also suggested to discard someof the items from the former questionnaire. As indicatedin Figure 1. the resulting instrument includes 88 items. Itwas pretes ted with nine users, with various backgroundsand responsibilities, in three organizations. Results of thepre-test indicated that the items were clearly formulated,and that they were well understood by the respondents.

Assessment of the measurement properties of theinstrument was performed via a field study of 22 Quebecorganizations. This convenience sample was selectedthrough personal contacts. The targeted respondents of thesurvey were of four types: user, user-developer, developer,or mauager of end-users. The applications had to have been

developed by end-user personnel, that is, personuel outsidethe IS department, without formal computer science

education, such as college or a university degree incomputer science. Partic ipants who had received basictraining on commercial software only (e.g. Dbase. Lotus .be it in-house courses or public types of seminars, wereconsidered end-users. No selection criterion was applied tothe type of applications, as long as their developer met theend-user selection criteria. Preliminary phone calls weremade to the IS directors to explain the project, solicit theirparticipation, and determine the number of end-users whowould receive the questionnaire. They were asked todistribute the questionnaire to end-users working with auser developed application, without discrimination withrespect to the perceived overall quality of the application.While it is possible that only the most satisfied userscompleted the questionnaire, this was beyond theresearchers’ control; yet, it was estimated that this wouldnot significantly affect the validity of the study.

Copies of the questionnaire were sent to the IS managers,with return envelopes and a cover letter explaining thepurpose of the research. Respondents returned thequestionnaire directly to the researchers. A follow-up (bymail) was made four weeks after the f i t mailing.

A total of 292 questionnaires were distributed., and 117were returned, out of which 110 were usable for dataanalysis purposes. The response rate was 100 percent forthe organizations contacted, and 38 percent for therespondents. Respondents’ experience with end-usercomputing differed widely; 25 percent had never developedan application, 55 percent had developed from one to tenapplications, and 20 percent had developed more than en.

The applications were developed with B a s e (18 of allapplications). Lotus (16 ), SAS (14 ). Dataease (9 ).Image (4 ), Foxbase (3 ). DB2 (1 ), and other end-usertools (35 ). The types of applications were as follows:inventory management (16 ). data entry (14 ).scheduling 11 ). accounting 11 ), file management(10 ). budgeting (9 ). customer management (7 ),statistical analysis (6 ). printer management (6 ). phonemauagement (6 ) and time management (4 ).

Results and discussion

While the development of the measure was done in a top-down fashion, starting with the UDA quality factors,followed by the criterion variables for each factor and thenby the items for each variable, the assessment of themeasurement properties was performed bottom-up.confirmatory factor analysis approach was chosen, usingthe partial least squares technique I‘M. [45]), as opposedto an exploratory approach using principal components

analysis. This choice was based on the prioridevelopment of the dimensions of user-developedapplication quality, and the need to test the hypothesizedfactor structure [5]. Compared to the other widely used“second generation” multivariate analysis techniqueLISREL, PLS s more suited for causal-predictive analysisemphasizing theory development. LISREL is morerecommended when the objective is to confirm that atheoretical model fits observed data [6]. Nonetheless,given the methodological nature of the research objectives,PLS was preferred here because of its greater robustness,that is, not requiring multivariate normally distributed dataand large samples [21].

Internal Consistency. The first measurementproperty to be assessed is the internal consistency ofoperationalization. This involves two related issues,namely unidimensionality and reliability [43]. The

525



multiple items that measure a criterion variable mustnecessarily be unidimensional if this variable is to betreated as a single value. The same c n be said, at a higherlevel, for the multiple criteria that underly a quality factor,that is, assessing that all criteria measure the underlyingconstruct UDA quality factor) to which they aretheoretically related. Unidimensionality is assessed in PLSby examining the loadings of the measures on theirassociated construct, using 0.5 as a cut-off point,following Rivard and Huff [3q.

Measures are reliable if they are free from random error.Reliabil ity is usually assessed by Cronbach's alphacoefficient, with a value greater than 0.7 e d cceptable[31]. However, this coefficient is calculated with theassumption that all measures weight equally on theircorresponding construct. A less restric tive definition ofreliability, the rho coefficient, is based on the ratio ofconstruct variance to the sum of construct and errorvariance [ U ] rho value greater than 0.5 indicates that

the construct variance accounts for at least 50 of themeasurement variance. Another reliability criterion is theaverage extracted variance, which should be 0.5 or more,that is, measures should am u n t for at least 50 of thevariance in their corresponding construct [22]. oth thereliability coefficient and the average extracted variane arecomputed with the loadings obtained from PLS [23], sshown in the bottom of Table 1

A PLS run was first done for each of the 23 criterionvariables, using item scores. Items were selected forremoval on the basis of their not loading at the 0.5 levelon their corresponding variable, or not providingacceptable values for the alpha and rho coefficients. and forthe average extracted variance. As shown in Table 1, this

resulted in the elimination of five criterion variables out ofthe original twenty-three, namely a c c e s s i b i l i t y,conciseness, clarity, modifiabilityand j l a b i l i ty and inthe deletion of a number of measurement items for theremaining variables. The results for the deleted variablesmight be explained by the fact that many applications aredeveloped by end-users for strictly personal ends, to answer

ad hoc needs which are very specific [42]. Theseapplications are often seen by the user-developer as beingonly temporary or 'throw-away , and thus not requiringthat comments be inserted in the code, that the applicationbe properly documented, or that other end-users and futuremodifications be taken into account.

Unidimensionality of the eight UDA quality factors wasassessed by doing a PLS run with the scores of theeighteen remaining criterion variables, obtained byaveraging the items remaining in each. The loading pattern

526

matrix presented in Table 2 globally confirms thehypothesized factor structure, with each variable save oneloading at 0.5 or more on ts associated factor. The oneexception is functionality whose loading on the reliabilitydimension was only 0.23 whereas it loaded on theeffectiveness dimension at the 0.44 level. As this criterionrefers to the completeness of outputs (reports) and inputs(data base), a case could be made that this is a featureshowing an application being effective rather th n reliable.

An alternative assessment of unidimensionality was madeby correlating a criterion variable with its correspondingfactor (whose modified score was computed by excludingthe variable in question). The right-hand column of Table 2 hows acceptable levels for all item-total correlations

0.30 r more [26]). with the exception of functionality.Note that some variables loaded on more than one factor,especially user-friendliness and understandability whichmight in fact constitute a single dimension. The two left-hand columns of Table 3 rovide the reliability indicators

for each factor. While the alpha value for portability anduser-friendliness is low, the rho coefficient is satisfactoryfor all dimensions of UDA quality. Given that this lastindicator is more appropriate in the context ofconfirmatory factor analysis, the eight dimensions of thehypothesized factor structure are udged here to be reliable.

Discriminant Validity. Given the hypothesizedfactor structure, discriminant va lidity is the degree towhich the UDA quality factors are unique from each other,or measure distinct concepts. It is assessed by examiningthe correlations between pairs of factors. The sharedvariance between two factors (the squared correlation)should be inferior to the average variance extracted by thecriterion variables that underlie each factor [22].

As shown in right-hand side of Table 3, all of the UDAquality factors satisfy the aforementioned test ofdiscriminant validity. There is nonetheless some sharedvariance between reliability and verifiability 0.28). ndbetween user-friendliness and understandability 0.27) s

was alluded to in the preceding assessment ofunidimensionality, but not enough to warrant questioningthe fundamental discriminating power of the eight factors.

Convergent Validity. A construct is shown to haveconvergent validity if there is measurement consistencyacross multiple operationalizations. The results ofmeasuring the same trait with different methods shouldthus correlate with one another [26]. or this purpose,three scales were added to the instrument, measuring theoverall quality of the user-developed application, thequality of the application's components (menus, screens,



Table 1Internal consistency assessments (PLS) of the criterion variables (N=llO)

QUALITY FACTOR initial final

Criterion n . o f alphaa ho AEVc n.of alpha rho EV

variable items items

RELIABILITY

Integritycoherence

Functionalityk - f r e e

security

EFFEcl-IvENEssM O l l E l U C X

Efficiency

PORTABILITY

GeneralizabilityAdaptability

ECONOMYProfitability

USER-FRIENDLINESSEase of useHelp feahmsAccessibility

UNDERSTANDAE3ILI

ConcisenessModularity

Clarity

UniformitystructurednmsInformativeness

VERIFIABILITYTestability

MAINTAINABILITYModifiabilityFlexibilityCompatibility

65475

12

11

4

823

532

667

2

332

.42 .55 .45

.73 .82 .48

.46 .69 .43

.52 .36 .26

.71 .76 .47

--- 1.00 1.00.48 .79 .66

.71 .82 .54

.62 .53 .29

.94 .97 .95

.37 .71 .45

.84 .89 .62

.24 .66 .40

.36 OO .61

.53 .71 .33

.61 .64 .35

.75 .80 .42

.66 .85 .74

.44 .72 .48

.36 .70 .44

.65 .85 .74

35324

12

11

4

620

500346

2

002

84 .90 .76.73 .82 .48.63 .79 .56

.67 .82 .51

.74 84 .57

--- 1.00 1.00.48 .79 .66

--- 1.00 1.001.00 1.00--

.71 .82 .54

.73 .82 .43

.94 .97 .95

84 .89 .62---_ ---- ----

----

.57 .75 .44

.62 .79 .48

.78 .85 .49

.66 .85 .74

.65 .85 .74

527

~



Table 2Unidimensionality assessment (PLS) of the quality factors (N=llO)

QUALITYCriterion

variable

FACTOR loading patte€n matlixa

1 2 3 4 5 6 7 8 rb

1. RELIABILITYsecurityIntegrityCoherenceFunctionality

Error-free

2.EFFEcTIvENEssPerformanceEfficiency

3. PORT BI LI TYGeneralizabilityAdaptability

4. ECONOMYProfitability

5. USER-FRIENDLINESSEase of useHelp features

6. UNDERSTANDABILITYModularity

S t r u ~ e d n e s s

Informativeness

Uniformity

7. VERIFIABILITYTestability

8. MAINTAINABILITYCompatibility

- - 1 o.50 - 1.0 - -

- - 1.0 - 1 oSO - - -

aA dash indicates that the loading is inferior to .5bcorrelation of the criterion variable with the quality factor pe.001 for all save for Functionality which is non significant)

reports, tc.) and the quality of the application c o m p a d toall others used.

The results of correlating these scales with each qualityfactor are presented in three middle columns of Table 4.One can see that most correlations are positive and

significant, but not in the case of ver ifiability andmaintainability which s e e m to be unrelated to the threeoverall quality criteria These are he two most technicaldimensions of quality and as such, would preoccupycomputer professionals but probably not typical end-usersu n l e s s they have been trained to do so [29].

528



Table 3Reliability and disc riminant validity assessments of the quality factor s (N=1 10)

Quality factor

alpha8 rhob squ redcorrelation matrix=

1 2 3 4 5 6 7 8

Reliability .61 .78 .48Effectiveness .57 .83 OS .71Portability .40 .78 06 .10 .64Economy -- 1.0 .04 .23 .13 .99

Ullderstandability .73 .82 .23 .02 .07 06 .27 56

Verifiability - 1 O .28 O 1 .02 .01 06 .04 .99Maintainability - 1 O O S .03 06 09 09 .OO .01 .99

Cronbach's alpha computed from the criterion variablesbReliability coefficient computed from the criterion variable loadingscSubdiagonals= s h a d ariance between actors R2)

Diagonal= averagevariance

extracted by the criterion variables

USer-friendlineSS .29 .76 .14 .04 .08 .03 .6 1

Table 4Internal consistency (PLS), convergent and nomological validity assessments (N=l 10)

loading correlations

Quality factor QUALlb QUAL2c Q U U d SATISF.e

ReliabilityEffectivenessPortabilityEconomy

user-fliendlinessUnderstandabilityVediabili yMaintainability

.74

.55

.60

.59

.68

.6448

.48

.56**

.43***

.36***

.32***

.30***

.47***

.3 1***

.32***

.09***

.33***

.21*

.23**

.24**- . lo

08

.17*.36***.29* *.20*

.29***

.34***-.04

.07

.27**

.35***

.34***

.24**

.23**

.25**-.01,15

.23**s.22**.19*

.32***

.31***.16*.03

APPLICATION QUALITY -- --- .29*** .32*** .36*** .39***

Cronbach's alphaReliability coef. (rho)Average extracted var.

= .70 (computed from the quality factors)= .82 (computed from the factor loadings)= .36 computed from the factor loadings)

Correlation of the quality factor with application qualityboverall quality of the application 1 s d e )CQuality of the application's components 1 scale)Quality of the application compared to all others used 1 scale)eEnd-user satisfaction (Doll and Torkzadeh, 1988.12 scales)

529



Nomological Validity. A construct exhibitsnomological validity when its predicted relationships withother constructs within a theoretical framework areconfirmed. The centr l aspect of nomological validation isascertaining predictive validity, whereby the constructunder study is related to a single antecedent or consequent

[4]. Redictive validity was tested by correlating each UDAquality factor with end-user satisfaction, assumed here toresult from application quality, using Doll and Torkzadeh’s[19] 12-scale measure (alpha=O.93).

The right-hand column of Table 4 indicates that alldimensions of quality are positively and significantlyrelated to end-user satisfaction, except for maintainability.Again, this feature could be seen as irrelevant in manycases User attitudes could thus be quite impervious to themore technical dimensions of application quality such as

maintainability and verifiability.

The Application Quality Construct. At the toplevel, the measurement properties of UDA quality as asingle construct was then assessed by making a final PLSrun using the scores of the eight quality factors, obtainedby averaging the criteria corresponding to each. The twoleft-haud columns of Table 4 provide assessments of theunidimensionality of the construct, including the factorloadings and item-total correlation. The lowest loadingswere those of verifiability 4) nd maintainability 4)

while all other values were satisfactory. The inclusion ofthese two factors in the UDA quality measure should thusbe subject to future question and revision.

A global score was obtained by averaging the eight factorscores. Evidence is found at the bottom of Table 4 for thereliability of the global construct (rho=0.82). its

convergent validity and its predictive validity, as itscorrelations with the three overall quality scales and withend-user satisfaction are all positive and significant. Yet,the fact that the eight factors extracted only 36% of thetotal variancecould be an added indication that the moretechnically-oriented factors constitute a dimension apartfrom the others. Future work on the instrument shoulddelve more deeply into this question.

Conclusion

With the increasing pressuresmade on organizations so thatthey better perform, it is likely that we will continue towitness the growth in the number of applicationsdeveloped by end-users. Along with this growth, the needfor better instruments for assessing these applications willalso increase. The measure proposed here is readily usableby end-users, and it can help them better assess their

applications. Having gone through a quite thoroughdevelopment and assessment process, i t can be qualified asboth reliable and valid, and as such may be used withconfidence. However, further refinements will be requiredto improve the measurement of the more technical aspectsof an application, such as Verifiability and maintainability.

A greater reliance on software engineering met ria mightbe quite helpful in that regard. furthermore, instead ofusing self-report evaluations of the quality of theapplications, it could be nteresting to have independentevaluators involved in the process.

References

[ l ] Amoroso. D.L.. Cheney, P.H.. “Quality End UserDeveloped Applications: Som e Essen tial Ingredients.”Data

[2] Andriole, S.J.. “Quality Assurance Through Verification,”S o f t w a r e Va l i d a t io n . V e r i f i c a t i o n , Te s t i n g . a n dDocumentation Petrocelle Books. Princeton,1986, pp.81-88.- Measuring Programmer Productivitv andSoftware Oualitv, John Wiley,1985.

[4] Bag oui. R.P. “An Examinationof the Validity of TwoModels of Attitude”. Multivariate Behavioral Research, 16,

[5] Bagozzi. R.P. ”A Holistic Methodology for ModelingConsumer Response to Innovation”, ODerations Research,3 1,1983, pp. 128-176.

[6] Barclay. D.. iggins. C.A., Thompson, R.L.. “The PartialLeast Squares (PLS) Approachto Causal Modeling: PersonalComputer Use as an Illustration,” Working Paper MIS-90-0007-3, School of Business Administration, UniversityofVermont, 1993.

[7] Bergeron, F.. Rivard, S., Raymond, L.. “Assessment ofEnd-User Computing from an Organizational Perspective”.Information Resources Management Journal, (6: 1). Winter1993, pp.14-25.

[8] Boehm, B.W., Brown. J.R.. McLeod,G., Lipow, Menit. M..“Characteristicsof Software Quality.” TR W Series on SoftwareTechno loey, North-Holland, Amsterdam, 1978.

(23: 1). Winter 1992, pp.1-12.

1981, pp. 323-359.

[9] Cavano, J.P.. McCall, J.A., “A Frameworkof SoftwareQuality,” Proceedingsof the Software Oualitv and AssuranceWorkshop, San Diego, Califomia.1978. pp. 133-139.

[lo] Churchill. G.A. Jr., “A Paradigm for Developing BetterMeasures of Marketing Constructs.” Journalof MarketingResearch,Vol. XV. February 1979. pp.64-73.

[ l l ] Cho, C.K., A n Introduction to Software Oualitv Control,Wiley Interscience, New York,1980.

[12] Claudle, S. L., Gorr, W.L., Newcomer, K.E., “KeyInformation Systems Management Issues for the PublicSector”. MIS Ouarterlv, (152 ). June1991. pp. 171-188.

[13] ComDuterworld, The Premier 100, September 12.1988,pp. 10-17.

[14] Computerworld, The Premier 100, September 11, 1989.pp. 10-17.

53



[la ComDuterworld, The Premier100. October 8 1990. pp.

[16] Computerworld. The Remier100, September30. 1991.

18-25.

pp. 1421.

[ 17 ] C o m D u t e r w o r l d . F o r ec a s t 1992. December 23 ,19911January 2. 1992. p. 1.

[18] Com uterworld The Premier100, September 14. 1992.pp.

19] Doll, W.J.. Torkzadeh. G. The M easurement of E nd-UserComputing Satisfaction . MIS Quarterly. 12, 2, September1988. pp. 259-274.

[20] Farbey. B.. Software Quality Metrics: Considerationsabout Requirements and Requirement Specifications,Information and Software T echnoloav. (3 2 1). January-February 1990.

[21] Fornell. C.R. and Bookstein. F.L. Two StructuralEquation Models: LIS= and PLS Applied to ConsumerExit-Voice Theory . Joumal of Marketing Research,19. 1982,pp. 440452.

[22] Fomell. C.R. an d Larcker. D.F. Structural EquationModels With Unobservable Variables and Measurement Error ,Joumal of Marketing Research, 18,1981. pp. 39-50.

[23] Fomell, C.R. Tellis G.J. nd Zinkhan . G.M. ValidityAssessment A Structural Equations Approach Using Partialleast Squares . AMA Educator's Proceedinas, Chicago, 1982.pp. 405409.

[24] Gilb. T.. Softw are Metrics, Winthrop Publishers,Cambndge. 1977.

[25] International Organization for Standardization,Information Technology Software Product Evaluation-

Quality Characteristics and Guidelines for their Use, UDC683.3.06.006.83. 1990.

[26] Kidder, LH . Research M ethods in Social Relations, Holt,Rinehart and W inston, New-York,1981.

[27] M c h n . E R . End-Users As Appl ica tion Developers .MIS Ouarterlv,3.3. December 1979. pp. 37-46.

[28] Meyer. K.. Harp er, M.. User Friend liness, MISOuarterly, (81). March 1984, pp.1-3.

[29] Nelson, R. R Educational Needsas Perceived by IS andEnd-User Personnel . MIS Quarterlv. 1 5.4 , December1991.pp. 503-525.

[30] Niederman, F., Brancheau. J.C.. Wetherbe. J.C..Information Systems Managem ent Issues for the1990s . MIS

Ouarterlv. (154). pp. 475500.

[31] Nunnally. J.C. Psvchometric Theory, McGraw-Hill. New-York. 1978.

[32] P lans, J.. La aualit6 informatiaue. Dunod.Paris, 1988.

[33] Petrova, E. Veeve rs, A., Role of Non-Stochastic-BasedMetrics in Quantification of Software Reliabil i ty.Information an d Software Techn oloav. (32: 1). January-February 1990. pp.71-77.

[34] aymond, L., Bergeron. F.. Gingras. L. ivard. S.. Laqualit6 des app lications de la micro-informatique: lesutilisateurs sont-ils de bons concepteurs? Gestio n, (13:4).

[35] Rivard, S.. Huff, S L An Empirical Studyof User AsApplication Developers . Information and Management, (82)February 1985. pp. 89-102.

1988, pp.3440.

r361 Rivard. S. and Huff. S.L. Factors of S uccess for End Userkomputing', Communications of the ACM.31.5. May 1988.pp. 552-561.

[37l Rivard. S.. Lebrun. P. Talbot, J.. Measu ring the Qualityof User-Developed Applications , Proceedinas of the 24thHawaii Intemational Conference o n Svstems Science. January

[38] Robillard. P.N.. Cou al. D.. Coallier, F.. ProfilingSoftware T h u g h the Use ofMetrics, Software Practice and

Exuer ience , ( 2 1 3 May1991. pp. 507-518.[39] Schneiderman, B.. Software Psvcholo w: Human Factorsin Computer and Information Svstems, Winthrop Publishers,Cambridge, 1980.

[40] Sipior, J.C.. S anders, G.L.. Definitional Distinctions andImpl ica t ions for Ma naging End User Comput ing .Information and Manapement, (16 3). March1989. pp.

[41] Sullivan-Trainor, M.. Dix. L.Z.. Winn ing the PaperChase , Computerworld, (2 7 1). December28. 19921January4. 1993. p. 39.

[42] Sum ner, M. User-Deve loped Applications: Wh at AreThey Doing? , Joumal of Information Svstems Manaaemenf'3.4. Fall 1986, pp. 3746.

1991. pp.471-478.

45-123.

[43] Venkatraman. N. and Grant, J.H. Construct Measurem entin Strategy Research: A Critique and Proposal . AcademvofManagement Review, 11. 1986. pp. 71-86.

[U] Werts. C.E. Rock, D.R. Linn.R L and Joreskog. KB . AGeneral Method of Estimating the Reliabil i ty of aComposite . Educational and Psvcholopical Measurement,38,Winter 1978. pp. 933-938.

[45] Wold, H. Soft Modeling: Th e Basic Design and SomeExtensions , in Svstems Under Indirect Observation. Joreskog.K.G. nd Wold, H. Eds.). North-Holland, Amsterdam,1982.

[ a ] Zloof. M.M.. Design Aspectsof the Query-by-ExampleData Base Management Language. in Databases: ImprovingUsability and Reswnsiveness. B. Schneiderman.Ed. academicPress, New York, 197%.

pp. 1-54.

531

Development of a Measure to Assess the Quality of User-Developed Applications

Documents

Transcript of Development of a Measure to Assess the Quality of User-Developed Applications