EJISE Volume 11 Issue 3

ISSN 1566-6379 167 ©Academic Conferences Ltd Reference this paper as: Brown, I. and Jayakody, R. “B2C e-Commerce Success: a Test and Validation of a Revised Conceptual Model.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 167 - 184, available online at www.ejise.com

B2C e-Commerce Success: a Test and Validation of a Revised Conceptual Model

Irwin Brown and Ruwanga Jayakody University of Cape Town, South Africa [email protected] Abstract: Since the advent of the Internet, B2C e-Commerce has grown substantially across the globe. Whilst much research has examined factors influencing adoption of e-commerce, not as many studies have investigated the post-adoption phenomenon of success. Those studies that have investigated IS success and the extensions required to accommodate e-commerce have mainly been conceptual. Few have attempted to test and validate the models empirically. The purpose of this study was to fill this gap. By drawing from the technology acceptance model, expectation-confirmation theory and IS success theory, a revised conceptual model was derived. The model and relationships were tested and validated using data gathered from 166 online consumers in South Africa. 7 interrelated dimensions of B2C e-commerce success were confirmed, namely service quality, system quality, information quality, trust, perceived usefulness, user satisfaction and continuance intentions. Direct relationships between dimensions were identified. These showed that user intentions to continue using an online retail site are directly influenced by perceived usefulness, user satisfaction and system quality. User satisfaction is directly influenced by service quality and perceived usefulness, whilst perceived usefulness is directly influenced by trust and information quality. Trust in the online retailer is directly influenced by service quality and system quality. The implications of these and other findings are discussed. Keywords : IS success; e-commerce success; B2C e-commerce; DeLone and McLean

1. Introduction

E-commerce is generally accepted as being a sub-set of e-business (Kim et al., 2006; Pavic et al., 2007). E-business activities and applications range from simple email to e-enabled supply chain management (Fusilier & Durlabhji, 2003; Parker & Castleman, 2007). E-commerce on the other hand is more narrowly defined in terms of the purchasing and selling products or services through the medium of the Internet (Grandon & Pearson, 2004). Business-to-consumer (B2C) e-commerce, the focus of this paper, refers specifically to the activity in which consumers buy products or services using the Internet medium (Pavlou & Fygenson, 2006). Despite the dot-com failures of the 2000s, B2C e-commerce has continued to grow steadily (Van Slyke et al., 2004). This growth has occurred not only in the developed countries, but in the developing as well, resulting in an increasingly global community of online shoppers (Cyr et al., 2004). South Africa is no exception in this regard. Goldstuck (2002, 2004) and World Wide Worx (2006) have reported consistent increases in South African online retail sales revenue (excluding airline tickets) over the past few years, rising from R 162 million in 2001 to R 514 million in 2005. This is indicative of a relatively healthy B2C e-commerce environment, which is even healthier if sales of airline tickets are also considered (World Wide Worx, 2006). Companies are making large investments in e-commerce applications but are hard pressed to evaluate the success of their e-commerce systems (DeLone & McLean, 2003). This growth in e-commerce is the reason behind recent attempts to measure the success of e-commerce. Much of the discussion on B2C e-commerce success has been based on earlier research on IS success by DeLone & McLean (1992). An example of this is the e-commerce success model developed by Molla & Licker (2001). Grounding conceptualizations of e-commerce success in IS literature is perfectly valid given that for all intents and purposes B2C e-commerce systems are information systems that have been extended for direct use by consumers (DeLone & McLean, 2004; Garrity et al., 2005; Pather et al., 2004). There is often a lack of conceptual clarity as to the theoretical basis for relationships between dimensions of B2C e-commerce success this being as a result of the conceptual weaknesses in the foundational IS success models (Seddon, 1997). Establishing the basis for these relationships and clarifying their nature is therefore an important endeavour (DeLone & McLean, 2003, 2004). There have been surprisingly few attempts at empirically validating and testing models of B2C e-commerce success such as that developed by Molla & Licker (2001) and DeLone & McLean (2003). Alternative e-commerce success models have been suggested. Torkzadeh & Dillhon (2002) identified a set of success factors for e-commerce. Unlike Molla & Licker (2001) and DeLone & Mclean (2003), no relationships between factors were identified. Garrity et al. (2005) too developed an e-commerce success model, but this tended to view success primarily in terms of dimensions of satisfaction. Quaddus & Achjari

mailto:[email protected]

Electronic Journal Information Systems E valuation Volume 11 Issue 3 2008 (167-184)

(2005) developed an e-commerce success framework that reflected an organizational perspective. In this study, a customer perspective was being sought, and so the DeLone & McLean (2003) model and its variants were deemed to provide a more appropriate base. The overall purpose of the study was to test and validate a revised conceptual model of B2C e-commerce success. The detailed objectives were to:

Re-examine the relationships between key dimensions of B2C e-commerce success in the light of established theories, and to develop a revised model;

Empirically validate and test the model using data gathered from a sample of online consumers in South Africa;

Add to the body of knowledge on B2C e-commerce in South Africa. Whilst there have been several major South African academic studies examining e-commerce success from an organizational perspective (Molla, 2002; Pather et al., 2006), few have investigated the consumer perspective. An exception is De Villiers & Van Der Merwe (2001). This study will build on the work of De Villiers & Van Der Merwe (2001).

In the next section, the conceptual background of the study is laid out and the research model is developed. The research methodology is then outlined. The data analysis and results are reported before these results are discussed and implications drawn. The limitations of the study are noted and ideas for further research put forward. Finally the paper is concluded.

2. Conceptual background

Theory in the field of IS success has stemmed primarily from the seminal work of DeLone & McLean (1992). Numerous studies have since sought to extend and/or validate this framework. A summary of some key studies and variables examined are shown in Table 1 below. The table shows 11 dimensions of IS success that have been identified. The dimensions of system quality, information quality and user satisfaction have been included in all of the studies. This indicates that they are central to measuring IS success and should be included in any success model. Seddon & Kiew (1996) attempted to partially validate the DeLone & McLean (1992) model and in so doing suggested the inclusion of perceived usefulness as a replacement to use, given that use is primarily a behaviour, and not reflective of success in contexts where usage is mandatory. Individual impact and organizational impact were included in the original DeLone & McLean (1992) model, but not in most subsequent studies. DeLone & McLean (2003) in a 10-year update of their 1992 IS success model suggested that individual, organizational and other impacts be collapsed into a single net benefits construct for the sake of parsimony. Rai et al. (2002) empirically tested and validated both the DeLone & McLean (1992) and Seddon (1997) model, but excluded the individual impact and organizational impact dimensions. Iivari (2005) also attempted to validate the DeLone & McLean (1992) model. In so doing he demonstrated that individual impact could be assessed with a perceived usefulness measure. He did not include organizational impact as a dimension. Molla & Licker (2001) conceptualized a model of e-commerce success by drawing from the work of DeLone & McLean (1992). In addition to information (or content) quality, system quality, use and user (e-commerce customer) satisfaction, they also highlighted support and service (service quality) and trust as additional factors to consider in a B2C e-commerce environment. They noted too that in a B2C e-commerce environment usage of a system is typically voluntary, unlike in work situations where usage of a system may be mandatory. DeLone & McLean (2003) in their updated model included the core dimensions of information quality, system quality and user satisfaction, as well as use/intentions to use, net benefits and service quality. They suggested that intention to use may be employed as an alternative to use as a success dimension. DeLone & McLean (2004) illustrated how the updated model could without modification be used to evaluate e-commerce success.

www.ejise.com ©Academic Conferences Ltd

168

Irwin Brown and Ruwanga Jayakody

Table 1: Dimensions of IS Success

Dimension D&M (1992)

S&K (1996)

Seddon (1997)

M&L (2001)

Rai et al. (2002)

D&M (2003/4)

Iivari (2005)

System Quality X X X X X X X Information Quality X X X X X X X User Satisfaction X X X X X X X Use X X X X X X Individual Impact X X X Organisational Impact X X Perceived Usefulness X X X [X] Net Benefits of Use X X Service Quality X X Trust X Intention to Use X Key: D&M – DeLone & McLean; S&K: Seddon & Kiew; M&L: Molla & Licker

Intention to use is primarily a pre-adoption construct (Davis, 1989). It gives an indication of users’ intentions concerning future use of an information system. It is thus not entirely suitable for post-adoption studies, where attempts are made to evaluate the success of an information system (Bhattacherjee, 2001). Bhattacherjee (2001) makes the case for assessing user intentions to continue using a system (continuance intention). Such a measure is more suited to post-adoption studies such as the investigation of IS success. This construct is furthermore appropriate as a success measure in the e-commerce context, where repeat customers are highly valued. Bhattacherjee (2001) found that perceived usefulness, user satisfaction and loyalty incentives were influences on continuance intention. In the next section, a research model is developed by drawing out the salient dimensions of e-commerce success and the relationships between them.

3. Research model

Based on the discussion in the conceptual background section above, the factors that are important in the context of B2C e-commerce systems include the core IS success measures of system quality, information quality (also referred to as content quality in e-commerce environments) and user satisfaction (also referred to as customer satisfaction in e-commerce environments) (Molla & Licker, 2001). Additional factors relevant to e-commerce specifically are trust, service quality (or support and service), loyalty incentives and continuance intention (Bhattacherjee, 2001; Molla & Licker, 2001). As suggested by DeLone & McLean (2003) [continuance] intention was chosen as an alternative to use, given the difficulties of measuring general e-commerce use. Use therefore was not included in our research model. For the same reason net benefits was not included in our research model, but rather perceived usefulness, which is a valid, easily assessable perceptual measure of net benefits (Iivari, 2005; Rai et al., 2002; Seddon, 1997). The relationships between these 8 chosen factors will now be discussed by examining each in turn.

3.1 Continuance intention

Research into IS success has been concerned with the quest for the dependent variable (DeLone & McLean, 1992). DeLone & McLean (1992) initially proposed the organizational impact of an IS as the ultimate dependent variable, preceded by individual impact. In the 10-year update, DeLone & McLean (2003) combined these into a net benefits construct and argued for this as the dependent variable. Given the difficulties associated with assessing usage and net benefits of use as success measures, continuance intention provides an appealing yet credible alternative as the ultimate dependent variable. The focus in this study is on how the various independent variables ultimately impact directly and indirectly on it. As a consequence only unidirectional relationships are considered. Consistent with many mainstream studies in IS, null hypotheses are not expressed (Bhattacherjee, 2001; Gefen et al., 2003; Iivari, 2005).

3.2 User satisfaction

User Satisfaction is the most general perceptual measure of information systems success (Seddon, 1997). In the e-commerce environment it is an important means of measuring customers’ opinions of the e-commerce system (DeLone & McLean, 2003). Customers in this study refer to only those who directly use e-commerce services. Wang et al. (2001) developed a measure of customer information satisfaction with e-commerce websites. The measure had dimensions of customer support, security, ease of use, transactions and payment, information content, digital products and services and innovation. Several of these dimensions

www.ejise.com ISSN 1566-6379 169


overlap with those in IS success models (DeLone & McLean, 2003; Molla & Licker, 2001). For example customer support (service quality), ease of use (system quality) and information content (information quality) appear as dimensions in both success and satisfaction measures. This indicates a lack of conceptual clarity between satisfaction and success in the IS literature (Garrity et al., 2005). As a consequence many studies that incorporate user satisfaction into IS success models employ single item measures for this construct (Rai et al., 2002). Bhattacherjee (2001) identified user satisfaction as a key determinant of whether a user intends to continue using an e-commerce system. The hypothesis suggested is:

H1: User Satisfaction has a positive effect on Continuance Intention for an e-commerce system.

3.3 Perceived usefulness

Rai et al. (2002) in validating the Seddon (1997) model demonstrate that perceived usefulness positively influences user satisfaction with an information system. The hypothesis suggested is: H2: Perceived Usefulness has a positive effect on User Satisfaction for an e-commerce system.

Bhattacherjee (2001) demonstrated perceived usefulness as a key determinant influencing customer intentions to continue using an e-commerce system, justifying the following hypothesis: H3: Perceived Usefulness has a positive effect on Continuance Intention for an e-commerce system.

3.4 Loyalty incentives

Bhattacherjee (2001) drew on expectation-confirmation theory (ECT) in the consumer behavior literature, technology acceptance model (TAM) in the IS use literature and agency theory in the organizational economics literature to propose satisfaction, perceived usefulness and loyalty incentives as the three key factors influencing customers’ decisions to continue using an e-commerce system. Loyalty incentives has not previously been included in IS success models, but based on Bhattacherjee (2001) it is hypothesized that:

H4: Loyalty Incentives have a positive effect on Continuance Intention for an e-commerce system.

3.5 Trust

Gefen et al. (2003) note that there should be a clear distinction between trusting beliefs and trusting behaviours (e.g., usage of an e-commerce system). In this paper trust is viewed as a set of trusting beliefs about e-commerce (Gefen et al., 2003). Molla & Licker (2001) included trust in their model of e-commerce success. Trust in an e-commerce system has several benefits including heightened perceptions of usefulness and greater intentions to use the system (Gefen et al., 2003). The latter argument may also be extended to encompass the impact of trust on intentions to continue using an e-commerce system. Intention to continue using a system is as much a behavioural intention as intention to use. Thus, the hypotheses suggested are:

H5: Trust has a positive effect on Perceived Usefulness of an e-commerce system. H6: Trust has a positive effect on Continuance Intention for an e-commerce system.

Molla & Licker (2001) provide support for the influence of trust on user satisfaction. Where an e-commerce vendor is perceived as trustworthy, this leads to heightened levels of user satisfaction. The hypothesis suggested is therefore that:

H7: Trust has a positive effect on User Satisfaction with an e-commerce system.

3.6 System quality

System quality in an e-commerce context, as with traditional IS, is reflected by usability, availability, reliability, adaptability and fast response time of the system (DeLone & McLean, 2003). According to Seddon (1997) system quality is concerned mainly with the consistency of the interface and the ease of use. As such ease of use features prominently in the operationalisation of system quality (Ifinedo, 2006; Rai et al., 2002; Seddon, 1997). In attempting to understand the relationship between system quality and other dimensions of IS success the TAM can be referred to (Davis, 1989). In the TAM, perceived ease of use has been found to influence both perceived usefulness and behavioural intentions to use a new system (Davis, 1989). Given the prominence of ease of use in the assessment of system quality, it therefore follows that:

H8: System Quality has a positive effect on Perceived Usefulness of an e-commerce system. H9: System Quality has a positive effect on Continuance Intention for an e-commerce system.

The positive impact of system quality on user satisfaction has been recognized and demonstrated in several prior IS success studies (DeLone & McLean, 2003; Rai et al., 2002; Seddon & Kiew, 1996). Molla & Licker


170


(2001) argue that this effect applies equally to e-commerce systems. There is thus ample evidence to support the hypothesis that: H10: System Quality has a positive effect on User Satisfaction with an e-commerce system.

A link between system quality (ease of use specifically) and trust has been demonstrated by Gefen et al. (2003). They posit that through investing time and effort in improving the ease of use and usability of websites, e-commerce vendors demonstrate to consumers their integrity and trustworthiness. Thus, the hypothesis supported is:

H11: System Quality has a positive effect on user Trust of an e-commerce system.

3.7 Information quality

Information quality is highlighted by DeLone & McLean (1992) as an important indicator of IS success. Information quality has received increased attention since the advent of the Internet and World Wide Web (WWW) (Lederer et al., 2000). Molla & Licker (2001) suggest that in the e-commerce context, it should be referred to as content quality, whilst DeLone & McLean (2004) suggest retaining the label information quality. Information and content quality are used interchangeably in this study. Lederer et al. (2000) found information quality to be a major influence on the perceived usefulness of websites, as suggested by Seddon (1997) and as validated by Rai et al. (2002) concerning traditional IS. Rai et al. (2002) also validated the relationship between information quality and user satisfaction as suggested by DeLone & McLean (1992) and Molla & Licker (2001). DeLone & McLean (2003) argue that information quality influences intentions to use a system, which may equally apply to intentions to continue using a system. The hypotheses suggested are therefore as follows: H12: Information Quality has a positive effect on Perceived Usefulness of an e-commerce system.

H13: Information Quality has a positive effect on User Satisfaction with an e-commerce system. H14: Information Quality has a positive effect on Continuance Intention with an e-commerce system.

3.8 Service quality

Pather et al. (2004) note the importance of service quality in the e-commerce context. Service quality (or support and service in Molla & Licker, 2001) is defined as the overall support delivered by the e-commerce service provider (DeLone & McLean, 2003). Molla & Licker (2001) postulate e-commerce satisfaction is affected by the level of support and service (or service quality) provided by the e-vendor. DeLone & McLean (2003) also show this relationship in their updated model of IS success. They further indicate that service quality influences intentions to use a system, which may equally apply to intentions to continue using a system. The hypotheses suggested are: H15: Service Quality has a positive effect on User Satisfaction with an e-commerce system.

H16: Service Quality has a positive effect on Continuance Intention for an e-commerce system.

Harris & Goode (2004) found that service quality plays an important role, not only in enhancing user satisfaction, but as an influence on trust and perceived usefulness. There is therefore support for the following hypotheses:

H17: Service Quality has a positive effect on Trust in an e-commerce system. H18: Service Quality has a positive effect on Perceived Usefulness of an e-commerce system.

The above set of hypotheses can be summarized into a research model, as shown in Figure 1 below.



Figure 1: B2C E-commerce success model

4. Research methodology

The methodology employed in this study was positivistic, quantitative and hypothetic-deductive. Hypotheses were derived from the extant literature on IS and e-commerce success, leading to the development of the framework in Figure 1. A survey instrument was then developed in order to assess and measure the different dimensions of success from an e-commerce user perspective. For the purposes of focusing the study, online retail as a specific type of B2C e-commerce activity was investigated. The data was analysed statistically to validate the instrument and test hypotheses. The instrument design, sample design, data collection procedure and data analysis procedure will be discussed in this section.

4.1 Instrument design

DeLone & McLean (2004) provides the following advice to researchers aiming to evaluate e-commerce success: “Researchers and practitioners should not let themselves be carried away by the hype of the new economy and led to believe that this new and rapidly changing environment requires entirely new measures of IS success. One should look first at the cumulative tradition, and determine which existing and validated success measures can be used in the e-commerce environment. As much as possible, tried and true measures should be enhanced and expanded with modifications or, where necessary, new measures should be considered. Selection of e-commerce success dimensions and measures should be contingent on the objectives and the context of the empirical investigation, but tested and proven measures should be used whenever possible. Completely new and untested metrics should be adopted only as a last resort” (p. 43). In keeping with this advice, measures that were tried and tested in previous IS success or e-commerce success studies were sought. The measures needed to be sufficiently generalisable to various types of B2C e-commerce systems. For example, online retail varies from online grocery shopping to airline ticket purchasing, the frequency of purchase (and therefore e-commerce usage) varying depending on product and context. A detailed search of relevant instruments used by previous researchers was conducted. Based on this search, the instruments shown in Table 2 below were used to create the questionnaire. Slight modification to suit the context of e-commerce was necessary for some instruments. Table 2: Instruments Used to Measure B2C E-commerce Success Dimensions

Measure Number of Items Source System Quality 7 Seddon (1997) Information (Content) Quality 7 Rai et al. (2002) Service Quality (Support & Service) 4 Wang et al. (2001) Trust 6 Suh & Han (2002) Loyalty Incentives 3 Bhattacherjee (2001) Perceived Usefulness 6 Suh & Han (2002) User (Customer e-commerce) Satisfaction 1 Rai et al. (2002) Continuance Intention 3 Bhattacherjee (2001)


172


The questionnaire was divided into two sections. In the first section, the 8 measures in Table 2 were laid out (See Appendix 1). The measures made use of a 5-point Likert scale, anchored by Strongly Disagree (1) at one end to Strongly Agree (5) at the other. All the original measures in Table 2 were designed in this way. The second section gathered general demographic information about the respondents such as age, income, gender, Internet experience and habits, and e-commerce experience. Internet experience and habits were assessed using measures derived from Tan & Teo (2000).

4.2 Sample design

This study was aimed at a population of e-commerce users, specifically online retail customers in South Africa. There were no specific demographic requirements, apart from the fact that respondents were required to have conducted online retail activity via the medium of the Internet. Of the estimated 3.4 million Internet users in South Africa at the end of 2004 (World Wide Worx, 2005), about 41% were estimated to have engaged in e-commerce activities such as shopping online (Webchek, 2004). The demographic profile of the typical e-commerce user in South Africa is thought to be young to middle-aged, university-educated, and affluent (De Villiers & Van Der Merwe, 2001; Brown & Buys, 2005). The gender split is estimated to be only slightly biased towards males. In order to reach a sample of this target population, postgraduate business students at a leading South African university were the primary target group. A few respondents (7) from outside of this group also participated. Postgraduate business students have been used in prior e-commerce studies in recognition that they very often have e-commerce experience (Gefen at al., 2003). In South Africa this is the case too. Brown & Buys (2005) found that this group conformed closely to the profile of the typical e-commerce user in terms of age, education and income. A realistic sample for a positivistic study of this nature was deemed to be in the region of 150 to 250 respondents (De Villiers & Van Der Merwe, 2001; Molla, 2002). This figure was deemed more than sufficient to carry out the envisioned statistical analysis (Molla, 2002).

4.3 Pilot study

Before the major study, a pilot study was conducted. The purpose was to establish the basic soundness of the instrument and to check for any problems related to wording and ambiguity in measurement items. The respondents for the pilot study were 10 IS Masters students. Through the pilot it was recognized that respondents needed to be instructed to base their responses on their most recent e-commerce experience, rather than their general perceptions. Asking questions that relate to the most recent service encounter enabled the researchers to elicit attitudes with respect to a specific experience (Shankar et al., 2003). The instrument made use of existing validated measures, so no pre-tests were conducted to refine the measures.

4.4 Data collection and analysis procedure

For the postgraduate students, questionnaires were distributed just before the end of their formal classes. This method was very effective as the questionnaires were returned immediately after completion. It also ensured a high response rate. In total, 183 questionnaires were returned of which 166 were useable. 17 were unusable, either because they were incomplete or because the respondents indicated that they had no experience of purchasing or shopping from online retail sites. Several statistical tests were employed to analyse the data. Descriptive statistics were used to describe the respondent profile, to ascertain the mean scores for key variables, and to determine correlations between the variables. To test for instrument reliability, the Cronbach alpha was used, whilst factor analysis was employed to determine construct validity. To test the hypotheses multiple linear regression equations were formulated.

5. Data analysis and results

In this section, the demographic profile of the respondents and other descriptive findings will be presented, followed by validation and refinement of the measuring instrument. The hypotheses will then be tested.

5.1 Demographic profile

Appendix 2 shows the self-reported demographic profile of the 166 valid responses in terms of gender, age, education, profession and income. 69% of the respondents were male, with 93% between the ages of 25 – 44. 86% had either an undergraduate and/or postgraduate degree. 69% were managers or professionals,



and 79% were in the upper income brackets, earning more than R 10,000 per month. This profile matches the typical profile of a South African e-commerce user, except for the larger proportion of male respondents. The male bias in the respondent profile was due to the majority of postgraduate business students being male.

5.2 Internet and e-commerce experience

The level of Internet and e-commerce experience amongst the sample is shown in Appendix 3. 72% had Internet access at home. The majority (71%) had over 5 years of Internet experience, but only 14% had over 5 years online shopping experience. Most (89%) used the Internet a few times a day or more, on average for more than an hour at a time. The Internet was mostly used for communications, followed by getting information and product support, downloading free resources, banking, online shopping and entertainment respectively. Respondents were asked what products and services they purchased online. The most common purchases were by category: (1) books, magazines and stationary, followed by (2) DVDs, movies and videos, then (3) music, and (4) flowers, gifts and cards. These results are consistent with those of De Villiers & Van Der Merwe (2001), who also found that books were the most popular product purchased online by South African consumers. The respondents were asked what online retail web site they were referring to in responding to the questionnaire. The most commonly referred to was Amazon.com. Other retail web sites referred to included Kalahari.net (a South Africa retailer similar to Amazon.com), FlySAA (a South African airline), Picknpay.co.za (the e-commerce site of a major South African supermarket chain) and Inthebag.co.za (the e-commerce site of a major South African supermarket chain). This is also consistent with De Villiers & Van Der Merwe’s (2001) findings where they stated: “It should be of concern for local Internet entrepreneurs that Amazon.com, an American site, is by far the most popular site for local Internet users” (p. 13).

5.3 Instrument validity, reliability and refinement

The measures in the survey were derived from existing instruments that had been previously validated in prior IS and e-commerce studies. It was still nevertheless necessary to confirm their validity and reliability. Some refinements were also anticipated, since the context of e-commerce is slightly different to the more traditional IS context where some of the measures had been developed.

5.3.1 Construct validity

Confirmatory factor analysis (CFA) is often used to assess construct validity (Molla, 2002). Since the questionnaire was devised from previously validated instruments, CFA was used to validate and refine the measurement model. The following commonly applied decision rules were used (Wang et al., 2001; Tan & Teo, 2000):

Use a minimum eigenvalue of 1 as a cutoff value for extraction; Delete items with factor loadings of less than 0.5 on all factors or greater than 0.5 on two or more

factors;

Use varimax rotation; Exclude single item factors from the standpoint of parsimony.

There were 8 major constructs identified in the research model of Figure 1, containing 37 measurement items, as shown in Table 2. One of the constructs, user (customer e-commerce) satisfaction contained only a single item and so was not included in the factor analysis. 7 factors were therefore expected to emerge. The iterative sequence of factor analysis and deletion was executed and repeated until distinct factors were identified (Wang et al., 2001). As a result of the CFA, 8 of the original 37 items mentioned in Table 2 were deleted to ensure construct validity. The CFA identified 6 distinct factors, namely Perceived Usefulness/Continuance Intention, Information (Content) Quality, System Quality, Trust, Loyalty Incentives and Service Quality (Support and Service) (See Table 3). The only anomaly that remained was the loading of Continuance Intention on the same factor as Perceived Usefulness. This attests to the strong influence of Perceived Usefulness on Continuance Intention. These constructs were still retained and treated as separate constructs, as their Variance Inflation Factors (VIFs) were well below the upper limit of 10 (Tan & Teo, 2000). Furthermore, there are strong conceptual grounds for treating Perceived Usefulness and Behavioural Intentions as distinct


174


(Davis, 1989; Bhattacherjee, 2001). Table 3 shows the optimal factor loadings achieved after elimination of items, while Table 4 illustrates which items were deleted, and the structure of the final measurement model. Table 3: Final factor loadings

Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6

Perceived Usefulness (PU)

PU1 0.10 0.76 0.04 0.06 0.16 0.05

PU2 -0.02 0.76 0.13 0.14 0.08 0.12

PU3 0.11 0.80 -0.08 0.13 0.17 0.10

PU4 0.17 0.80 -0.08 0.00 0.05 0.14

PU5 0.18 0.68 0.05 0.11 -0.10 0.24

PU6 0.13 0.77 -0.10 0.28 0.15 0.15

Information (Content) Quality (IQ)

IQ1 0.11 0.29 0.07 0.74 0.11 0.13

IQ2 0.16 0.08 0.00 0.78 0.00 0.10

IQ3 0.19 0.20 -0.01 0.62 0.27 0.17

IQ5 0.03 0.09 -0.01 0.66 0.18 0.31

IQ7 0.27 0.15 0.05 0.75 0.05 0.02

System Quality (SQ)

SQ4 0.19 0.06 -0.01 0.21 0.56 0.17

SQ5 (Reversed) 0.20 0.23 0.05 0.10 0.68 0.18

SQ6 (Reversed) 0.13 0.12 0.02 -0.06 0.79 0.11

SQ7 (Reversed) 0.10 0.15 0.09 0.20 0.71 0.17

Continuance Intention (CI)

CI1 0.16 0.56 0.02 0.23 0.26 0.26

CI3 (Reversed) -0.10 0.55 -0.06 0.17 0.36 0.17

Trust (TR)

TR1 0.12 0.27 -0.04 0.17 0.34 0.70

TR2 0.18 0.24 0.01 0.05 0.14 0.77

TR3 0.31 0.26 -0.02 0.14 0.29 0.64

TR4 0.30 0.18 0.17 0.26 0.04 0.61

TR6 0.22 0.15 0.07 0.25 0.22 0.70

Loyalty Incentives (LI)

LI1 0.12 0.02 0.90 0.02 0.10 0.08

LI2 0.17 -0.04 0.92 0.03 0.08 0.06

LI3 (Reversed) -0.04 -0.03 0.81 0.02 -0.06 -0.02

Service (Support & Service) Quality (SS)

SS1 0.73 0.13 0.05 0.14 0.33 0.20

SS2 0.73 0.19 0.12 0.15 0.29 0.23

SS3 0.76 0.12 0.12 0.27 0.06 0.25

SS4 0.76 0.13 0.05 0.24 0.12 0.22

Expl.Var 2.95 4.69 2.46 3.21 2.81 3.04

Prp.Totl 0.10 0.16 0.08 0.11 0.10 0.10

Table 4: Refined Instruments

Construct Original Instrument (No. of Items) Refined Instrument (No. of Items) System Quality (SQ) 7 4 Information (Content) Quality (IQ) 7 5 Service Quality (Support & Service) (SS) 4 4 Trust (TR) 6 5 Loyalty Incentives (LI) 3 3 Perceived Usefulness (PU) 6 6 User (Customer e-commerce) Satisfaction (US) 1 1 Continuance Intention (CI) 3 2



5.3.2 Instrument reliability

In order to assess reliability of the refined instrument, the Cronbach Alpha was calculated for each construct (Tan & Teo, 2000). Reliability is indicated if the Cronbach Alpha is greater than 0.7. The lowest Cronbach Alpha was 0.71, thus demonstrating that all measures exhibited reliability (see Table 5).

Table 5: Reliability test

Construct Items Cronbach Alpha Perceived Usefulness (PU) 6 0.89 Information Quality (IQ) 5 0.83 System Quality (SQ) 4 0.76 Trust (TR) 5 0.86 Loyalty Incentives (LI) 3 0.86 Service Quality (SS) 4 0.88 Continuance Intention (CI) 2 0.71 User Satisfaction (US) 1 N/A

5.4 Descriptive statistics for B2C e-Commerce success dimensions

Table 6 illustrates the mean scores for the 8 dimensions and the correlations between them. The coefficients in bold italic are significant at p < 0.05. The mean scores for all dimensions are above 3 on a scale of 1 to 5 indicating that on average the e-commerce systems were perceived as successful. The lowest score of 3 was for loyalty incentives (LI), and the highest was for continuance intentions (CI), with a mean of 4. All the dimensions were significantly correlated with each other (p < 0.05), with the exception of loyalty incentives (LI), which only correlated with service quality (SS). It is perhaps an indication that including LI in the e-commerce success model was ill-conceived. As DeLone & Mclean (2004) note, “…despite the multidimensional and contingent nature of e-commerce success, an attempt should be made to significantly reduce the number of different measures used to measure success, so that research results can be compared and findings validated” (p. 44). The 7 remaining factors are all interrelated, indicating they are part of a higher order measure of B2C e-commerce success. The correlations between them do not exceed 0.63, which is indicative that they are nevertheless distinct constructs.

Table 6: Descriptive Statistics for B2C e-Commerce Success Dimensions

Mean PU IQ SQ CI TR LI SS US

PU 3.7 1.00 0.41 0.37 0.63 0.50 0.01 0.37 0.44

IQ 3.4 1.00 0.38 0.41 0.50 0.08 0.51 0.40

SQ 3.8 1.00 0.46 0.55 0.11 0.49 0.45

CI 4.0 1.00 0.50 0.00 0.36 0.48

TR 3.7 1.00 0.12 0.62 0.53

LI 3.0 1.00 0.20 0.13

SS 3.4 1.00 0.55

US 3.9 1.00

5.5 Hypothesis testing

The hypotheses were tested using multiple linear regression equations to represent the relationships shown in Figure 1. The following equations were built: CI = h1* US + h3 * PU + h4 * LI + h7 * TR + h9 * SQ + h14 * IQ + h16 * SS…….…………(i) US = h2 * PU + h6 * TR + h10 * SQ + h13 * IQ + h15 * SS……………………………….....(ii) PU = h5 * TR + h8 * SQ + h12 * IQ + h18 * SS……………………………………………....(iii) TR = h11 * SQ + h17 * SS……………………………………………………………………..(iv) h1 to h18 represent the beta values corresponding to the hypotheses they represent. Recursive simultaneous equation models, such as equations (i) to (iv) can be estimated by using multiple linear regression applied to each equation separately (Pindyck & Rubinfeld, 1998). Table 7 below summarises the results of the regression tests. Hypotheses were supported if the p values were less than 0.05. 9 of the 18 hypotheses were supported and are highlighted in bold italics in Table 7. Table 7 shows that the three major variables influencing continuance intentions are perceived usefulness, user satisfaction and system quality,


176


with perceived usefulness being the primary contributor. User satisfaction in turn is influenced by perceived usefulness and service quality, with service quality having the higher beta value. Perceived usefulness is primarily influenced by trust and information quality. Trust is strongly influenced by service quality and system quality, in that order.

Table 7: Hypothesis testing

Hypothesis Independent Variable Dependent Variable Beta value

p level (*p<0.05)

Hypothesis Supported?

H1 User Satisfaction (US)

Continuance Intention (CI) 0.17 *0.0223 Yes

H2 Perceived Usefulness (PU)

User Satisfaction (US) 0.18 *0.0131 Yes

H3 Perceived Usefulness (PU)

Continuance Intention (CI) 0.43 *0.000 Yes

H4 Loyalty Incentives (LI)

Continuance Intention (CI) -0.05 0.431 No

H5 Trust (TR) Perceived Usefulness (PU) 0.34 *0.0004 Yes

H6 Trust (TR) User Satisfaction (US) 0.16 0.0676 No

H7 Trust (TR) Continuance Intention (CI) 0.11 0.1900 No

H8 System Quality (SQ) Perceived Usefulness (PU) 0.11 0.1848 No

H9 System Quality (SQ) Continuance Intention (CI) 0.17 *0.0150 Yes

H10 System Quality (SQ) User Satisfaction (US) 0.14 0.0752 No

H11 System Quality (SQ) Trust (TR) 0.31 0.0000 Yes

H12 Information Quality (IQ)

Perceived Usefulness (PU) 0.19 0.0175 Yes


User Satisfaction (US) 0.05 0.5112 No


Continuance Intention (CI) 0.09 0.2114 No

H15 Service Quality (SS) User Satisfaction (US) 0.28 0.0008 Yes

H16 Service Quality (SS) Continuance Intention (CI) -0.08. 0.3115 No

H17 Service Quality (SS) Trust (TR) 0.47 0.0000 Yes

H18 Service Quality (SS) Perceived Usefulness (PU) 0.01 0.9215 No

The supported hypotheses are illustrated in Figure 2 below.



Figure 2: Refined e-Commerce success model

6. Discussion and implications

The model shown in Figure 2 is a very much refined version of Figure 1. As illustrated in Table 6 all the 7 dimensions in Figure 2 are significantly correlated, indicating that they are all dimensions of the same higher order e-commerce success construct. The contribution of multiple linear regression analysis has been to identify the most significant relationships while controlling for interrelationships between factors and indirect effects. The strong influence of perceived usefulness and user satisfaction on continuance intention and the negligible influence of loyalty incentives are as expected from Bhattacherjee (2001). The influence of system quality on continuance intentions mirrors the equivalent perceived ease-of-use to usage-intentions relationship in the pre-adoption TAM model (Davis, 1989). System quality is often operationalised as the ease and/or difficulty of system use (Rai et al., 2002; Seddon, 1997), which allows for this parallel to be drawn. The influence of information quality is shown to be primarily on perceived usefulness and not continuance intentions. This finding is indicative that information quality indirectly influences continuance intentions through perceived usefulness, as reasoned by Lederer et al. (2000). This rationale may apply also to the trust variable which has a strong influence on perceived usefulness, but not on continuance intentions. Perceived usefulness and service quality both have an influence on user satisfaction. Trust indirectly affects user satisfaction through perceived usefulness. Trust, finally, is strongly influenced by service and system quality. The relationship between system quality and perceived usefulness is surprisingly absent in the refined model of Figure 2. This apparent anomaly can be attributed to the inclusion of the trust variable, whereby system quality indirectly affects perceived usefulness through trust. These findings show that the inclusion of service quality (DeLone & McLean, 2003) and trust (Molla & Licker, 2001) in the traditional IS success model significantly alters the nature of relationships between success dimensions. These relationships are reflective of the new e-commerce context, where service quality and trust are essential attributes for successful B2C e-commerce. The inclusion of continuance intentions rather than use or intentions to use is also a valuable refinement, as customer intentions to continue using an e-commerce system are reflective of repeat customers. Given the importance of these three variables to e-commerce success, more sophisticated conceptualizations and measures can be used to investigate their full impact. For example, Gefen et al. (2003) identify structural assurances, familiarity with the e-vendor, situational normality and calculative-based beliefs as important factors in enhancing trust. Concerning service quality, several studies suggest the use of a comprehensive instrument such as SERVQUAL to assess this dimension (DeLone & McLean, 2004; Molla & Licker, 2001; Pather et al., 2004). DeLone & Mclean (2004) recommend that fewer and not more dimensions are needed to describe IS success. The inclusion of continuance intention was not as an extension to their model, but as a better means of assessing usage intentions in a post-adoption context such as when evaluating IS success (Bhattacherjee, 2001). The inclusion of trust as an additional variable was justifiable given its importance to e-commerce (Gefen et al., 2003; Molla & Licker, 2001). The addition of loyalty incentives on the other hand was found to be unnecessary. This additional factor did not correlate with most of the other interrelated dimensions. It also did not influence continuance intentions as hypothesized. DeLone & Mclean’s (2004)


178


stance that additional dimensions beyond those in their updated model will not add much value to the e-commerce success concept is therefore partially vindicated. In developing an instrument to assess the different dimensions of success, existing validated measures were drawn upon and modified appropriately. The use of existing measures proved successful as the instrument, after refinement, was shown to be both valid and reliable. DeLone & McLean’s (2004) exhortation to researchers to use existing measures so as to build up a cumulative tradition was therefore justified. The model of Figure 2 provides practitioners with a holistic, multidimensional perspective on e-commerce success. There is a tendency in practice to focus on different success dimensions in isolation, and not recognize their strong interrelationships. By considering the different dimensions in the model, and looking closely at their interrelationships, a better understanding of how to measure and improve e-commerce offerings can be obtained. The model may also be used to diagnose and address problems with B2C e-commerce offerings. For example, if an online vendor is experiencing problems related to lack of trust amongst potential or existing e-customers, a two-pronged strategy aimed at simultaneously enhancing service quality and system quality will help improve the levels of trust.

7. Limitations and further research

The sample was drawn largely from a group of MBA and other postgraduate students at a leading South African university. Most of the respondents were working professionals or managers and were representative of the typical e-commerce user in South Africa. There was however a predominance of males in the sample, whereas there is no major gender bias evident amongst e-commerce users in South Africa (De Villiers & Van Der Merwe, 2001). This was a limitation of using such a group of students. Future research might draw from a wider sample of e-commerce users to ensure the sample is even more representative of the typical e-commerce user in South Africa. The instrument validation process revealed that continuance intention and perceived usefulness loaded on the same factor during CFA. This was indicative of the strong relationship between perceived usefulness and behavioural intentions, as identified in the pre-adoption TAM model, and in post-adoption studies such as by Bhattacherjee (2001). To ensure greater discriminant validity, it may be necessary to revisit the continuance intention instrument, given that the perceived usefulness instrument has been validated in a wide variety of contexts. For the sake of parsimony, it may also be suggested that either perceived usefulness or continuance intentions be included in the e-commerce success model. This would be in line with the call by DeLone & McLean (2004) for a reduction in the number of dimensions of e-commerce success. Future research could therefore assess alternative models of e-commerce success to identify the best-fit model. Rai et al. (2002) conducted such a study for traditional IS success models. Some dimensions of e-commerce success were difficult to operationalise. For example, user satisfaction with e-commerce has been identified by Wang et al. (2001) as consisting of 7 dimensions. Many of the dimensions overlap with those identified in the e-commerce success model of Figure 1. A single-item measure was therefore used to assess overall user satisfaction as suggested by Rai et al. (2002). Future research ought to bring conceptual clarity to this overlap of dimensions between e-commerce user satisfaction and success. This study has incorporated continuance intentions as the ultimate dependent variable. Future research might attempt to incorporate appropriate measures for use and net benefits into the model, either as an alternative for continuance intentions or in addition to it. To add explanatory richness and greater conceptual clarity to the phenomenon of B2C e-commerce success, it is perhaps timely at this juncture to conduct an inductive, qualitative enquiry using methods such as the grounded theory methodology. Even in well-researched areas such as IS success, grounded theory studies may be conducted to garner a better understanding of the social processes at play (Glaser, 1992).

8. Conclusion

IS success is a multi-faceted and multi-dimensional construct. In a B2C e-commerce context, customer service and trust in an online vendor are of utmost importance. It stands to reason therefore that service



quality and trust are two additional dimensions to consider in addition to the traditional dimensions of system quality, information quality, user satisfaction, perceived usefulness, and use/intentions to use. Given the variety of products and services on offer to e-commerce consumers, using frequency of use as a general measure of e-commerce success is problematic. Frequency of use is also dependent on the nature of products and services being purchased. For example, online grocery shopping might be conducted more frequently than purchase of airline tickets for some. Assessing customer intentions to continue using an e-commerce system is perhaps a better measure of success, as it reflects repeat business regardless of product/service. It also treats behavioural intentions as a post-adoption phenomenon, as is required for evaluating success of a system. The pre-adoption intention to use measure from the TAM model is not as suitable to use for this evaluative purpose. The validation of the initially formulated model in Figure 1 showed that the 7 major dimensions making up e-commerce success are all interrelated. However, only 9 of the 18 original hypotheses were supported. This was because the hypothesis testing delineated direct and indirect relationships between the core set of e-commerce dimensions. Overall the ultimate dependent variable, continuance intentions, was found to be directly influenced by perceived usefulness, user satisfaction and system quality. User satisfaction was directly impacted by service quality first and foremost and then perceived usefulness secondly. This affirms the strong association between a satisfied customer and good quality service. Perceived usefulness was influenced by trust and information quality. Trust in turn was influenced by service and system quality. The confirmation of 7 interrelated dimensions for e-commerce success and a set of 9 direct relationships provides for a parsimonious way of understanding and evaluating B2C e-commerce success. Much of the research in this domain has been of a conceptual nature. This study adds to the body of knowledge by empirically validating a refined model of B2C e-commerce success upon which future studies can build.

References Bhattacherjee, A (2001). “An Empirical Analysis of the Antecedents of Electronic Commerce Service Continuance”,

Decisions Support Systems, Vol. 32, Iss. 2, pp 201–214. Brown, I and Buys, M (2005). “Customer Satisfaction with Internet Banking Web Sites: An Empirical Test and Validation of a Measuring Instrument”, South African Computer Journal, Vol. 39, pp 29 – 37. Cyr, D, Bonanni, C and Ilsever, J (2004). “Design and e-Loyalty across Cultures in Electronic Commerce”, Proceedings

of the 6th International Conference on Electronic Commerce, 2004. Delft: Netherlands, pp 351-360. Davis, F (1989). “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology”, MIS

Quarterly, Vol. 13, Iss. 3, pp 319-339. DeLone, W and McLean, E (1992). “The Quest for the Dependent Variable”, Information Systems Research, Vol. 3, Iss.

1, pp 60-65. DeLone, W and McLean, E (2003). “The DeLone and McLean Model of Information Systems Success – A 10 Year

Update”, Journal of Management Information Systems, Vol. 19, Iss. 4, pp 9-30. DeLone, W and McLean, E (2004). “Measuring e-Commerce Success: Applying the DeLone & McLean Information

Systems Success Model”, International Journal of Electronic Commerce, Vol. 9, Iss. 1, pp 31 – 47. De Villiers, C and Van Der Merwe, J (2001). An Investigation into the Adoption of Electronic Commerce by South African

Consumers – Who They Are and What They Think. Technical Report, University of Pretoria. Fusilier, M and Durlabhji, S (2003). “No Downturn Here: Tracking e-Business Programs in Higher Education”, Decision

Sciences Journal of Innovative Education, Vol. 1, Iss. 1, pp 73-98. Garrity, E, Glassberg, B, Kim, Y, Sanders, G and Shin, S (2005). “An Experimental Investigation of Web-based

Information Systems Success in the Context of Electronic Commerce”, Decision Support Systems, Vol. 39, pp 485-503.

Gefen, D, Karahanna, E and Straub, D (2003). “Trust and TAM in Online Shopping: An Integrated Model”, MIS Quarterly, Vol. 27, Iss. 1, pp 51-90.

Glaser, B (1992). Emergence vs. Forcing. Basics of Grounded Theory Analysis. Sociology Press, Mill Valley. Goldstuck, A (2002). The Goldstuck Report: Online Retail in South Africa, 2002. Pinegowrie: World Wide Worx. Goldstuck, A (2004). Online Retail in South Africa 2004. Pinegowrie: World Wide Worx. Grandon, E and Pearson, J (2004). “Electronic Commerce Adoption: An Empirical Study of Small and Medium US

Businesses”, Information & Management, Vol. 42, pp 197-216. Harris, L and Goode, M (2004). “The Four Levels of Loyalty and the Pivotal Role of Trust: A Study of Online Service

Dynamics”, Journal of Retailing, Vol. 80, pp 139–158. Ifinedo, P (2006). “Extending the Gable et al. Enterprise Systems Success Measurement Model: A Preliminary Study”,

Journal of Information Technology Management, Vol. 17, Iss. 1, pp 14 – 33. Iivari, J (2005). “An Empirical Test of the DeLone-McLean Model of Information System Success”, DATA BASE for

Advances in Information Systems, Vol. 36, Iss. 2, pp 8-27. Kim, M, Kim, W and Oh, S (2006). “Past, Present and Future of e-Business”, International Journal of Contents, Vol. 2,

Iss. 1, pp 1-4.


180


Lederer, A, Maupin, D, Sena, M and Zhuang, Y (2000). “The Technology Acceptance Model and the World Wide Web”, Decision Support Systems, Vol. 29, Iss. 3, pp 269–282.

Molla, A. (2002). E-Readiness and E-Commerce Success: Developing and Exploring an Antecedent Model in Developing Countries Context. Unpublished PhD Thesis, University of Cape Town.

Molla, A and Licker, P (2001). “E-Commerce Systems Success: An Attempt to Extend and Respecify the Delone and Mclean Model of Information Systems Success”, Journal of Electronic Commerce Research, Vol. 2, Iss. 4, pp 1-11.

Parker, C and Castleman, T (2007). “New Directions for Research on SME e-Business: Insights from an Analysis of Journal Articles from 2003 to 2006”, Journal of Information Systems and Small Business, Vol. 1, Iss. 1-2, pp 21-40.

Pather, S, Remenyi, D and Erwin, G (2004). “E-Commerce Success: The Quest for IS Effectiveness Measurement: A Conceptual Framework for the e-Commerce Environment”, South African Computer Journal, Vol. 32, pp 34 – 43.

Pather S, Remenyi D and de la Harpe A (2006). “Evaluating e-Commerce Success – A Case Study” The Electronic Journal of Information Systems Evaluation, Vol. 9, Iss. 1, pp 15-26, available online at www.ejise.com.

Pavic, S, Koh, S, Simpson, M and Padmore, J (2007). “Could e-Business Create a Competitive Advantage in UK SMEs?” Benchmarking: An International Journal, Vol. 14, Iss. 3, pp 320-351.

Pavlou, P and Fygenson, M (2006). “Understanding and Predicting Electronic Commerce Adoption: An Extension of the Theory of Planned Behavior”, MIS Quarterly, Vol 30, Iss 1, pp 115-143.

Pindyck, R and Rubinfeld, D (1998). Econometrics Model and Economic Forecasts (4th Edition), McGraw-Hill, Singapore.

Quaddus, M and Achjari, D (2005). “A Model for Electronic Commerce Success”, Telecommunications Policy, Vol. 29, pp 127-152.

Rai, A, Lang, S, and Welker, R (2002). “Assessing the Validity of IS Success Models: An Empirical Test and Theoretical Analysis”, Information Systems Research, Vol. 13, Iss. 1, pp 50-69.

Seddon, P (1997). “A Respecification and Extension of the DeLone and Mclean Model of IS Success”, Information Systems Research, Vol. 8, Iss. 3, pp 240-253.

Seddon, P and Kiew, M (1996). “A Partial Test and Development of DeLone and Mclean’s Model of IS Success”, Australian Journal of Information Systems, Vol. 4, Iss. 3, pp 66-88.

Shankar, V, Smith, A and Rangaswamy, A (2003). “Customer Satisfaction and Loyalty in Online and Offline Environments”, International Journal of Research in Marketing, Vol. 20, Iss. 2, pp 153–175.

Suh, B and Han, I (2002). “Effect of Trust on Customer Acceptance of Internet Banking”, Electronic Commerce Research and Applications, Vol. 1, pp 247–263.

Tan, M and Teo, T (2000). “Factors Influencing the Adoption of Internet Banking”, Journal for the Association of Information Systems, Vol. 1, Iss. 5, pp 1-42.

Torkzadeh, G and Dhillon, G (2002). “Measuring Factors that Influence the Success of Internet Commerce”, Information Systems Research, Vol. 13, Iss. 2, pp 187-204.

Van Slyke, C, Belanger, F and Comunale, C (2004). “Factors Influencing the Adoption of Web-Based Shopping: The Impact of Trust”. DATA BASE for Advances in Information Systems, Vol. 35, Iss. 2, pp 32-49.

Wang, T, Tang, T and Tang, J (2001). “An Instrument for Measuring Customer Satisfaction toward Web Sites that Market Digital Products and Services”, Journal of Electronic Commerce Research, Vol. 2, Iss. 3, pp 20-48.

Webchek (2004). Online Shopping Trends – October 2004, available online at http://www.webchek.co.za/library_trends2.html. (Accessed 18 July 2006).

World Wide Worx (2005). Great Expectations Stall on the Internet, available online at http://www.theworx.biz/access05.htm (Accessed 18 July 2006).

World Wide Worx (2006). Online Retail in South Africa 2006 – Broadband and Experience the Keys to Online Retail, available online at http://www.theworx.biz/retail06.htm (Accessed 18 July 2006).

Appendix 1: Instrument Measures

(Answered with respect to the most recent e-commerce experience)

Perceived Usefulness (PU)

PU1. Using this online shopping site enhances the productivity of my shopping activities. PU2. Using this online shopping site has a critical role in supporting my shopping activities. PU3. Using this online shopping site makes it easier to do my shopping activities. PU4. Using this online shopping site enables me to accomplish shopping activities more quickly. PU5. Using this online shopping site improves my performance (e.g. saving time or money) of shopping activities. PU6. I find this online shopping site useful for my shopping activities.

Information (Content) Quality (IQ)

IQ1. This online retail site provides the precise information I need. IQ2. This online retail site provides responses to questions and queries that are exactly what I need. IQ3. This online retail site provides sufficient information to enable me to do my tasks. * IQ4. This online retail site has errors that I must work around.


http://www.ejise.com/

http://www.webchek.co.za/library_trends2.html

http://www.theworx.biz/access05.htm

http://www.theworx.biz/retail06.htm


IQ5. I am satisfied with the accuracy of information provided by this online retail site. * IQ6. The output options (printer-friendly options, page sizes allowed for, etc.) are sufficient for my use. IQ7 .The information provided by this online retail site is helpful regarding my questions or problems.

System quality (SQ)

* SQ1. This online retail site is user friendly. * SQ2. Compared to other sites, this online retail site is easy to become familiarized with. * SQ3. I find it easy to get this online retail site to do what I want it to do. SQ4. I find it easy to become skilful at using this online retail site. SQ5. I believe that this online retail site is cumbersome to use. SQ6. To use this online retail site requires a lot of mental effort. SQ7. I get frustrated when I use this online retail site.

Continuance Intention (CI)

CI1. I want to continue using this online retail site rather than discontinue its use. * CI2. My intentions are to continue using this online retail site rather than use any other alternative means of shopping. CI3. I would like to discontinue the use of this online retail site.

Trust (TR)

TR1. This online retail site is trustworthy. TR2. I trust in the benefits of the decisions made by this online retail site, i.e.: I trust the specials offered and recommendations made by the site. TR3. This online retail site keeps its promises and commitments. TR4. This online retail site keeps customers’ best interests in mind. * TR5. This online retail site would do the job right even if not monitored. TR6. I trust this online retail site.

Loyalty incentives (LI)

LI1. This online retail site offers incentives for its continued use, such as frequent flier miles or bonus points. LI2. I get rewarded for my continued patronage of this online retail site. LI3. This online retail site generally does not give me any loyalty incentives for my continued use of its service.

Service quality (support and service) (SS)

SS1. I am satisfied with the customer support provided by this online retail site. SS2. I am satisfied with the after-sales service provided by this online retail site. SS3. This online retail site understands my problems and requests. SS4. This online retail site responds to my requests fast enough.

User (customer e-Commerce) satisfaction (US)

US1. I am satisfied with making purchases online from this online retail site. * Dropped Items


182


Appendix 2: Demographic profile of respondents

Count %

Gender

Male 114 69%

Female 51 31%

Unknown 1 1%

Age Under 18 3 2%

18 – 24 4 2%

25 – 34 113 68%

35 – 44 42 25%

45 -54 4 2%

55 + 0 0%

Highest Education Qualification

High School 4 2%

College Diploma 9 5%

Technikon Diploma 11 7%

Undergrad. Degree 49 30%

Postgrad. Degree 93 56%

Occupation

Full-time Student 36 22%

Professional 76 46%

Technician/Artisan 4 2%

Manager 38 23%

Unemployed 1 1%

Self-employed 4 2%

Other 5 3%

Missing 2 1%

Monthly Taxable Income

R 0 - R 1,999 11 7%

R 2,000 - R 4,999 5 3%

R 5,000 - R 9,999 11 7%

R 10,000 - R 19,999 54 33%

R 20,000 + 76 46%

Missing 9 5%




184

Appendix 3: Internet and e-Commerce experience

Count %

Internet Access at Home? Yes 119 72%

No 46 28%

Missing 1 1%

Internet Experience (yrs) < 1 2 1%

1 – 3 9 5%

3 – 4 15 9%

4 – 5 22 13%

> 5 118 71%

Online Shopping Experience (yrs) < 1 31 19%

1 – 3 64 39%

3 – 4 33 20%

4 – 5 15 9%

> 5 23 14%

Frequency of Internet Use Few times/month 8 5%

Once a week 10 6%

Few times/day or more 148 89%

Intensity of Internet Use per Average Day (hrs) < ½ 12 7%

1/2 – 1 49 30%

1 – 3 68 41%

> 3 37 22%

Extent of Internet Use (Scale of 1 to 5) Mean StdDevGet Information and Product Support 4.0 1.1

Communications 4.8 0.8

Download Free Resources 3.3 1.3

Entertainment (e.g. Online games) 1.9 1.2

Purchasing/Shopping 2.6 0.8

Banking 3.2 1.1

ISSN 1566-6379 185 ©Academic Conferences Ltd Reference this paper as: Goethals, F. G. “Important Issues for Evaluating Inter-Organizational Data Integration Configurations.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 185 - 196, available online at www.ejise.com

Important Issues for Evaluating Inter-Organizational Data Integration Configurations

Frank G. Goethals IESEG School of Management, Lille, France [email protected] Abstract: Partnering companies can share data via various configurations. Typically problems become evident as partnering companies seek to exchange data. These problems take a different form for different data integration configurations and are thus of great relevance when evaluating different configurations. This paper identifies issues to be taken into account when evaluating inter-organizational data integration configurations. Eight problems are discussed: difficulties to identify which information flows to automate; to relate different viewpoints on boundary objects; to agree on data formats; to distribute investments among the parties; to deliver appropriate service levels; to preserve the value of the data sharing; to clarify data ownership and to do all of this in the frame of changing relationships. For several problems it is illustrated how they surface in a completely centralized and in a completely decentralized inter-organizational data-integration scenario. Keywords: business-to-business integration, inter-organizational data integration problems, boundary objects, service levels, data ownership, data functionality

1. Introduction

Optimization efforts often focus on inter-organizational activities. Such schemes heavily rely on information sharing for two principal reasons: (1) as information is being shared, data inconsistencies across enterprises can be eliminated so that all companies accurately perceive the current state and (2) as new information is being shared, new business practices become possible. The process-paradigm indicates companies can execute a task if they are in a specific state. Information and Communication Technology (ICT) makes it possible to transmit information about a multitude of states in real-time. For example, in the past only two states were recognized in the ordering process: ‘order placed’ and ‘delivery received’. Nowadays a customer company can be informed that an order was received, accepted, scheduled, picked, loaded in the truck, etc. A number of ‘information sharing problems’ are associated with setting up an inter-organizational system to share data. Section 2 identifies eight problems and illustrates them in the context of the following practices:

Vendor Managed Inventory (VMI). The Bullwhip Effect is a significant problem that is discussed extensively in Supply Chain Management literature. Variability in demand is magnified at each stage up the supply chain (i.e., from reseller, over seller to manufacturer). Case studies have proven that, through Vendor Managed Inventory, the Bullwhip-effect can be strongly reduced (Smaros et al. 2003). VMI requires an intensive sharing of stock and sales data among different companies in the supply chain. Information exchange in this configuration is so extensive that it is unrealistic to assume the data could be shared and processed manually, without directly connecting the computer systems of the different companies.

Product Lifecycle Management (PLM). According to CIMdata (2005), PLM involves the collaborative creation, management, dissemination and use of product definition information across the Extended Enterprise from concept to end of life of the product. Takalikar (2004) asserts companies with PLM get better products to market faster and provide better support to the customers.

- A specific case: e-procurement via Tradcom. Tradcom (Muylle & Croon 2003) offers a virtual trading environment for companies in Belgium, the Netherlands and Luxembourg to trade indirect goods and services. Neef (2001, p. 25) defines such indirect goods as ‘any commodity or service that a company buys that does not result directly in finished goods’. Tradcom’s suppliers tend to enjoy a long term relationship with Tradcom. Many of the suppliers now co-own Tradcom. Computer systems between suppliers and Tradcom are highly integrated. Customer data couplings are looser. Since this trade concerns indirect goods, customers typically do not want to invest heavily in linking their systems directly to the platform.

After presenting eight problems associated with sharing data in Section 2, Section 3 illustrates how the problems take a different form in different data sharing configurations. This leads to the conclusion that those problems have to be considered when evaluating inter-organizational data integration configurations. The



goal of the paper is not to give an exhaustive list of issues that have to be considered. Rather, eight problems are discussed which were found to be of relevance in several cases. It should be noted that this paper investigates only the exchange of explicit knowledge. Tacit knowledge is at least until some point in time not stored in computer systems (see Nonaka 1994). As such, there are a number of distinct characteristics which make it inappropriate to treat explicit and tacit knowledge as one and the same.

2. Problems associated with sharing data

Case studies (such as the Tradcom case study) and literature (Ljung 2004; Sheu et al. 2001; Tang et al. 2006; etcetera) indicate that a number of important problems become evident when companies begin to share information. This section identifies eight frequently encountered problems. Each problem is defined and its relevance is illustrated by referring to one or more of the practices presented above: VMI, PLM and e-procurement via Tradcom. This paper focuses on identifying principal problems. Further research is required to elaborate rectification projects. The eight problems we discuss are the following:

1. First and foremost, companies have to define what information flows are valuable from a business viewpoint: what data does a company want to use, when should it get that data for the data to be useful, etc. Identifying what information flows to automate is not an easy task. 2. When one wants to identify information flows, one will stumble across another problem: if companies want to share data about an object, they have to acknowledge they may use the object in different tasks and they, therefore, may have another viewpoint on the object. Thus they need to map their viewpoints before they can actually go about sharing data.

3. At this stage partners know what data they intend to share. The next phase requires defining the data format.

4. To realize the information sharing, investments will need to be made. Partners have to agree on who will bear costs for installing, maintaining and upgrading the systems. 5. Once the investments are made, business continuity can only be ensured if the data is provided by the systems as needed. The systems should thus offer appropriate service levels in terms of availability, response time, etc. 6. Sharing data gives the parties involved new sources of power. Data receivers may forward the data to a third party or may inadequately secure their systems, the data provider may not pay enough attention to data quality, etc. Partners must preserve the value of the data sharing. 7. A party should be designated that can decide what can or has to happen with specific data, and what is prohibited. This requires clarifying who the designated data owner is.

8. All of the problems described above have to be dealt with in the frame of changing relationships. New partners may be added, and partners may be removed from the network.

2.1 Valuable information sharing practices have to be identified

Partners have to identify what information sharing practices add value.

2.1.1 Problem description

Identifying the content, timing and parties involved for an electronic data transmission is far from straightforward. When starting a Business-to-Business integration (B2Bi) effort it is very likely that the existing information flows between the partners have not been ‘architected’ or made explicit (Goethals et al. 2006b). This means many existing information flows are ‘invisible’, complicating their digitalization. Moreover, creativity and coordination are needed to redesign existing information flows and identify valuable new information flows and ways to realize them. A party can be required to capture new or more fine-grained data internally. New data content can be identified that only exists at the level of the Extended Enterprise.

2.1.2 Illustration of relevance

The introduction of Electronic Data Interchange (EDI) enabled companies to digitally transmit existing documents such as purchase orders. This way, the traditional replenishment process was automated. Later on, creative minds introduced VMI as a substitute for the traditional replenishment process, relying upon the

www.ejise.com ©Academic Conferences Ltd 186

Frank G. Goethals

introduction of new information flows that were also based on EDI. Sellers only used to receive (and to have knowledge about the reseller’s purchase orders. With VMI, sellers regularly receive information about the resellers’ sales orders and stock states (Angulo et al. 2004; Danese 2004). Pursuing VMI requires agreement on what messages to transmit and the timing of message transmissions. Messages may only be transmitted after a request or regularly as a subscription with a publication once a week, daily, several times per day, etcetera. PLM acknowledges regular transmissions of information between different designers are valuable. Sharing information among different designers requires many decisions to be made, the timing of transmission for example. If one is reworking an artifact that is being used by others in their decision making process, should a new version of the design then be transmitted to the others every 15 minutes, every hour, every day or every week?

2.2 Partners have different viewpoints on objects

The determination of what data to share is complicated by the fact that the partners have a different perception of the objects on which they want to share data.


Successful data exchanges between two parties generally fit with theories on boundary objects (or trading zones) (Star & Griesemer 1989; Chrisman 1999). Boundary objects are objects that both inhabit several intersecting social worlds and satisfy the informational requirements of each of them. ‘Boundary objects are objects which are both plastic enough to adapt to local needs and the constraints of the several parties employing them, yet robust enough to maintain a common identity across sites’ (Star & Griesemer 1989, p 393). A ‘flight’ is an example of a boundary object:

A pilot flying an airplane needs to know where a flight is going to, but not who is sitting where on the plane.

The stewards have to know where everyone is to be seated on the plane.

The catering company needs to make sure that the right type and quantity of food is available for a specific flight. Appropriate catering is a function of the number of people seated on the plane, what is related to the identity of the people on the plane,

known by the stewards; and possibly of

the location where the plane is heading to/from, known by the pilots.

Carlile (2002) asserted that knowledge is localized and embedded. Integrating the data about an object is problematic as different user groups have different perceptions of the object, having common and diverse needs and interests. They execute different tasks using the object data and different business rules pertaining to the object are relevant. User groups’ views are interrelated and information in different views needs to be aligned. The ultimate level of transparency is to integrate enterprises so that only one electronic version of an ‘order’ is saved in a shared database and that Purchase Order and Sales Order melt entirely together. Making this vision reality is difficult because companies involved interpret the concept of ‘order’ differently. That is, an ‘order’ is a boundary object. Simply stated, different companies can have a number of common and specific events that may affect the state of the object. For common events, it has generally been acknowledged that the lifecycles of the different perceptions of the boundary object should be compatible. Snoeck et al. (2004) for example show that both companies should agree whether an order first has to be paid for and then has to be delivered; or the other way around. This does not create an additional burden on the goal to store only one copy of the boundary object. However, when it comes to the specific events, the boundary object is brought in a different state for one company, but not in another state for the other company. For example, checking the customer history can categorize the order into the state ‘urgent handling’ or the state ‘postponed handling’ for the supplier, while the customer leaves it in the state ‘PO sent’. Consequently, not all information on the boundary object simply can be stored in a single shared electronic object.


Carlile (2002) studied four principal functions that are dependent upon each other in the creation and production of a product: (1) sales/marketing, (2) design engineering, (3) manufacturing engineering and (4)



production. He found that knowledge specialization makes it complex to work across functional boundaries and to accommodate the knowledge created in another practice. People perceive product data differently. Designers make a Computer Aided Design (CAD) file and machine operators focus on Numerical Control (NC) files. Workmen that install a part are only interested in the outside appearance of the part and whether that part fits. A designer considers the same object as vector graphics. Consumers of a product want to see how the product appears in different settings (e.g., a refrigerator placed in different kitchens). Engineering Bill of Materials (BOM) can be changed to create a manufacturing BOM by adding information about machines that will be used during production. Although all data concerns the same product, only to some extent are the various parties interested in the same information. To determine the relationship between data one has and data others need, the events that happen to an object have to be investigated. Several parties influence the state of the object and the data that is valid is thus determined by many parties together. All partners need data that is valid to them. This validity does not require consistency, as events happen to the object that are not relevant to a party. Data that is valid for one company is not necessarily valid for another company. For instance, an engineering company decides to forward its entire engineering BOM to a number of suppliers, each of which is only interested in a subpart of the BOM. Updates to the BOM are relevant for one party, but not for another. That is, without the update the data is not consistent, but still valid for some suppliers.

2.3 An appropriate data format has to be defined

Once the information companies want to share is known, partners have to look for a way to share data so that it offers desired ‘functionality’.


Choosing a data format is not easy. Partners can have data in systems they have developed themselves or that were created by vendors. These systems are likely to support different data formats and their interfaces may not be documented. Even though many XML industry standards have been developed to exchange data, many companies have legacy investments in EDI systems. These make the choice for XML messages less evident. An important decision companies have to make concerns how they annotate and structure data. Different formats can be adopted to transmit data. Different formats, however, enable different ‘functionality’. If data is meant to be ‘fully functional’ on another computer system, using the data will be greatly facilitated if the data is annotated. That can be done by sending data in a standard XML format. Hence, the sender can try to standardize and enhance the data structure, thus transforming unstructured data into semi-structured data. Senders may not like the recipient to use information for unknown or undesired purposes. Hence, they may transmit data that is not fully functional. The receiver can be hampered because the data is not annotated and highly unstructured. Data that is highly structured at the sender’s site, such as prices of products in a relational database, can be transmitted in unstructured documents such as highly graphical brochures. An information owner who wants to prevent poaching and the like (see Section 2.6) may adopt a format that reduces functionality, thus limiting the value of the transmission for the information receiver (Clemons & Hitt 2004).


The Tradcom case indicates that the data format that companies internally use often matters. This is especially relevant for companies that limit relationships to short term contracts. Tradcom has a short term relationship with the customers. Therefore, it allows the customers to send orders in the format they use internally, such as SAP (Goethals et al. 2007a). This lowers the burden on the customers to do business with Tradcom. However, not every internally used format is accepted by Tradcom. The platform ‘understands’ messages from counterparts in a number of vendor formats (such as SAP). Customer’s proprietary system formatted messages are generally not understood (Goethals et al. 2007a). From the supplier-side, where there are long term relationships, Tradcom-specific XML protocols have been developed. All suppliers have


Frank G. Goethals

enabled their systems to speak that Tradcom-specific language. Their internal data storage format thus is of no direct relevance. Companies can seek to restrict ‘data functionality’. As an example, a supplier may not want potential purchasers to load the entire product catalogue into their ERP systems. This would make it too easy for them to compare different suppliers’prices. Still, it is valuable for the purchaser that data on products he buys from the catalogue are made persistent in his system. A solution to this problem was defined by SAP and Ariba (SAP 2003) and implemented at Tradcom. With the Open Catalog Interface (OCI) or ‘punch-out’ solution, a purchaser sees the catalogue on a webpage formatted in HTML. The catalogue does not enter the ERP system of the purchaser. The purchaser can select the products he wants to order via a web interface and can have a standardized message sent to his ERP system so that only the data on the desired products is posted to his ERP system. Many suppliers do not want purchasers to download the entire catalogue into their system because this would make it too easy for the customers to benchmark offerings. This is, however, not always the case. If the competitive strength of the company is just that it has the lowest prices, it will try to make its price data as accessible as possible. In case the supplier tries to differentiate his products from those of competitors using other variables than the price, the supplier will try to pull the attention of the customers to the appropriate variables; for example by presenting a nice picture on the catalogue-webpage. Short term buyer-supplier relationships tend to work this way. Suppliers in longer term relationships may make prices available in an accessible format, even if these do not show the best side of the supplier. When considering PLM, for a designer a bit map picture is not very useful: they require vector formats to be able to make changes to the product’s design.

2.4 Different parties have to make investments

Investments are needed to enable information sharing. Partners have to agree who will bear the costs for installing, maintaining and upgrading systems.


Investments can be made primarily by one company or split among the partners and/or third parties. If one party changes or withdraws the systems he possesses, other parties may have to make further investments in order to ensure connectivity. Without their investments, information flows will fail if interfaces for example have become invalid. Long term relationships partners are better placed to know who currently uses a modified system and are able to negotiate, plan and control changes. It can be desirable that new entrants to the network do not need to make significant capital investments. If big investments are required, this complicates the decision to commit to the network because they then are locked-in to a relationship that may not end in a partnership. On the other hand, a party that leaves can take a part of the investments with him so that the rest of the Extended Enterprise suffers more than proportionally from the departure of that party. That is, they lose more than just the connection with that party: they also lose connection with other parties. Han et al. (2004) showed companies tend to be reluctant to share information via a non-neutral infrastructure. The natural fear is that the information will be used beyond a contractual relationship. This seems significant in cases of coopetition (i.e., cooperation between competitors). It is thus not only important to decide who pays for the investment just because it negatively influences who wants to share data (i.e., as it costs money you do not want to share data), but also because it has a positive influence (i.e., as you own the storage space yourself you are willing to share data).


The spread of investments by collaborators depends on the chosen information sharing configuration. Therefore, no general comment on VMI or PLM can be given. In the Tradcom-case (Muylle & Croon 2003), customers did not have to make big investments: they only had to enable their systems to send purchase orders in their proprietary format to Tradcom. Their suppliers had to make bigger investments: they needed to communicate in a Tradcom-specific XML format with Tradcom.



Tradcom took care of translating between the customer’s formats and Tradcom’s standardised format. A group of suppliers together own Tradcom and they are thus the parties that invest most in the communication. Customers can easily enter and leave the marketplace whereas suppliers are closely tied to Tradcom. Tradcom reduces investments for individual suppliers because the lowest cost option is to have suppliers connect only to Tradcom, and this interfaces with systems of all group’s customers. If a new customer enters the marketplace, the individual suppliers do not have to make new investments. If a new supplier enters the marketplace, other existing suppliers do not have to make investments. Neither do their customers. Should one or several of the Tradcom-owning suppliers ‘leave’ Tradcom and take the Tradcom platform with them, the customers and all other suppliers would be disconnected.

2.5 Partners become dependent upon service levels provided by data sharing systems

The value of data sharing systems depends on the service levels that are realized.


A significant service level requirement is ‘availability’. Lack of data inhibits the functioning of companies. User’s service level expectations have to be managed so that unexpected temporary events that cause lower availability do not create negative sentiments (Eliadis 2007). Other service level issues that require attention are system response times, the variability in response time, the maximum time span specific data is inaccessible, etcetera (Hartley 2005). Current ‘embodied’ solutions to deal with contingencies, such as calling a colleague in the partnering company, can become less evident once the computer systems are integrated. This is because computer systems integration can weaken the personal relationships that exist between companies (Levina and Vaast 2006). This increases the dependence upon sustained peak performance of the computer systems.


In the context of VMI, unavailability of the reseller’s stock and sales data prevents the seller from accurately defining the ‘purchase order quantities’ (i.e., the ‘sales order quantities’ from his viewpoint). The network may be able to function without this information, but only for a limited time span. Similarly, companies sharing product data can usually execute their tasks if they have access to not-so-recent product data. Still, sometimes it is important an update is considered in the execution of a task. The infrastructure then should allow the timely communication of this update. Product data files can be quite large. Files of more than 100 MB are common. If such a file is ‘suddenly’ needed, the requesting system may have to wait a long time for the transmission to complete, disrupting the system’s execution.

2.6 Partners must preserve the value of data sharing activities

The partners should handle data in line with how other partners would like them to handle it so that they can build trust.


There are two main situations that create trust between companies: the receiver has faith in the quality of the data and the sender has faith in the receiver preserving the confidentiality of the data. This trust can be marred by the partner himself, or by a third party. This is shown in Figure 1.

Receiver (R)Sender (S)Sender (S) Receiver (R)

Third party (T)

Himself (H) PoachQualityHimself (H) Quality Poach

Steal Corrupt/StealThird party (T) Corrupt/Steal Steal

Figure 1: Sender and receiver are both responsible for preserving trust

Receivers have several responsibilities. A party that obtains new data has the power to ‘poach’. According to Clemons and Hitt (2004, p94), ‘poaching’ comprises three components: ‘(1) the exchange of information between two parties, as a natural by product of contractual exchanges for other goods or services,


Frank G. Goethals

necessary for the performance of contractual obligation; (2) the subsequent use of this information by the receiving party, outside the purposes for which the information was provided, and for its own benefit or economic gain; and (3) at the expense of, or creating economic damage to, the party that provided the information’. By extension, poaching cannot only concern ‘production data’, but also meta-data: partners can pass on data about what data is being shared. Going one step further, receiving parties may reject poaching but inadequately secure their systems so that unauthorized parties can gain access to the data. In essence, a network is only as secure as its weakest module or connection. Security requires authentication and authorization (Logan 2004). Authorizations typically are discussed in the context of who is trying to access the data content. Goethals et al. (2006a) found authorizations tend to depend on what task the data will be used for, the physical location the message has to be sent to, the medium over which the message is transmitted, the moment the message is sent, the data content, the message format, whether it concerns a single record or a batch of records, whether the data is coarse-grained or fine-grained, etcetera. Authorizations are thus a complex matter. Both the receiving and sending parties’ systems should be trustworthy. The sending party should make sure that it provides data of the expected quality. Therefore, this party has to prevent stored data corruption and ensure accurate data entries. This can be a significant task for operators that do not know or understand the relevance to users. A special problem concerning the preservation of the functionality of the data has to do with non-repudiation. Typically, a party executes tasks in response to received messages. Parties that have taken part in information transmissions should not deny their participation later. In a B2B context, the concept of non-repudiation is important. This concept embraces two ideas: the sender cannot deny that he sent the message and the receiver cannot deny that he received the message (Kremer et al. 2002). ‘Privacy’ is an often mentioned topic in the context of partnering companies. Clarke (1999) asserts information privacy requires that information about individuals should generally not be available to others and that, where such data is possessed by another party, the individual should be able to control the data and its use to a considerable extent. This leads to the conclusion that a customer that gives personal information to a specific company, not automatically agrees this information can be forwarded to partners of this company. Partners clearly should handle data with care, fully respecting the agreement the originating organization has with its customer. As the partner does not have a direct relationship with the customer, chances of poaching are elevated. Poaching can be limited by using a data format with limited functionality (Clemons & Hitt 2004). Logically, one would not expect poaching to show up in an Extended Enterprise context as poaching potentially damages trust. Still, Clemons and Hitt (2004) state actual poaching is difficult to observe, turning it more plausible. They assert poaching is more likely to turn up if there is weak intellectual property protection (e.g., the impossibility to ‘return’ the information at the end of the contract) and if poaching was not prohibited in a contract. These are two conditions that often apply in a close partnership. One could make a similar assumption about the problem of securing the systems. One could assume that partners ensure the security of their systems because they are dealing with confidential information. In one study, Dynes et al. (2005) indeed found firms do not formulate big security requirements for their suppliers. Yet, in an earlier study Dynes (2004) found that, in general, companies are auditing the information security status of potential partners. One difficulty with such assessments is that they are said to slow down partnering.


Applying Figure 1 to the PLM example suggests the following risks: (RH) The party can consciously share his partner’s designs with a competitor of that partner. (RT) If a party’s systems are not secured well enough, intruders may gain access and steal the product designs that are owned by this party’s partner. (SH) For one party it can suffice that data are generally precise, while for another party extreme accuracy may be relevant. Rounding done by the former party (implicitly) results in inaccurate data for the latter party. (ST) If an external party can hack the systems and can change measurements, the plans become worthless. If such problems re-occur this makes it hard for companies to have confidence in the data that is being shared.



Similar risks exist in the VMI case.

2.7 Data ownership has to be unambiguous

A data owner has to be appointed. A party decides what can or has to happen with the data and what cannot. Often it is not clear who is the designated data owner.


On the one hand, several parties may want to decide what can or must happen with the data. On the other hand, no party may be designated as data owner or may feel responsible. Data ownership is defined in the California State University Data Warehouse Glossary as the “responsibility for determining the required quality of the data, for establishing security and privacy for the data and determining the availability and performance requirements for the data”. Data owners have the authority, accountability and responsibility to create and enforce organizational rules and policies for business data. They have the right to determine access, creation, standardization and modification rights (Alstyne et al. 1994). If ownership is not well defined, data that is considered to be valid may in fact not be valid; may fall in the wrong hands, etcetera. Levina and Vaast (2006, p28) indicate that, in community-like practices, boundary objects are co-produced and represent negotiated outcomes of a joint practice. If different partners collaboratively change a view on an object, it becomes obscure who actually owns that specific view on the object. As a result, the ownership of intellectual property is one of the most critical legal risks that confront Extended Enterprises (Bader 2008).


If a subpart of a product is created by a partner it is not always clear who is the owner of the subpart-data: the producer of the final product, or the designer of the subpart. Similarly it may be unclear who the owner of the stock data is in case of VMI. Unless consignments are used, the stock is the property of the reseller. This would suggest the reseller is the data owner. The stock is, however, managed by the seller. So, the seller could be seen as being the data owner. The fact that the reseller and the seller have to agree on a stocking plan, inventory turns, replenishment rates, delivery frequency and the like shows that both parties have something to say about the stock. The key issue here is defining responsibility, accountability and authority. Data ownership problems also show up in cases where there are data that only exist at the level of the entire Extended Enterprise (i.e., at the level of the collection of the collaborating legal entities). In Tradcom’s case, suppliers sell their products through Tradcom to a multitude of customers. Although individual suppliers only get orders with respect to their own products, there is information available at Tradcom-level about orders that contain products from different suppliers and about customers that entered the platform to do business with one supplier but in time started doing business with other suppliers. The question is who owns this data and has responsibility for managing its quality. This information can be very valuable yet easily overlooked because it is by nature not really owned by any of the individual parties.

2.8 The involved parties change over time

All of the problems above have to be dealt with in the frame of changing relationships. Partners can be added to and can be removed from the network.


The parties that provide data and the parties that use data change over time. Data that is valid changes if partners change. Also, there are changes in the agreement on which parties can access data: some parties can probably no longer access the data. Partners that are dropped from the Extended Enterprise are likely to become competitors of the Extended Enterprise. Retained access to the Extended Enterprise data may hamper the competitive position of the Extended Enterprise. However, it can be desirable that they still get access to specific data, for example to offer after-sales service. Similarly, it is often important for members of the Extended Enterprise to maintain access to former partner’s data.


Frank G. Goethals


When a supplier of a semi-finished product is replaced, the valid product data set typically also changes. That is, in the design of the assembled product, the design of the semi-finished product has to be replaced by the design of comparable items made by the new provider, or at least its interface. While the old design data may not be valid for newly produced products it remains useful to maintain old products and the after-sales service provision. This implies that different versions of the component design have to be managed and related to specific variants of the produced product. In general, the former supplier of the component no longer should have access to the other product data as they could disperse this data across the new network they have entered. Nevertheless, if this supplier still is to create spare-parts for the product under consideration, they shall need to be given access to specific parts of the data warehouse. With VMI the seller truly obtains insight in the reseller’s business (Monitor Technologies 2005). This can be illustrated with the fact that if a transmission of sales and stock data fails, the seller, in practice, still has quite a good idea of the sales and stock levels at the reseller’s site because he knows his business so well. This fact is very desirable while both companies do business with each other. However, once their relationship is suspended, the seller could abuse the knowledge gained from this relationship.

3. Discussion: the problems are importa nt when evaluating data sharing configurations

The eight problems presented above should be taken into consideration when deciding upon an inter-organizational data integration configuration. Many configurations exist that have radically different properties in terms of how the eight problems show up. According to Goethals et al. (2007b), one can distinguish between different data integration configurations on the basis of

1. how decisions with respect to the data sharing are made: decisions with respect to data sharing can be taken centrally or decentrally. A central consortium such as RosettaNet can define a global data model, what messages can be exchanged, etcetra. Alternatively, parties can bilaterally agree on the content that will be shared, on an XML Schema they shall use, etcetera.

2. how data is transmitted: Data can be transmitted directly from data provider to data receiver or data transmissions can happen via a central system. Such a central system then can take care of format mappings, guaranteed delivery and other services.

3. how data is stored: The data that is to be shared may be stored in one central repository that is shared by all partners, or can be stored in different locations of different partners.

A data integration configuration is characterized by the extent to which it is rather centralized or rather decentralized on these three dimensions. Depending on the degree of (de)centralization, a configuration has different properties in terms of the problems presented in Section 2. Each configuration has good points and bad points to note for each problem. In CIMdata (2005), an entirely centralized data integration configuration is presented where companies create a single shared data repository that contains all the data the companies want to share. This configuration can be evaluated in terms of the problems presented above. For example, it was stated that timing, content and responsibilities for transmissions as well as data access rights have to be determined. When deciding upon necessary updates of the data in the shared storage space and access rights, a completely centralized configuration has the disadvantage that an update to the central data is relevant for one party, but may need to be hidden for another party. Also, for all information requirements that should be fulfilled there is this third party, the shared storage space, which needs to be involved. This central party can have established policies that are considered rigid and inappropriate by local offices. This may reduce motivation to introduce new information flows (Streeter 1973). However, the fact that there is an overall view available on what information sharing is happening makes it easier to manage the whole system. As another example, when it comes to the data format, the intermediary can take care of a big number of tasks, among which the transformation of message formats and the transformation of content. The product number used in one system, for example, can be reformatted to suit product numbers in another system. However, adapting the ‘functionality’ of the data may be a difficult matter in this setting. That is so, not only



because it is harder for a central office to understand local functionality requirements and restrictions, but also because the data can be available in the commonly owned shared space in a fully functional form. Furthermore, when considering service levels, there is a single point of failure (Iamnitchi 2000). All parties use the capacity of that same central point and the capacity that is available for one party thus depends upon the capacity used by all other parties. Still, as many parties co-invest in central systems, they may reasonably expect a high-quality shared space, operated by specialists who know how to intervene in case of problems. As a final example: as there is only one site with data, there is only one site that needs to be protected from security breaks (Yu 2000). However, if this security system fails a non-trusted party can immediately gain access to all data. Centralized configurations can be contrasted with entirely decentralized data integration configurations , where partners set up direct data exchanges between the system of the partner that wants to use the data and the system of the other partner where the data is available. In this case, parties autonomously can decide what data they want to share and are less concerned with high-level restrictions. Mintzberg, (1992) considered this to be a stimulus for innovation. However, setting up many point-to-point connections complicates management of the entire integration effort. Also, parties do not share the capacity of a central system. This lowers the chance that another party is using some capacity one had planned for. This, however, eliminates the possibility to use capacity that is currently not used by other parties. Capacity can thus be available in the wrong place (HP 2003). Furthermore, implementing security policies in a highly heterogeneous environment is harder (Iamnitchi 2000). Several data stores and transmission systems have to be secured and it is unclear what the security-level is of the Extended Enterprise as a whole and it is hard to identify whether anyone is poaching. There is no central intermediary that can oversee the unusual behavior of a party that are breaking trust. The illustration above highlights that companies are confronted in different ways with the problems from Section 2 if they choose another data integration configuration. Consequently companies have to find out how important a specific problem is to them and consider the identified problems when evaluating different configurations. Many configurations exist (Goethals et al. 2007b). The goal of this paper is not to give an evaluation of all integration configurations. This paper discusses problems that repeatedly have shown up when such configurations were installed and identified that these problems take on different forms when different configurations are used. Further research is needed to evaluate different configurations with respect to the problems discussed in Section 2. Management is advised to consider problems in the context of a specific configuration they have in mind.

4. Conclusions

This paper discusses eight problems that show up when partnering companies decide to share information. Each problem occurs in different practical situations, as illustrated with examples from the PLM and the VMI domains and the Tradcom B2B trading platform. It is important to deal with each of these problems. It was shown that the problems take a different form in different Business-to-Business data integration configurations. Companies that need to choose an inter-organizational data integration configuration should determine appropriate weights for the problems. One of the problems concerns the distribution of investments and benefits. The amount of counterparty-specific investments companies are willing to make depends upon the expected duration of the use of the investment and thus upon the expected duration of the relationship between the companies. Furthermore, a data-owner is needed at inter-organizational level which determines who can access what data and those that may not. The value created from using specific information sharing configurations depends on service levels that are realized and the way partners handle data. Senders are dependent upon their partners to reject poaching and to secure their systems. Receivers are contingent upon the senders for transmitting the data in an appropriate quality. Data format problems and obscurity about appropriate data flows further complicate the issue.

Acknowledgements

I gratefully acknowledge the help from Prof. David Newlands in writing this paper.


Frank G. Goethals

References Alstyne, M. Van, Byrnjolfsson, E. and Madnick, S.E. (1994) “Why not one big database? Principles for data ownership”,

MIT Sloan School Of Management Working Paper Series 3695-94, p 39. Angulo, A., Nachtmann, H. and Waller, M. (2004) “Supply Chain Information Sharing in a Vendor Managed Inventory

Partnership”, Journal of Business Logistics, Vol. 25, No. 1, pp101-120. Bader, M.A. (2008) “Managing intellectual property in inter-firm R&D collaborations in knowledge-intensive industries”,

International Journal of Technology Management, Vol. 41, No. 3/4, pp311-335. Carlile, P.R. (2002) “A pragmatic view of knowledge and boundaries: boundary objects in New Product Development”,

Organization Science, Vol. 13, No. 4, pp442-455. Chrisman, N. (1999) “Trading Zones or Boundary Objects: Understanding Incomplete Translations of Technical

Expertise”, Proceedings of the Society for Social Studies of Science Annual Meeeting, San Diego, October 28-31, p15.

CIMData (2005) “Eurostep’s Share-A-space – Making your intellectual assets the backbone of your information management strategy”. [online], Eurostep, http://www.eurostep.com/files/Share-A-space_CIMdata_2005.pdf.

Clarke, R. (1999) “Internet Privacy Concern Confirm the Case for Intervention”, Communications of the ACM, 42, 2, (February 1999) pp60-67.

Clemons, E.K. and Hitt, L.M., (2004) “Poaching and the misappropriation of information: transaction risks of information exchange”, Journal of Management Information Systems, Vol. 21, No. 2, pp87-107.

Danese, P. (2004) “Beyond Vendor Managed Inventory: the Glaxosmithkline Case”, Supply Chain Forum: An International Journal, Vol. 5, No. 2, pp32-40.

Dynes, S. (2004) “Security and Privacy: at odds with speed and collaboration?”, Executive Round Table Series, [online], Tuck School of Business, http://mba.tuck.dartmouth.edu/digital/Programs/CorporateEvents/SecurityAndPrivacy/Overview2.pdf p. 13.

Dynes, S., Brechbühl, H. and Johnson, M.E. (2005) “Information security in the extended enterprise”, Proceedings of the 4th Workshop on the Economics of Information Security, Cambridge, MA.

Eliadis, H. and Rand, A. (2007) “Setting expectations in SaaS: The importance of the service level agreement to SaaS providers and consumers”, [online], Software & Information Industry Association, http://www.siia.net/Software/pubs/saas_0207.pdf, p. 14.

Goethals, F. G., Vandenbulcke, J., Lemahieu, W., Snoeck, M. and Cumps B. (2005) “Two Basic Types of Business-to-Business integration”, International Journal of E-Business Research, Vol.1, No. 1, pp1-15.

Goethals F. G., Lemahieu W., Snoeck M. and Vandenbulcke J. (2006a) “Dependencies between Data Decisions in the Software Process”, Proceedings of the SPW/ProSim Workshop, Shanghai, May 20-21, 2006, LNCS Volume 3966, pp48-55.

Goethals, F., Lemahieu, W., Snoeck, M. and Vandenbulcke, J. (2006b) “Management and Enterprise Architecture Click”, Information Systems Frontiers, Vol. 8, pp67-79.

Goethals, F. G., Lemahieu, W., Snoeck, M. and Vandenbulcke, J. A. (2007a) “The data building blocks of the enterprise architect”, Future Generation Computer Systems. Vol. 23, No. 2, pp269-274.

Goethals, F. G., Snoeck, M. and Lemahieu, W. (2007b) “The Business-to-Business Integration solution space”, Pre-ICIS Workshop on E-Business, Montreal-Canada, December 9, 2007.

Han, K., Kauffman, R.J. and Nault, B.R. (2004) “Information exploitation and interorganizational system ownership”, Journal of Management Information Systems, fall 2004, Vol. 21. No. 2, pp109-135.

Hartley, K.L. (2005) “Defining effective service level agreements for network operation and maintenance”, Bell labs technical journal, Vol. 9, No. 4, pp139-143.

HP Invent (2003) “Architectures for storage consolidation”, [online], HP, http://h30046.www3.hp.com/uploads/whitepapers/Architectures_for_Storage_Consolidation_whitepaperF.pdf.

Iamnitchi, A. (2000) “Centralized vs. Decentralized design in Internet-Scale Applications”, Proceedings of the International Conference on Parallel Processing.

Kremer, S., Markowitch, O. and Zhou, J. (2002) ”An intensive survey of fair non-repudiation protocols”, Computer Communications, Vol. 25, No. 17, pp1606-1621.

Levina, N. and Vaast, E. (2006) ”Turning a community into a market: a practice perspective on information technology use in boundary spanning”, Journal of Management Information Systems. Vol. 22, No. 4, pp13-37.

Ljung, R., Johansson, M. and Rosén, J. (2004) “Sharing product data through life in the Extended Enterprise”, Product Data Journal, Vol. 11, No. 2, pp40-42.

Logan, C. (2004) “Security principles: identity, authentication, authorization”, Current IS&T Issue, Vol. 20, No. 2. pp 5. Mintzberg, H. (1992) Structure in fives, Prentice Hall, p 312. Monitor Technologies (2005) “Remote and Vendor Managed Inventory Management Systems for Bulk Solids”, [online],

Monitor Technologies, www.monitortech.com/webseminar/RMI_V2doc.pdf Muylle, S. and Croon, E. (2003) “Venturing into e-commerce in the MRO market”, ECCH, p 19. Nonaka, I. (1994) “A dynamic theory of organizational knowledge creation”, Organization Science, Vol. 5, No. 1, pp14-37. Neef, D. (2001) E-procurement, Prentice Hall Upper Saddle River, p 1996. SAP (2003) “Open Catalog Interface (OCI) Release 4.0”, [online], SAP,

https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/com.sap.km.cm.docs/library/icc/Open%20Catalog%20Interface.pdf.

Sheu, R.K., Chang, Y.S. and Yuan, S.M. (2001) “Managing and sharing collaborative files through WWW”, Future Generation Computer Systems, Vol.17, No.8, pp1039-1049.


http://www.springerlink.com/(xdpfszfax3cojerxtat244no)/app/home/contribution.asp?referrer=parent&backto=issue,5,38;journal,87,3923;linkingpublicationresults,1:105633,1



Smaros, J. Lehtonen, J.M., Appelqvist, P. and Holmström, J. (2003) “The impact of increasing demand visibility on production and inventory control efficiency”, International journal of physical distribution & logistics management. Vol. 33, No. 4, pp336-354.

Snoeck, M., Lemahieu, W., Goethals, F., Dedene, G. and Vandenbulcke, J. (2004) “Events as atomic contracts for application integration”, Data and Knowledge Engineering 2004 Oct; Vol. 51, No. 1, pp81-107.

Star, S.L. and Griesemer, J.R. (1989) “Institutional Ecology, Translations and Boundary objects: amateurs and professionals in Berkeley’s museum of vertebrate zoology, 1907-39”, Social Studies of Science, Vol. 19, pp387-420.

Streeter, D.N. (1973) “Centralization or dispersion of computing facilities”, IBM Systems Journal, No. 3, pp283-301. Takalikar, S. (2004) “Leveraging PLM for manufacturing excellence”, Express computer, April 2004, [online],

expresscomputeronline http://www.expresscomputeronline.com. Tang, M., Lee, B.S., Tang, X and Yeo, C.K (2006) “The impact of data replication on job scheduling performance in the

Data Grid”, Future Generation Computer Systems, Vol. 22, No. 3, pp254-268. Yu, E. (2000) Centralize or decentralize? “A requirements engineering perspective on Internet-scale architectures”,

Proceedings of the International Conference on Parallel Processing.

ISSN 1566-6379 197 ©Academic Conferences Ltd Reference this paper as: Jokela, P., Karlsudd, P. and Östlund, M. “Theory, Method and Tools for Evaluation Using a Systems-based Approach.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 197 - 212, available online at www.ejise.com

Theory, Method and Tools for Evaluation Using a Systems-based Approach

Päivi Jokela 1, Peter Karlsudd 2 and Martin Östlund 1 1School of Communication and Design, University of Kalmar, Sweden 2School of Human Sciences, University of Kalmar, Sweden [email protected] [email protected] [email protected] Abstract: This work introduces an evaluation model for examining organisations that in one form or another depend on modern information technology for their core activities. The evaluation model, named SUV, is based on a systems science approach and has been developed at the eHealth Institute (eHälsoinstitutet) at the University of Kalmar. The central mechanism of the model, the SUV diagram, partitions the scope of evaluation into seven categories and three levels. The seven categories have been selected on a systems science basis with inspiration from systems thinking. The rationale for making this the starting point for the evaluation model is that it may be justifiably stated that the framework provided by systems thinking truly encompasses the breadth of human activity. Making use of this general framework, the SUV diagram provides a general roadmap to guide the evaluation effort. The three levels (organisation, technological and individual) were selected to add detail to the analysis complement the categories and enrich the analysis of each category as well as to the dynamic interplay among them. The SUV methodology is based on a continuous evaluation process whose driving-force is the wish of the interested parties to develop their own activities. It is a methodology within whose framework any and all methods for data gathering deemed appropriate for the evaluation at hand can be used. The use of multiple methods is explicitly encouraged for the purpose of gaining a multi-perspective view of the organisation/activity under scrutiny. Based on the accumulated findings from pilot studies, the model was operationalised in the form of an IT application for electronic surveys. The application is expected to contribute in making the evaluation process more efficient and add structure to this process. Keywords: systems science, systems thinking, evaluation methodology, multi-method, electronic surveys

1. Introduction

Taking appropriate strategic or tactical action based on evaluation results is essential for any organisation to be able to adapt and survive in a changeable environment. However, the process of evaluation is fraught with problems and its findings are often regarded as incomplete, erroneous or biased (Barroso et al., 2006, Hartford et al., 2007). One reason is that systems involving human interaction are inevitably messy and complex. One way to handle this is to divide the system into smaller parts and examine each one at a time. This is a common tactic to manage complexity, but the downside is that the relationships between the parts and the ones between the part and the whole are neglected. Systems science provides a framework within which complex systems can be understood and it offers structure and method to describe and analyze parts within their context and systems as wholes (von Bertalanffy, 1968, Boulding, 1985, Checkland, 1981). The usefulness of the systems science view and the holistic perspective in examining organisations has been both theoretically justified (von Bertalanffy, 1968, Checkland, 1981, Beer, 1979) and is supported by empirical findings from diverse fields (Sandelin and Sarvimäki, 2000, Marine, 2002, Cohen et al., 2007). The systems science approach can be used to examine a system at any level. Within the boundaries of a system there exist an infinite number of subsystems (von Bertalanffy, 1968, Boulding, 1985, Checkland, 1981, Edwards, 2005). The process of deconstruction of the system should be guided by the stated purpose for the study and be regarded as part of defining its scope. This is similar to what Mulej (2007) lables requisite holism/realism. The inherent complexity of any human system, with its infinite number of subsystems and complex contexts, implies that any description or understanding of, a system must be based on incomplete evidence (Backlund, 2002, Jacucci and Hanseth, 2006, Simon, 1969). Furthermore, human activity is also fleeting, constantly changing and morphing, both at surface level and at a deeper level (Cohen et al., 2007). Still, all but the most temporary and volatile systems tend to have some kind of stable core, which contributes to order in the system, although the relationship between structure and order is not fully understood (Bohm and Peat, 2000).





By letting the purpose guide the process of analysis, a useful, albeit incomplete, representation can be constructed of the system under scrutiny. In practice, this means that some parts of the system will be analyzed in greater detail while others will be analyzed in less detail. This kind of goal-directed, focused analysis is necessary to be able to manage the process and also the product of analysis. At the same time one must not lose track of the whole picture. The marriage between details and the whole (or the wholes) is the main challenge to be overcome in order to fulfil the potential of a systems science based approach (Boulding, 1985, Checkland, 1981, Koestler, 1967, von Bertalanffy, 1968). Systems science aids us in this process by providing a language to describe the systems and their parts; and to provide structure and guidance in the traversal of systems levels when shifting between different levels of granularity (Cottam et al., 2005, Edwards, 2005). The model described here, systematic evaluation – also called the SUV model, takes a systems science based approach to evaluation and uses systems thinking based methods to perform evaluative work and analyse findings. SUV can be characterised as theory-based evaluation with a holistic approach. SUV-evaluation is mainly formative, in a sense that it is aimed to provide evaluation participants with feedback and to develop new knowledge, which is needed in order to continuously improve the evaluand and its performance. Involvement of the participants and interactivity between the evaluator and participants are essential throughout the evaluation process; from the beginning when the diverse evaluation issues and goals are established to the end when action must be taken to make changes in the organisation, based on the evaluation outcomes. SUV has been developed by the authors while working at the e-Health Institute (eHälsoinstitutet), University of Kalmar. Previous work on the model is described in (Brandt et al., 2003). The SUV tool is especially suitable for organisations that use (as opposed to having it as their main business) modern information and communication technology; for example: e-learning (Harun, 2002) and health services (Zhao et al., 2002). This, of course, applies to most organisations today (in the western world), and in the near future any organisation that does not, will have a hard time surviving. In the methodology based on the SUV model, the use of multiple methods is encouraged. One method that is given special attention here is a web-based questionnaire tool has been constructed to provide technical support and practical guidance in formulating evaluation questionnaires. The basic principles of the SUV model may be summarised as follows:

Analysing holistically by systematically considering the parts and their mutual relations.

Analysing activities starting from three levels: Individual , Technology and Organisation , and seven categories: Goal Seeking, Hierarchy and Relations, Differentiation and Entropy, Inputs, Transformation Process, Outputs and Regulation .

Enabling previous evaluations to be analysed by the same principles

Within the holistic framework the use of complimentary methods of data capture is encouraged.

The following section will give a brief orientation into the theory on which the model is based. Then follows a description of the method and the program supporting the model and creating the conditions for evaluation construction, implementation, analysis and innovation.

2. Evaluating on a systems-based approach

Systems theory is a theoretical approach for formulating general laws for systems, regardless whether they are physical, biological, social etc. Within organisation theory systems thinking involves that an organisation, like an organism, is affected by its environment to which it has to adapt for its survival (Boulding, 1985, Churchman et al., 1969, Litterer, 1973, von Bertalanffy, 1968). Systems have been defined in the literature in various ways. A frequently quoted specification by Hall and Hagen (1969) reads “A system is a set of components (objects) containing relations between the components (objects) and between their properties.” A system to be evaluated can be defined as a whole organisation but also as a department or a specific computer system. It is important to notice that the definition of system boundaries as well as the division into separate system levels does not follow any strict rules or principles but is a result of the goals and attitudes of the evaluator (Katz and Kahn, 1980). In an open system information and material are continuously interchanged with the environment. This continuous interaction with the environment makes it complicated to


Päivi Jokela, Peter Karlsudd and Martin Östlund

distinguish an open system from its surroundings (Checkland, 1981). The aim of the evaluation model is to facilitate the choice of limitations by using an area division derived from systems theory.

2.1 Main principles for a system

The properties of a system may be explained according to the following basic principles:

1. Holism: A change in part of a system affects the whole system (Boulding, 1985, Litterer, 1973, von Bertalanffy, 1968).

2. Non-summativity: The whole is more than the sum of the parts (ibid). 3. Equifinality: All open systems can attain the same state or result via different starting-points and approaches (ibid).

4. Multifinality: A common starting-point and/or the use of different roads may lead to different results (ibid). 5. Circular and manifold causality, non-linearity. A causal connection is not linear but a more refined explanation model is needed, in that many parts of the system may be affected on different occasions (de Shazer, 1994).

2.1.1 Holism

Evaluating an activity on a system-theoretical basis entails seeing the whole by systematically studying the parts and then feeding these results back into the activity from a holistic perspective. Most systems may be regarded as hierarchically organised, consisting of a long series of subordinate partial systems which all interact within the superordinate system (Boulding, 1985, Edwards, 2005, Koestler, 1967, Litterer, 1973, von Bertalanffy, 1968).

2.1.2 Non-summativity

Every partial system is of importance to the whole and should thus be examined separately but also in relation to the other parts. A hackneyed but relevant expression is that “no system is stronger than its weakest link”. Therefore it is essential that all parts of importance to the activity become visible and are included in the evaluation while at the same time the holistic perspective must be the guiding principle (Boulding, 1985, Litterer, 1973).

2.1.3 Equifinality and multifinality

In contrast to closed systems, all open systems lack an unambiguously defined point of equilibrium. Activities and organisations are never static but change more or less continuously. This is the reason why an evaluation must be considered from a dynamic perspective and why the result must always be regarded as more or less preliminary. One important concept in systems theory is equifinality, which means that all open systems can reach the same condition or result from different starting-points and by different approaches. In other words, an activity may attain the same goal by different methods. Similarly, the principle of multifinality may be explained as a common starting-point where similar conditions may lead to different results. Even though an organisation, for instance, gives the same directions and provides the same conditions for its departments the result may turn out to be completely different (Boulding, 1985, Litterer, 1973, von Bertalanffy, 1968).

2.1.4 Causality

A cause-effect relation or a linear causality, as it may also be formulated, is common in Western thinking. An explanation model like this briefly means that all events derive from one cause. This event may in turn give rise to other events. Linear causality thus means that one cause leads to one event (Petit and Olsen, 1994). Linear causality is not suitable for systems theory, as it provides a simplified image of reality. In systems thinking various causality theories have been brought forward, one of the most frequently mentioned being circular causality. The gist of it is “that an event is considered in relation to other events taking place simultaneously, interfering with each other and affecting each other reciprocally. An individual’s actions are affected by the environment, while these actions simultaneously affect the environment.” (Schöjdt and Egeland, 1994), p. 83). The consequence of this is that all events may be regarded as both cause and effect. All parts or events are dependent on one another (Petit and Olsen, 1994). Circular causality as an explanation model is not sufficient, however, for studying more complex systems. A more refined explanation model is manifold causality, according to which it is meaningless to try to find one cause of an event, but that there may be many causes behind what has happened. If circular and manifold causality are combined, an



even more thoroughgoing and refined causal explanation is attained, entailing that all parts of a system affect each other directly and indirectly (ibid.).

2.2 Human activity systems

Organisations that can be the target of a SUV-evaluation are human activity systems, in a sense that they can be modeled as a range of human activities that are related to each other within an organisational framework and these activities are supported or mediated by information technology. The perception of such systems is associated with the individual observers’ image of the world, or according to Checkland’s terminology their Weltanschauung (Checkland, 1981). Although these perceptions may sometimes converge into an agreement of what constitutes a meaningful system, there is no objective method available to create a joint description of the system. The special nature of the human activity system hinges on the concept that the system is aware of itself, in the same way that a person is aware of him- or herself. If a member of the human activity system was to observe the system he/she is part of, this would alter the system. Consequently, any member of the human activity system is barred from participating in a group observing this system with the purpose of building absolute, objective public knowledge about the system. However, a pre-requisite for building public knowledge is that anyone and everyone can participate in observing something. The exclusion of the members of the human activity systems from observing group makes it impossible for knowledge about human activity systems to ever reach the level of objective public knowledge. This implicates that the knowledge that is needed to evaluate human activity systems must be gained by using a different methodology. In this work, we postulate that the process of evaluative inquiry can be organised as a learning system, where multiple issues originating from different Weltanschauung are identified, presented and debated within a systems framework. This discourse is the basis of shared knowledge as well as new knowledge created jointly by the evaluation participants, and the deeper understanding may eventually lead to improvements of the studied organisation. (Jackson and Kassam, 1998, Checkland, 1981)

2.3 The main principles of the evaluation model

In the SUV evaluation model structures and processes of importance are divided into seven categories and three levels. The categories may be illustrated as the spokes of a wheel, which give stability and direction to the activity; the levels show the recurring layered structure of an organisation (Figure 1).

Figure 1: The seven categories and three levels in a SUV evaluation.

The three levels of the model can be seen as a nested holarchy, a concept that was introduced by Koestler (1967) and further developed by Edwards (2005). Each level not only includes but also transcends the previous levels. The complexity increases successively when one moves towards higher levels (Edwards, 2005). In the SUV model, the fundamental level comprises individuals and interactions between them. At the next level, information technology is added to the system so that the technology level includes individuals,



computer systems and networks as well as the human-computer interactions. The third level emphasises how the individuals and IT-systems are embedded in the organisational structure. The seven categories are gleaned from work in the field of systems science on the epistemological view of the world as a system. The categories can, with due justification, be said to encompass the breadth of human and natural activity, albeit at a high abstraction level (Beer, 1979, Checkland, 1981, Skyttner, 2001, von Bertalanffy, 1968). For each category an example is given to illustrate the type of issues that are covered by the category and how they can be operationalised in the form of questions on the personal, technical and organisational levels, respectively. The examples are from projects at the eHealth Institute.

2.3.1 Goal-seeking

As organisations consist of a number of different parts and hierarchical levels there is often an amalgamation of goals that have to be defined. The structure of some goals are more overarching and static and will be relevant to the activity for a long time, whereas others are more mobile and changeable. It is important to consider organisational goals from different hierarchical levels, as well as making both the formal and informal organisation goals visible. (Boulding, 1985, Litterer, 1973, von Bertalanffy, 1968).

Project: Evaluation of a web-based customer service application for 3D-mediated advice-giving on pharmaceutical products

Status: Planned

Personal What are the personal goals of the pharmaceutical counsellors when using the service?

Technical What technical equipment and software will be required for users of the service?

Organisational Can the introduction of this service broaden the range of questions that can be handled over distance?

2.3.2 Hierarchy and relations

In most organisations it is possible to identify different hierarchical levels and specialised functions. Elucidating these structures and the distribution of responsibility among them creates important conditions for the evaluation. Since the holistic perspective is a matter of cooperation and mutual relations between the parts, the whole cannot build on a collection of autonomous elements. In this part of the evaluation the external and internal boundaries of the organisation are made clear. (ibid)

Project: Evaluation of a distance learning course for medical professionals

Status: Completed

Personal What is the work climate like in the student groups?

Technical What is the structural relationship between the different technical solutions that are used in examination with regard to authentication and security issues?

Organisational How are responsibilities and task distributed among the persons involved in running and administering the course?

2.3.3 Differentiation and entropy

When the parts of an organisation begin to specialise to be able to perform certain tasks, the redundancy (surplus) as well as the entropy of the system decreases. In the long run the result of this differentiation is that the various parts become more dependent on each other while simultaneously, to some extent, the total efficiency of the system increases. A balanced degree of work distribution creates stability and safety but too much differentiation reduces the flexibility of the organisation and may make reality vulnerable. (Boulding, 1985, Litterer, 1973, Simon, 1969, von Bertalanffy, 1968).



Project: Evaluation of a web-based monitoring system for home care patients

Status: Underway

Personal What skills do the various professionals who are in contact with the patients have?

Technical What alternative technical systems are available should the primary system fail?

Organisational What routines are in place for the training of the different categories of professionals involved in running the system?

2.3.4 Inputs

The input category makes visible the resources and conditions available to implement a certain process (Boulding, 1985, Litterer, 1973, von Bertalanffy, 1968). This comprises human resources like the knowledge, competencies and experiences of the individuals involved in the process. It is also important to study the attitudes and motivations of the people involved, as well as their demands, expectations and fears with regard to the process. The physical circumstances include the ergonomic design of the work place and the access to work tools. The IT environment comprises hardware, software and network resources as well as IT security.

Project: Evaluation of a web-based customer service application for 3D-mediated advice-giving on pharmaceutical products

Status: Planned

Personal How many real life birthing sessions has the user participated in?

Technical What technical capabilities are there in the hardware to sense and react to the physical actions performed by the midwifery student?

Organisational Is the simulator training co-ordinated with the clinical training?

2.3.5 Transformation process

The transformation process is the actual interaction where the resource inputs are transformed into products and services (ibid). This kind of interaction may take place between, for example, management – employee, employee – customer or human – computer.

Project: Evaluation of a health information web site Status: Completed

Personal How does the service support the user’s knowledge learning process?

Technical In what way do the technical features of the service support the user’s learning?

Organisational Is the organisation well-adapted to support the purpose of increasing its users’ knowledge on health issues?

2.3.6 Outputs

The outputs are a result of the processes taking place in the organisation and are usually synonymous with the services or products it delivers (ibid). They thus occupy a central position in following-up systems. In an evaluation it is important to remember that there may be several different goals for what may be considered a good result. One important question that has to be asked is whether the results created in the evaluation of the outputs will be effectively fed back into the activity.

Project: Evaluation of a pharmaceutical telephone-based call centre service

Status: Completed

Personal How satisfied are the clients with the information they have received?

Technical What level of audio quality is delivered?

Organisational How can the average call time be optimised?



2.3.7 Regulation

Regulating the parts and process included is necessary for directing the transformation process towards the goals formulated. Regulation takes place synchronously and continuously on three levels. The first “primary” regulation is chiefly based on policy, norms and fixed rules in the organisation and its activities. The second or “secondary” regulation may use parts of the feedback occurring in a continuous evaluation in order to better adapt the contents to external and internal demands. The third “adaptation” level is attained when the organisation has been made part of the environment by opening up new paths and goals for its activities. All the three levels are essential to an open system, because static goals would mean that the system dies of itself. (Boulding, 1985, Churchman et al., 1969, Litterer, 1973, von Bertalanffy, 1968). The result and the efforts made around the evaluation may be regarded as the regulation of systems and partial systems, especially on the second and third levels, which makes the evaluation very important for maintaining the vitality of the system.

Project: Evaluation of an information exchange system for the co-ordination of activities for special needs kids

Status: Planned

Personal In what way can the system users contribute to the way the service is developed?

Technical In what way can the features of the system be adapted to suit the individual preferences of users?

Organisational How are proposals for improvements and changes managed?

2.4 Evaluation research

Evaluation is an important control instrument in all activities, especially in goal-oriented and decentralised ones, where a clear feedback both to decision-makers and to individuals in the operative activity is required. There are a number of different models and strategies in the evaluation field, some of which models evaluate products and others processes (Langerth Zetterman and Lindblad, 2001). Depending on which type of evaluation model is used, the possibilities of different interested parties to have their needs and interests elucidated and noticed are affected (Guba and Lincoln, 1989, House, 1980, Madaus and Kellaghan, 2000, Patton, 1986). It is not uncommon that evaluations focus more on models and methods than on a theoretically founded research methodology, and frequently criteria set up in advance are applied. In a comparison with what has been mentioned above the SUV model may be positioned as a theoretically founded model, which gives the evaluator the freedom to choose and create new criteria in both product- and a process-oriented evaluations. There are many points of contact between evaluation and research (Karlsson, 1999). An essential overarching similarity is that researchers and evaluators use the same methods for data gathering and analysis. The term evaluation research is relatively frequently used in describing the application of scientific research methods in evaluation contexts. However, there are differences between evaluation and research. One such difference is that evaluation often takes place in a task framework that is narrower than the theory frame used by researchers. This means that the evaluator is under stricter control than the researcher. A further difference is that the evaluation task often involves assessment, which is not expected of research. SUV enables both of these directions depending on which approach is employed. An evaluation effort based on systems science theory broadens the issue and opens up to a more analytical, critical and reflective approach. Theory-based evaluation has been recommended and developed by Chen and his colleagues in reaction to the more product-oriented evaluation tradition. If the theoretical foundation is too narrow, there is a risk that the approach of the evaluation is confirmatory rather than descriptive and explanatory (Smith, 1994). Closely related criticism has been vented by Patton (1989), who argues that theory-based evaluations have gone too far in reducing the focus on and generalisation of causal connections, criticism that he somewhat tones down in a later work (Patton, 1996). It is important to emphasise that evaluation according to the SUV model does not focus on causal connections but rather on pointing to a number of possible influential factors. The evaluation field contains basically three directions. The first is goal-result evaluation. This means evaluating towards goals in terms of measurable results. The process is viewed as an implementation of decisions which need not be motivated. The results are often presented as quantitatively measurable results. The second direction is process evaluation , which is conducted against criteria for what characterises a good process. The interest is geared towards the implementation, for instance in order to assess work



methods and contents. The result is assumed to be positive if it can be established that the implementation corresponds to the requirements agreed on. The third direction is interactive evaluation , which is characterised by the participation of the interested parties. Questions on the implementation are formulated on the basis of what different interested parties want to know. The interest focuses on what different interested groups consider important to evaluate from their various perspectives. Interactive evaluation forms a pattern developed in the 1980s and 1990s, by which cooperation between different interested parties increases the relevance of the evaluation questions and results and strengthens their influence (Cronbach et al., 1981). The goal of the SUV model is to thoroughly involve all the interested parties’ experiences and ideas. Earlier on, the idea prevailed that evaluation meant being able to stay neutral vis-à-vis the various interests by making objective measurements of goals which everyone was assumed to share. Today most evaluation researchers agree that this is not feasible – possibly except for narrowly delimited situations or relatively uncontroversial issues. It is probably more usual to be faced with different views on what should be examined and against which criteria this should be done. In a situation like this it is important to find forms which may increase the chances of various groups to make themselves heard. In the SUV model goal-seeking is done at an early stage and continues throughout the evaluation process, one important principle being to view the goals from different system levels. All the individuals involved should be offered the opportunity to participate in goal-seeking. The object is to be able to make formal and informal as well as short- and long-term goals visible. According to Franke-Wikberg (1992), the evaluation aim is formulated primarily from one vertical and one horizontal perspective. The first vertical evaluation, “Type 1”, is top-down, while vertical evaluation, “Type 2”, is bottom-up. Horizontal evaluation takes place on the same organisation level. An example of this is a personnel group’s internal evaluation, where the personnel “own” the results, deciding among themselves what they want to convey to other levels within the organisation. To these three levels may also be added a level outside the organisation consisting of various external interested parties. There is reason to point out that evaluation according to these different descriptions can function in parallel. However, confusion often arises or attempts to hold up one type as the only right one (Karlsson, 1999). The SUV model enables formulating the aim from all of these starting-points, which is often an advantage, since it sheds light on the issue to be evaluated from different perspectives.

3. Methodology

The SUV methodology is based on a continuous evaluation process whose driving-force is the wish of the interested parties to develop their own activities. What are in several evaluation methods termed process and product are studied from several points of view on the basis of a holistic approach. By continuously viewing the strengths, deficiencies and connections of the evaluated systems major and minor changes may be carried out continuously. The five basic principles of the SUV methodology may be summarised as follows:

A holistic approach

All interested parties take an active part in the evaluation A flexible model enables the use of several different theories and methods

The result of the evaluation is used for improving and developing activities

A continuous evaluation process

The evaluation model described here is valid for use in both quantitative and qualitative approaches and can be incorporated in inductive and deductive studies. An example of deductive use is to construct a questionnaire based on the SUV categories and levels. SUV also encourages to building up explanations via an inductive approach in a continuously explorative process, which to some extent reminds of the theory-generated approach called Grounded Theory as formulated by Glaser & Strauss (1967). An example of inductive use is the post factum categorisations of previously performed evaluations not using the model, see 3.2. The SUV model is not restricted to any one method for gathering data. It is a methodology within whose framework any and all methods deemed appropriate for the evaluation at hand can be used. Moreover, the use of multiple methods, used simultaneously, in parallel or sequentially; is explicitly encouraged for the purpose of gaining a multi-perspective view of the organisation/activity under scrutiny. A survey tool has been developed for questionnaire construction, web based delivery of surveys and compilation and analysis



of results. The survey tool is used exclusively for quantitative studies and while it serves as a good example of how the SUV principles can be put to practical use, it represents only one out of many methods that can be used with SUV. The choice of methods depends on the purpose of the evaluation and should be given careful consideration at the outset of the evaluation project. Figure 2 presents an overview of the conceptual foundations for the SUV methodology and its inclusive meta-structure within which any suitable method of data inquiry and any suitable approach for data analysis can be used.

Compilation of results

SUV

Syst

ems

Scie

nce

Com

pila

tion

...Surveytools

Obser-vation

Inter-view

Parti-cipatio

nMetho

d nMet

hods

Evaluation

Evaluation practiceSystems science practice

Systems science theory Evaluation theory

SUV-A Analysis

Analysis of results

SUV-O OperationalizationOperationalization of a systems science based

approach

Complementaryapproaches

Approach 1

Approach 2

Approach n

Figure 2: SUV is based on Systems Science Theory and Practice. The figure also illustrates the independence of method. Figure 3 describes the process of an evaluation study according to the SUV methodology. The iterative quality in the process of evaluation is emphasized with the resulting measures and innovations serving as input in the next cycle of evaluation. In a large scale evaluation project, several parallel processes of the kind illustrated in the figure can be active at the same time starting and ending at different points in time. Going backwards and forwards between sub-steps in an iterative fashion will (and should) also occur. All the processes of evaluation should, however, follow the same general progression starting with introductory delimitation and problematisation and ending in proposed measures and innovations.



Figure 3: SUV, a continuous evaluation process

3.1 Step 1 – Introductory delimitation and problematisation

This step includes an introductory discussion about what is to be evaluated, what is the background of the task and for what context is the effort planned. Preliminary system delimitation is made in the form of a sketch of what and who should be included in the evaluation. The evaluator should also formulate the overarching goal as clearly as possible and make a list of what is considered to be most relevant to the evaluation. It may also be useful to reflect on evaluator’s pre-understanding and how it may affect the planning of the study. The following questions may serve as guidelines during the introductory phase:

Why should there be an evaluation? What knowledge is desired?

Which interested parties are involved in the evaluation?

What should to be the role of the evaluator? How should the evaluation be conducted?

How should the evaluation be utilised?

3.2 Step 2 – Deconstruction of previous evaluations

It is possible, and often also beneficial, to feed back previously conducted evaluations into the SUV model, and these results can then be sorted into and interpreted within the categories the model includes. Even if this kind of revised categorisation cannot be absolute, it may give indication on how accurately the previous evaluations have covered the entire problem area, and also on which previous questions may be important to follow up.

3.3 Step 3 – Problematisation and delimitation

The third step in the model is to make a more exact formulation of problems. By determining scope and level the system will be delimited, the choice of what to evaluate depends on the goal of the task. Anything outside the system boundaries is defined as the environment and will not be included in the evaluation. However, it is important to keep in mind that there is a continuous interaction between the system and its environment. The evaluator’s list of what is considered to be most relevant to the evaluation (see step 1) can be refined and completed. Following steps can be used as guidelines in this phase:



3.3.1 The evaluation question

Formulate the evaluation questions as clearly as possible; both the overarching problematisation and more specific questions.

3.3.2 Delimitation of the system

Set clear boundaries for the system and specify what is to be included in the evaluation within these boundaries. It is also important to clarify which individuals are to be involved in the evaluation.

3.3.3 Safeguard the three levels

Make sure that the three levels, individual, technology and organisation, are represented in the evaluation.

3.3.4 Scope and timeframe

Clarify what resources are available in the form of individual effort, time, and access to respondents. Based on the evaluation’s scope and available resources, make a time schedule for the evaluation process.

3.4 Step 4 – Method choice/Question construction

The SUV method allows for a range of different forms of data gathering, such as observation, focus group, interview, questionnaire and document analysis. In the same way as the system delimitation, the choice of appropriate methods depends on the evaluation goal and also on questions that are defined in step 3. Using various methods in the same evaluation usually makes the study more refined and valid. The seven categories and three levels in the SUV circle are used as a framework for methodological design and also for question construction. Preferably, questions should be constructed for every single one of the fields in the SUV circle. Such full coverage will indicate that the evaluation takes into account all the important properties that characterize a complex system.

3.5 Step 5 - Dialogue, communicating questions

In SUV model, it is essential that the evaluation questions as well as obtained results are directly communicated to the participants. To enable a more formative and communicatively oriented evaluation, asynchronous and synchronous discussion conferences can be used to continually interpret and discuss findings; this approach can also contribute to the evolving evaluation method. The web-based questionnaire tool contains several functions to convey questionnaires to respondents. Among the functions may be mentioned advance notice, automatic reminder and a report function for processing the gathered data.

3.6 Step 6 – Visualisation of data

In this step, the gathered empirical data is compiled and presented in suitable ways, both quantitatively and qualitatively. The previous categorization of questions into SUV categories and levels, can be used to structure the results in preparation for their subsequent analysis. One example of such structuring can be seen in figure 4, where the results from multiple choice questions are presented for each of the fields in the SUV circle (this functionality is included in the survey tool previously presented).



3.4

4.1

3.5

2.7

3.2

2.8

4.2 3.7 3.2

3.

3.3.14.4

Figure 4: Visualisation of the mean values from multiple-choice questions.

3.7 Step 7 – Analysing the result

Assessment or analysis is the central task of evaluation (Karlsson, 1999). The analysis is often the least common denominator in different definitions of evaluation where the evaluation method constitutes the framework of the analysis. At the analysis/assessment of the result the evaluator may adopt a more or less objective or subjective attitude. The objective attitude includes general evaluation principles setting up norms, while the result in the subjective attitude is more dependent on the interpreter. It is also possible to choose the dialogically inter-subjective standpoint without getting stuck in objective or subjective reasoning (ibid.). The analysis may instead be shaped after a discussion between evaluators who try to get an insight into and understand each other’s perspective. Consequently, this qualitative analysis takes place in a dialogue with different interested parties. Comparison is one normative analysis approach used in evaluation contexts. There are basically three ways of viewing results. The first is to make a comparison with an explicitly defined reference point of some kind, such as a goal, idea or theoretical criterion. The second is comparing evaluation objects of the same kind, usually by ranking. The third is to compare transformation over time, in other words the development of the evaluation object. In SUV the starting-point of comparisons consists of the categories and levels stipulated in the model, which enables the use of all three approaches. A comparison emanating from its theoretically founded variables may be discussed using the following questions as starting-points:

What qualitative differences exist within the various categories and levels?

Which categories/levels received low and high values, respectively? Between which categories/levels are the differences the greatest or the smallest?

How does the result change over time?

On the basis of this orientation any reasons behind the differences displayed are analysed. It is also possible to further refine the categories which have indicated deficiencies or those categories which may serve as positive examples.

3.8 Step 8 – Taking action

At the very planning of an evaluation the issue of how to utilise the obtained results should be discussed. On the basis of the analysis in step 7 an assessment can be made of which categories require further scrutiny or immediate action to be taken. The SUV model makes it possible to start this process at an early stage and to continuously adjust and attend to various phenomena and conditions, even during the evaluation process. The fact that evaluation results are not always utilised for further development of the organisation and its activities may be due to various reasons. There may be external environment or background circumstances which affect the structure and implementation of the evaluation. It may also be due to how the evaluation is conducted and whose questions have been actualised. Further reasons may be that the results of the



evaluation are not considered relevant or that they are presented in a deficient way or that the intended receivers lack the knowledge, motivation and resources to operationalize the evaluation results (Karlsson, 1999). There are various measures that can be taken in order to support the utilisation of the evaluation results. To start with, it is essential to create good representativeness and maintain a active dialogue with those who are in various ways involved in the evaluation process. Another important contribution to enhance the positive attitudes is to emphasise the importance of communication and interaction among the participants, The results should be reported in a way that makes them available and interesting. In some cases, it may be important to raise the general competence level in the organisation, for instance by educating personnel and by encouraging discussion and critical reflexion in practice. An important measure for enhancing positive attitudes is to set aside resources which enable the application of those evaluation results which are considered important (Karlsson, 1999).

4. Web-based questionnaire tool

The questionnaire application was constructed by the authors to create a tool for question generation, data gathering and analysis that would make the SUV method accessible and usable also for the non-systems theory savvy user. The questionnaire instrument includes the possibility of choosing between different question types and between different formats, as shown in Figure 5. The questions may be divided into background questions, SUV questions and open questions.

Figure 5: Different questions types and formats in the web-based questionnaire tool.



Background questions are used in the statistical analysis to see the differences between one group and another. When searching for possible explanations it is important to consider which variables are relevant to the current evaluation. It is common to ask questions on sex, age, education, computer experience etc., but every organisation has its own special conditions which may be of importance to use as background variables. The multiple questions provided with a six-grade answering scale are termed SUV questions and are formulated so that each question belongs to a specific field in the SUV graph. The user marks in which field the question has been constructed and the tool states the frequency of questions in each field, as shown in Figure 6. This illustrates the distribution of questions among the different fields and also among the different categories and levels.

Figure 6: Examples of questions and of how the tool visualises their location and distribution within each question field.

There is also a possibility for the respondent to add further views on the SUV questions. In questionnaires distributed in paper form such a possibility is often offered at the end of the questionnaire, whereas the digital SUV tool makes it possible to attach comments immediately after each SUV question. The advantage is that this enables catching the spontaneous response, which may otherwise be lost if the respondent has to wait till the end of the questionnaire. To create a clear structure for result presentation and analysis according to the model there is a specially adapted result presentation module in the program. The result presentation is based on the SUV graph and on the collected quantitative data which is used in the statistical calculations. The arithmetic mean as well as median value of the answers to the SUV questions can be shown in each in each field, category and level, respectively. The result is also presented in greater detail by including the frequency distribution within the various answer alternatives in a table and visualising them in a suitable graph. For background variables either a bar chart or a histogram is used, depending on whether it is a discrete or continuous variable. If the number of respondents is sufficient the connection between background variables (groups) and SUV questions may be studied by means of a chi-two test (χ2) (Curwin and Slater, 2004). It is important to emphasise that the statistical analysis is mainly used to looking for possible significant connections in the gathered material, the primary aim thus not being to draw general conclusions that are supposed to be relevant to all organisations. The questionnaire tool also comprises a question archive where one may simply include and edit pre-constructed and pre-categorised questions. Usually, questions suitable for elucidating the various categories can be reused after a slight modification

5. Experiences and further elaboration

Constructing questions with the support of the SUV model does not automatically result in a holistic evaluation. It is certainly quite possible to use the principles of question construction to a smaller extent, but for SUV to achieve reliable reality the guideline is that all categories at all levels are well covered, as our pilot



evaluations demonstrate. It is also a clear advantage if the evaluation is conducted on a number of occasions and preferably complemented by several different data gathering methods. This complementation might suitably be performed after the first more explorative study. To continuously measure the status of the activity according to the same evaluation model enables a longitudinal comparison on the basis of a common theoretical reference frame. Formulating questions that cover all categories and levels in the model may sometimes be difficult and tedious, and it certainly requires more effort of the evaluator. On the other hand, the proposed structure also provides that all parts of importance to the activity are included in the evaluation, and at the same time the holistic view of the problem area becomes the guiding principle. In our opinion, a specially adapted model and methodology are required to fully acknowledge the complexity of organisations where the use of information and communication technology forms an integral part of core activities. As mentioned above, the SUV methodology can be used with several different methods. The same broad approach also applies to the analysis. The result may reflect a number of theories depending on the perspective/s adopted. A theory discussion is a necessary element if the method is to be used in a strictly scientific study. Within the next few years the eHealth Institute is going to use SUV as its evaluation model in the projects conducted by the institute. The method will most likely be developed further in evaluations of e-related projects.

Acknowledgements

Thanks to Prof. Göran Petersson at eHälsoinstitutet for his continuing support for the SUV evaluation project. Our gratitude to the reviewers for their valuable comments.

References Backlund, A. (2002) The Concept of Complexity in Organisations and Information Systems. Kybernetes, 31(1). Barroso, J., Sandelowski, M. & Voils, C. I. (2006) Research Results have Expiration Dates. Ensuring Timely Systematic

Reviews. Journal of Evaluation in Clinical Practice, 12(4), 454-462. Beer, S. (1979) The Heart of Enterprise, New York, John Wiley. Bohm, D. & Peat, D. (2000) Science, Order and Creativity, London, Routledge. Boulding, K. E. (1985) The World as a Total System, Beverly Hills, Sage Publications. Brandt, P., Jokela, P., Karlsudd, P., Révay, P. & Wennberg, L. (2003) Evaluation model for e-phenomena – a case study.

Proceedings of the MicroCAD 2003 International Scientific Conference 6-7 mars 2003. Miskolc, Hungary. Checkland, P. (1981) Systems Thinking, Systems Practice, New York. Churchman, C. W., Ackoff, R. L. & Arnoff, E. L. (1969) Analysis of the Organization. IN A, L. J. (Ed.) Systems, Control

and Adaptation Volume II (1980). NewYork, John Wiley & Sons. Cohen, S., Kaimenaki, E. & Zorgios, Y. (2007) Assessing IT as a Key Success Factor for Accrual Accounting

Implementation in Greek Municipalities. Financial Accountability & Management, 23(1), 91-104. Cottam, R., Ranson, W. & Vounckx, R. (2005) Life and Simple Systems. Systems Research and Behavioral Science, 22,

413-430. Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, D. C., Hornik, R. C., Philips, D. C., Walker, D. F. & Weiner, S.

S. (1981) Toward Reform of ProgramEvaluation: Aims, Methods, and Institutional Arrangements, San Francisco, Jossey-Bass Publishers.

Curwin, J. & Slater, R. (2004) Quantitative Methods, London, Thomson Learning. de Shazer, S. (1994) Words Were Originally Magic, New York, W W Norton & Company.

Edwards, M. G. (2005) The Integral Holon – A Holonomic Approach to Organisational Change and Transformation. Journal of Organizational Change, 18(3).

Franke-Wikberg, S. (1992) Utvärederingens mångfald: några ledtrådar för vilsna utbildare, Stockholm, FoU-enheten, UHÄ.

Glaser, B. & Strauss, A. L. (1967) The discovery of grounded theory: Strategies for qualitative research, Chicago, Aldine. Guba, E. G. & Lincoln, Y. S. (1989) The Fourth Generation Evaluation, Newbury Park, Sage Publications. Hall, A. D. & Hagen, R. E. (1969) Definition of System. IN Litterer, J. A. (Ed.) Systems, Control and Adaptation, Volume II

(1980). NewYork, John Wiley & Sons. Hartford, K., Carey, R. & Mendonca, J. (2007) Sampling Bias in an International Internet Survey of Diversion Programs in

the Criminal Justice System. Evaluation & the Health Professions. Harun, M. H. (2002) Integrating e-Learning into the Workplace. The Internet and Higher Education, 4, 301-310. House, E. R. (1980) Evaluating with quality, London, Sage Publications. Jackson, E.T & Kassam, Y. (1998) Knowledge Shared: Participatory Evaluation in Development Cooperation, Ottawa,

IDRC/CRDI.




Jacucci, E. & Hanseth, O. (2006) Taking Complexity Seriously in IS Research. Information Technology and People, 19(1), 5-11.

Karlsson, O. (1999) Utvärdering – mer än metod. Tankar och synsätt i utvärderingsforskning, Stockholm, Norstedts tryckeri.

Katz, D. & Kahn, R. L. (1980) The Definition and Identification of Organisations. IN A, L. J. (Ed.) Organizations, Structure and Behavior. NewYork, John Wiley & Sons.

Koestler, A. (1967) The Ghost of the Machine, London, Arkana. Langerth Zetterman, M. & Lindblad, S. (2001) Learning about e-learning. A strater about Internet discourses and

borderless education. NFPF:s 29. congress. Pedagogikens mångfald. Lärande inför och utanför institutionerna. Stockholm, Sweden.

Litterer, J. A. (1973) The Analysis of Organizations, New York, John Wiley & Sons. Madaus, G. M. & Kellaghan, T. (2000) Models, Metaphors and Definitions. IN Stufflebeam, D. L. (Ed.) Evaluations

Models. Viewpoints on Educational and Human Services Evaluation. Hingham, MA, USA, Kluwer Academic Publishers.

Marine, R. J. (2002) A systems freamework for evaluation of faculty web-work. New Directions for Institutional Research year, 63-72.

Mulej, M. (2007) Systems Theory: A Worldview and/or a Methodology Aimed at Requisite Holism/Realism of Humans’ Thinking, Decisions and Action. Systems Research and Behavioral Science, 24, 347-357.

Patton, M. Q. (1986) Utilization-Focused Evaluation 2nd edition, London, Sage Publications. Patton, M. Q. (1989) A context and boundaries for a theory-driven approach to validity. Evaluation and Program

Planning, 12 (4), 375-377. Patton, M. Q. (1996) Utilization-Focused Evaluation: the new century text, Thousand Oaks, CA, Sage Publications. Petit, B. & Olsen, H. (1994) Om svar anhålles. En bok om interaktionistiskt förändringsarbete Andra reviderade

upplagan, Stockholm, Mareld. Sandelin, B., S & Sarvimäki, A. (2000) Evaluation of patient-focused health care from a systems perspective. Systems

Research and Behavioral Science, 17:6, 513-525. Schöjdt, B. & Egeland, T. (1994) Från systemteori till familjeterapi, Lund, Studentlitteratur. Simon, A. H. (1969) The Architecture of Complexity. IN A, L. J. (Ed.) Systems, Control and Adaptation Volume II (1980).

NewYork, John Wiley & Sons. Skyttner, L. (2001) General Systems Theory. Ideas & Applications, Singapore, World Scientific. Smith, N. L. (1994) Clarifying and expanding the application of program theory-driven evaluation. Evaluation Practice,

15(1), 83-87. von Bertalanffy, L. (1968) General System Theory, New York, George Braziller. Zhao, Y., Yagi, Y., Nakajima, I. & Jujozi, H. (2002) IP Telephony – New Horizon for Telemedicine and e-Health. Journal

of Medical Systems, 4(4), 277-283.

ISSN 1566-6379 109 ©Academic Conferences Ltd Reference this paper as: April, G. D. and Pather, S. “Evaluating Service Quality Dimensions within e-Commerce SMEs.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 109 - 124, available online at www.ejise.com

Evaluating Service Quality Dimensions within e-Commerce SMEs

Graham D. April and Shaun Pather e-Innovation Academy, Cape Peninsula, University of Technology, Cape Town, South Africa [email protected] [email protected] Abstract: With the continued growing investment in WWW technologies by e-Commerce businesses the measurement of Information Systems (IS) effectiveness in this business sector has become increasingly important over the last decade. As business users, especially in the SME sector, have become reliant on outsourced IS service providers for a wide range of services, the quality of service rendered by the latter is an important issue which impacts on IS effectiveness. Researchers have since the 1990s recognised the importance of service quality as a measure of IS performance. The literature suggests that IS service delivery to e-Commerce businesses needs to be evaluated differently to that of traditional brick-and-mortar businesses. There is however a paucity of research regarding IS evaluation in e-Commerce environments, including that of the application of service quality principles. It is thus difficult for managers of IS service providers in this context to develop a complete picture of the effectiveness of the IS they deliver. This paper reports on a study which investigated whether IS service quality criteria and dimensions applied in large brick-and-mortar organisations, are also applicable to SME e-Commerce businesses in the tourism sector in South Africa. In pursuit of this objective an IS-adapted SERVQUAL instrument was tested in an e-Commerce SME environment. The research results indicate that, although SERVQUAL principles are applicable to the e-Commerce SME context, the service quality dimensionality is different. The research derived four new dimensions for service quality expectations of e-Commerce SMEs viz., Credibility, Expertise, Availability and Supportiveness. A fifth dimension is the Tangibles dimension, which is retained from SERVQUAL. Furthermore the results indicate that the Credibility dimension was the most important dimension in this research context, while the Tangibles dimension was the least important. Keywords: information systems, evaluation, e-commerce, WWW, service-quality, SME, SERVQUAL, IS outsourcing

1. Introduction

Over the past 25 to 30 years researchers have developed several approaches to evaluating Information Systems (IS) effectiveness, e.g. IS usage, user information satisfaction (UIS), quality of decision-making, productivity from cost/benefit analysis, and system quality (Pather et. al. 2004). Since the 1990s IS researchers introduced a new perspective to IS effectiveness measurement, namely Service Quality. The latter has subsequently been recognised as an important performance metric in the delivery of IS (Pitt et al., 1995; DeLone & McLean, 2003; Kettinger & Lee, 2005). However, IS research in the area of service quality, has focused mainly on traditional brick-and-mortar organisations in which there is usually an in-house IS department or function. Many authors have empirically researched IS service quality as a measure of the performance of this function (e.g. Pitt et al., 1995; Van Dyke et al., 1997; Pitt et al., 1997; Kettinger & Lee, 1997; Watson et al., 1998; Kang & Bradley, 2002; Jiang et al., 2002; Bharati & Berg, 2003). However, an IS literature search revealed no such studies in an e-Commerce context1. That is, no empirical research could be found which investigates the performance of IS service providers when servicing e-Commerce businesses. This concern has been substantiated Studies in the e-Commerce environment have focused mainly on the service link between the e-Commerce business and its end-customers, i.e. website users. Furthermore, Small and Medium Enterprises (SMEs) generally outsource their IS functions in order to access the necessary IS expertise (Al-Qirim & Bathula, 2002) which are generally not available within the business. Thus external IS service providers are usually employed by these SMEs. The high reliance of these SMEs on outsourced IS service providers for the support of business critical systems implies that their service expectations could be different from those in a large brick-and-mortar organisation. Furthermore, IS service quality expectations of business units within a large organisation are influenced by issues such as corporate culture and standardised service levels agreements. However there are no such influences when an IS provider has a diverse number of SME clients, each with individualised expectations of service quality. Consequently, IS providers should be enabled with a firm understanding of service expectations of their clients in the SME e-Commerce businesses environment.

1 This concern is also substantiated by DeLone & McLean (2004), Pather et. al. (2004) and Hong & Zhu (2006).




In light of the foregoing, the research reported on in this paper examines the IS service quality expectations of SME e-Commerce businesses i.e. the clients of IS service providers. The main research objective was to investigate and determine service quality dimensions for this business environment. The principle research question of the study was: What are the Service Quality dimensions for the evaluation of service expectations of e-Commerce SMEs?

2. Literature review

2.1 The e-Commerce SME research context

The case for the research question has been made by a combination of issues raised in different studies. Firstly, Molla and Licker (2001) argue that further research is required to investigate whether IS effectiveness measurement in the e-Commerce context, should be approached differently to that of traditional IS. Secondly, the literature also suggests that the application of e-Commerce in small businesses is different from that of large businesses (Lui & Arnett, 2000; Stansfield & Grant, 2003). Although the principles of e-Commerce trading are the same regardless of the size of the business, differences arise especially with regards to how IS are managed. In-house IT units in large businesses are generally able to support and manage IS, but this is not the case for many medium, and almost all small and micro enterprises. According to Rohde (2004), SMEs also generally outsource their IS functions to external IS service providers. Consequently, the level of service quality delivered by these service providers is considered a critical success factor for SMEs (Kim et. al., 2003). Together, all these factors suggest that empirical research of IS service quality in the context of e-Commerce SMEs, would be a useful contribution to the IS effectiveness research field.

2.2 IS effectiveness research

The measurement of IS effectiveness has been widely discussed in the IS literature, and has been a long-standing concern for both academics and IS practitioners (Grover et al., 1996). IS effectiveness can be defined as the extent to which a system achieves the goals for which it was designed (Lui & Arnett, 2000). However, in today’s competitive world, IS are also expected to contribute to achieving the organisation’s mission, improve productivity and facilitate service delivery (Elpez & Fink, 2006). Researchers have had difficulty finding appropriate metrics to measure IS effectiveness (Pather et al., 2004), and many researchers have resorted to surrogate measures (Elpez & Fink, 2006). From the multitude of IS effectiveness measures, Grover et al. (1996) identified some of the more prominent measures used in the literature, viz., IS usage, user information satisfaction, quality of decision making, productivity from cost-benefit analysis, and system quality. DeLone and McLean (1992) developed a framework for classifying the multitude of effectiveness measures into six categories. They called this framework the DeLone and McLean (D&M) IS success model (Figure 1). DeLone and McLean’s research brought about some structure to IS effectiveness research (Seddon & Kiew, 1996).

Figure 1: D&M IS Success Model (Source: DeLone & McLean, 1992)


Graham D. April and Shaun Pather

Ten years after the publication of their original IS Success Model, DeLone and McLean (2003) reviewed more than 100 journal articles dealing with IS success measurement. They subsequently revised their model, producing the “Updated DeLone and McLean IS Success Model” (Figure 2). Some of the revisions to the original model were based on suggestions and re-specifications from other researchers in the field.

Figure 2: Updated D&M IS success model (Source: DeLone & McLean, 2003) Importantly, amongst various amendments to the IS success model in 2003, was the introduction of a service perspective. IS effectiveness researchers up to the mid 1990s have shown bias towards a product perspective of evaluating effectiveness, while ignoring the service-based perspective (Whyte & Bytheway, 1996; Lomerson & Tuten, 2005). One exception was the study conducted by Pitt et al. (1995), whose research was the basis for DeLone and McLean (2003) adding the Service Quality measure to their updated model (Figure 2). Pitt et al. (1995) believed that the prominence of the service-based dimension had increased since the advent of the personal computer (PC). They assert that the PC had resulted in more IS users interacting with the IS department more often. Wilkin et al. (2001) offered two ways of interpreting how service applies to the IS function. Firstly, an IS can be seen to be more than just a technical product, it can also generate value from its “capacity to serve the needs of its end-users/stakeholders” (Wilkin et al., 2001:113). Thus the whole system provides service to the stakeholders by serving their needs and providing pertinent information. The second view of service deals with the service or support delivered by the IS department or external service providers. Examples of support tasks that IS users expect the IS department to assist them with included hardware and software selection, installations, problem resolution, connection to LANs, systems development and software education (Pitt et al.,1995).

2.3 Service quality as an IS performance measure

The foundation of Service quality research in the IS literature is rooted in the work conducted by researchers in the marketing field. In the marketing literature the authors Parasuraman, Zeithaml and Berry (1985, 1988, 1991, 1993, 1994a, 1994b) have been particularly influential. These authors developed a measuring instrument called SERVQUAL, to measure service quality from the customers’ perspective. The SERVQUAL scale comprises of five service quality dimensions (Table 1) and 22 items. Table 1: SERVQUAL dimensions (Adapted from Parasuraman et al., 1988)

Dimension Dimension Meaning and Attributes Reliability Ability to perform the promised service dependably and accurately Responsiveness Willingness to help customers and provide prompt service. Assurance Knowledge and courtesy of employees and their ability to inspire trust

and confidence. Empathy Caring, individualised attention the firm provides its customers. Tangibles Physical facilities, equipment, and appearance of personnel.



Although SERVQUAL had its detractors (e.g. Cronin & Taylor, 1994)2, Parasuraman et al. (1988:31) believed that the SERVQUAL instrument could be “adapted or supplemented to fit the characteristics or specific research needs of a particular organisation”.

Pather et al. (2004), having conducted an extensive literature review in this area, found that Kim (1990) was one of the first IS researchers to introduce the service quality perspective to IS user satisfaction research. The appropriateness of the SERVQUAL instrument for the IS domain was subsequently researched by authors interested in measuring user satisfaction of the IS department. Watson et al. (1993) proposed that the SERVQUAL scale be adapted to measure IS service quality as a surrogate measure for user satisfaction. Pitt et al. (1995) subsequently tested the use of the SERVQUAL scale empirically across three IS organisational types. They concluded that the “study provides evidence that practitioners can, with considerable confidence, use SERVQUAL as a measure of IS success” (Pitt et al., 1995:182). However, Van Dyke et al. (1997) challenged the assessment of Pitt et al. (1995), and questioned the validity and usefulness of SERVQUAL in the IS domain. In response to this critique, Pitt et al. (1997) criticised Van Dyke et al. (1997) for using arguments without empirical backing and for not offering viable alternatives. They re-iterated that the developers of SERVQUAL used rigorous empirical research in their development of the model Kettinger and Lee (1997) in extending the debate agreed with the counter-arguments of Pitt et al. (1997). Kettinger and Lee (1997) adapted the 22-item SERVQUAL instrument for the IS environment, to reflect IS specific issues such as software, hardware and computer technology. After empirically testing the adapted scale in the IS domain, the authors used factor analysis techniques which resulted in a condensed 13-item scale called IS-SERVQUAL which omitted 9 of the original SERVQUAL items and had only four dimensions (the original Tangibles dimension was omitted). This derived scale was subsequently used in other IS service quality studies such as Kang and Bradley (2002) and Park and Kim (2005). The concept of the “zone of tolerance” (ZOT) (Parasuraman et al., 1994b), was another service-quality concept that captured the attention of IS researchers. The ZOT is a concept which allows for the multi-level nature of the user expectation measure. It provides for the measurement of the difference in user expectation between what the user considers an adequate level and that of a desired level of service. In a recent paper Kettinger and Lee (2005) again used the IS-adapted 22-item SERVQUAL scale to empirically test the ZOT concept in the IS domain. This time they derived an 18 Item scale across four dimensions. Unlike their previous findings (Kettinger and Lee, 1997) the Tangibles dimension was retained, and the Assurance and Empathy dimensions merged to form a new dimension which the authors labelled “Rapport”. Thus the dimensions for the derived 18-item scale, which the authors called IS-ZOT-SERVQUAL, are Reliability, Responsiveness, Rapport and Tangibles. However, both the Kettinger and Lee (1997, 2005) studies were conducted in the context of large brick-and-mortar organisations. The foregoing review supports the assertions in the introduction of this paper regarding the dominance of IS service quality studies within large organisations only. In the next section, the research design and methods are discussed. In particular, we describe the use of a survey design, the delineation of the research population, and the statistical procedures that were applied to analyse the data.

3. Research design and methodology

Following on studies such as Kang and Bradley (2002), Kettinger and Lee (1997, 2005), Pitt et al. (1995) and Watson et al. (1998), the empirical work for this study was conducted using survey design principles. The main section of the survey questionnaire consisted of the IS-adapted SERVQUAL questions that Kettinger and Lee used in their study (Kettinger & Lee, 2005). Kettinger and Lee’s adaptation entailed rewording of the 22 items3 to reflect the IS environment. These scale items were then reformatted for the current research, to focus only on the expectations of the clients of IS service providers i.e. e-Commerce

2 Cronin and Taylor (1994) critiqued the use of the SERVQUAL “perception-minus-expectations” (P-E) measure in favour of a “perception-only” measure. They called their perception-only measuring instrument SERVPERF. Parasuraman et al. (1994a) refuted many of these concerns, and defended the managerial diagnostic capability of SERVQUAL over SERVPERF. 3 In their 2005 study, Kettinger and Lee tested the 22-item SERVQUAL scale by adapting the wording for the IS environment. We have used this IS-adapted 22-item scale in this study.



SME managers. Additionally an open-ended question was included in the survey instrument to draw possible inferences about any additional service quality expectations in the research context.

3.1 Research population

The first consideration in choosing the research population was that homogeneous data from the same sector was needed, so that data from different respondents would be comparable. It would thus also be unproblematic to aggregate the data for statistical analysis. The tourism sector was selected because of the vast number of SMEs in this sector in South Africa (Warden & Williams, 2003). The tourism sector was also well suited to the application of e-Commerce and was an early adopter of e-Commerce in this country (Wynne & Berthon, 2001). As this research focused on the service requirements of e-Commerce enabled businesses, the research population chosen for this study, from within the tourism sector, was e-Commerce enabled bed-and-breakfast and self-catering accommodation businesses in the Western Cape4. An e-Commerce enabled business, for the purposes of this study, was defined as a business that was on an adoption level of 2 or higher on the Subba Rao and Metts (2003) e-Commerce adoption stage model (refer to Table 2).

Table 2: Subba Rao and Metts (2003) e-Commerce adoption stage model levels

Adoption Stage Level Stage Characteristics 1. Presence Web Content; Window to the Web; No Integration; E-mail 2. Portals Profiles; two-way Communication; E-mail; Order Placing; Cookies; No

payment Transactions 3. Transactions Integration B2B/B2C; Communities; E-Marketplaces; Auctions; 3rd Party E-

Marketplaces; Low level Collaboration; Payment Transactions 4. Enterprises Integration E2E; Full Integration; E-Business; E-commerce + CRM + SCM; Value

Chain Integration; High level Collaboration

In summary, the criteria for eligibility to participate in the survey were that the business had to be: A bed-and-breakfast or self-catering accommodation business; Based in the Western Cape, South Africa;

An SME; and

At a minimum level 2 on the adoption model of Subba Rao and Metts (2003).

3.2 The survey

The Capestay website (http://www.capestay.co.za) which is a comprehensive directory for all types of accommodation in the Western Cape was used as a basis for random sampling. This website featured category listings which allowed for the separate listings of the bed-and-breakfast and self-catering businesses. The website hosts a dedicated homepage for each listing, which provides detailed information about the establishment, as well as an online booking facility. This implied that the businesses listed would be on a minimum of Level 2 (Portals Stage) on the Subba Rao and Metts (2003) e-Commerce adoption stage model. The listings on this directory website thus provided a representative list from which to select a sample for the research. The www.capestay.co.za directory website had 1177 unique contact email addresses for bed-and-breakfast and self-catering accommodation establishments based in a wide geographic spectrum of the Western Cape region. The survey questionnaire accompanied by a cover letter was emailed to the listed 1177 email addresses. Only 48 responses were received. This is a response rate of only 4%, which although was lower than expected5, is a trend corroborated by Sheehan (2001) for email surveys.

3.3 Methodology

The survey instrument, based on the IS-adapted SERVQUAL instrument, used the same dimensions and items which had been applied in the context of large brick-and-mortar businesses in other studies. The validity of these dimensions and respective items for the research context (e-Commerce SMEs) was statistically tested using Exploratory Factor Analysis and re-affirmed using Confirmatory Factor Analysis. 4 The Western Cape is one of nine provinces in the Republic of South Africa. 5 Previous experience with the micro business sector in this region indicated that owners and managers were generally not very responsive to surveys.


http://www.capestay.co.za/

http://www.capestay.co.za/


The survey respondents were asked to rate the importance of individual service quality items taking into account their relationship with IS service providers. The objective was to assess the respondents’ service quality expectations. The mean values of all the item responses per dimension were then used to rank the service quality dimensions. Thereafter EFA was used to statistically conclude if any service quality items should be eliminated and/or if the service dimensionality needed to be adapted for the research context.

4. Data analysis

The statistical data analysis for this research followed an approach similar to the approach used by Kettinger and Lee (2005). The analysis was undertaken as follows:

1. Perform an Exploratory Factor Analysis (EFA), using Principal Component Analysis, to ascertain if there are new or different service quality dimensions in this research context. This essentially entails eliminating low-scoring instrument items and regrouping others.

2. Perform a Confirmatory Factor Analysis (CFA) on the adapted survey instrument to re-affirm the results of the EFA.

3. Assess the validity and reliability of the adapted instrument, using Cronbach alpha and the results of the CFA.

The results of the foregoing are reported on in the following sections.

4.1 Exploratory factor analysis

EFA using a Principal Component Analysis (PCA) extraction method, is used “to derive the minimum number of factors that account for the maximum portion of the total variance in an exploratory manner” (Kettinger & Lee, 2005:612). An exploratory approach, similar to the approach used by Kettinger and Lee (2005), was also used in the current research. The following factor selection criteria was used in the EFA:

1. The use of Oblique rotation i.e. using Oblimin Rotation Method;

2. Factor loading should be greater than or equal to 0.5;

3. No multiple loadings are allowed i.e. no items (rows) with multiple factor loadings greater or equal to 0.5; 4. No single loadings i.e. no factors (columns) having only one high loading item.

Items not fulfilling the selection criteria were eliminated, as indicated by a strikethrough line in Table 3. A bold font with shading indicates the highest factor loading for each of the remaining items in Table 3.



Table 3: Exploratory factor analysis Component

1 2 3 4 5

Relia1 provides you with services as promised? .426 .743 -.055 .156 -.453 Relia2 is dependable in handling your service problems? .028 .639 .141 .175 -.116

Relia3 performs services right the first time? .223 .500 .194 .304 -.732 Relia4 provides you with services at the promised time? .280 .582 .028 .656 -.616

Relia5 provides you with reliable technology and systems?

.276 .161 .229 .647 -.134

Resp1 keeps you informed about when service will be made? .377 .136 .224 .182 -.719

Resp2 delivers prompt service to you? .312 .037 .290 .123 -.838

Resp3 has the willingness to help you? .429 .661 .137 .397 -.595

Resp4 has the readiness to respond to your requests?

.796 .134 .273 .290 -.342

A1 has staff that instils confidence in you? .870 .112 .319 .650 -.222

A2 makes you feel safer in computer transactions? .673 -.032 .301 .728 -.479

A3 has staff that is consistently courteous? .842 .148 .452 .440 -.366

A4 has staff that has the knowledge to answer your questions? .549 .674 .155 .349 -.164

E1 gives you individual attention? .882 .179 .191 .235 -.200 E2 has staff that deal with you in a caring fashion?

.873 .043 .406 .458 -.421

E3 has your best interest at heart? .719 -.278 .288 .640 -.261

E4 has staff that understands your service needs? .286 .115 .154 .912 -.220

E5 has convenient business hours? .039 .053 .216 .253 -.650

T1 has up to date technology? .426 -.336 .248 .312 -.327 T2 has visually appealing premises and facilities? .220 .022 .894 .116 -.237

T3 has staff who appear professional? .426 -.036 .832 .192 -.150

T4 has useful support materials (such as documentation, training, videos, etc.)? .140 .151 .794 .415 -.471

Extraction Method: Principal Component Analysis. Rotation Method: Oblimin with Kaiser Normalization.

4.2 Regrouping of instrument items

As indicated in Table 3, eight items were eliminated, with all five dimensions being affected. After regrouping the remaining 14 items according to the highest factor loading, only one of the original SERVQUAL dimensions emerged from the Principal Component Analysis i.e. Tangibles as Component 3 in Table 3. Components 1, 4 and 5 had a mixture of items while Component 2 had only two of the original Reliability items. Table 4 summarises these item groupings



Table 4: Item regrouping

COMPONENT 1 = Supportiveness Resp4 has the readiness to respond to your requests? A3 has staff that is consistently courteous? E1 gives you individual attention? E2 has staff that deal with you in a caring fashion? COMPONENT 2 = Credibility Relia1 provides you with services as promised? Relia2 is dependable in handling your service problems?

COMPONENT 3 = Tangibles

T2 has visually appealing premises and facilities? T3 has staff who appear professional? T4 has useful support materials (such as documentation, training, videos, etc.)? COMPONENT 4 = Expertise Relia5 provides you with reliable technology and systems? E4 has staff that understands your service needs?

COMPONENT 5 = Availability Resp1 keeps you informed about when service will be made? Resp2 delivers prompt service to you? E5 has convenient business hours?

The derived dimensions shown in Table 4 contained a mixture of items from the original set of five dimensions in the IS-adapted SERVQUAL scale. These derived dimensions were then relabelled. The labels were intuitively chosen based on the criteria that the grouped items represented, i.e. each group of data was labelled based on the imagery of meaning they evoked when examined comparatively and in context (See Strauss & Corbin, 1998:105 for a more detailed explanation of labelling or coding textual data).6 The research by Wilkin and Castleman (2003) also helped guide the formulation of appropriate labels for the derived dimensions, viz., Expertise, Credibility, Availability, and Supportiveness. The rationale for these labels are as follows: Component 1 had a mixture of Responsiveness, Assurance and Empathy items. These items related to the supportive interaction and communication between the service provider and its customer. An appropriate dimension label was Supportiveness to frame the supportive service quality aspects of the service provider towards its customers. Component 2 contained only items from the original Reliability dimension. However, one other Reliability item had moved to Component 4. The remaining two items dealt with issues of service provider credibility regarding keeping service promises and handling service problems. An appropriate dimension label to frame these items was Credibility . Component 3 contained only items from the original Tangibles dimension. Although one of the original Tangibles items was omitted, the remaining items were still best framed by the Tangibles dimension label, which focused on appearances and support materials. Thus this dimension label was retained. Component 4 had a mixture of Reliability and Empathy items. These items related to the capability of the service provider to deliver reliable technology and systems that fulfil the customer needs. An appropriate dimension label for these items was Expertise, framing the ability of the service provider to provide the required systems. Component 5 had a mixture of Responsiveness and Empathy items. These items related to the timely delivery of service. An appropriate dimension label was Availability to frame issues dealing with duration and promptness of service.

6 A similar approach was followed by Kettinger and Lee (2005). One of the derived dimensions in their research had a mixture of Empathy and Assurance items. This derived dimension was then relabelled “Rapport” by the authors because “the construct items focus on an IS service provider’s ability to convey a rapport of knowledgeable, caring, and courteous support” (Kettinger & Lee, 2005:612).



4.3 Confirmatory factor analysis (CFA)

The next step in the data analysis was to perform a CFA on the data. A CFA using a Principal Axis Factoring extraction method was performed on the newly derived 14-item instrument. This was performed to confirm the dimensionality of this instrument (refer to Table 4). However, unlike Kettinger and Lee (2005), who used a second set of sample data (holdout sample) to confirm the dimensionality, this research used the same set of collected research data since the total number of usable responses were relatively small. The process followed in this research was therefore not as refined as the approach used by Kettinger and Lee (2005) and is noted as a limitation. The results of the CFA are presented in Table 5. Table 5: Confirmatory factor analysis

Factor

1 2 3 4 5

SU

PP

OR

TIV

EN

ES

S Resp4 has the readiness to respond to your requests? .797 .291 .138 -.272 .300

A3 has staff that is consistently courteous? .871 .456 .095 -.413 .486

E1 gives you individual attention? .826 .176 .150 -.303 .248

E2 has staff that deal with you in a caring fashion?

.839 .405 .063 -.512 .491

TA

NG

IBLE

S

T2 has visually appealing premises and facilities? .240 .808 .038 -.328 .197

T3 has staff who appear professional? .400 .779 -.037 -.176 .237

T4 has useful support materials (such as documentation, training, videos, etc.)? .161 .746 .282 -.513 .448

CR

ED

IBIL

ITY

Relia1 provides you with services as promised? .445 -.032 .469 -.419 .134

Relia2 is dependable in handling your service problems?

.091 .056 .875 -.122 .094

AV

AIL

AB

ILIT

Y Resp1 keeps you informed about when service will be

made? .344 .204 .159 -.835 .203

Resp2 delivers prompt service to you? .331 .314 .102 -.645 .156

E5 has convenient business hours?

.122 .195 .138 -.499 .273

EX

PE

RT

ISE

Relia5 provides you with reliable technology and systems? .263 .201 .010 -.192 .485

E4 has staff that understands your service needs?

.247 .159 .144 -.205 .826

Extraction Method: Principal Axis Factoring. Rotation Method: Oblimin with Kaiser Normalization. Garson (2007) defines a rule-of-thumb for factor loadings values to be weak if the value is less than 0.4, strong if it is more than 0.6, otherwise it is moderate. Applying this rule-of-thumb to Table 5, three of the factor loadings are considered moderate and 11 are considered strong. The results of the Confirmatory Factor Analysis thus appear to confirm the derived dimensions of the EFA in Table 3.

4.4 Validity and reliability of the derived instrument

Instrument validity entails verifying that the constructs measured by the instrument are real and reliable, and that the instrument is measuring the right content (Straub et al., 2004). Content validity is defined by Straub et al. (2004) as the indication of whether the instrument is a true representation of all the ways that could be used to measure the content of the given construct. These authors assert that content validity is established



through literature reviews and expert judges or panels. The items used in this survey instrument were based on SERVQUAL items originally tested for applicability in the IS domain by Pitt et al. (1995). Therefore the rationalization used in Pitt et al. (1995) to argue the Content Validity of the measuring instrument also applies to this research. Straub et al. (2004) posit that the two main components of Construct Validity, viz., convergent Validity and discriminant validity, can be deduced from the CFA results. The “strong” factor loadings indicate good convergent validity, because the items converge strongly to the derived dimensions. Also discriminant validity can be deduced because the factor loadings indicate that the items do not overlap across different dimensions. Reliability is an evaluation of the measurement of accuracy of the instrument and can be viewed as the extent to which the respondent can answer the same questions or close approximations the same way each time (Straub, 1989). A Cronbach alpha measurement can be used to determine reliability of a measurement instrument (Straub, 1989). A Cronbach alpha measurement of 0.7 and greater is considered reliable (Straub et al. 2004). The Cronbach alpha measurement for the derived instrument was calculated at 0.837 thus indicating good reliability.

4.5 Relative importance of the derived dimensions

The mean of the survey responses (rated 1 to 5, with 5 being most important) for each survey item, was used to deduce the importance ranking of the specific item i.e. the item with the highest mean value was regarded as the most important. The average score for each item of the derived factors (see Table 4) are tabulated in Table 6. The mean of all items that comprised each of the factors was then calculated. These mean values are indicated by a bold shaded font in Table 6. Table 6: Mean values for regrouped items

FACTOR Mean per item

Mean for Factor

FACTOR 1 = Supportiveness

Resp4 has the readiness to respond to your requests? 4.69 4.25

A3 has staff that is consistently courteous? 4.00

E1 gives you individual attention? 4.35

E2 has staff that deal with you in a caring fashion? 3.96

FACTOR 2 = Credibility Relia1 provides you with services as promised? 4.92

4.86 Relia2 is dependable in handling your service problems? 4.79

FACTOR 3 = Tangibles T2 has visually appealing premises and facilities? 2.15

2.83

T3 has staff who appear professional? 3.29

T4 has useful support materials (such as documentation, training, videos, etc.)? 3.04

FACTOR 4 = Expertise Relia5 provides you with reliable technology and systems? 4.81

4.74 E4 has staff that understands your service needs? 4.67

FACTOR 5 = Availability

Resp1 keeps you informed about when service will be made? 4.56 4.58

Resp2 delivers prompt service to you? 4.75

E5 has convenient business hours? 4.42

The results in Table 6 were used to deduce the relative importance of the derived dimensions in the research context. The ranking of the service quality dimensions in the research context was as follows:

1st – Credibility



2nd – Expertise

3rd – Availability 4th – Supportiveness

5th – Tangibles

5. Analysis of responses to the open-ended question

In this the last part of the data analysis section, the responses to the open-end question in the survey instrument are analysed i.e.

How can your e-Commerce service providers better assist or support your business to benefit from e-Commerce?

The verbatim responses are included in the Appendix. In taking an interpretive approach (Klein & Myers, 1999) the text was analysed for both literal and underlying meaning. However, no conclusions from the responses could be made on whether additional service quality dimensions are applicable to the research context. However, the responses seem to support the derived dimensions of service quality in the e-Commerce SME context. The impacted dimensions for each of the verbatim responses are also included in the Appendix. A summary of the results of the number of responses impacting on these dimensions are displayed in Table 7.

Table 7: Service Quality (SQ) dimension impacts of the responses to open-ended question

SQ Dimension No. of responses impacting on the SQ dimensions Credibility 11 Expertise 11 Availability 5 Supportiveness 4 Tangibles 2

The results in Table 7 thus appear to confirm the results of the importance ranking of the derived dimensions in the previous section.

6. Data interpretation

The EFA results indicate that all the SERVQUAL items are not needed in the research context i.e. bed-and-breakfast and self-catering accommodation eCommerce SME's in the tourism sector. Eight items were consequently omitted. These items are not necessarily unimportant, but the results suggest that these are not required in the measurement of service quality expectations in the business environment studied. The remaining items are sufficient for gathering data about the relative importance of the derived service quality dimensions. The derived dimensions, viz., Credibility, Expertise, Availability, Supportiveness and Tangibles, indicate the expected service quality focus in the research context. An explanation of the meaning and attributes of these dimensions are tabulated in Table 8.

Table 8: Meaning and Attributes of the Derived Service Quality Dimensions

Derived Dimension

Meaning and Attributes of the Derived Dimension

Credibility

The service provider should be credible in maintaining its service promises and delivering the exact system requirements as agreed with the client. The service provider should also be dependable when handling service problems after system installation.

Expertise

The service provider should have the expertise to deliver reliable systems and know-how to its clients. The service provider should also have the ability to understand the service needs of the client, and how to fulfil those needs.

Availability

The service provider should be available when service is required, and should respond promptly to service requests. The service provider should also be able to inform the client about time, duration and status of service requests.

Supportiveness

The service provider should have the readiness to help clients in a caring and supportive manner.

Tangibles

The service provider should have visually appealing premises, staff that appear professional, and supply useful support materials.



Figure 3 illustrates the change of service quality dimensionality for the specific business environment studied. The Tangibles dimension label was retained from SERVQUAL, although one of its original items was omitted. The remaining items were still best framed by this label. The other SERVQUAL dimension labels, viz., Reliability, Responsiveness, Assurance and Empathy, were replaced by Credibility, Availability, Expertise and Supportiveness.

Figure 3: Change of service quality dimensions for research context

7. Conclusion

The objective of this study was to investigate the application of an IS-adapted SERVQUAL instrument in the SME e-Commerce sector. Evidence has been provided that in the SME sector chosen for this study, service quality expectations differ from that of larger corporate organisational settings. The results of the study have yielded a validated instrument with a new set of service-quality dimensions. The dimension differences between the e-Commerce and brick-and-mortar contexts could be due to issues of trust and uncertainty in an environment that relies on outsourced IS service providers. Additionally issues related to the separation of human contact and required technology expertise could also have an influence. These issues are expanded on in the framing of each of the new service quality dimensions that follows. Credibility was identified as being the most important service quality dimension. The criteria for this dimension entail that the IS service provider be credible in maintaining its service promises and delivering the exact system requirements as agreed with the client. The service provider should also be dependable when handling service problems after system installation. This could be as a result of the technical dependence that the SMEs have on their IS service providers. The high ranking of this dimension could also be related to uncertainty of the security and reliability of the online environment, especially after the “dot-com” crash (Razi et al., 2004). Thus IS service providers need to instil confidence in their e-Commerce SME clients. One of the implications of this is that they should refrain from making service promises that they know will be difficult to honour. This also has a bearing on the extent to which SMEs successfully adopt e-Commerce, since they can only be expected to embrace the technology if service providers are perceived to be credible. Expertise was identified as the second most important service quality dimension. The criteria for this dimension suggest that the IS service provider is expected to have the expertise to deliver reliable systems and technical know-how to its clients. This dimension also incorporates the service provider’s ability to understand the service needs of the client, and how to go about fulfilling those needs, and is possibly related to the SMEs technical dependence on IS service providers. Barnes et al. (2004) report that outsourcing IS expertise was one of the problems during the “dot-com” crash. They assert that this was a weakness in e-Commerce businesses with respect to initial technology choices and on-going management and development. The lack of technical expertise by the SMEs could place additional requirements on the service provider in solving service problems. The service provider would need to have the expertise to also get to the root cause of service problems without technical advice from the client. It is very important that the IS service providers consistently provide a high level of expertise, and in so doing allow the e-Commerce SMEs to concentrate on their core business activities. The third ranked service quality dimension is Availability. The criteria for this dimension suggest that the IS service provider be available when service is required, and respond promptly to service requests. The service provider should also be able to inform the client about time, duration and status of service requests. The online environment places large importance on availability of Internet technologies. System downtime could mean revenue loss for the e-Commerce SME. These businesses firstly require reliable systems that seldom fail. But if the systems do fail, they expect prompt service and reparations. The businesses expect



the service providers to be available when there are system failures, and to have the processes and infrastructure in place to keep them updated on reparation progress. The fourth ranked service quality dimension is Supportiveness. The criteria for this dimension underscore the willingness of the IS service provider to assist clients with a caring and supportive disposition. This dimension underscores the need for service providers to be able to empathise with the IS related problems that underlie business problems confronting managers. IS service providers are evaluated by clients based on mutual interests, shared approaches to problem solving, and a compatible management culture and style (Dibbern et al., 2004). This is the foundation on which the service relationship is structured. It is thus important for IS service providers to provide such supportiveness in order to foster goodwill and trust in their clients. The last dimension, and lowest in terms of importance is Tangibles. The criteria for this dimension suggest that the IS service provider have visually appealing premises, staff that appear professional, and supply useful support materials. The finding that Tangibles is the least important dimension in the research context is not surprising considering the nature of an outsourcing relationship. SME businesses in the e-Commerce environment are less dependant on visual contact with the service provider. IS service providers in this environment are able to manage technical and other problems remotely. Consequently less physical interaction required. Thus IS service providers should focus on the delivery of reliable systems rather than on attractive and professional visual appearances of staff and premises. Lastly an understanding of these service quality dimensions is not only useful to the IS service providers, but to the e-Commerce SME business managers themselves. Business managers with a higher level of prior experience, and greater familiarity with the subject of evaluation may be more confident about the realisation of their expectations (Khalifa & Liu, 2003). These dimensions and their criteria serve to highlight to the business managers what reasonable service quality expectations are in this business environment. The study provides the foundation for much needed research in the SME sector to enhance the ability of businesses to conduct e-Commerce business on a satisfactory platform of IS. The study highlights those areas in which SMEs require support from their IS service providers. Future work will entail further testing of the derived instrument in other SME sectors, as well as application of the instrument amongst client’s of specific IS service providers.

Acknowledgements

The authors gratefully acknowledge: Financial support from the National Research Foundation (NRF) of South Africa, and of the Cape Peninsula University of Technology (CPUT) for the financial contribution to this study.

N.B. Any opinion, findings and conclusions or recommendations expressed in this paper, are those of the authors and do not necessarily reflect the views of the NRF. Corrie Uys, e-Innovation Academy (CPUT) for statistical support.

References Al-Qirim, N.A.Y & Bathula, H. 2002. IT and eCommerce Outsourcing in Small to Medium-Size Enterprises in New

Zealand: An Exploratory Research. Proceedings of the 15th Bled Electronic Commerce Conference eReality: Constructing the eEconomy, Bled, Slovenia, 17-19 June 2002.

Barnes, D., Hinton, M. & Mieczkowska, S. 2004. Avoiding the fate of the dotbombs: lessons from three surviving dotcom start-ups. Journal of Small Business and Enterprise Development, 11(3):329-337.

Bharati, P. & Berg, D. 2003. Managing information systems for service quality: a study from the other side. Information Technology & People, 16(2):183-202.

Cronin, J.J., Jr. & Taylor, S.A. 1994. SERVPERF versus SERVQUAL: reconciling performance-based and perception-minus-expectations measurement of service quality. Journal of Marketing, 58(1):125-131.

DeLone, W.H. & McLean, E.R. 1992. Information systems success: the quest for the dependant variable. Information Systems Research, 3(1):60-95.

DeLone, W.H. & McLean, E.R. 2003. The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems, 19(4):9-30.

Dibbern, J., Goles, T., Hirschheim, R. & Jayatilaka, B. 2004. Information systems outsourcing: a survey and analysis of the literature. The DATA BASE for Advances in Information Systems, 35(4):6-102.

Elpez, I. & Fink, D. 2006. Information systems success in the public sector: stakeholders’ perspectives and emerging alignment model. Issues in Informing Science and Information Technology, 3:219-231.

Garson, G.D. 2007. Factor analysis. [online] http://www2.chass.ncsu.edu/garson/pa765/factor.htm [14 March 2008].


http://www2.chass.ncsu.edu/garson/pa765/factor.htm


Grover, V., Jeong, S.R. & Segars, A.H. 1996. Information systems effectiveness: The construct space and patterns of application. Information & Management, 31:177-191.

Jiang, J.J., Klein, G. & Carr, C.L. 2002. Measuring information system service quality: SERVQUAL from the other side. MIS Quarterly, 26(2):145-166.

Kang, H. & Bradley, G. 2002. Measuring the performance of IT services: an assessment of SERVQUAL. International Journal of Accounting Information Systems, 3(1):151-164.

Kettinger, W.J. & Lee, C.C. 1997. Pragmatic perspectives on the measuring of information systems service quality. MIS Quarterly, 21(2):223-241.

Kettinger, W.J. & Lee, C.C. 2005. Zones of tolerance: alternative scales for measuring information systems service quality. MIS Quarterly, 29(4):607-623.

Khalifa, M. & Liu, V. 2003. Determinants of Satisfaction at Different Adoption Stages of Internet-Based Services. Journal of the Association for Information Systems, 4(5):206-232.

Kim, D.R., Cheon, M.J., Beugre, C.D. & Coverdale, T.K. 2003. Information systems outsourcing: exploration on the impact of outsourcing service providers’ service quality. Issues in Information Systems (IIS), 4(2):528-534.

Kim, K.K. 1990. User Information Satisfaction: Toward Conceptual Clarity. Proceedings of the International Conference on Information Systems (ICIS), Copenhagen.

Klein, H. K. & Myers, M. 1999. A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23(1):67-94.

Lomerson, W.L. & Tuten, P.M. 2005. Examining evaluation across the IT value chain. Proceedings of the 2005 Southern Association of Information Systems Conference, Savannah, Georgia, USA, 25–26 February 2005:124-129.

Lui, C. & Arnett, K.P. 2000. Exploring the factors associated with Web site success in the context of electronic commerce. Information & Management, 38(1):23-33.

Molla, A. & Licker, P. 2001. E-Commerce systems success: an attempt to extend and respecify the DeLone and McLean model of IS success. Journal of Electronic Commerce Research, 2(4):131-141.

Parasuraman, A., Berry, L.L. & Zeithaml, V.A. 1993. Research note: more on improving quality measurement. Journal of Retailing, 69(1):140-147.

Parasuraman, A., Berry, L.L. & Zeithaml, V.A. 1994b. Alternative scales for measuring service quality: a comparative assessment based on psychometric and diagnostic criteria. Journal of Retailing, 70(3):201-230.

Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1985. A conceptual model of service quality and its implications for future research. Journal of Marketing, 49(4):41-50.

Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1988. SERVQUAL: a multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64(1):12-40.

Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1991. Refinement and reassessment of the SERVQUAL scale. Journal of Retailing, 67(4):420-450.

Parasuraman, A., Zeithaml, V.A. & Berry, L.L. 1994a. Reassessment of expectations as a comparison standard in measuring service quality: implications for further research. Journal of Marketing, 58(1):111-124.

Park, J. & Kim, J.S. 2005. The impact of IS sourcing type on service quality and maintenance efforts. Information & Management, 42(1):261-274.

Pather, S., Remenyi, D. & Erwin, G. 2004. E-commerce success: the quest for IS effectiveness measurement: a conceptual framework for the e-commerce environment. South African Computer Journal, 32:34-43.

Pitt, L.F., Watson, R.T. & Kavan, C.B. 1995. Service quality: a measure of information systems effectiveness. MIS Quarterly, 19(2):173-188.

Pitt, L.F., Watson, R.T. & Kavan, C.B. 1997. Measuring information systems service quality: concerns for a complete canvas. MIS Quarterly, 21(2):209-223.

Razi, M.A., Tarn, J.M. & Siddiqui, F.A. 2004. Exploring the failure and success of DotComs. Information Management & Computer Security, 12(3):228-244.

Rohde, F.H. 2004. IS/IT outsourcing practices of small- and medium-sized manufacturers. International Journal of Accounting Information Systems, 5:429-451.

Seddon, P.B. & Kiew, M-Y. 1996. A Partial Test and Development of DeLone and McLean's Model of IS Success. Australian Journal of Information Systems, 4(1):90-110.

Sheehan, K. 2001. E-mail survey response rates: a review. Journal of Computer-Mediated Communication, 6(2). http://jcmc.indiana.edu/vol6/issue2/sheehan.html [14 March 2008].

Stansfield, M. & Grant, K. 2003. An investigation into issues influencing the use of the internet and electronic commerce among small-medium sized enterprises. Journal of Electronic Commerce Research, 4(1):15-33.

Straub, D., Boudreau, M. & Gefen, D. 2004. Validation guidelines for IS positivist research. Communications of the Association for Information Systems, 13:380-427.

Straub, D.W. 1989. Validating instruments in MIS research. MIS Quarterly, 13(2):147-169. Strauss, A.L. & Corbin, J. 1998. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded

Theory, 2nd ed. London: Sage. Subba Rao, S. & Metts, G. 2003. Electronic commerce development in small and medium sized enterprises: a stage

model and its implications. Business Process Management Journal, 9(1):11-32. Van Dyke, T.P., Kappelman, L.A. & Prybutok, V.R. 1997. Measuring information systems service quality: concerns on the

use of the SERVQUAL questionnaire. MIS Quarterly, 21(2):195-208. Warden, S. & Williams, R. 2003. Benefit’s for small medium and micro enterprises (SMMEs) in the Western Cape in the

tourism industry for adopting electronic commerce. Proceedings of the 5th Annual Conference on World Wide Web Applications, Rand Afrikaans University, 10-12 September 2003.


http://www.uow.edu.au/ajis/vol41p9.html

http://jcmc.indiana.edu/vol6/issue2/sheehan.html


Watson, W.T., Pitt, L.F. & Kavan, C.B. 1998. Measuring information systems’ service quality: lessons from two longitudinal case studies. MIS Quarterly, 22(1):61-79.

Watson, W.T., Pitt, L.F., Cunningham, C.J. & Nel, D. 1993. User satisfaction and service quality of the IS department: closing the gaps. Journal of Information Technology, 8:257-265.

Whyte, G. & Bytheway, A. 1996. Factors affecting information systems’ success. International Journal of Service Industry Management, 7(1):74-93.

Wilkin, C. & Castleman, T. 2003. Development of an Instrument to Evaluate the Quality of Delivered Information Systems. Proceedings of the 36th Hawaii International Conference on System Sciences, Big Island, HI, USA, 6-9 January 2003.

Wilkin, C., Carr, R. & Hewett, W. 2001. Evaluating IS quality: exploration of the role of expectations on stakeholders' evaluation. In Van Grembergen, W. (ed). Information Technology Evaulation: Methods and Management. Hershey PA: Idea Group Publishing: 111-129.

Wynne, C. & Berthon, P. 2001. The impact of the Internet on the distribution value chain: the case of the South African tourism industry. International Marketing Review, 18(4):420-431.

Appendix

Verbatim responses to Open-ended Survey Questions How can your e-Commerce service providers better assist or support your business to benefit from e-Commerce?

The verbatim responses to this question are listed below together with the service quality dimensions which are impacted by the responses: “Don’t let the server go down – loss of business… SP should do what they say they can and will do.”

IMPACTED DIMENSIONS: Credibility, Expertise “SP should care about me – I should be important enough to them – they should know who I am. Had a bad experience with “Webmail” – did not deliver on promise – wasted R6000.”

IMPACTED DIMENSIONS: Credibility, Supportiveness “SP should deliver on promises of increased customer awareness and bookings. Have listings on 23 website – get booking from only 3. Have invested R30 000 in listings.”

IMPACTED DIMENSIONS: Credibility “User-friendly ‘templates’. “

IMPACTED DIMENSIONS: Tangibles “Keeping us informed of technological advances that may be of benefit.”

IMPACTED DIMENSIONS: Expertise, Availability “They could give themselves a face! We have never met most of them - purely email or phone conversations.”

IMPACTED DIMENSIONS: Supportiveness, Tangibles “By constantly reviewing my website and ensuring that the website appears at the top of search engine listings.”

IMPACTED DIMENSIONS: Availability, Expertise “Our e-Commerce service provider does keep as well abreast of new technologies, systems and developments. We feel secure in their hands, and rely heavily on their expertise going forward. They have a good understanding of our business model. This we feel is the most important part of an e-Commerce provider, that they take the time to understand and develop with you (not for you) an e-Commerce solution to suit your business.”

IMPACTED DIMENSIONS: Credibility, Expertise “I rely on good websites with loads of hits leading to loads of enquiries. All my business comes from websites. Most I try are a waste of money but a couple of South African ones always come up trumps. I now don't subscribe to any new offers unless they give a free trial and can put their money where their mouth is.”

IMPACTED DIMENSIONS: Credibility, Expertise “Note: I am with MWEB who have consistently provided the best service around at almost the best rates.”

IMPACTED DIMENSIONS: Credibility “Probably more than 90% of our business is sourced from our secondary website listings. I think the small B&B market is well catered for in SA. I'm generally very satisfied with the services provided.”

IMPACTED DIMENSIONS: Credibility


http://www.research.deakin.edu.au/performance/pubs/reports/database/dynamic/output/person/person.php?person_code=carrxro&cat_code=

http://www.research.deakin.edu.au/performance/pubs/reports/database/dynamic/output/person/person.php?person_code=hewetwi&cat_code=



“More information for latest developments and current improvements. I am happy with my service provider right now but need to sort out Telkom and the skyrocketing costs of dial-up. Honestly, these are necessary evils.”

IMPACTED DIMENSIONS: Credibility, Expertise “Make sure that the business advertised reach more possible clients. Help me in my marketing endevour and win customers for my tourist business.”

IMPACTED DIMENSIONS: Expertise “I find my dealings with the e-Commerce service providers outstanding and personal attention exceptional!”

IMPACTED DIMENSIONS: Availability, Supportiveness “Don’t interrupt service delivery.”

IMPACTED DIMENSIONS: Availability “They close over December – January, but I suppose they need to take a break.”

IMPACTED DIMENSIONS: Availability “Up to date with technology and developments in that area.”

IMPACTED DIMENSIONS: Expertise “Update and Upgrade website. Confidentiality. Improve search engine ratings.”

IMPACTED DIMENSIONS: Credibility, Expertise “SP should inform me of webpages that are not working for me – are they viable.”

IMPACTED DIMENSIONS: Credibility, Expertise “None, very happy with the service that I receive, good feedback & reasonable special offer advertising – capestay.co.za”

IMPACTED DIMENSIONS: Credibility, Supportiveness “By taking a pro-active approach. “

IMPACTED DIMENSIONS: Credibility, Expertise “Ensure that search engines find our website.”

IMPACTED DIMENSIONS: Expertise

ISSN 1566-6379 125 ©Academic Conferences Ltd Reference this paper as: Ashrafi, R. and Murtaza, M. “Use and Impact of ICT on SMEs in Oman.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 125 - 138, available online at www.ejise.com

Use and Impact of ICT on SMEs in Oman

Rafi Ashrafi and Muhammed Murtaza Sultan Qaboos University, Oman [email protected] [email protected] Abstract: This paper presents the results of an exploratory study carried out to learn about the use and impact of Information and Communication Technologies (ICT) on Small and Medium Sized Enterprises (SMEs) in Oman. The study investigates ICT infrastructure, software used, driver for ICT investment, perceptions about business benefits of ICT and outsourcing trends of SMEs. The study provides an insight on the barriers for the adoption of ICT. Data on these aspects of ICT was collected from 51 SMEs through a survey instrument. The results of the study show that only a small number of SMEs in Oman are aware of the benefits of ICT adoption. The main driving forces for ICT investment are to provide better and faster customer service and to stay ahead of the competition. A majority of surveyed SMEs have reported a positive performance and other benefits by utilizing ICT in their businesses. Majority of SMEs outsource most of their ICT activities. Lack of internal capabilities, high cost of ICT and lack of information about suitable ICT solutions and implementation were some of the major barriers in adopting ICT. These findings are consistent with other studies e.g. (Harindranath et al 2008). There is a need for more focus and concerted efforts on increasing awareness among SMEs on the benefits of ICT adoption. The results of the study recognize the need for more training facilities in ICT for SMEs, measures to provide ICT products and services at an affordable cost, and availability of free professional advice and consulting at reasonable cost to SMEs. Our findings therefore have important implication for policy aimed at ICT adoption and use by SMEs. The findings of this research will provide a foundation for future research and will help policy makers in understanding the current state of affairs of the usage and impact of ICT on SMEs in Oman. Keywords: Information and communication technologies (ICT), Small and Medium Sized Enterprises (SMEs), developing countries, Gulf Cooperative Council (GCC), Middle East, Oman

1. Introduction

Today organizations of all types are utilizing Information and Communication Technologies (ICT) around the globe, not only for cutting costs and improving efficiency, but also for providing better customer service. Governments too, around the world, are adopting ICT to provide better services to their citizens. The adoption of ICT by organizations requires a business environment encouraging open competition, trust and security, interoperability and standardization and the availability of finance for ICT (UNCTAD 2004). Most of the large and international organizations in Oman have effective computer systems to efficiently conduct business. A number of large organizations have spent huge amounts of money on installing computer systems to support their business processes. However, the situation has not been the same with SMEs - similar to other parts of the world for various reasons (Parker and Castelman 2007), (Shiels et al 2003), and (Fink and Disterer 2006). No studies have been carried out on this topic in any of the GCC countries. So why Oman? Authors believe that Oman is a country with unique features such as Oman is a developing country, an oil producing country but is neither as poor as some of the other developing countries nor as rich as other oil producing countries. It does not intensely depend on foreign labor as some of the other GCC countries and is expanding its small industrial base. With its small population and a large geographical area Oman has international standard grade roads to link all major cities of the country. Similarly education and health facilities are expanded to rural and remote areas. Telephone, land lines or mobile, and Internet services are available to more than 80% of the population. The female workforce represents 17% of the total workforce of Oman which is a very distinctive feature as compared to other Arab countries of the region. Oman has limited resources (monetary and human) and has to use these resources in an efficient way learning from the experiences of other countries. These and many other geo-political, social and other factors make Oman an interesting case to study in what aspects Oman is different or similar to other countries in ICT adoption for SMEs. The government of Oman has taken various measures to diversify the economy for sustainable development of the country and one of the major steps is to transform Oman into a digital society. The adoption of ICT will have significant positive consequences on SMEs and consequently on the economy of Oman. There is a dearth of data and research about the size and contribution of SMEs towards Oman’s economy. Therefore, through this research, we would like to learn about the effects and usages of ICT on SMEs in Oman and their current and future perceptions towards ICT. This study is based on the review of literature on the topic and the Net Impact Canada 2006 study (Illuminas 2006). We hope that the findings of this research will




provide a foundation for future research and will help policy makers in understanding the current state of affairs of the usage and impact of ICT on SMEs in Oman. We collected data on the usage and impact of ICT on Omani SMEs through a survey questionnaire of 51 randomly selected SMEs in Muscat, the Capital of Oman. Definitions: It is important to note that the term "ICT" in context of this research refers to the wide range of computerized information and communication technologies. These technologies include products and services such as desktop computers, laptops, handheld devices, wired or wireless intranet, business productivity software such as text editor and spreadsheet, enterprise software, data storage and security, network security and others. SME: There is no official definition of SMEs available. In this research, we have adopted the following definitions of SMEs. Businesses with less than ten employees as a Micro Enterprise, between ten and fifty as Small Enterprises, and between fifty to two hundred and fifty employees as Medium sized enterprises. This criterion is consistent with other similar studies (Kapurubandara et al 2006) and was used to identify and qualify SMEs for the purpose of this research. In the next section we review selected literature on the areas related to the topic, followed by diffusion of ICT in GCC countries and Oman, objectives of the study and research design of the study. In the later sections we present findings of our study with conclusions, limitations and directions for future research. An earlier version of this paper was presented at and included in the proceedings of the 2008 International Conference on Information Resource Management (Conf-IRM) held at Niagara Falls, Ontario, Canada during May 18-20, 2008.

2. Literature review

Importance of SMEs: It has been widely recognized that small and medium enterprises (SMEs) not only play an important role in the economy of a country, but are crucial to the country’s economic stability. In New Zealand SMEs make up more than 99% of all businesses and account for about 60% of employment. In the USA more than half of all the employment comes from firms with fewer than 500 employees (Baldwin et al 2001). In the UK, SMEs employ 67 % of the workforce (Lange et al 2000). In most EU member states SMEs make up over 99% of enterprises, 67% of jobs and 59% of GDP. In most countries SMEs generate a substantial share of GDP and a key source of new jobs as well as a breeding ground for entrepreneurship and new business ideas. The United States of America, UK, Japan, Australia, New Zealand, Canada and other developed, as well as developing, countries are making policies to facilitate the growth of SMEs. Realizing the importance of ICT New Zealand spend about 10% of her GDP on ICT, making it the top ranking country in the world (Clarke 2004). Estimates from the World Bank indicate that SMEs have contributed over 55% percent of GDP in OECD countries and between 60 to 70 percent of GDP in middle-income and low income countries generating 60 to 70 percent employment (Oman Economic Review 2007). The above facts show that SMEs play a very important role in the growth of economy of a country, and Oman is not an exception. ICT Diffusion in SMEs: There are a number of studies that discuss adoption of Internet and e-business in SMEs in developed countries (Lucchetti and Sterlacchini 2004), (Love et al 2004), (Schubert and Leimstoll 2006 and 2007a, b), (Koellinger 2006), (Stroeken 2001), (Morikawa 2004), (Caldeira and Ward 2002), (Gregor et al 2004), and (Doczi 2002). Governments around the globe recognize the importance of adoption of ICT by SMEs and they have created special groups to study various aspects of ICT adoption in SMEs. Despite the importance of ICT and emphasis by various governments to encourage SMEs to adopt ICT, it has been reported that SMEs have been slow in adopting ICT for various reasons (Houghton and Winklhofer 2004), (Smallbone et al 2001), (Dawn et al 2002), and (Lawson et al 2003). We wanted to find out reasons for the slow adoption of ICT in Oman. Barriers to ICT Adoption: Large organizations have enough resources to adopt ICT while on the other hand SMEs have limited financial and human resources to adopt ICT. (Duan et al 2002) identified lack of ICT skills and knowledge in SMEs as one of the major challenges faced by all European countries, particularly in the UK, Poland and Portugal, in their study. (Houghton and Winklhofer 2004) have reported a slow response of SMEs relating to adoption of ICT. (Shiels et al 2003) found that characteristics of the firm and industry sector are contributory factors to the adoption and exploitation of ICTs by SMEs. (Kapurubandara et al 2006) have categorized internal and external barriers that impede adoption of ICT by SMEs in a developing


Rafi Ashrafi and Muhammed Murtaza

country. The internal barriers include owner manager characteristics, firm characteristics, cost and return on investment, and external barriers include: infrastructure, social, cultural, political, legal and regulatory. ICT Diffusion in developing countries: There are very few studies about ICT adoption in developing countries (Temtime et al 2003), (Mutula et al 2006), (Yeh et al 2007), (Ssewanyana et al 2007), (Kapurubandara et al 2006). (Lal 2007) investigating adoption of ICT in Nigerian SMEs, found that one of the major factors inhibiting ICT diffusion and intensive utilization is poor physical infrastructure. In developing countries some of the ICT adoption challenges include legal and regulatory issues, weak ICT strategies, lack of R& D, excessive reliance on foreign technology and ongoing weaknesses in ICT implementation (Dutta et al 2003).

3. ICT diffusion in Gulf Cooperation Council (GCC) countries and Oman A recent market survey shows that GCC countries’ current IT spending are $5 billion annually and are expected to be doubled by 2010. Oman is emerging as one of the region’s strong and fast growing markets. Oman’s IT market is expected to grow from $230 million in 2005 to over $400 million by 2010 (GulfBase 2008) and (Inno Vest Group 2008). Currently, the average ICT spending in the region are between 2% to 4% of GDP as compared to an average of 8% in developed countries. Also, PC penetration is below the world average. The highest growth rate of internet usage access over the past six years is an encouraging sign in the Middle East (Patel 2007). This shows that GCC countries including Oman have made strides in advancing the development of ICT sector yet they have to go a long way ahead to reach to developed world averages. Most of the ICT activities in these countries like in other developing countries are based on government’s initiatives and policies.

3.1 Use of ICT in Oman:

An Arab Gulf country in the Middle East, Oman has a population of 3.1 million. According to a recent (UN 2008) report, among some of ICT adoption indicators, Oman has 12.22 Internet per 100 users, 5.06 PC per 100 users, 69.6 Cellular subscribers per 100 users, 10.65 main telephone lines, and 0.58 broadband per 100 users. Oman has a Web Index of 0.4849 and ranked 52 in the world, an Infrastructure Index of 0.1559, Human Capital Index 0.7659, and e-Government Index of 0.4691. Oman has an e-participation index of 0.2045 and rank 60 among the world. It is at the bottom of the list of the GCC countries appearing on most of the above mentioned ICT indicator indices. Realizing the importance of ICT for the economic development of the country, Oman’s government has placed a great emphasis upon creating a digital economy as key drivers for the sustainable growth of the country. Oman is heavily investing in ICT as one of the building blocks to diversify the economy. In September 2003 Oman established Knowledge Oasis Muscat (KOM 2008), a technology park as one of the initiatives taken to help develop a knowledge-driven economy, attract investment, and to serve as an incubator for local start up companies in the ICT sector. Also, in 2006 the government of Oman created an Information Technology Authority (ITA 2008) for developing a national IT strategy, help facilitating and implementing an ICT infrastructure and overseeing the implementation of Oman’s digital strategy. The government has encouraged private sector to open universities and technical colleges in order to increase ICT literacy in the country. Government of Oman has established a Sanad program to encourage entrepreneurship and develop SMEs by providing them necessary finance, guidance and training. From the private sector a number of organizations are contributing in the growth of SMEs. For example Shell has established a $10 million Intilaaqah Enterprise Fund to provide capital and on-going support for SMEs in Oman. Microsoft has signed a Memorandum of understanding with Ministry of Education to provide training in schools. Also, there are a number of other initiatives in place in order to transform Oman into a digital society. It is hoped that all these steps will increase diffusion of ICT in Oman in general including SMEs in near future. According to an estimate of HSBC Middle East Bank there are only 15,000 to 20,000 SMEs in Oman generating only 10-20% employment (Oman Economic Review 2007). These estimates show that there is a significant potential for the SMEs in Oman to grow in terms of contribution to GDP and employment, and to be competitive at regional and international levels. Since there is no data available or a study carried out on the impact of ICT on SMEs in Oman, we were interested to know about the present technological infrastructure, the reasons behind ICT investment, the restrictions and barriers for adoption of ICT, and the implementation methods and benefits of ICT investments. These aspects essentially formed the basis of our research.



4. Objectives of the study

Realizing the importance of SMEs for the economy of Oman and the impact of ICT on improving the performance of SMEs, data was collected on various aspects of ICT usage and utilization. The investigation took place during January and April 2008 and is the first step towards an exploratory study on learning more about the use and impact of ICT on SMEs in Oman. Through this study we aim to find answers to the following questions:

1. What is the level of usage of ICT in terms of ICT infrastructure, internet connection, IT staff and types of website used in SMEs?

2. What Enterprise software SMEs used?

3. What are the drivers for ICT Investment? 4. What percentage of budget is allocated to ICT?

5. What are long term business goals for investment in ICT?

6. How is competition among SMEs? 7. What are the barriers towards to ICT Investment?

8. What approach SMEs use to differentiate their business from competitors?

9. What are realities and perceptions about business benefits of ICT implementation? 10. What are out-sourcing trends?

11. What internal capabilities and processes SMEs have in place for managing ICT?

5. Research design

We performed the following tasks to conduct the research and determine the usages and effects of information and communication technologies in SMEs in Oman:

1. A questionnaire was prepared based on the review of current literature and the Net Impact Canada 2006 study (Illuminas 2006) to determine the usages, effects and perceptions of Omani SMEs towards ICT.

2. The questionnaire contained 22 questions related to business aspects of the organization, ICT infrastructure, use of internet, website, drivers for ICT investment, barriers to ICT adoption, competition, and benefits of ICT.

3. The questionnaire was distributed to a number of SMEs in Muscat area, the capital of Oman. Fifty one completed surveys were received from the companies who have adopted ICT. Those companies who do not use any form of ICT are not included in our study. The questionnaires were completed by the founder, general manager or accounts manager because of their ability and understanding of the issues investigated in the questionnaire. In very few cases more than one individual representing the enterprise filled out the questionnaire - this actually ensured preciseness.

4. Based on the 51 completed survey questionnaires, simple statistics were carried and logical inferences were made to determine the general usages and effects of ICT on SMEs in Oman.

6. Research findings

Based on our definition of SMEs, 41% of the respondents can be classified as Micro Enterprises (less than 10 employees), 41% as Small Enterprises (between ten and fifty) and 18% Medium sized Enterprises (between fifty and two hundred fifty). This shows that more than 80% of SMEs in our sample belong to Micro and Small Enterprises. Table 1: Types of SMEs based on number of Employees

Types of SMEs Percentage Micro Enterprises 41 Small Enterprises 41 Medium Sized Enterprises 18

6.1 ICT usage

We investigated IT infrastructure, Internet connection type, IT staff, usage of enterprise software, and type of website used as a measure for ICT usage.



6.1.1 ICT infrastructure

Figure 2 shows that desktop, laptop or handheld computers are used by 100% of the surveyed SMEs. The reason is that our sample is based on only those SMEs who have adopted some form of ICT in their business (e.g. computers). During our survey we noticed that a large number of Micro Businesses (we estimate 80% or more) do not use any form of ICT. It is not surprising as in the UK, 30% of micro businesses do not use computers at all (Pritchard 2006). The reason seems to be the older generation of Owner/Manager who are not used to ICT and/or are low educated foreign labor force (with no or little knowledge of ICT). Business productivity software such as Microsoft Word, Excel and PowerPoint were used by 82% of the surveyed firms. About 67% of the firms use enterprise software such as CRM, Inventory Management, E-Commerce or ERP. Wired computer network solutions such as servers, routers and firewall were utilized in 45% of the SMEs. Data storage and security solutions (such as file servers, storage area network or web-based storage) were used by 25% of the surveyed firms, and wireless networking technologies (such as access points and wireless routers) and network security solutions by only 14% of the organizations surveyed.

100%

45%

14%

82%

67%

25%

14%

Sol

utio

ns

Usages

Netw ork Security Solutions

Data Storage Solutions

Enterprise Softw areApplications

Business productivitysoftw are

Wireless netw orkinghardw are/softw are

Wired netw orkinghardw are/softw are

Desktop computers, laptopsor handheld devices

Figure 1: ICT infrastructure

6.1.2 Internet connection type

The type of internet connection in organizations largely indicates the required bandwidth and frequency or purpose of usage. In Oman the SMEs have shown no reluctance in subscribing high-speed broadband internet connection. This might be because of the relatively lower costs and higher speeds as compared to the dial-up, but nevertheless, business heavily used internet as indicated by the surveyed enterprises. Overall, 86% of SMEs in our survey used Internet. Figure 2 shows that 49% of the respondents use high speed broadband (ISDN, ADSL, DSL), 35% use dial up connection, 2% use Satellite and 16% have no internet connection. This is consistent with other studies. In a recent survey of SMEs in the UK, 78% of SMEs use Internet in their business (Harindranath et al 2008).

35%

49%

0%

2%

16%

Typ

e

Percent

No Internet

Satellite Connection

Very High Speed (T1, T2)

High-Speed Broadband (ISDN,ADSL or DSL)

Dial Up (less thank 56 Kbps)

Figure 2: Type of Internet connection

6.1.3 IT staff

One may assume that small and medium sized enterprises often lack the supporting workforce needed, such as IT staff, for the quality and quantity of the required support; the expertise of such staff are too little and



basic. However some do have a special IT department with multiple supporting staff. The decisive factors that determine the existence of such staff in SMEs around the world are the size of the firm and more importantly the relative complexity of the adapted ICT solutions. The same is the case with SMEs in Oman as around 65% of the surveyed firms do not have full-time IT staff (similar to 63% SMEs in Canada), and 35% of the firms have IT/IS departments with full-time IT staff.

Usage of enterprise software

Investment in ICT infrastructure enables businesses to take advantage of the large number of different technologies available in the market. No matter how small the business and how basic the infrastructure, businesses always look to automate and computerize the essential business functions to save costs and time and to eliminate the need for support staff. Almost 84% of the surveyed firms use finance and accounting enterprise software such as Tally and Peachtree. It was found that even some of the smallest family run businesses used such software. Inventory management software is being used by 69% of the firms and 31% of the enterprises use customer relationship management (CRM) software as illustrated by Figure 3.

84%

69%

31%

16%

8%

4%

14%

Typ

e of

Sys

tem

Usage (%)

Resource Planning

e-Commerce

Supply Chain Management

Human ResourceManagement

Customer RelationsManagement

Inventory Management

Finance/Accounting

Figure 3: Enterprise system usage It was observed that organizations that invested in enterprise software are of medium size, in terms of the size of workforce or annual turnover within the SME category, or they are micro enterprises with high inventory turnover. For example, one small retail business run by a single person (owner) used enterprise software for automating accounting, inventory and customer related business processes. Figure 3 shows that a majority of SMEs in Oman are utilizing basic Enterprise software such as finance/accounting and inventory management in their businesses.

6.1.4 Types of website used

Unlike SMEs in developed countries, the SMEs in Oman have not managed to utilize and use commercial websites. Results shows that only 2% of the firms have commercial websites which helps them reach new customers. As for informational websites, 27% of the surveyed enterprises run websites which simply introduce the business and publish their contact information. About 73% of the businesses had no website, the main reason was lack of internal staff and high maintenance costs in the long term. This shows that SMEs are not fully utilizing Internet. This shows that ICT usage within SMEs who have adopted ICT is relatively moderate in common technologies (such as desktop, laptop, productivity software etc) but limited in the more sophisticated technologies such as wireless, data storage and network security solutions (in those companies who have adopted ICT). About 86% of surveyed SMEs use Internet but only 27% have a website and only 2% have commercial website. Also, 65% do not have full time ICT staff. The results seems to be very similar to other studies (Illuminas 2006) and (WestFocus 2007)

6.2 Investments in ICT

Many of the decision makers in small and medium firms are not always aware of relevant ICT that could revolutionize their business. This in itself is a barrier to ICT investment. As for those enterprises that have already invested and are willing to continue, what really encourages them to do so? And what is the proportion of their investment? Also, what is the nature and true reason behind their investments? We have used amount of budget dedicated to ICT, drivers for ICT Investment, number of competitors, long term ICT



investment goals and organization’s strategy for competitive advantage as key elements to learn about reasons for ICT investment.

6.2.1 IT budget

Figure 4 shows that 70% of the SMEs assign less than 10% of their annual budget to ICT investment, only 2% of the firms invest between 10% and 20%, and 8% of the surveyed firms assign greater than 40% of their budgets to ICT.

70%

20%

2%8%

<10%

Between 10% and 20%

Between 20% and 40%

>40%

Figure 4: Proportion of budget assigned to ICT

6.2.2 ICT investment drivers

Regardless of the relative proportion of budget invested in ICT certain stimuli exist that encourage and push decision makers in SMEs to invest in ICT. The main driving forces for ICT Investment are to provide better and faster customer service (65%), and to stay ahead of competition (69%). These findings are in consistent with WestFocus that found three biggest perceived benefits of ICT usage were keeping up with the competition, faster response to customers and improved quality. Figure 5 shows stimuli for ICT investments in SMEs.

45%

65%69%

25%

2%6%6%

Reasons

Per

cent

Demands of stakeholders

Better and faster customer supportand services

To stay ahead of competition

Follow ing the strategy set by topmanagement

Advice from consultants

Demand of your suppliers

Government requirements

Figure 5: ICT investment drivers

6.2.3 Number of competitors

Most of the business decisions are based on the competition in the market. Figure 6 shows that 50% of the enterprises have more than twenty direct competitors. About 16% of SMEs have between ten and nineteen competitors, 16% between five to nine competitors and 14% have between one to four competitors. This shows that there is reasonably strong competition among SMEs in Oman, and perhaps that is one of the main reason that SMEs would like to be more competitive by adopting ICT.



14%

16%

16%

50%

4%

1-4

5-9

10-19

20+

Don't Know

Figure 6: Number of direct competitors

6.2.4 Long term goals for ICT investment

Most of the surveyed firms’ long-term plan with regards to their ICT investment was to increase their market share, grow their sales revenue and cut costs and expenses. About 86% of the firms expect to increase sales revenue, 67% expect to increase their market share and about 55% expect to cut costs as shown in figure 7.

86%

67%

55%

Strategy

Per

cent

Growth of sales revenue

Increase market share

Cutting costs

Figure 7: Long term plans The above results show that Omani SMEs have strong competition and are devoting a reasonable percentage of their budget to ICT in order to provide better and faster customer service. Also to realize benefits of ICT investment, SMEs expect to increase sales revenue and market share as well to reduce costs. Most of the SMEs aim to provide highest quality products and services to their customers as well as establishing long term relationships with customers. (Harindranath et al 2008) also found increasing sales and reducing costs as main reasons for investing in ICT.

6.3 Competitive strategy

A common way of achieving business goals is to differentiate one’s business from the competition. The surveyed SMEs have chosen the approach of providing the highest quality products and services (80%) to their customers as the principal method for differentiating as well as establishing long-term relationships with customers (47%), as illustrated in Figure 8.



31%

80%

47%

27%

Per

cent

Approach

By providing the mostinnovative products orservicesBy establishing the bestrelationship withcustomers By providing the highestquality products orservicesBy providing the lowestcost products or services

Figure 8: Approaches to achieve competitive advantage

6.4 Barriers to ICT investment

While some of the SMEs in Oman are clearly aware of ICT and its benefits, there exist certain restrictions and barriers to ICT investment. Figure 9 shows that 75% of the firms feel that a lack of necessary internal skills is a major barrier. This is consistent with other studies that SMEs do not have enough human resources (Wymer and Regan 2005). There has been a recent increase in technological colleges and the general investment by the Omani government in the ICT industry to overcome shortage of IT staff. About 63% feel that the monetary costs of ICT solutions and implementation are too high. (Harindranath et al 2008) also identified cost as the single most factor threatening future investment in ICT. Almost 63% of the decision-makers within the surveyed firms feel that there is not enough information available at their disposal about relevant and effective technologies. This finding also confirms findings of (Harindranath et al 2008), (Chibelushi 2008) and (WestFoucs 2007) study who found concerns over costs and uncertainty over the business benefits-followed by a lack of internal expertise. This shows that there is a need for free advice and relevant information for SMEs. Of the respondents, 31% feel they simply have no time to implement the projects. About 47% of the firms are uncertain about retaining their ICT investment and 18% of the managers feel there is not enough support from the top-management in the firms. Other barriers identified include: government regulations and requirements (6%), and bad experiences in the past (8%). A few of the firms also complained about lack of infrastructure in certain areas in the country. One business specifically complained about the unavailability of internet access in some of the areas far away from Muscat, the capital of Oman. Figure 9 shows some of the main barriers to ICT adoption of SMEs in Oman.

63%

31%

75%

47%

63%

18%

Barrier to Investment

Per

cent

age

Monetary costs of implementation

Time to implement ICT project

Lack of necessary internal skills

Uncertain about retain on investment

Lack of available information about relevanttechnologies

Lack of top-managementsupport/direction/planning

Q

Figure 9: Barriers to ICT investment



6.5 Realization of business performance improvements

We solicited opinions about the actual and expected benefits of ICT investment in improving the business in a series of questions. Table 2 shows that only 27% SMEs have realized business impact beyond their geographical area. The reason is that only 27% of SMEs have website. It is worth noting that 80% have realized reducing costs due to ICT, while 53% have improved revenue and 57% have an impact on customer relationship due to ICT. Almost all of the SMEs are more positive of future expected benefits of ICT implementation and investment. This shows that those SMEs who have adopted ICT have realized benefits due to ICT and are very positive in continuing to invest and harvest those benefits. (Harindranath et al 2008) also found that of the SMEs were satisfied with their ICT investment and perceived their ICT investment as offering good value for money.

Table 2: Actual and expected business performance improvements due to ICT

Impact of ICT

Actual Business Performance

Expected Business Performance

Yes No Yes No Beyond their geographical area 27% 73% 41% 59% In cutting costs 80% 20% 84% 16% On revenue 53% 47% 55% 45% On customer relationship 57% 43% 67% 33%

6.6 ICT implementation and outsourcing

One of the solutions to solve the problem of unavailability of internal skills many organizations have the option of “outsourcing” their ICT related work. Therefore, what we know as outsourcing has become one of the most common practices performed by almost all organizations, both private and public, small and large, in order to remain competent within their fields of business or work.

6.6.1 Proportion of activities outsourced

Majority of the SMEs that participated in this research outsourced a portion of their ICT activities and work. Figure 21 shows that 57% of SMEs outsourced more than 50% of their activities, 14% outsourced between 10 and 25 percent of their activities, and 14% outsourced less than 10% of their ICT activities. This relates to the shortage of IT staff, lack of internal skills, lack of relevant information and guidance on relevant technology solutions and high costs of ICT mentioned earlier sections. (Harindranath et al 2008) study who found that 50% of the firms in their survey used external consultants in ICT matters.

14%

14%

8%

57%

8%

Act

ivity

Proportion Outsourced

None

>50%

Between 25% and 50%

Between 10% and 25%

<10%

Figure 10: Proportion of ICT activities outsourced



6.6.2 Resources used for ICT implementation

It is important to know what ICTs were adopted and used within SMEs, but it is also important to know which internal or external resources were used to implement these technologies. Most of the enterprises surveyed used internal resources to implement basic technologies such as the installation and setup of desktop or laptop computers, data storage hardware or business productivity software. As the technologies become more sophisticated, like the use of enterprise software applications, wireless networking or the use of mobile phone applications, the firms tended to use external resources. Figure 22 shows that 33% organizations use internal resources for Desktop or laptops, as well as 24% for data storage solutions and 24% for business productivity software. About 59% of the SMEs outsourced more sophisticated technologies such as enterprise software applications. Also, it seems that some of the organizations lack internal capacity for basic technologies such as Desktop or laptops (47%), business productivity software (41%) and data storage (35%), as they used outsourcing for these activities A number of firms use both internal and external resources to implement some of the technologies. It was observed that SMEs often use external resources to initially implement the technology, and then tend to use internal resources to maintain and upgrade the implemented technologies.

24%

10%

24%

33%

12% 10%6%

35%

10%

41%

47%

8%

18%

59%

8%

2%

24%20%

4%

16%10%

Dat

a st

orag

eso

lutio

ns

Wire

less

netw

orki

ngso

lutio

ns

Bus

ines

spr

oduc

tivity

softw

are

Des

ktop

PC

s,la

ptop

s or

hand

held

PC

s

Net

wor

k se

curit

yse

rvic

es

Net

wor

king

hard

war

e/so

ftwar

eso

lutio

ns

Ent

erpr

ise

softw

are

appl

icat

ions

ICT Solutions

Tre

nd P

erce

ntag

e

Internal

External

Both

Figure 11: Resourcing trends – Internal vs. external

This shows that most of the SMEs lack skills and resources in Computers, ERP, Business software, data storage and network solutions. They look for external help in almost all areas of ICT. This emphasizes need for ICT training for SMEs. (Harindranath et al 2008) and (Chibelushi 2008) also found that lack of ICT expertise as one of the main barrier in their study.

7. Conclusions

We found that ICT usage within SMEs in Oman is moderate in common technologies, but limited in the more sophisticated technologies such as wireless, data storage and network security solutions. It was noticed that Omani SMEs are taking a comprehensive approach to their ICT investment focusing on both strategic and operational aspects of their business. The results of our study show that Omani SMEs are making reasonable investment in ICT and that there is a modest competition among the SMEs. The main driving forces for ICT investment was to provide better and faster customer service, to stay ahead of competition and following top management strategy. The competitive strategy for the majority of SMEs was to provide high quality products and services to their customers and to establish long term relationships with customers.



Nearly half of the respondents have realized business benefits of ICT adoption such as better customer relationships, increase in revenue and in reducing costs. Two thirds of the respondents are very positive about increasing their business performance in the future. With regard to barriers to ICT investment, three fourths of the firms feel that a lack of necessary internal skills is a major barrier. More than half of the respondents feel that the costs of implementation are too high. Lack of availability of relevant information and advice on suitable and effective technologies is also one of the major barriers. Other barriers include SMEs having no time to implement ICT projects, lack of top management support, bad experience in the past and government regulations and requirements. These findings are consistent with other studies e.g. (Harindranath et al 2008). This emphasizes the need for more training facilities in ICT for SMEs, measures to provide ICT products and services at an affordable cost, and availability of free professional advice and consulting at reasonable cost to SMEs. Our findings therefore have important implication for policy aimed at ICT adoption and use by SMEs. More than half of the participants outsourced over 50% of their activities. This can be related to lack of in-house capabilities in ICT identified as a major barrier. These results also confirm findings of (Harindranath et al 2008) and (Chibelushi 2008) and re-emphasize the need of ICT training for SMEs. Overall, it seems that only a small number of SMEs in Oman are aware of the benefits of ICT adoption. The findings of our research show that the SMEs lack necessary ICT knowledge and skills and mechanism to find and receive advice and support. There is a need for more focus and concerted efforts on increasing awareness among SMEs on the benefits of ICT adoption in order for SMEs to be more productive and competitive. Also, there is a need for providing affordable ICT products, services, solutions and relevant professional advice for SMEs. There is a need for government and professional trade organizations (such as Chamber of Commerce and Industry) to address the gaps and issues identified in this study. The findings of this research will provide a foundation for future research and will help policy makers in understanding the current state of affairs of the usage and impact of ICT on SMEs in Oman.

8. Limitations of the research and directions for future research

This study was a preliminary exploratory study to learn about a few selected aspects of ICT adoption in a GCC country, Oman. There are number of issues such as legal, regulatory, interventions from the government, just to name a few, in the adoption of ICT that needs further investigation. A detailed study should take a more comprehensive approach considering a wide range of areas of ICT adoption. These results are based on a small sample of 51 SMEs, out of that 80% belongs to Micro and Small enterprises. Data was collected from SMEs who use some form of ICT in their business. Those organizations that do not use computers were excluded from the study. Our conservative estimate is that currently, more than 80% of Micro businesses do not use any form of ICT for reasons mentioned earlier in the paper. The results show a general trend and practices of the use and impact of ICT on SMEs in Muscat, the Capital of Oman. The results should be interpreted or used with these perspectives in consideration. A larger sample is needed to further validate these trends.

References Baldwin, J.R., Jarmin, R.S., and Tang, J. (2001) “The Trend to Smaller Producers inManufacturing in Canada and the

US”, Statistics Canada Working paper. Caldeira, M.M., and Ward, J.M. (2002) “Understanding the Successful Adoption and Use of IS/IT in SMEs: an

Explanation from Portuguese Manufacturing Industries”, Information Systems Journal, Vol. 12, No. 2, pp 121-152. Chibelushi, C. (2008) “ICT industry challenges in adopting ICT: a case study from the West Midlands, UK”, in

proceedings of the 2008 International Conference on Information Resource Management, May 18-20, Niagara Falls, Canada.

Clarke, J. (2004) "Information Systems Strategy and Knowledge-based SMEs in the Australian Biotechnology Industry: Does IS need to reorient its Thinking?" 15th Australian Conference on Information Systems, Hobart, Australia, 1-3 December.

Dawn, J., Podonik, P., and Dhaliwal, J. (2002) “Supporting the E-business Readiness of Small and Medium-Sized Enterprises: Approaches and Metrics”, Internet Research Vol. 12, No. 2, pp139-195.

Doczi, M. (2002) “Information and Communication Technologies and Social and Economic Inclusion", Information technology Policy Group, Ministry of Economic Development, Manatu, Ohanga, NZ, March.

Duan, Y., Mullins, R., Hamblin, D., Stanek, S., Sroka, H., Mavhado, V., Araujo, J. (2002) “Addressing ICTs Skill Challenges in SMEs: Insights from three country investigations”, Journal of European Industrial Training, Vol. 26, No. 9, pp. 430-441.



Dutta, S., and Coury, M, E. (2003) “ICT Challenges for the Arab World”, Chapter 8, in Dutta, S., Lanvin, B. & Paua, F. (ed.) “The Global Information Technology Report 2002-2003: Readiness for the Networked World”, World Economic Forum/Oxford University Press, New York, pp116-131.

Fink, D. and Disterer, G. (2006) “International Case Studies: To what extent is ICT infused into the Operations of SMEs?”, Journal of Enterprise Information Management, Vol. 19, No. 6, pp 608-624.

Gregor, S., Fernandez, W, Holtham, D, Martin, S, Vitale, M, & Pratt, G. (2004) "Achieving Value from ICT: Key Management Strategies", Department of Communications, Information Technology and the Arts, ICT Research Study, Canberra.

GulfBase (2008) “GCC Common market set to enrich ICT sector” [online] http://www.gulfbase.com/site/interface/NewsArchiveDetails.aspx?n=49939 [11 March 2008]

Harindranath, G., Dyerson, R. and Barnes, D. (2008) “ICT Adoption and Use in UK SMEs: a Failure of Initiatives? The Electronic Journal Information Systems Evaluation Vol. 11, No. 2, pp. 91-96.

Houghton, K.A. and Winklhofer, H. (2004) “The Effect of Website and E-commerce Adoption on the Relationship between SMEs and Their Export Intermediaries”, International Small Business Journal, Vol. 22, No. 4, pp. 369-388.

Illuminas (2006) “Net Impact Canada 2006”, [online], http://www.netimpactstudy.com/ca/ {20 March 2008] Inno Vest Group (2008) “The Technology Small and Medium-Sized Enterprises (SMEs) in Middle East Region”, [online],

http://www.theinnovestgroup.com/43.html [12 March 2008] ITA (2008), “Information Technology Authority, Oman” [online] http://www.ittf.gov.om/english/about_ita.html [15 March

2008] Kapurubandara, M., and Lawson, R. (2006), “Barriers Adopting ICT and E-commerce with SMEs in Developing

Countries: An Exploratory Study in Sri Lanka”, CollECTeR ’06, 9 December, 2006, Adelaide, [online], http://www.collecter.org/archives/2006_December/07.pdf [13 March 2008]

Koellinger, P. (2006) “Impact of ICT on Corporate Performance, Productivity and Employment Dynamics”, Special Report No 01/2006, European Commission Enterprise & Industry Directorate General, Berlin.

KOM 2008, Knowledge Oasis Muscat [online] http://www.kom.om/ev.htm [12 March 2008] Lal, K. (2007) “Globalization and Adoption of ICTs in Nigerian SMEs”, Science, Technology Society, Vol.12, No. 2,

pp217-244. Lange, T., Ottens, M., and Taylor, A. (2000) “SMEs and Barriers to skills development: A Scottish Perspective”, Journal

of European Industrial Training, Vol. 24, No. 1, pp. 5-11. Lawson, R., Alcock, C., Cooper, J., Burgess, L. (2003), “Factors Affecting the Adoption of Electronic Commerce

Technologies by SMEs: An Australian Study”, Journal of Small Business Enterprise and Development Vol. 10, No. 3, pp. 265-276.

Love, E.D., Irani, Z., Edwards, D.J. (2004), “Industry-centric Benchmarking of Information Technology Benefits, Costs and Risks for Small-to-Medium Sized Enterprises in Construction", Automation in Construction, Vol. 13, No. 4, pp. 507-524.

Lucchetti, R. and Sterlacchini, A. (2004) “The Adoption of UCT among SMEs: Evidence from an Italian Survey", Small Business Economics Vol. 23, No. 2, pp.151-168.

Morikawa, M. (2004) “Information Technology and the Performance of Japanese SMEs”, Small Business Economics Vol. 23, No. 3, pp.171-177.

Mutula, S.M., and Brakel, P. (2006) "E-readiness of SMEs in the ICT Sectors in Botswana with Respect to Information Access", The Electronic Library, Vol. 24, No. 3, pp.402-417.

Oman Economic Review, 2007, “SME Forum – Making Life Easy”, [online]. http://www.oeronline.com/php/2007_dec/sme_forum.php [8 March 2008]

Parker, C.M., and Castelman, T. (2007) "New Directions for Research on SME-eBusiness: Insights from an Analysis of Journal Articles from 2003 to 2006", Journal of Information Systems and Small Business, Vol. 1, No. 1/2, pp. 21-40,.

Patel, F. (2007) “The Arab business world’s ICT opportunity”, Arabian Business World (23 August 2007). Pritchard, S. (2006) “How can so many businesses cope without computers” Financial Time London (19 Sep. 2006). Schubert, P., and Leimstoll, U. (2006) “The Importance of ICT: An Empirical Study in Swiss SMEs” in 19th Bled

Conference eValues, Bled, Slovenia, June 5-7. Schubert, P., and Leimstoll, U. (2007)a “Importance and Use of Information Technology in Small and Medium-Sized

Companies”, Electronic Markets, Vol. 17, No. 1, pp. 38-55. Schubert, P., Fisher, J., and Leimstoll, U. (2007)b “ICT and Innovation in Small Companies” 15th European Conference

on Information Systems, St. Gallen, Switzerland, 1226-1239, 7-9 June. Shiels, H., McIvor, R., and O’Reilly, D. (2003) “Understanding the Implications of ICT adoption: Insights from SMEs”,

Logistics Information Management, Vol.16, No. 5, pp. 312-326. Smallbone, D., North, D., Vickers, I., and Roper, S. (2001) “Innovation and the Use of Technology in Manufacturing

Plants and SMEs: An Inter-Regional Comparison”, Proceedings 24th ISBA National Small Firms Conference: Exploring the Frontiers of Small Business (Hinckley, Leicestershire: ISBA), pp. 289-308.

Ssewanyana, J., Busler, M. (2007) “Adoption and Usage of ICT in developing Countries: Case of Ugandan Firms”, International Journal of Education and Development using ICT, Vol. 3, No. 3, [Online] journal, available at http://ijedict.dec.uwi.edu/viewarticle.php?id=349 [March 13 2008]

Stroeken, J.H.M. (2001) “The Adoption of IT by SMES: The Dutch Case”, Journal of Enterprising Culture, Vol. 9, No. 1, pp. 129-152.

Temtime, Z.T., Chinyoka, S.V., Shunda, J.P.W. (2003) "Toward Strategic Use of IT in SMEs: A Developing Country Perspective", Information Management & Computer Security Vol. 11, No. 5, pp. 230-237.

UN (2008) “E-Government Survey 2008, From e-Government to Connected Governance”, Doc # ST/ESA/PAD/SER.E/112, United Nations, New York


http://www.gulfbase.com/site/interface/NewsArchiveDetails.aspx?n=49939

http://www.netimpactstudy.com/ca/

http://www.theinnovestgroup.com/43.html

http://www.collecter.org/archives/2006_December/07.pdf

http://www.kom.om/ev.htm

http://www.oeronline.com/php/2007_dec/sme_forum.php

http://ijedict.dec.uwi.edu/viewarticle.php?id=349



UNCTAD (2004) “ICT as an Enabler for Growth, Development and Competitiveness”, Plenary session, June 17, [online], http://www.unctadxi.org/templates/Event____69.aspx [13 March 2008]

WestFocus (2007) “Abandon Heroes –ICT Adoption and Use in SMEs” [online] http://www.westfocus.org.uk/SiteManager/UserFiles/downloads/14_Abandoned_Heroes_Project_Report_final.doc [5 June 2008]

Wymer, S.A. and Regan, E.A. (2005) “Factors influencing e-commerce adoption and use by small and medium businesses” Electronic Markets, Vol. 15, No.4, pp. 438-453.

Yeh, Quey-Chen, and Chang, A, J-T (2007) "Proliferation of Computers among Taiwanese SMEs", International Journal of Entrepreneurship and Innovation Management, Vol. 7, No. 1, pp 51-73.

http://www.unctadxi.org/templates/Event____69.aspx

http://www.westfocus.org.uk/SiteManager/UserFiles/downloads/14_Abandoned_Heroes_Project_Report_final.doc

ISSN 1566-6379 139 ©Academic Conferences Ltd Reference this paper as: Barclay, C. and Osei-Bryson, K-M “The Project Objectives Measurement Model (POMM): an Alternative View to Information Systems Project Measurement.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 139 - 154, available online at www.ejise.com

The Project Objectives Measurement Model (POMM): an Alternative View to Information Systems Project Measurement

Corlane Barclay 1 and Kweku-Muata Osei-Bryson 2 1University of the West Indies, Jamaica 2Virginia Commonwealth University, USA [email protected] [email protected] Abstract: The information systems (IS) project management profession has been faced with numerous socio-technical challenges. As part of its analysis, research has expressed discontent with the traditional measures used to assess the success or failure of these projects, i.e. conformance to time, schedule and specification requirements espoused by the project management (PM) standard bodies. Despite this, research has also revealed that industry continues to place high reliance on this approach in determining the outcome of their projects. These developments imply, in part, a misalignment between research and practice and a scarcity of appropriate measurement tools that are aligned to the realities of different project contexts. The research presents a Project Objectives Measurement Model (POMM) that attempts to address some of these concerns through the development of project measures that are aligned to key project stakeholders’ values and objectives within the unique project contexts. It is argued that objectives are the key performance criteria of the project hence measures must be aligned to these criteria and formal procedures should be in place to assure that these objectives and measures are carefully developed and reflective of the persons to which the project matters, the stakeholders. The POMM is grounded on several principles of the Value Focused Thinking (VFT) and Goal Question Metric (GQM) techniques. The evaluation of the proposed model was performed in two parts: a team of industry experts examined the principles of model and provided feedback on its practicability to practice, and a case study of a Caribbean educational institution’s IS graduate programme development was used to illustrate the procedures of the model. The research provides theoretical and practical implications for IS evaluation particularly within the project management and performance measurement domains. The research aims to extend the debate on suitable evaluation methods for IS projects while providing project practitioners with an alternative approach that can enhance their decision making processes during the life of the project. Keywords : IS project; project objectives measurement model (POMM); success criteria; IS project management; Caribbean

1. Introduction

Information systems (IS) projects play an important, often strategic role in contemporary organisations. Given this prominence, executives and other project stakeholders have a vested interest in assessing the real value that these projects provide (Brynjolfsson, 1993, Melville et al., 2004). It is therefore important to find appropriate methods to evaluate them, including the value that it provides to relevant stakeholders. Current analysis indicates several challenges with regards to current practices. There is growing disenchantment with the traditional methodology (i.e. conformance to time, cost and scope requirements) which has been described as an incomplete measurement paradigm (Atkinson, 1999, Atkinson et al., 2006, Wateridge, 1998). Despite its perceived limitation this approach continues to be pervasive in practice (White and Fortune, 2002, KPMG, 2005). A compounding concern is the varied perception of performance, as success may mean different things to different stakeholder which may lead to disagreement on whether a project is successful or not (Shenhar et al., 2001, Shenhar et al., 2002). Against this background, we argue that an effective evaluation approach for IS projects should consider active alignment of stakeholders’ value to the measures. The research explores an alternative measurement system for assessing IS project performance that extends the traditional approach. We therefore propose the Project Objectives Measurement Model (POMM) which adapts the principles of the Value Focused Thinking (VFT) method (Keeney, 1992) to elicit and structure stakeholders’ project values and objectives, and the Goal Question Metric (GQM) method (Basili and Weiss, 1984) to derive measures that are linked to the identified project values and objectives. The POMM can be used to provide guidance to practitioners in the design of project metrics/measures that are closely aligned to the realities of their IS projects in a manner that transcends the restrictive frames of traditional approaches. Further it aims to enable project stakeholders to be better equipped to see the project is, identify missing links or inconsistencies in project design, and take appropriate corrective actions where necessary, thus increasing the likelihood of achievement of the objectives or project performance criteria.




With continued calls to extend the conceptual foundations of project management (PM) through the analysis of similar events in other disciplines and increase the alignment with practice (Engwall, 2003, Winter et al., 2006b, Smithson and Hirschheim, 1998) the study seeks to extend the knowledge base in the discipline. The research uses empirical investigation to help demonstrate our claims. First, a preliminary evaluation of POMM is performed by a team of IS PM experts and stakeholders. Second, a case study illustration of POMM using an IS graduate programme project is presented. Findings indicate that POMM has credence in practice and can be suitable in assisting practitioners in strengthening their evaluation process.

2. Research Background

2.1 Trends in evaluating projects

An appreciation of the full context within which the project is being performed will assist those involved in project management to deliver a project (APM, 2006), as these activities continue to play key role in organisational change. Projects are unique, transient endeavors undertaken to achieve a desired outcome (APM, 2006) including a unique product, service or results (PMI, 2004) and refer to single or multiple endeavours. Assessing outcome of these projects is of extreme importance to the key stakeholders involved (Cleland, 1986). However, with the continued problems with assessing the value of IS (Irani, 2002), the true contribution of IS projects are often not realized or identified. Thus, it is essential to find more insightful analysis of project performance or outcome to assess where we are and whether we need corrective action to get back on track. A review of key project evaluation literature revealed that the primary responses to examining the performance of projects have been to develop alternative success criteria to assess these projects (Atkinson, 1999, Morris and Hough, 1987, Nelson, 2005), and critical success factors (Fortune and White, 2006, Belassi and Tukel, 1996, Pinto and Slevin, 1987, Shenhar et al., 2001, Shenhar et al., 2002) while others have focused on the business value contribution of these investments (Kaplan and Norton, 1992, Kumar, 2003, Fitzgerald, 1998). In an analysis of the literature over the last forty (40) years, Jugdev & Muller (2005) showed the evolution of our understanding within the framework of the project and product life cycles in the determination of outcome perspectives. The four (4) evolving research themes were categorized into project implementation and handover, critical success factor (CSF) lists, CSF frameworks and strategic project management paradigms (Jugdev and Muller, 2005).

Table 1: Summary of success factors & criteria

Key Categorization

Outcome Perspectives Literature

Project Management & Project Team

Minimization of project cost and project duration; strong project commitment; communication; monitoring and feedback; personnel & competence; planning,; clear project objectives; conformance to budget, time, scope requirements; project functionality, project efficiency,

(Atkinson 1999); (Bryde et al. 2005); (Freeman et al. 1992) (Morris et al. 1987); (Pinto et al. 1987); (Nelson 2005), (Shenhar et al. 2001; Shenhar et al. 1997; Shenhar et al. 2002)

Management & Executive

Top management involvement, politics (Pinto et al. 1987); (Morris et al. 1987)

Client and Other Stakeholders

Satisfaction; endorsement; acceptance; user involvement; utility; use; safety; impact on customer; customer service; increased responsiveness

Belassi et al. (1996); Bryde et al. (2005); Kumar (2005) ; Pinto et al. (1987); (Standish 1994); Lim et al. (1999), Nelson (2005), Morris et al. (1987) ;

Project product or service

New product or market; safety; commercial performance; technical performance business and direct success; financial rewards; implementability; flexibility

Dvir et al. (1992); Lim et al. (1999) ; Fitzgerald (1998) ; Freeman et al. (1992)

Preparation for the Future

Value, personal growth, learning, readiness for the future

Bryde (2005), Nelson (2005) ; Freeman et al. (1992) ; Kaplan & Norton (1992)

2.2 Managing stakeholders’ expectations

Success may mean different things to different stakeholders (Shenhar et al., 2001, Shenhar and Levy, 1997, Belassi and Tukel, 1996) which may lead to difficulties in managing these perspectives (Agarwal and Rathod, 2006) and determining the success of the project outcome. This has interesting implications, particularly the opportunity to synthesize the diverse views and transform these into suitable measurement frameworks, hence, facilitating richer analysis rather than simple listing of objectives.


Corlane Barclay and Kweku-Muata Osei-Bryson

The stakeholders group is diverse, and may include individuals that are internal and external to the project and organisation such as contractor, sponsor, project team and client (PMI, 2004). They are typically most involved in the project and have a vested interest in its outcome or contribution (PMI, 2004). These stakeholders may have certain expectations and consequently engage in behavior that may constructive or destructive (Bourne and Walker, 2006). Importantly, Courtney (2001) noted that open, honest, effective dialogue among stakeholders is a critical aspect in the development of multiple perspectives, which implies that reaching effective measurement tool is made easier open honest and effective dialogue about stakeholders’ viewpoint.

2.3 Rethinking the profession

An interdisciplinary colloquium in the UK met to discuss and analyse initiatives that may enrich and extend the current project methodologies to better meet these and other contemporary challenges (Winter et al., 2006b) as it recognized that bridging the process of academic and practitioner perspectives is still imperative (Winter et al., 2006a). The need for new and better ways to think about projects and their management (Crawford et al., 2006) (Cicmil et al. 2006) revealed that current views do not adequately explain the richness of what actually happens in contemporary project environments This complements the view that current practice does not sufficiently address sources of uncertainty within the project and thus more sophisticated efforts are needed (Atkinson et al., 2006). The participants therefore called for increased focus on concepts and theories that resonates with the realities of practice thereby providing practitioners with practical concepts and improve alignment with contemporary thinking (Winter et al., 2006b) and recommended that PM research embraces theory about practice, theory for practice and theory in practice (Winter et al., 2006b).

3. Towards a new methodology for practice

An objective is defined as a lens or system of lenses that forms an image of an object (Merriam Webster Dictionary, 2008). In this case, the stakeholders’ lens forms an image of the project that results in the perception of its performance. Hence, project objectives therefore can be considered the stakeholders’ criteria through which performance outcome perceptions are based. Assessing the outcome of the projects based on these factors appears to better match the realities of practice. It is therefore critical that the objectives are robustly defined and reflects the views of the key participants.

3.1 Value Focused Thinking (VFT) approach

Understanding what is important to key stakeholders can enhance the decision-making process in the project. The VFT is a decision technique developed by Keeney (1992), it identifies the values of the stakeholders in a given activity. The technique has been applied successfully in other research, and is a proven methodology that has been used in various disciplines to aid in the analysis of strategic decisions (Sheng et al., 2005). The VFT is useful in uncovering hidden strategic objectives of diverse managerial processes, and has been used successfully in the operations management disciple to help chart a clear decisive path for the fundamental objectives and values of the activities thereby providing an unobstructed view to carefully assessing performance of organisational activities (Keeney, 1992, Keeney, 1996). Keeney (1996) suggested that the VFT enriches the decision making process. Further, while it is important to identify objectives, simply listing objectives is shallow as there is need for greater depth, clear structure, and a sound conceptual base in developing objectives for strategic decision contexts (Keeney, 1996).

Figure 1: Steps in the VFT



The application of the VFT technique includes the following steps (Keeney, 1992, Keeney, 1996):

Develop list of objectives. This involves identifying the values and converting values to objectives. Values are those principles that encompass what a person care about or value in a specific situation. An objective is characterized by three features: a decision context, an object and a direction of preference. This essentially means explicating the objective within its context based on the nature of the problem and determining exactly what the stakeholder is ultimately trying to achieve. It is also possible to derive more than one objective from a specific value statement (Drevin et al., 2007).

Structure objectives. Objectives are classified into two types: (1) fundamental objectives - the end that the decision-maker values in a specific decision context, and (2) mean objectives – the methods to achieve the ends which are context dependent. This implies that consideration of the particular nature or purpose will determine how the fundamental and mean objectives are formed.

Develop means-ends network. To perform this step, asking why each objective is important will help to distinguish between fundamental and means objectives which can be represented by the means-end objective network. Aided with this tool, decision makers are better able to see the relationship among the project objectives and can make more informed decisions in the project

3.2 GQM approach

Subsequent to the identification of core objectives of the project, the identification of suitable quantifiable metrics aligned to the objectives is essential. The GQM approach is a metric generation technique that develops performance metrics aligned with the goals (objectives) of the activities (Basili and Weiss, 1984). It utilizes a top-down method for the identification of metrics needed for certain goals by asking questions linked to these goals. It is a practical approach to bounding the measurement problem as it allows organisations to focus on its own context and culture. The method asks, how do you decide what you need to measure in order to achieve your goals (Basili and Weiss, 1984)? Applying these principles into the IS project evaluation context strengthens the generation of stronger metrics and measures associated with the project objectives. The scope of the GQM application in this research is primarily to identify the metrics subsequent to the application of the VFT to identify project objectives. These steps include (Basili and Weiss, 1984, Solingen and Berghout, 1999):

1. Formalize measurement goals

2. Identify quantifiable questions 3. Define the measures to be used

4. Prepare plan for implementing and interpreting the measures

4. The project objectives measurement model (POMM)

The Project Objectives Measurement Model (POMM) involves the elicitation of objectives and measures that reflect the strategic and tactical vision of the project from the perspectives of its multiple stakeholders. Three key questions are reflected throughout the framework: 1. Do the project measures reflect the fundamental objectives identified?

2. Do the project objectives reflect the project contexts? 3. Does the evaluation process reflect the realities of the project?



Figure 2: The POMM conceptual framework

POMM involves the following several steps:

1. Identify key stakeholders of the project, taking into consideration the roles involved, the organisations or personnel that may be impacted by the project and its results

2. Elicit project values and objectives for each key stakeholder, including standard objectives relating to time, cost and scope

3. Apply VFT method to determine the fundamental and means project objectives

4. Prioritize project stakeholder fundamental objectives

5. Develop, review and refine (where necessary) the project means-end network 6. Apply GQM method to elicit project measures

7. Develop, review and refine (where necessary) the project objective-measure network

8. Implement, monitor and take corrective actions throughout the project 9. Determine the cumulative outcome of the project

The Objective-Measure Network (Figure 3) is an output of these steps. It is an extension of the Means-End Network (Keeney, 1992) and represents the relationship between the means objectives, fundamental objectives and project measure of the project, highlighting the dependence of objectives on each other and the measures that are associated with them. This provides an additional aid to stakeholders and project managers in particular to analyse achievement of objectives, through-out the project cycle. Depending, on the project context the hierarchy of objectives may change (i.e. a fundamental objective may be a means objectives or vice-versa in another context). For example, cost containment objective may be a fundamental for a cost reduction project yet it may be only a means to another objective in another setting. This further highlights the benefit of this approach through its focus on understanding the rationale or importance of the objective within each decision context, which are solely guided by a restrictive set of objectives.



Figure 3: An illustration of the objective-measure network

5. Towards tighter coupling with practice

Whetten (1985) provided researchers with some important considerations in the theory development process. He explained that in the theory development process it is important to clearly state the “what”, “why” and “how” of the research contribution. The “what” includes the concepts that are important in the research, “how” describes the casual relationships between the concepts or constructs, while “why” explains the justification for the research. Essential in the concept development is the need for comprehensiveness (are all relevant factors included) and parsimony (are there factors that add little value to our understanding) in the explication of theoretical models. The when, where and who are also important as it these provide the temporal and contextual factors, the boundaries of generalisability or limitations of the research and the intended audience of the research. He further explained that this facilitates better understanding on what is going on through an appreciation of where and when it is happening. Such an approach for theory development provides tighter coupling of knowledge claims and research solutions. Against this background, eight (8) experts and project stakeholders in the PM and IS fields were asked to assess the POMM in order to establish its suitability, usefulness and practicability for practice, including comparing with the current standard practice of measuring IS projects, which these are sometimes poorly defined. This exploratory investigation is aimed at achieving tighter coupling with practice through an interactive design cycle in which practitioners’ feedback are used to help refine the proposed model. This approach sets the stage for an explication of the principles of POMM through the case study illustration.

5.1 A preliminary evaluation of POMM

Two (2) rounds of interviews with IS PM experts were conducted to assess the practicability, usefulness and limitations of the POMM for practice. The interviews also included discussions based on issues and responses raised throughout the sessions. The first round was used to gather the perspectives of the practitioners, and the second interview round was used to clarify points and gather more supporting evidence for the arguments provided by the participants. The stakeholders played several different roles within their organisations (Table 2) and had shared experiences in managing and assessing IS projects. More importantly, these individuals have an important stake in finding alternative methods outside their current practice of adopting the standard methodology. This factor was the key motivation for their selection in this empirical evaluation combined with their availability to critically discuss the issues. They were asked to critique the model and offer their perspectives and interpretation through a questionnaire developed based on the objectives of our investigation and underlined by the principles highlighted by Whetten (1985).



Table 2: Stakeholders demographic data

Role (# of individuals) Years of Experience

Industry

Project Consultant (2) 10-20+ Information Systems, Software Development, Telecommunication

Project/Programme Manager (4)

5 – 15+ Financial Services, Telecommunications

Executive Management (2) 15+ Banking, Information Services

5.1.1 Findings & discussion

The interviews took a structured path; the model was distributed to them initially, during the meetings it was explained and supported with visual aids. Subsequently, they were allowed to asked questions to clarify their understanding of the model and its objectives. Upon confirmation that their understanding was consistent with the model’s intent they were asked to critically assess it.

5.2 Expected benefits

The model “takes into consideration a wider set of objectives through the attention to the breadth of key stakeholders” responded an executive manager. Another suggested that its use may provide some insights into IS projects that have failed by the traditional approach yet successful upon implementation and use, or vice-versa. The interviewee went on to claim that “few [of their] projects completely fail even they deliver some value to the organisation”. This underscores the need to have a method to help identify the values important to these stakeholders and the organisation as a whole, and provide a process through which these can be translated into quantifiable measures. Some of the project managers expressed that the POMM’s philosophy was aligned to their current thinking as their experiences have shown that a project outcome may be successful in terms of increased revenue or market share, despite it being over budget or late for example. In addition, the model provides beneficial results through the improved identification of project objectives, strengthening of the project design process and the evaluation mechanisms and process. The case was also made for improved decision-making through the analysis of factors that may not have been previously considered or identified. An example was cited of a project that has been implemented but has had severe environmental challenges; it was proposed that with the use of the POMM approach “some of these issues may have been brought to the fore, and additional stakeholders may have been taken into consideration”. Improved stakeholder management was also highlighted as this could be enhanced through the open communication and the perception that their perspectives are deemed important in the elicitation of key project objectives. “This [approach] could be use when we are developing our business case”. An executive noted that with such an approach the project justification may become clearer and thus created a better opportunity for stakeholders buy-in while possibly identifying possible trade-offs and help mitigate some of the potential project risks. This is an important value proposition as it highlights the value of the model’s flexibility in enhancing project justification, design and the evaluation of the project itself. The detailed investigation of the project stakeholder objectives through the POMM approach allows improved analysis at the onset of the project selection process which can help eliminate or reduce unknowns.

5.3 Factors impacting successful adoption

Some of the essential factors that may influence the model’s successful adoption in practice were similar to the discussed on the extant literature on critical success factors (e.g. Pinto and Slevin, 1987, Morris and Hough, 1987). The participants highlighted executive support and buy-in and stakeholder commitment to the process as important consideration in the model’s success in their respective organisations. Other key factors suggested were the level of awareness of the techniques of the POMM and conformance to its standards by stakeholders. Hence it is anticipated that the learning curve may be steep during early adoption. A project manager suggested that organisational buy-in may be difficult as project managers and organisations may find it difficult to move from the standards e.g. PMI Body of Knowledge (PMI, 2004) “It is difficult to walk away from the [project management] bible”. This was reinforced by another expert who expressed reluctance in adopting it in their project because of the continued use of the established practice of conformance to time, cost of specifications. “I am more comfortable with a method that is tried and proven”. Notably, this was despite agreement of its practical benefits. This finding reinforces the prominence



of standard bodies’ methodology and its strong influence on practice (Morris et al., 2006). Additionally, this phenomenon may be further explained by several technology adoption theories, particularly intuitionalism (c.f. DiMaggio and Powell, 1983). Some of the participants saw stakeholders as a restrictive set, primarily focusing on the most influential (i.e. executive management) without much initial considerations for wider project context such as the clients, external consultants and other third parties. We propose that more formal approaches to identify project stakeholders are needed for the critical process (Sharp et al., 1999). The main concerns raised were the issue of prioritization of objectives and the model’s adaptability to small organisations. The interviewees noted despite best efforts some objectives may be more critical than others and it is essential not to be swayed solely by the most influential project stakeholder. Keeney (1992) stated that one the principal benefits of the VFT technique was avoiding conflicting decisions as the rationale for the objectives become known. This point becomes more apparent in the illustration of the POMM, because in asking the question of values and why each are important helps to classify (and prioritize) the objectives into means and fundamental objectives wherein all the fundamental objectives will be top priority. Alternatively, multi-criterion decision techniques such as the Analytic Hierarchy Process (AHP) is useful in prioritizing diverse perspectives within a particular context (Saaty, 1990) since it can help resolve complex problems involving high stakes and diverse perceptions (Bhushan and Rai, 2004). The issue of adaptability was raised primarily because of the assumption that smaller organisations may not have the requisite resources or skill-sets to use the model effectively. This is indeed an important consideration; however a key consideration of the POMM is its applicability to different contexts, organisations or project sizes. The model is accompanied by simple steps that may used by any individual or organisation. As noted earlier a challenge may be the initial learning curve, however the potential benefit of greater project clarity and justification may outweigh that. Additionally, where organisations notice a knowledge gap, this can be resolved through the hiring of qualified consultants or investing in training programmes for staff.

6. A practical illustration of POMM

A single case study was used to illustrate the concepts and contributions of POMM. The case study approach is recommended for the in-dept study of an object or model in the business environment (Hevner et al., 2004). The case study approach embodies an account of past or current phenomenon drawn from multiple sources of evidence (Leonard-Barton, 1990, Yin, 2003) and involves an empirical inquiry that investigates a contemporary phenomenon within its real-life context; when the boundaries between phenomenon and context are not clearly evident; (Yin, 2003). Several researchers advocate this methodology for investigating real-life events, including organisational and managerial processes (Orlikowski, 2000, Darke et al., 1998, Yin, 2003) and allows the researcher to retain the meaningful and holistic characteristics of real-life events (Yin, 2003). Additionally, the case study method is applicable because of its suitability in providing enhanced understanding to organisational contexts and IT-related innovations (Darke et al., 1998). The illustration of the POMM makes the case for its usefulness in providing an alternative evaluation technique for IS project outcome. The study considers a IS graduate programme development project at a Caribbean university, which includes activities such as the programme design and students’ thesis development process and review of the thesis outcome. This activity was chosen for several reasons: 1) it clearly makes the case that the traditional method of assessing project outcome is insufficient because the success of the graduate programme involves more that conformance to time, cost and specifications as determined by the University; and 2) there is a great possibility that each set of stakeholders have their unique set of objectives combined with common set of the programme’s project objectives. Therefore, the steps of POMM may be clearly seen through this view. Identify Key Stakeholders. The thesis development process involves numerous stakeholders. The task involved considering the roles and actors that may be affected by the programme’s evolvement. This process was achieved through documents’ review and elicitation through discussions. Documents relating to the programme were reviewed to identify key roles that are involved in the programme, and specific stakeholders were asked to consider the process and the history of the programme to help identify groups or individuals that played a part in its development and management. The stakeholder entities were later grouped into personal, school community, academic and business communities were identified. The principal stakeholders therefore included advisors, administrators, internal and external examiners, executive sponsor, corporate sponsors or organisations. The stakeholder groups interviewed were students, advisor,



academic director, internal examiner, executive sponsor and an organisational representative. This was done to obtain a fair representation of the stakeholder groups and their availability in being interviewed. Elicit Project Values and Objectives . The first stage involved interacting with stakeholders in an open forum to garner the essential elements that were important to them in the dissertation programme. Using the guideline set out by Keeney (1996), the participants were asked questions such as what do you value or want from the dissertation project, and what are your ultimate objectives of the dissertation project. Upon completion of all interviews, the notes were reviewed and a complete list of all the objectives identified was prepared. The product of this was a list of 46 project objectives. The second stage involved reviewing the objectives and converting them into common or consistent form as several of the objectives represented the same thing but were stated differently. This resulted in 26 objectives (see figure 3) which laid the foundation for the next step in the process. Determine the Relationship between Objectives . Each participant was asked why each of their identified objectives was important. During this exercise it was discovered that this approach may result in additional objectives as the participants get an opportunity to elucidate more on what they value. For example, in discussing the importance of a high quality dissertation development process, the advisor noted that ignorance of stakeholders’ (e.g. student, committee role) may lead to conflicts and they did not want any occurrence of unfairness or lack of objectivity to be experienced. Therefore along with an objective for clarity among stakeholders’ role, the issue of objectivity in the review process was also highlighted. Also, as expected the exercise resulted in identifying the relationships among the objectives. For example, the student identified the need to improve research skills or enhance their understanding of research methodologies, however in asking why these were important in addition to completing the dissertation, the value of growth in the IS community or to produce relevant research were identified. Nine (9) fundamental objectives were identified and the list of all the objectives and relationships were converted to the project means-end network (see figure 4). Determine the Priorities of the Project Fundamental Objectives. A simple rating method was used based on the decision context. The stakeholders were asked to rate the importance of the fundamental project objectives in determining a successful IS PhD programme. The level of importance ranged from 1-9, based on the number of project fundamental objectives identified, and was defined as follows: High = 7-9, Medium = 4-6, and Low = 1-3. Each stakeholder was asked to explain their decisions in determining the ratings and an average score was produced as the final perspectives on priority of the objectives. Interestingly, none of the objectives received a rating below five which implies that the stakeholders valued each of the fundamental objectives and likert-like scale with closer range could be used. The higher priority objectives included: exhibit intellectual independence, competence in the research process IS content knowledge and a formal process for IS PhD programme development, see fig. 3. Table 3: Fundamental project objectives & priorities

List of Project Fundamental Objectives Average

Rating Weight

Exhibit intellectual independence 9.00 0.124

Develop competence in research process 9.00 0.124

Possess doctoral-level IS content knowledge 8.67 0.119

Develop/enhance a formalised process of IS dissertation management (inclusive of development, review, etc)

8.33 0.115

Conform to ethical standards of the profession 8.33 0.115

Maximize the number of completed dissertation 7.67 0.106

Create a legacy of research 7.33 0.101

Dissertation effort must provide practical and relevant research to the Caribbean 7.33 0.101

Maximize opportunity to become established in the research community 7.00 0.096



Review Means-End Network . A review of the network was performed to ensure completeness of the objectives and accuracy and completeness of the relationships. The main questions asked during this process were: did it make sense and did it accurately reflect the views of the stakeholders. Minor refinements were performed to address missing links and consistencies among the project objectives. Elicit Project Measures . The participants are asked to explain how they would measure the achievement of the stated objectives, those they provided initially and the full composite of project objectives. The GQM approach was used to analyse the data, elicit and identify measures associated with the project objectives. There are two strategies that may be employed in obtaining the measures by identifying the measures linked to the project means objectives or linked to the project fundamental objectives. This examination analysed all the project objectives to identify suitable measures. In eliciting the measures, the purpose of the objective is first identified, followed by the its issue, object or aim of the objective, and viewpoint through which the objective is seen (Basili et al., 1994). The set of questions are then developed from which the measures of viewing the objectives derived and the process was repeated for all the objectives. For example, the objective of Possess Doctoral-Level IS Content Knowledge is analysed by identifying purpose of the objective (i.e. attain), the issue (i.e. possession), the object of the objectives (i.e. doctoral-level IS content knowledge), and the viewpoints through which the objectives may be viewed (i.e. student, advisor). Questions include (1) What are the characteristics that indicate or reflect doctoral-level IS content knowledge? and (2) Is formal assessments of students’ content knowledge performed? Therefore metrics include (1) Score on comprehensive exam, (2) GPA on dissertation course work, and (3) Level of comprehensiveness of literature review section. The approach allows for objective and subjective ratings of the objectives (Basili et al., 1994), so an objective measure would be the exam results and a subjective measure of an advisor would be the rating on the level of comprehensiveness of the dissertation or sections of it. It was observed that the exercise of identifying metrics was also made easier with the use of the VFT technique. It became evident that the VFT provided a clearer vision of what the stakeholders needed (richer objectives), and it was observed that the act of developing questions and finding associated metrics was made easier with the aid of the project means-end network. The project measures are later summarized in the Fundamental Objectives-Measures network, a product of the POMM displayed in Table 4. The reader may observe that each objective has multiple measures, and that measures appear to have different scales. Through the use of appropriate transformation rules, scores for all measures could be converted to a uniform scale (e.g. High = 7-9, Medium = 4-6, and Low = 1-3). For a given fundamental objective, the scores of its associated measures on the uniform scale would be synthesized using some method (e.g. averaging) resulting on an overall score for the fundamental objective.



Figure 4: Project means-ends network - IS Thesis Programme Project



Table 4: Project fundamental objectives-measures network

Project Fundamental Objectives Measures Develop competence in research process

Subject course grades

Rating on dissertation research development

Possession of doctoral-level IS content knowledge

Course work grades

Comprehensive exam results

Rating on completed dissertation

Rating on Literature Review section of dissertation

Possess intellectual independence

# of independent research activities

# of independent publications

Ratio of independent activities/publications to collaborative activities/publications

Rating on level of intellectual growth during programme

Develop/enhance a formalized dissertation mgmt process

Detailed procedures on review and assessment process

Detailed procedures on accountabilities, structure, how the programme works

Results of benchmark against other universities

Results of independent audit of programme

Conform to ethical standards of the profession

Rating of the standard of the completed dissertation

Rating of conformance to standards set-out by IS profession

Maximize dissertation completion

# of completed dissertation

Ratio of completed dissertation to enrolled students

Average latency of completion

Provide strong legacy of research

# of graduates

Rating of cadre of students/graduates

Creation of facilities that promote Caribbean research/IS research (e.g. a conference)

Produce practical & relevant research

# of publications

# of practitioner-based publications

# of research adopted in practice

Rating on the level of interest shown by practice (e.g. business community)

Establish self in research community # of publications

# of citations of published works

# of consultations and/or requests

# of invitations (reviews, research community service)

Review Objective-Measure Network . The participants were later asked to review the means-ends network to determine whether 1) it represented what they wished to convey 2) there were any missing relationships, objectives or measures, and 3) it was an understandable tool that would be useful to them in generating richer project objectives. The stakeholders conveyed that the model and approach was useful as it highlights the objectives, relationships and measures that may be used to assess the achievement of same. Monitor Project Objectives . The performance of the project is monitored against the achievement of the objectives based on the measures identified. Equipped with the set of objectives and measures, the university (e.g. programme coordinator) may assess the progress or achievement of the respective objectives and determine whether any refinement or corrective actions are needed throughout the remainder of the project. For instance, in monitoring intellectual independence of students, the number of independent publications would be evaluated or ratio of independent and collaborative activities. These may be further compared against agreed baselines that are included in the programme’s policies and procedures (objective – formalization of programme). If performance is below expected standards, corrective measures such as research workshops may be conducted and independent research activities reevaluated. Evidently, as seen by the diverse objectives some may be achieved within the short term and others over a longer term, for example, course grades and procedures can be evaluated during the life of the programme, the number of graduates at the first cycle and number of publications per student over a longer period. This is aligned with



trends of including the product life cycle assessments as part of the evaluation of outcome of the project (Nelson, 2005). Determine the Project Outcome . In the determination of the project objectives and measures practitioners now need a strategy to define the final outcome of the project in terms of success or failure or anything in between. Such an approach includes an assessment of the achievement of all the fundamental objectives and the use of a matrix as defined in Table 5 to define the achievement of an outcome. Table 5 includes a set of decision rules that could be used to assess the overall project outcome. For example, achievement of all fundamental objectives equates to a highly successful project, achievement of some of the fundamental objectives combined with all the other equates to a partially successful project outcome. Achievement of none of the fundamental objectives would equate to failure. All of the possible combinations are not shown, but rather an illustration of the decision rules that may guide practitioners in the determination of a clear project outcome. Table 5: An example of project outcome determination

High priority fundamental objectives

Medium priority fundamental objectives

Low priority fundamental objectives

Project Outcome Results

All All All Highly successful All Most All Highly successful All Most Most Highly successful All Some All Highly successful Most All All Mostly successful Most Most All Mostly successful Most Most Most Mostly successful Most Some Some Mostly successful Most None None Partially successful Some All All Partially successful Some Most Most Partially successful Some Some All Challenged Some Some Some Challenged None All All Challenged None None All Failed *** All other Combinations *** Challenged or Failed

Alternately the overall assessment of the project could be determined by using the weighting model as determined in the Determine the Priorities of the Project Fundamental Objectives step (see Table 3) and scores for the fundamental objectives as were determined in the Elicit Project Measures step. Other options include multi-criteria decision analysis (MCDA) methods such as AHP or Electre, which are useful strategies for assisting in the determination of the final outcome of the project. These methods have been cited for being most reliable or user-friendly based on a review of the literature. Hence, there is improved likelihood they can be easily adopted in this environment.

7. Concluding remarks

The POMM is proposed as a tool that can aid in the evaluation process of IS projects through the elicitation, development and alignment of project objectives and measures reflective of the values of the key project stakeholders. An exploratory assessment of the strengths and implications of the proposed model by a team of IS PM experts was performed. They concluded that the model can provide value to organisations through the identification of stronger set of project objectives, or even at the project selection/justification phase through improved project design, and is an improvement of the current standard methodology practiced in their organisations. The case study method was used to illustrate the procedures and concepts of POMM using an IS thesis programme. While this case may not have fallen into the typical business domain, we proffer that IS plays a role in every sector and the model can be applied within any domain seeking to evaluate their IS projects. This research represents the first in a series of studies to extend the debate on perspectives of IS project evaluation tools. It is anticipated that this study will help project practitioner rethink their evaluation methods, and focus on a more complete approach that engenders stronger project design and improved accounting of project delivery. We argue that this will aid stakeholder satisfaction. Future research directions involve enhancing the generalisability of the study through further examination and refinement of the POMM and its application in diverse IS project case settings.



References Agarwal, N. and Rathod, U. (2006) Defining 'success' for software projects: An exploratory revelation, International

Journal of Project Management, Vol. 24, No. 4, pp 358-370. APM (2006) The APM body of knowledge, The Association of Project Management Atkinson, R. (1999) Project management: Cost, time and quality, two best guesses and a phenomenon, its time to accept

other success criteria, International Journal of Project Management, Vol. 17, No. 6, pp 337-342. Atkinson, R., Crawford, L. and Ward, S. (2006) Fundamental uncertainties in projects and the scope of project

management, International Journal of Project Management, Vol. 24, pp 687–698. Basili, V. R., Caldiera, G. and Rombach, H. D. (1994) The goal question metric approach, Encyclopedia of Software

Engineering, Basili, V. R. and Weiss, D. M. (1984) A methodology for collecting valid software engineering data, IEEE Transactions on

Software Engineering, Vol. 10, No. 6, pp 728-738. Belassi, W. and Tukel, O. I. (1996) A new framework for determining critical success/failure factors in projects,

International Journal of Project Management, Vol. 14, No. 3, pp 141-151. Bhushan, N. and Rai, K. (2004) Strategic decision making: Applying the analytic hierarchy process, Springer-Verlag,

London Bourne, L. and Walker, D. H. T. (2006) Using a visualising tool to study stakeholder influence - two Australian examples,

Project Management Journal, Vol. 37, No. 1, pp 5-21. Brynjolfsson, E. (1993) The productivity paradox of information technology, Communications of the ACM, Vol. 36, No. 12,

pp 67-77. Cicmil, S., Williams, T., Thomas, J. and Hodgson, D. (2006) Rethinking project management: Researching the actuality

of projects, International Journal of Project Management, Vol. 24, pp 675–686. Cleland, D. I. (1986) Project stakeholder management, Project Management Journal, Vol. 17, No. 4, pp 36–44. Courtney, J. F. (2001) Decision making and knowledge management in inquiring organizations: Toward a new decision-

making paradigm for DSS, Decision Support Systems, Vol. 31, pp 17–38. Crawford, L., Morris, P., Thomas, J. and Winter, M. (2006) Practitioner development: From trained technicians to

reflective practitioners, International Journal of Project Management, Vol. 24, No. pp 722–733. Darke, P., Shanks, G. and Broadbent, M. (1998) Successfully completing case study research: Combining rigor,

relevance and pragmatism, Information Systems Journal, Vol. 8, No. pp 273-289. DiMaggio, P. J. and Powell, W. W. (1983) The iron cage revisited: Institutional isomorphism and collective rationality in

organizational fields, American Sociological Review, Vol. 48, No. 2, pp 147-160. Drevin, L., Kruger, H. A. and Steyn, T. (2007) Value-focused assessment of ICT security awareness in an academic

environment, Computers & Security, Vol. 26, No. 1, pp 36-43. Engwall, M. (2003) No project is an island: Linking projects to history and context, Research Policy, Vol. 32, No. pp 789-

808. Fitzgerald, G. (1998) Evaluating information systems project: A multidimensional approach, Journal of Information

Technology, Vol. 13, No. 1, pp 15-27. Fortune, J. and White, D. (2006) Framing of project critical success factors by a systems model, International Journal of

Project Management, Vol. 24, No. 1, pp 53-65. Hevner, A. R., March, S. T., Park, J. and Ram, S. (2004) Design science in information systems research, MIS Quarterly,

Vol. 28, No. 1, pp 75-105. Irani, Z. (2002) Information systems evaluation: Navigating through the problem domain, Information & Management,

Vol. 41, No. 1, pp 11-24. Jugdev, K. and Muller, R. (2005) A retrospective look at our evolving understanding of project success, Project

Management Journal, Vol. 36, No. 4, pp 19-31. Kaplan, R. S. and Norton, D. P. (1992) The balanced scorecard: Measures that drive performance, Harvard Business

Review, Vol. 70, No. 1, pp 71-79. Keeney, R. L. (1992) Value-focused thinking: A path to creative decision-making, Harvard University Press. pp. 432 Keeney, R. L. (1996) Value-focused thinking: Identifying decision opportunities and creating alternatives, European

Journal of Operational Research, Vol. 92, No. pp 537-549. KPMG (2005) Global IT Project Management Survey, KPMG Information Risk Management, pp. 1-35. Kumar, R. L. (2003) Understanding the value of information technology enabled responsiveness, Electronic Journal of

Information Systems Evaluation, Vol. 1, No. 1, pp. Leonard-Barton, D. (1990) A dual methodology for case studies. Synergistic use of longitudinal single site with replicated

multiple sites, Organisation Science, Vol. 1, No. 1, pp 248-266. Melville, N., Kraemer, K. and Gurbaxani, V. (2004) Review: Information technology and organisational performance: An

integrative model of it business value, MIS Quarterly, Vol. 28, No. 2, pp 283-322. Merriam Webster Dictionary (2008) [online] http://www.merriam-webster.com/ Morris, P. W. G., Crawford, L., Hodgson, D., Shepherd, M. M. and Thomas, J. (2006) Exploring the role of formal bodies

of knowledge in defining a profession – the case of project management, International Journal of Project Management, Vol. 24, No. pp 710–721.

Morris, P. W. G. and Hough, G. H. (1987) The anatomy of major projects: A study of the reality of project management, John Wiley & Sons. pp. 338

Nelson, R. R. (2005) Project retrospectives: Evaluating project success, failure and everything in between, MIS Quarterly Executive, Vol. 4, No. 3, pp 361-372.



Orlikowski, W. J. (2000) Using technology and constituting structures: A practice lens for studying technology in organisations, Organization Science, Vol. 11, No. 4, pp 404-428.

Pinto, J. K. and Slevin, D. P. (1987) Critical success factors in successful project implementation, IEEE Transactions Engineering Management, Vol. 34, No. 1, pp 22-27.

PMI (2004) A guide to project management body of knowledge, Project Management Institute Saaty, T. L. (1990) How to make a decision: The analytic hierarchy process, European journal of operational research,

Vol. 48, No. 1, pp 9-26. Sharp, H., Finkelstein, A. and Galal, G. (1999) Appeared in 10th International Workshop on Database & Expert Systems

Applications, Vol No. pp 5. Sheng, H., Nah, F. F.-H. and Siau, K. (2005) Strategic implications of mobile technology: A case study using value-

focused thinking, Journal of Strategic Information Systems, Vol. 14, No. pp 269–290. Shenhar, A. J., Dvir, D., Levy, O. and Maltz, A. C. (2001) Project success: A multi-dimensional strategic perspective,

Long Range Planning, Vol. 34, No. pp 699-725. Shenhar, A. J. and Levy, O. (1997) Mapping the dimensions of project success, Project Management Journal, Vol. 28,

No. 2, pp 5-13. Shenhar, A. J., Tisler, A., Dvir, D., Lipostevsky, S. and Lechler, T. (2002) Refining the search for project success factors:

A multivariate typographical approach, R & D Management, Vol. 32, No. 2, pp 111-126. Smithson, S. and Hirschheim, R. (1998) Analysing information systems evaluation: Another look at an old problem,

European Journal of Information Systems, Vol. 7, No. 3, pp 158-174. Solingen, R. v. and Berghout, E. (1999) The goal/question/metric method, a practical method for quality improvement of

software development, McGraw-Hill Wateridge, J. (1998) How can is/it projects be measured for success? International Journal of Project Management, Vol.

16, No. 1, pp 59-63. Whetten, D. A. (1985) What constitutes a theoretical contribution? Academy of Management Review, Vol. 14, No. 4, pp

490-495. White, D. and Fortune, J. (2002) Current practice in project management - an empirical study, International Journal of

Project Management, Vol. 20, pp 1-11. Winter, M., Smith, C., Cooke-Davies, T. and Cicmil, S. (2006a) The importance of ‘process’ in rethinking project

management: The story of a UK government-funded research network, International Journal of Project Management, Vol. 24, No. pp 650–662.

Winter, M., Smith, C., Morris, P. and Cicmil, S. (2006b) Directions for future research in project management: The main findings of a UK government-funded research network, International Journal of Project Management, Vol. 24, No. pp 638–649.

Yin, R. K. (2003) Case study research: Designs and methods, Sage Publications


ISSN 1566-6379 155 ©Academic Conferences Ltd Reference this paper as: Ben-Menachem, M. and Gavious, I. “Economic Desirability and Traceability of Complex Products.” The Electronic Journal Information Systems Evaluation Volume 11 Issue 3 2008, pp. 155 - 166, available online at www.ejise.com

Economic Desirability and Traceability of Complex Products

Mordechai Ben-Menachem and Ilanit Gavious Ben-Gurion University, Beer-Sheva, Israel [email protected] [email protected] Abstract: The real values and benefits of Information Technologies are difficult to quantify and frequently even to identify accurately. Existing financial models such as Net Present Value have proven insufficient for complex products, for long-term corporate goals. IS projects and software-rich products are decided upon while ignoring critical financial aspects, as the distance between the corporate product vision and the reality that engineers see may be very large. This paper maps between economics vis-à-vis IS-based product management via an inter-disciplinary approach, looking at the needs and exigencies of corporate management, IS project, products and software engineering. The basis for the article is a discussion of the difficulties in evaluation of the economic desirability of complex, software-rich products. It presents a dynamic corporate-level model for economic profit evaluation designed to deal with the unique characteristics of such products, over many variants and versions, and the entire lifecycle. Given the extreme uncertainty of costs, benefits, risks and timeframes projections, the model facilitates real time reporting via an information system designed for management of Products, Portfolios and Projects. Whereas existing project management techniques such as Earned Value Management provide a general basis for managing project level activity, our model provides a longer-term view to assess economic affects of corporate strategies over time. This is provided by a dynamic, Management Information System based aggregation of all product information, over an entire product lifecycle, with the objective to provide a knowledge base for corporate dynamic decision-making. Concomitantly, the model fulfils Sarbanes-Oxley Act of 2002 requirements for management assertion traceability of valid and accurate measures. These aspects co-joined, from Sarbanes-Oxley, back through multiple products, over myriad versions, and through automated requirements, design and testing tools, all combine to form an auditable management feedback loop that can be leveraged at multiple corporate management levels. The paper represents a significant step towards quality product decision-making via a model that is meaningful, while also useful as it is leveraged through an automated tool set. Keywords: economic profit, information systems, IS management, IS evaluation, product management, project management, traceability

1. Introduction

In the modern economy management seeks enhanced measurement capabilities of economic benefits provided by complex, software-rich, products. This economic benefit is usually derived from costs’ savings, revenue increase and/or enhanced resource-usage efficiency. Project managers are required to provide corporate management with a supporting environment for quality decision-making; while products compete for resources within an environment of limited resources. Extant research examines issues of project efficiency maximization. For example, a recent study by Serich (2005) shows how the combination of Concurrent Engineering with Prototyping enhances project efficiency. However, what is still lacking is the multi-dimensional facet of the economic value of the product to the organization. Notably, economic value cannot be limited to the period in a product/system lifecycle devoted to initial development, but needs to be viewed in a longer context. Corporate environments are characterised by multi-layered product portfolios; wherein each separate product is created, maintained, enhanced and/or expanded via the facility of a project. Basic data concerning these processes begin from the project level but for corporate decision-making, need to be aggregated to product, and frequently portfolio, level. This demands a dynamic decision-making instrument. One of the most common financial tools used for investment decision making is Net Present Value (NPV). Literature referring to economic modelling for software-rich products is very sparse; whether in computing literature, finance literature or management literature (e.g., Turnbull, 2003; Greiner, 2003; Armour, 2005). Boehm’s (1981) well-known book “Software Engineering Economics” deals mostly with software engineering aspects, and not the economic or management aspects. Tockey (2005) is the first book to address this area specifically. His orientation is for the software professional, and is based on an expectation that the usual software practitioner is unfamiliar with economic aspects. This gap between the two professions creates an economic reality in which Information Systems (IS) projects, the basis of every modern organization's economic existence, are being decided upon while ignoring many critical financial aspects.




This article presents a dynamic model for economic evaluation of the desirability of complex products. Throughout this article, the term ‘complex products’ refers to software-rich products, containing in their lifecycle, multiple versions (changes over time) and multiple variants (changes in use). The model is based on analysis of critical differences between complex and more ‘traditional’ products. These differences are found in costs and expected benefits from the product as well as associated risks and timeframes. Our model applies these differences as measurable information, collected from real objects and events, by an information system. Existing project management techniques, such as Earned Value Management (EVM), provide a general basis for managing project level activity. A project management approach is sufficient in an environment of incremental development or a ‘one up’ project. Our model provides a longer-term view to assess economic affects of corporate strategies over time. This is provided by dynamic, Management Information System (MIS) based aggregation of all product information, over an entire product lifecycle. The objective is to provide a knowledge base for corporate dynamic decision-making. Concomitantly, our model supplies business goals traceability via specific procedures. This is consistent with the Sarbanes-Oxley Act of 2002 (Section 203) requirement for enterprises to implement traceability and change-management, from management assertions through components. The paper is organized as follows. Section 2 discusses principles of economic evaluation of complex products, focusing on the “traps” in economic modelling. Section 3 presents a multi-dimensional analysis of costs and benefits, with incorporation of risks and timeframes for complex products. Thus providing basis for an economic profit model adapted for the unique characteristics of complex products. Section 4 presents our dynamic model for evaluation of complex products' economic desirability, designed for constant updating via a functioning Management Information System. This dynamism is shown to be crucial in such products because of the extreme uncertainty embedded in projections of costs, benefits, risks and timeframes. Section 5 contains summary and concluding remarks.

2. Economic evaluation of complex products’ desirability

Managers generally base business decisions on evaluation of costs and expenses versus revenues, or other potential benefits. Management needs to perceive which resource investments may produce an optimal result, which should be economically expressible. One possible way to represent this conceived optimum might be via estimation of the expected return on a particular investment, as compared to alternative investments – though a specific project may be chosen even though economic models may not show it the optimal investment (for instance from considerations such as market competitiveness, customer services, quality or other not-readily-quantifiable corporate needs). This is made much more difficult in a software-rich project. Such projects should be measured using standards of performance similar to traditional, tangible asset-based projects. However, while they are expected to stand under the same magnifying glass of economic profit measures, they are not expected to produce the same “kinds” of profit as benefits may frequently be qualitative (e.g., customer satisfaction). Furthermore, looking at the entire lifecycle of a complex product, rather than a single project, complicates the decision-making process. The use of a financial model to measure product desirability is valuable. A financial model supplies a clear framework for forecasting costs slated for an investment and evaluating its expected benefits. It enforces a well-designed implementation plan from all stakeholders, with viewpoint1 commonality. A model helps organizations understand and concur on expected results. At the least, such a model is a common tool to show all stakeholders, from the “lowest level” to senior management that an investment has economic value. However, a significant “trap” in economic modelling is the inherent inaccuracies embedded in forecasting future cash flows and cost of capital over investment lifecycle. This is made much more problematic in software-rich products as development times may be quite long. Moreover, economic value cannot be limited to the period in a lifecycle devoted to initial development, but also needs to account for on-going system evolution (ASD C3I 2002; Ben-Menachem & Marliss 1997; BPM Forum 2004). The temporal gap between the project’s initial requirements analysis and the concomitant requirements as the system or product matures, then serves to “forcibly grow” the aforementioned inaccuracy. This is more than just requirements drift, which tends to infer changing of agreed upon requirements while in development, and includes new 1 As defined by IEEE Standard 1471: [3.1.9] view: a representation of a whole system from the perspective of a related set of concerns. [3.1.10] viewpoint: a specification of the conventions for constructing and using a view. A viewpoint acts as a pattern or template from which to develop individual views by establishing the purposes and audience for a view and the techniques for its creation and analysis.


Mordechai Ben-Menachem and Ilanit Gavious

requirements resulting from the learning processes inherent to development. The temporal issue has additional effects, as while the product and technology evolve, other effects occur, such as changes in the business environment. Another trap is that quantifiable methods for estimating intangible results are a major challenge; while frequently, these intangibles may be the most significant (for example, enhanced Information Retrieval ease/speed or an ability to make higher-quality business decisions). Despite these weaknesses, financial modelling is usable even in complex products, provided the model and the modelling techniques are adapted to the product. We make this adaptation in a multi-dimensional costs and benefits analysis, with the incorporation of risks and timeframes, for complex products.

3. Multi-dimensional costs and benefits analysis of complex products Our model for economic evaluation of complex products' desirability is based upon the traditional NPV method of evaluating the desirability of investments. This section shows the basis for our model and critical points of difference between it and NPV. These differences are in the areas of costs and expected benefits from the product as well as associated risks and timeframes.

3.1 Costs analysis

A deep understanding of costs is critical for good product planning. The Information Technology industry is rife with reports concerning projects’ failure rates (Clancy, 1998; Taylor, 2001). Enhanced understanding of costs involved in a project will enhance product managers’ ability to effectively plan products, as well as enhance abilities to react to changes in real time. Studies that deal with this issue (e.g., Boehm, 1997) tend to address initial development exclusively, with little or no discussion of on-going costs. Complex products' characteristic costs divide into those that are typically predictable and those that are known ex post facto. The former include cost of hardware (equipment of all kinds), communications networks (infrastructure and connection costs), software licenses, manpower (technical and non-technical personnel, organic to the organization or external “consultants” employed for temporary tasks), training (staff and users) and installation (including data conversion). Although predictable, these costs pose a high degree of uncertainty, temporal and otherwise. For example, whereas consultants may be hired for exactly the times needed, employees’ availabilities are not pre-determinable by project needs (only) but also by the availability of the needed personnel; that is, the correct and needed personnel must be available, many of whom may be specialists. The second type of costs includes aspects of a more complex structure. This complexity causes them frequently to display unpredictably quantifiable aspects. Software (and withal, software-rich products) differs from “traditional” product development projects in four broad areas, Commercial-Off-The-Shelf (COTS) tools and modules, quality assurance activities, development processes and system uncertainty. COTS tools or modules: It is very rare to develop an entire bespoke system/product. Almost all, whether information or reactive systems, include some vendor-acquired modules and some bespoke. In addition, most work environments (development and evolution) include vendor-developed software tools (these may be deliverable with the project or non-deliverable, used only for development). Some applications may contain both types, e.g., reporting tools and acquired processing modules (the concept of module infers that it becomes an active part of application operation; while a tool is either passive or external). Packaged modules and tools predetermine a great deal of the design structure of the chosen solution and highly affect all development and evolution processes used to produce and maintain product suitability. A high quality, flexible set may reduce both development and evolution costs. Poor choices will cause many unforeseen and unforeseeable occurrences. Frequently, a great deal of expert and specialized knowledge is needed to utilize these purchased items effectively. This knowledge may also be acquired and not necessarily with the purchased product. Components’ purchase is common in every project – but there is a very significant difference. In traditional projects one purchases a component with the clear intention that it be fundamentally a ‘Black-Box’. In software, this is almost never the case. Almost all purchased software components are ‘Grey-Boxes’ where functionality is acquired and then needs to be studied (and frequently modified) before it can be utilized, or utilized optimally. This difference is a very large cost component, above and beyond the purchase cost that must be accounted for and planned for.



Quality Assurance activities: Large portions of the resources of software-rich projects are used to correct errors produced by the project (typically, fifty percent of overall expenditures (Ben-Menachem & Marliss, 1997)). As part of, and in parallel with, the quality assurance process, every project contains many verification and validation processes, both to capture errors and to allow timely correction after discovery. Generally, software quality is not easily quantifiable (despite many attempts). Notwithstanding best efforts, every software system contains some unknown quantity of defects. Moreover, defects also occur in parts of a system that were previously satisfactory, as requirements and environments change and evolve. Many costs of non-quality can be computed, but only in retrospect. Software development processes: Theoretically, discussions of software development process may not seem significantly different from many other process-oriented industries - for example, the highly published processes in the automotive industry (e.g., “common platforms use”). However, when designing an automobile, the concept is to design it once and then manufacture as many “copies” as possible (even if the copies are not identical, they are technologically non-dissimilar; differing usually in superficial aspects such as colour schemes). Software does not have manufacturing. It is always designed and produced ‘one-up’, then constantly changed and evolved afterwards. Automobiles are also not designed to be updated while in use, while software must be constantly updated to remain viable (Lehman & Ramil, 2001). System uncertainty: A most prevalent attribute of software-rich projects is high uncertainty; degrees of uncertainty ensure unforeseen costs. Lehman (1990), in his seminal article concerning computer system uncertainty shows that: “In the real world, the outcome of software system operation is inherently uncertain with the precise area of uncertainty also not knowable”. Additionally, Ben-Menachem & Marliss (2004) showed that this uncertainty principle extends beyond programs to systems and systems-of-systems (e.g., ERP) and more than that, is not static and is a natural aspect of the ways that systems evolve. To whit, “…all change increases system uncertainty, which grows exponentially as the configuration item view extends toward systems and systems-of-systems; anomalies propagate, creating unforeseen states, and inherent system uncertainty explodes”. Unforeseen costs refers to costs which, while the experienced project manager knows that such will occur and may even be able to “guess-timate” their approximate quantities, what these will be, when they will occur and exactly for what will the money actually be spent, is impossible to define ex-ante.

3.2 Benefits analysis

Potential benefits of complex products may be quite broad. Occasionally, the most significant benefits will be strategic. Strategic and tactical benefits should be differentiated, as the former frequently are not directly measurable while the latter may be straightforward (see Tockey, 2005). Strategic might be overall “change the company” concepts such as ERP implementation, while tactical might be enhancement of a corporate function (e.g., customer management). Many systems surprise the producers, bringing benefits other than those originally defined, some of whom may actually prove of greater value than the original plan. Not all benefits are, or can be, tangibly defined. For example, if the new system enables more efficient report production, using less manpower and/or less time, than the direct benefit is clear and can be quantitatively measured. However, indirect benefits may be more difficult to measure - staff that previously produced this report may now be free to work on enhanced services. These are not as easily measured. Additionally, benefits are not actualized at once. Projects may take significant time to implement (three to five years is not uncommon) while many other factors may affect the corporate environment during the period. Direct cause-effect analysis is impracticable, as it may be difficult to isolate a project’s benefits from other causes. However, this does not preclude indirect analysis and/or use of simulations and projections. Pre-project evaluation of benefits and costs should be based on projection of the (proposed) situation with and without the project, and not before and after the project. This is to isolate (proposed) changes to the organization caused by the project (see also, Tockey, 2005, pp. 305-306) and even more so, by the product that results from the project (or projects) creating and evolving it. Whereas benefits analysis of an individual project is difficult and complex, it is significantly more so for an entire product lifecycle, at least as a product may typically be developed via many individual projects over its lifecycle.



3.3 Risks of complex products and their effect on the investment

Longstaff, Chittister, Pethia & Haimes,(2000, pp. 45) state: “We cannot simultaneously measure the risks associated with software and information assurance when no protective actions are taken and measure the efficacy of deploying risk assessment and management on the system because the system has fundamentally changed.” This extends Heisenberg’s Uncertainty Principle, to software, and has quite profound and far-reaching effects on economic modelling of software-rich systems (“…since the measuring device has been constructed by the observer, and we have to remember that what we observe, and what we observe is not nature itself but nature exposed to our method of question.” “…when searching for harmony in life one must never forget that in the drama of existence we are ourselves both players and spectators.”, Heisenberg, 1991). What seems the most obvious characteristics of software, to both practitioners and users, are the lack of wear-down and the ease of changing it. Both perceptions are incorrect and are a major source of risk (Broekman & Notenboom, 2003). The mis-perception results from viewing software via an inapplicable paradigm. The past few centuries used an industrial paradigm for filtering worldview. The industrial paradigm assumes glacial change; so slow and gradual as to be barely recognizable except over relatively long periods of time; in hindsight. Today’s reality differs. “We live in a world where the only certainty is change. Changes occur so quickly that in most situations we must resign ourselves to them without first being able to assess what their risks or benefits might be” (Asuaga, 2001). Changes have levels of risk associated with them (Broekman & Notenboom, 2003), and as they occur so quickly, how can they be economically modelled, effectively? If software always has threats, hazards and risks, what are the roles of the project, and project management, in identifying, preventing, mitigating and controlling them? Basic comprehension of risks’ centrality is critical to economic modelling (Armour, 2005). In order to comprehend the difficulty inherent to the task of risk management and its concomitant affects on economic modelling, one must understand that this is not limited to “the expected” – that is, to the risks inherent in original product development and deployment (see section 3.1 “System Uncertainty”). Brooks (1995) added to this the inherent instability of computer systems: “Fixing a defect has a substantial (20 to 50 percent) chance of introducing another”. From this is derived the concept that after a complexity threshold is exceeded, fixing one flaw tends to create new flaws. Risks in complex products exist on at least two levels, product oriented risks and risks associated with the projects that create and maintain the product. These are separate and not a linked hierarchy of inter-related risks.

3.4 Timeframes

The timeframes in which a product’s desirability is evaluated changes as a function of the type of project(s) utilised, the type of product/system that the project(s) are intended to develop, the product’s defined (and undefined) lifecycle and the changeability of the external environment over the time that the product is in development. Many researchers have shown that software projects almost always take more time than was initially expected (Brooks, 1995; Clancy, 1998; Taylor, 2001). The project’s original definition frequently changes over time, sometimes without management knowledge. In addition, Verhoef (2002) shows that project costs and risks (and withal, concomitant product risks) “…dramatically increase when the development schedule is compressed…”. Thus, despite the basic management desirability to shorten individual projects which compose a product development (and/or evolution), as suggested by Clancy (1998), the utility of this technique has a ‘lower bound’ when this increases risk and defects. To summarize, an economic evaluation of investments – particularly for items of questionable tangibility – should examine issues both quantitatively and qualitatively. It is not always simple to acquire the data needed for a proper quantitative analysis; or at least, not a sufficient population of such data as to ensure an analysis of reasonable value. In such a case, qualitative data can prove relevant and significant. This is particularly poignant for complex products where fuzzy values are prevalent. Quantitative tools tend to be more appropriate to economic valuation of traditional projects based upon “hard” tangible assets while, qualitative (or a combination of quantitative and qualitative) tools tend to be more appropriately used for “soft” or intangible assets. As such, our model uses a combination of quantitative and qualitative tools. Thus, this combination of quantitative, qualitative and risk together with opportunities creates a much more complex situation than that for which NPV was designed. The organization, after a complex product implementation (many complex products consist of a system of systems), can arguably be a quite different



organization than it was before. This “degree of impact” on the enterprise implies that the optimal model will supply more than just a numeric measurement; but rather, it will show a strategic view of product value to the enterprise, including qualitative values, information states, etc.

4. The model

We present a model for economic evaluation of complex products' desirability based on estimation of their incremental contribution to corporate value. Standard financial models, such as NPV and ROI, are static, have difficulty mapping between quantitative and qualitative data, are not oriented towards a view of systems (or systems of systems) and not designed for constant updating via a functioning MIS. A model needed for complex products must be dynamic. The large extent of risks and system uncertainty compels the corporation to base decisions upon verifiable, event-related data, as supplied by both Financial and Inventory Management Information Systems (see section 4.1.1, below), mapped with strategic Business Goals and Product Plans. Notably, this complexity is layered on top of the Enterprise Projects Management system. This basis, upon multiple management systems, allows application of the model to products and to portfolios of products, at enterprise level. Product planning generates a project or series of inter-related projects. In the framework of our model, at product initiation, an initial estimation of economic profit from the product is performed based on projections of costs, benefits, risks and schedules. All these projections are product-level; as a product management process and not a part of the development (technical) process. This will usually be a part of the approval process. Each year or period (whatever periodicity management chooses as optimal) a comparison needs to be performed between projected and realized schedules, costs and benefits. Projections for the future are then updated via new information from this period and new profit computations are executed. An intelligent application of future projections adjustment demands cooperation between two parties; the firm’s financial officers and software engineers. This collaboration is important for a thorough analysis from both perspectives, finance and engineering, needed to complement each other. During the evolutionary process of complex products there will be many versions and releases (Lehman & Ramil, 2001). The computation as per our model, is intended to be realized from “any point in time” until “any point in time” – inclusive of any chosen timeframe. The model is a corporate-level management instrument, for viewing a product (or portfolio). One of the commonly used project management tools is Earned Value Management (EVM) (Boehm & Li, 2003; Ernst, 2006; DI-MGMT-81466, 2004; Lipke, 2002). EVM views tasks’ schedule, resources, requirements and costs. Each and all together, are addressed as managerial aspects of the task that can be itemised. It is a bottom-up project view and does not address overall product separating the project that creates the product or a new version of it from an overall corporate view. We now present the model; depicted by two flow graphs (Figure 1 and Figure 3). Figure 1 presents the basic economic profit computation, with its information sources and processes. Figure 3 presents the iteration process of updating profit computations, with relationship to business goals and overall product planning. The graphical language used is standard notation, from Data Flow Diagramming techniques.2

4.1 The basic economic profit computation

Figure 1 presents a flow graph, divided into three sections. The left section deals with costs’ acquisitions (actual and projected), the middle section deals with acquisition and analysis of benefits (realized and projected) and the right section deals with the economic profit computation, albeit with the updated and dynamic inputs allowing needed model sophistication.

4.1.1 Costs’ acquisitions

Costs acquisition divides into two processes; quantitative cost elements and conversion of the costs into monetary terms. The first process recognizes quantitative cost elements and has two data sources. The first source is an on-line inventory system for software assets that shows enterprise software artefacts with their history and

2 A “bubble” is a process (always begins with a verb); a rectangle is a “source or sink” entity; a parallel with arced ends represents a data stores and arrows represent flow (with heavy arrows depicting main flow path).



complexity, and the accumulated costs incurred (Ben-Menachem & Marliss, 2004). The second is a Products Management MIS (see Figure 2). This system back-ends to MS-Projects for task tracking, providing data relating to business goals, lifecycle events, present and projected resources, and business and product risks. Note the absence of time as a controlling element. Time is provided for by any standard project management tool (e.g., Primavera or MS-Projects). The Inventory system stores values concerning realized costs, while the Products MIS shows projected costs. The second process converts all costs (such as data stored in terms of ‘hours’, or other non-financial terminology) into monetary terms for use in the profit computation. It is aided by a financial reporting system (Ben-Menachem & Gavious, 2007).

4.1.2 Benefits’ acquisition and analysis

Costs are actual or projected values that are fairly straightforward to collect and simple to understand. Benefits are much more complex, whether actualized already or projected (see section 3.2). The first process is to capture system benefits. The main source for understanding benefits the definers of the system intended is from the list of system goals. However, this will cover only those intended benefits defined as primary goals of the system. Many systems have additional benefits that are indirect results of their development. In many cases, these may even be difficult to discern. We recommend that the system be examined, in a recursive relationship, to discover additional, indirect benefits. These benefits are then critically examined and quantitative benefits are separated from qualitative. Quantitative benefits are converted into monetary terms, via information from the financial reporting system and supported by Expert opinion; while in parallel, the qualitative benefits are converted to metric values via the Goals-Question-Metric Paradigm (Mashiko & Basili, 1997) or some equivalent tool. This analysis provides additional value to the model as a management support for tracing system benefits’ development and for auditing of systems’ value to the organization.

4.1.3 Computation of economic profit

The Economic Profit computation is executed based upon updated projected cash flows of costs and benefits. Realized costs and benefits are relevant for the adjustment of projected costs and benefits, however irrelevant for decision-making; i.e., they are no longer relevant for evaluating the viability of continuing with the project. To account for risk and the time value of money, a risk-adjusted discount rate needs to be intelligently estimated, with the required differences per industry or business environment, by the firm's financial officers in collaboration with software engineers. This collaboration is needed for a risk analysis from both perspectives of finance and engineering, resulting in a best estimation of a risk premium to be accounted for in the cash flows' discount rate (Tockey, 2005).



Financial experts

Inventory

Economic profit

System goals

The system (recursively)

Expert opinion

Parallel analysis for tracing &

audit

GQM

Recognize system benefits

Convert quantitative to monetary

terms

Convert qualitative to metrics

Differentiate quantitative

from qualitative

benefits

Convert to monetary

terms

Recognize quantitative

cost elements

Financial reporting system

Products Management

MIS

BenefitsCosts

Risk adjusted

discount rate

Strategic (direct) & primary

indirect

Figure 1: The basic economic profit computation

The processes described above need to be based upon sophisticated information processing tools. Until recently, tools to provide corporate management with realistic, real-time data that maps both existing systems and projects in progress were lacking. Recent development of a discipline called the Paradigm of Change (henceforth ‘PofC’) (Ben-Menachem & Marliss, 2004) provides an integrated toolset designed for data collection from projects, information management concerning projects (of all types, i.e., projects that create new systems as well as those that update existing systems) and knowledge management for decision support concerning complex products. Figure 2 shows the Products Management MIS focus of the PofC (one of several possible foci that PofC provides).



Figure 2: Screen shot of products management MIS

Figure 1 includes a series of data-models (Products Management MIS and Inventory) drawn directly from PofC and from financial reporting system which has information derived from PofC. In addition to these, the Analysis, GQM and System functions base their operations on PofC-derived knowledge.

4.2 The iteration process

Figure 3 presents a superset of the economic profit computations presented in Figure 1. It shows the evolutionary profit computations process, automatically updated with aid of the information system; hence making it a useful enterprise management tool. The Product Plan and the Business Goals from which the product plan is derived, are general inputs to this process. The product plan has a defined scope and objective. Scope is critical to comprehend economic profit computation, vis-à-vis the goals defined by the business. Project plans are created by engineering staff (engineering management) in terms of performance, time (milestones), risks management and budget. The ‘Projected Costs’ and ‘Projected Benefits’ in Figure 3 are projected at product initiation, but in realistic practice today, seldom revisited categorically for continual feedback as to product viability. The basic, ‘initial’ economic profit (I) computation is performed at ‘initiating project’ (the project that initiates the product) start, usually as part of the process of product approval. Each period (as chosen by management) a new computation is performed based on adjusted Cash Flow (CF) projections (economic profit (Sn)). As shown, the process of performing these computations is now much simplified by the fact that the input data for the computations are all based upon the PofC Management Information System. The succeeding computations are performed ‘iteratively’ per period. Note the two feedback loops in Figure 3. The inner loop (dashed line) is internal to the model process. It shows that the computed profit is based upon constantly updated data/information from the MIS and that each new computation is, itself, also stored in the MIS for future use. Thus, the accumulation of computed profits becomes a knowledge base with which upper management can deal for constant updating of business goals processing.



Notably, this becomes a positive feedback loop (as opposed to passive feedback) because it represents Past-Present-Future value of model use. Accounting systems, for instance, show a picture of the present based upon accumulated data from the past. However, such normative management information is augmented by showing the combined processes of profit evaluations at every action-point during product evolution. Additionally, these action-points can be compared one with others, across projects and across products, so that decision-points and action-points can be aggressively compared for enhancing the processes of making improved business decisions. This leads to Future, where the Past and the Present combine positively to enhance both the business model itself and the processes of improving the modelling technique. Finally, as shown in Figure 3, the model contains an element called ‘defects’. Every technology-rich project creates defects while in production. This is the state-of-the-art and the inherent nature of all technology-rich projects. Their quantity can be limited by very careful development processes, and their effects can be minimized if technology management so desires. These are choices with associated short-term costs; however it is generally accepted that the long-term corporate benefits justify them. Product defects may interfere with an ability to satisfy critical requirements or key functionality and always negatively affect customer satisfaction. Defects also negatively affect schedules and future product cash flows. All this negativity must be reflected in projections. Some project management methods can deal with estimations of defects and resources consumed, such as parametric estimating, backfiring or several function point estimation techniques. This relates to specifics of software coding and is outside the scope of this article.

Product plan

Realized CF (Sn)

Defects

DefectsBusiness

goals

Projected benefits

Projected costs

Adjusted projections

Economic profit (Sn)

Economic profit (I)

CF projections

iterations

Products Management

MIS

The system

Knowledge-base of economic profit processes

Figure 3: The iteration process

The suggested model helps learn from history (“mistakes”) in estimating product economic desirability, where costs are typically underestimated and benefits are typically overestimated (Tockey, 2005). Experience improves firm's costs and benefits analysis and hence its investment decisions. Beyond management’s need to perceive which resource investments may produce an optimal result is now added a regulatory requirement for lifecycle traceability. Sarbanes-Oxley Act of 2002 (Section 203) requires that IT be made transparent to the business and fit within mainstream business management. “Specific assertions … have a general relationship to assertions about internal controls for both manual and information processing systems”. “An assertion can either be explicit or implicit” and based upon “specific procedures that can be tested and used to evaluate the assertions made by management.” Furthermore, “Assertions made regarding disclosure/internal controls can be evaluated by testing stated or measurable criteria.” The criteria are based upon “specific assurance objectives” such as business goals that are traced via change management, mapped from management assertions to products and product components.



To accomplish these regulatory requirements, IT and business goals need to align and this alignment must be traceable, throughout the entire product lifecycle – that is, from the first requirements statement until the product is finally retired from all use. Use of automated requirements gathering and analysis tools, together with design and testing traceability, allows auditability of this alignment, both immediately and long-term. Economic evaluation, performed periodically via auditable information, from a traceable resource (MIS) and recorded for continual re-evaluation, endows corporate management with verifiable tools to affirm this regulatory requirement. Figure 1 shows this feedback and how this 'loop' is utilised for weaving the regulatory requirement into the corporate fabric – requirements, audits, economic evaluation, new product planning, etc.

5. Conclusion

Products and product portfolios should be economically valued, using economic terminology and standard financial computations. Economic models, common to the financial-world, apply to information systems (software-rich products); for they can be made to apply with adjustments. The application of economic models inherently enhances a corporation's long-term viewing capabilities. Software-rich products need to be managed with a view extending significantly farther than competitors conceived. Notably, many analysts have claimed that the primary driving force to market power is a long-term view. Organizations need a long-term knowledge base, to attain a long-term view. The on-going management process, with all its decision-points, increases the necessity for long-term product information. Moreover, today’s regulatory environment (e.g., Sarbanes-Oxley Act of 2002) necessitates the availability of such information. Withal, how can this long-term view be consistently achieved, even in corporate environments that have not been previously built upon such concepts? Or, what can a corporation do, when it perceives that this kind of product view is not yet its corporate strength, and wishes to enhance capabilities in this area? This article discusses these issues and provides a dynamic model for economic evaluation of complex products’ desirability, via an inter-disciplinary approach, by mapping products, with the needs and exigencies of corporate management, product managers and software engineers. The model facilitates real time reporting via an information system designed for management of Products, Portfolios and Projects. This dynamism is shown to be management crucial because of the extreme uncertainty embedded in projections of costs, benefits, risks and timeframes associated with software-rich, complex products. As an additional significant benefit, the model fulfils Sarbanes-Oxley Act of 2002 requirements for management assertion traceability of valid and accurate measures, with auditability. The model is designed to be based upon quality information, from automated data-gathering utilities, that is, the information is both of high quality and highly auditable, allowing corporate management to leverage this information to maximum benefit. What is no less significant is that this information can be used consistently, over related products, and to make long-term comparisons between product lines, while taking all their aspects properly into account. These aspects co-joined, from Sarbanes-Oxley, back through multiple products, over myriad versions, and through automated requirements, design and testing tools, all combine to form an auditable management feedback loop that can be leveraged at multiple corporate management levels. The paper represents a significant step towards quality product decision-making via a model that is meaningful, while also useful as it is leveraged through an automated tool set.

References Armour, P. G. (2005) "Project Portfolios: Organizational Management of Risk", Communications of the ACM, Vol 48, pp.

17-20. Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (ASD C3I). (2002). Repairing

Latent Year 2000 Defects Caused by Date Windowing. Investment and Acquisition Directorate, US Department of Defense, Version 1.0, April.

Asuaga, A. (2001) "Engineering a New Society", IEEE Computing, June, pp. 102-104. Ben-Menachem M., & Gavious, I. (2007) "Accounting Software Assets: A Valuation Model for Software", Journal of

Information Systems of the American Accounting Association, Vol 21, No.2, pp. 117-132. Ben-Menachem, M., & Marliss, G. (1997) Software Quality: Producing Practical, Consistent Software, Int’l Thomson

Computer Press.




Ben-Menachem, M., & Marliss, G. (2004) "Inventorying Information Technology Systems: Supporting the "Paradigm of Change"", IEEE Software, Vol 21, pp. 34-43.

Boehm, B. (1981) Software Engineering Economics, Englewood Cliffs, NJ: Prentice Hall PTR. Boehm, B. (1997) COCOMO II Model Definition Manual. University of Southern California, Version 1.4. Boehm, B. & Li, G. H. (2003) "Value-Based Software Engineering: A Case Study", Computer, Vol 36, No.3, pp. 33-41. BPM Forum. (2004) Software Drain or Business Gain: Assessing Application Value, Relevance and Cost to Your

Company. http://www.bpmforum.org/PDF/Software_Drain.pdf; December. Broekman, B., & Notenboom, E. (2003) Testing Embedded Software, Addison Wesley. Brooks, F. (1995) The Mythical Man-Month, Addison Wesley. Clancy, T. (1998) CHAOS, Standish Group. DI-MGMT-81466. (2004) Contract Performance Report. Department of Defense, December 2004. Ernst, K. D. (2006) Earned Value Implementation Guide. Department of Defense, October 2006. Greiner, L. (2003) "Analytics Unlock the Secrets to ROI", Computing Canada, Vol 29, pp. 16-17. Heisenberg, W. (1991) “The Copenhagen Interpretation of Quantum Theory”; as reproduced in The Treasury of Physics,

Astronomy and Mathematics, ed. Timothy Ferris, Little, Brown and Company, pp. 86-96. Lehman, M. M. (1990) "Uncertainty in Computer Applications", Communications of the ACM, Technical Correspondence,

V33N5, pp. 584-586. Lehman, M. M., & Ramil, J. F. (2001) "Rules and Tools for Software Evolution Planning and Management", Annals of

Software Engineering, Vol 11. Lipke, W. H. (2002) EVM and Software Project Management - Our Story. Cross Talk. Longstaff, T. A., Chittister, C., Pethia, R., & Haimes, Y. Y. (2000) "Are we forgetting the risks of information technology?",

Computer, December, pp. 43-51. Mashiko, Y., & Basili, V. (1997) "Using the GQM Paradigm to Investigate Influential Factors for Software Process

Improvement", Journal of Systems Software, Vol 36, pp. 17-32. Sarbanes-Oxley Act of 2002. Available at: http://fl1.findlaw.com/news.findlaw.com/hdocs/docs/gwbush/sarbanesoxley072302.pdf Serich, S. (2005) "Prototype Stopping Rules in Software Development Projects", IEEE Transactions on Engineering

Management, Vol 52, No.4, pp. 478-485. Taylor, A. (2001) IT Projects Sink or Swim. British Computer Society Review, September. Tockey, S. (2005) Return on Software, Addison Wesley. Turnbull, I. (2003) "Finding the Return on HR Technology Investments", Canadian HR Reporter, Vol 16, pp. 17-18. Verhoef, C. (2002) "Quantitative IT Portfolio Management", Science of Computer Programming, Vol 45, pp. 1-96.

http://fl1.findlaw.com/news.findlaw.com/hdocs/docs/gwbush/sarbanesoxley072302.pdf

EJISE Volume 11 Issue 3

Documents

Transcript of EJISE Volume 11 Issue 3