Evaluating I .. T.. Investments: An Organisational Perspective

by

Pamela Simmons

(Serial No.. 101)

ISBN 0 85590 726 6

This paper should not be quoted or reproduced in whole or in part without the consent of the author,. to whom all comments and enquiries should, be directed.

© Simmons, P, 1993

FACULTY OF BUSINESS WORKING PAPERS·

This series of Working Papers has been established to encourage the dissemination of research and work in progress undertaken by the staff and postgraduate students of the Faculty of Business, Swinburne University of Technology. Each paper is refereed before publication. Comments are welcome and should be directed to the author(s) of the relevant paper.

Enquiries concerning the Faculty of Business Working Papers should be directed to:

The Editor Working Papers Faculty of Business Swinburne University of Technology PO Box 218 Hawthorn Vic 3122

Additional copies of this paper are available at cost upon request.

The views and opinions expressed in Working Papers are those of the individual authors, and publication' does not imply endorsement by the Editor or the Faculty of Business.

;

.\ ' t .

{ "

J

I ' 1 • 1

I

l i

EV ALUA TING t To INVESTMENTS:

AN ORGANISATIONAL PERSPECTIVE

Pamela Simmons Department of Information Systems Swinburne University of Technology

March 1993

ABSTRACT

This study focuses on the way in which organisations evaluate their investments in infonnation technology (IT). The objective is to bridge the gap betwen the mainly theoretical work done on evaluation and the ex-periences and needs of organisations.

A comprehensive benefits and measurement framework is described. The five types of benefit cover all benefits that are likely to result from an IT investment and a set of measures, based on the literature and the experience of organisations, is proposed. If the same measures are al£o used to review the project after implementation and'ifionitored regularly as part of a benefits m&nagement programme, they provide feedback to the business managers to ensure that the expected benefits are achieved. Taking this proactive step increases the likelihood of investments in IT resulting in enhanced business performance.

This study has a1so revealed that organisational context influences evaluation policy and practices. Differences in attitudes towards effectiveness. evaluation and measurement can be eX'Plained by the industry sector, the culture and the management approach of the organisation. A complete model of the evaluation process demonstrates the complex nature of the evaluation of IT invesunents in organisations ..

1. Introduction Organisations have invested in IT (Information Technology) at ever increasing levels. Expenditure on information processing and communications rose 14.6% annually from 1975 to 1985 and annual investment in IT now represents half of total corporate capital spending (Keen, 1991). Until recently) the assumption has been held, largely unquestioned, that this investment would lead to more efficient operations which would in turn be reflected in the standard measures of business performance. But studies of IT expenditure and business performance have not generally been able to substantiate the assumption that increased investment in IT can be linked to improved business performance.

This paper summarises some of the problems found in linking IT investment to improved business performance and focuses on the problem of establishing suitable measures for the impact of IT investments on business performance. It describes a study which examined how organisations evaluate their investments in information techt:lology and so bridges the gap between the mainJy theoretical work done on evaluation and the experiences and needs of organisations. This research focused on the way in which organisations evaluated individual projects, both at the justification stage and later, after the project had been implemented. It explored the problems -they faced and the type of measures they considered to be useful.

A comprehensive framework of benefits and measures is described. It has been developed in order to overcome some deficiencies found in other evaluation frameworks reported in the literature; it was then validated using the actual j~stifications of twenty five projects. -

The study has also identified the importance of organisational context when determining the most appropriate evaluation approach. Research in the area of IT evaluation has been dominated by the positivist paradigm. The interpretive approach taken in this research has allowed a new issue to emerge: the-importance of the organisational context. Great differences have been found in the attitudes of organisations towards effectiveness, measurement and evaluation. These differences can be explained by the industry sector, the culture and the management approach of the organisation. A model of the complete evaluation process in" organisations has been developed. It shows the way in which the investment decision is related to the formal decision criteria, informal influences, project characteristics and attitudes to effectiveness and evaluation.

2. Investment in IT and Organisational Performance Studies investigating the relationship between IT expenditure and various measures of business performance at all levels of analysis have generally not been able to demonstrate that increased IT investment was related to improved business performance. (Kauffinan and Weill, 1989)

At the national level, productivity does not appear to have increased as a result of investment in technology. An analysis of investment and productivity in the service sector, based on U.S. government statistics, demonstrated that although the finance

2

,\, \

I , \ I

I (

and insurance sector had experienced the greatest increase in technology intensity between' 1970 and 1985, productivity performance in those industries declined by 1.4% over that period (Roac~ 1988).

At the business unit level, the central finding of a study of data for U. S. and western manufacturing companies from 1978 to 1984 was that IT capital had little if any marginal impact on output or labour productivity while all other inputs into production, including non-IT capital, had significant positive impacts on output and la~our productivity (Loveman, 1988). In fact, Loveman concluded that "profit maximising :firms in this sample would best have invested their marginal dollar in non­IT factors of production II (Loveman, 1988, p.3).

It is therefore not surprising that senior executives are now questioning the wisdom of continuing to invest in IT at such high Jevels. The respondents to a Datamation survey (1991) expressed doubts concerning the rea] benefits of IT investment; these doubts are also reflected in these research resuJts. A study of Australian and New Zealand organisations re~ealed that IS managers in these countries have similar concerns to their American counterparts; the problem of determining the return on IS investments was considered to be important by 65% of the managers questioned. (Broadbent, Lloyd, Hansell and Dampney~ 1992)

Reasons for difficulty in establishing the return on IT investments The literature offers a number of explanations as to why the expected improvement in business perfonnance resulting from the increased investment in IT is not evident.

- Organisations have responded to the price decline of IT by applying increasing amounts of IT capital and information workers to marketing and services, rather than focusing on reducing the costs of producing goods (Baily, 1988). The value of such services produced is not captured using the usual economic indicators of business performance.

- It is difficult to assign monetary value to information. (Loveman 1988, Keen 1991)

- Using traditional financial measures such as return on assets as an indicator of the effectiveness of IT investments does not provide an accurate picture. Investment in IT increases the asset base of the organisation but the investment allows an increased volume of business to be handled by fewer staff. Thus return on assets as a measure of business performance does not favour expenditure based on reducing recurrent expenditure and building a capital base. (Keen 1991)

- Some organisations would be better performers than others, with or without IT. The deterministic approach taken in most studies has neglected such intraorganisational factors and this may explain some of the unsatisfactory results reported in the literature (Barna) Kriebel and Mukhopadhayay, 1989). However, a study by Weill (1989) did include an indicator of the organisation's ability to make effective use of the technology. He found that organisations with high tlconversion effectiveness" were relatively better performers for the same strategic IT investment.

The first three points suggest that possibl~ problems exist with the way in which the impact of an IT investment, flO business performance is measured and. the fourth

3

suggests that improvement in business perfonnance does not follow automatically from an investment in IT. Thus, determining a suitable measure (or measures) is an essential prerequisite to demonstrating the link between investment in IT and business performance. Such measures can then be employed to assist managers leverage the maximum benefit from an IT investment.

Measuring the Impact of IT on Business Petformance. It is evident from the preceding section that economic indicators alone may not provide an accurate representation of the entire benefit obtained from an IT investment. Financial benefits address only one aspect of the va1ue derived from an infonnation system. Moreover, this is often an aspect which is of value to only a limited group of people. Decisions regarding what is to be measured and when measurement should take place will determine the "amountll of value that results.

Value The value that an organisation obtains from an investment in IT extends beyond purely financial benefits and includes other less easily quantified benefits such as better decision making, improved communications and added flexibility (Strassman, 1988) .. By taking a goal-centred view of effectiveness, the financial objectives (e.g. sales revenue, profit contribution, return on investment) are supplemented by objectives relating to customers (customer satisfaction, regulatory agency compliance) and to organisational deVelopment (morale). (Hamilton and Chevany, 1981)

Parker and Benson (1988) also extended the purely economic view of value to encompass the broader objectives of the organisation, including such dimensions as strategic direction and exposure to technological risk.

In seeking to define value, Legge (1984) raised a number of issues - whether value is intrinsic or attributed, what its elements comprise and whether there is a distinction between "meritl! (context-free evaluation) and IIworth II (context-determined evaluation). Ifvalue is attributed by people rather than being intrinsic to an entity, the only value found is that which people feel it shou1d have and consciously look for. Or, as Strassman (1988) expressed it, "questions of IT's value are settled when one decides how to measure it". (p.18)

The va1ue of an information system is inextricably linked with the value of the jnformation produced by'that system (Ahituv, 1989). In fact, Marchard and Horton (1986) asserted that "information and knowledge are factors of production as authentic and as critical as capital and labour" (p.21). But there is considerable difficulty in placing a value on the output of infonnation systems. '1'nrormation, unlike tangible .

..I goods, has no intrinsic value. It has value only when integrated with a structure that is capable of exploiting it effectively (Loveman, 1988). Chismar and Kriebel (1985) stated that "information has value for an individual only in relation to a stipulated decision context: 'value' is not an inherent property of , information I but derives from the decision problems involved and the economic consequences of outcomes for the decision maker II (p.46)

A number of characteristics distinguish information from other products and contribute to the problem of assigning value to it.

4

( \

.I

(

.I I

I \ \1

( r \ (

i \

~ Unlike other goods, one does not lose possession of an information good when it is sold; it does not Itdepletell as other goods do; often the more one uses data, the more valuable it becomes. (Marchard and HortOD, 1986)

- The value of information depends on external factors, The time that the information is provided) the number of people possessing the information and its accessibility all strongly influence its value (Ahituv, 1989)

- The standard procedures for making investment decisions are centred on the production of tangible goods, but give no guidance when the output is in the form of information. (Strassman, 1988)

The difficulty of assigning monetary value to information is considered by some to be at the root of the apparent lack of impact of IT on productivity (Loveman 1988, Keen 1991).

Stakeholders Values, being views about what constitutes "desirable consequences ", are likely to vary in context or priority from group to group (Legge; 1984). The stakeholders of an information system (e,g. internal sponsors, external stockholders and lenders, auditors, users, system developers, operators, unions, competitors and government regulators) will have individual requirements and conflicting criteria for effectiveness (Ginzberg and Zmud, 1988). The traditional method of evaluating the effectiveness of an IT investment has been to consjder only the impact on the stockholder (the economic impact) but the impact on other stakeholders (e.g. external customers, the users of the system) are also important.

Type of measure T~e selection of appropriate perfonnance measures is important as the measures adopted largely determine the amount of value that is uncovered: they could also influence the behaviour of an organisation and where ii: places its emphasis (Gold, in Carlson and McNurlin, 1989). However, the literature provides conflicting advice.

- What should be measured? Strassman (1988) suggested that managers want to measure the impact of IT on profit in e,osting financial measures. However, Gold suggested looking beyond the traditional financial indicators and measuring what management think is important.

- If it is recognised that value has dimensions beyond the purely economic, should these different dimensions (e.g. strategic support) be aggregated with the economic factor into a single measure of value, as inParker and Benson's Information Economics method (parker and Benson, 1988)? The meaning that can be attributed to such a figure is questionable and the process conceals the assumptions upon which the aggregation is founded. Other authors (e.g. Buss, 1983) proposed that the separate dimensions of value be retained .with the results possibly displayed graphically.

- Quantifiability is an issue that is particularly problematic in the evaluation of IT investments. The greatest impact of IT results from enhanced organisational effectiveness rather than improved efficiency (Eason 1988, Keen 1975, Strassman 1988) but organisational effectiveness is not easily defined or measured. It is a multidimensional concept. Some dimensions are amenable to measurement, economic performance, for example. Other objectives of the organisation, for example, improving quality or customer service, cannot be

5

Time

measured in the same objective way. But measurement does not necessarily imply the use of the standard accounting measures. The process of measurement involves lithe definition and calibration of a metric, determination of a reference, standard and goal, monitored observations, comparison of observations against a standard and a record of the result". (Kauffinan and Kriebel, 1988, p.99) Therefore we are not restricted to standard accounting measures but can develop appropriate measures provided that it is done in a structured, rigorous way.

Another aspect of measurement is the time at which the impact of an investment is measured) particularly the time required for the effect to be reflected in traditional economic measures. The impact of increased efficiency is evident fairly quickly as a result of cost savings or staff reductions. But other investments have a much longer term effect. Investments in strategic activities and in the IT corporate infrastructure are more often concerned with the long term health of the organisation than with providing a rapid economic return. The traditiona] accounting measures do not include this investment in the future.

Thus, it is evident that the traditional economic measures of business performance fail to capture some aspects of the impact of an information system on the business. Additional indicators of value are required.

Need for Organisational Input Much of the research reported in the literature has focused on the measurement of business value of IT investments from a theoretical viewpoint with the proposed approaches derived from the fields of economics) accounting, operational research and organisation behaviour. There is little evidence of input based on the knowledge and experience of practitioners in organisations.

A gulf exists between the extremely sophisticated attempts at evaluation emanating from academic researchers and consultants and the vast majority of the evaluation carried out by non-specialist management. (Blacker and Brown, 1988) There is a need for evaluation approaches that have credibility within organisations, that are compatible with the corporate culture and that are linked to corporate objectives.

The study described in this paper addresses the following questions: 1. To what extent do organisations attempt to evaluate the effectiveness of their

IT investments? 2. What criteria are used to justifY and review the effectiveness of IT investments? 3. To what degree are organisations satisfied with the processes used? 4. How are different types of benefit treated?

Drawing on the experience of organisations to answer these questions, comparisons were then made between their experiences and perceived needs, and the evaluation methods and frameworks proposed in the literature.

6

Ii

3 .. The Research Design The objective of this study is to increase understanding of how organisations evaluate their IT investments and to determine the meaning of IIvalue'l within the orgarusation. Organisations are complex assembHes of people, politics and technology, subject to many internal and external forces. Investment decisions depend on the response of individual s to many types of information and pressures. An interpretive approach has been adopted for this research because it emphasises the importance of understanding the processes taking place (Hirschheim, 1985); it reveals the perceptions of individuals and the outcomes that are valued by organisations (Markus and Robey, 1988); and assists the development of theory (Blaikie, 1980). The focus of interest is on understanding the process of evaluation, rather than on the establislunent of a deterministic relationship or the testing of hypotheses. Case studies have been used to capture reality in greater detail than is possible in _a survey and the interactive nature of the information gathering has allowed additional issues to emerge - an example 'of grounded theory (Glaser and Strauss, 1967).

The interpretive approach taken to this research has led to an increased understanding of the way in which organisations evaluate their IT investments; it has enabled the mainly theoretical work that has been reported in the literature to be supplemented by a different perspective derived from the experiences of practitioners who have had to tackle the problems in their work.

40 The Case Organisations Nine organisations were selected for study, three in each of the banking, manufacturing and public sectors. The organisations are al1 substantial and seemed representative when cOlnpared with the survey of 200 organisations conducted by Broadbent et al. (1992). In all but one organisation the business function, often jointly with the IS department, is accountable for return on IS investments. The organisations general1y exhibit a fairJy high degree of IS maturity in that their IS strategies were developed in dose collaboration with the business areas (this was probab1y related to the criteria used to select the cases).

The very rich resource of information, gained in the interviews with IS and business managers in each organisation and from the study of the justification of many projects, yields interesting and useful answers to the research questions.

s. Framework of Benefits and Measures A framework of benefits was firstly dev.eloped from the literature (using suggestions of McDowell (reported in CaIson and McNurlin, 1989), Berger 1988, Weill and Olson 1989 and Keen 1991). The five types of benefit are sUnurlarised in the following table. .

7

Types of Benefit

Type 1 Increased Efficiency Benefits attributed to cost avoidance or reduction

Type 2 Increased Effectiveness E.g. provicling better information for decision making

Type 3 Added Value Benefits which enhance the strategic position of the organisation (e.g. increased market share due to better product differentiation or the creation of entry barriers)

Type 4 Marketable Product Development of a marketable product or service

TypeS Development of E.g. conununication networks, corporate IT hardware. database environments that infrastructure provide little direct benefit themselves

but are required as a foundation for other systems.

Table 1. Benefits Classification Framework

The justifications of twenty five projects were analysed and compared with this framework of benefits. The' cost of the projects range from $39,000 to $17 million and they inc1ude transactional, informational, strategic and foundation systems. Each project resulted in a number o~benefits of different types leading to eighty six benefits in all. It was found that these five c1asses of benefit covered the full range of benefits u~ed by the case organisations to justify their projects. The project managers described the methods used to measure these different types of benefit and their degree of satisfaction with the approaches used.

Type i.Increased Efficiency These benefits resulted in cost avoidance or reduction, either. directly (e.g .hardware, maintenance, inventory holdings) or through staff savings. They were the most easily quantified and respondents expressed a high level of satisfaction with their ability to measure the benefits. In some organisations, it was important that improved productivity led to real staff savings. In one public sector organisation which was expe~ting shortly to be SUbjected to commercial competition, productivity gains had to be realised in an actual reduction in staff numbers. In others management was confident that any free time created would be spent on higher value work. Another public sector organisation had a strategic objective that its staff spend more time on operational duties and, therefore, time saved on administrative activities was seen to be important although it would not lead to a reduction in staff numbers.

Type 2. increased Effectiveness Respondents found it more difficu1t to quantifY this type of benefit. In many cases, it was possible to identify how better information could lead to direct economic savings through considering how the information was used. For example:

- The information produced by a new Human Resources system would allow sick leave to be better contro11ed and therefore reduced.

- More accurate raw marterials inventory would reduce the cost of inventory. - Increased time available for negotiation would lead to better contract prices.

8

r

1 I

I I I

,I

(

, I

\ '

Other quantitative, but nonpeconomic, measures were considered less satisfactory.

Type 3. Added Value These benefits enhance the strategic position of the organisation and they are generally directed outside the organisation, towards its customers or its trading partners. With one exception, these strategic benefits were not quantified. The measures that would be used usually relate to factors external to the organisation (e.g. customer attitudes, market share) and are influenced by many other factors. For example, the improvement of customer service is considered by the banking organisations to be strategically important but measuring the impact of a system on this very sUbjective outcome is fraught ~ith difficulty. However, the difficulty in measuring this type of benefit was not necessarily a major problem; if the IT project was an essential part of an overall business strategy, measurement of the impact of the system alone became less important. Instead the question was posed - What was the most cost effective way of providing the IT function?

Type 4. Marketable Product Marketable products are sometimes identified as by-products of systems being developed. The information contained within a system could have an additional market or the system itself may become the marketable product. Both cases were encountered in this study and market prices were readily established for each. Thus this type of benefit was easily measured in economic terms.

Type 5. IT Infrastnlcture Investments in the corporate IT infrastructure were seen as enabling moves (i. e. as prerequisites for other developments) and a direct economic return was flot usually expected. Quantifying the benefit was not usually considered necessary; decisions had already been made by senior management that these investments in the IT infrastructure were required to aHow future developments to take place.

The nleasurement frameworks proposed in the literature The measurement frameworks in the literature provide only limited assistance to organisations. Although individual frameworks provide guidance on some aspects of evaluation, none provides organisations with a comprehensive guide for evaluating the full range of benefits. Measurement frameworks that were based on a single objective of a system failed to allow for the fact that many systems have more than one major objective and win produce a number of different types of benefit. Also approaches to evaluation have been suggested using classifications based on an IT view of systems, using terms such as transactional and informational. Business managers are taking a greater role in IT investment decisions and it is important that the tenninology used is, readily meaningful to them. Classifications based on the business purpose of the system (e.g. internal support or customer focused) are more easily understood than those based on IT perceptions and terminology. .

A Benefits and Measurement Framework Measures for the framework of benefits which overcome the problems described above are shown in Table 2.

9

Benefit Measure Type 1 & Increased Efficiency Economic, through cost avoidance or _ .. ~

reduction Type 2 - Increased Effectiveness Economic (considering how information is

used) Key Performance Indicators (ifvalue based management has been implemented)

Type 3 - Added Value Usually no direct measure for ITsystem alone Evaluate return from the entire business strategy

Type 4 - Marketable Product Economic, through establishment of market price

Type 5 - IT Infrastructure Usua11y no direct measure Corporate policy decision

Table 2. Benefits and Measurement Framework

These measures are based on a synthesis of those described in the theoretical literature (Berger, 1988; McDowel1 (reported in CaIson and McNurlin, 1989); Weill and Olson 1989; Keen, 1991) but they also reflect the experiences of the organisations in this study. They should be meaningful to organisations and provide a useful guide as to how all types of benefit may be measured.

6. Evaluation metbods Eleven evaluation methods which have been proposed in the literature as being suitable for evaluating the benefits derived from an IT project were assessed for their suitability for the justification and review of projects. The results are summarised in Table 3.

Only five methods were appropriate for evaluation at the justification stage. - Work Value Model (Sassone and Schwartz, 1986). Thls method, derived from

the field of operations research, recognises that one of the major benefits of inforination systems is the capacity to restructure work patterns and enable office workers to spend more time on higher value work. However, obtaining the extensive and detailed information about work patterns can be time . consuming and costly .

.:. Quantitative Cost Benefit Analysis. This method for justifying project expenditure encompasses a family of techniques in which the costs of a project are weighed against the benefits, usually also expressed in economic terms. Although there is general agreement in the literature that the application of a quantitative CBA to IT investments is inappropriate (e.g. Knutson and Nohin 1974, Keen 1981, Chismar and. Kriebel 1985, Blackler and Brown 1988, Loveman 1988), it remains a popular technique (Simmons, 1991).

- Excess Tangible Cost (Litecky 1981). Recognising the difficulty in analysing the costs and benefits of decision support systems, Litecky proposed that if the costs were greater than the tangible benefits, the excess or net cost be identified and managers can then weigh this against~he additional intangible benefits. This enables management to focus on these benefits and their equivalent economic value.

10

( )

( J

(

I .

IT PROJECT EVALUATION METHODS

Discipline Method Rererence Process ObJectlvel RelatlveJ Justlflcatlonl Benefits Included Comments on Subjective Absolute Review Implementation

Ei'conomics Production Frontier Chismar, Kriebel Compares erriciency oV 0 R R Economic Difficult to identify Analysis (inc. DEA) production units appropriate measures

Business Value linkage Banker & Kautrman structural modelling o~ 0 A R All that can Relationships poorly understood

--- Impact Analysis relationship between be quantified Complex models, unique inputs & outputs and modelled to application

Operations Res Work Value Model Sassone Estimates value of '0 A J Efficiency Data collection expensive changed work practice9 effectiveness

Cost Avoidance Edelman Compares operat'! costs 0 A R Ertlcfency Control groups required with & without system

Return on Management strassman Estimates contribution of 0 A R All that show up Isolation of appropriate value added by management In fin. statement costs is dirflcult

of organisation

Accounting Cost Benefit Analysis various Analysis of economic 0 A J+R Economic No major difficulties costs & benerrts only

SESAME (I.e. EPDP) lincoln Comparison with O+S A R Economic Subjective estimates or manual hypothetical equivalent costs system

Excess Tangible Cost Utecky Compares net cost with O+S A J Economic No major difficulties Intangible benefits Intangible

Information Economics Parker & Benson CSA enhanced to include O+S R J Economic No major difficulties other dimensions of valUle Intangible

(Risk)

Subjective Analytic Hierarchy Muralidhar, Santhanam Comparison of projects S R J All No major difficulties Assessment Process & Wilson on multiple criteria

User Information 'ves, Baroudi Surrogate measure orr S A R To user only No major difficulties Satisfaction & Olson value of system

Table 3. Summary or Evaluation Methods

- Information Economics (parker and Benson, 1988). Thls method uses a quantitative Cost Benefit Analysis to provide an analysis of the economic benefits and provides a structured and very detailed guide on how all other aspects of value should be rated. These are then combined into a single score. One organisation in the study was using ltiformation Economics but the economic and non-economic factors were not aggregated into a single score; they considered that this was not meaningful and would not assist the decision makers.

- Analytic Hierarchy Process (Muralidhar, Santhanam and Wilson, 1990). This method is intended to be used for ranking competing projects according to a number of criteria. It is based on pair.;.wise comparisons between projects for each criterion and the resulting matrices are mathematically manipulated to give an overall ranking of the projects under consideration. However, the process does not provide absolute indicators of value nor is the outcome expressed in business or financial terms.

In this study, it 'Yas found that organisations almost invariably used a quantitative CBA supplemented by a subjective judgement of other business factors to justify IT projects. This is really an extension ofLitecky's approach (replacing the simple net excess cost with a thorough cost benefit analysis) and Wing (1992) used the term "Business Case ll

to describe this approach. It is not prescriptive and the way in which the non­economic benefits are presented depends on the requirements of the organisation.

A growing number of organisations wish to evaluate whether the expected business benefits do eventuate and the review is based on the business outcomes predicted in the justification. A disadvantage with Information Economics is that the non-:economic benefits are not expressed in terms of business performance and so do not provide a benchmark for reviewing an investment after implementation.

If an organisation uses the generic Business Case approach to justifY projects, the framework of benefits and measures described above would assist in identifying and where appropriate quantifying the benefits.

7. A Complete Evaluation Model This study has revealed the complex nature of the evaluation of IT investments in organisations and that the investment decision is affected by many factors. Some are obvious (the justification method, the specific project objectives, the fonnal decision criteria) but others are less so. Organisational attitudes towards effectiveness and evaluation, the persuasive power of individuals, the political organisational environment and the structural positioning of the IT department all playa part. The evaluation model in Figure 1 shows how these many factors impact the final investment decision.

This model shows how organisational context (industry sector, organisational culture,and management style) was found to influence attitudes towards effectiveness and evaluation and the impact of other influences (personal, political and structural) on decision making. The method used to justify projects was determined by these attitudes towards effectiveness and evaluation. The form of the actual project proposal

12

depended on the justification method adopted, the specific objectives of the project under consideration anc;l the way different types of benefit were measured. The actual investment decision was detennined by the fit between the project proposal and the fonnal decision criteria, but personal, political and structural factors also influenced the decision.

Orgailisational

context

Figure 1. A Complete Evaluation Model

Measurement

framework

Specific project

objectives

The cornponents of this model and the inter~relationshlps uncovered in this study are described in more detail below.

Organisational Context It quickly became evident that there were great differences between organisations in their attitud~s towards effectiveness, measurement and evaluation. These distinctions in attitudes were related to the industry sector, the culture and the management style of the organisation. Industry sector - In the banking organisations, IT was seen as a costly and strategic resource and measurement of the effectiveness of these investments was important. Increasing staff productivity and improving customer seIVice were major objectives and these priorities were reflected in the criteria used to evaluate investment proposals. In the manufacturing/process organisations, IT was seen as providing support to the business rather than as a strategic resource. In the public sector, attitudes towards· evaluation were rel~ted to the extent to which the organisation was exposed to private competition. The expectation of facing such competition in the near future was having a considerable impact on attitudes towards obtaining value from IT investments. Organisational culture - When interviewing the IS and business managers, significant differences between the cultures of the organisations in the study were evident. Two' organisations in particular stood out as having a "unitary culture" (Legge, 1984); in both cases the organisations' objectives were well understood and all members were strongly committed to them. Deal and Kennedy (1988) used the term "strong culture" to describe such an environment. There V!as little emphasis on formal procedures and ' this was reflected in their approach to project justification and review. However, in

13

some other organisations there was not the same strong feeling of a common purpose. There was a concern that some business managers placed their own or departmental interests above those of the organisation and influenced IT investment decisions so that the outcome would not necessarily provide the best value from an organisational perspective. Such a cu1ture was described by Legge (1984) as "pluralist". Management style - Organisations which had adopted value based management and total quality management approaches were also particularly concerned about measuring the effectiveness of IT investments as measurement is integral to these methods of management. The establishment of Key' Value Drivers and their Key Performance Indicators, which are essential components of value based management, (Rappaport, 1986) provides the opportunity for demonstrating that an IT Investment impacts an indicator which has already been proven to affect business performance.

Thus there is not one "best" way for organisations to justifY and review their investments; industry sector, organisational culture and management approach all influence the choice of a suitable approach to evaluation for a particular organisation.

Attitudes towards Effectiveness and Evaluation All but one organisation (which is in the public sector, with no prospect of having to face private competition) were concerned about the value being derived from their investments in IT and considered the evaluation of the effectiveness of IT investments to be an important issue. The effectiveness of IT investments was generally not judged solely on economic terms; support for business strategy and objectives was also important. Measurement of effectiveness was important when a project was being proposed but attitudes towards measurement in the context of a post implementation review were more ambivalent. All organisations sought a quantitative indicator of economic effectiveness but the evaluation of the non-economic aspects of value depended on organisational factors. Those who had implemented value based management considered measurement to be very important and use Key Performance Indicators as measures of organisational value. Other organisations could only use qualitative descriptors for the less tangible benefits. It was evident that attitudes towards the measurement of the effectiveness of IT were changing. In this study five out of the nine organisations had either recently introduced new justification procedures or were planning to do so.

Justification Method There is not one "right II or IIbest" method for justifYing IT investments. Five organisations each considered that their justification procedures were successful, but the approaches used differed greatly. The organisations held widely differing attitudes towards effectiveness and evaluation and therefore also had different requirements of a justification process. However, the following general. conclusions can be drawn.

Organisations required a justification method that provided an absolute (rather than comparative) measure of a project's value. It was accepted that value had a number of C:limensions and therefore a number of measures were needed. Although projects that made a strategic contribution were identified and their strategic nature influenced the decision making, organisations still wanted to use one justification approach for assessing all projects. Therefore, the ju'stification method must be flexible enough to cater for different types of project. A few' organisations saw post implementation reviews as providing an opportunity to ensure that the objectives of the project were

14

achieved; they required performance measures from the justification process that could be later used as benchmarks after implementation.

Very few of the justification methods described in the literature (see Table 3) satisfy these requirements. In particular, most methods are directed towards a single type of benefit; rather than providing a genera) approach for project evaluation which encompassed all types of benefit.

Information Economics has been widely reviewed and one of the case organisations had implemented a simplified version, but for many organisations this method would not be satisfactory:

- It does not use absolute measures that are meaningful in a business context. Projects are rated on many factors but apart from a measure of economic performance, objectiye measures of business performance are not used. Projects are subjectively assigned scores relative to given scales. It is not possible or meaningful to use these scores to review a project after implemef)tation to detennine whether the expected benefits had been achieved.

- Information Economics is a highly structured process and is only likely to be suitable for organisations that themselves have a'high degree of structure. and embrace formalised processes.

It was found that all but one organisation used a justification approach which was based on a quantitative cost benefit analysis supplemented by qualitative or quantitative descriptors of the less tangible -benefits. This generic approach was tenned the "Business Case" method by Wing (1992). There is no generally accepted mechanism used to combine the different dimensions of value and organisations employed methods that were appropriate to their culture and current priorities. Organisations who had adopted value based management used Key Performance Indicators as measures of the non-economic factors~ other organisations for which measurement was less important used qualitative descriptors. For the organisations which recognised the value to be obtained from perfonlung post implementation reviews, business performance measures that can be verified after implementation were used in the justification. The decision making process was detennined by the organisational culture; it copld be very structured and formalised or informal, depending on consultation and consensus.

Measurement Framework If organisations adopt the generic Business Case approach to justifying projects, guidance is still required on how the specific 'benefits can be measured. The set of measures developed in this research provides such a framework. It covers all types of benefits that are likely to be used in the justification of a project and it recognises that although a project will have a main objective; the actual justification will usually rest on an integrated perception of the value of a number of different types of benefit.

Specific Project Objectives Factors that are specific to the project wer~ also important. Some organisations categorised systems according to the type of contribution made to the business, usually classifying them as either ,p~oviding internal support or supporting a business strategy. These organisations recognised that these strategic systems were less likely to show a direct economic return and therefore judged the project proposal differently. When a . ~roject was an integral part of an overall business strategy, the wisdom of evaluating

15

such a project in isolation from the other related business activities was frequently questioned. This supports the socio-technical view of systems, which perceives technology as an integral part of organised social systems) not just a contextual variable (Mayntz, 1984).

Each project also created its own challenges in defining and estimating the expected benefits. In some cases, it was almost impossible to estimate in any quantitative way the impact of a system. Office automation projects and some decision support systems presented these problems) and a pilot development and implementation provided an alternative mechanism for demonstrating the benefits.

Proje.ct Proposal The form of the project proposal was detennined by the justification method used, the specific objectives of the system and the framework of measures used to describe the expected benefits. Almost all organisations provided guidelines on how project proposals should be presented and these provided flexibility to cater for different types of projects. Most project proposals adhered to these guidelines.

There was great variation in the extent and type of guidance provided. Guidelines which had been specifically developed for the justification of IT investments reflected most closely the organisation's decision criteria. Those that were part of the IS development methodology tended to focus on system issues rather than on business value; some organisations used the standard corporate capital expenditure request procedures but it was felt that these were inappropriate for most IT investments.

Fornlai Decision Criteria Organisations generally sougQt a quantified indicator of economic effectiveness and some evidence of support for other organisational objectives. The relative importance of these different types of indicator depended on organisational factors and on the type . of project. ,Decision makers did not expect a single measure of value.

Similar economic indicators were used (net present value, internal rate of return or return on investment) but mechanisms for assessing contribution to other organisational objectives varied. Organisations which had implemented value based management already had a means of demonstrating the impact of an IT investment on organisational performance. Other organisations tended to rely on qualitative descriptors; those with a strong "unitary" culture took a more "visionary" approach towards the support of corporate goals. Hurdle rates were often applied to the economic indicators although these tended to be flexible, depending on the current situation in the organisation and sometimes varying across different parts of the organisation.

Other Influences Although organisations had formal criteria for evaluating investment proposals) other factors also played an important part in the decision making process. The persuasive power of individuals, the political organisational environment and the structural positioning of the IS department were al1 cited as affecting the outcome of investment decisions. In organisations ,with a unitary culture, personal influence was likely to have a positive impact- on the deCision as staff members placed the organisation's objectives above their own. But many organisations did not operate in this way and there was a ri.sk that one powerful stakeholder will use his or her influence to produce an outcome

16

j

r

I )

) \

that was not in the best interests of the organisation. The risk was particularly high if the IS department reported t~ another functional area, e.g. finance; in this case IS investment was often biased tovJards financial systems which may not have been making optimal use of the ·organisation's resources.

Other forms of personal influence can be more positive. Pre-selling is often necessary if projects do not have a convincing economic case and ,a strong project champion increases the chance ofa project being approved. The importance of this personal influence could be of concern, but it was also argued that a strong project champion was required to see through the organisational change that is often required and therefore the benefits of such a project are more ]~kely to be achieved.

Thus personal, political and structural factors influenced the formal decision making in both positive and negative ways. In a unitary organisation, their impact on investment decisions was more likely to be positive.

Investment Decision Most organisations had a fonnal planning cycle to coordinate and prioritise their IT developments for the forthcoming year; senior IS and business managers reviewed the proposals and prioritised them in some way. Although .organisations had developed evaluation criteria, most organisations in this study had no formal mechanisms for combining the economic benefits with other qualitative benefits in order to rank their overall worth to the organisation. In this study the exceptions were the one organisation that used Information Economics and another organisation that had developed an algorithm that combined economic benefit (in terms of payback period) with strategic contribution. The organisations that itnplemented Information Economics refiained from taking the final step of combining economic and qualitative factors in a single score. It was considered more meaningful for managers to be presented with scores for the two dimensions of economic and qualitative factors and to apply their judgement to the relative importance of each. But whatever'mecharusm was used to reach the final decision, the involvement of business and IS managers in an open discussion of the economic and strategic merits of different investments increased the likelihood of the right systems being developed.

Review The post implementation review of IT projects was generally considered to be much less important than the irutial evaluation that was part of the project justification. In the past, these reviews have tended to focus on the development process and, . therefore, were not seen as worthwhile by the business managers (Kumar, 1990). Also the difficulty in isolating the impact of an IT project from that of the other related business irutiatives raised the question of the wisdom of evaluating many IT proje~ts in . isolation.

The organisational context was an important factor in explaining attitudes to review. ~ organisations which had a strong unitary culture, managers were already confident that good value was being obtained from their IT investments and studies to prove it were not considered necessary. In other cases, particularly where the business environment was changing and becoming more difficult, reviews were organised in response to senior management's concern that the organisation's resources had not been used to best effect.

17

This study revealed that reviews can also be used to ensure that the expected business benefits are delivered. If the full benefits of an IT investment are to be achieved, organisational change is required. In four organisations, regular reviews were being used to provide feedback to the business managers on relevant business performance indicators. This information encouraged managers to carry through the necessary organisational changes. These organisations had recognised that lithe pursuit of organisational enhancement through the use of IT demands organisational change" (Eason, 1988, p.23). A good set of measures is required ifreviews are to be used in this proactive way and organisations who have adopted value based management will already have developed relevant measures of business performance.

8. Conclusion This study addressed the problem of developing meaningful measures of the business benefit of IT investments and focused on evaluation within an organisational context.

The benefits and measurement framework (summarised in Table 2) provides guidance to the large number of organisations which use the generic Business Case approach. The five types of benefit cover all benefits that are likely to result from an IT investment and a set of measures, based on the literature and the experience of organisations, is proposed. This set of measures would assist in identifying and, where appropriate, quantifying the benefits. The same measures could also be used to review the project after implementation and if monitored regularly as part of a benefits management programme, provide feedback to the business managers to ensure the expected benefits are achieved: Taking this proactive step increases the likelihood of investments in IT resulting in enhanced business performance.

The model of the evaluation process (see Figure 1) demonstrates the complex nature of the evaluation of IT investments in organisations. Although much has been written on the subject of evaluation, it is often not clear where the particular contribution fits into the whole task of project justification and evaluation. This study has revealed the importance of the organisational context and the model of the evaluation process shows how the different aspects of evaluation are linked.

9. Implications for Future Research This research work has a number of implications for future research, both for information systems research generally and for research into the evaluation of IT investments specifically.

- It has demonstrated the value of an interpretive approach. The major part of information systems research has been based on the positivist paradigm but this research provides an example of how much information can be elicited from a detailed study of a small number of organisations. The knowledge gained from this study could not have been derived from the more common approach which depends upon a survey of a large number of organisations.

- It has also demonstrated the importance of using the knowledge and experience of the people working in organisadons. Research into the practical problems relating the application of IT in organisations benefits from the inc1usion of this

18

I ,

body of knowledge; it is complementary to the more traditional theoretical body of knowledge but it is just as essential.

This research has highlighted the importance of the organis'ationa] context when considering the evaluation of IT investments. Researchers need to be aware that factors such as industry sector, organisational culture and management approach influence attitudes towards effectiveness, evaluation and measurement and hence evaluation needs. Previous work in the field has not considered these issues.

- The framework of benefits and measures developed in this study provides researchers and practitioners with a comprehensive framework that includes a11 types of benefits likely to result from an IT investment and with suggested measures for each type of benefit.

- The Complete Evaluation Model provides a clear depiction of the evaluation process in organisations; it shows the relationships between the many components in this complex process and in particular shows how the organisational factors influence the IT investment decision. This model presents many opportunities for further work on these organisational influences.

10. Acknowledgments I would like to express my gratitude to the organisations who agreed to participate in this study and to the staff who "gave their time to be interviewed. Without their generous assistance t this study could not have been done.

Tills "lork was completed with the assistance of a grant from the Departnlent of Employment, Education and Training.

19

BmLIOGRAPHY

Ahituv N. (1989), Assessing the Value of Infonnation: Problems and Approachs, Proceedings of the 10th International Conjerence on Injonnation Systems, Boston, Mass.

Baily M.N. and Chakrabarti A. (1988), Innovation an4 the Productivity Crisis, Brookings Institution, Washington, DC

Barna A., Kriebel C.H., Mukhopadhayay T (1989), A New Approach to Measuring the Business Value of Injormation Technologies, Working paper #88-89-34, Graduate School of Industrial Administ~tion, Carnegie Mellon University, Pittsburgh, P A

Berger P. (1988), Selecting Enterprise Level Measures of IT Value, Measuring Business Value of IT, ICIT Research Study Team #2, ICIT Press, 1988

Blaclder F. and Brown C. (1988), Theory and Practice in Evaluation: The Case of the New ITs, Information System Assessment: Issues and Challenges, N. Bjorn-Andersen and G.B. Davies (eds.), North Holland

Blaikie N.H.W. (1980) The Interpretive Alternative to Positivistic Sociology, Reactions to Positivism, E Veit-Brause et a1. (eds), Warun Ponds, Deakin University Press

Broadbent M., Lloyd P., Hansell P.and Dampney C.N.G. (1992), Roles, Responsibilities and Requirements for Managing IS in the 1990s, International J oumal of Injormation Management, March

Buss M.D.I. (1983), How to Rank Computer Projects, Harvard Business Review, lan-Feb~ 1983

\, Carlson W.M. and McNurlin B.C. (1989)~ Measuring the Value of I.S., IS " Analyser Special Repon, 1989.

Chismar W.,G. and Kriebel C.H. (1985), A Method for Assessing the Economic Impact of IS Technology on Organisations, Proceedings of the 6th International Conjerence on IS, Indiannapolis, Dec ·1985

Datamation (1991), Datamation Spending Survey, April 15, 1991. Deal T. and KeIll1edy A. (1988), The Rites and Rituals of Corporate Life, Penguin

Books, London, 1988 Bason K. (1988), In/ormation Technology and Organisational 'Change, Taylor and

, Francis, London, 1988 Ginzberg M.J. and Zmud R.W. (1988), Evolving Criteria for IS Asessment,

Information System Assessment: Issues and Challenges, N.Bjom-Andersen and G.B. Davies (eds.), north Holland, 1988

Glaser B.G. and Strauss A.L. (1967), The Discovery of Grounded Theory: Strategiesfor Qualitative Research, Aldine Publishing Co., N.Y.

\Hamilton S. and Chervany N.L. (1981), Evaluating IS Effectiveness - Part 1: Comparing Evaluation Approaches,Ma1U2gement In/ormation Systems Quarterly, Sept., 1981

Hirschheim R. A. (1985), Infonnation Systems Epistemo10gy:An Historical Perspective, Research Methods in Injonnation Systems, E. Mumford et ale (eds.), Elsivier Science Publisher~ B.V., North-Holland

20

\ ~ I

Kauffman R.J. and Kriebel C.H. (1988), Measuring and Modelling the Business Value of IT, Measuring Business Value of IT, ICIT Research Study Team #2~ ICIT Press, 1988

Kauffman R.J. and Weill P. (1989), Methods for Evaluating the Performance Effects of Investments in IT, Graduate School of Management, University of Melbourne, Working Paper # 13, 1989

Keen P. G. W. (1975), Computer Based Decision Aids: The Evaluation Problem, . Sloan Management Review, Spring 1975.

Keen P.G.W. (19~1), Value Analysis: Justifying DSS, Management Information Systems Quanerly, March, 1981

Keen P.G.W. (1991), Shaping the Future: Business Design Through IT, Harvard Business School Press, 1991

Knutsen K.E. and Nolan R.L. (1974), On Cost/Benefit of Computer-Based Systems, Managing the Data Resource Function, Ed. by R.L. Nolan, West Publishing Co, 1974, pp277-292.

" Kumar K. (1990), Post Implementation Evaluation of Computer Based Information , Systems .. Current Practices, Communications of the ACM, Feb 1990, Vol

33, No 2 Legge K. (1984), Evaluating Planned Organisational Change, Academic Press,

(Harcourt Brace, Jovanovich, Publishers), London, 1984 Litecky C.R. (1981), Intangibles in Cost Benefit Analysis, Journal of Systems

Management, February 1981. \ Loveman G. W. (1988), An Assessment of the Productivity Impact of Information , Technologies, Working Paper, Management in the 1990s, Massachusetts

Institute of Technology, Sloan School of Management Marcha:rd D.A, and Horton Jnr. P.W, (1986)11 In/otrends.· Profiting from your

lnforrnation Resource; John Wiley and Sons) NY, 1986 Markus M.L. and Robey D. (1988), Infonnation Technology and Organisational

Change: Causal Structure in Theory and Research, Management Science, (34:5), May) pp. 563-598

Mayntz R. (1984), Infonnation Systems and Organisational Effectiveness: The Sociological Perspective) in Beyond Productivity: IS Development for Organisational Effectiveness, ed. Th M. A. Bemelmans, Elsevier Science, Publishers B. V. North Holland

Muralidhar K.,Santhanam R. and Wilson R.L. (1990), Using the Analytic Hierarchy Process for Infonnation System Project Selection, Infonnation and Management 18(1990), pp87 -95

Parker M.M. and Benson R.J. (1988), Injormation Economics: Linking Business Peifonnance to Injonnation Technology, Prentice Hall, Englewood Cliffs, NJ, 1988.

Roach S.S. (1988), Technology and the Services Sector: The Hidden Competitive Challenge, Technoloy Forecasting and Social Change, Vol 34, No 4, Dec 1988, pp387-403.

Sassone P.G. and Schwartz A.P. (1986), Cost Justifying Office Automation, Datamation, February 15 1986.

Simmons P.M. (1991), Measuring the Business Value of IS Investments: A Pilot Survey of Industry Attitudes and Practices, Staff Paper #80, Faculty of Business, Swinburne Institute of Technology, Melbourne

21

Strassman P. (1988), Management Productivity as an IT Measure, Measuring Business Value of IT, ICIT Research Study Team #2, ICIT Press, 1988

Strassman P. (1990), The Business Value-o/Computers, The Information Economics Press, New Canaan, CT, 1990

Weill P. (1989), The Relationship Between Investment in lnfonnation Technology and Firm Performance in the Manufacturing Sector, Working Paper No. 18, University of Melbourne, Graduate School of Management

Weill P. and Olson M. (1989), Managing Investment in IT: Mini Case Examples and Implications, Management Information Systems Quarterly, March, 1989

Wing P.J. (1991) The Relationship between an Organisation's Position on the IS Life Cycle' and the Approach Adopted to Justify Proposed Investments in IT, Unpublished thesis fo~ Master of Economics, University of New England, December, 1991

22

)

)

I "

No 7

, No 8

No 9

No 10

No 11

No 12

No 13

No 14

No 15

No 16

. SWINBURNE INSTITUTE OF TECHNOLOGY

FACULTY OF BUSINESS

STAFF PAPERS PUBLISHED TO DATE

1981

1981

1981

1981

1982

1984

1984

1984

1984

1984

tA Note on Customs. Unions Theory : The Viner Controversy RIpt by D J Thomas

tDisequilibrium . and the Expectations­Augmented Phillips Curve' by M Grant

fA View of Ideological Pressures in the Context of Managerial Power' by M Brown

'Short Term Prediction of Student Numbers in the Victorian Secondary Education System' by M G Nicholls

'The Lega1 Protection of Geographical Trade Names: Prognosis for a Case of Champagne' by B R Clarke

'Corporate Planning Practice in Major American and Australian Manufacturing Companies' by N Capon, Christodoulou? J U Farley and J Hulbert

'A Modified Markovian -Direct Control Model in Fixed Time Incorporating a New Objective Function Specification' by M G Nicholls

'Government Intervention in the Labour Market - A Case Study of the Referral and Placement Activity of the Commonwealth Employment Service in a Major Metropolitan Areal by J B Wielgosz

'Big Business in the US and Australia: A Comparative Study' by N Capon, C Christodoulou, J U Farley and J M Hulbert

'Modelling the Demand for Tertiary Education - An Exp]oratory Analysis based on a MO,dified Human-capita] Approach' by M G NichoJ]s

No 17- 1984

No 18 1984

No 19 1984

No 20 1985

No 21 1985

No 22 1985

No 23 1985

No 24 1985

No 25 1985

No 26 1985

No 27 1986

'Pormal Corporate Planning Practices of Major Australian Manufacturing .. -Companies' by C Christodoulou

'The Australian Short Run Demand for Money Function - Further Theoretical Considerations and Empirical Evidence using Bayesian Techniques' by E J Wilson

'Alternative Job Search and Job Finding Methods : Their Influence on Duration of Job Search and Job Satisfaction' by J B Wielgosz and S Carpenter

'A Comprehensive Study of Strategic Planning in Australian Subsidiary and Non­Subsidiary Companies' by C Christodoulou and P T Fitzroy

'Towards an Optimal Taxation Structure ln Australiat by D Thomas

'A Suggested Theoretical Basis for the Interpretation of the Effects of Income on the Demand for Tertiary Education' by M G Nicholls

'Austrian Economics and Australian Patents' by B Oakman

'Ensuring a Future for your Organisation t by C Christodoulou

'The Long Search : A Pursuit of Organizational Understanding from the Perspective of "System" Thinkers' by M Brown

'Managing the Introduction of New Technology' by· J Newton

'Positive Economic Analysis and the Task of State Enterprise Efficiency and Contror by P Xavier

\

I '

No 28 1986

No 29 1986

No 30 1986

No 31 1986

No 32 1987

No 33 1987

"No 34 1987

No 35 1987

No 36 1987

No 37 1987

No 38 1987

'Profitability of Horizontal Takeovers in the Australian Industrial Equity Market : 1978 to 19821 by M A Johns and N A Sinc1air

'A Comparative Examination of Subsidiary and Non-Subsidiary Strategies' by C Christodoulou

'Solving Linearly, Constrained Nonlinear Programming Problems' by F Ghotb

'An Economic Appraisal of Recent Reforms in Public Enterprise Pricing Policy in Victoria' by P Xavier

'Australian Manufacturing Companies and Academic Institutions : A Comparative Analysis of Strategic Planning· by N H Kelly and R N Shaw

'Centralisation of Information and Exchange with Special Reference to the South Australian Winegrape Industry' by C Hunt~ P Tiernan and E Wilson

'The Impact of Home Office Culture on Subsidiary Strategic Planning' by C Christodoulou

'A Comparison between Guarantees Standby Credits and Performance Bonds by A Johns

'The Effects of Uncertainty and Incomplete Information in a Foreign Exchange Market Subject to Noisy Rational Expectations' by E. J Wilson

'Inflation Accounting for Australian Public Enterprises - Economic Rationale and Financial Implications' by B Graham and P Xavier

'Financial Targets and Dividend Requirements for Commonwealth Government Business Enterprises - Are they Appropriate and how should they be Determined and Measured?' by B Graham and P Xavier

No 39 1988

No 40 1988

No 41 1988

No 42 1988

No 43 1988

No 44 1988

No 45 1988

No 46 1988

No 47 1988

. No 48 1989

No 49 1989

No 50 1989

'Whither Co-operative Federalism? An Analysis of the Commonwealth Government's Plan for Reform of Companies, Securities and Futures Legislation' by P J Pascoe

'An Analysis of the Pricing of Section 23 Expert Reports' by A Johns

'Food Laws : Reviewing the Regulatory Framework' by S Edmonds

'Share Prices and Divestiture' by J Barker

'Marketing Education in Malaysia : ImpJications for Australian Tertiary Institutions' by C T Selvarajah

'Forecasting the Demand for Tertiary Education using Econometric and Markovian Models' by M G Nicholls

'Power Pays - An Analysis of the Relationship between Managerial Power and Interdepartmental Relations' by M Brown

'Workers Participation - Concepts, Issues and Prospects : An Australian Perspective' by C T Selvarajah and S Petzall

'Performance Indicators for Telecommunications and Price-Cap Regulation' by P Xavier

'Multinational Enterprises and Host Nation Response' by C Selvarajah

'Women in Management' by C T Selvarajah and S Petzall

'The Cultural, Political and Legal Environment of International Business' by C T Selvarajah

-~ ", ...

. 1

I ' \

) ,

J

No 51 1989

No 52 1989

No 53 1989

No 54 1989

No 55 1989

No 56 1989

No 57 1989

No 58 1989

No 59 1989

No 60 1989

No 61 1989

No 62 1989

No 63 1989

'Australia's Human Capital and Labour Markets - Their Role in Achieving a More Competitive Economy' by M S De Lorenzo

fA Comparative Analysis of the Strategy and Structure of United States and Australian Corporations' by N Capon, C Christodoulou, J U Farley and J M Hulbert

'Strategies during Market Transition - A Study of Detergent Marketing in Australia 1930s to 1960s' by D ChIng

'Some Legal and Economic Aspects of Third World Debt' by L Kloot

'Access to Corporate Documents - Section 265B of The ,Code' by S KapnouUas

'Minding 'Everybody's Business : Performance Indicators for Australia Post' by P Xavier

'Some Legal Aspects of Electronic Funds Transfer! by L Kloot-

'Japanese Worth Ethics' by C T Selvarajah and S Petzall

'StrategY7 Policy -and Operational Planning' by C T Selvarajah

'Technology' b:y C; T Selvarajah

'The Development of a National EFT System Network in Australia' by A Richardson

'International Marketing Research : A Review' by D ChIng

'An Exploration of the Relationship between Training System Effectiveness and the Environmental 'Variables' by C T Selvarajah

No 88 1992

No 89 1992

No ,gO 1992

No 91 1992

No 92 1992

No '93 1992

No 94

No 95 1993

No 96 1993

No 97 1993

No 98 1993'

No 99 1'993

No 100 1993

'Control of Information Technology Costs by Alloca,ting Costs to Users' by L Kloot

'Seeking Effective Management Education' by C Christodoulou

'Fine Print in Contracts : Can You Rely on it?' by B Clarke and S Kapn oullas

'The Development of a Mathematical Model . for, the 'Optimisation of the Operations of a

National Glass Manufacturer' by M Nicholls

'The Development of a Model of an Ingot Mill in an Aluminium Smelter' by M Nicholls

'Software and Computer Services: Some Sectoral and International Issues' by L Arossa

'An Outline of Indonesian Environmental Law" by P Holland

'A Stitch in Time: Environment Impact Analysis Legislation in Indonesia' by P 'Holland

'Paradise Sustained: The Bali Substainable Development Projecf by P Holland

'The Best Laid Plans: Administration and Enforcement of the Bali Sustainable Development Strategy' by P Holland ,.

'No Higher Than a Coconut Tree: Contro]s on , Tourist Development on the Bali Foreshore'

by P Holland

'Product Portfolio Models: Conceptualisation and Implementation Problems' by A Koch

'Teaching' Organisation Behaviour to Eastern European Managers: A Process of Adaptation to Change' by S Long

! ;

! ' I

i I .

1 . I

J / I

!

No 64 1990

No 65 1990

No 66 1990

No 67 1990

No 68 1990

No 69 1990

No 70 1990

No 71 1990

No 72 1990

No 73 1990

No 74 1991

No 75 1991

'Interest Rates, Their Changes and the Australian All Ordinaries Index - An Empirical Results (with some theoretical justificationy by D L Dowe

'Current Monetary Policy : A Review of Current Literature' by M L Freebairn

'Performance Indicators for Public Telecommunications Operators : Will They Serve to Improve Performance?' by P Xavier

'Mosaic mac-b : A Comprehensive Framework for Marketing Planning' by D ChIng

'The Death of the Reasonable Man' by B R Clarke

'The Implications of the Development of a National EFT System Network in Australia' by L Kloot and A Richardson

'Victoria's Government Business Enterprise . Debt: The Scope for Reduction' by P Xavier

'Training System Effectiveness' by C T Selvarajah

'Relationship of Size of Organisation to Training System Effectiveness' by C T Selvarajah

'The Relationship of the Structural VariabJe, Industry Type, to Training System Performance Criteria' by C T Selvarajah

'Preliminary Study of Export Success in the Australian Scientific Instruments Industry' by Ling Ping Wang

'Management Accounting System,s : Can They Exist in Local Government in Victoria?' by L Kloot

No 76 1991

No 77 1991

No 78 1991

No 79 1991

No 80 1991

No 81 1991

No 82 1992

No 83 1992

No 84 1992

No 85 1992

No 86 1992

No 87 1992

'A Content Analysis of Advertisements in the Chinese Times, 1902-1914' by D Ch'ng

'Endogenous Money Supply and Monetary Policy : An Interpretation of Recent Literature' by G Messinis

'Improving Accounting Research Performance Based upon Consulting Activities' by B McDonald and H Paterson

'A Critical Analysis of DB2's Support for Automated Integrity Check' by G Menon

'Measuring the Business Value of IS Investments: A Pilot Survey of Industry Attitudes and Practices' by P Simmons

'Present Status of Marketing Research in Malaysia' by D ChIng .

'Australian Government Policy on Higher Education Impact on Accounting Education' by J Wells

'Employers and the Public Employment Service A Case for Mandatory Vacancy Notification' by J Wielgosz

'The Community Service Obligations of Victorian Government Business Enterprises' by P Xavier

'Measuring Australia Post's Economic Performance' by P Xavier

'Management Training Employment Outcomes and Eq~aJ Opportunities Towards the Year 2000' by B Lasky

'The Exercise of Intellectual Property Rights and A buse of Dominant Market Position in the European Economic Community' by S Edmonds

No 88 1992

No 89 1992

No 90 1992

No 91 1992

No 92 1992

No 93 1992

No 94 1993

No 95 1993

No 96 1993

No 97 1993

No 98 1993

No 99 1993

No 100 1993

'Control of Information Technology Costs by Allocating Costs to Users' by L Kloot

'Seeking Effective Manag.ement Education' by C Christodoulou

'Fine Print in Contracts : Can You Rely on it?' by B Clarke and S Kapnoullas

'The Development of a Mathematical Model for - the Optimisation of the Operations of a National Glass Manufacturer' by M Nicholls

'The Development of a Model of an Ingot Mill in an Aluminium Smelter' by M Nicholls

'Software and Computer Services: Some Sectoral and International Issues' by L Arossa

'An Outline of Indonesian Environmental Law' by P Holland

'A Stitch in Time: Environment Impact Analysis Legislation in Indonesia' by P Holland

'Paradise Sustained: The Bali Substainable Development Project' by P Holland

'The Best Laid P1ans: Administration and Enforcement of the Bali Sustainable Development Strategy" by P Holland

'No Higher Than a Coconut Tree: Controls on Tourist Development on the Bali Foreshore' by P Holland

'Product Portfolio Models: Conceptualisation and Implementation Problems· by A Koch

'Teaching' Organisation Behaviour to Eastern European Managers: A Process of Adaptation to Change' by S Long

D3001078987

( ..... .

" ,

i

\ J i \ '

No 101 1993 'Evaluating LT. Investments: An Organisational Perspective' by P Simmons