Geog 469GIS Workshop

Evaluating Implementations of GIS

Outline

1. How do we evaluate a GIS (project) implementation?

2. What is a successful GIS implementation? How do we define the “success” of GIS implementation?

3. What are some additional perspectives on GIS implementation?

1. How do we evaluate a GIS (project) implementation?

Why is “project” in parentheses?

Evaluating GIS

• Ad-hoc method– If time is limited, evaluate project based on your

insight and previous interaction with clients

• Benefit-cost analysis– Compare benefit to cost to determine whether the

project is financially justifiable (at any stage) to move forward

• Usability testing– Test whether clients’ needs have been met; use

prepared questionnaire, interview, observation

Ad-hoc

• What does your experience tell you?– Data– Software– Hardware– Personnel– Timing of Procedures and Institutional

Arrangements– Motivations

Benefit-cost analysis• Enumerate all possible costs spent on this project; classify them into

one-time cost (data, hw, sw) and operating cost (maintenance, overhead, salary, training)– Cost of leasing workstation for 10 weeks can be obtained from

estimates • Enumerate all possible benefits; classify them into tangible (cost

reduction, efficiency) and intangible one (improved decision making, better public image, equity); assign monetary value wherever possible– Benefit of using GIS for making map can be calculated from saved time

multiplied by salary rate (= actual time spent - time spent if manual method is used)

• Discounting: If B & C estimates are based on different time period, discount them given inflation rate

• BCR: Calculate the ratio of benefit to cost to determine whether the ratio is over 1 (i.e. economically justifiable)

• Time: If possible, project the cost/benefit into the foreseeable future; draw the curve; determine payback period

Usability testing• Develop test objectives

– What am I testing? e.g. effectiveness of map design, accuracy of data geocoded

• Acquire participants– Sampled from potential end users– You can screen them through pre-survey form (ask their age, education

level, gender, occupation, and so on wherever appropriate)• Collect qualitative and quantitative performance and preference

measurements– Questionnaires

• Which map do you prefer? Does prepared map design work?– Task scenario if applicable

• Describe the context for use of the information product• Recommend improvements to the design of information product• Read materials on usability testing at http://www.lib.washington.edu

/usability/

Use an Evaluation to…

• Draw conclusions– About significance of what you found– About the effectiveness of your methods– About whether needs were met– About whether research question was

answered

• Provide recommendations– For improving the completed work – For additional work

2. What is a successful GIS implementation? How can we define the “success” of GIS implementation?

Elements of GIS project success and failure

Activity Success Failure

Planning Rigorous “Run and gun”

Requirements Focused Diffused

Appraisal of effort Realistic Unrealistic

Staffing Dedicated High turnover

Funding Adequate Inadequate

Time estimates Thoughtful Rushed or prolonged

Expectations Balanced Exaggerated

Validity of implementation success

• Technical validity– The system to be implemented works

• no bug, data seems clean

• Organizational validity– Congruence between the organization and the system to be

implemented• The system is accepted and used by the organizational members

• Organizational effectiveness– Improvement in efficiency and effectiveness

• Improved decision making– Even perceived as a process of organizational change

• Some GIS changes the way the organization does their business

From Obermeyer and Pinto 1995

What are the criteria that define the success of GIS implementation?

• It will be useful to address this question based on two different perspectives

• Content model– GIS project is viewed as collection of elements

• Process model– GIS project is seen as an activity composed of phases

GIS Implementation: Content Models

• Focus on key (critical) success factors

• Technical aspect– Competent technical support– Staff’s competence in GIS knowledge

• Managerial aspect– Clearly defined goals– Sufficient resource allocation– Top-management support– Implementation schedule– Adequate communication channels– Feedback capabilities

GIS Implementation: Process Models

• Focus on key phases in GIS implementation process

• Initiation– Planning phase of the implementation process – Related to “strategies”

• Implementation– Actual process (or the action) of the implementation– Related to “tactics”

Conceiving GIS implementation as the two-stage “process” helps us look at dynamic nature of GIS in the context of strategic goals of organizations

Strategy/tactics effectiveness matrix

High acceptance, misuse

High probability of implementation success

High probability of implementation failure

Low acceptance, low use

Effectiveness of strategy

Effectiveness of

tactics

highlow

low high

From Schultz, Slevin, and Pinto (1987)

Successful GIS should be both strategically and tactically effective

• Success in the short-term might become failure in the medium- to long-term

• Early false start does not necessarily imply that the project is destined to fail

• Bear in mind the dynamic nature of the environment in which GIS is embedded when making assessments

3. What are some additional perspectives on GIS implementation?

• The implementation of GIS is not merely a technical process, nor one that simply requires the formulation of rational management strategies.

• It is as much a social and political process which takes place within unique organizational cultures subject to changing conditions.

– Adapted from Campbell and Masser 1995

How do organizations interact with technology such as GIS? • Depends on how you see organizations, for example, as…

– Machine, system, or culture

Three perspectives (theories) on organization activity…• Technological determinism

– Widespread adoption and implementation of GIS is dependent upon the quality of the technology

• Managerial rationalism– GIS implementation success is based on good management

strategy that produces results (e.g. corporate approach)

• Social interactionism– Good understanding of the social organization (social attitudes or

institutional developments) leads to GIS implementation success

Technological determinismCharacteristic Technological determinism

Propensity for adoption Inevitable if a good technology

Reason for adoption To solve an operational problem that has been identified

Style of implementation Technical process

Constraints on implementation Technical worth of the innovation and stupidity of users

Likely outcome of implementation Positive, greater efficiency and more rational decision-making

Perception of technology Machine and methods

Perception of organizations Machine, organism

From Campbell & Masser, “GIS and Organizations” 1995

Managerial rationalismCharacteristic Managerial rationalism

Propensity for adoption Inevitable if a good technology and rational management

Reason for adoption To solve an operational problem that has been identified

Style of implementation Guided by a rational management strategy

Constraints on implementation Poor management and the technical worth of the innovation

Likely outcome of implementation Positive, greater efficiency and/or more rational decision-making

Perception of technology Machine and methods

Perception of organizations System

Social interactionismCharacteristic Social interactionism

Propensity for adoption Uncertain, depends on trends in society

Reason for adoption To enhance symbolic status or power

Style of implementation An organizational process that is problematic and uncertain

Constraints on implementation Social and political processes

Likely outcome of implementation Uncertain, at best a mixture of positive and negative results

Perception of technology Machine, methods and knowledge

Perception of organizations Cultures

Case studies suggest…

• A significant number of GIS (projects) failed when…– User needs are not well-defined– GIS design is not well matched to specific tasks– Ownership and control of GI is not well determined– Not clear how information is to be used in decision-

making process– Information sharing is not well coordinated – Organizational members are not ready to adopt GIS

or resistant to using GIS

Review of critical issues to successful GIS implementation

• Identification of simple applications producing information is fundamental to the work of potential users

• Awareness for limitations of the organization in terms of accepted practices and available resources is crucial to know limits

• User-directed implementation set within a framework based on the commitment and real participation of staff throughout the organization

• Ability to create stability or cope with change fosters success through adaptive management…continual review is critical to successful implementation.

Campbell & Masser 1995