L643: Evaluation of Information Systems
Week 9: March 3, 2008
2
Announcement Guest lecture by Dr. Jonathan Plucker, Director of
the Center for Evaluation and Education Policy on April 14th
In Week 11, choose one of the following articles: Hernon, P., & Calvert, P. (2005). E-serve quality in
libraries: Exploring its features and dimensions. Library & Information Science Research, 27(3), 377-404.
Negash, S., Ryan, T., & Igbaria, M. (2003). Quality and effectiveness in web-based customer support systems. Information & Management, 40(8), 757-768
3
Exhibit 1: Hilton’s First Kiosk
4
Exhibit 2: Hilton’s Improved Kiosk
5
Hilton’s Self-Service Kiosk Case Study Bill, the VP for Customer-Facing Technology,
suggested to conduct a pilot study of the kiosk and ordered to deploy 14 kiosks in New York & Chicago
Bill asked you to come up with a proposal for evaluation studies. What would you propose?
6
Updated D&M IS Success Model (2002, 2003)
InformationQuality
System Quality
ServiceQuality
IntentionTo Use
Use
UserSatisfaction
NetBenefits
Creation Use Consequences
7
Updated D&M IS Success Model (2002, 2003)
InformationQuality
System Quality
ServiceQuality
IntentionTo Use
Use
UserSatisfaction
NetBenefits
Creation Use Consequences
8
IS Success Model (DeLone & McLean, 1992)
9
Technology-to-Performance Chain(Goodhue & Thompson, 1995)
Precursors ofutilization
Taskcharacteristics
Technologycharacteristics
Task-technology
fit
Utilization
Performanceimpacts
Individualcharacteristics
Expected consequences
Affect toward use
Social norms
Habits
Facilitating conditions
Feedback
Feedback
10
Summary Productivity concerns: Is the worker more
productive because s/he is using the system? Easy to determine for factory-like information work,
e.g., data entry More is better, so more items processed = more
productivity Generally not true of professionals and managers
Quality is of greater concern than quantity Measurement of productivity of knowledge workers
Generally based on their own perceptions of their productivity
11
IS Success Model (DeLone & McLean, 1992)
12
Measuring Payoffs of IT Investment (Kohli & Sherer, 2002)
IT Expenditure IT Assets IT Impacts OrganizationalPerformance
The IT ConversionProcess
The IT UseProcess
The competitiveProcess
Figure 1, p. 247
Technical risk Project risk
Organizational/Political risk
Competitive riskDisaster/security riskCollaborative risk
13
Measuring Payoffs of IT Investment (Kohli & Sherer, 2002)
Operational IT Managerial IT Strategic ITInvestment Financial
investment inFTE’s (employees)Equipmentconsulting
Financial investment & budgeting forApplicationsTrainingEducation
Financial investment & budgeting forCollaborative technologiesElectronic data interchangeERP
IT Assets # ofWorkstationsModemsInformation kiosksTrainers
Process redesign projectsPerson hours investedChange management initiatives
Hubs and routersKnowledge-based applicationsTeams working on strategic systemsIndustry & vendor partnerships
14
Measuring Payoffs of IT Investment (Kohli & Sherer, 2002)
Operational IT Managerial IT Strategic ITIT Impacts # of
Customers servicedHits on the websiteReturning customers
# ofMissed deadlinesReporting errorsMid-project process redesignsProduct recalls
# ofActual usage by period by userExtend of integration of IT into corporate decision making such as reports requested
Organiza-tional impacts
ProfitabilityROIROA
Employee turnoverMaintenance expensedowntime
Market shareRankingIndustry awardsCustomer service ratingStock price
15
Performance Impacts of IT (Devaraj & Kohli, 2003)
Independent Variablesof Technology Usage•# of report executed•Processing (CPU) time•# of records accessed
Control Variables•Midicare•Medicaid•Casemix•Patient income•# of employees•Age of hospital•Outpatients
Dependent Variables of Hospital Performance•Mortality•Revenue per admission•Revenue per day
16
Proving Your Project’s Worth (Tynan, 2005) IT = utility/infrastructure
Invisible (Star & Ruhleder, 1996)
What’s your take-aways from this article?
Star, S. L., & Ruhleder, K. (1996). Steps towards an ecology of infrastructure: Design and access for large information spaces. Information Systems Research, 7(1), 111-134
17
Intranet Model and Metrics (Jacoby, 2007) Intranet Efficiency and Effectiveness Model
(IEEM) Front-end: user-based factors, e.g., accessibility
and site navigation Back-end: site-based factors, e.g.,
personalization, information search, etc. People, processes, and technology: knowledge-
worker-based factors, e.g., their vision, purpose, and products or service
18
Intranet Model and Metrics (Jacoby, 2007)
19
20
IT and the Modalisation of Organisational Behaviour (Corea, 2006)
Modalisation = the modification of an organization’s capacity to carry out desired actions
IT enables action as well as constraints action
21
IT and the Modalisation of Organisational Behaviour (Corea, 2006)
Being a desired stateBeing the reverse/opposite (of the desired) state
Not being a desired stateNot being the reverse/opposite (of the desired) state
22
IT and the Modalisation of Organisational Behaviour (Corea, 2006)
Having to Act Having Not to Act
Not Having to ActNot Having Not to Act
Causing to Act Causing Not to Act
Not Causing to Act
Not Causing Not to Act
Legitimization Power
MotivationWanting to Act Wanting Not to Act
Not Wanting to ActNot Wanting Not to Act
23
Group Activity Work individually to fill out the worksheet Form teams based on the Group Project Propose a new model to measure
organizational impacts of IS Tell us how you can apply the model to
evaluate organization impacts of a real-world IS (your case)
Give a name to the model
Top Related