L643: Evaluation of Information Systems Week 11: March 17, 2008.
-
Upload
dwain-gibbs -
Category
Documents
-
view
215 -
download
0
Transcript of L643: Evaluation of Information Systems Week 11: March 17, 2008.
L643: Evaluation of Information Systems
Week 11: March 17, 2008
2
Measurement Instrument
It is not necessary to use the instruments discussed in class
However, it would be helpful to consider the existing instruments Review the ones we discussed Search the IS World survey instrument database:
http://www.isworld.org/surveyinstruments/surveyinstruments.htm
3
Pilot Testing of Instruments
Instrument providers: Give instructions to the testers
E.g., their roles, actions, etc Provide your instrument Provide a feedback sheet/ or ask a list of questions
afterwards (debriefing) Testers
Try to answer the questions Ask questions when instructions are not clear Be honest and sincere Provide constructive feedback
4
Pilot Testing of Instruments
First Pilot Test Session
Second Pilot Test Session
Unicoop (5) Zipcar (5)
Need to send an elec copy to [email protected] and cc’ the msg to [email protected]
Library Thing (3) Evergreen (4)
John Fluevog Boots & Shoes (4)
Quadrem (4)
5
Announcements
The Evaluation Design Proposal, Measurement Instrument, and Instrument Memo will be due at 5pm on Thursday, March 26th, instead of next Monday
Any group will need a computer lab?
6
Evaluation Project ProcessProblem/Purpose
Hypotheses or Questions
Definitions
Literature Review
Sample
Instrumentation
Procedures/Design
Data Collection & Analysis
Recommendation
Pilot (test)
See also UCLA’s report
7
Survey Research (Babbie, 2004)
Choose appropriate question forms Open-ended questions (qualitative) Closed-ended questions (quantitative)
The response categories provided should be exhaustive The answer categories must be mutually exclusive
When would be appropriate to use open-ended Qs? When would be appropriate to use closed-ended
Qs?
8
Survey Research (Babbie, 2004)
Questions: Avoid asking multiple questions in a single item Qs must be relevant Qs must be concise Try to avoid confusions/ biased items & terms
Respondents: whether they are capable of answering the
questions must be willing to answer
9
Survey Research (Babbie, 2004)
Format Clear instructions Easy to understand Ordering items Pre-testing
10
Survey Research
Types of surveys Telephone surveys
http://www.pewinternet.org/reports.asp (Pew Internet & American Life)
Mail questionnaire Low response rates How to increase response rates?
11
Survey Research (Babbie, 2004)
Administration of survey: Self-administration:
Via snail mail Via Web (e.g., Survey Monkey)
Interview survey Telephone survey
12
Assessing Program Impact: Alternative Designs (Rossi, et al., 2004)
Bias in estimating program effects Selection bias (nonequivalent comparison design)
Targets drop out Targets refuse to participate
Environmental trends Interfering events, e.g., natural disaster Maturation – people grow
13
Risks of Quantitative Studies (Nielsen, 2004)
List 3 major problems of quantitative studies Considering these problems, how would you
treat most of the IS research based on quantitative studies? (disregard them or accept as it is?)
Can you think of any situations in which quantitative studies would be useful?
For your group project, what type of research design would be useful? Why?
14
Quality and Effectiveness in Web-based Customer Support Systems (Negash, et al., 2003)
Information Quality
•Informativeness
•entertainment
System Quality
•Interactivity
•AccessService Quality
•Tangible
•Reliability
•Assurance
•Responsiveness
•Empathy
Effectiveness
•User satisfaction
H1
H2
H3
15
Quality and Effectiveness in Web-based Customer Support Systems (Negash, et al., 2003)
SERVQUAL: Tangibles: physical features of the system Reliability: the system’s consistency of
performance and dependability Responsiveness: readiness of the system to
provide service Assurance: knowledge & courtesy expressed in
the system Empathy: the care and individualized attention
http://www.12manage.com/methods_zeithaml_servqual.html
16
Quality and Effectiveness in Web-based Customer Support Systems (Negash, et al., 2003)
Information Quality
•Informativeness
•entertainment
System Quality
•Interactivity
•AccessService Quality
•Tangible
•Reliability
•Assurance
•Responsiveness
•Empathy
Effectiveness
•User satisfaction
H1
H2
H3-.051
.449
.348
17
Zones of Tolerance (Kettinger & Lee, 2005)
Perceived Service
Adequate Service
Desired Service
Expected Service
SERVQUAL ZOT SERVQUAL
Minimum service level
Desired service level
Perceived service level
Perceived adequacy
Perceived superiority
18
E-service quality in libraries (Hernon & Calvert, 2005)
Adaptations of SERVQUAL to include e-SERVQUAL instrument
SERVQUAL focuses on 5 dimensions: Assurance Empathy Reliability Responsiveness Tangibles
19
E-service quality in libraries (Hernon & Calvert, 2005)
LibQUAL focuses on 4 dimensions: Access to information Affect of service Library as place Personal control
20
E-service quality in libraries (Hernon & Calvert, 2005)
Customization/personalization Importance #4 Unimportance #2
Security/privacy/trust Importance #5 Unimportance #3
Web site aesthetics Importance #10 Unimportance #1
21
E-service quality in libraries (Hernon & Calvert, 2005)
ideal
perceived
Retain
Improve
Reallocate
resources
Revisit
22
Choosing Measures to Evaluate … (Ryan, et al., 2001)
Are you measuring something simply because you can? Consider cost & time Just why you are measuring something
Pre-test the measure’s proposed definition Training the relevant staff (see the 5 Qs on p.
123)
23
Choosing Measures to Evaluate … (Ryan, et al., 2001)
Data collection methodologies: Qualitative techniques:
Case study Content analysis Policy analysis Focus groups Interviews
Quantitative techniques: In-house, mail or electronic surveys
24
Choosing Measures to Evaluate … (Ryan, et al., 2001)
Data collection methodologies: Automated techniques:
Pop-up surveys Network traffic measures Web log file analysis
Are there confidentiality and privacy concerns?
How can the burden of data collection be balanced ore reduced?