S519: Evaluation of Information Systems

Understanding Evaluation

Ch1+2

Definition of Evaluation

Systematic determination of the quality or value of something (Scriven, 1991)

What can we evaluate? Projects, programs, or organizations Personnel or performance Policies or strategies Products or services Processes or systems Proposals, contract bids, or job application

Lessons-learned and methods aretransdisciplinary

Terminology (Davison, Glossary)

Evaluand That which is being evaluated (e.g. Program,

policy, project, product, service, organization) In personnel evaluation the term is evaluee

Issues of Evaluation Evaluation is for

Find areas for improvement Generate an assessment of overall quality

Answer the question of „Merit“ or „Worth“ (Scriven, 1991) Merit is the „intrinsic“ value of something = „quality“ Worth is the value of something to an individual, an

organization, an institution – contextualized merit -- = „value“

Choosing the right group

Accountability evaluation It is important to conduct an independent

evaluation i.e. Nobody on the evaluation team should have a

significant vested interest in whether the results are good or bad

Organization learning capability evaluation It can (better) be dependent evaluation

i.e. Organizational staff, consultants, managers, customers, trainers, trainnees etc. can join.

The steps involved (D-p4)

Step1: understanding the basic about evaluation (ch1)

Step2: defining the main purposes of the evaluation and the „big picture“ questions that need answers (ch2)

Step3: Identifying the evaluative criteria (ch3) Step4: Organizing the list of criteria and

choosing sources of evidence (collecting data) (ch4)

The steps involved (D-p4)

Step5: analyzing data dealing with the causation issue (which cause

what, why), to avoid „subjectivity“ (ch5+6) importance weighting: weight the results (ch7) Meric determination: how well your evaluand has done

on the criteria (good? Unacceptable?) (ch8) Synthesis methodology: systematic methods for

condensing evaluation findings (ch9)

Staticistical analysis: Salkind (2007)

The steps involved (D-p4)

Step6: result Putting it all together: fitting the pieces into the KEC

framework (ch10) Step7: feedback

Meta-evaluation: how to figure out whether your evlauation is any good (ch11)

The Key Evaluation Checklist (Davidson, 2005, p. 6-7)

I. Executive Summary

II. Preface III. Methodology

1. Background & Context

2. Descriptions& Definitions

3. Consumers

4. Resources

5. Values

6. Process Evaluation

7. OutcomeEvaluation

8 & 9. Comparative Cost-Effectiveness

10. Exportability

11. Overall Significance

12. Recommendations& Explanations

13. Responsibilities 14. Reporting& Follow-up

15. Meta-evaluation

Step 1: Understand the basic of evaluation

Identify the evaluand Background and context of evaluand

Why did this program or product come into existence in the first place?

Descriptions and definitions Describe the evaluand in enough detail so that virtually anyone

can understand what it is and what it does

How: collect background information, pay a firsthand visit or literature review

Are you ready for your first evaluation project?

Some tips before you start Make sure that your evaluand is not difficult to

access (geolocation, inanimate objects) Make your evaluand a clearly defined group

(avoid abstract and complex system) Avoid political ramification (assess your boss pet

project, university administration) To avoid your invovlement in the evaluand (to

assess a class which you are teaching, etc.)

Previous Projects

Metadata discussion group Brown Bag discussion group Twitter SLIS website Information Visulization Lab website Media and Reserve Services in Wells Library IU CAT SoE website Chemistry library website

Step1: Output report

Output: one or two page overview of the evaluand and findings What is your evaluand Background and context of your evaluand Description of your evaluand

Try to be as detail as possible

Step2: Defining the Purpose of the Evaluation (D-Ch2)

Who asked for this evaluation and why? What are the main evaluation questions? Who are the main audience?

Evaluation purposes

A. what is (are) the main purpose(s) of the evaluation?

To determine the overall quality or value of something (summative evaluation, absolute merit) i.e. Decision making, funding allocation decision, benchmarking

products, using a tool, etc.

To find areas for improvement (formative evaluation, relative merit) To help a new „thing“ to start To improve the existing „thing“

Big picture questions

Big picture questions: B. What is (are) the big picture question(s) for

which we need answers? Absolute merit

Do we want to invest this project? relative merit

How does this project compare with the other options? Ranking

Step2: Output report

Your step2 output report should answer the following questions: Define the evaluation purpose

Do you need to demonstrate to someone (yourself) the overall quality of something?

Or Do you need to find a file for improvement? Or do you do both?

Once you answer above questions, figure out what are your big picture questions: Is your evaluation related to the absolute merit of your evaluand? Or the relative merit of your evaluand