Systematic determination of the quality or value of something (Scriven, 1991) What can we evaluate?...
-
Upload
sylvia-hall -
Category
Documents
-
view
215 -
download
0
Transcript of Systematic determination of the quality or value of something (Scriven, 1991) What can we evaluate?...
Understanding Evaluation
Systematic determination of the quality or value of something (Scriven, 1991)
What can we evaluate? Projects, programs, or organizations Personnel or performance Policies or strategies Products or services Proposals, contract bids, or job application Almost everything in our daily life because
before you make decision, you do the evaluation first. Comparison is a kind of evaluation.
Definition of Evaluation
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage
Evaluand That which is being evaluated (e.g.
Program, policy, project, product, service, organization, almost everything)
In personnel evaluation the term is evaluee
Terminology (Davison, Glossary)
S519
Evaluation is for Find areas for improvement Generate an assessment of overall quality
Answer the question of „Merit“ or „Worth“ (Scriven, 1991) Merit is the „intrinsic“ value of something =
„quality“ Worth is the value of something to an individual, an
organization, an institution – contextualized merit -- = „value“
Issues of Evaluation
S519
Accountability evaluation It is important to conduct an
independent evaluation i.e. Nobody on the evaluation team should
have a significant vested interest in whether the results are good or bad
Choosing the right group
S519
Step1: understanding the basic about evaluation (ch1)
Step2: defining the main purposes of the evaluation and the „big picture“ questions that need answers (ch2)
Step3: Identifying the evaluative criteria (ch3)
Step4: Organizing the list of criteria and choosing sources of evidence (collecting data) (ch4)
The steps involved (D-p4)
S519
Step5: analyzing data dealing with the causation/correlation issue
(which cause what, why), to avoid „subjectivity“ (ch5+6)
importance weighting: weight the results (ch7) Meric determination: how well your evaluand has done
on the criteria (good? Unacceptable?) (ch8) Synthesis methodology: systematic methods for
condensing evaluation findings (ch9)
Staticistical analysis: Salkind (2007)
The steps involved (D-p4)
S519
Step6: result Putting it all together: fitting the pieces
into the KEC framework (ch10) Step7: feedback
Meta-evaluation: how to figure out whether your evlauation is any good (ch11)
The steps involved (D-p4)
S519
The Key Evaluation Checklist (Davidson, 2005, p. 6-7)
I. Executive Summary
II. Preface III. Methodology
1. Background & Context
2. Descriptions& Definitions
3. Consumers
4. Resources
5. Values
6. Process Evaluation
7. OutcomeEvaluation
8 & 9. Comparative Cost-Effectiveness
10. Exportability
11. Overall Significance
12. Recommendations& Explanations
13. Responsibilities 14. Reporting& Follow-up
15. Meta-evaluation
S519
Identify the evaluand Background and context of evaluand
Why did this program or product come into existence in the first place?
Descriptions and definitions Describe the evaluand in enough detail so that virtually
anyone can understand what it is and what it does
How: collect background information, pay a firsthand visit or literature review
Step 1: Understand the basic of evaluation
S519
Some tips before you start Make sure that your evaluand is not
difficult to access (geolocation, inanimate objects)
Make your evaluand a clearly defined group (avoid abstract and complex system)
Avoid political ramification (assess your boss pet project, university administration)
To avoid your invovlement in the evaluand (to assess a class which you are teaching, etc.)
Are you ready for your first evaluation project?
S519
Metadata discussion group, Brown Bag discussion group, Twitter, SLIS website, Information Visulization Lab website, Media and Reserve Services in Wells Library, IU CAT, SoE website, Chemistry library website
How about something else related to real-world problems
For social good (http://dssg.io/projects/) Industry and Government talks at KDD214 (
http://www.kdd.org/kdd2014/) Maybe this will inspire you with a great project to
work on
Previous Projects
Utility To ensure that an evaluation will serve the practical
information needs of intended users Feasibility
To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal
Propriety To ensure that an evaluation will be conducted
legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results (IRB: http://research.iu.edu/rschcomp/revlocation.html)
Accuracy To ensure that an evaluation will reveal and convey
technically adequate information about the features
Standards for Evaluation (Patton, 1997, p. 17)
http://www.wmich.edu/evalctr/jc/ S519
Systematic inquiry Competence Integrity/honesty Respect for people Responsibilities for general and
public welfare
Guiding Principles for Evaluators (American Evaluation Association, 1995)
S519
Output: one or two page overview of the evaluand and findings What is your evaluand Background and context of your
evaluand Description of your evaluand
Try to be as detail as possible
Step1: Output report
S519
Who asked for this evaluation and why?
What are the main evaluation questions?
Who are the main audience?
Step2: Defining the Purpose of the Evaluation (D-Ch2)
S519
A. what is (are) the main purpose(s) of the evaluation?
To determine the overall quality or value of something (summative evaluation, absolute merit)
i.e. Decision making, funding allocation decision, benchmarking products, using a tool, etc.
To find areas for improvement (formative evaluation, relative merit)
To help a new „thing“ to start To improve the existing „thing“
Evaluation purposes
S519
Big picture questions: B. What is (are) the big picture
question(s) for which we need answers? Absolute merit
Do we want to invest this project? relative merit
How does this project compare with the other options?
Ranking
Big picture questions
S519
Your step2 output report should answer the following questions:
Define the evaluation purpose Do you need to demonstrate to someone (yourself) the
overall quality of something? Or Do you need to find a file for improvement? Or do you do both?
Once you answer above questions, figure out what are your big picture questions:
Is your evaluation related to the absolute merit of your evaluand?
Or the relative merit of your evaluand
Step2: Output report
S519