Overview of Chapters 11 – 13, & 17
description
Transcript of Overview of Chapters 11 – 13, & 17
![Page 1: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/1.jpg)
Overview of Chapters 11 – 13, & 17
![Page 2: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/2.jpg)
Clarifying the evaluation request and responsibilities
1. An organization has requested that you to complete an evaluation. What questions do you feel are important to ask before beginning the evaluation?
2. Are there times when you would decline a request for evaluation? Why?
3. What are some of the advantages and disadvantages in having an evaluation conduct by an external evaluator? By an internal evaluator?
(take 5 minutes with a partner to discuss)
Chapter 11
![Page 3: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/3.jpg)
1. Identify the stakeholdersKey questions:Who are the sponsors and clients?Who are the managers and staff?Are there other interest groups, such as agencies, elected
officials, the public at large?
2. Identify the purpose for the evaluationKey questions: Why is the evaluation being requested?What question will it answer?How will the results be used? By whom?
Clarifying the Evaluation Request and Responsibilities
![Page 4: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/4.jpg)
3. Determine if the Evaluation is AppropriateKey questions:How will the results be used?Will the results produce trivial information?Is the program too new?Is the evaluation purpose ethical?
Clarifying the Evaluation Request and Responsibilities
![Page 5: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/5.jpg)
Steps to follow: 1. Clarify the intended program model or theory
2. Examine the program in implementation to determine whether it matches the model and could achieve the goals or objectives
3. Explore different approaches to determine if they meet the stakeholders information needs
4. Agree on evaluation priorities and intended uses of the study
Clarifying the Evaluation Request and Responsibilities
![Page 6: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/6.jpg)
Internal vs. External evaluator
Internal: know the organization, history, and decision-making style and would be around to encourage use of evaluation results
External: bring a greater objectivity and sometimes specialized skill for a particular project or evaluation
Who will evaluate the program?
![Page 7: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/7.jpg)
Setting Boundaries and Analyzing the Evaluation Context
1. After determining who the stakeholders are, how do you determine who should be involved in the evaluation process?
2. What should the evaluator consider in analyzing the political context in which an evaluation will occur?
(take 5 minutes to discuss with a partner)
Chapter 12
![Page 8: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/8.jpg)
The evaluator should identify and communicate with each stakeholder group or its representative:
1. To learn about the groups concerns2. To understand how they will use the
evaluation results
This will help the evaluator weigh their input during the evaluation
Setting Boundaries and Analyzing the Evaluation Context
![Page 9: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/9.jpg)
Do evaluators need to be aware of Squeaking wheels &Powerful Stakeholders?
You need to bring about a “democratic dialogue” if possible, with a balance in the number of people involved and diversity within the group.
Setting Boundaries and Analyzing the Evaluation Context
![Page 10: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/10.jpg)
Greene (2005) identified four groups:1. People who have decision authority over
the program2. People who have direct responsibility for
the program3. People who are the intended beneficiaries
of the program
Setting Boundaries and Analyzing the Evaluation Context
![Page 11: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/11.jpg)
Scriven (2007) adds to the list:
Political supporters or opponents
Community Leaders
Public in General
Setting Boundaries and Analyzing the Evaluation Context
![Page 12: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/12.jpg)
Political Context:
1. Who will gain to gain/lose?2. Which individuals and groups have power in the
setting?3. How should the evaluator relate/communicate to the
groups?4. Will all groups cooperate?5. Which groups have a vested interest in the
outcomes?6. What safeguards should be incorporated into the
evaluation agreement?
Setting Boundaries and Analyzing the Evaluation Context
![Page 13: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/13.jpg)
Identifying and selecting the evaluation questions and criteria
1. What is the focus of evaluation questions?2. What are good sources for evaluation
questions?3. Who should be involved in the divergent
and convergent phases? Why?
Chapter 13
![Page 14: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/14.jpg)
Evaluation Questions:
Provide focus to the evaluationSpecify the information the evaluation will
provideGuide choices for data collection, analysis,
and interpretation
Identifying and selecting the evaluation questions and criteria
![Page 15: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/15.jpg)
Good Sources:
Existing Standards in the FieldResearch LiteratureContent ExpertsEvaluators Own Experience
Identifying and selecting the evaluation questions and criteria
![Page 16: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/16.jpg)
Divergent: Cronbach (1982), “Opening one’s mind to be entertained at least briefly as prospects for investigation” (p. 210).
Fitzpatrick, et al. (2011), “throw a broad net and learn from many possible sources” (p. 316).
Identifying and selecting the evaluation questions and criteria
![Page 17: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/17.jpg)
Convergent: Three reasons for reducing the range of variables.
1. Budget2. Manageability/Complexity3. Attention Span of Audience
The evaluator aims for the maximum bandwith.Cronbach (1982)
*Typically completed by an advisory group with representatives from different stakeholder groups.
Identifying and selecting the evaluation questions and criteria
![Page 18: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/18.jpg)
1. What considerations are important in tailoring the reporting of evaluation results to audiences?
2. What are some of the ways results can be communicated to stakeholders?
3. What should be included in an evaluation report?
(take 5 minutes to discuss with a partner)
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding
![Page 19: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/19.jpg)
Different audiences have different information needs and knowledge levels.
Reading abilityFamiliarity with the programAttitude toward the programRole in decision makingFamiliarity with evaluation practices/methodsAttitude and interest in evaluationExperience in using evaluation results
Torres, Preskill, and Pointek (2005)
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding
![Page 20: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/20.jpg)
If possible (Patton, 2008) points out that evaluation data are sued more if the evaluator discusses and negotiates the format, style, and organization of reports with primary users.
Other Key Points: Avoid Jargon, Use simple direct language, Use examples, anecdotes,
be interesting…
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding
![Page 21: Overview of Chapters 11 – 13, & 17](https://reader036.fdocuments.in/reader036/viewer/2022062501/568161e5550346895dd2046f/html5/thumbnails/21.jpg)
Key Elements of an Evaluation Report (p. 470)I. Executive SummaryII. Introduction to the ReportIII. Focus of the EvaluationIV. Brief Overview of Evaluation Plan &
ProceduresV. Presentation of ResultsVI. Conclusion and RecommendationsVII. Minority Reports or RejoindersVIII. Appendices
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding