Designing and conducting summative evaluations
-
Upload
larry-cobb -
Category
Education
-
view
116 -
download
0
description
Transcript of Designing and conducting summative evaluations
Designing and Conducting Designing and Conducting Summative EvaluationsSummative Evaluations
Larry F. Cobb
ObjectivesObjectivesDefine the purpose of summative
evaluationDescribe the two phases of
summative evaluation and the decisions resulting from each phase
Design a summative evaluation to examine organizational benefits of instruction they have implemented
Contrast Formative and summative evaluation by purpose and design
What’s the Difference?What’s the Difference?
What is Summative EvaluationWhat is Summative Evaluation
Summative evaluation is defined as the design of evaluation studies and the collection of data to verify the effectiveness of instructional materials with target learners.
The Purpose of Summative The Purpose of Summative EvaluationEvaluation
Make “go-no-go” decisions• Keep current materials?• Look for something better suited to meet
organization’s specific instruction needs?• Summative evaluations are used to
judge the impact of a plan of instruction on the organization’s initial problem.
Evaluators – Why they should be Evaluators – Why they should be ExternalExternal
The primary evaluator in a summative evaluation is rarely the designer or developer of the instruction.
Evaluator is often unfamiliar with the materials, the organization requesting the evaluation, or the setting in which the materials are evaluated.
Preferred because they have no personal investment and will likely be more objective
Two Main Phases of Summative Two Main Phases of Summative EvaluationEvaluation
1.Expert Judgment Phase
2. Field Trial Phase
Expert Judgment PhaseExpert Judgment PhasePurpose:
Do the materials have the potential for meeting this organization’s needs?
There are several activities that decide whether the candidate instruction is promising.1. Congruence Analysis2. Content Analysis 3. Design Analysis4. Utility and Feasibility Analysis5. Current User Analysis
Expert Judgment PhaseExpert Judgment PhaseCongruence Analysis
Analyzing the congruence among1.An organization’s state needs and goals and those addressed in candidate instruction2.An organization’s target learners’ entry skills and characteristics and those for which candidate materials are intended3.An organization’s resources and those required for obtaining and implementing candidate instruction
Expert Judgment PhaseExpert Judgment PhaseContent Analysis
During this activity, an identified expert is used to judge material for accuracy and completeness to determine if they are inline with the organization’s stated goals. An instructional analysis of the stated goal is a very cost effective method.
Expert Judgment PhaseExpert Judgment PhaseDesign Analysis
The design analysis is an evaluation of the adequacy of the components of the instructional strategy included in the candidate material. Checklist are great during this activity.
Expert Judgment PhaseExpert Judgment PhaseUtility and Feasibility Analysis
Factors such as the availability of a learner guide or syllabus and an instructor’s manual are taking into consideration during this activity. This is also the time when you get information for the people that determine that the evaluation was necessary.
Expert Judgment PhaseExpert Judgment PhaseCurrent User Analysis
This final analysis seeks to get information about the candidate material from the organizations that are experienced in using them. The names of current users can often be obtained from the publishers of the materials.
Steps for Conducting Expert Steps for Conducting Expert Judgment Phase of Summative Judgment Phase of Summative EvaluationEvaluation
Field Trial PhaseField Trial PhasePurpose:
Are the materials effective with target learners in the prescribed setting?
Outcomes Analysis:1.Impact on Learners2.Impact on Job3.Impact on Organization
Field Trial Phase (continued)Field Trial Phase (continued)Management Analysis:1.Are instructor and manager attitudes
satisfactory?2.Are recommended implementation
procedures feasible?3.Are costs related to time, personnel,
equipment, and resources reasonable?
Steps for Conducting Field Trial Steps for Conducting Field Trial Phase of Summative EvaluationPhase of Summative Evaluation
Comparison of Formative and Comparison of Formative and Summative Evaluation (Table 12.3)Summative Evaluation (Table 12.3)
Formative Evaluation Summative Evaluation
PurposeLocate weaknesses in instructor in order to revise it
Document strengths and weaknesses in instruction in order to decide whether to maintain or adopt it
Phases or StagesOn-to-oneSmall groupField Trial
Expert judgmentField trial
Instructional Development History
Systematically designed in-house and tailored to the needs of the organization
Produced in-house or elsewhere not necessarily following a systems approach
Materials One set of materialsOne set of materials or several competing sets
Position of EvaluatorMember of design and development team
Typically an external evaluator
OutcomesA prescription for revising instruction
A report documenting the design, procedures, results, recommendations, and rationale
Change AgentChange AgentAs a change agent, is very important for
us to constantly evaluate the instructional strategy ensure that is staying the course with what the organization has in mind while being usable and feasible for the organization. The summative evaluation does just that.
Larry F. CobbLarry F. [email protected]