DESIGNING AND CONDUCTING FORMATIVE EVALUATIONSPresented by Larry F. Cobb

April 30, 2014

BACKGROUND

In this presentation, we will address formative evaluations, how they should be used, and how to design and conduct them

OBJECTIVE

To discuss and give clarity to the following subjects: Formative Evaluation Designs Role of Subject-Matter, Learning, and Learner Specialists in Formative

Evaluation One-to-One Evaluation with Learners Small-Group Evaluation Field Trial Formative Evaluation in the Performance Context Formative Evaluation of Selected Materials Formative Evaluation of Instructor-Led instruction Data Collection for Selected Materials and Instructor-Led instruction Concerns Influencing Formative Evaluation Problem Solving during Instructional Design

DICK AND CAREY’S MODEL

WHAT IS FORMATIVE EVALUATION? A formative evaluation is the collection of data and

information during the development of instruction that can be used to improve the effectiveness of the instruction.

The process is used by designers to make their instructions more effective and effiecient.

They were originally used as a process to improve instruction after the first draft of instruction was developed.

It was later found better to try out earlier components of the design process, thereby avoiding a lot of problems that would otherwise not be discovered until after the draft of the instruction was complete.

FORMATIVE EVALUATION DESIGNS The evaluation design needs to yield information about

the location of and the reasons for any problems. This prevents the evaluation that is too narrow or too broad.

The instructional strategy is the framework for the design. The components will help you create questions that will allow you to evaluation instruction better.

FORMATIVE EVALUATION DESIGNS CON’TThese are questions that can be asked to evaluate the components of the instructional strategy:

1. Are the materials appropriate for the type of learning outcome?

2. Do the materials include adequate instruction on the subordinate skills, and are these skills sequenced and clustered logically?

3. Are the materials clear and readily understood by representative members of the target group?

4. What is the motivational value of the materials? Do the learners find the materials relevant to their needs and interests? Are they confident as they work through the materials? Are they satisfied with what they have learned?

5. Can the materials be managed efficiently in the manner they are mediated?

ROLE OF SUBJECT-MATTER, LEARNING, AND LEARNER SPECIALISTS IN FORMATIVE EVALUATION

It’s important to have the instruction reviewed by specialists.SMEs may be able to comment on the accuracy and currency of the instruction.Learning specialist may be able to critique your instruction related to what is known about enhancing that particular type of learning.Learner specialist may be able to provide insights into the appropriateness of the material for the eventual performance context.

ONE-TO-ONE EVALUATION WITH LEARNERSThe purpose of one-to-one evaluation is to identify and remove the most obvious errors in the instruction and to obtain initial performance indications and reactions to the content by learners

There are three criteria:

1. Clarity – Is the message, or what is being presented, clear to individual target learners

2. Impact – What is the impact of the instruction on individual learner’s attitude and achievement of the objectives and goals

3. Feasibility – How feasible is the instruction given the available resources (time/context)

SMALL GROUP EVALUATION

Small group evaluation have two primary purposes. The

first one is to determine the effectiveness of changes made

following the one-to-one evaluation and identify any

remaining learning problems that learners may have.

Secondly, small group evaluation is used to determine

whether learners can use the instruction without interacting

with the instructor.

FIELD TRIAL

The field trial is the third stage in formative evaluation, referring to the evaluations of the program and product in the setting in which it is intended to be used. It is used to determine whether the changes/revisions in the instruction made after the small group stage were effective and to also see whether the instruction can be used in the context for which it was intended.

FORMATIVE EVALUATION IN THE PERFORMANCE CONTEXTThe purpose of the performance context formative

evaluation is to make three fundamental determinations1. Do the learners find that it is appropriate to use their new skills in

the workplace and have they been doing so

2. If they have been used, what has been the impact on the organi_ation

3. What suggestions do the learners and other that they work with have for improving the instruction

FORMATIVE EVALUATION OF SELECTED MATERIALS Selected materials formative evaluation concludes that

the instructor chooses existing materials be there is seldom time to or resources to conduct all faces of the evaluation. In this case, the instructor should proceed directly to the field trial.

The instructor who conducts a field trial is able to observe the progress and attitudes of learners using a set of adopted or adapted materials.

FORMATIVE EVALUATION OF INSTRUCTOR-LED INSTRUCTION Formative evaluation of instructor-led instruction is done

to determine whether the instruction is effective and decide how to improve it. It is functions just like the field trial.

To prepare, the instructor must be concerned with the entry skills and prior knowledge, the posttest knowledge, and that attitudes of learners.

This is an opportunity to evaluate instructional procedures.

DATA COLLECTION FOR SELECTED MATERIALS AND INSTRUCTOR-LED INSTRUCTION

Information dealing with collection of data in a field trial of original instructional materials applies equally well to date collection procedures used in the evaluation of selected materials and instructional procedures.

CONCERNS INFLUENCING FORMATIVE EVALUATIONThere are several concerns about the formative evaluation

context and the learners who participate in the evaluation that the designer should keep in mind when planning and implementing data-collection procedures

1. Context concerns

2. Concerns about Learners

3. Concerns about Formative Evaluation Outcomes

4. Concerns with Implementing Formative Evaluation

PROBLEM SOLVING DURING INSTRUCTIONAL DESIGN The purpose is to make a decision about what to do with

a particular unit of instruction. The designer could collect enough date in the formative evaluation about the illustrations to make at least a tentative decision about their continued use in instructoin.

LIFELONG LEARNER SUMMARY

As a change agent, formative evaluations must be done regularly to ensure that information and content is kept current.

LARRY F. COBB

Email: lfcobb@alasu.edu