people.virginia.edupeople.virginia.edu/~jlk4p/itma/11_softeval/lesson8.docx · Web viewThe...

Assignment 8 – Creating My Own Evaluation Criteria

Jack [email protected]/22/2013

Activity 1

Checklist Description Mnemonic or Evaluative?

Why?

Job daily task list Mnemonic The list is used to document the things I need to get done at work; items just get checked off.

Baking cookies – Selecting a recipe to follow.

Mnemonic Before starting, I check off that I have all the ingredients required. Then I just check off each step in the recipe from the first to the last as I do them.

Walking the dog - Run down the list of things I need to have with me on the walk: keys, leash, baggie, and hat.

Mnemonic Just checking off items on my mental list.

Home maintenance project usually requires comparing estimates from multiple contractors.

Evaluative I need to weigh several different things in each estimate: proposed work, time, cost, references. It’s not simply selecting the least expensive contractor. Each gets evaluated and then they are compared to make the selection.

Leaving the house to go anywhere requires making sure I’ve got the keys, my cellphone, my wallet, and put the dog in the kitchen and put the gate up.

Mnemonic Just checking off items in my mental list that need to be done any time I go anywhere.

mailto:[email protected]

Activity 2

Develop a checklist that can be used in conjunction with the expert review provided by Dr. Sherman.

Software Evaluation Checklist Part 1

Computer–Supported Instructional Contexts1. How often are there opportunities for learners to create things in the software?2. How much does the software simulate real world experiences that allow the learner

to make decisions?3. How often does the software simulate real world experiences that allow the learner

to explore within the environment?4. How much does the software allow the learner to explore and access reference-type

information?5. How often does the software provide direct instruction with specific examples

applicable to the skills, knowledge, and/or attitudes?6. How much does the software offer practice with feedback over specific skills?7. How often does the software engage learners in competition, cooperation, puzzles,

or strategies for the sake of entertainment?8. How much does the software allow learners to communicate with other people via

text, audio, video and/or binary files?9. How often is this software used in real-world situations or settings?

Context Types Evident1. The software ORIENTS the learner about a need to learn a set of skills.2. The program offers specific INSTRUCTION in the form of an introduction,

information, activities, practice, feedback, and/or review.3. The software provides opportunities to TRANSFER skills, knowledge, and/or

attitudes using conditions different than the initial instruction of them.Motivational Strategies Incorporated

Incongruity/conflict Concreteness Variability Humor Inquiry Participation Experience Present worth Future usefulness Need matching Modeling Choice Learning requirements

Difficulty Expectations Attributions Self-confidence Natural consequences Unexpected rewards Positive outcomes Negative outcomes Scheduling

Effective Instructional Design Component Strategies Used1. The software introduces instruction supporting the specified learning outcomes.2. The program contains information/activities related to the learning skills indicated

for the specified learning outcomes.3. There are practice and feedback opportunities for the specified learning outcomes

in the software.4. The program helps students in learning the specified outcomes by offering review of

the instructional content.5. The software assesses the overall learning of the specified outcomes.6. The program provides opportunities for the learners to apply previously learned

specified outcomes in a new transfer environment.Specific Elements Established For Conditions

Gain learner attention Articulate in some way the SKA already needed to succeed within the new learning

environment Identify opportunities in which learners will relate what is about to be learned to

what they already know how to do Inform learners of objectives Present a big picture and fit objectives into it Present the utility of the SKA to be learned Clearly identify the incentives for learning the SKA and succeeding within the

learning environment Establish clearly-perceived learner accountability, roles and tasks within the

learning environment Establish clearly-perceived instructor roles and learner support mechanisms Employ specific context approaches if applicable Establish the appropriate conditions for the types of new SKA facilitated Structure activities conforming to situated learning guidelines and strategies Relate all SKA to a clearly-defined content domain Provide learner guidance for learners as they apply information present to SKA

being facilitated Present a variety of clear, concrete examples and non-examples Provide opportunities for learners to explore the learning environment with

minimal instructor guidance and intervention Provide initial learner guidance Practice must match performances and conditions indicated within objectives All should get practice

Feedback as immediate as possible Provide opportunities for learners to summarize the key ideas, what they learned

how to do, and how they personally learned it Restate objectives Assessment must match performances and conditions indicated within objectives Present new context approach which elicits the same performances under different

conditions Make the utility of succeeding within the new context apparent Present cues within the new context which aid learners in selecting and applying the

appropriate previously-learned SKA Clearly identify the incentives succeeding within the new context

Skill-Specific Conditions Established For Outcomes Use the term or definition in a sentence Relate the information to preexisting knowledge Present all terms clearly using the fewest number of words to convey meaning Use a variety of concrete examples when possible, emphasizing the clear and well-

defined features that relate directly to the information Explain clearly how learners will be expected to recall the information while it is

initially presented Make information readily accessible to learners, and provide opportunities for them

to explore nice to know info associated with the knowledge Practice verbal information with immediate feedback Present varied examples or instances of concepts and rule applications, calling

attention to the distinctive features of examples, definitions, and procedures Present non-examples or non-instances of the concept Encourage learners to recall previously learned information or examples that

illustrate concepts or rules being presented Clearly communicate the definition of defined concepts, using the fewest number of

words Break down the process of performing or applying rules into steps, and clearly

communicate these steps to the students Demonstrate an application of the rule for the students Provide learners with opportunities to play with concepts and rules within

simulated or real environments Present a variety of contexts or experiences that allow the students to practice

applying the rules or identifying/describing concepts, providing guidance early on Clearly identify examples of choices made by people who posses the desired attitude Clearly identify instances in the students’ lives in which making choices are based

on the attitude being presented Make students aware of the personal benefits gained by making choices based on

attitudes Allow students the opportunity to practice making choices associated with the

desired attitude and give them feedback Verbally guide learners through the routine Visually present examples of routine execution Practice motor skills with immediate feedback

Encourage the use of mental practicePractice With Feedback Available

1. Learners are informed when they submit a correct or incorrect response.2. Learners are informed what the correct response is.3. When learners submit an incorrect response, they are informed of the correct

answer and offered information about the nature of their error.4. Learners are informed of the correct response regardless of whether they got it

right or not and are presented with additional information related to the correct response.

5. Learners are informed about the consequences of specific responses and may receive recommendations about future choices.

Effective General Message Principles Applied1. Screen display follows horizontal-vertical and left-right organization common to the

culture of the intended audience.2. Screens contain easily identifiable relevant information.3. Attention is drawn to those parts of a message intended to stand in contrast to the

other screen elements.4. Meaningful perceptual units are clustered in physical as well as temporal space.5. The rate of presenting sequential information is slow enough for accurate

perception, scrutiny, elaboration, and comprehension, but not too slow to prevent attention from wandering.

6. Unfamiliar presentation modes are modeled for the user to develop familiarity.7. Messages are kept simple and not obscured by too much non-critical detail.

Text Usage1. Line breaks are dictated logically by the text, not by space available.2. Marginal headings and annotations are used to improve text organization.3. Text and background contrast are clear.4. Standard text formatting is used throughout, even for titles, headings, etc.

Pictures and Illustrations Usage1. A picture’s function is clearly designated.2. If necessary, specific instructions are included to interpret a picture and its intended

function.3. Illustrations are closely related to the context of the text.4. Representational pictures (e.g. line drawing of elephant) relevantly resemble their

referent in terms of shape, detail, size, color, and context. Analogies are incorporated for users of limited experience with the referent.

5. Organizational pictures incorporate devices that stress temporal and spatial relationships.

Diagrams, charts and graphs usage1. Abstract ideas and relationships are presented graphically in diagrams, charts, or

graphs.2. Diagrams are organized so that sequence of the parts viewed is in the necessary

sequence.3. Thicker lines designate stronger relationships.4. Attention is drawn to captions.5. The relative importance of elements is represented by relative sizes in the diagram.

6. Estimates or magnitude are made linear (as opposed to two or three dimensional).7. The position of the elements corresponds to the perceived relationship among the

elements.8. Graphs depicting precise amounts utilize a Cartesian graph.9. Graphs depicting comparative amounts not requiring precision utilize pie charts.10. Trends are generally illustrated using line graphs.11. Comparisons are illustrated by bar graphs or pie charts.

Audio Usage1. Speech is utilized to effectively communicate ideas that have to be understood in a

particular sequence.2. Speech is not used to relay complex content better suited for text or text with

illustrations.3. Narration is closely related to the context of the text.

Navigation1. Clearly and quickly deciphered navigation schemes are used.2. A common navigation scheme is used throughout the program.3. Menu organization is broad and shallow.4. Unfamiliar navigation modes are modeled for the user to develop familiarity.5. Complex or detailed navigation choices are explained to the learner.


Checklist Used In Conjunction With Dr. Sherman’s ID Expert Review

Evaluator: ___________________________________________________ Date: __________________________

Product Name: ________________________________________________________________________________

Description: _______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Intended Audience: ___________________________________________________________________________

Place a check mark in front of each section heading that is applicable to the software being evaluated.

___ Computer–Supported Instructional Contexts (10%)Check the appropriate answer for each question below based on your examination of the software.

1. How often are there opportunities for learners to create things in the software?__ None __ Few __ Adequate __ Plenty

2. How much does the software simulate real world experiences that allow the learner to make decisions?

__ None __ Few __ Adequate __ Plenty

3. How often does the software simulate real world experiences that allow the learner to explore within the environment?


4. How much does the software allow the learner to explore and access reference-type information?


5. How often does the software provide direct instruction with specific examples applicable to the skills, knowledge, and/or attitudes?


6. How much does the software offer practice with feedback over specific skills?__ None __ Few __ Adequate __ Plenty

7. How often does the software engage learners in competition, cooperation, puzzles, or strategies for the sake of entertainment?


8. How much does the software allow learners to communicate with other people via text, audio, video and/or binary files?


9. How often is this software used in real-world situations or settings?__ None __ Few __ Adequate __ Plenty

___ Context Types Evident (10%)For each statement below check the appropriate option that reflects your observations for the software.

1. The software ORIENTS the learner about a need to learn a set of skills.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

2. The program offers specific INSTRUCTION in the form of an introduction, information, activities, practice, feedback, and/or review.

__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

3. The software provides opportunities to TRANSFER skills, knowledge, and/or attitudes using conditions different than the initial instruction of them.


___ Motivational Strategies Incorporated (10%)Check the appropriate expectations found for each motivational strategy in the software in the table below.

Motivation Strategy

ExpectationsDoesn’t Meet

Marginally Meets

Meets Exceeds

Attention Incongruity/ConflictConcretenessVariabilityHumorInquiryParticipation

Relevance ExperiencePresent worthFuture usefulnessNeed matchingModelingChoice

Confidence Learning requirementsDifficulty

ExpectationsAttributionsSelf-confidence

Satisfaction Natural consequencesUnexpected rewardsPositive outcomesNegative outcomesScheduling

___ Effective Instructional Design Component Strategies Used (10%)Check the appropriate option for each statement based on your experience with the software.

1. The software introduces instruction supporting the specified learning outcomes.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

2. The program contains information/activities related to the learning skills indicated for the specified learning outcomes.


3. There are practice and feedback opportunities for the specified learning outcomes in the software.


4. The program helps students in learning the specified outcomes by offering review of the instructional content.


5. The software assesses the overall learning of the specified outcomes.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

6. The program provides opportunities for the learners to apply previously learned specified outcomes in a new transfer environment.


___ Specific Elements Established For Conditions (5%)Refer to the items marked as Marginally or Definitely in the Context Types Evident section and check the appropriate specific elements expectations found in the software for the skills, knowledge, and/or attitudes (SKA).

Specific Elements Established


Marginally Meets

Meets Exceeds

Orients Gains learner attention.Articulate the SKA



Marginally Meets

Meets Exceeds

already needed to succeed within the new learning environment.Identifies opportunities where learners will relate what is about to be learned to what they know how to do.

Instruction Informs a learner of objectives.Presents a big picture and fits objectives into it.Presents the relevance of the SKA to be learned.Clearly identifies the incentives for learning the SKA and succeeding within the learning process.Establishes clearly perceived learner accountability, roles, and tasks.Establishes clearly perceived instructor roles and learner support mechanisms.Employs specific context approaches if applicable.Establishes appropriate conditions for the type of new SKA facilitated.Structures activities around situated learning guidelines and strategies.Relates all SKA to a clearly defined content domain.Provides learners



Marginally Meets

Meets Exceeds

guidance as they apply information presented to SKA being facilitated.Presents a variety of clear, concrete examples and non-examples.Provides opportunities for learners to explore the environment with minimal instructor intervention.Practice provides initial learner guidance.Practice opportunities match performances and conditions indicated within objectives.All learners get practice.Feedback is as immediate as possible.Provides opportunities for learners to summarize key ideas, what they learned how to do, and how they personally learned it.Restates the objectives.Assessment matches performances and conditions indicated within objectives.

Transfer Presents new context and/or approach under different conditions, which elicits the same performances.Makes the utility of succeeding within the new context apparent.Presents cues within the



Marginally Meets

Meets Exceeds

new context, which aids learners in selecting and applying the appropriate previously-learned SKA.Clearly identifies the rewards succeeding within the new context.

___ Skill-Specific Conditions Established For Outcomes (5%)Based on the learning outcomes check the appropriate expectations of conditions found in the software for the skills, knowledge, and/or attitudes (SKA).

Specific Conditions Established


Marginally Meets

Meets Exceeds

Verbal Information

Uses term or definition in a sentence.Relates the information to preexisting knowledge.Presents all terms clearly using the fewest number of words to convey the meaning.Uses a variety of concrete examples when possible, emphasizing the well-defined features that relate directly to the information.Explains clearly how learners will be expected to recall the info while it is initially presented.Makes information readily accessible to learners, and provides opportunities for them to explore info



Marginally Meets

Meets Exceeds

associated with the knowledge.Practices verbal information with immediate feedback.

Intellectual Skills

Presents varied examples of concepts and rule applications, calling attention to distinctive features of examples, definitions, and procedures.Presents non-examples of the concept if they help to clarify the concept.Encourages learners to recall previously learned info or examples that illustrate concepts or rules being presented.Clearly communicates the definition of defined concepts, using the fewest number of words.Breaks down the process of performing or applying rules into steps, and clearly communicates these steps to the students.Demonstrates an application of the rule for the students.Provides learners with opportunities to play with concepts and rules within simulated or real environments.Presents a variety of



Marginally Meets

Meets Exceeds

contexts or experiences that allow the students to practice applying the rules or identifying concepts, with guidance early on.

Attitudes Clearly identifies examples of choices made by people who possess the desired attitudes.Clearly identifies instances in the students’ lives in which making choices are based on the attitude being presented.Makes students aware of the personal benefits gained by making choices based on attitudes.Allows students the opportunity to practice making choices associated with the desired attitude and gives them feedback.

Motor Skills

Verbally guides learners through the routine.Visually presents an example of routine execution.Practices motor skills with immediate feedback.Encourages the use of mental practice.

___ Practice With Feedback Available (10%)Based on review of the software check the option that best represents your opinion of each statement below.

1. Learners are informed when they submit a correct or incorrect response.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

2. Learners are informed what the correct response is.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

3. When learners submit an incorrect response, they are informed of the correct answer and offered information about the nature of their error.


4. Learners are informed of the correct response regardless of whether they got it right or not and are presented with additional information related to the correct response.


5. Learners are informed about the consequences of specific responses and may receive recommendations about future choices.


___ Effective General Message Principles Applied (10%)Based on your use of the software indicate your opinion for each of the following statements.

1. Screen display follows horizontal-vertical and left-right organization common to the culture of the intended audience.


2. Screens contain easily identifiable relevant information.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

3. Attention is drawn to those parts of a message intended to stand in contrast to the other screen elements.


4. Meaningful perceptual units are clustered in physical as well as temporal space.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

5. The rate of presenting sequential information is slow enough for accurate perception, scrutiny, elaboration, and comprehension, but not too slow to prevent attention from wandering.


6. Unfamiliar presentation modes are modeled for the user to develop familiarity.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

7. Messages are kept simple and not obscured by too much non-critical detail.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

___ Text Usage (6%)If there is text in the program, then indicate your opinion for each of the statements below. Otherwise ignore this section and automatically give the full weight value for text usage to the software.

1. Line breaks are dictated logically by the text, not by space available.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

2. Marginal headings and annotations are used to improve text organization.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

3. Text and background contrast are clear.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

4. Standard text formatting is used throughout, even for titles, headings, etc.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

___ Pictures and Illustrations Usage (6%)If there are pictures/illustrations in the program, then indicate your opinion for each of the statements below. Otherwise ignore this section and automatically give the full weight value for pictures/illustrations usage to the software.

1. A picture’s function is clearly designated.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

2. If necessary, specific instructions are included to interpret a picture and its intended function.


3. Illustrations are closely related to the context of the text.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

4. Representational pictures (e.g. line drawing of elephant) relevantly resemble their referent in terms of shape, detail, size, color, and context. Analogies are incorporated for users of limited experience with the referent.


5. Organizational pictures incorporate devices that stress temporal and spatial relationships.


___ Diagrams, Charts and Graphs Usage (6%)

If there are diagrams/charts/graphs in the program, then indicate your opinion for each of the statements below. Otherwise ignore this section and automatically give the full weight value for diagrams/charts/graphs usage to the software.

1. Abstract ideas and relationships are presented graphically in diagrams, charts, or graphs.


2. Diagrams are organized so that sequence of the parts viewed is in the necessary sequence.


3. Thicker lines designate stronger relationships.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

4. Attention is drawn to captions.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

5. The relative importance of elements is represented by relative sizes in the diagram.


6. Estimates or magnitude are made linear (as opposed to two or three dimensional).


7. The position of the elements corresponds to the perceived relationship among the elements.


8. Graphs depicting precise amounts utilize a Cartesian graph.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

9. Graphs depicting comparative amounts not requiring precision utilize pie charts.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

10. Trends are generally illustrated using line graphs.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

11. Comparisons are illustrated by bar graphs or pie charts.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

___ Audio Usage (6%)If there is audio in the program, then indicate your opinion for each of the statements below. Otherwise ignore this section and automatically give the full weight value for audio usage to the software.

1. Speech is utilized to effectively communicate ideas that have to be understood in a particular sequence.


2. Speech is not used to relay complex content better suited for text or text with illustrations.


3. Narration is closely related to the context of the text.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

___ Navigation (6%)If there is navigation in the program, then indicate your opinion for each of the statements below. Otherwise ignore this section and automatically give the full weight value for navigation to the software.

1. Clearly and quickly deciphered navigation schemes are used.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

2. A common navigation scheme is used throughout the program.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

3. Menu organization is broad and shallow.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

4. Unfamiliar navigation modes are modeled for the user to develop familiarity.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally

5. Complex or detailed navigation choices are explained to the learner.__ Not at all __ Marginally __ Satisfactorily __ Exceptionally


The development of a criteria of merit checklist based on the content of the ID expert review evaluation report seemed appropriate to choose categories (criterias) around the key sections of the report that were tied to specific evaluative content. So the first two sections were not really appropriate for the checklist. Neither were the summary evaluation sections. In addition C3 was excluded because the content did not seem as if it could be done in a checklist format. This left the 13 categories found in the checklist above.

Sub-criteria was determined by reviewing what each section of the report was evaluating. If the section referenced a table in one of the additional appendices, content from that table was used for the sub-criteria. Choosing which column content from the table to use as the sub-criteria was usually based on the column element appropriate to the references made from the report section. For example, the Context Types Evident category is based on section C2 of Appendix A: Notes on the Content and Organization of an ID Expert Review Evaluation Report. This references the context function column of Appendix B: Effective Instructional Design Strategies, so orienting, instructional, and transfer were the sub-criteria desired. In this particular situation, I felt it would be better to use a statement rather than a single word to facilitate anyone not familiar with using the checklist. Descriptive statements are used whenever possible in the checklist above for this reason.

The sub-criteria for the categories based on the sections found under the Instructional Message Display Description and Evaluation section of the ID expert review report are derived from the recommended description points provided in each section.

people.virginia.edupeople.virginia.edu/~jlk4p/itma/11_softeval/lesson8.docx · Web viewThe...

Documents

Transcript of people.virginia.edupeople.virginia.edu/~jlk4p/itma/11_softeval/lesson8.docx · Web viewThe...