1
An Evaluation Plan and Tool Kit for the Archibald Bush Innovative Teaching and
Technology Strategies Grant
Valerie Ruhe and J.D. Walker, Center for Teaching and Learning Services and
Digital Media Center
University of Minnesota
Evaluation Plan Evaluation: A formal appraisal of the
quality of an educational phenomenon (Popham, 1993)
Gathering evidence to determine how well the innovation is “working”.
What does the grant say about the evaluation plan?
Foci of the Bush Grant Course-level Evaluation
Student Outcomes Learning Process:
Student EngagementReflective Learning
Responsible Learning
The Bush grant re: Evidence Grades—ABC and DFW rates (outcomes). Faculty Reflection Logs and conversations
with consultants (process) Surveys e.g. compare engagement across
groups: with/without innovation and/or: Surveys to compare across time, e.g.
before/after the innovation.
Survey Items: Closed-ended (Examples) Student Engagement Mostly, I come to class because I want the
certificate/degree. Most of the time, I enjoy this class.
Reflective Learning I enjoy applying theories or concepts from this class to new
situations. I sometimes find myself thinking about the lecture after it
has ended.
Responsible Learning I usually come to class prepared. I usually review my lecture notes after class.
Survey Items: Open-ended What are the most important benefits of
this innovation for you? What are the drawbacks? What kind of problems have you had
with the technology? Be specific. What suggestions do you have for
improvement?
Tailoring the Evaluation Plan to Meet Your Needs What do you want to know? Collect the kind of evidence to find out what
you want to know, for example: Business: experimental design with two
treatment groups and long pre- and post-intervention surveys.
Architecture: a series of short surveys, interviews and observations, then analyze them for recurring themes.
Optional Evaluation Methods Scoring Rubrics for student assignments Class Observation: take notes, analyze them
for themes, can inform surveys. Narrative: To learn about learners’ reflective
practices. Triangulation is important. Interviews: Open-ended items, can deviate,
tape-record and analyze for themes. Focus groups: Validity checks.
Implementing the Plan Decide what you want to know Choose your preferred methods/tools Write surveys and/or interview protocols Collect data Data entry: training? Data analysis
Tools: Scoring Rubrics “Scoring rubrics…guide the analysis of the
products and/or processes of students' efforts…[and] provide a description of what is expected at each score level.” (Moskal, 2000)
a systematic way of evaluating qualitative data; a way of reducing subjectivity
Tools: Scoring RubricsDevelopment process:
break general concepts down into evaluation criteria
develop descriptions of degrees to which, and ways in which, the criteria can be satisfied
conduct pilot testing on sample data
Tools: Surveys
Survey questions should: be interpreted by respondents in the same
way; ask for information that respondents are
able to provide; provide information that is interpretable by
you.
Tools: Questionnaires
Time estimates: How many hours per day do you typically study?
Less than 1 hour 1 – 1.5 hours 1.5 – 2 hours 2 - 2.5 hours More than 2.5 hours
Less than 2.5 hours 2.5 – 3 hours 3 – 3.5 hours 3.5 – 4 hours More than 4 hours
Tools: QuestionnairesScale issues: How would you rate your
instructor’s knowledge of the subject matter of this class?
poor fair adequate good very good exceptional
Tools: QuestionnairesScale issues: How effective were the
small group discussions in helping you to learn the course material?
very effective effective don’t know ineffective very ineffective
Tools: QuestionnairesItem order effects: Which of the following
activities helped you to learn the course material?
Lectures Large group discussions Small group discussions Group assignments Course readings Studying for quizzes
Tools: QuestionnairesOpen versus closed-ended questions:
How easy or difficult to use did you find each of the components of our course website?
Tools: QuestionnairesOpen versus closed-ended questions:
How easy or difficult to use did you find these components of our course website?
Discussions very easyQuizzes easyContent modules difficultAudio files very difficultPowerPoint filesetc.
Tools: QuestionnairesInterpretability: Which of these statements
best describes your professor?
very good teacher, but not very approachable about the same on approachability and
teaching quality very approachable, but not a very good teacher a born teacher extremely approachable
Top Related