1

An Evaluation Plan and Tool Kit for the Archibald Bush Innovative Teaching and

Technology Strategies Grant

Valerie Ruhe and J.D. Walker, Center for Teaching and Learning Services and

Digital Media Center

University of Minnesota

Evaluation Plan Evaluation: A formal appraisal of the

quality of an educational phenomenon (Popham, 1993)

Gathering evidence to determine how well the innovation is “working”.

What does the grant say about the evaluation plan?

Foci of the Bush Grant Course-level Evaluation

Student Outcomes Learning Process:

Student EngagementReflective Learning

Responsible Learning

The Bush grant re: Evidence Grades—ABC and DFW rates (outcomes). Faculty Reflection Logs and conversations

with consultants (process) Surveys e.g. compare engagement across

groups: with/without innovation and/or: Surveys to compare across time, e.g.

before/after the innovation.

Survey Items: Closed-ended (Examples) Student Engagement Mostly, I come to class because I want the

certificate/degree. Most of the time, I enjoy this class.

Reflective Learning I enjoy applying theories or concepts from this class to new

situations. I sometimes find myself thinking about the lecture after it

has ended.

Responsible Learning I usually come to class prepared. I usually review my lecture notes after class.

Survey Items: Open-ended What are the most important benefits of

this innovation for you? What are the drawbacks? What kind of problems have you had

with the technology? Be specific. What suggestions do you have for

improvement?

Tailoring the Evaluation Plan to Meet Your Needs What do you want to know? Collect the kind of evidence to find out what

you want to know, for example: Business: experimental design with two

treatment groups and long pre- and post-intervention surveys.

Architecture: a series of short surveys, interviews and observations, then analyze them for recurring themes.

Optional Evaluation Methods Scoring Rubrics for student assignments Class Observation: take notes, analyze them

for themes, can inform surveys. Narrative: To learn about learners’ reflective

practices. Triangulation is important. Interviews: Open-ended items, can deviate,

tape-record and analyze for themes. Focus groups: Validity checks.

Implementing the Plan Decide what you want to know Choose your preferred methods/tools Write surveys and/or interview protocols Collect data Data entry: training? Data analysis

Tools: Scoring Rubrics “Scoring rubrics…guide the analysis of the

products and/or processes of students' efforts…[and] provide a description of what is expected at each score level.” (Moskal, 2000)

a systematic way of evaluating qualitative data; a way of reducing subjectivity

Tools: Scoring RubricsDevelopment process:

break general concepts down into evaluation criteria

develop descriptions of degrees to which, and ways in which, the criteria can be satisfied

conduct pilot testing on sample data

Tools: Surveys

Survey questions should: be interpreted by respondents in the same

way; ask for information that respondents are

able to provide; provide information that is interpretable by

you.

Tools: Questionnaires

Time estimates: How many hours per day do you typically study?

Less than 1 hour 1 – 1.5 hours 1.5 – 2 hours 2 - 2.5 hours More than 2.5 hours

Less than 2.5 hours 2.5 – 3 hours 3 – 3.5 hours 3.5 – 4 hours More than 4 hours

Tools: QuestionnairesScale issues: How would you rate your

instructor’s knowledge of the subject matter of this class?

poor fair adequate good very good exceptional

Tools: QuestionnairesScale issues: How effective were the

small group discussions in helping you to learn the course material?

very effective effective don’t know ineffective very ineffective

Tools: QuestionnairesItem order effects: Which of the following

activities helped you to learn the course material?

Lectures Large group discussions Small group discussions Group assignments Course readings Studying for quizzes

Tools: QuestionnairesOpen versus closed-ended questions:

How easy or difficult to use did you find each of the components of our course website?

Tools: QuestionnairesOpen versus closed-ended questions:

How easy or difficult to use did you find these components of our course website?

Discussions very easyQuizzes easyContent modules difficultAudio files very difficultPowerPoint filesetc.

Tools: QuestionnairesInterpretability: Which of these statements

best describes your professor?

very good teacher, but not very approachable about the same on approachability and

teaching quality very approachable, but not a very good teacher a born teacher extremely approachable