1

 

Evaluation of Education Evaluation of Education Development ProjectsDevelopment Projects

CCLI PI MeetingCCLI PI MeetingAugust 15, 2008August 15, 2008

2

Caution

The information in these slides The information in these slides represents the opinions of the represents the opinions of the individual program directors and not individual program directors and not an official NSF position.an official NSF position.

3

Session Goals

The session willThe session will

Increase your understanding of evaluation Increase your understanding of evaluation

Enable you to collaborate more effectively Enable you to collaborate more effectively with evaluation experts in preparing and with evaluation experts in preparing and performing effective project evaluation plansperforming effective project evaluation plans

It will It will NOT NOT make you an evaluation expertmake you an evaluation expert

4

Perspective

Ideas here applicable to your project Ideas here applicable to your project whether in Week 5 or Year 5whether in Week 5 or Year 5

Consider your evaluation in light of what Consider your evaluation in light of what you discover hereyou discover here

Make adjustments to try to get the best Make adjustments to try to get the best evaluation data possibleevaluation data possible

5

Definitions AssessmentAssessment

Measurement of something (e. g., students’ Measurement of something (e. g., students’ ability to solve problemsability to solve problems

Usually does not carry a value judgmentUsually does not carry a value judgment Assessment instrument (tool)Assessment instrument (tool)

The method or device for measuring (e. g., a The method or device for measuring (e. g., a test; survey; portfolio)test; survey; portfolio)

Evaluation (e-VALUE-ation)Evaluation (e-VALUE-ation) A judgment of something based upon results A judgment of something based upon results

from one or more assessmentsfrom one or more assessments Frequently compared to an expected outcomeFrequently compared to an expected outcome

6

Evaluation and Assessment Evaluation (assessment) is used in many waysEvaluation (assessment) is used in many ways

Individual’s Individual’s performance (grading)performance (grading) Program’s Program’s effectiveness (ABET; regional effectiveness (ABET; regional

accreditation)accreditation) Project’s Project’s progress & accomplishmentsprogress & accomplishments (monitoring & (monitoring &

validating)validating)

Session addresses Session addresses project evaluationproject evaluation May involve evaluating individual and group May involve evaluating individual and group

performance – but in theperformance – but in the context of the project context of the project

Project evaluation Project evaluation Formative Formative – monitoring progress– monitoring progress Summative Summative – characterizing final accomplishments– characterizing final accomplishments

7

Evaluation and Project Goals & Outcomes

Evaluation starts with carefully defined project goals Evaluation starts with carefully defined project goals & expected (measurable) outcomes& expected (measurable) outcomes

Goals & expected outcomes related to:Goals & expected outcomes related to: Project managementProject management outcomes outcomes

Initiating or completing an activityInitiating or completing an activityFinishing a “product”Finishing a “product”

Expected sExpected student outcomestudent outcomesModifying a learning outcomeModifying a learning outcomeModifying attitudes or perceptionsModifying attitudes or perceptions

Workshop focuses on student outcomesWorkshop focuses on student outcomes

8

Evaluation and Project Evaluation and Project Goals/Outcomes/QuestionsGoals/Outcomes/Questions

9

Developing Student Behavior Goals & Outcomes

Start with one or more overarching statements Start with one or more overarching statements of project intentionof project intention Each statement is a Each statement is a GOALGOAL

What is your overall ambition? What do you What is your overall ambition? What do you hope to achieve?hope to achieve?

Convert each goal into one or more specific Convert each goal into one or more specific expected measurable resultsexpected measurable results Each result is an Each result is an EXPECTED OUTCOMEEXPECTED OUTCOME

How will achieving your “intention” reflect in How will achieving your “intention” reflect in student behavior?student behavior?

10

Goals – Objectives – Outcomes -- Questions

Converting goals to expected outcomes Converting goals to expected outcomes may involve intermediate stepsmay involve intermediate steps Intermediate steps maybe called Intermediate steps maybe called objectivesobjectives

More specific, more measurable than goalsMore specific, more measurable than goalsLess specific, less measurable than Less specific, less measurable than

outcomesoutcomes Expected outcomes lead to Expected outcomes lead to questionsquestions

These form the basis of the evaluationThese form the basis of the evaluation Evaluation process Evaluation process collects and interpretscollects and interprets

data to answer evaluation questionsdata to answer evaluation questions

11

Exercise

Identification of Goals and Expected Outcomes

Read the abstract Read the abstract Note - Goal statement removedNote - Goal statement removed

Suggest two plausible goalsSuggest two plausible goals One focused on a change in learning One focused on a change in learning One focused on a change in some other One focused on a change in some other

aspect of student behavioraspect of student behavior

Use student not instructor focusUse student not instructor focus

12

Abstract The goal of the project is …… The project is developing computer-The goal of the project is …… The project is developing computer-

based instructional modules for statics and mechanics of materials. based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of project is being disseminated through … The broader impacts of the project are …the project are …

Substitute sampleSubstitute sample““organic chemistry” for “statics and mechanics of materials”organic chemistry” for “statics and mechanics of materials”““Interactions” for “external forces and internal reactions”Interactions” for “external forces and internal reactions”““molecules” for “objects”molecules” for “objects”

Focus student not instructor perspectiveFocus student not instructor perspective

13

PD’s Response - Goals Goals may focus onGoals may focus on

Cognitive changesCognitive changes Knowledge-driven Knowledge-driven Skill-driven Skill-driven

AAffective (or affective (or attitudinalttitudinal)) changes changes Success changesSuccess changes Diversity ChangesDiversity Changes

14

PD’s Response

Goals - Cognitive Change

GOALGOAL: Improve understanding or : Improve understanding or skillsskills

In the context ofIn the context of the course the course Describe verbally the effect of external Describe verbally the effect of external

forces on a solid objectforces on a solid object Solve textbook problems Solve textbook problems

In application In application beyond coursebeyond course Solve out-of-context problems Solve out-of-context problems Visualize 3-D problemsVisualize 3-D problems Communicate technical problems orallyCommunicate technical problems orally

15

PD’s Response Goals on Attitudinal Changes

GOALGOAL: Improve: Improve

Interest in the courseInterest in the course Attitude about Attitude about

ProfessionProfession CurriculumCurriculum DepartmentDepartment

Self- confidenceSelf- confidence Intellectual developmentIntellectual development

16

PD’s Response Goals on Success Changes

Goals: Improve Goals: Improve Recruitment ratesRecruitment rates Retention or persistence ratesRetention or persistence rates Graduation ratesGraduation rates

17

PD’s Response Goals on Diversity

““Broaden the participation of Broaden the participation of underrepresented groups”underrepresented groups”

GOAL:GOAL: To change a target group’s To change a target group’s Cognitive abilities (understanding Cognitive abilities (understanding

or skills)or skills)

AttitudesAttitudes

Success ratesSuccess rates

18

Exercise

Transforming Goals into Expected Outcomes

Write one expected measurable outcome Write one expected measurable outcome for each of the following goals:for each of the following goals:

1.1. Increase the students’ understanding Increase the students’ understanding of the concepts in staticsof the concepts in statics

1.1. Improve the students’ attitude about Improve the students’ attitude about engineering as a careerengineering as a career

19

PD’s Response Expected Outcomes

Conceptual understandingConceptual understanding Students will be better able to solve Students will be better able to solve

conceptual problems that don’t require the conceptual problems that don’t require the use of formulas or calculationsuse of formulas or calculations

Students will be better able to solve out-of-Students will be better able to solve out-of-context problems.context problems.

AttitudinalAttitudinal Students will be more likely to describe Students will be more likely to describe

engineering as an exciting careerengineering as an exciting career The percentage of students who transfer out The percentage of students who transfer out

of engineering after the statics course will of engineering after the statics course will decrease. decrease.

20

ExerciseTransforming Expected Outcomes

into Questions

Write a question for these expected Write a question for these expected measurable outcomes:measurable outcomes:

1.1. Students will be better able to solve Students will be better able to solve conceptual problems that do not require the conceptual problems that do not require the use of formulas or calculationsuse of formulas or calculations

2.2. In informal discussions, students will be In informal discussions, students will be more likely to describe engineering as an more likely to describe engineering as an exciting careerexciting career

21

PD’s Response

Questions

Conceptual understandingConceptual understanding

Did the students’ ability to solve Did the students’ ability to solve conceptual problems increase ?conceptual problems increase ?

Did the students’ ability to solve Did the students’ ability to solve conceptual problems increase conceptual problems increase because because of the use of the 3D rendering and of the use of the 3D rendering and animation softwareanimation software??

22

PD’s Response - Questions

AttitudinalAttitudinal Did the students discussions indicate Did the students discussions indicate

more excitement about engineering as more excitement about engineering as a career?a career?

Did the students discussions indicate Did the students discussions indicate more excitement, about engineering as more excitement, about engineering as a career a career because of the use of the 3D because of the use of the 3D rendering and animation software?rendering and animation software?

23

Reflection

What is the most surprising idea What is the most surprising idea

you heard in this session?you heard in this session?

24

Evaluation ToolsEvaluation Tools

25

FLAG (Field-Tested Learning Assessment Guide)

A Primer of Assessment & EvaluationA Primer of Assessment & Evaluation

Classroom Assessment Techniques (CATS)Classroom Assessment Techniques (CATS) QualitativeQualitative QuantitativeQuantitative

Searchable & Down-Loadable ToolsSearchable & Down-Loadable Tools

Resources in AssessmentResources in Assessment

http://www.flaguide.orghttp://www.flaguide.org

26

FLAG Classroom Assessment Techniques (CATS)

Attitudinal SurveyAttitudinal Survey Concept TestsConcept Tests Concept MappingConcept Mapping Conceptual Diagnostic TestsConceptual Diagnostic Tests InterviewsInterviews Performance Assessment PortfolioPerformance Assessment Portfolio Scoring RubricsScoring Rubrics Weekly ReportsWeekly Reports

http://www.flaguide.orghttp://www.flaguide.org

27

SALG (Student Assessment of Learning Gains)

Assesses perceived degree of “gain” students Assesses perceived degree of “gain” students made in specific aspects of the classmade in specific aspects of the class

Spotlights course elements that best support Spotlights course elements that best support student learning and those needing improvementstudent learning and those needing improvement

Web-based instrument requiring 10-15 minutes to Web-based instrument requiring 10-15 minutes to useuse

Easily modified by the instructorEasily modified by the instructor Provides instant statistical analysis of resultsProvides instant statistical analysis of results Facilitates formative evaluation throughout course Facilitates formative evaluation throughout course

http://www.salgsite.orghttp://www.salgsite.org

28

Concept Inventories (CIs)Concept Inventories (CIs)

29

Introduction to CIs A tool that measures A tool that measures conceptual conceptual

understandingunderstanding Series of multiple choice questionsSeries of multiple choice questions

Questions involve Questions involve single conceptsingle concept Formulas, calculations, or problem solving not Formulas, calculations, or problem solving not

requiredrequired

Possible answers include Possible answers include ““detractorsdetractors”” Common errors Common errors Reflect commonReflect common “ “misconceptions”misconceptions”

Force Concept Inventory (FCI) is the prototypeForce Concept Inventory (FCI) is the prototype

30

  Exercise

Evaluating a CI Tool

Suppose you where considering an Suppose you where considering an existing CI for use in your project’s existing CI for use in your project’s evaluationevaluation

What questions would you consider in What questions would you consider in deciding if the tool is appropriate?deciding if the tool is appropriate?

31

PD’s Response

Evaluating a CI Tool Nature of the toolNature of the tool

Is the tool relevant to what was taught? Is the tool relevant to what was taught? Is the tool competency based? Is the tool competency based? Is the tool conceptual or procedural?Is the tool conceptual or procedural?

Prior validation of the toolPrior validation of the tool Has the tool been tested? Has the tool been tested? Is there information on reliability and validity? Is there information on reliability and validity? Has it been compared to other tools? Has it been compared to other tools? Is it sensitive? Does it discriminate novice and expert?Is it sensitive? Does it discriminate novice and expert?

Experience of others with the toolExperience of others with the tool Has the tool been used by others besides the Has the tool been used by others besides the

developer? At other sites? With other populations? developer? At other sites? With other populations? Is there normative data?Is there normative data?

32

Decision Factors for Other Tools

Would these questions be different Would these questions be different for another tool?for another tool?

Interpreting Evaluation Data

34

Hypothetical Concept Inventory Data

Pre Post Pre Post1 35 32 29% 23%2 35 32 34% 65%3 35 32 74% 85%

- - - - -

Quest

35

Exercise

Alternate Explanation For Change

Data suggests that the understanding Data suggests that the understanding of Concept #2 increasedof Concept #2 increased

One interpretation is that the One interpretation is that the intervention caused the changeintervention caused the change

List some List some alternative explanationsalternative explanations Confounding factorsConfounding factors Other factors that could explain the Other factors that could explain the

changechange

36

  PD's Response

Alternate Explanation For Change

Students learned concept out of class (e. g., in Students learned concept out of class (e. g., in another course or in study groups with students another course or in study groups with students not in the course)not in the course)

Students answered with what the instructor wanted Students answered with what the instructor wanted rather than what they believed or “knew” rather than what they believed or “knew”

An external event (big test in previous period or a An external event (big test in previous period or a “bad-hair day”) distorted pre- or post-test data“bad-hair day”) distorted pre- or post-test data

Instrument was unreliableInstrument was unreliable Other changes in course and not the intervention Other changes in course and not the intervention

caused improvementcaused improvement Students not representative groupsStudents not representative groups

37

Exercise

Alternate Explanation for Lack of Change

Data suggests that the understanding Data suggests that the understanding of Concept #1 did not increase of Concept #1 did not increase

One interpretation is that the One interpretation is that the intervention did cause a change but it intervention did cause a change but it was masked by other factorswas masked by other factors

List some List some confounding factorsconfounding factors that that could have masked a real changecould have masked a real change

38

 PD's Response

Alternate Explanations for Lack of Effect

An external event (big test in previous period An external event (big test in previous period or a “bad-hair day”) distorted pre- or post-or a “bad-hair day”) distorted pre- or post-test datatest data

The instrument was unreliableThe instrument was unreliable Implementation of the intervention was poorImplementation of the intervention was poor Population too smallPopulation too small One or both student groups not One or both student groups not

representative representative Formats were different on pre and post testsFormats were different on pre and post tests

39

Reflection

What is the most surprising idea What is the most surprising idea

you heard in this session so far?you heard in this session so far?

40

Evaluation PlanEvaluation Plan

41

Exercise

Evaluation Plan

Read the evaluation plan Read the evaluation plan and suggest improvementsand suggest improvements

42

Exercise - Evaluation Plan The results of the project will be evaluated in several ways. The results of the project will be evaluated in several ways.

First, students will be surveyed at the end of the semester on First, students will be surveyed at the end of the semester on the content, organization, continuity of the topics, and the the content, organization, continuity of the topics, and the level of difficulty. Since the course can be taken as dual level of difficulty. Since the course can be taken as dual enrollment by local high school students, it is expected that enrollment by local high school students, it is expected that the course will attract more students to the college. Due to the course will attract more students to the college. Due to the urban setting of Grant College, a majority of these the urban setting of Grant College, a majority of these students will be working part time, and they will be asked students will be working part time, and they will be asked about the appropriateness of the content and its relevancy in about the appropriateness of the content and its relevancy in their jobs.their jobs.

Second, the professors teaching the subsequent advanced Second, the professors teaching the subsequent advanced bioinformatics lecture courses will be asked to judge the bioinformatics lecture courses will be asked to judge the students’ ability to apply bioinformatics. While the projects students’ ability to apply bioinformatics. While the projects in these upper level classes focus on different application in these upper level classes focus on different application areas, the problems frequently involve concepts initially areas, the problems frequently involve concepts initially learned in the new bioinformatics laboratory. The current learned in the new bioinformatics laboratory. The current advanced bioinformatics instructors have taught the course advanced bioinformatics instructors have taught the course for several years and are qualified to compare the future for several years and are qualified to compare the future students’ abilities with their previous students.students’ abilities with their previous students.

43

PD’s Responses - Evaluation Plan

Tie evaluation to expected outcomesTie evaluation to expected outcomes Include measures of student learningInclude measures of student learning Include capturing the demographics of Include capturing the demographics of

the population the population Use an “external” evaluator for objectivityUse an “external” evaluator for objectivity Describe processes for formative Describe processes for formative

evaluation and summative evaluationevaluation and summative evaluation Consider including beta test at one or Consider including beta test at one or

more other sitesmore other sites Include impact statementInclude impact statement

44

References NSF’s User Friendly Handbook for Project NSF’s User Friendly Handbook for Project

EvaluationEvaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htmhttp://www.nsf.gov/pubs/2002/nsf02057/start.htm

Field-Tested Learning Assessment Guide (FLAG)Field-Tested Learning Assessment Guide (FLAG) http://www.flaguide.orghttp://www.flaguide.org

Online Evaluation Resource Library (OERL)Online Evaluation Resource Library (OERL) http://oerl.sri.com/http://oerl.sri.com/

CCLI Evaluation Planning WebinarCCLI Evaluation Planning Webinar http://oerl.sri.com/ccli_resources.htmlhttp://oerl.sri.com/ccli_resources.html

SALG (Student Assessment of Learning GainsSALG (Student Assessment of Learning Gains http://http://www.salgsite.orgwww.salgsite.org

45

Questions?Questions?

46