Evaluation Research and Engineering Education
description
Transcript of Evaluation Research and Engineering Education
![Page 1: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/1.jpg)
Evaluation Research and Engineering Education
Lesley Jolly
For AaeE
ERM wiki at www.aaee-scholar.pbwiki.com
![Page 2: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/2.jpg)
What is evaluation research?
• Periodic assessment of results– Appropriateness, effectiveness, efficiency,
impact, sustainability
• Identifies intended and unintended effects
• Identifies what worked and what didn’t
• Provides level of judgement about overall worth of intervention
![Page 3: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/3.jpg)
What has it got to teach us?
• Systematic approach to what we’re doing already
• Extracting lessons learned
• Legacy of sustained evaluation frameworks for ongoing data collection
• Arguments against ‘popularity contest’ course and program evaluation
![Page 4: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/4.jpg)
Step 1: describe the program
• Aims and objectives– Need to be clearly articulated– Does everyone involved share the same aims
• Program logic diagram– Describes how we think the program
produces results– May be called logframe
![Page 5: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/5.jpg)
Program logic identifies what we need to evaluate
![Page 6: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/6.jpg)
Evaluating PBL tutor training
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS
Training
materials
Staff time
Staff support
workshops
Rooms/ space
Library support
[no
Administrative
support]
Published
research
Compiling
materials
Putting
Materials
online
Running
workshop
Staff research
Review and
edit materials
Staff reflection
Numbers of
new tutors
Numbers of
Returning
tutors
Level of
satisfaction
Growing
library of
examples and
materials
Tutors facilitate
rather than
inform
Tutors
Understand
PBL
Tutors develop
Knowledge
independently
Tutors make
PBL -
Appropriate
contributions toCurriculum
review
Students take
Responsibility
for learning
Students learn
to focus
research and
Synthesise
results
![Page 7: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/7.jpg)
Monitoring facilitates data collection
• Can be process (outputs) or impact (short to medium outcome) monitoring
• Need to develop indicators of progress• Targets may be included in indicators or
separate e.g.– In 2010 50% of 2nd yrs will have used the new facility
for more than 20 hrs– OR percentage of students using facility (indicator)– 50% in 2010 (target)
![Page 8: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/8.jpg)
PBL Tutor training
• GOAL: provide timely well-placed supportive guidance to encourage tutors to scaffold and facilitate student learning
• OBJECTIVES: at the end of training the successful tutor will be able to
• articulate a good understanding of the objectives and methods of PBL
• guide student learning through providing appropriate support and guidance rather than information
• contribute to curriculum development within a staff team
![Page 9: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/9.jpg)
Monitoring Tutor TrainingFOCUS PERFORMANCE
INDICATORSDATASOURCES
DATACOLLECTIONMETHODS
RESPONSIBILITY FOR COLLECTION
TIMEFRAME
Objective 1: tutors articulate a good understanding of the objectives and methods of PBL
OutcomesTutors facilitaterather thaninform
Tutor contributions to discussion
OnlineDiscussionlists
Content analysis of lists
Course co-ordinator Twice persemester
OutputsLevel ofsatisfaction
Likert: training hasclarified purposes of
PBL
Exit survey Paper survey post training
Trainer Everysemester
Objective 2: tutors guide student learning through providing appropriate support and guidance rather than information
OutcomesTutors facilitaterather thaninform
Students takeresponsibility forlearning
Tutor contributions to discussion
Student contributions to discussion
OnlineDiscussionlists
OnlineDiscussionlists
Content analysis of lists
Content analysis of lists
Course co-ordinator
Course co-ordinator
Twice persemester
Twice persemester
![Page 10: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/10.jpg)
Evaluation asks formative and summative questions
• Are the questions cohesive and logical?
• Do evaluation questions link to monitoring data?
• Have ethical issues been addressed?
• What mechanisms are in place to gather the learnings generated by evaluation?
• What needs to be retained from this evaluation process in future years?
![Page 11: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/11.jpg)
Vary questions to suit program
QUESTIONS SOURCES OF INFORMATION FROM MONITORING
SOURCES OFINFORMATIONFROM EVALUATIONDATA COLLECTION
DATA ANALYSISAND REPORTINGMETHODS
Is the program as implemented appropriate, and is it being appropriately monitored?
Has the purpose of the program been achieved?
Is the program being implemented in the most efficient and cost effective way possible?
Has the program produced sustainable results and outcomes?
![Page 12: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/12.jpg)
PBL Tutor Training• Appropriateness
• Did the training model the target behaviour?
• Has the purpose of the training been achieved?
• Indicator 1: changes in Tutor behaviour indicating deeper knowledge of and commitment to PBL
• Indicator 2: changes in student behaviour
• Efficiency• Was the time invested by staff good value?
• Sustainability• what needs to retained as core material from year
to year to retain benefit of training
![Page 13: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/13.jpg)
Owen, J. (2006) "Program Evaluation" Allen & Unwin: Crows Nest
Making use of evaluation
![Page 14: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/14.jpg)
Dissemination and reporting
• Findings may be communicated throughout project to multiple audiences– Evidence, conclusions, judgements,
recommendations.
• Different occasions call for different styles– Oral, interactive workshops, posters, reports,
summaries, papers, conference presentations
• Must be well timed
![Page 15: Evaluation Research and Engineering Education](https://reader035.fdocuments.in/reader035/viewer/2022081603/568150c4550346895dbee3c5/html5/thumbnails/15.jpg)
Developing capacity
• “process use of evaluation”– Taking part develops skills– Taking part sensitises staff to issues
• Improved communication– Ongoing 360 degree dissemination– Development of local discourse