Post on 14-Apr-2017
Induction to Pedagogy
Evaluation of the Learning Resources
Introduction
SYNERGY Exchange (http://synergyexchange.eu/)
Created for “SYNERGY”, an ERASMUS+ project
Module Overview & Main Objectives
PARTS OBJECTIVES
Introduction Module Review Main Objectives
Unit I:
Basic Definitions
Basic definitions Identification of main evaluation criteria Examples of their application
Unit II:
Methods of Evaluation
The 5 WH- questions model Presentation and implementation
Unit III:
Other Models and Tools
The MERLOT model The LORI model The EFQUEL network
Unit IV:
Practical Tips & Final Remarks Create your own evaluation criteria
Induction to Pedagogy
Evaluation of the Learning Resources
Unit I: Basic Definitions
SYNERGY Exchange (http://synergyexchange.eu/)
Created for “SYNERGY”, an ERASMUS+ project
4
What is evaluation?
Evaluating is the process of determining merit, worth, or significance; an evaluation is a product of that process.
(Scriven 2007)
5
What is a learning resource?
A learning resource is information represented, accessible or stored in a variety of media and
formats (texts, videos, software etc.), which assists an individuals’ learning as defined by the learning outcomes and/or the overall learning objectives.
6
Why do we evaluate resources?
The main purpose of an evaluation is to determine the quality of a learning resource. The quality of a learning resource is dependent on: • its properties and features • the context of the resource such as time, place and
human relationships. Quality is also connected to how the resource is used, when it is used, and in relation to whom it is used.
7
Why do we evaluate resources?
• We undertake an evaluation to check and measure the quality of something.
• Quality depends on two different types of factors: • There are intrinsic factors, i.e. internal
features of the object we have to evaluate;
• There are external factors, i.e. external and contextual conditions.
• Quality, therefore, is not a fixed idea with a fixed value; quality is a relative concept, which varies depending on the intrinsic and extrinsic factors.
8
How do we evaluate?
Evaluation is not a single and isolated process. It is a systematic approach, based on a number of criteria such as:
• Potential Effectiveness as a Learning Tool; • Ease of Use (for Practitioners and Learners); • Accuracy; • Appropriateness; • Clarity; • Completeness, etc. See Unit II for an example of
evaluation process and set
of criteria
9
FOR/ BY WHOM (target groups)
TO DO WHAT (objectives)
IN WHICH CONDITIONS (constraints)
Example: • FOR 5 welders • BY 1 external
expert
Example: To be able to satisfy a new customer.
Example: • Very close deadline • Limited time by the trainer • Very busy period for
production
The 3 main aspects to evaluate?
We can start approaching evaluation by analysing three main aspects, namely the objective(s); the target group(s) and the context
10
Exercise 1
To complete this quiz, rate the importance of the factors which are provided in the table, based on the descriptions given. Rating is between 1 and 5, where 5 is most important and 1 is least important.
FACTOR DESCRIPTION RANKING
TARGET GROUPS • Training for7 administrative staff from 2
companies
• To be facilitated by an external tutor
2
1
OBJECTIVES • Prepare the administrative staff of 2 companies
to manage customers from the Russian market
5
CONSTRAINTS • 3 months to deliver the programme;
• A mininum of a grade A2 to required from each
learner to pass the programme;
• There is a low-medium budget allocated.
4
5
5
11
Induction to Pedagogy
Evaluation of the Learning Resources
SYNERGY Exchange (http://synergyexchange.eu/)
Created for “SYNERGY”, an ERASMUS+ project
Unit II: Methods of Evaluating
12
The Scope of this unit is to perform a more in-depth analysis of the learning resources without getting into complex and time-consuming studies and preliminary preparation.
Adapt a well-known outline for structuring written texts to the evaluation process, as it is based on a small set of key points that can correspond to key aspects of resources to be evaluated.
Suggest possible improvement.
Scope & Objectives
Objectives
13
This methodology is an adaptation of the 5-W question model which can be used to help in planning, writing and revising text: It is based on 5 key factors (who, what, why, when,
where) to identify and analyse the 5 most important factors that:
(a) define the context of the learning resource by establishing how it was used;
(b) affect and determine its quality.
The 5 ‘W’ Question Model
14
• Who are the learners? (age; role; background, etc.)
• Who is their possible trainer? (internal, external etc.) WHO
• What kind of resources do they need most?
• Are online resources easy to access for them? WHAT
• Outline the training objectives.
• Is the training being delivered a new or existing curriculum? WHY
• When can the training be performed?
• When is the deadline for completing the assignments? WHEN
• Planning where the training will take place and ensuring all equipment (incl. technical) are in place for training. WHERE
Asking the 5 ‘W’ Questions
15
• The following slide shows an example of how the 5-WH grid can be completed.
• The grid guides you to focus your attention on different factors and elements that can determine the quality of the resources you have to evaluate. • WHO corresponds to the target group the resources
are addressed to; • WHAT and WHY correspond to the learning objectives
and content of the learning; • WHEN and WHERE correspond to the planning and
implementation of the learning.
Sample Answers for the 5 ‘W’ Questions
16
• 9 individual aged 25-47; administrative staff, no previous experience in project designing.
• No internal trainer available. WHO
• They need resources for project design.
• They are familiar to online resources and know how to access them.
WHAT
• This training is from an established curriculum, therefore the training outline has already been developed to adhere to external quality standards.
WHY
• The training will take place from 9.00am until 5.30pm on Wednesdays for six consecutive weeks. WHEN
• The training will take place in a local training centre.
• Wi-fi connection is available and there are facilities for PowerPoint.
WHERE
Answering the 5 ‘W’ Questions
17
Possible findings from the 5 ‘W’s
• On the basis of the previous assessment using the 5-WH method, one can deduce that the proposed learning resources should be: Intended to support and guide the learning
process of individuals involved in a training programme;
Available and accessible online; Relevant to the learner’s needs; Practical and should be applicable to real-world
working situations, or contain real-world case-studies.
18
Exercise using the 5 ‘W’s
The following slides contains another short exercise relating to the 5 ‘W’s. • In the table provided, one column contains 5
statements which form sample answers to the 5 ‘W’ Questions.
• In the next column, there is a list of learning resources. • The aim of this exercise is to match the data from the 5
statements and identify which learning resource is most suitable.
• You have 10 minutes to complete this exercise.
19
Sample 5 ‘W’ Answers Sample Resources
1. 10 low-skilled workers of a company working in
cleaning and gardening sector, all aged over 55
Online resources
including vdeos and
simulations could be
very useful but are not
suitable to the target
group;
Leaflets with images,
graphs and lists of
content & check-lists
could be better.
Mobile learning can
be used (where
applicable) to help to
organise the workers’
learning tasks
2. Health, safety and first-aid course; 2 of the learners
have to become “health & safety manager” of their
unit
3. To comply with safety regulations which are legally
required
4. Training will take place at a time that suits the
workers because it is urgently needed by this is the
company’s peak time for contract work
5. The company has access to training facilities and
computer labs where resources can be accessed
Exercise 2: Find the Right Resource
20
Induction to Pedagogy
Evaluation of the Learning Resources
SYNERGY Exchange (http://synergyexchange.eu/)
Created for “SYNERGY”, an ERASMUS+ project
Unit III: Other models and tools
21
The scope of this unit is to provide a set of criteria for assessing the quality of learning resources. These criteria are used in conjunction with the following tools:
• MERLOT • LORI
This unit also aims to explore the opportunities and services available for e-learning resources via the EFQUEL network.
Analyse the quality of selected learning resources to decide which are more respondent to the identified needs.
Scope & Objectives
Objectives
22
Introducing MERLOT
• MERLOT stands for Multimedia Education Resource for Learning and Online Teaching.
• MERLOT is an online repository. It has been developed by an international consortium of institutions (and systems) of higher education, industry partners, professional organizations and individuals.
• MERLOT is a community of staff, volunteers, and members who work together in various ways to provide users of OER (Open Educational Resource) teaching and learning materials with a wealth of services and functions that can enhance their instructional experience.
23
Introducing MERLOT
MERLOT’s partners and members are devoted to identifying, peer reviewing, organising and making available existing online learning resources in a range of academic disciplines for use by higher education faculty and students. These disciplines include:
1. Arts
2. Business
3. Education
4. Humanities
5. Mathematics and Statistics
6. Science and Technology
7. Social Sciences.
These resources can be accessed by visiting www.merlot.org
24
MERLOT Quality Insurance Process
• All MERLOT materials are examined in various ways to ensure that they are useful for the MERLOT community.
• The Quality Assurance Process in based on Peer Reviewing. • The Peer Review process is led by an Editor, and includes
editorial board members and peer reviewers.
• The MERLOT editorial boards provide leadership, tools, and training in developing evaluation standards and processes for peer review which include: • Quality of content • Potential effectiveness as a teaching tool • Ease of use
25
Evaluation Standards Key Questions
Quality of Content
1. Does the resource present valid (correct) concepts, models and skills?
2. Does the resource present educationally significant concepts, models and skills for the discipline?
Potential Effectiveness as a
Teaching-Learning Tool
1. In what stage(s) in the learning process/cycle
could the materials be used?
2. What is(are) the learning objective(s)? What should students be able to do after successfully learning with these materials?
3. What are the characteristics of the target group(s)?
Ease of Use 1. How easy it is for teachers and students to use
the resource for the first time?
MERLOT Quality Criteria
26
The second model and method is LORI. • LORI stands for Learning Object Review Instrument. • It is available through e-LERA (www.eLera.net), and is a
collaboration between a community of teachers, students, university faculty, researchers and instructional designers who all share an interest in the use and evaluation of learning resources.
• LORI is an effective tool for evaluating learning resources. • The reviews, in turn, facilitate teachers’ searches for relevant
and high quality resources, and also provide a means of developing best practices in the development of learning resources.
Introducing LORI
27
eLera is a collaboration among a group that researches and
evaluates e-learning. Their specific interests include online learning resources, e-
portfolios and learning design specifications, and related topics.
eLera provides tools and information for learning resource evaluation and research, maintains a database of learning resource reviews, and supports communication and collaboration among researchers, evaluators and users of online learning resources.
Introducing eLera
28
• LORI provides an easy-to-follow list of broad, yet targeted categories that allow reviewers to quickly assess a learning resource’s strengths and weaknesses.
• Each item has an associated 5-point rubric, and reviewers are also encouraged to provide comments with their reviews.
• Version 1.5 of LORI includes the 9 items which are as follows:
1. Content Quality 2. Learning Goal Alignment 3. Feedback and Adaptation 4. Motivation 5. Presentation Design 6. Interaction Usability 7. Accessibility 8. Reusability 9. Standards Compliance
LORI Quality Dimensions
29
1. Content Quality Authentic, relevant, accuracy, balanced presentation of
ideas, and appropriate level of detail
2. Learning Goal
Alignment
Alignment among learning goals, activities, assessments,
and learner characteristics.
3. Feedback and
Adaptation
Adaptive content or feedback driven by differential
learner input or learner modelling
4. Motivation Ability to motivate and interest an identified population
of learners
5. Presentation Design Design of visual and auditory information for enhanced
learning and efficient mental processing
LORI Quality Dimensions
30
6. Interaction Usability Ease of navigation, predictability of the user interface,
and quality of the interface help features
7. Accessibility Design of controls and presentation formats to
accommodate disabled and mobile learners
8. Reusability Ability to use in varying learning contexts and with
learners from differing backgrounds
9. Standards
Compliance Adherence to international standards and specifications
LORI Quality Dimensions
31
• EFQUEL is the leading international network in the field of quality in education.
• EFQUEL’s services cover training, review services and conformity checks for higher education and other educational organisations and consulting, as well as certification through EFQUEL’s E-Quality Label.
Introducing EFQUEL
For more information, visit: www.efquel.org
32
• EFQUEL has developed a service portfolio reacting to its members’ needs.
• The services are divided into three categories: 1. Quality services: Quality Services are designed to help members to improve educational quality through using ICT 2. Knowledge services: These provide members with comprehensive information on current developments in several areas of educational innovation, such as quality for open education, MOOCs, or quality management for technology-enhanced learning. 3. Training services: EFQUEL organises workshops, expert seminars, short courses and summer schools in EU countries
Introducing EFQUEL
33
• The E-Quality Review Services are providing organisations with individual experts’ reviews and consultancy, as well as compliance checks for courses, programmes or departments.
• Often these are considered as a first step into certification.
e-Quality Label & Review Service
34
Delivering an official certificate
Being free
Meeting the needs of the target group
Validated by a recognized authority
Easiness of use
Designed to be re-used
Being adaptive and feedback driven
From the list below, select the statements which, according to the 2 models
analysed, are quality criteria for learning resources Answer True or False:
TRUE
FALSE
FALSE
Exercise 3:
TRUE
TRUE
TRUE
FALSE
35
Induction to Pedagogy
Evaluation of the Learning Resources
SYNERGY Exchange (http://synergyexchange.eu/)
Created for “SYNERGY”, an ERASMUS+ project
Unit IV Practical Tips & Final Remarks
36
After completing this module, you are now more familiar with the models and tools you can use to evaluate learning resources. You are now equipped with the basic knowledge which allows you to contribute to resource evaluation and to provide useful feedback. From this module, you have learned that: 1. There are several models and methodologies to assess the quality of
learning resources, but the main aspects to take into considerations can be summarised into 3 categories: Objective(s); Target group(s); Constraints.
2. Every lesson has to be adapted to the fit the needs of the learners and the learning environment.
3. Remember that feedback is crucial. You should always be open to receiving feedback and using this feedback to revise and improve the resource.
Conclusion & Summary
37
Thank you for your interest and attention
www.projectsynergy.eu facebook.com/pages/Synergy-for-the-SME-Business-Community www.linkedin.com/company/erasmus-synergy-project
38
• www.merlot.org
• Scriven, M. (2007). The logic of evaluation. In H.V. Hansen, et. al.(Eds),
Dissensus and the Search for Common Ground, CD-ROM (pp. 1-16).
Windsor, ON: OSSA.
• Leacock, T. L., & Nesbit, J. C. (2007). A Framework for Evaluating the
Quality of Multimedia Learning Resources. Educational Technology &
Society, 10 (2), 44-59.
• Prince Edward Island Department of Education (2008). Evaluation and
Selection of Learning Resources: A Guide, Charlottetown, Prince
Edward Island - Canada - http://www.gov.pe.ca/educ
• Student Achievement and Supports Branch Saskatchewan Ministry of
Education (2013). Learning Resources Evaluation Guidelines,
Saskatchewan Ministry of Education – Canada,
http://www.wncp.ca/media/49329/wncptool_%20assessing_validating.p
df
References from this Module