Post on 09-Mar-2016
description
Skills for Climate Change – Project Evaluation
FINAL PHASE – Learning Tools Review
Dr Jonathan Pratt
Steve Matthews
Ann Komzolik
20 September 2012
Contact: Dr Jonathan Pratt
jonathan.pratt@emergentresearch.co.uk
07725 358933
Emergent Research and Consulting Ltd
Canterbury Innovation Centre
University Road
Canterbury
Kent
CT2 7FG
Registered in England and Wales.
Company number: 07452200
Registered address: 31 St George’s Place, Canterbury, Kent, United Kingdom. CT1 1XD.
Executive Summary
� This evaluation report for the Skills for Climate Change (SFCC) project focuses on a review of the
five learning tools developed for the programme by June 2012, namely:
1. Future-proof Your Career
2. Introduction to the Code for Sustainable Homes
3. Environmental Technology Systems Awareness (Level 3)
4. Solar Renewable Energy Systems Science
5. Waste Management and the Environment
� The review is based on learner feedback forms from over 300 learners, interviews with tutors
and a panel of peer reviewers with experience of FE, e-learning environments and/or the main
topics covered by the learning materials developed.
� The learner feedback suggests high levels of satisfaction with the learning materials developed in
each of the five areas and the learning delivered under the SFCC project. Average feedback
scores for the materials overall and in terms of clarity and ease of understanding were
consistently 8 out of 10 or above.
� For three of the tools where more detailed feedback is available, learners report significant
improvement in their understanding and awareness of relevant climate change and skills issues
following the training. The increase in scores is particularly notable for the Environmental
Technology Systems Awareness and Solar Renewable Energy Systems Science tools, suggesting
that the desired key learning outcomes for these tools were being achieved.
� Peer reviewers suggested that the learning tools developed are of a high quality and for the
most part are clear and accessible for their intended audiences. They noted however that there
is room to improve the interactivity of the materials particularly if they are used in a tutor-led
environment.
� Two of the tools were primarily designed to be used online and this may have contributed to a
perceived lack of interactivity when used in a more traditional classroom setting.
� Most tutors using the materials report that they have worked well and that they have received
good informal feedback from learners. Over time, some attempts have been made by tutors to
design more interactive sessions around the materials.
� Learner feedback for the Solar Renewable Energy Systems-Science tool suggested that the level
and amount of information covered was proving difficult for a minority of learners. Tutors
delivering the learning suggested their delivery has evolved over time to try and take account of
these concerns.
� Reviewers also suggested that the online tools would benefit from more testing of learner
understanding (and particularly their ability to apply what they have learnt) and the use of more
video and animations to provide more variety in the nature of the material.
� Following the interim evaluation report, efforts have been redoubled to pilot the tools more
extensively online. However, to date, learner feedback has been limited. In future, it would be
worth considering incentives to feedback such as offering respondents a prize draw or High
Street tokens.
Contents
Page
1. Context 1
2. Future-proof Your Career 2
3. Introduction to the Code for Sustainable Homes 6
4. Environmental Technology Systems Awareness 9
5. Solar Renewable Energy Systems Science 12
6. Waste Management on Site 15
7. Conclusions 17
Annex 1: Peer Reviewers 18
Annex 2: Peer Review Notes 19
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 1
1.0 Context
This evaluation report for the Skills for Climate Change focuses on a review of the five learning tools
developed for the programme by June 2012. The Skills for Climate Change Project has developed
new learning materials that focus on previously unmet needs and sought to deliver them in ways
(online and mobile) not often used in the construction and building services sector.
By June 2012 the project had developed tools in five areas
1. Future-proof Your Career (Level 2)
2. Introduction to the Code for Sustainable Homes (Level 2/3)
3. Environmental Technology Systems Awareness (Level 3)
4. Solar Renewable Energy Systems Science (Level 3/4)
5. Waste Management and the Environment (Level 2)
This report updates the interim report reviewing the learning tools. The SFCC project has a clear
focus on innovation but such innovation also represents higher risks for the developers (e.g. in terms
of effectiveness in delivering learning, likelihood of take-up etc). The first phase of our project
evaluation highlighted that feedback on the tools had been quite ad hoc to date and there remained
an opportunity to formally review and refine them within the timescale of the project.
Emergent therefore sought to review the materials in three ways:
1. Learner feedback – Emergent designed more detailed learner feedback forms for each of
the five tools and sent them to the providers. Providers then returned what learner feedback
they had managed to collect by the end of June.
2. Tutor feedback – Emergent undertook qualitative telephone interviews with tutors leading
sessions using four of the five tools and designed a tutor feedback form for tutors using the
Environmental Technology Systems Awareness materials at the National Skills Academy.
3. Peer Review – Emergent identified peers working in similar or related fields and/or in
designing e-learning packages (at the Suscon centre in Kent) and arranged for them to
review the materials in order to provide constructive feedback (See Annex 1 for details of
the Peer Reviewers)
In total, some 333 learner feedback forms were returned, although 136 were the generic, simpler
forms that had been previously used by the project. Around 80 of the forms were from pilots of the
first two tools undertaken in 2011. The table below summarises the responses by tool.
Tool Responses
1. Future-proof Your Career 56* & 30
2. Introduction to the Code for Sustainable Homes* 47
3. Awareness of ETS systems 127
4. Solar Renewable Energy Systems Science 41
5. Waste Management and the Environment* 32
*Generic feedback forms
In addition, 7 online feedback forms were received from virtual users of the Future-proof Your
Career learning tool.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 2
2.0 Future-proof Your Career
The Future-proof Your Career tool was originally developed as an e-learning tool, then adapted to
create a set of toolbox talks and flashcards (and more recently as a more traditional classroom based
course by one of the delivery partners). It is aimed at all construction & building services engineering
workers. Discussions with the provider leading on tutor-led delivery of these materials suggest that
most beneficiaries to date have been between the ages of 18-25, are current or recent apprentices
and likely to have or be studying for a level 3 qualification.
The project evaluation framework outlines the short, medium and long term desired outcomes for
the tool. The short-term outcomes summarise what might be called learning outcomes. For Future-
proof Your Career, these are principally around improving workforce awareness and understanding
of why skills for climate change are needed; what type of skills are needed and how to set about
developing the skills.
Learner Feedback
Some 30 learners filled one of the more detailed feedback forms. On average they gave the
materials an overall score of 8.5 out of 10 (see Figure 1). They gave similar scores for the materials in
terms of more specific measures such as clarity, realism and usefulness. The scores for interest and
relevance were very slightly lower.
Figure 1: Learner Ratings for ‘Future-proof Your Career’ Materials
Base: 30 learners
The majority of learners providing feedback suggested that the amount of information given and the
level of difficulty of the materials were about right (Figure 2a and 2b overleaf). However, a minority
(20%) of learners on this particular tool felt that it may be pitched a little low (too easy).
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 3
Figure 2a: Amount of Information Figure 2b: Level of difficulty
Base: 30 learners
Feedback from the 56 learners who filled in the more generic original feedback forms was also
positive with more than two thirds (68%) rating the training as very good or excellent. Students were
most likely to rate the trainers particularly highly (73%). They were most critical of the length of the
workshop (16% rated this aspect of the training average or poor).
Figure 3 highlights that on a six point scale (from ‘very well’ at point 6 to ‘not at all’ at point 1),
learners suggest on average that the course significantly improved learner understanding in each of
the three desired learning outcome areas for the tool.
Figure 3. Understanding Before and After the Training (Mean score out of 6)
Base: 30 learners
Key 1=Do not understand at all
6= Understand Very Well
Verbatim comments from learners were also mostly positive, although several students noted that
they would like to have seen more multimedia content, with video and audio clips for example, and
interactivity in terms of tasks and activities:
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 4
I feel this has helped me a lot.
Really enjoyed this course.
Today’s presentation was very helpful and understanding.
Could make it more interesting, put video clips, something interactive.
Bit tedious, get clips and don't patronise us with information that's common sense, give us a
task.
97% of the learners providing detailed feedback (i.e. 29 of the 30) said that they would recommend
the learning materials to a friend or colleague.
By the middle of July, it was reported that there were over 100 registrations for the online ‘Future-
proof Your Career’ course, promoted by the Federation of Master Builders. However, only 7
completed feedback forms were received by 19th
July. This number is too small to analyse
quantitatively. Qualitatively, however we can say that there is no clear evidence that learner scores
for the materials online will be significantly lower (or higher) than for tutor-led delivery.
Tutor Feedback
While the materials have primarily been designed to be used online and at the learners own pace,
the learning commissioned to date has been delivered in a tutor-led classroom environment in quite
intensive single sessions (usually combining elements of SFCC Tools 1 and 2) with tutors going
through the material with the support of a PowerPoint presentation based on the flashcards.
Feedback from the tutor was that the materials worked well, the material is pitched at about the
right level, and is clear but could be a bit more concise. When the materials talk about
competencies, for example, they can be a little repetitive. Some examples may not always be
relevant to learners; however it is easy for a tutor working in the sector to augment them with their
own examples. Overall, the tutor felt that the materials clearly help learners achieve the learning
objectives set. They suggested that the students really seemed to take on board the idea that each
and every one of us can contribute to reducing climate change and its impacts
The main perceived weakness of the materials was the lack of guidance on or incorporation of more
interactive and engaging activities and encouraging them to work in groups for different elements of
the learning for example. It should be noted however that the tools were originally designed to be
delivered online (with or without tutor support) not in a more traditional tutor-led lecture or
classroom style.
Peer Review1
The reviewers felt the materials were pitched at the right level for the intended audience (builders,
plumbers, electricians etc), were clear and well written, well referenced and included some excellent
graphical elements.
They noted that the content was quite basic to start off with but built learner’s awareness of the
need to adapt their working practices as they progress through the materials. They also suggested
that it clearly gives the message that this will require bolt-on courses and continuing professional
development rather than complete retraining. The reviewers also felt the materials should be able
1 The Peer Reviewer’s notes for each tool are outlined in more detail in Annex 2.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 5
to impart an understanding of the Government’s commitment to reducing carbon and how this can
be achieved in new build and existing homes.
The reviewers also felt the idea of working based on flashcards is a good one but that these would
be improved if they explained a bit more of the terminology, maybe with small print notes. They
noted that some of the materials (e.g. around feed-in-tariffs) will also need updating over time.
Using the online tool, navigation was found to be easy between screens and the idea of testing as
you move along works well. The reviewers thought the learning checks were relatively easy and
would give learners some confidence to move on.
The e-learning reviewer also made a number of observations and suggestions that apply more
generally across all three online tools (Future-proof Your Career, Introduction to the Code for
Sustainable Homes and Environmental Technology Systems Awareness). They suggested:
• The facility to turn off audio commentary (rather than all sound) would be a valuable
addition to the learning materials.
• The inclusion throughout the tools of pertinent video and animations would also be a useful
addition in supporting a greater range of learning styles and bringing more of the content ‘to
life’.
• The ability for the user to select their own colour/contrast settings would also make the
materials more ‘accessible’ (although with the audio commentary much of the need for a
screen reader -although not all- has been removed).
The e-learning reviewer also noted that whilst an iphone app has been developed for one tool, the
particular authorware used means the main online content is not accessible in a meaningful way via
iPads or iPhones and that when accessed through other mobile devices (e.g. Blackberry & android-
via web browser) it renders poorly on smaller screens. The SCORM2 scoring, they suggested, could
also have been used to provide more information on learner progress than a simple pass/fail against
each whole tool.
The peer reviewers also suggested that as the majority of the intended outcomes for the tools are
for the learner to “know” or “be aware” that the learning checks used in the online tools (Multiple
Choice Questions) don’t always successfully allow the learner to demonstrate this.
2 Sharable Content Object Reference Model (SCORM) is a collection of standards and specifications for web-based e-learning
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 6
3.0 Introduction to the Code for Sustainable Homes
The Introduction to the Code for Sustainable Homes was originally developed as an e-learning tool,
then adapted to create a set of toolbox talks, flashcards and a mobile phone application (and more
recently as a classroom based course by one of the delivery partners). It is aimed at all construction
& building services engineering workers.
The evaluation framework suggests that the desired short term (learning) outcomes for the tool
were to develop workforce awareness and understanding of what the Code is and how it works, why
it is a key driver of the need for workers constructing new homes to adapt and extend their skillset
and how understanding the Codes aims and approaches can also be applied to work on existing
homes.
Learner Feedback
As with the first tool, most of the learning undertaken to date for this tool has been in a tutor-led
environment rather than online. Discussions with the provider suggest that most beneficiaries to
date have been between the ages of 18-25, are current or recent apprentices and likely to have or
be studying for a level 3 qualification.
Unfortunately, no learners have completed the more detailed feedback forms for this tool, although
most of the recent learners who completed the detailed feedback on the Future-proof Your Career
learning tool (see Section 2) also completed the Introduction to the Code for Sustainable Homes
Training. It is likely that their responses were coloured by both experiences and had they filled in
forms for both courses their responses may have been similar to those outlined in Figures 1-3.
However, some 47 learners in the previous pilot phase of the project filled out a more generic
feedback form. The majority (68%) rated the learning as either very good or excellent (Figure 4).
Figure 4: Learner Ratings of ‘Introduction to the Code for Sustainable Homes’ Workshops
Base: 47 learners
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 7
Learners were most likely to rate the trainers particularly highly but were less likely to rate the
length of the workshop and the content as highly.
Again comments from learners were mostly positive, although several students noted that they
would like to have seen more visual content with examples of installations
Could do with photographic examples of installation e.g. solar & wind & how cabling would
be fed through building materials.
Would help if there were a video to show demonstration.
It might be better if it had a practical element to the workshop.
A few students also felt that it was a lot to take in a short period of time…
Trainer went through stuff slightly too fast. Venue was too big; found it difficult to hear the
trainer.
Information given very informative but content too much in short time.
Both criticisms are unfortunate, given that the learning materials were originally designed to be
online and interactive, and for learners to be able to work at their own pace.
Tutor Feedback
The learning commissioned to date has been delivered in a tutor-led classroom environment in quite
intensive single sessions (usually combining elements of SFCC Tools 1 and 2) with tutors going
through the material with the support of a PowerPoint presentation based on the flashcards. This is
not what was originally envisaged by the learning tool designers. The tool was designed to be used
online and at the learners own pace.
Given this context, feedback from the tutor was that the materials worked well, the material is
pitched at about the right level (might be more stretching for learners who were not apprentices),
and is clear. Some examples may not always be relevant to learners; particularly as most are more
involved in retrofits than new homes, however most would see the potential relevance to their
work. Overall, the tutor felt that the materials clearly help learners achieve the learning objectives
set.
Not surprisingly, given the change in delivery method, the main perceived weakness of the materials
from the tutors’ perspective was the lack of guidance on or incorporation of more interactive and
engaging activities.
Peer Review
The peer reviewers felt the learning materials were very informative and full of useful information
and that the flow was good, gradually increasing knowledge. The information presented would
clearly allow the learner to understand the reasons behind the Code, how it is applied and what they
could do to ensure buildings meet standards and especially to understand which elements are
mandatory. It was thought that even the small business owner concentrating on small build and
domestic extensions would understand what can be achieved to raise the energy efficiency of the
whole dwelling, how this relates to energy, water, materials etc. and be able to advise his customers
accordingly.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 8
On the toolbox talks, the reviewers noted that it is very important to have the key message at the
end and suggested they should be applied in every flashback to enhance the summary. They also
noted that there was quite a lot of technical terminology and suggested that small print comments
on the meaning of technical words might help, as would avoiding abbreviations (such as DER and
TER) wherever possible. However, it is perhaps reasonable to expect the toolbox tutor delivering the
talk to be able to explain the acronyms and technical terms. The main tool also has a Glossary
section.
In terms of online interaction, the way you could click onto speech bubbles in the online tool to gain
more information and onto hot spots in houses was thought to work well. The reviewers noted that
once again navigation was easy between screens. They also suggested that the learning checks were
good but could still be improved to show more application of the knowledge gained.
The e-learning reviewer noted that presentation and images were of high quality and support the
page content well. He noted however that the tool is very large and that the progress bar did not
help learners should they progress through the modules in a different order. They suggested that
the tool might benefit from being split into distinct units.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 9
4.0 Environmental Technology Systems Awareness (L3)
This learning tool was originally developed in both tutor-led and Virtual Learning Environment (VLE)
formats. The learning tool content provides the learning required to achieve the Level 3 QCF unit
‘Understand the Fundamental Principles and Requirements of Environmental Technology Systems’.
This unit is used in the City & Guilds 2399-01 Level 3 Award in Environmental Technology Systems
Awareness (QCF) and the EAL Level 3 Award in the Fundamental Principles and Requirements of
Environmental Technology Systems (QCF). The learning tool has been mainstreamed through the
National Skills Academy for Environmental Technologies. The National Skills Academy has also
produced an ‘Introduction to Environmental Technology Systems’ learner manual using the SfCC
learning tool as the core resource.
The project evaluation framework identifies a number of desirable short term (learning) outcomes
for the tool, namely that it should improve workforce awareness and understanding of key types of
environmental technology system, the fundamental working principles of each technology, the
fundamental location/property features required for the potential to install each technology to exist,
the fundamental regulatory requirements for each technology and the advantages and
disadvantages for each technology.
Learner Feedback
Discussions with the provider suggested that most learners were current apprentices in Building
Services Engineering (electricians, plumbers & air conditioning engineers) and the majority are 19-
25.
The learner feedback forms received for this tool suggested a quite high level of satisfaction overall
(with a mean of 9.0 out of 10). Scores for the clarity of the materials were highest; while scores for
the apparent relevance of the materials to learners’ current work were noticeably lower (Figure 5).
Figure 5: Mean Score (Out of 10) for different aspects of the ETS awareness tools
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 10
Base: 127 learners
91% of learners felt the materials were about right in terms of ease of understanding, 7% thought
they were too easy. 88% the learners responding felt that amount of information given in the
materials was about right (Figures 6a and 6b).
Figure 6a: Amount of Information Figure 6b: Level of difficulty
Base: 127 learners
Encouragingly, learners reported a significant improvement in their levels of understanding in areas
corresponding to the desired learning outcomes for the tool (Figure 7).
Figure 7. Understanding Before and After the Training (Mean score out of 6)
Base: 127 learners Key 1=Do not understand at all
6= Understand Very Well
Verbatim feedback from learners (on tutor-led courses) was mostly positive, although a number
suggested it could be more interactive and/or practical.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 11
More practical learning such as witnessing the process instead of reading about it which is
much harder to understand and take in.
More interactive.
More hands on examples.
More on ‘costing’ systems.
Tutor Feedback
While the materials have primarily been designed to be used online and at the learners own pace,
the learning commissioned to date has been delivered in a tutor-led classroom environment in quite
intensive sessions over 2 days with tutors going through the material with the support of a Power
Point presentation based on the flashcards. A booklet using the same materials has also been
prepared for the students to take away.
Discussions with one of the tutors revealed that they had incorporated some of their own Q&A
sequences into the presentation to make it more interactive.
The tutor felt the learning materials were at an appropriate level, were clear and easy to understand
and contained about the right level of information. The examples were mostly relevant and realistic,
although the diagrams for wind turbines only showed large industrial level turbines rather than the
smaller ones students are more likely to meet in a domestic environment. Despite this, the tutor felt
the visuals were a key strength of the materials.
The tutor suggested that the materials could do with bringing the session together at the end of the
day to engage in some kind of activity to use the learning. He noted, however, that the tool is
primarily about awareness and will never therefore be that practical (which is why it works well as
an introduction to other courses which are).
Peer Review
The reviewers noted that this tool includes a wide range of technologies and clearly explains how
and when they were suitable. The depth of the information was felt to be relevant to a competent
workforce and would increase their knowledge and understanding of new technologies whilst noting
that further installation training would be required. The reviewers felt the materials will demystify
the technologies with language and diagrams that the trained builder/plumber/electrician etc.
would understand.
They also noted that the learning tool covers a lot of material and the narrative can be repetitive.
They suggested replacing some of the text with narration and enlarging the images or introducing
animation or videos instead. There was also a suggestion that some more introductory information
on the relative costs and benefits, short, medium and long term of different technologies would help
learners at this point.
The e-learning reviewer noted that the tool again allows progress in a non-linear fashion but that
feedback regarding progress (the progress bar) can be inaccurate because of this.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 12
5.0 Solar Renewable Energy Systems-Science (Level 3)
This tool was developed as a tutor-led learning tool (with the Centre for Efficient and Renewable
Energy in Buildings at London South Bank University). It is an introductory, non-accredited course
that aims to be used with both FE and vocational trainers as well as more advanced practitioners in
the industry.
The evaluation framework suggests that the short-term desirable (learning) outcomes are the same
for both groups; i.e. to develop a more in-depth understanding of the solar resource and the factors
that affect it, of solar thermal hot water system science and of solar photovoltaic system science.
Learner Feedback
Of the 41 learners who provided detailed feedback forms, only three identified themselves as
vocational trainers (by filling in a distinct form). In reality the proportion is likely to be higher. Across
both groups learners gave the Learning Tool an average score of 8.3 out of 10. Scores for relevance
and interest/engagement were slightly lower but still quite high (Figure 8).
Figure 8. Mean Score (Out of 10) for different aspects of the Solar Systems Science Tool
Base: 41 learners
Whilst the majority of learners felt that the learning tool materials were pitched at the right level in
terms of level of difficulty and amount of information (Figure 9, overleaf), a sizeable minority felt
they were too difficult (12%) or included too much information (20%).
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 13
Figure 9a: Amount of Information Figure 9b: Level of difficulty
Base: 41 learners
Once again, it is encouraging to note that learners report significant improvements in their
understanding of solar systems science in areas relating to the short term learning objectives for the
tool (Figure 10). Indeed the difference in scores is more pronounced than for some of the other SFCC
learning tools.
Figure 10: Understanding Before and After the Training (Mean score out of 6)
Base: 41 learners Key 1=Do not understand at all
6= Understand Very Well
Comments from learners included:
A lot to take in but v enjoyable and a good learning curve.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 14
The equations are quite complicated. Could have done with a little more time/explanation.
I found the course very good and knowledgeable and explained very well but I have no
previous experience or knowledge of PV and solar panels. A beginners guide at the start
would have been a big help.
Possibly extend the course for another 1/2 day to reduce the intensity of the training.
Tutor Feedback
The lead tutors on the course reported that they feel the training had gone very well and they had
had good informal feedback from the learners. They acknowledged that there is a lot of information
covered with the learners in a very short space of time.
They noted that the way the learning has been delivered has evolved over time, partly in response to
feedback. The training is completed on a single day to try and minimise disruption to the learners’
working week but the sessions now vary more in style to maintain interest. Learners are shown a
range of solar renewable energy systems in operation (on the roof of the CEREB centre) and there is
a workshop at the end of the day when learners get to put their learning into practice. Learners are
also given more materials to take away with them, including access to live data from the CEREB solar
renewable systems and a spreadsheet calculation tool they can use in their working lives. The tutors
noted that there is a lot of complex material but that they now focus more on people working at
their own pace and leaving some of the more advanced mathematical and scientific context to the
supporting materials that people can access if they want to but do not need to go through.
They would also like to consider more blended learning delivery in future, providing learners with
some materials before they start so that they can orientate themselves prior to the tutor-led
delivery.
Peer Review
The peer reviewers noted that there was a lot of useful information in the learning materials and
that the flow was good, gradually increasing the learners knowledge. They noted that the Power
Point slides included high quality schematics and visual presentations and that the video sections,
whilst of quite a basic quality, offered a good opportunity for learners to consolidate their learning.
They suggested that the learning objectives were clearly stated and met in the materials and noted
that there were questions posed at the end of the presentation to ensure understanding of the
content.
However the reviewers suggested there might be too much to complete in one day (assuming that
all the materials in the presentation slides are used). They also noted that the level of material
seemed to be well above the Level 3 it has been assigned (although nevertheless probably
appropriate for the target audience of trainers and advanced practitioners).
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 15
6.0 Waste Management and the Environment
This tool was originally developed as a set of toolbox talks in response to a need identified by
strategic project partner Higgins Construction PLC, who have since incorporated the materials into
their own training programmes internally and with supply chain companies.
The evaluation framework identifies a number of short term desirable (learning) outcomes for the
tool. Namely that learners are aware of the fundamental requirements of waste management and
waste transfer legislation; can understand and apply the ‘waste hierarchy’ and why it should be
applied; understand the need to reduce energy use on site and know actions that they can take to
reduce energy usage; understand the need to protect the local environment during construction
work and know actions that they can take to protect the environment.
Learner Feedback
The focus for the delivery of this Learning Tool to date has been on workers undertaking an NVQ
Level 2 in ‘Removing Non-Hazardous Waste’ in the Olympic Park site (staff fitting and furnishing the
Olympic village). It has been used in conjunction with a specific scheme to take on staff who have
been unemployed or economically inactive. Beneficiaries are likely to be general labourers ranging
from 20s to 40s and many may not have Level 2 qualification. The learners have therefore mostly
been non-typical construction workers.
Unfortunately, the bulk of the training using this tool took place before the more detailed learner
feedback forms were available. However, some 32 learners responded to the generic feedback
sheets on the course. Whilst this is a small number, their average scores for the different elements
of their workshop experience are very high (Figure 11).
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 16
Figure 11: Learner ratings for the ‘Waste Management and the Environment’ Tool
Base: 32 learners
Some 94% rate their overall experience as very good or excellent. No learners reported that any
aspect of their training was average or poor.
Verbatim feedback from learners included:
Constructive and knowledgeable information from trainers explained and answered
questions brilliantly.
The training was very good and it gave you the knowledge to carry towards future jobs and
everyday tasks.
I thought that I knew about waste management was basically recycling until this induction. It
was interesting to learn how I could be doing a bit more.
Learnt a lot. More knowledge about rules on the construction site.
Tutor Feedback
Discussion with the current provider reveals that the materials are being delivered using the toolbox
talks developed directly by the project (interspersed through their own longer waste management
course). The tutor suggested that the materials have been very useful in introducing their own
course and what it will cover in more detail.
The tutor suggested that some of the content could have been covered in more depth, particularly if
it was used with industry savvy labourers rather than completely new recruits. They also suggested
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 17
that it currently requires an ‘interesting’ trainer – a lack of interactivity meaning that a less confident
trainer may struggle. However, a series of suggested Q&As have since been added to the reverse of
toolbox talk flashcards.
Peer Review
The peer reviewer for this tool felt that the toolbox talks were appropriate for their intended
audience, including those with little prior knowledge of the subject. They noted that the materials
clearly summarise the key issues and provide good introductions to the broader sustainability issues
raised. The materials also provide clear, practical information on how to comply with legislation.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 18
7.0 Conclusions
The learner feedback suggests high levels of satisfaction with the learning materials developed for
each of the five learning tools and for the learning delivered (by tutors) under the SFCC project.
Average feedback scores for the materials overall and in terms of clarity and ease of understanding
were consistently 8 out of 10 or above.
For three of the tools where more detailed feedback is available, learners report significant
improvement in their understanding and awareness of relevant climate change and skills issues
following the training. The increase in scores is particularly notable for the Environmental
Technology Systems Awareness and Solar Renewable Energy Systems Science tools, suggesting that
the desired key learning outcomes for these tools were being achieved.
In general the peer reviewers found the online learning materials easy to use with clear and concise
instructions. They noted the use of interesting images and graphics. They also liked the idea of
Flashcards as a simple pack of cards that helped you remember various facts, although they were
not convinced that the cards match with the depth of knowledge in the online courses. The
reviewers concluded that the tools were fit for purpose but made a number of suggestions for
further improvements.
The peer reviewers suggested, for example, that there is room to improve the interactivity of the
materials particularly if they are used in a tutor-led environment. Both tutors and learners report
that the materials lack a little in terms of interactivity and can be somewhat static and un-engaging
(although they are clearly written and include good static visual diagrams and pictures). This is
disappointing given the SFCC programmes aim to be innovative and dynamic in terms of engaging
small businesses.
Two of the tools were primarily designed to be used in an online setting and this may have
contributed to a perceived lack of interactivity. Following the interim evaluation report, efforts have
been redoubled to pilot the tools in an online setting, although the learner feedback from online
users has been limited to date. Nevertheless this raises significant questions for the project going
forward. Will the main legacy for the project be the online tools or the more static visual
presentation aids? If the former, then the lack of feedback means that a key aspect of the tools
remains relatively untested. If the latter, then the materials may need to be refreshed with this in
mind, introducing periodic suggestions for group and individual activities and tests.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 19
Annex 1: Peer Review Team
Cristina Blanco-Lion
Cristina is a sustainable construction researcher, practitioner and trainer. Most recently she has
been working on the accreditation of sustainable construction courses at Suscon, the sustainable
construction training centre in Kent. Previously she has worked as a researcher for the Sustainable
Urban Design Group and as an Architect for Blanco & Nunez Associates. She has an MA in
Architecture, Design and Sustainability from the University of East London.
Steve Humphries (e-learning)
Steve is highly experienced in both designing and delivering e-learning courses for level 3 and level 4
learners. He has administered VLEs since 2002 and overseen a number of projects customising and
implementing institution wide customisations of Moodle (eg Personal Learning Plans), GPS check in
block, unified mark book and qualification management. Steve has authored, delivered and gained
Edexcel accreditation for a BTEC level 4 "award in e-learning and the VLE. He is also a Hi5 award
winner for use of screen capture for learning (JISC and National Excellence Gateway recognition). He
is currently Learning Technology Manager for North West Kent College.
Sally Clark
Sally is the General Manager at Suscon, the Sustainable Construction Training Centre based at the
North West Kent College. Sally has 5 years experience of working in housing contracts for a local
authority, planning and developing tenant involvement in major refurbishment/retrofit works. She
spent 7 years working for a Regeneration Partnership running EU and Government funded research
into the skills base and projected skills needs. For the last three years has led the SusCon project
which developed, tested and accredited 30 new units of sustainable construction training. More
recently, she has been heading up the development of the Higher Apprenticeship in Sustainable
Building at SusCon.
Noha Nasser
Noha is an Associate of Suscon. She is a consultant with a BSc in Architectural Engineering and a PhD
in Sustainable Urban Design, Conservation and Regeneration. She has 10 years academic experience
including as Project Director of CABE urban design summer school 2007-9. Her specialisms include
community engagement, urban design and designing low carbon retrofit schemes for individual
dwellings.
Charles Nouhan (Waste Management Tool Reviewer)
Charles is an Associate of Suscon. He is a former US based real estate broker with a Masters in
Environmental Management. He also has 10 years experience as a Recycling Manager at a District
Council and was the Chair of the Kent Waste Minimisation and Recycling Forum/Kent Waste
Partnership for four. He delivers occasional training to Council customer service reps and refuse
collection supervisors and crews.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 20
Annex 2: Peer Review Notes
Tool 1: Future-proof Your Career Criteria Comments
Level/Pitch • Perceived audience are builders, plumbers, heating engineers, plasterers and electricians – pitch is about right although less text and more
visuals/narration may be easier to grasp
• Pitched at Level 2 but at points seems be more like Level 3
• The unit is a bit didactic with insufficient application or testing of knowledge
Relevance of material • Simple explanation of drivers of climate change and growing professional and consumer demand
• Basic introduction to areas for climate change intervention that is useful to a beginner or climate-change related occupations looking to skill up
• Raises awareness of the practical monetary and carbon savings to consumers
• Clearly suggests ways of seeking greater specialisation in climate change skills
Quantity of
information/level of detail
E-learning review
comments in italics
• Content was basic and built learner’s awareness of the need to adapt working practices
• Informative and comprehensive content identifying key policies and features. Good referencing of further information and qualifications required.
• Good use of interactive web links in ‘Resources’ menu but could also be embedded in main body of modules where relevant. Glossary could be
expanded to act as an aide-memoire for main content
• URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading or possible further research in
the subject area. The separate resource section means that the links contained therein are often de-contextualized.
• The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good.
The images are relevant and add value to each screen.
Clarity/ease of access
E-learning review
comments in italics
• Clear and straightforward language and font.
• Easy navigation between screens but not easy to go back to previous screens or start from the beginning once completed Unit
• Materials are easily accessed through a desktop computer however if using a mobile device the quantity of content and format used e.g. SWF generated
by using Articulate have inherent issues with regard to mobile access
Quality of presentation
E-learning review
comments in italics
• Excellent graphic of cross-section of home with buttons for more info in Module 2 section on approaches to energy efficiency
• Some of the more technical images are too small e.g. Module 3 technical skills solar water heating.
• Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should
be an option. The images are of a good quality. No printer-friendly format for future reference
Engagement/interactivity
E-learning review
comments in italics
• Interactivity is achieved through clicking to visit the next screen/more info and through a drag-and-drop exercise.
• A bit too much prevalence on giving information rather than encouraging application
• More activities similar to the drag and drop activity in Module 2 on ‘advice and guidance’ order of approach to energy efficiency methods which creates
a more interactive learning experience would enhance the tools
Educational assessment • Learners will understand that re-training is not necessary but often bolt-on courses will help. Learners will understand government’s commitment to
reducing carbon and how this can be achieved in new build and existing homes.
• Some of the material will need updating; Flashcard 3 for example needs to note that the Feed in tariff was reduced in December 2011. Flashcard 9 could
do with revisiting. This is a key diagram and a Pyramid instead of rectangles would improve the emphasis (Reduce energy demand at the bottom of the
pyramid would be highlighted as the MAIN action!).
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 21
Tool 2: Introduction to the Code for Sustainable Homes Criteria Comments
Level/Pitch • Pitched at a level that even the small businessman concentrating on small build and domestic extensions will understand what can be achieved to raise
the energy efficiency of the whole dwelling
• The information is more technical but should be accessible to construction industry trades
• Pitched at Level 2 but could be increased to Level 3 with greater application and interpretation testing concepts, principles and methods
Relevance of material • Very informative information – flow was good gradually increasing knowledge. The information would allow the learner to understand the reasons
behind the Code, how it is applied, what they could do to ensure buildings meet standards and especially to understand which elements are mandatory.
• Material supports learners to advise customers on energy, water, materials etc.
• Range of design solutions to achieve higher code standards made clear in text but could be supported by visuals
Quantity of
information/level of detail
E-learning review
comments in italics
• Thorough and detailed content explaining at great length the various criteria of the higher Code standards – a little too text heavy could benefit from
replacing text with narration and visuals
• There is quite a lot of technical terminology: is it possible to add a small print comment on the meaning of that technical word. Is it also possible to avoid
abbreviations such as DER and TER (Flashcard 11)?
• URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading. The separate resource section
means that the links contained therein are often de-contextualized.
• The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good.
The images are relevant and add value to each screen.
Clarity/ease of access
E-learning review
comments in italics
• Easy navigation between screens. Good means of learning from mistakes i.e. with corrections identified by reviewing tests
• Test is useful but only tests knowledge not application of knowledge (too easy?).
• Keeping track of topics and sub-topics got slightly confusing and tedious – suggest option of a summary page for each topic theme ‘in a nutshell’ or
subdivisions on the menu bar
• This tool is very large. It contains a significant amount of content and would perhaps benefit from being split into a number of tools particularly in light of
any scorm data being passed back to the system. This tool also allows progression in a non-linear fashion. Feedback regarding progression therefore is
not meaningful to the user should they progress in the order 2, 5, 7 etc… as the progress bar at the base will be inaccurate.
Quality of presentation
E-learning review
comments in italics
• The speech bubbles and hotspots on the houses to gain more information are very successful
• Too much text – needs more visual material
• Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should
be an option. The images are of a good quality and support the page content well.
Engagement/interactivity
E-learning review
comments in italics
• Learning checks are easy to use as are hot spots
• Simple navigation between screens but limited engagement and interactivity overall
• Interactivity is achieved through clicking to visit the next screen/more info and through MC/drag-and-drop quizzes.
Educational assessment • A useful introduction to the Code and its various categories and criteria of measurement and intervention
• Meets identified learning aims but could benefit from learning checks aimed at applying knowledge
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 22
Tool 3: Environmental Technology Systems Awareness Criteria Comments
Level/Pitch • The level is not really addressed at novices/beginners but a more technical audience e.g. micro-combined heat and power systems connection. Key
features and concepts should only be included if to be addressed to a general audience. Good example is biomass fuelled system
• A large amount of knowledge and information but not always easy to digest. Could be supported by better diagrams and animation/video
Relevance of material • Included a wide range of technologies and clearly explained how and when they were suitable
• This unit will help de-mystify the technologies with language and diagrams that the trained builder/plumber/electrician etc. would understand.
Quantity of
information/level of detail
E-learning review
comments in italics
• The depth of information is relevant to a competent workforce and would increase their knowledge and understanding of new technologies whilst
noting that further installation training would be required.
• Unlike unit on ‘Future-proof Your Career’ knowledge of costs to install, maintain, long term monetary benefits and savings, as well as carbon
savings/efficiency, would be helpful for learners to decide between technologies
• An overall assessment of how each technology compares to the other would be useful
• Comprehensive glossary – to be replicated in other units
• URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading or possible further research in
the subject area. The separate resource section means that the links contained therein are often decontextualized.
• The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good.
The images are relevant and add value to each screen.
Clarity/ease of access
E-learning review
comments in italics
• Easy navigation between screens
• Keeping track of topics and sub-topics got slightly confusing and tedious – suggest option of a summary page for each topic theme ‘in a nutshell’ or
subdivisions on the menu bar
• This tool is of a good size. This tool again allows progression in a non-linear fashion although feedback regarding progression through the tool (progress
bar) can be inaccurate because of this.
Quality of presentation
• Narration is repetitive – Would suggest replacing text with narration and enlarging image, animation or embedding a video instead
• Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should
be an option.
• The images are of a good quality and although generally of a good size there are examples when some images may be too small.
Engagement/interactivity
E-learning reviewer
comments in italics
• Some interactivity is achieved through clicking to visit the next screen/more info and through MC/drag-and-drop quizzes.
• Information often presented in a static way – overall quite limited interactivity in this unit
Educational assessment • Meets the intended learning objectives but unit remains quite didactic, particularly if used as slides in a classroom setting rather than as an online tool
(as was originally intended).
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 23
Tool 4: Solar Renewable Energy Systems Science Criteria Comments
Level/Pitch • Pitched at undergraduate level.
• Well above level 3 delivery.
• Technical information based on physics of solar technology.
• Videos were rather basic (at lower level than information on slides)
Relevance of material • Very informative information – flow was good gradually increasing knowledge
• Material support learners to understand how solar renewable technologies work.
• Materials support learners to understand the limitations of the solar technologies and relate to orientation, tilt, seasonal changes and occupant
behaviour.
Quantity of
information/level of detail
• Very thorough and detailed explanation
• If all modules covered in one day - too much detail to absorb.
Clarity/ease of access
• Powerpoint
• Videos
Quality of presentation
• Excellent quality powerpoint presentation
• Clear schematics – excellent visual presentations
• Videos were basic (lower level than presentation) but provided an opportunity to consolidate learning.
Engagement/interactivity
• Learners without prior overview training would struggle to remain engaged.
• Although Powerpoint by itself does not allow very much interactivity – a good presenter can ensure learners are engaged.
• Videos – good use of visuals
Educational assessment • A good in depth look at the science behind solar renewable technologies.
• Learning objectives clearly stated and met within the presentation.
• Assessment tools not described but questions posed at the end of the presentation to ensure understanding of content.
Skills For Climate Change Project Evaluation – Learning Tools Review
Emergent Research & Consulting 24
Tool 5: Waste Management on-site (Toolbox Talks only) Criteria Comments
Level/Pitch • Is appropriate for those with little prior knowledge of the subject.
Relevance of material • The ‘What’ sections clearly summarise the core issues. Other sections offer good introductions to the broader sustainability issues raised by the subjects
in each toolbox talk.
• Where appropriate, the do and don’t sections provide practical information on how to comply with the legislation
Quantity of
information/level of detail
• Level of detail in each toolbox talk is probably about right for the audience.
Clarity/ease of access
• Tool box talk structure is good/very good in terms of course introduction, topic layout and course summary.
Quality of presentation
• Good
Engagement/interactivity • NA
Educational assessment • The toolbox talks fulfil the needs identified and would be fine to use in their current form – but some amendments are recommended if it is practical to
do so:
Toolbox Talk 25: Should point out that SIC codes relate to industries not occupations. The Why? section could include a statement such as ‘The recovery of
materials for reuse or recycling ensures that an adequate supply of essential materials will be available in the future as they become scarce’. Post delivery
checklist Q 3 could include reduced site vehicle traffic as it relates to both site safety and impact on those neighbouring the site..
Toolbox Talk 27: In the Why? Section, strictly speaking CO2 already in the atmosphere is driving current global warming, CO2 emissions could accelerate the
rise in global temperature averages in future. Further section, note that cement production is the source of 5% of man-made CO2 emissions not global CO2
emissions
Toolbox Talk 28: See first comment on Why section in Talk 27. Further section, there could be a mention of how masonry rubble such as brick, block and
cement are recycled to produce similar materials that have a significant recycled content with carbon-saving and other benefits.
Toolbox Talk 29: See first comment on Why section in Talk 27. There is no mention of the sustainable use of water in plastering/dry-lining operations.
Sustainable use of water reduces multiple environmental impacts (e.g. excess water use, pollution) and reduces the costs and time associated with over-use
and clean-up operations.
Toolbox Talk 30: See first comment on Why section in Talk 27.
Toolbox Talk 31 & 32: See first comment on Why section in Talk 27. There is no mention of the larger concept of conservation. Although not necessary for
the module an overall understanding of the concept by practitioners will result in across-the-board environmental savings, particularly in the area of energy
use.
Toolbox Talk 33: See first comment on Why section in Talk 27. The Why section also makes specific reference to Higgins Construction PLC. The first bullet
point should be modified for more general use.