Design & Engineering Services
Usability of In-Home Energy Displays
HT.11.SCE.013 Report
Prepared by:
Design & Engineering Services
Customer Service Business Unit
Southern California Edison
December 2012
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page i Design & Engineering Services December 2012
Acknowledgements
Southern California Edison’s Design & Engineering Services (DES) group is responsible for
this project. It was developed as part of Southern California Edison’s HVAC Technologies
and System Diagnostics Advocacy Program (HTSDA) under internal project number
HT.11.SCE.013. DES project manager Alvaro Mendoza conducted this technology evaluation
with overall guidance and management from Jerine Ahmed. For more information on this
project, contact [email protected].
Disclaimer
This report was prepared by Southern California Edison (SCE) and funded by California
utility customers under the auspices of the California Public Utilities Commission.
Reproduction or distribution of the whole or any part of the contents of this document
without the express written permission of SCE is prohibited. This work was performed with
reasonable care and in accordance with professional standards. However, neither SCE nor
any entity performing the work pursuant to SCE’s authority make any warranty or
representation, expressed or implied, with regard to this report, the merchantability or
fitness for a particular purpose of the results of the work, or any analyses, or conclusions
contained in this report. The results reflected in the work are generally representative of
operating conditions; however, the results in any other situation may vary depending upon
particular operating conditions.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page ii Design & Engineering Services December 2012
EXECUTIVE SUMMARY Southern California Edison directed the Western Cooling Efficiency Center (WCEC) of the
University of California, Davis, to conduct a research project on the usability of In-Home
Energy Displays (IHEDs). In two related studies conducted during 2011-2012, this project
evaluated IHEDs available in the consumer and utility markets. It also developed a mockup
prototype of an IHED that was then experimentally tested by varying a set of common
display features.
IHEDs are used to save energy by providing feedback to users on their energy consumption.
In addition, they have been used as a part of demand response programs, and have
facilitated home automation. The success of residential IHEDs as key players in engaging
and influencing users to reduce energy consumption relies heavily on the ability and
motivation of people to read the displayed information and respond to it with energy saving
actions. For these reasons, the ease of using energy feedback is key to success. Ease of use
is affected by design characteristics of the IHED interface, such as numerical vs. graphic
information, size of fonts, and ease of navigation. Effectiveness of these devices is also
influenced by other environmental and social variables, such as the energy saving goals of
users, social comparisons with neighbors and peers, and incentives to monitor and decrease
energy usage. The studies in this project address both interface design and social variables.
The, ability of occupants to perform certain tasks and successfully extract information using
IHEDs has not been explicitly examined.
The project evaluates the usability of a device to help reduce energy use by utility
ratepayers by providing them with detailed information on energy use that they can use to
make informed decisions. The project explores currently available IHED device interface
features, as well as promising potential features that have not yet been demonstrated; and
evaluates their effectiveness in a controlled experimental setting.
Energy savings provided by IHEDs depend on their usability that depends on the
characteristics of the devices, and how well their interfaces are designed. To design the
appropriate evaluations and usability tests for IHED interfaces, the WCEC research team
conducted first a market review of currently available IHEDs. A list of available displays was
developed to characterize their features, along with a characterization table to summarize
available IHEDs, organized by feature.
Additionally, expert heuristic usability testing was conducted with a sample of four IHEDs.
This market review was used to design the simulation prototypes used later in the test of
IHEDs usability. The main part of the study examined users’ reactions and responses to
simulated energy feedback interfaces that were representative of devices available on the
market. The IHED interfaces used in the study were abstracted from the heuristic expert
review of market devices from the first part of the study. The interfaces present increasing
levels of complex information to a user. Participants were questioned on their ability to
retrieve information from the IHED experimental interface, put that information in a
meaningful energy use and reduction context, and use that information to make energy use
decisions. If participants can retrieve and understand the information and develop a plan to
save energy based upon this information, then energy savings should result from this
technology. Recommendations on desirable specs for IHED interfaces are based on the
results.
A population that is interested in energy savings can be encouraged to adopt energy
efficient behaviors with an IHED, but issues of installation and design must be considered in
more detail by producers, utilities, and other consumers.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page iii Design & Engineering Services December 2012
The market review indicates the presence of an in-flux market, with low penetration and
wide variation in device offers and features. There are no set standards for features,
installation, compatibility, and connectivity. This fluid market probably confuses individual
consumers about the product, its function, and its effectiveness, making adoption less
likely. Despite the confusion in the marketplace, one can still purchase and install IHED
devices, and obtain feedback on energy consumption.
The usability study indicates that users can have difficulty interpreting basic information
presented in graphs, and do better when simple numbers and messages are provided. This
finding should be considered when evaluating graphic and information rich interfaces that
may be confusing to users.
Results also indicate that there are some pitfalls in using social data comparisons
(comparisons with neighbors); users seem to lose motivation to save energy when
neighbors are seen as using more.
Finally, there is strong evidence for the effectiveness of general energy saving tips and
specific energy usage diagnostic messages as a type of IHED information. IHEDs in the
market today do not present such messages, but our findings suggest that they would be
very effective in making IHED information actionable.
The following recommendations are based on the results from this project:
Gadget vs. Tool Framing: IHEDs or other energy consumption feedback devices
should be provided in the context of a planned energy savings action program such
as providing goals to reduce energy use.
Conduct additional research on the impact of general energy saving tips and
diagnostic information and messages that are provided directly in the IHED while
people are checking their energy usage.
Engage manufacturers to develop and incorporate diagnostics algorithms into their
devices, since this was found to be quite beneficial, but it is not possible with today’s
technology.
Expand research on the effects and desirability of social comparison information,
including possible “boomerang” effects.
Expand research on the impact of complex, graphic and information rich IHED
devices, focusing on users’ ability to extract correct and actionable information.
Evaluate simple, clear messages as part of the IHED information. These may be
more useful and satisfying to users.
IHED Usability scores, like the one proposed here, should combine traditional
usability measures with an effectiveness usability measure, gauging whether the
IHED provides information that facilitates energy saving actions.
Conduct further research into deployability and usability of particular models prior to
including them in any incentive programs.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page iv Design & Engineering Services December 2012
ACRONYMS
CO2 Carbon Dioxide
HAN Home Area Network
HTSDA HVAC Technologies and System Diagnostics Advocacy Program
IHED In-Home Energy Display
kWh kilowatt Hours
UCD University of California, Davis
WCEC Western Cooling Efficiency Center
WV West Village Residential Complex
ZNE Zero Net Energy
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page v Design & Engineering Services December 2012
CONTENTS
EXECUTIVE SUMMARY ______________________________________________________ II
INTRODUCTION __________________________________________________________ 1
BACKGROUND __________________________________________________________ 2
ASSESSMENT OBJECTIVES __________________________________________________ 3
TECHNOLOGY/PRODUCT EVALUATION ________________________________________ 4
Review of Market Available IHEDs ................................................................. 4
IHED Spectrum ...................................................................................... 5 Characterization of Market IHEDs ........................................................... 10 Heuristic Evaluation .............................................................................. 11
TECHNICAL APPROACH/TEST METHODOLOGY _________________________________ 17
Basic Information ...................................................................................... 17
Contextualizing Information ........................................................................ 17
Decision Making Tools ................................................................................ 17
Test Plan .................................................................................................. 18
Information ......................................................................................... 19 Meaning/Context .................................................................................. 19 Actions/Decisions ................................................................................. 19 Mockup Designs ................................................................................... 20
RESULTS_______________________________________________________________ 22
A final evaluation of the devices Initial Evaluation .................................... 23 Information Accessibility ....................................................................... 25 The Effect of Social Information ............................................................. 27 Taking Action ....................................................................................... 30
DISCUSSION ___________________________________________________________ 34
CONCLUSIONS _________________________________________________________ 37
RECOMMENDATIONS ____________________________________________________ 39
APPENDIX A ___________________________________________________________ 40
Research Team ......................................................................................... 40
APPENDIX B ___________________________________________________________ 42
APPENDIX C ___________________________________________________________ 43
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page vi Design & Engineering Services December 2012
APPENDIX D ___________________________________________________________ 44
Usability Controlled Experiment .................................................................. 44
Questionnaire Items ............................................................................. 44
APPENDIX E ___________________________________________________________ 49
APPENDIX F ___________________________________________________________ 67
GLOSSARY ____________________________________________________________ 69
REFERENCES ___________________________________________________________ 70
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page vii Design & Engineering Services December 2012
FIGURES Figure 1. IHED Hardware (Clamp-on sensor, transmitter, display)
(source: http://www.electricity-monitor.com/wireless-
energy-monitors-c-37.html) .......................................... 6
Figure 3. Classification of the HAN Market (Source: GreenTech
Media) ........................................................................ 9
Figure 5. Sample Mockup Display with three navigation screens.
The first two screens were available for navigation when
participants were first presented with the study. The
third screen was available for the second part of the
study, and appeared as a “pop-up” on the display. ......... 21
Figure 6. Demographics: Age distribution of participants. (n=249) ... 22
Figure 7. Demographics: education level of Usability Testing
Controlled Experiment participants. (n=249) ................. 23
Figure 8. Evaluations of the mockup IHED interface along a series
of hedonic and utility dimensions. ................................ 24
Figure 9. Frequency of respondents’ subjective valuation of IHEDs,
in US Dollar ranges. (N =249) ..................................... 24
Figure 10. Percentages of correct, incorrect, and “unsure”
responses to 4 multiple choice questions testing ability
of respondents to extract information from IHED
interface. .................................................................. 26
Figure 11. Participants did better at responding to questions in
screen 1 about cost (1.1kW; top left corner screen 1)
and cost (16 cents; data point in screen 1), than to
questions in screen 2 requiring responding to a data
point (500kW; bottom screen 2) and projecting a trend
(116 dollars, bottom screen 2). .................................... 26
Figure 12. Comparison of responses to 2 information questions,
comparing participants who saw displays with vs.
without budgetary information. Budgetary information
included a cost projection trend drawn in red in the
graph, but it was a more complex screen. Questions are
“hourly cost” and “projected cost” in Table 4 above. ....... 27
Figure 13. Comparison of response patterns to the question on
comparative energy usage (“The household ‘s energy
use seems to be”….), across three conditions of social
information available. ................................................. 28
Figure 14. Comparison of response patterns to the question on
normative energy usage, across three conditions of
social information available. ........................................ 29
Figure 15. Comparison of response patterns to the question on
Absolute judgment on energy usage, across three
conditions of social information available. ...................... 30
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page viii Design & Engineering Services December 2012
Figure 16. t-test p <.001 from no extra info to Tips or Diagnostics.
No significant difference found between Tips and
Diagnostics ................................................................ 31
Figure 17. t-test p <.001 from no extra info to Tips or Diagnostics.
No significant difference found between Tips and
Diagnostics. ............................................................... 31
Figure 18. t-test p <.001 from no extra info to Tips or Diagnostics.
No significant difference found between Tips and
Diagnostics. ............................................................... 32
Figure 19. t-test p <.001 from no extra info to Tips or Diagnostics.
No significant difference found between Tips and
Diagnostics ................................................................ 32
Figure 20. Respondents’ liking and usefulness ratings of a series of
IHED mockup features. ............................................... 34
Figure 21. Usability scores compared across kind of actionable
information provided (none, general tips, specific
diagnostics), and kind of social comparison provided
(usage higher than neighbors, Usage lower than
neighbors, No social information) . ............................... 35
Condition 1. Basic Information Only x Specific Diagnostic
Information ............................................................... 49
Condition 2. Basic Information Only x General Tips ......................... 50
Condition 3. Basic + Budget x Specific Diagnostic Information ......... 51
Condition 4. Basic + Budget x General Tips ................................... 52
Condition 5. Basic + Social Information Higher Than Neighbor x
Specific Diagnostic Information .................................... 53
Condition 6. Basic + Social Information Higher Than Neighbor x
General Tips .............................................................. 54
Condition 7. Basic + Budget + Social Information Higher Than
Neighbor x Specific Diagnostic Information .................... 55
Condition 8. Basic + Budget + Social Information Higher Than
Neighbor x General Tips .............................................. 56
Condition 9. Basic + Social Information Lower Than Neighbor x
Specific Diagnostic Information .................................... 57
Condition 10. Basic + Social Information Lower Than Neighbor x
General Tips .............................................................. 58
Condition 11. Basic + Budget + Social Information Lower Than
Neighbor x Specific Diagnostic Information .................... 59
Condition 12. Basic + Budget + Social Information Lower Than
Neighbor x General Tips .............................................. 60
Condition 13. Basic x No Energy Saving Suggestions ...................... 61
Condition 14. Basic + Budget x No Energy Saving Suggestions ........ 62
Condition 15. Basic + Social Information Higher Than Neighbor x
No Energy Saving Suggestions ..................................... 63
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page ix Design & Engineering Services December 2012
Condition 16. Basic + Budget + Social Information Higher Than
Neighbor x No Energy Saving Suggestions .................... 64
Condition 17. Basic + Social Information Lower Than Neighbor x No
Energy Saving Suggestions ......................................... 65
Condition 18. Basic + Budget + Social Information Lower Than
Neighbor x No Energy Saving Suggestions .................... 66
TABLES Table 1. Multiple choice questions gauging ability of participants to
extract correct information from IHEDs. ........................ 25
Table 2. Complete wording of questions evaluating users’ liking and
perceived usefulness of IHED features .......................... 34
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 1 Design & Engineering Services December 2012
INTRODUCTION Currently, homeowners have very little access to information about how much energy they
are consuming. Paper utility bills have limited usefulness in allowing a customer to track
their energy use and compare it with earlier usage or with others. Most utilities have on-line
functionality that allows users to view historical energy use, by bill (or month). The recent
advent of “smart meters” (meters that collect usage data at more frequent intervals and can
be configured to provide this information to utility or third-party applications) makes it
possible to provide hourly historical energy consumption data. This still falls short of the
desire to view current energy use.
Energy feedback in-home displays and web interfaces allow residential occupants to monitor
their energy use at home or check their energy use remotely. In-Home Energy Displays
(IHEDs) are increasingly being developed, produced, and offered in the market as a tool to
reduce energy consumption and energy waste by residential users, as well as to facilitate
demand response and home automation. Past research on displays shows a wide range of
effectiveness rates in producing energy savings1, that may partly depend on the ease of use
of specific devices and on their different available features such as feedback format,
information and content provided, ability to set individual goals, comparisons of energy use
with other users2, or past use, etc. The usability of these displays, the ability of occupants
to perform certain tasks and successfully extract information, has not been explicitly
examined. However, similar studies of usability in thermostats have revealed a significant
amount of confusion leading to inability to perform simple tasks in a reasonable amount of
time3
Additionally, many energy feedback displays are now offered as integrated systems that not
only provide information, but also allow the user to remotely control appliances and other
energy - consuming services. An even more complex feature of energy feedback displays
that holds promise is the ability to provide specific diagnostic information about household
energy use at the appliance/service level. Specific diagnostic information may be key,
especially considering that general “tips” or advice for saving energy can be ignored by
consumers, even when they have asked for them4 . The ideal, of course, is that the feature
offered by an energy feedback interface both motivates and enables energy conservation
choices by users.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 2 Design & Engineering Services December 2012
BACKGROUND There are obstacles to the successful implementation of energy feedback devices and
interfaces. The market of energy feedback devices –whether in-home displays, networked
systems with online dashboard access and apps, or whole house integrated control systems,
is currently in a state of flux and confronts the potential consumer with a number of
uncertainties. There are no standardized formats for the content provided by an energy
feedback display, nor is there a single standardized way for the device to connect with home
appliances, smart-meters, or household electrical panels. (Standards such as Zigbee, Wi-Fi,
SEP have all been used with different systems.) Residential household consumers motivated
to acquire an energy feedback device for their household may find it difficult to determine:
Whether a device works with their electrical system or utility
What information is available from the display
What features are available
How the device helps in energy savings.
If a consumer makes the decision to buy a device, its installation can prove quite
challenging. If consumers are expected to proactively seek and purchase market available
devices, it may be important to evaluate how they respond to the scope of existing offers
and their characteristics.
Confusing market offers and usability problems may be significant obstacles to the adoption
of these devices and their success in the market. In the current state of development and
innovation it may well be that the increasing complexity of device features is making
consumers hesitant to engage with these products, and frustrated when they use them for
the first time5.
This project used a controlled usability experiment to explore differences in effectiveness
between different features available in market IHEDs. Researchers created a mockup
simulated IHED and varied the amount, complexity, and kinds of information offered. The
study then measured users’ ability to extract correct information and use the information to
make energy saving decisions.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 3 Design & Engineering Services December 2012
ASSESSMENT OBJECTIVES The goal of the project was to evaluate the usability of a device that can help reduce the
energy consumed by SCE ratepayers by providing them with detailed information on energy
use that they can use to make informed decisions. The project explores IHED device
interface features currently available in the market, as well as promising potential features
(such as diagnostic messaging) that have not yet been demonstrated, and evaluates their
effectiveness in a controlled experimental setting.
This study intends to advance our understanding of IHED usability barriers, and provide
recommendations on preferred usability design for this kind of device. The study also
evaluates whether different forms of information provided in IHEDs are helpful in motivating
and enabling residential end users to make decisions that may lead towards energy savings.
The results of the review of available models and characterization of capabilities are
summarized in Table 1.
Findings of this study are followed by recommendations about:
The most effective IHED features for design
Future research projects
How to choose devices for pilot programs
It also suggests the research and development of features that, while not currently available
in the market, appear to hold promise as enablers of effective consumer choice regarding
energy use.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 4 Design & Engineering Services December 2012
TECHNOLOGY/PRODUCT EVALUATION The first step in evaluating the usability of IHEDs, was to examine the products available in
the consumer and utilities’ market and to identify common characteristics of IHEDs that can
be subject to a usability evaluation.
The wide range of offers in the IHED market, and its fluidity (with products being constantly
dropped or updated), makes this a particularly difficult task. The continuum extends from
first generation IHEDs that offer only limited data on energy used and cost, to automated
home control systems that go way beyond energy in their information and control tools.
This makes it almost impossible to determine a single standard usability evaluation protocol
that is valid across the spectrum. This report provides an extensive list of IHED and energy
monitoring systems available in the market for individuals, utilities, or developers, and
provides a characterization table of available features for each IHED identified. This list
serves, mostly, as an indication of the heterogeneity of the IHED market.
The second step involved a heuristic examination of four devices acquired by the research
team. Given the market conditions, the research team restricted the heuristic evaluation of
displays to the subset of those available for end user purchase, since they were the only
ones available. Thus, the heuristic evaluation focused only on devices that were available to
be purchased and installed by individual residential customers, and that focused exclusively
on electricity feedback6. It is the characterization of this subset of IHEDs, used to develop
the usability testing protocol, to be used in the controlled experiments.
REVIEW OF MARKET AVAILABLE IHEDS An extensive list of IHEDs and other energy monitoring systems available in the
market was compiled by the research team (see Appendix A). First, the authors of
this report who have expertise in feedback and control devices reviewed the recent
literature on the topic (e.g., Darby 2006, Fischer 2008, Neenan 2009, Froehlich
2010, Ehrhardt-Martinez, 2010, Karlin 2011, LaMarche 2012). Some of the devices
included in this list come from these reports. The list was supplemented with devices
mentioned in technical magazines, online e-zines consulted during fall/winter 2011
and from online searches for energy feedback displays. Examples of the e-zines and
magazines consulted are:
http://www.greentechmedia.com/
http://news.cnet.com/greentech/
http://gigaom.com/cleantech/
http://www.smarthome.com/_/index.aspx
http://www.automatedbuildings.com/
http://www.ashrae.org/resources--publications/periodicals/ashrae-journal
http://www.homeenergy.org/
Researchers also contacted manufacturers, energy consultants and other energy
feedback researchers to help complete the inventory. The authors tried to be
comprehensive in the survey, gathering all the information to choose the best
devices based on range of capabilities and range of different feedback formats. This
market is in rapid and constant change and this list is a snapshot in time. WCEC
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 5 Design & Engineering Services December 2012
found a wide variety of devices with different functions and interface options that
currently have almost no market penetration.
After collecting extensive information on all kinds of devices and systems available in
the market, the team created an extensive list of market available IHEDs, provided
in Appendix B. This list shows the broad spectrum of devices and their features, as
they are described by the vendors. The following section analyzes the variability in
the IHED spectrum in more detail.
IHED SPECTRUM
IHEDs are devices whose main function is to show the instantaneous and cumulative
energy consumption of a home or an appliance. Most of them also display the cost of
the energy used, calculated based on either data input by users or information
coming from Utilities. IHEDs have been investigated by utilities in several Energy
Efficiency (EE) and Demand Response (DR) projects in the past few years. Examples
include, SMUD (small commercial thermostat and IHED for DR and EE, 2010), TXU
(Smart thermostats BrightenSM iThermostat for DR and EE, 2010), Bluebonnet
Electric Cooperative (Control 4 home automation for DR and EE, 2009), Reliant of
Texas (Nest, 2012), and BC Hydro (2009).
Goals of these projects were to increase customers’ awareness and reduce energy
consumption and peak demand. Through price signals (variation of the cost of
electricity in peak periods) sent to the displays, users were encouraged to voluntarily
decrease their energy consumption. This approach is significantly different from
taking direct control of customers’ equipment to match Utility’s needs and arguably
more acceptable for consumers. Traditionally IHEDs have been designed to monitor
electricity, but recent models also display gas and water consumption. The first
generation of these devices simply displayed power (kW) being used instantly and
cumulative energy (kWh) used in a period of time. An example of such architecture
is shown in Figure 1. The device records power directly from the utility meter
through a clamp-on sensor.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 6 Design & Engineering Services December 2012
FIGURE 1. IHED HARDWARE (CLAMP-ON SENSOR, TRANSMITTER, DISPLAY) (SOURCE:
HTTP://WWW.ELECTRICITY-MONITOR.COM/WIRELESS-ENERGY-MONITORS-C-37.HTML)
The Smart Grid is a class of technology used to bring utility electricity delivery
systems into the 21st century, offering many benefits to utilities and consumers --
mostly seen in big improvements in energy efficiency on the electricity grid and in
the energy users’ homes and offices. With the deployment of the Smart Grid and
availability of less expensive communication components, new companies started
developing products that are more sophisticated.
The second generation of IHEDs is part of a network of connected devices creating a
Home Area Network (HAN in Figure 2). Sensors gather information on energy use
from the Utility meter and appliances; and that information is displayed through
multiple interfaces. These systems are usually connected to the Internet and have a
web interface, as well as a physical interface in the house. Some more modern
products also have smart phone/tablet applications. Figure 2 shows how IHEDs are
now part of a more complex network that partially resides inside the wall of the
house but more and more relies on remote services allowed by Internet connection.
These services span from remote climate and lighting control to security and
entertainment.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 7 Design & Engineering Services December 2012
IHEDs are sometimes integrated with controls (e.g., smart thermostats) to share the
hardware in order to be less expensive. The new network now has all the capabilities to
provide users with comprehensive information and a convenient way of acting on them.
Connection to the web and availability of interval meter data also allows third parties to
interact with the system, offering services for the customer such as demand response
(e.g.,EnerNoc), energy analysis and benchmark (e.g., Opower, Efficiency 2.0 ), fault
detection and performance diagnosis (EcoFactor, NEST). Some companies also plan on
extending service including remote health monitoring. Integration and interoperability are
important challenges for developers.
FIGURE 2. HAN ARCHITECTURE
The main features offered by more recent devices (introduced since 2010) are listed
in Table 1. Frequency is shown to be Common if it appears in almost all devices. It is
shown as Uncommon if it appears in only 25% or fewer devices, and it is shown as
Rare if it is only in a few devices.
TABLE 1: CHARACTERISTICS OF IHEDS ON THE MARKET
FEATURE FREQUENCY
ENERGY MONITORING
-Energy and Cost monitoring Common
ENERGY CONTROL
-Climate Control (smart thermostat) Common
-Lighting control Uncommon
-Smart Windows Rare
-Appliance load control (through smart plugs, load control modules and smart appliances)
Common (smart plugs)
to Rare (appliances)
-Automatic load shift depending on energy price Rare
ENERGY/BEHAVIORAL SERVICES
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 8 Design & Engineering Services December 2012
-General or Tailored Tips on how to save energy Rare
-Energy Budget Rare
-Social Context (comparison with neighbors or friends)
Rare
-Remote Home Auditing Rare
-Continuous commissioning (...) Rare
ENTERTAINMENT
-Entertainment (integration with TV, audio, wireless audio distribution, etc.)
Uncommon
SECURITY
-Security Uncommon
-Remote house monitoring (alarm, cameras, remote door opening, system status, intercom)
Uncommon
HOME HEALTH CARE:
-Remote health care (monitoring, assisted living, emergency aids, medication schedules)
Rare
-Baby monitor and intercom Rare
INTEGRATION W/GENERATION
-Integration with Solar/Wind systems Rare
DEMAND MANAGEMENT (FOR UTILITIES)
-Demand Response (for Utilities) (price signal, load control, text messages)
Uncommon
OTHERS
-Pool/Spa pump control Uncommon
-Irrigation (schedule the irrigation system) Rare
-Electrical Vehicle charging Rare
Devices differ significantly in terms of features offered, choices in interface design
and market strategy. Most of the players actively look for collaboration with Utilities
as the first channel to increase sales. A few others target high end users (home
automation companies) or plan to use their existing sales channels (e.g., security or
thermostat companies).
Despite continuous activity and attraction of capital from different investors, the
market penetration of these devices is still very low. In 2010-2011, several start-ups
were acquired by big players such as Verizon, Motorola and Comcast. Other players
in the Demand Response (demand aggregators) started a process of vertical
integration with HAN manufacturers. At the same time GE, Whirlpool and other
manufacturers are working on a new generation of smart appliances (less expensive
and integrated with the smart grid).
However, the lack of a single standard protocol, the unclear value proposition for the
customers and the high cost of these systems ($300-$1000 according to the survey)
are casting shade on the potential market for these products in the next few years.
The current market (2011-12) is changing fast with big players entering (GE, 3M,
and Cisco) and some leaving it (Google and Microsoft) because the market is so
immature. Figure 3 shows the classification of this market according to a report of
the web magazine GreenTech Media.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 9 Design & Engineering Services December 2012
FIGURE 3. CLASSIFICATION OF THE HAN MARKET (SOURCE: GREENTECH MEDIA)
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 10
Design & Engineering Services December 2012
CHARACTERIZATION OF MARKET IHEDS
The information provided above and in Appendices B and C provides a systematic
overview of the most representative IHEDs and HAN systems available in the
market, and the features they describe in their websites or other promotional
materials and in product reviews. It is difficult to characterize each system
exhaustively, because only specs that each company decides to promote in their site
are available, and the amount of information provided can vary widely across
different companies.
At this point, this is a very fluid market, and there is no standard set of specifications
that companies are required or expected to provide. For example, if a product
indicates that it can provide “current usage information,” this can be either
instantaneous kW, kWh for the last hour, cost for the last hour, cumulative energy so
far today, etc. In addition, it is not always clear whether the company is describing
an existing product’s features, promising future features, or describing features that
are available only with access to other technologies or products.
It is extremely difficult to keep track of all the companies and products (probably
more than a hundred) being developed. The technical specs are listed in a
spreadsheet in Appendix C. The spreadsheet includes the following categories:
Market: Company Headquarters
Sells to Utilities
Sells to Consumers
Sector: Residential (R), Small Commercial (SC), Large Commercial (LC)7
Product and Service Offering
Display
Mob App
Web App
Circuit-Box monitor
Meter Reader
Smart Plug/Strip
Load Control Module
Smart Thermostat
Lighting Control
Security
Entertainment
Healthcare
Demand Response: Yes-unspecified (x), Load Control (LC), Price Signal (PS)
Integration with Generation
Integration with EV
Architecture and communication:
Gateway: proprietary, ADSL gateway
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 11
Design & Engineering Services December 2012
Communication Meter-Device: Zigbee (ZB), Z-Wave (ZW), RDS (RD), Wi-Fi
(WF), Wi-Max (WM), Powerline (PW), Others (O)
Communication Device-Utilities: Radio (R), Web (WB), Power Line (PW), AMI
(AMI),
Communication Device-3rd Parties: Radio (R), Web (WB)
Meter Connection: clamp-on (CO), CT
Interface:
Displays energy: Whole house (W), single appliance (A), circuit breaker
(C)
Displays Real time Energy Information
Displays Current Price
Displays historical data
Displays also: gas (G), water (WT)
Displays other messages: temperature, weather forecast, text
messages
Goal setting
Provides suggestions on how to reduce energy
Social context
Data Storage and Analysis:
Customer Data stored: locally (L), cloud (C)
Partners
The team also developed a summary of the main features available in IHEDs,
grouped by frequency (common features vs. rare features).
TABLE 2. SUMMARY OF COMMON, UNCOMMON AND RARE FEATURES IN MARKET IHEDS
Hardware and Software IHED Feature Summary
Real-Time Time Averaged
(Day) Time Averaged
(month) Projection
(month) Thresholds &
Alerts Display Styles Data Collection
Common $/hr, kW $,kWh $, kWh, CO2 Numeric only
(hardware devices), bar charts
Zigbee, CT-Clamp
Uncommon CO2/hr Day-day
comparison, CO2, social comparison
Social comparisons, comparison to
previous month, comparison to
target
$, kWh Preset color
changes Line graphs
Rare Change in
consumption (delta kW)
$ & CO2 savings, peak power
demand Peak demand
Comparison to target
User-settable color changes, budget based threshold,
user advice
Pie charts, speedometers
HEURISTIC EVALUATION
In order to develop a usability test procedure for IHEDs, the research team
heuristically evaluated a sample of market-available residential IHEDs. Four energy
feedback displays were acquired and examined in detail by the research team,
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 12
Design & Engineering Services December 2012
adapting usability guidelines as developed by Peffer et al., 20128 developed for
programmable thermostat usability. These guidelines include evaluating the device
for
Walkup usability/visibility: User must be able to see most important features
from the initial view of the device.
Feedback: The device offers confirmation of settings or other operations conducted
by the user.
Standards: Terms, icons, and common actions available correspond to standard
information and possible operations that are expected by the user.
Terminology: Terms and abbreviations are comprehensible to the user, or
definitions are easily available.
Broad and shallow decision tree: The device does not present the user with too
many layers (information screens) that make it difficult to find information or
controls.
Hierarchy of display: The device presents important and more frequently used
features more prominently than less used features and information.
Affordances: Affordances of the device are easy to identify by the user (no hidden
navigation buttons, or covers). Navigation options and buttons are clearly
identifiable.
Recovery from errors: The device promptly offers feedback and capability to
correct errors or recover settings.
Researchers decided to focus on IHEDs that were available for purchase and
installations by individual residential consumers, and that focused exclusively on
electrical usage feedback. The four IHEDs purchased correspond to three different
kinds of devices:
1. first generation IHED that display numerical information of energy use at a
whole house level (TED 5000-C and Envi Energy Monitor);
2. Smart power strips that allow for energy monitoring at the plug-level and
present information via an online or mobile interface (UFO Power Center);
and
3. home systems that allow for power strip, thermostat and high voltage plug
monitoring, and present the information through a color and graphics-rich
IHED. The last two kinds also include the possibility of remote controlling of
plug loads.
The 4 IHEDs were acquired and evaluated by the research team.
TED 5000-C Home Electricity Monitor
Envi Energy Monitor
UFO Power Center
Energy Hub Home Base
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 13
Design & Engineering Services December 2012
TED 5000-C Envi Energy
Monitor
UFO Power Center Energy Hub
FIGURE 4. THE FOUR EVALUATED IHEDS : TED5000, ENVI, UFO WITH ONLINE INTERFACE, ENERGYHUB SYSTEM
Research team members installed each of the energy saving devices in their homes
or offices, and activated as many device features as possible. The team members
then accessed the device’s features, while evaluating them according to the criteria
described above (when applicable). Researchers also noted additional usability issue
not covered by the guidelines above, especially regarding installation problems and
interpretations problems.
First, however, the devices were categorized according to the feature offered, as
seen in the table below. (The UFO is described twice: UFO HW refers to the hardware
plug strip that provided some feedback on site, UFO SW refers to the mobile app
software that provided feedback that is more sophisticated).
Below is a characterization of the features available for each of the devices acquired.
(Blank spaces indicate that the feature is not available).
TABLE 3. CHARACTERIZATION OF IHEDS ACQUIRED AND HEURISTICALLY EVALUATED
DEVICE TED HW UFO HW UFO SW ENVI-R HW
Real-Time Cents-per-hour; Watts
Watts (color, high. Med. Low threshold)
Watts per plug & total ;
watts
Time Averaged (Day)
Cumulative Cost; Cumulative kwh
cost; energy cost; energy
Time Averaged (Week)
cost; energy energy
Time Averaged (month)
Cumulative Cost; Cumulative kwh
cost; energy cost
Time Averaged (short-term)
15 minute cost; energy
yesterday
night/day/evening bars
Projection (month) Cumulative Cost;
Cumulative kWh ; Remaining days
Detail (day - kw) current; lowest; highest
Detail (day - $) current; lowest; highest
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 14
Design & Engineering Services December 2012
DEVICE TED HW UFO HW UFO SW ENVI-R HW
Detail (day-CO2 lbs/hr)
current; lowest;
highest; Month cumulative
Other Voltage high/low,
time & day of month
time and temperature
Alerts / thresholds no user settable thresholds
notification of
momentarily higher or lower usage
display styles numeric light numeric and graphical (bar charts dial graphs)
numeric and graphical for retrospective data
notes sensitive to WiFi easy setup
granularity house plug plug house + appliances with additional
plug-level devices
controls none none plug-level; scheduling
none
meter interface CT clamps, power
cables & power line communications
none none CT clamps +
wireless broadcast
Real-Time Cents-per-hour; Watts
Watts (color, high. Med. Low threshold)
Watts per plug & total ;
watts
Time Averaged (Day)
Cumulative Cost; Cumulative kwh
cost; energy cost; energy
Time Averaged (Week)
cost; energy energy
Time Averaged (month)
Cumulative Cost; Cumulative kwh
cost; energy cost
Time Averaged (short-term)
15 minute cost; energy
yesterday
night/day/evening bars
Projection (month) Cumulative Cost;
Cumulative kWh ; Remaining days
Detail (day - kw) current; lowest; highest
Detail (day - $) current; lowest; highest
Detail (day-CO2 lbs/hr)
current; lowest;
highest; Month cumulative
Other Voltage high/low,
time & day of month
time and temperature
Alerts / thresholds no user settable notification of
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 15
Design & Engineering Services December 2012
DEVICE TED HW UFO HW UFO SW ENVI-R HW
thresholds momentarily higher or lower usage
display styles numeric light numeric and graphical (bar charts dial graphs)
numeric and graphical for retrospective data
notes sensitive to WiFi easy setup
granularity house plug plug house + appliances
with additional plug-level devices
controls none none plug-level; scheduling
none
meter interface CT clamps, power
cables & power line communications
none none CT clamps +
wireless broadcast
After acquiring and installing the devices, researchers became aware of usability
issues unique to each device. Those unique findings are presented here by device,
and common issues are addressed in the discussion section. Several noteworthy
issues affected the acquisition and installation of the devices, and are described here
because they are relevant to the market penetration of IHEDs. Consumers motivated
to purchase IHEDs probably find the same set of barriers, if they engage in the IHED
market as it stands today.
The market penetration of these devices is still very low (fewer than 1% of
homes), due in part to the lack of standard protocols, the unclear value
proposition for the customers and the high cost of these systems.
Devices in the market vary significantly in terms of featured offered, choices
in interface design and market strategy. At this point, this is a very fluid
market.
It is not always clear whether the company is describing an existing product’s
features, or promising future features, or features that are available only with
access to other technologies or products.
Sales of IHEDs online are most commonly targeted to utilities and building
developers. Consumers may invest significant time researching a given
product online before it becomes evident that they may not actually be able
to purchase it for their homes.
For IHEDs that do or do not allow purchase by individual consumers, it may
be hard for consumers to understand what they are purchasing. Research
found that some heavily promoted features were not actually available on the
products purchased, but rather were “upcoming”, or would be available
“soon”. Examples of such features were remote control of plugs, or web
portals for feedback.
Consumers may also find it difficult to know whether the device they are
purchasing works with their system. Product websites were often vague about
Smart Meter connectivity (“works with most SmartMeters”), and failed to
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 16
Design & Engineering Services December 2012
inform consumers that compatible Smart Meters might still not work if
wireless capabilities were not activated by the utilities.
Consumers may be confused about the scope of energy monitoring possible.
IHEDs based on power strips fail to inform consumer that large energy users
are left out of the monitoring (Air conditioners, washers, dryers).
Product websites were also ambiguous about the need for electrician assistant
for installation (“we recommend an electrician assist you, but you can also do
it yourself”). Installation usability should be evaluated if planning involves
user acquiring their own devices.
Most IHEDs are not designed to work with other IHEDs or other devices within
the home. Interoperability is not addressed.
There is no proof that these systems save energy, especially for new players.
Usability, efficacy, energy savings are all currently unstudied.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 17
Design & Engineering Services December 2012
TECHNICAL APPROACH/TEST METHODOLOGY This study examines, in controlled experimental conditions, users’ reactions and responses
to simulated energy feedback interfaces that are representative of devices available on the
market. The energy feedback interfaces used in the pilot and the full study were abstracted
from the heuristic expert evaluation of representative market devices described earlier in
this report, and correspond to varying levels of complexity. Experiment participants then
respond to questions that gauge their ability to retrieve basic information, put that
information in a meaningful energy use context, and use that information to make energy
use decisions.
As part of the experimental design (described in more detail below), different participants
are randomly assigned to one of eighteen different energy interface conditions. Each
presents a different combination of features. Features presented in the device fall into three
categories:
BASIC INFORMATION Display of basic energy use and cost features, corresponds to the “common” set of
features in our heuristic evaluation. Information provided is almost never absent,
from even the simplest of devices. All experimental conditions include this basic
information, and the condition that includes only basic information is the control
condition in this study. Basic information includes current power usage (W),
cumulative day usage (kWh), pattern of hourly and daily usage (graph with daily
hours on x axis, and cents-dollars and kWh on y axes), and cumulative monthly
usage (graph with days of the month on x axis, and dollars and kWh in y axes).
CONTEXTUALIZING INFORMATION The display’s information allows the user to compare basic information with an
external standard, potentially leading to a motivation to reduce energy use. In this
study, contextualizing information is presented in the form of either a social
comparison with similar households, or a pre-set energy usage goal. Research
indicates that social comparisons, or “anchors”9 (Kahneman, 2011) act as descriptive
social norms that provide a motivation for people to act in accordance with the
norm1011. Goals act as strong motivators for individual action and change, and they
have been shown to be effective motivators in the domain of household energy
behaviors121314.
DECISION MAKING TOOLS The display presents general tips for energy saving that may be used if users are
motivated to reduce energy usage. The display also includes diagnostic tips to alert
the user that a hypothetical appliance is wasting energy because of some form of
malfunction.
This controlled randomized trial test experiment is designed to determine whether
and/or how much each layer of information (Basic, Contextual and Diagnostic)
increases the individual’s perception of the need to reduce energy use or the
magnitude of any actions they feel are appropriate.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 18
Design & Engineering Services December 2012
TEST PLAN The experiment was a randomized controlled trial. The test varied the display
characteristics of a simulated energy feedback interface, including available basic
information, set budgets, and social information. Participants were randomly
assigned to either a device with basic information alone (control), a set budget, or
social comparison information (usage is higher or lower than neighbors). In a second
part of the experiment, subjects were randomly assigned into three groups: those
shown no additional information, those shown general tips for energy saving, and
those shown specific diagnostic information for energy saving.
Using the energy feedback interface they are assigned to, participants are asked to
complete several rounds of information retrieval tasks (finding and reporting
information provided in the feedback interface) and decision making tasks (choosing
appropriate actions to reduce energy usage depending on the feedback). After
completing three rounds of tasks, participants are asked questions about their
experience with the interface and some demographic questions.
Participants in the study were recruited in three different locations in the cities of
Davis and Fairfield, in California. One location was a campus cafeteria, one was a
grocery store and one was a large mall. The study was conducted at each of these
locations using a stand or table in a public high-traffic spot. Passers-by were asked
to participate in exchange for a five dollar gift card. Researchers stayed at a table
with six small tablet computers that were programmed to display the mockups as
well as an integrated questionnaire. Participants used these tablets to access the
energy feedback displays and responded to the questions and tasks required by the
study.
Participants were randomly assigned to one of six energy feedback display formats,
and to one of three energy reduction information conditions (see design summary in
Table 3 below). Devices and questions were presented using MediaLabTM
experimental software that allows display of the information, automatic recording of
the questions, and tracking of device navigation (hyperlinks followed and time spent
at each screen).
After reading the consent form (see Appendix H) and accepting to participate in the
study, participants read the first set of instructions that asked them to familiarize
themselves with the energy feedback display. The feedback display then appeared on
the screen, and participants had some time to explore the display by clicking through
the device’s screens and seeing what information was provided, and what were the
different feedback navigation paths. After the participant had explored the display,
he/she was directed to the first round of questions. The full questionnaire is available
in Appendix F.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 19
Design & Engineering Services December 2012
TABLE 4. FULL EXPERIMENTAL DESIGN SCHEMATIC TABLE.
FIRST PART: KIND OF ENERGY FEEDBACK PROVIDED
BASIC
ONLY
BASIC
+
BUDGE
T
BASIC
+SOCIAL
INFORMATION
HIGHER THAN
NEIGHBOR
BASIC
+SOCIAL
INFORMATION
LOWER THAN
NEIGHBOR
BASIC
+BUDGET +
SOCIAL
+INFORMATION
HIGHER THAN
NEIGHBOR
BASIC
+BUDGET
SOCIAL
+INFORMATION
LOWER THAN
NEIGHBOR
SECOND
PART:
Energy Saving Suggestions Provided
No Energy Saving Suggestions
13 14 15 17 16 18
General Tips 2 4 6 10 8 12
Specific
Diagnostics
1 3 5 9 7 11
The numbers in the cells represent the 18 different conditions that participants were
randomly assigned to (See Appendix G).
Participants responded to three types of questions: Informational, Meaning/Context
and Actions/Decisions.
INFORMATION
Questions such as “according to the display, how much energy has the household
spent so far this month?” or “what is the projected electricity bill for the household?”
These questions are intended to gauge whether users can identify the kind of
information provided in the display and how hard or easy it is for them to access it.
Right and wrong answers to the tasks allowed the researchers to determine whether
users can obtain accurate information from the device. Ease of use is gauged by
measuring how quickly respondents are able to answer the questions, and how many
navigation screens they had to go through to reach the answer.
MEANING/CONTEXT
Questions such as, “is the energy use of the house shown in the previous screen low,
average, or high?” or, “does the energy use of the house suggest to you that energy
use should be reduced?” These questions are intended to gauge whether the
information provided in the feedback interface can be put into a meaningful context
by the participants.
ACTIONS/DECISIONS
Participants were provided with a variety of energy saving options with an increasing
cost or effort (the options were “change your thermostat setpoint”, “call a
professional to maintain your air conditioner”, and “replace your air conditioner”),
and asked which they felt was appropriate given the feedback. These questions are
intended to gauge whether information in the device can be used to inform and
motivate energy saving decisions.
After responding to this first round of questions, participants were allowed to access
the feedback interface again. The interface now included extra information on energy
reduction in either tip or diagnostic format. After seeing this info, they answered a
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 20
Design & Engineering Services December 2012
second round of questions on information, meaning and action. Finally participants
answered questions (five-point scale from strongly disagree to strongly-agree) rating
their experience with the experimental device (was it fun, frustrating, easy, hard,
boring, interesting, etc.) as well as demographic questions.
MOCKUP DESIGNS
As described in the design summary, participants were randomly assigned to view
different sets of IHED screens. The mockup screens were created based on common
ways of presenting features in market IHEDs. Figure 5 below presents two possible
sets of screens that participants were able to see. The first set corresponds to the
control condition, in that participants saw only basic information. That was the
simplest set of screens available. The second set of screens corresponds to the basic
+ budget + social information higher than neighbor usage + general tips condition.
This set of screens is an example of the most complex possible IHEDs. Simpler
versions would have presented less features, in the same kind of visual organization.
A full set of screens for all 18 conditions of the study is provided in Appendix G.
A description of each category (basic, budget, social information) is provided below,
with color references that can be used to look for the features in the sample screens.
That said, use of color was not one of the variables of interest in the study. The
colors chosen to present information (purple, orange and gray) were the ones
considered to be least likely to convey an evaluative judgment of consumption. In
general, cultural norms are that green indicates positive and environmentally friendly
behaviors, whereas red represents behaviors that are dangerous or must be stopped.
The use of green was avoided, and red only used in alerts regarding budget
information. Colors were uniform across all conditions, and, again, were not tested in
this study.
The basic information included numerical information (see large white numbers)
describing current power use, and energy usage for a day. It also included a graph
with “yesterday’s hourly usage” (gray color) and “today’s hourly usage” (orange
color). By clicking on “more info” the user can access a second screen, showing daily
usage for the month, and cumulative usage for the month (in orange). Basic
information corresponds to common features found in most of the IHEDs available in
the market.
Budget information included all of the above, plus a red alert on the top left of the
screen, alerting user to being “over budget” by 15%. It also included a red projected
line, in the bottom half graph of the second screen that represented the budget.
Social information presented included purple color graph information on the second
screen, representing neighbors’ usage. The purple overlay indicated similar
neighbors’ usage, whereas the orange indicates self usage. Social information was
presented in two ways: consumption higher than that of neighbors (as in the screen
sample below), or consumption lower than that of neighbors.
Finally, the “take action” screen included three messages recommending either
general tips for action, or suggestions based on specific measurements for each
users’ system (example shows general tips). The three “general tips” were
Replace your air conditioner! Older, less efficient air conditioners can use as
much as 30% more energy.
Call a professional to maintain your air conditioner! Not providing regular
maintenance for your air conditioner wastes 30% of its energy use
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 21
Design & Engineering Services December 2012
Change your thermostat setpoint! For each degree the thermostat is set
below 75°, 3% to 5% more energy is used.
The three “diagnostic tips” were:
Replace your air conditioner! You air conditioner is 17 years old, and uses
30% more energy than a new Energy Star model. According to our
measurements, important components of your air conditioner are worn or
near breakage, and it may only last two more years.
Call a professional to maintain your air conditioner! According to our
measurements, your air conditioner is wasting 30% of its energy use because
of airflow restrictions produced by lack of regular maintenance.
Change your thermostat setpoint! According to our measurements, your
thermostat is currently set at 68°. 21-35% more energy is being used than if
you changed it to 75°.
FIGURE 5. SAMPLE MOCKUP DISPLAY WITH THREE NAVIGATION SCREENS. THE FIRST TWO SCREENS
WERE AVAILABLE FOR NAVIGATION WHEN PARTICIPANTS WERE FIRST PRESENTED WITH
THE STUDY. THE THIRD SCREEN WAS AVAILABLE FOR THE SECOND PART OF THE STUDY, AND
APPEARED AS A “POP-UP” ON THE DISPLAY.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 22
Design & Engineering Services December 2012
RESULTS Participants were recruited at three field sites in Davis and Fairfield, CA. After removing
ineligible participants, 249 responses were saved to a final data table. The respondents
were 50% female, and typically college age as shown in the demographic figures.
FIGURE 6. DEMOGRAPHICS: AGE DISTRIBUTION OF PARTICIPANTS. (N=249)
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 23
Design & Engineering Services December 2012
FIGURE 7. DEMOGRAPHICS: EDUCATION LEVEL OF USABILITY TESTING CONTROLLED EXPERIMENT
PARTICIPANTS. (N=249)
The experiment included four broad sections addressed in subsections below:
An initial evaluation of the devices;
A section to test the participants’ abilities to access important numerical information
from the device;
The effect of social comparison information;
An assessment of intent to take action;
A FINAL EVALUATION OF THE DEVICES INITIAL EVALUATION
The initial evaluation was performed after allowing each subject to be familiarized
with a mockup and explore its features. This evaluation is therefore a “first look”
evaluation of an IHED, similar to the type of response one might have while browsing
similar products in a home-improvement store. The evaluation is completed using a
series of scale (1-6) responses to statements, where a 1 represented “strongly
disagree”, 2 represented “moderately disagree”, 3 represented “slightly disagree”, 4
represented “slightly agree”, 5 represented “moderately agree” and 6 represented
“strongly agree. ”
As shown in the figure, the subjects generally thought that the mockup designs were
easy and even fun to use, they would like to have the information at home, and
would find it useful. They did not find the screens frustrating or too simple. There
were also mixed responses on their plans to purchase a similar device, although 60%
of responses indicated that they were positively disposed towards purchasing such a
device. When asked about how much such a device might be worth, 60% of subjects
felt that it was worth more than $50, as shown in the figure below. Only 22% of
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 24
Design & Engineering Services December 2012
respondents felt that the display was worth over $100 probably the lowest possible
cost for a mass produced product. These results indicate that only about 20% of
demographically similar people might purchase an unsubsidized energy display in the
future.
FIGURE 8. EVALUATIONS OF THE MOCKUP IHED INTERFACE ALONG A SERIES OF HEDONIC AND UTILITY
DIMENSIONS.
FIGURE 9. FREQUENCY OF RESPONDENTS’ SUBJECTIVE VALUATION OF IHEDS, IN US DOLLAR RANGES.
(N =249)
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 25
Design & Engineering Services December 2012
INFORMATION ACCESSIBILITY
One of the primary functions of an IHED is to provide residents with pertinent energy
usage and cost data. The ability of individuals to extract simple information about
their home energy use and spending on energy is therefore a crucial test of IHED
usability. To test subject’s ability to extract information we constructed four multiple
choice questions. The subject was shown the display and then asked a question
about the energy usage shown on the display. The subject was free to press the
“back” button to revisit the energy display as many times as they wished to find the
information before answering the multiple choice question. To analyze the data the
answers were coded as Correct, Incorrect, and Unsure. The responses are shown in
the pie chart matrix below.
Subjects were much better at determining hourly cost and power than energy used
or projected cost. The relatively poor performance on the quiz indicates the
importance both of clear IHED design, and improved energy education of IHED users
to ensure that they can easily extract pertinent information from their IHED.
Participants were the most successful with the question where the answer can be
found as a numerical cypher in the screen (power = 1.1 kW). Participants also did
well at one question involving a direct reading of a graph’s data point (hourly cost =
16 cents), but failed at a question that required the same skill (energy use =
500kWh). The difference between these two questions, other than the unit of
measurement (cents vs. kWh), is that the one that subjects performed poorly on,
required an extra navigation screen to get to. It is possible that participants fail to
navigate to the screen with the correct energy information.
The other question with poor performance required projecting an imaginary trend
from a graph (projected cost this month = $116). This result is an indication that
simplicity in information provided is significantly clearer to consumers than
aesthetically pleasing colorful graphics. Participants may have particular difficulty
projecting trends from graphs and navigating screens to reach all information
necessary.
Referring back to the IHED mockups, the questions and correct answers were:
TABLE 1. MULTIPLE CHOICE QUESTIONS GAUGING ABILITY OF PARTICIPANTS TO EXTRACT CORRECT INFORMATION FROM
IHEDS.
QUESTION WORDING CORRECT ANSWER
Power How much power is the household using right now?
1.1 kW
Energy Approximately, how much energy
has been used so far this month in kWh?
500kWh
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 26
Design & Engineering Services December 2012
FIGURE 10. PERCENTAGES OF CORRECT, INCORRECT, AND “UNSURE” RESPONSES TO 4 MULTIPLE CHOICE
QUESTIONS TESTING ABILITY OF RESPONDENTS TO EXTRACT INFORMATION FROM IHED
INTERFACE.
FIGURE 11. PARTICIPANTS DID BETTER AT RESPONDING TO QUESTIONS IN SCREEN 1 ABOUT COST
(1.1KW; TOP LEFT CORNER SCREEN 1) AND COST (16 CENTS; DATA POINT IN SCREEN 1), THAN TO QUESTIONS IN SCREEN 2 REQUIRING RESPONDING TO A DATA POINT (500KW;
BOTTOM SCREEN 2) AND PROJECTING A TREND (116 DOLLARS, BOTTOM SCREEN 2).
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 27
Design & Engineering Services December 2012
Budgetary information was hypothesized to help individuals make the monthly
projection estimates with more ease. The budget line is shown in the display, for the
budget condition. However, as shown below, although the budget did help subjects
answer the monthly cost question, more individuals were unsure of their answer, and
importantly, the added complexity of the screen led to a reduced percentage of
correct responses for the prior question of the energy used to-date this month. Low
rate of correct responses may be due to difficulties interpreting projections in graphs,
increased confusion as more information is presented in a screen, and difficulties in
reaching the information in navigation, since it was presented in the second screen
FIGURE 12. COMPARISON OF RESPONSES TO 2 INFORMATION QUESTIONS, COMPARING PARTICIPANTS
WHO SAW DISPLAYS WITH VS. WITHOUT BUDGETARY INFORMATION. BUDGETARY
INFORMATION INCLUDED A COST PROJECTION TREND DRAWN IN RED IN THE GRAPH, BUT IT
WAS A MORE COMPLEX SCREEN. QUESTIONS ARE “HOURLY COST” AND “PROJECTED COST”
IN TABLE 4 ABOVE.
THE EFFECT OF SOCIAL INFORMATION
Social information plays a potentially important role in IHED design. Neighbors (or
similar households) can provide contextualizing information that is otherwise
unavailable from a household’s energy data on its own. For example, although
individual-level data can answer the question: “Has my usage gone up over time?” it
cannot answer the question, “is my usage higher than average, or particularly high?”
This social information can play a role both in making individuals aware of the
potential to reduce their energy use, as well as motivating reductions due to a social
normative effect.
Currently some existing services for utilities, are already offering services that
contrast energy usage with sets of similar households near a customer.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 28
Design & Engineering Services December 2012
To test the importance of social information three social variants were tested in the
experiment. The three variants were no social information, an overlay showing
neighbors that are 15% higher than the home, and an overlay showing neighbors
that are 15% lower than the home (see Appendix G for screens). A series of
questions in the survey then probed the subject’s interpretation of the presented
home energy use to determine if the usage of neighbors had an effect.
FIGURE 13. COMPARISON OF RESPONSE PATTERNS TO THE QUESTION ON COMPARATIVE ENERGY USAGE
(“THE HOUSEHOLD ‘S ENERGY USE SEEMS TO BE”….), ACROSS THREE CONDITIONS OF
SOCIAL INFORMATION AVAILABLE.
The first question tested the ability of subjects to correctly interpret the screen, and
shows a clear impact of the additional information for most subjects. Neighbors using
higher energy levels resulted in the median subjects indicating that this was the
case, whereas lower neighbors resulted in a shift in responses. No neighbors shown
resulted in many subjects indicating that they believe the usage was either similar or
they cannot tell. In general, participants correctly understood the comparative
information provided when they saw screens where the neighbor was using more
energy, most said they were using less than neighbor, and vice versa.
Once we establish that participants correctly understood the information, we asked a
second question to test whether they were interpreting neighbors’ energy use as a
norm to follow. The second question, then, probed the normative value of the
neighbor information by asking if the household was using more or less energy than
it “should be.” This distinction indicates that the peer effect can play a role in
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 29
Design & Engineering Services December 2012
motivating reductions: if my neighbor is using less, and I take that as a norm to
follow, I might use less too. Again, subjects responded as hypothesized to the
information, and notably responded most strongly when their house was shown as
using more than their neighbors.
FIGURE 14. COMPARISON OF RESPONSE PATTERNS TO THE QUESTION ON NORMATIVE ENERGY USAGE,
ACROSS THREE CONDITIONS OF SOCIAL INFORMATION AVAILABLE.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 30
Design & Engineering Services December 2012
FIGURE 15. COMPARISON OF RESPONSE PATTERNS TO THE QUESTION ON ABSOLUTE JUDGMENT ON
ENERGY USAGE, ACROSS THREE CONDITIONS OF SOCIAL INFORMATION AVAILABLE.
The final question in this set asked for an overall evaluation of energy use in the
presented household. The results indicate that when the IHED showed the house as
using more energy than the neighbors the subjects felt that the usage was high, but
the reverse was not true when the house was shown using less than the neighbors –
in this case the subjects felt that the usage was average (not low).
TAKING ACTION
One purpose of an IHED is to motivate resident action in saving energy, by either
taking conservation measures or purchasing or maintaining their equipment. To test
the efficacy of the IHED mockup designs in motivating this type of behavior a series
of hypothetical questions probed each subject’s belief that it would be appropriate to
take action to save energy due to the presented information, were it shown in their
own home.
In addition, subjects were split into two groups to receive either general energy
saving tips or specific energy related diagnostic information. The tips and diagnostic
groups are additionally compared to test the hypothesis that diagnostic information
is more relevant to residents than generic energy saving tips. In these questions,
subjects were allowed to choose as many energy saving options as they wanted, to
better reflect reality.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 31
Design & Engineering Services December 2012
FIGURE 16. T-TEST P <.001 FROM NO EXTRA INFO TO TIPS OR DIAGNOSTICS. NO SIGNIFICANT
DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS
FIGURE 17. T-TEST P <.001 FROM NO EXTRA INFO TO TIPS OR DIAGNOSTICS. NO SIGNIFICANT
DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 32
Design & Engineering Services December 2012
FIGURE 18. T-TEST P <.001 FROM NO EXTRA INFO TO TIPS OR DIAGNOSTICS. NO SIGNIFICANT
DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS.
FIGURE 19. T-TEST P <.001 FROM NO EXTRA INFO TO TIPS OR DIAGNOSTICS. NO SIGNIFICANT
DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 33
Design & Engineering Services December 2012
The results of the action questions show the importance of tips and diagnostic
information in converting the meaning of energy information (high vs. low) into
intent to actually change a household behavior or improve energy using equipment.
In A) subjects with diagnostics were the most certain that the house is using more
energy than it should. In B) those with diagnostics felt that they knew what actions
to take to save energy. In C) those with diagnostics were more likely to choose more
effortful and costlier conservation measures, and in D) those with diagnostics felt
that ideally (without some of the constraints of their real situation) they would be
even more aggressive with conservation measures.
The trend in each question indicates that tips were more motivating than no
information and diagnostics were more motivating still. The distributions were tested
using an unpaired Welch Sample t-test that tests whether the mean value of the
distributions is different. To test the groups unsure answers were removed, and
multiple-selection answers were added up to form a score for each individual. For
each question, each information type was tested against no additional information
and also against each other. The results were very consistent: the addition of either
Tips or Diagnostic information led to a statistically significant increase in the mean
score for the question, indicating that they have a real effect on knowledge and
motivation. The difference between Tips and Diagnostics was apparent in the mean
values but not statistically significant.
It is worth noting that there are currently no residential IHEDs that display diagnostic
information. This section of the study tested a hypothetical possibility that the
researchers thought might potentially encourage energy saving actions. It is quite
interesting to see that both kinds of diagnostic information can significantly increase
motivation to engage in energy savings.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 34
Design & Engineering Services December 2012
DISCUSSION To evaluate the effectiveness of the individual measures as well as the performance of the
IHEDs overall, two measures were taken. The first measure is from subject evaluations of
the information they were presented with in terms of whether they liked the information,
and if they found it useful. The questions and answers are presented below.
TABLE 2. COMPLETE WORDING OF QUESTIONS EVALUATING USERS’ LIKING AND PERCEIVED USEFULNESS OF IHED
FEATURES
QUESTION NAME WORDING
kWH_EVAL I liked that I could see the electricity usage expressed in kWh.
kWh_USEFUL I think that seeing the electricity usage expressed in kWh was useful.
kW_EVAL I liked that I could see the power being used expressed in kW.
kW_USEFUL I think that seeing the power being used, expressed in kW, was useful.
DOLLARS_EVAL I liked seeing the electricity consumption expressed in dollars and cents.
DOLLARS_USEFUL I think that seeing the electricity consumption being expressed in dollars and cents was useful.
BUDGET_EVAL If I had a display at home, I would like it to mark a budget goal that I can see.
BUDGET_USEFUL If I had a display at home, I think a budget marker would be useful.
COMPARISON_EVAL If I had a display at home, I would like it to show my neighbors' energy information.
COMPARISON_USEFUL If I had a display at home, I think seeing my neighbors' energy information would be useful.
FIGURE 20. RESPONDENTS’ LIKING AND USEFULNESS RATINGS OF A SERIES OF IHED MOCKUP
FEATURES.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 35
Design & Engineering Services December 2012
The experimental subjects on average both liked and found all of the information useful. In
summary, 80% had positive views of energy and power information. Over 90% of subjects
found financial (costs and budgeting) good (and more than half of those strongly agreed
with the statements), whereas only 60% found the social comparisons to be liked or useful
information types.
To test the projected usability of similar IHEDs, a combined usability score was generated
for each screen type. The usability score follows the basic concept that the most usable
IHED is easy to use as well as provides the required knowledge to transform the information
into action, termed “actionable knowledge”, and the motivation to act. In the case of our
study “actionable knowledge” corresponds to the question asking people whether they
would know what to do if they wanted to save energy, and “actionable norms” corresponds
to scores on the question asking whether they think, based on the IHED, that they should
be saving energy.
The score is then based on the formula:
Translated into the measures recorded in the survey this becomes:
The average usability scores are shown below organized by display.
FIGURE 21. USABILITY SCORES COMPARED ACROSS KIND OF ACTIONABLE INFORMATION PROVIDED
(NONE, GENERAL TIPS, SPECIFIC DIAGNOSTICS), AND KIND OF SOCIAL COMPARISON
PROVIDED (USAGE HIGHER THAN NEIGHBORS, USAGE LOWER THAN NEIGHBORS, NO
SOCIAL INFORMATION) .
The usability score shows two main trends- an increase in usability with additional tips and
diagnostics (as shown on the x-axis), and an increase in usability with additional social
information (shown in the side-by-side bars). The chart shows that the two information
)__(Re_
__ NormsActionableKnowledgeActionable
quiredEffort
GainednInformatioScoreUsability
)__(_
__ NormsActionableKnowledgeActionable
SpentTime
ScoreQuizScoreUsability
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 36
Design & Engineering Services December 2012
types also interact to improve usability – with the highest scores having both social
information and diagnostics. When the house was shown to have a higher energy use than
its neighbors it was always more usable than the opposite – most likely due to the
normative effect. The case with no neighbors was much more usable in the diagnostic
scenario. This is likely because the diagnostic information largely plays the same role as
social information in motivating improvements, but it was also easier to complete the quiz
when there was less social information on the screen. Finally, the highest usability score
(.34) is more than a 50% increase over the lowest score (.22), suggesting that IHED design
can have a large impact on resident actions.
This experiment provided valuable insight into the cognitive and sociological processes that
take place between the data underlying IHED design and a resident’s intention to take
action to reduce their energy use. The summary findings are:
80% of subjects enjoyed trying out the IHED, and would like to have one at home.
However, only 20% were seriously interested in buying one, and only 22% thought
that the device was worth at least $100.
70% of subjects can correctly answer questions about the instantaneous power draw
based on a numerical value, but only 30% can correctly answer questions about
monthly data using a chart. This indicates that both clarity of design and user
education will play an important role in the success of IHEDs.
Social (similar neighbor) information was successful in creating a performance norm
that subjects used to evaluate the house. However, in an evaluation of social
information, subjects had mixed responses to it, and 40% reported that they did not
like it or find it useful.
Direct evaluations of many common energy metrics showed that subject preferred
financial metrics (current cost and budgetary information) by a wide margin over
energy data or social information.
An ad-hoc usability measure indicated that some display styles are much more
effective than others are. In general, the added complexity of diagnostic information
and social information was worth the additional effort required to extract the
information. The usability measure also showed that the addition of diagnostic
information makes social comparisons less important to the overall usability.
Social information showing that the household was using less energy than the
neighbors were, consistently caused subjects to reduce their motivation to take
action, and led to a decrease in usability.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 37
Design & Engineering Services December 2012
CONCLUSIONS The market survey found the following problems with usability of the IHED:
Penetration of these devices is still very low (fewer than 1% of households possess
an IHED), in all likelihood because of the fluid state of the market itself, resulting in
confusing information for consumers.
There is no evidence that using the information in an IHED can help save energy,
and consumers may find it hard to know how to use an IHED to save energy, and
how to verify that the IHED feedback has resulted in savings for them.
Consumers may be confused about the scope of energy monitoring possible, since
promotion of the devices is vague regarding availability, installation compatibility and
features.
It may be hard for consumers to purchase or to understand what they are
purchasing.
Because of technical installation issues, and issues of compatibility with utilities’
meters, other wireless home systems, and price, it is likely that, in the long term,
the IHED market will drift towards mass installation on the part of building
developers or the utilities, rather than individual purchase on the part of consumers.
Respondents in the usability study were more likely to interpret the information
incorrectly when it was presented in graphic form. This should be a warning flag,
since many of the new generation IHEDs are graphics and color rich, and rely on
timeline graphs to present information. More research is needed to determine
whether these graphs are actually usable and useful, besides making the device
“look cool” or “fun”.
The controlled usability experiment sheds some light on features that are more likely to
result in appropriate use of these devices. That said, because of the high potential for
confusion and failure, the researchers stress the need for further research in these areas
before choices on devices are made, or mass deployments are considered.
The usability controlled experiment found that both simplicity of design and user
education play an important role in the success of IHEDs.
Social (similar neighbor) information was successful in allowing residents to
understand the meaning of their energy usage numbers. However, subjects had
mixed responses to it, and a significant fraction reported that they did not like it or
find it useful. Comparisons can also be a double edged sword: when the household
was using less energy than the neighbors, subjects’ motivation to take energy saving
actions was decreased.
The study also found that general tips are more motivating than no information and
specific diagnostics are more motivating still.
It may be worth it to invest time and resources in developing and testing energy
feedback technologies that provide specific energy saving diagnostics to residents.
Diagnostic algorithms exist, although they have never been applied to residential
systems.
The specific diagnostics that would need to be implemented probably includes
detection of when things are left on too long (including the air conditioner) and a loss
of efficiency of the air conditioner and furnace over time. This would require
additional infrastructure (communications with the thermostat, for example) that
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 38
Design & Engineering Services December 2012
currently does not exist. Although these algorithms do not exist, this study makes
the case that they should be developed.
Overall, the study indicates that IHED development, marketability and effectiveness are at
an immature state. The information provided by some systems on the market is interesting
and complex, but users may have a hard time understanding what the goal of the IHED is,
what its information means, and how they are supposed to use it. If the market is to
coalesce into a feedback system that is useful for energy savings, we may need to know
more about how to educate communities and consumers about IHED capabilities and goals,
and more about how to design a device that provides consumers with simple and direct
information that they can act upon.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 39
Design & Engineering Services December 2012
RECOMMENDATIONS
Gadget vs. Tool Framing: Make sure that IHEDs or other energy consumption
feedback devices are provided in the context of a planned energy savings action
program such as providing goals or other motivations to reduce energy use.
Conduct additional research on the impact of general energy saving tips and
diagnostic information and messages that are provided directly in the IHED while
people are checking their energy usage. We found that they can be more effective
than simply providing energy use, although there are currently no systems that
provide diagnostic messages
Engage manufacturers to develop and incorporate diagnostics algorithms into their
devices since this was found to be quite beneficial, but it is not possible with today’s
technology.
Expand research on the effects and desirability of social comparison information,
paying particular attention to the possibility of “boomerang” effects where a
comparison decreases motivation to save energy.
Expand research on the impact of complex, graphic and information rich IHED
devices, focusing in particular on users’ ability to extract correct and actionable
information from it.
Evaluate simple, clear messages as part of the IHED information. Results suggest
that these may be more useful and satisfying to users.
IHED Usability scores, like the one proposed here, should combine traditional
usability measure (ease of navigation, ability to gather correct information) with an
effectiveness usability measure, gauging whether the IHED provides information that
facilitates energy saving actions.
Conduct further research into deployability and usability of particular models prior to
including them in any incentive programs.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 40
Design & Engineering Services December 2012
APPENDIX A
RESEARCH TEAM
Kristin Heinemeier Ph.D., P.E.
Dr. Heinemeier has over 20 years of experience in building operation-phase efficiency issues. Previously, she worked with Portland Energy Conservation, Inc., a non-profit firm that is an established leader in the area of building commissioning, was the technical director of the Brooks Energy and Sustainability Laboratory at Texas A&M University, and developed products for Honeywell Home and Building Controls. Prior to
that, she was a researcher at Lawrence Berkeley National Laboratory. She was formerly the administrator of the California Commissioning Collaborative, and the
chair of ASHRAE’s technical committee on Building Commissioning. Kristin has a Ph.D. in Building Science, and is a Licensed Mechanical Engineer.
Tai Stillwater, Ph.D.
Dr. Stillwater is a Postdoctoral Scholar at the UC Davis Institute of Transportation Studies. He works in the Behavior and Sustainability Feedback Systems group that he helped develop as a PhD student in Transportation Technology and Policy. He has been an Emerging Venture Analyst at the Energy Efficiency Center and worked at the Plug-in Hybrid Demonstration Project, focusing on vehicle interfaces and energy use.
His dissertation examined the impact of novel vehicle interfaces on driving behavior
and fuel economy. Tai was the 2007-8 CH2M Hill Fellow in transportation and received the 2007-8 Achievement Rewards for College Scientists Scholarship. He holds an M.S. in Transportation and Technology Policy from UC Davis and a B.S. in Mechanical Engineering from UC Berkeley. At Berkeley, he worked on composite parts for the ATLAS particle detector at LBNL, and he participated in the Cal Human Powered
Vehicle Team, breaking multiple world speed records with a carbon fiber vehicle he helped design and manufacture called the "Bearacuda", that was accepted into the U.S. Bicycle Hall Of Fame. Tai has also worked at the California Energy Commission and with Chevron Energy Solutions, where he organized the 2008 West Village Contributor’s Forum.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 41
Design & Engineering Services December 2012
Claudia Barriga Ph.D.
Dr. Barriga is the WCEC's Behavioral Research Associate. In collaboration with Kristin Heinemeier, she is in charge of coordinating the center's new Behavioral Research Initiative. She is responsible for social science research that looks at people's
motivations, goals, and values as they impact their behavior and choices on energy consumption and use of HVAC systems. Claudia is currently leading a project to look at the behavior of homeowners and small-business owners on HVAC maintenance decision-making processes. She also collaborates in research that examines the usability of IHEDs and the impact of attitudes and goals on residential energy use and response to energy use feedback. Claudia holds a Ph.D. and an M.S. in Communication from Cornell University, and a B.S. in Psychology from Universidad
Diego Portales, Chile. While at Cornell University she collaborated in research on barriers that prevent people from practicing well-known food safe behaviors, and she conducted research on the mental processing of moral, gender and science information in narrative form.
Marco Pritoni M.I.E.
Marco Pritoni is a Ph.D. student in Mechanical and Aeronautical Engineering at UC
Davis. Prior to joining the program, Marco received his Masters’ Degree in Industrial Engineering from the University of Bologna, Italy and after working in industry for 6 years joined the LBNL as a senior research associate for a year. He is currently a graduate student researcher for UC Davis Western Cooling Efficiency Center and an Emerging Ventures Analyst for the UC Davis Energy Efficiency Center. He is a Southern California Edison Fellow and Johnson Controls Fellow.
Dina Biscotti Ph.D.
Dr. Biscotti is a Postdoctoral Scholar at the University of California, Davis Energy
Efficiency Center. She completed her Ph.D. in Sociology at UC Davis in 2010 and her research specialization is in economic and organizational sociology. In collaboration with EEC Director Nicole Biggart, Dina is leading a research project to study the diffusion of energy efficiency technologies through social institutions like churches, schools, and community-based organizations. With the UC Davis California Lighting Technology Center and the UC Davis Western Cooling Efficiency Center, she is working
on several interrelated research projects to identify structural barriers to the adoption of energy efficiency technologies and practices, along with strategies for overcoming those barriers.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 42
Design & Engineering Services December 2012
APPENDIX B Appendix B, the full list of market available IHEDs, is attached separately as a pdf
document.
Appendix B.pdf
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 43
Design & Engineering Services December 2012
APPENDIX C Appendix C, the characterization spreadsheet of market IHEDs, is attached separately as an
excel document.
Appendix C.xlsx
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 44
Design & Engineering Services December 2012
APPENDIX D
USABILITY CONTROLLED EXPERIMENT
QUESTIONNAIRE ITEMS
Variable names are presented in brackets
1. Previous Familiarity with Energy Consumption
[KNOW_MONEY] I spend approximately $ _______ per month on energy (average).
[KNOW_kWh] I use approximately _____kWh monthly. 2. IHED mockup display evaluation. Respondents state their agreement with a series of statements
using a 6 point scale, where
1 = Strongly Disagree
2 = Moderately Disagree
3 = Slightly Disagree
4 = Slightly Agree
5 = Moderately Agree
6 = Strongly Agree
[EASY] The energy display seemed easy to use and understand.
[STRAIGHTFWD]The energy display was straightforward to use.
[LIKETOHAVE]I would like to have an energy display like this at home.
[PLANTOBUY]I plan to buy such an energy display eventually.
[LIKE]I like this energy display.
[FUN]The energy display was fun to use.
[ENTERTAIN]The energy display was entertaining.
[COOLTOHAVE]It would be cool to have an energy display like this one at home.
[INTERESTINGHOME]The energy display’s information would be interesting to have at home.
[SAVEENERGY] I think I could save energy if I had this energy display.
[FRUSTTECH] I find new technologies like this display frustrating.
[LOOKGOOD] The energy display looked nice
[TOOSIMPLE] The energy display was too simple
[USEFUL] I think this energy display could be very useful.
3. [WORTH] To me, this device is worth ___________ 4. Respondents were then instructed that they could consult the display to respond to the next set
of questions. The following questions were presented, and respondents were allowed to go back and look at the display in order to choose their responses. These items were multiple choice.
[POWER] How much power is the household using right now? o 1.1kW 16kWh 26kWh 16 cents $80 DK
[ENERGY] Approximately, how much energy has been used so far this month, in kWh?
o 4 26 500 800 1.1 DK
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 45
Design & Engineering Services December 2012
[HOUR_COST] How much has this household spent in the last hour? o 1 dollar 16 cents 26 dollars 3 cents 11 dollars DK
[PROJ_COST] If the household continues using energy at the current rate, how much will they spend this month?
o 16 25 80 500 116 DK
[ENERGY_Overall_Meaning] Overall it appears that this household’s energy use is: o Extremely Low o Low o Average o High o Extremely High o I cannot tell
[ENERGY_Compared_Meaning] The household’s energy use seems to be: o Much lower than most homes o Lower than most homes o Similar to most homes o Higher than most homes o Much higher than most homes o I cannot tell
[ENERGY_Normative_Meaning] The household appears to be using: o Much less energy than it should o Less energy than it should o As energy as it should o More energy than it should o Much more energy than it should o I don’t know
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 46
Design & Engineering Services December 2012
[ACTIONABLE_Norm] If you saw this information for your household, you would think:
o You need to use much less energy o You need to use slightly less energy o You are using an appropriate amount of energy o You could afford to use a little more energy o You could afford to use a lot more energy o Would not know how to decide
[ACTIONABLE_Possibilities] Which of the following actions do you think would be more useful in getting this household’s usage to an appropriate level?
o A small easy action, like turning off electronics when not using them o A moderate, no cost action, like cleaning your refrigerator or AC coils. o A moderate, low cost action, like changing incandescent light bulbs for
efficient ones o A moderate, costly action, like hiring a maintenance contractor to optimize
your air conditioner o A big, costly action, like replacing your appliances with new Energy Star
efficient ones o No change is needed
[ACTIONABLE_Knowledge] If you saw this energy display, and you WANTED to SAVE ENERGY, you would know exactly what to do to save energy
o Strongly Disagree o Moderately Disagree o Slightly Disagree o Slightly Agree o Moderately Agree o Strongly Agree
5. After participants were introduced to the second phase display, that included either general tips or specific diagnostics, participants were asked the following questions.
[ACTIONABLE_ NORM_ AFTERTIPDIAG] The new information in the display makes me think that the household is using:
o Much less energy than it should o Less energy than it should o As energy as it should o More energy than it should o Much more energy than it should o I don’t know
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 47
Design & Engineering Services December 2012
[ACTIONABLE_KNOWLEDGE_AFTERTIPDIAG] If you saw the device with this new information and you WANTED to SAVE energy, you would know exactly what to do to save energy
o Strongly Disagree o Moderately Disagree o Slightly Disagree o Slightly Agree o Moderately Agree o Strongly Agree
[ACTIONABLE_OPTIONS_REALISTIC] Please think of what you realistically might do: which of these options would you choose to implement if you saw the same information on your IHED at home?
o No action o Raise my thermostat 2-4 degrees o Raise my thermostat 5-7 degrees o Clean my AC o Hire someone to maintain my AC o Replace my AC
[ACTIONABLE_OPTIONS_IDEAL] Now please think of what your “ideal/best” self would do: which of the options provided would you choose to implement if you saw the same information on your IHED at home?
o No action o Raise my thermostat 2-4 degrees o Raise my thermostat 5-7 degrees o Clean my AC o Hire someone to maintain my AC o Replace my AC
6. Participants were then asked a set of questions about the usefulness and desirability of features in the IHED mockup. (Respondents used the 1-6 agreement scale.)
[kWh_EVAL] I liked that I could see the electricity usage expressed in kWh.
[kWh_USEFUL] I think that seeing the electricity usage expressed in kWh was useful.
[kW_EVAL] I liked that I could see the power being used expressed in kW
[kW_USEFUL] I think that seeing the power being used, expressed in kW, was useful
[DOLLARS_EVAL] I liked seeing the electricity consumption expressed in dollars and cents
[DOLLARS_USEFUL] I think that seeing the electricity consumption expressed in dollars and cents was useful.
[BUDGET_EVAL] If I had a display at home, I would like it to mark a budget goal that I could see.
[BUDGET_USEFUL] If I had a display at home, I think a budget marker would be useful.
[COMPARISON_EVAL] If I had a display at home, I would like it to show my neighbors’ energy information.
[COMPARISON_USEFUL] If I had a display at home, I think seeing my neighbors’ energy information would be useful.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 48
Design & Engineering Services December 2012
7. Finally, participants were asked a set of demographic questions
[AGE] I am ________ years old.
[SEX] You are (male) (female)
[EDUCATION] What is your highest level of education? o Elementary School o High School o Some College o Completed College o Graduate Degree
[HOMEOWN] Do you (or your family) own the place you live in? o Yes o No
[BILLPAYER] Are you in charge of paying energy bills at your home? o Yes o No
[MOTIVATION_SAVE] Overall, how interested are you in using less energy at home? o Not interested at all o A little interested o Moderately interested o Very interested o Extremely interested
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 49
Design & Engineering Services December 2012
APPENDIX E Appendix E, the full set of experimental condition screens, is provided below.
CONDITION 1. BASIC INFORMATION ONLY X SPECIFIC DIAGNOSTIC INFORMATION
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 50
Design & Engineering Services December 2012
CONDITION 2. BASIC INFORMATION ONLY X GENERAL TIPS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 51
Design & Engineering Services December 2012
CONDITION 3. BASIC + BUDGET X SPECIFIC DIAGNOSTIC INFORMATION
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 52
Design & Engineering Services December 2012
CONDITION 4. BASIC + BUDGET X GENERAL TIPS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 53
Design & Engineering Services December 2012
CONDITION 5. BASIC + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X SPECIFIC DIAGNOSTIC
INFORMATION
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 54
Design & Engineering Services December 2012
CONDITION 6. BASIC + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X GENERAL TIPS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 55
Design & Engineering Services December 2012
CONDITION 7. BASIC + BUDGET + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X SPECIFIC
DIAGNOSTIC INFORMATION
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 56
Design & Engineering Services December 2012
CONDITION 8. BASIC + BUDGET + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X GENERAL TIPS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 57
Design & Engineering Services December 2012
CONDITION 9. BASIC + SOCIAL INFORMATION LOWER THAN NEIGHBOR X SPECIFIC DIAGNOSTIC
INFORMATION
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 58
Design & Engineering Services December 2012
CONDITION 10. BASIC + SOCIAL INFORMATION LOWER THAN NEIGHBOR X GENERAL TIPS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 59
Design & Engineering Services December 2012
CONDITION 11. BASIC + BUDGET + SOCIAL INFORMATION LOWER THAN NEIGHBOR X SPECIFIC
DIAGNOSTIC INFORMATION
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 60
Design & Engineering Services December 2012
CONDITION 12. BASIC + BUDGET + SOCIAL INFORMATION LOWER THAN NEIGHBOR X GENERAL TIPS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 61
Design & Engineering Services December 2012
CONDITION 13. BASIC X NO ENERGY SAVING SUGGESTIONS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 62
Design & Engineering Services December 2012
CONDITION 14. BASIC + BUDGET X NO ENERGY SAVING SUGGESTIONS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 63
Design & Engineering Services December 2012
CONDITION 15. BASIC + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X NO ENERGY SAVING
SUGGESTIONS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 64
Design & Engineering Services December 2012
CONDITION 16. BASIC + BUDGET + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X NO ENERGY
SAVING SUGGESTIONS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 65
Design & Engineering Services December 2012
CONDITION 17. BASIC + SOCIAL INFORMATION LOWER THAN NEIGHBOR X NO ENERGY SAVING
SUGGESTIONS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 66
Design & Engineering Services December 2012
CONDITION 18. BASIC + BUDGET + SOCIAL INFORMATION LOWER THAN NEIGHBOR X NO ENERGY
SAVING SUGGESTIONS
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 67
Design & Engineering Services December 2012
APPENDIX F UNIVERSITY OF CALIFORNIA, DAVIS
ONLINE CONSENT TO PARTICIPATE IN A RESEARCH STUDY
STUDY TITLE: Energy Feedback Device Evaluation
This is a research study conducted by Kristin Heinemeier from the Western Cooling
Efficiency Center at UC Davis, and funded by Southern California Edison. You have the right
to know about the procedures, risks, and benefits of the research study.
Participating in research is your choice and voluntary. You have the right to know about the
procedures, risks, and benefits of the research study. If you decide to take part, you can
change your mind later and leave the study. No matter what decision you make, there will
be no penalty to you.
If you decide to take part in this study, you can decide to stop at any time.
ABOUT THIS RESEARCH STUDY
We hope to learn more about the way in which energy feedback devices are used. About
200 people will take part in this study.
If you decide to participate in this study, you will be asked to familiarize yourself with a
simulated energy feedback device (presented in a tablet) and answer questions about the
information provided in the device. The whole process is expected to take no more than 10
minutes.
Discomfort and Risks
There is minimal risk to participating in this study.
We will not ask for your name or any other identifiable information, so your privacy and the
confidentiality of your responses are protected. For more information about risks, ask the
Researcher.
Benefits
You will not directly benefit from taking part in this research. The information we get from
this study will help us understand how people use energy feedback and determine the best
way to design these devices in the future, so that they can effectively help residents save
energy.
costs
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 68
Design & Engineering Services December 2012
There is no cost to you beyond the time and effort required to complete the procedure(s)
described above.
COMPENSATION
To show our appreciation for your help, you will receive a five dollar gift card once you
complete the study.
WHO CAN ANSWER MY QUESTIONS ABOUT THE STUDY?
If you have questions, please ask us. You can talk to the Researcher about any questions or
concerns you have about this study at:
Claudia Barriga at phone number: 530-758-1882
Tai Stillwater at email: [email protected]
For questions about your rights while taking part in this study call the Institutional Review
Board at (916) 703-9167 or write to IRB Administration, CTSC Building, Suite 1400, Room
1429, 2921 Stockton Blvd., Sacramento, CA 95817. Information to help you understand
research is on-line at www.research.ucdavis.edu/IRBAdmin.
By clicking on “Accept” below, you indicate that you have read the consent document, and
agree to participate in the study,
Accept
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 69
Design & Engineering Services December 2012
GLOSSARY
IP Internet Protocol
OpenADR A protocol that standardizes, automates and simplifies demand response to
enable utilities to cost-effectively meet growing energy demand, and
customers to control their energy use.
SEP Protocol that offers IP-based control for advanced metering infrastructure and
home area networks
WiFi A common protocol for wireless data transfer
ZigBee A specification for a suite of high level communication protocols using small, low-power
digital radios based on an IEEE 802 standard for personal area networks.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 70
Design & Engineering Services December 2012
REFERENCES
1 Ehrhardt-Martinez, K., Donnelly, K.A., Laitner, J.A. (2010) Advanced Metering Initiatives
and Residential Feedback Programs: A Meta-Review for Household Electricity-Saving
Opportunities. American Council for an Energy-Efficient Economy Washington, DC.
2 Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007). The
Constructive, Destructive, and Reconstructive Power of Social Norms. Association for
Psychological Science, 18(5), 429-434.
3 Meier, A., Aragon, C., Pfeffer, T., Perry, D., & Pritoni, M. (2011). Usability of residential
thermostats: Preliminary investigations. Building and Environment, 46(10), 1891-1898.
4 Haakana, M., Sillanpaeae, L., & Talsi, M. (1997). The effect of feedback and focused advice
on household energy consumption. European Council for an Energy Efficient Economy
ECEEEE Summer Study
5 Lewis, S. C. R. (2011). Energy in the Smart Home. In R. Harper (Ed.), The Connected
Home: The Future of Domestic Life. London: Springer-Verlag.
6 Note that although we focused exclusively on electricity feedback, utilities and third-party
suppliers/integrators are also interested in learning about the possibilities for using a
common system for different applications or to monitor different types of energy, furthering
the concept of integrated demand side management and the Smart Grid. Customers would
not want to pay for separate systems, e.g., electrical energy monitoring, gas monitoring,
home security, entertainment, automation, etc. if the systems can be
integrated/consolidated. This emerging technology is not yet prevalent in the market, and
thus it was beyond the scope of our study.
7 Distinction between Small and Large commercial is as defined by vendor.
8 Peffer, T., Perry, D., Pritoni, M., Aragon, C. & Meier, A. (2012). Facilitating energy savings with programmable thermostats: evaluation and guidelines for the thermostat user interface. Ergonomics, (1-17). DOI:10.1080/00140139.2012.718370
9 Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.
10 Jacobson, R. P., Cialdini, R. B., & Mortensen, C. R. (2010). Bodies Obliged and Unbound:
Differentiated Response Tendencies for Injunctive and Descriptive Social Norms. Journal
of Personality and Social Psychology, 100(3), 433-448.
11 Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007). The
Constructive, Destructive, and Reconstructive Power of Social Norms. Association for
Psychological Science, 18(5), 429-434.
Usability of In-Home Energy Displays HT.11.SCE.013
Southern California Edison Page 71
Design & Engineering Services December 2012
12 Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2007). The effect of tailored
information, goal setting and tailored feedback on household energy use, energy-related
behaviors, and behavioral antecedents. Journal of Environmental Psychology, 27(4), 265-
276.
13 McCalley, L. T., deVries, P. W., & Midden, C. J. H. (2010). Consumer response to product
integrated energy feedback: behavior, goal level shifts, and energy conservation.
Environment and Behavior, 43(4), 525-545.
14 Wilson, C., & Dowlatabadi, H. (2007). Models of individual decision making and behavior:
Lessons for designing interventions to reduce residential energy use. Environment, 32, 1-
59.
Top Related