Findings from the Alpha Test of the …...Opower/Honeywell solution. The results from this...
Transcript of Findings from the Alpha Test of the …...Opower/Honeywell solution. The results from this...
PG&E’s Emerging Technologies Program
Findings from the Alpha Test of the Opower/Honeywell Thermostat Pilot ET Project Number: ET11PGE1072
Project Manager: Sarah Bresko Pacific Gas and Electric Company Prepared By: Michael Perry Freeman, Sullivan & Co. 101 Montgomery set, 15th floor San Francisco, CA 94104
Issued: April 13, 2012
Copyright, 2012, Pacific Gas and Electric Company. All rights reserved.
PG&E’s Emerging Technologies Program
ACKNOWLEDGEMENTS
Pacific Gas and Electric Company’s Emerging Technologies Program is responsible for this project. It was developed under internal project number ET11PGE1072. Michael Perry conducted this technology evaluation for Pacific Gas and Electric Company with overall guidance and management from Sarah Bresko. For more information on this project, contact [email protected].
LEGAL NOTICE
This report was prepared for Pacific Gas and Electric Company for use by its employees and agents. Neither Pacific Gas and Electric Company nor any of its employees and agents:
(1) makes any written or oral warranty, expressed or implied, including, but not limited to those concerning merchantability or fitness for a particular purpose;
(2) assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, process, method, or policy contained herein; or
(3) represents that its use would not infringe any privately owned rights, including, but not limited to, patents, trademarks, or copyrights.
PG&E’s Emerging Technologies Program
ABBREVIATIONS AND ACRONYMS
HVAC Heating, Ventilation and Air Conditioning
PC Personal Computer
App Software application—either on an iPhone or a PC
PG&E’s Emerging Technologies Program
TABLES
Table 1: Frequency of Responses to Overall Impression Questions
and Average Response ................................................ 12
Table 2: Degree of Agreement Questions About Supporting Materials ................................................................... 14
Table 3: Number of Times Respondents Have Used Each App .......... 15
Table 4: Tallies of Responses to Degree of Agreement Questions ..... 18
Table 5: Ranking of Reasons for Saving Energy .............................. 19
Table 6: Frequency of Responses to Overall Impression Questions
and Average Response ................................................ 19
Table 7: Number of Times Respondents Have Used Each App .......... 20
PG&E’s Emerging Technologies Program
CONTENTS
EXECUTIVE SUMMARY 6
INTRODUCTION 8
BACKGROUND 8
Emerging Technology/Product 9
ASSESSMENT OBJECTIVES 10
TECHNOLOGY/PRODUCT EVALUATION 10
TECHNICAL APPROACH/TEST METHODOLOGY 10
Field Testing of Technology 11
Test Plan 11
Instrumentation Plan 11
RESULTS 11
First Survey of Participants 12
Overall Impression Responses and Comments 12
Web Portal Enrollment 13
Call Center 13
Installation and the Installer 13
iPhone App 13
Internet App 13
Supporting Materials Ratings 14
Frequency of use of Apps 15
Final open-ended Comments 15
Second Survey of Participants 16
High priority Issues 16
Overall Impression Responses and Comments 19
Frequency of use of Apps 20
Focus Group Results 20
First Question 20
Second Question 22
Third Question 22
Fourth Question 23
Fifth Question 23
Sixth Question 24
Seventh Question 24
Eighth Question 25
Ninth Question 25
Tenth Question 25
Leader Board 25
Data Analysis 25
EVALUATIONS 26
PG&E’s Emerging Technologies Program
RECOMMENDATIONS 26
APPENDIX A SURVEY INSTRUMENTS 27
First Survey 27
Second Survey 29
PG&E’s Emerging Technologies Program
EXECUTIVE SUMMARY
This emerging technology report documents the results from the Alpha test of the Honeywell communicating thermostat pilot and Opower behavioral software. In this wave,
17 employees of PG&E had the thermostat installed at their home. The thermostat includes programming that allows it to be remotely programmed through an iPhone app or a PC-based app. It also has the functionality to push messages to participants through the
iPhone app, alerting them to important aspects of their energy use.
Honeywell technicians installed the devices and made sure that participants’ iPhones were set up with the app. Participants received instructions on how to use the device and the apps.
Participants were surveyed twice over the internet. The first survey took place after the device had been in place for about two weeks and the second survey took place when the device had been in place for about five weeks. Participants also took part in a focus group
one week after the second survey in which they were asked for their opinions about the device along many dimensions.
The Alpha test is a first step in a broader effort to test the Opower/Honeywell solution in PG&E’s service territory.
PROJECT GOAL
The goal of the Alpha test is to test functional readiness of the thermostat and behavioral
software solution, to assess customer satisfaction with the installation process and with the thermostat and to assess customer engagement with the thermostat.
Goals of the broader project include:
• Driving a higher percentage of customers to program their thermostats with a custom schedule;
• Motivating customers to set a more efficient temperature when home;
• Increasing set-back temperatures (i.e., automating systems when customers are away from home); and
• Expanding the length of the set-back period.
PROJECT DESCRIPTION
The evaluation of the Alpha test consisted of four main steps, three of which are
documented here. The first step consisted of recruiting and installing Honeywell thermostat devices at the homes of 17 volunteers. Each volunteer is a PG&E employee. The recruitment and installation processes are documented here only insofar as participants provided information about them through surveys and focus groups.
The other three steps in the project consisted of two surveys about the customer experience and a focus group, also about the customer experience.
The Alpha test is not focused on energy savings, load balance or cost savings. It is focused
on fixing problems prior to the larger Beta test of the project. The Beta test may provide
PG&E’s Emerging Technologies Program
energy savings by allowing customers to better control their energy use associated with climate control.
PROJECT FINDINGS/RESULTS
The main findings of the first survey are that:
• Respondents were impressed with the installation process and the installers, although some thought the installers should be more knowledgeable about the internet app and iPhone app;
• Respondents would like to have that ability to program the thermostat using the
device itself;1
• Several respondents had difficulties getting the iPhone app up and running at the beginning;2
• Call center reps came off as inexperienced; and
• Several respondents thought the way the thermostat responded to programming and the messages it sent could be improved.
The main findings of the second survey are that:
• 13 out of 15 respondents thought that they had reduced energy usage;
• 14 out of 15 respondents thought that the pilot should be rolled out to 750 customers;
• Customers were generally glad to have been part of the pilot;
• My Thermostat was the most popular name;
• The internet app is less well-liked and well-understood than the iPhone app; and
• The iPhone app is both well-liked and the most popular reason given for why the
respondent was able to save energy. Respondents also generally say that it is easy to use.
Focus group findings were varied. Overall ratings of the device by participants ranged from a 2 on a scale of 10 to an 8 on a scale of 10. Participants ranged from quite positive to
mildly negative in their view of how exciting, engaging and high-tech the device was. Most participants failed to notice messages being pushed to them through the iPhone app.3
PROJECT RECOMMENDATIONS
Recommendations from the Alpha test are:
1 This was mentioned by respondents to the first survey and in the focus group. 2 The process for loading the iPhone app was a temporary, testing solution only. For the Beta, the smart phone application will be downloaded in
the typical way all apps are downloaded.
3 Participants were specifically asked if they had noticed the messages being sent to them and most had not. This question was a lead-in to a
question about what types of messaging would be most useful.
PG&E’s Emerging Technologies Program
• That Opower or Honeywell follow up with respondents who indicated either that the thermostat isn’t working or that one of the apps isn’t working. The most important
steps are to identify the problem and communicate back what the problem is, how it can be fixed and whether that problem is likely to occur often in the broader pilot;
• That Honeywell add a step to the installation process in which the installer fully verifies that the iPhone app controls the thermostat before the installation is
considered complete;
• That PG&E work with Opower, Honeywell and FSC to revise the supporting materials that installers leave behind based on the suggestions below;
• That the solution be modified to work with Android smart phones as well (a step that
is being taken for the Beta test).
INTRODUCTION
This report describes the procedures and results obtained in the Alpha test of the
Opower/Honeywell solution. The results from this evaluation will support PG&E, Honeywell, and Opower in the launch of Beta test of the same thermostat by providing useful user feedback, and evaluation of the thermostat’s functionality and communication quality.
The Alpha test assessment was designed to allow for testing of the thermostat in a friendly
environment in which PG&E could simulate the customer experience and receive important feedback about device operation, while not risking customer relations problems that might arise due to technical problems with the technology being tested.
BACKGROUND
Honeywell and Opower, Inc. have collaborated to develop a solution that includes an internet-enabled thermostat and behavioral software that allows residential customers to better manage their climate-control energy use.
The Alpha test of the project, documented in this report, consists of a pilot program in which 17 PG&E employees were recruited and had the thermostat installed at their home. The feedback from these employees will be used to modify the recruitment and installation processes and to alert Honeywell and Opower to technical problems that must be addressed
prior to the Beta test, which will be comprised of PG&E customers rather than employees.
The two main goals for the Beta test will be to develop understanding of:
• Energy savings potential from reducing residential heating, ventilation and air conditioning (HVAC) consumption through behavioral changes; and
• Residential consumer preferences and attitudes towards enabling technologies.
PG&E’s Emerging Technologies Program
HVAC consumption is targeted for this project because it represents approximately 9% of electric consumption and 37% of gas consumption for residential customers in California.4
However, the percentage of total electric and gas consumption can vary drastically depending on the climate zone. HVAC consumption represents 30% of total electric consumption in several of PG&E’s climate zones, but can range as high as 45% for 2-story homes in specific climate zones.5
While many customers have programmable thermostats, research shows that the majority of customers do not know how to properly set their thermostats.6 This suggests that there
may be substantial savings available by making it easier for customers to set their thermostats and by providing reminders when it might be advantageous for them to change their settings.
The thermostat and software are meant to address this possibility by providing an intuitive programming interface and messaging, each delivered through an app on the customer’s iPhone or PC.
EMERGING TECHNOLOGY/PRODUCT
The solution is a combination of a Honeywell internet enabled thermostat with Opower
behavioral solution software, which includes normative messaging and recommended schedules through an iPhone Application (referred to as an app). The thermostat can also be programmed through an app on a participants PC. Through this messaging, the thermostat keeps customers better informed about their energy choices and the cost
ramifications of those choices. The solution design is intended to make it easy for customers to act and, by using psychological mechanisms and consistent encouragement and feedback, to motivate customers to reduce energy consumption.
The product in the home works as a standard programmable thermostat. The innovation is the simplicity of its programming process and its ability to interact with Opower software. The software nudges customers towards using less HVAC energy by making suggestions. The suggestions alert customers to times when the customer could save energy by allowing
the home to heat up more in the summer or cool down more in the winter.
The thermostat would replace either a conventional non-programmable thermostat, programmable thermostat or a thermostat.
The primary market barrier currently is that the customer must have an iPhone to use with
the thermostat, if they want to take full advantage of the mobile control. Opower is currently developing the software that will allow the thermostat to work with other types of smartphone. Though a smartphone will always be required for mobile control, a web
interface is also available for remote access and control.
4 California Energy Commission. “California Statewide Residential Appliance Saturation Study Final Report Executive Summary.” 2010 5 eQuest – Building Energy Use and Cost Analysis Software, developed by James J. Hirsch & Associates (JJH), version 3.64 was the latest release,
http://www.doe2.com/ 6 California Energy Commission. “California Statewide Residential Appliance Saturation Study Final Report Executive Summary.” 2010
PG&E’s Emerging Technologies Program
ASSESSMENT OBJECTIVES
The data from the Alpha test of the project consists of responses to two online surveys and a focus group, each of which included subsets of the 17 PG&E employees who had the
thermostat installed at their home. Each survey and the focus group included a different subset of participants.
The data is evaluated through summary tables of survey responses as well as qualitative
judgments of comments made to open-ended survey responses and focus group questions. The primary objectives of the data evaluation are to test functional readiness, customer satisfaction and customer engagement of the thermostat.
Energy savings will not be determined in the Alpha test of the project, but will be
determined as part of the broader project.
TECHNOLOGY/PRODUCT EVALUATION
The evaluation of the thermostat in the Alpha test was done by PG&E employees in their homes. This was an ideal test bed as it reflects what PG&E customers will experience if they
choose to take part in a broader pilot of this technology.
The criteria that PG&E employees had to satisfy in order to take part were that they had to have central heating and air-conditioning and an iPhone, and live in a single-family home
with only one zone for climate control.
The assessment of the Alpha test has been performed by a team including staff from both PG&E and FSC. FSC is a contractor providing advice and assistance with many aspects of the Alpha and Beta tests, including sampling, recruiting and installation. FSC will be the
evaluator of the Beta test, and therefore will also produce estimates of energy savings impacts from the technology. For the Alpha test assessment, FSC has extensive experience in survey research and focus groups evaluating customer-facing programs in the utility
industry.
TECHNICAL APPROACH/TEST METHODOLOGY
The technical approach to the test is that of a field test in which the primary data of interest are all aspects of the participant experience. These include:
• How satisfied participants were with the recruitment and installation process;
• How much they liked the thermostat device and how useful they found it;
• How much they liked the iPhone and PC apps and how useful participants found the apps;
• Whether participants liked the messages that were sent to them through their iPhones;
• Whether participants thought that the technology would lead them to save
energy; and
PG&E’s Emerging Technologies Program
• Whether participants thought that the technology should be introduced to PG&E customers in a Beta test.
This data has been evaluated mainly qualitatively to assess the technology and its installation process. Due to the small sample size and the qualitative nature of the data collected, there is no formal quantitative analysis of results from the Alpha test. Quantitative analysis will be performed as part of the Beta test evaluation.
FIELD TESTING OF TECHNOLOGY
This is a test of a thermostat technology, its associated iPhone and PC apps and the installation process for all three.
The sites for testing were homes of PG&E employees that satisfy similar eligibility criteria to the customers who will take part in the Beta test. Therefore this test bed
provides unbiased results.
Test sites may have a variety of different incumbent technologies, ranging from traditional thermostats to other types of programmable thermostat.
The system is controlled by the participant.
TEST PLAN
There is no baseline measurement for comparison. The implicit technology baseline is each participant’s previous thermostat and the test will rely on participants to make that comparison.
The energy consumption associated with the HVAC being controlled by the test technology will not be measured in the Alpha test.
The variables used to measure performance are responses to survey and focus group questions about the technology and its associated installation process. These
questions are detailed in the results section.
The monitoring period for the Alpha test is 6 weeks.
Data are gathered via online survey and through voice recordings and notes taken during focus group sessions.
INSTRUMENTATION PLAN
This section is not applicable because the only data collected are survey and focus group data.
RESULTS
This section is divided into three subsections describing results from the first survey, second
survey and focus groups, respectively. Copies of the survey instruments are provided in Appendix A.
PG&E’s Emerging Technologies Program
FIRST SURVEY OF PARTICIPANTS
Survey invitations were sent out to all 17 Alpha test participants on December 23, 2011. Reminder emails were sent to non-respondents on December 27. A total of 15 employees responded by January 3, 2012. Those responses are summarized here.
The main findings of the first survey are that:
• Respondents were impressed with the installation process and the installers, although some thought the installers should be more knowledgeable about the
internet app and iPhone app;
• Respondents would like to have that ability to program the thermostat using the device itself; Several respondents had technical problems with the iPhone app;7
• Call center reps will require additional training to understand nuances of the
program; and
• Several respondents thought the way the thermostat responded to programming and the messages it sent could be improved.
OVERALL IMPRESSION RESPONSES AND COMMENTS
The survey contained several questions about participants’ overall impressions of aspects of the installation and features of the device and supporting software. Table 1 summarizes the responses to those questions. It shows the tally of the number of times respondents gave each rating on a scale from 1 to 10, with 1 meaning “very negative” and 10 meaning “very
positive.” There were no responses below 3 for any aspect of the installation or features.
TABLE 1: FREQUENCY OF RESPONSES TO OVERALL IMPRESSION QUESTIONS AND AVERAGE RESPONSE
Rating (1-10)
Feature 3 4 5 6 7 8 9 10 Average Rating
Web portal enrollment
| || ||| |||| | ||| 8.5
Call center | |||| ||| || ||||| 8.4
Installation of
your thermostat
| | |||| |||| | ||| 8.6
Installer of
your thermostat
| |||| |||| |||| 8.8
Thermostat || | | ||| ||| |||| 8.0
iPhone app | ||| ||| ||| |||| 7.9
Internet app
on your PC | ||| | || 8.6
7 The process for loading the iPhone app was a temporary, testing solution only. For the Beta test, the smart phone application will be
downloaded in the typical way all apps are downloaded.
PG&E’s Emerging Technologies Program
Overall, responses were quite positive. The two lowest average scores were for the thermostat and the iPhone app. Each of those had more than one rating of 5 or below.
None of the other aspects or features received any rating below 5.
For all but one of the overall impression questions, respondents were asked to provide comments on why they gave the score they did. Only respondents giving scores of seven or lower responded to these questions (this was not a restriction, it just happened that way).
There were many useful comments which are discussed below.
WEB PORTAL ENROLLMENT
There were three comments about web portal enrollment:
• One respondent said that her husband signed up for the program using an email
address different from the one associated with their iPhone. This caused a problem at installation, and the respondent felt that it wasn’t clear that it needed to be the same address;
• Another respondent went through the enrollment process, and then when the installer showed up the app had disappeared from his/her phone, forcing the respondent to go through the process again; and
• A third respondent had trouble getting the Opower site to operate on his/her
iPhone. The respondent loaded it two or three times to “get it to be resident” and then it stopped working.
CALL CENTER
All three comments about the call center indicated that the representatives needed
additional training about the details of the program
INSTALLATION AND THE INSTALLER
Although the survey instrument differentiated between installers and the installation process, respondents did not. Respondents focused their comments on the installers.
Respondents were quite pleased with installers’ punctuality and respectfulness, and with the speed of installation. However, three respondents were disappointed that the installer was unable to help them install or use the iPhone app and could only refer the respondents to email instructions they had received. One of these respondents was specifically
disappointed because they said they had been told that “you will know everything about this program when the installer leaves.” Another of these respondents indicated that their device was still non-functional for controlling their heat and that that was costing them a lot
of money.
IPHONE APP
Two respondents said that they downloaded the iPhone app and it did not work. Another found the app messages to be confusing.
INTERNET APP
Only one respondent commented on the internet app as part of this section. The comment was that the app does not provide insight into the ramifications of the settings selected.
PG&E’s Emerging Technologies Program
SUPPORTING MATERIALS RATINGS
The installers left behind a set of supporting materials to help participants use and understand the thermostat and apps. In the survey, respondents were asked to give
numerical answers indicating on a scale from 1 to 10 how strongly they agreed with a set of statements about the materials. A response of 1 indicates “strongly disagree” and a response of 10 indicates “strongly agree.” The tallies of these responses are shown in Table 2.
TABLE 2: DEGREE OF AGREEMENT QUESTIONS ABOUT SUPPORTING MATERIALS
Rating (1-10, Strong Disagree-Strongly Agree)
Characteristic of Materials 1 2 3 4 5 6 7 8 9 10 Average
Provide useful information
| ||| || |||| || || 7.2
Are easy to understand | || ||| || |||| | | 7.5
Do a good job describing
how the Smart Thermostat can help me control my energy use
and save money
| ||| || || ||| ||| | 6.9
Are missing important information
| ||| ||| ||| | | ||| 5.1
Responses to these questions were less positive than the overall impression questions. It appears that there is room to improve the materials along every dimension respondents were asked about.
As a follow-up, respondents were asked what additional information should be added to the supporting materials. Respondents gave these suggestions for additions to the material:
• How to install and use the app on the iPhone;
• Material indicating that the thermostat can only be programmed via the iPhone app or the internet app—that programming on the device itself is disabled (two respondents mentioned this issue);
• Material indicating that the thermostat literally interprets “wake-up time” as time
to turn the heat on. If the respondent wants the house to be warm before that, then he/she must set a wake-up time before the actual wake-up time. A similar point applies to “arrive home time;”
• Instructions on how to have different temperatures at different times while at home;
• Instructions on how to use the Vera router (FSC note: not sure what this is);
• A one-pager with all necessary steps for set-up;
• Clear examples of when to use the iPhone app and when to control the unit using its own controls; and
• Examples of when the HVAC will go on and off based on programmed arrival times, going to bed times, etc.
PG&E’s Emerging Technologies Program
FREQUENCY OF USE OF APPS
Respondents were asked how many times so far they had used the iPhone app and internet app to control their thermostat. The tallies of these responses are shown in Table 3. In
general, respondents use the iPhone app much more than the internet app.
TABLE 3: NUMBER OF TIMES RESPONDENTS HAVE USED EACH APP
App 5+
times
3-4
times
1-2
times
Not at
all
iPhone App 12 1 2 0
Internet App
2 3 2 8
FINAL OPEN-ENDED COMMENTS
At the end of the survey, respondents were asked for anything else they would like to share about their experience so far. They gave the following responses:
• “The app installation process was very confusing. Also, there wasn't really any explanation of how to install the app on multiple devices;”
• “Suggested feature for Opower: leverage the geolocation feature of the iPhone to
offer option to let system detect that we are outside the home;”
• “Overall everything went pretty smoothly;”
• “The phone app is great. Easy and makes sense. The only thing I don't like is the physical thermostat. It is too big and it is too GREEN. Why does the light
not completely go out if I am not using it? I don't need the glowing nightlight in my kitchen;”
• “I made a mistake when I was setting my schedule, and I thought I had set the temperature very low during the day while we are usually out of the house. But
we had some contractors working replacing a bunch of windows, and they left doors open, one of them told me the heat was continually blasting the whole time they were working. This went on for a few days before I found out. I felt secure
that I had done the settings correctly. It would be great if the app could send you a text message, something like ‘Your heat is set at 68 degrees while you are gone from home today.’ No explanation necessary, just a fact that looks right or wrong. You could even let people choose the kinds of messages they'd like to
receive.”
• “Enrollment and putting the app on my iPhone was very easy. I don't have the URL for the PC app but would like to try that. The 2 guys who installed my
thermostat were very professional;”
• “So far the use / controls have been flawless;”
• “I would add a feature to dim the light on the thermostat;”
• “Outside of the installation of the app and the issue I had there, everything has
been very favorable. I appreciated the reminder call for the appointment, the warning that the crew was 45 minutes away and the fact that the crew sent to my home was courteous (even wore booties inside my home so as not to scratch of floors - nice touch that showed me they cared);”
PG&E’s Emerging Technologies Program
• “As a customer, I was initially a bit apprehensive about the installation process because the guys installing it were being trained. This apprehensiveness quickly
faded because the gentlemen supervising them were very meticulous and supervised closely. I'd like to say that the installers were VERY professional. They arrived at the beginning of the 4-hour window, they wore shoe covers to protect my carpets, they explained everything very thoroughly and I felt
comfortable having them in my home. On the iPhone application it's not 100% clear to me how the leaderboard is calculated. Although this is a cool feature, I'm not sure I would change my behavior much to influence my position on the board. I'm already trying to maximize savings;”
• “Did not know that we had control on PC - entire program seemed to be iPhone focused. Having a good video to watch - about how to use would be helpful. Need to call help desk during non-business hours when I am home;”
• “Very easy, The web interface could be more informative, but I am a tech person and like more detail;”
• “The thermostat should have the option to program it directly. Versus stating when I get up, leave, return, etc...I would rather state when I would like the
equipment to turn off and on.”
SECOND SURVEY OF PARTICIPANTS
Invitations for the second survey were sent to participants on January 13, 2012. Reminder emails were sent to non-respondents on January 20. A total of 15 employees responded by January 24, 2012. Those responses are summarized here.
The main findings of the survey are that:
• 13 out of 15 respondents thought that they had reduced energy usage through use of the thermostat and apps;
• 14 out of 15 respondents thought that the pilot should be rolled out to 750 customers;
• Customers were generally glad to have been part of the pilot;
• My Thermostat was the most popular name;
• The iPhone app was more well-liked and heavily used compared to the internet app; and
• The iPhone app is both well-liked and the most popular reason given for why the
respondent was able to save energy. Respondents also generally say that it is easy to use.
HIGH PRIORITY ISSUES
There were several high priority questions that are addressed individually. All follow up comments to these questions are shown in Appendix B.
First, to the question of whether customers believed that they had reduced energy usage as a result of the thermostat. The tally was 13 yes and 2 no.
PG&E’s Emerging Technologies Program
Next, Question 24 asked whether the pilot should be expanded to 750. The tally was 13 yes and 2 no. One of the no responses was from a respondent who fairly clearly meant to say
yes based on his follow-up comment.
The most popular name (Q22) by a wide margin was My Thermostat, with 8 first place votes and 1 second place vote. Next were EasySet Thermostat (0 first, 6 second, 3 third), and Mobile Access Thermostat (MAT; 1 first, 4 second, 3 third); followed by MobileStat (2 first, 1
second, 3 third) and Intellistat (1 first, 2 second, 3 third).
Question 15 asked “Besides the support materials left at your home after installation, would you like to receive additional educational communications or information regarding your Smart Thermostat?” The responses were 3 yes, 7 no and 3 don’t know. See the Appendix
B for further details on what customers would like to see.
Table 1 below contains the tallies of the answers to question 8, which is about each respondent’s degree of agreement with a set of statements. The answers were mixed,
although the most positive response came to the question about whether the respondent was glad to have signed up.
PG&E’s Emerging Technologies Program
TABLE 4: TALLIES OF RESPONSES TO DEGREE OF AGREEMENT QUESTIONS
Statement
Degree of Agreement Rating Tally
Average 1 (Strongly Disagree) 2 3 4 5 6 7 8 9
10 (Strongly Agree)
The thermostat, iPhone, and web applications raise my concerns about data privacy.
|||| ||||| || | | 2.7
The thermostat, iPhone, and web applications have met my expectations.
| || | || ||| ||||| 8.4
The thermostat, iPhone, and web applications give me greater control over my energy usage.
| | | || || || ||||| 8.1
The thermostat on the wall is easy to use.
| || | | ||| |||||| 7.9
The iPhone application is easy to use.
| | ||| |||| ||||| 8.7
The web application is easy to use.
| ||||| | ||| | ||| 6.4
The iPhone app and web applications provide useful information.
||| | | ||| || || ||| 7.0
I’m glad I signed up for the trial.
| ||| || |||||||| 9.3
I make adjustments to my temperature settings when coming home early or late.
| || | |||| | || |||| 7.0
My initial pre-programmed settings have kept me comfortable.
| || | | |||| || || | 6.3
I like the messages I receive when I make temperature adjustments.
|| || ||| | ||| | | || 5.9
The thermostat, iPhone, and web applications have made me more aware of how much energy I use.
|| || | |||| || || 6.3
Those who answered yes to whether they had saved energy were asked to rank in order the
reasons they thought they had done so. Table 2 shows the tally of responses. Remote access through the iPhone, followed by the initial temperature profile were the main reasons. The respondent who ranked “other” first gave this reason: “I was able to reduce my temperature setting to 62 at night with no discomfort, versus the 65 previous night time
setting.” There were two other written in reasons provided under other.
PG&E’s Emerging Technologies Program
They were both ranked third and they were:
• “can check my temp and adjust if I am not coming home.”
• “Easy Scheduling Control”
TABLE 5: RANKING OF REASONS FOR SAVING ENERGY
Feature Ranking Average
Ranking 1st 2nd 3rd 4th 5th
Setting up my temperature profile
upon enrollment || |||| ||| ||| 2.1
Remote access to my thermostat through my iPhone
|||| |||| ||| 1.2
The messages I have received || |||| ||| ||| 4.1
The Leader Board feature of the iPhone app
|| |||| |||| || 3.5
Other | |||| |||| ||| 4.1
Four of 15 respondents had problems connecting with or controlling their thermostat
remotely. Four of 15 respondents had to contact customer service—three of which were
included in the customers who had trouble connecting.
OVERALL IMPRESSION RESPONSES AND COMMENTS
The survey contained four questions about participants’ overall impressions of the
thermostat, supporting software and customer service. Table 3 summarizes the responses
to those questions. It shows the tally of the number of times respondents gave each rating
on a scale from 1 to 10, with 1 meaning “very negative” and 10 meaning “very positive.”
There were no responses below 5.
TABLE 6: FREQUENCY OF RESPONSES TO OVERALL IMPRESSION QUESTIONS AND AVERAGE RESPONSE
Rating (1-10)
Feature 3 4 5 6 7 8 9 10 Average Rating
Thermostat
| |||| | |||| |||| 8.5
iPhone app
|| | |||| |||| | || 8.3
Internet app on your PC
| | | ||
| 7.3
Customer Service
||
| | 8.2
Overall, responses were quite positive, although there were fewer total responses about the
internet app and customer service. The iPhone app, thermostat and customer service
received very high ratings. The internet app received the lowest scores.
PG&E’s Emerging Technologies Program
For all but one of the overall impression questions, respondents were asked to provide
comments on why they gave the score they did. These are shown in Appendix B.
FREQUENCY OF USE OF APPS
Respondents were asked how many times so far they had used the iPhone app and internet
app to control their thermostat. The tallies of these responses are shown in Table 3. In
general, respondents used the iPhone app much more than the internet app.
TABLE 7: NUMBER OF TIMES RESPONDENTS HAVE USED EACH APP
App 15+
times 6-15 times
2-5 times
Not at all
iPhone App 7 6 2 0
Internet App 2 3 2 8
FOCUS GROUP RESULTS
On January 30 and 31, FSC conducted focus groups with Alpha test participants. Eight
members of the Alpha test participated in the January 30 group and one member
participated on January 31. Results from the focus groups are inherently non-quantitative
and results are presented as synopses and paraphrases of what was actually said.
The responses of each participant are paraphrased below. Participants were called on in a
random order for their response. Respondents are referred to by a randomly-assigned
number for anonymity. Respondent 9 was the participant from the January 31 group.
FIRST QUESTION
For the first question of the focus group, the FSC moderator asked participants “On a scale
from 1 to 10, 1 is boring, 10 is exciting, how would you rate it, both hardware and
software?”
• Respondent 6: rating: 8, but only if it would interact with her more. Would like more messaging on iPhone, displaying progress toward a goal. She would like to look at how she was consuming energy relative to how she was in the past. She learned she had set her thermostat wrong and she didn’t get a message that told
her she shouldn’t have been running as much as she was.
PG&E’s Emerging Technologies Program
• Respondent 4: rating: 5. It would be nice to have a two hour default so she could raise a couple of degrees and then go back. She also likes the idea of
having a comparison to prior usage.
• Respondent 3: rating: Hardware was a 2 to a 3. Could be higher if it was something like a Nest product that didn’t look and feel like a traditional thermostat. Application gets a 5. The product did a good job of getting him to
set up settings. Would be better if it pushed notices to him that told him that he had a message over the IP. After he set it up, the only time he went back into it was to show a friend he could do it. He didn’t realize that there was a leader board until Sarah [Bresko] told him later. He didn’t know where he stood relative
to others. The product didn’t push useful information to him. He would like to have control with other end-uses through the device, not just the thermostat, such being able to lock front doors. It would be better if it was something that
gave him a reason to go into it other than just adjusting the thermostat.
• Respondent 2: rating: Hardware a 7 and application an 8. Could improve if he could ask questions and get answers, if it was really smart and gave some guidance and feedback. He had a bad experience on the programming side. His
daughter was there when it was installed and she didn’t really get the education and when he tried to do the programming himself it wasn’t obvious how to program it. He would like to be able to tailor his settings to be a bit off of when
he wakes up or goes to sleep. It would be good to get a text message when he’s away asking whether he wants to change his setting. It would be good if tied into other applications. He thinks the thermostat is low touch and maybe could/should be high touch. The comparison stuff didn’t make much sense to
him. He didn’t understand it.
• Respondent 1: rating: solid 8 for both hardware and software. He really liked the leader board aspect but the more effort he put into saving gas, the worse that he was getting on the leader board. He liked the built-in intelligence, it was
very understandable. He liked the cool factor of controlling his thermostat remotely. One thing to think about implementing is a geo location feature that knows when everyone is away from the house and the software would know this
and adjust accordingly.
• Respondent 8: rating: – Software and 8, hardware a 7. He liked the cool factor of remote control. His wife complained that she wanted to control things as well but it only allowed one iPhone in this pilot. After awhile he didn’t see the benefit
of the tool unless they were out of schedule – such as away at a movie and they would turn off the schedule. Also, he could preheat house when coming home off-schedule. He would really like an estimated bill feature for the end of the
month that would tell him where he lies relative to a budget goal. He had similar experience with leader board as Respondent 1. It wasn’t clear who he was being compared to. He had to have people come back and install a repeater because it didn’t work well from the beginning. Then in the last week, his iPhone app failed.
So he has had two failures.
• Respondent 5: rating: Device a 3 and the app a 5. The device could be a lot more high-tech and engaging. It looks like every other thermostat. The device is big and bulky, not sleek. She had trouble programming, and she said her
furnace accidentally ran 24/7 for 3 days. She wants override ability. She would like to be able to program it on the thermostat rather than just remotely or on the PC. She thinks energy use is actually increasing rather than decreasing
because of lack of control of settings. She is looking for a lot more flexibility. The leader board didn’t provide any context. What are you being measured on.
PG&E’s Emerging Technologies Program
The recommendations it gave she had already implemented. She didn’t see any value in the information.
• Respondent 7: rating: none – More lenient because it’s an evolving journey. It’s a nicer thermostat but it’s not Nest. Integration into the iPhone was very helpful as a family. He and wife were very engaged with the application. The leader board – not sure if there is value yet because the comparison group is small. The
ability to know what the temperature is and engage with it remotely is very good. Overall experience is an 8, but to get to a 10, get rid of the bugs. It needs more push messaging – there is no push messaging. Needs to push notifications either of something going wrong or to draw you in. Eventually you will need a self-
aware thermostat, like a Nest. Hardware shouldn’t be dependent on human interaction.
• Respondent 9: Rate both app and thermostat as an 8. She would want the
thermostat to show the time. Her prior thermostat let her set things on four periods M through F and then on Saturday and Sunday. Would like thermostat to be more interactive like her prior thermostat. Likes the leader board on the app. Would like app to be a little more intuitive – if you lowered temperature by 3
degrees, you would save $X. Push more information out. Could do more with the leader board (smiley face when doing well). Wants it to show what temperatures are when they are set.
SECOND QUESTION “At what point would more push messaging or tracking of where you are start to bother
you? Who would worry about this?”
Respondents 8 and 5 had worries about the security of the data. Push messaging doesn’t
bother them as long as information is secure. As long as the benefits are there in terms of
reduced bills they are in favor of it.
Respondent 7 says we are constantly giving away information but those who will be
involved will consider the balance between intrusion and value.
Respondent 1 says the amount of interaction you want needs to be part of the design.
Some want more, some want less. Need to be able to self select.
THIRD QUESTION “Knowing what you know now, would you recommend to your friends that they replace their
current thermostat with this new one? If yes, what would you say its advantages are and if
you wouldn’t recommend it, why?”
• Respondent 3: Depends on the cost. If it was free, he would recommend it because
most people don’t know how to program their thermostat and this would be enough
if free. He would recommend to friends that are technologically challenged. It gets
people over the hardware challenge that a standard thermostat has.
• Respondent 2: Yes, but based on some caveats. If a family has a very diverse
schedule.
• Respondent 1: Yes. It saves some energy and provides education for the kids,
especially if leader board is working correctly. He also likes the cool factor.
PG&E’s Emerging Technologies Program
• Respondent 4: Yes, he would recommend it but would like to see something that
covers the whole house. Also he would like to have something that zones the house.
• Respondent 5: Maybe. If they didn’t already have a programmable thermostat to
begin with, she would recommend it but would also suggest looking at other things
on the market, such as Nest. She would probably recommend Nest instead.
• Respondent 8: Yes, at a reasonable cost, but it depends on current usage and how
much they would save. He likes the remote capability. H would think that this is a
step toward controlling other end uses.
• Respondent 6: if there was no cost and nothing to lose, she would say yes. The
highest value items are the remote control features. It is also easier to program
than on thermostat itself.
• Respondent 9: when away on vacation, she could see what the house temperatures
were. She has cats so she could set the temperature up in the evenings to take care
of the cats. Also she likes to know what current temperature is when away.
FOURTH QUESTION The fourth question asked about preferences for the iPhone app versus the PC app. All participants preferred the iPhone app.
FIFTH QUESTION “Is this ready to be tested on a larger group of 750 customers? If not, what needs to be fixed before doing this?”
• Respondent 7: yes, but the software must be bullet proof. Can be NO hiccups
between hardware and software.
• Respondent 3 doesn’t think we have a good handle on whether this really saves
energy. In his case, he thought it cost him more and not knowing this it could be a
customer nightmare. He would develop better functionality before going larger. He
wants push messaging to be part of it and he would look for a different, better
thermostat. He also would want to see broader control of other end uses. By the
time they deploy this, he is afraid this will be obsolete.
• Respondent 4 would like to see it add other end-uses first. She wants to see cost
broken down by end-use. Her bill went up and she wants to know why.
• Respondent 8: roll it out, but he wants more features. Installations need to be
correct the first time. Need to have installs on the weekend. Taking days off work
to get installed is bad. If there are problems, need to get technical people out there
very quickly, like that day. He would also like to see color coded information saying
whether you are or aren’t improving relative to goal. He says yes to next step but
doesn’t know if next step should really be focused on energy savings, but maybe
about other things like remote control—maybe that is the sell. There is also a need
to work through troubling installations. He also wants more interface options and
more messaging. He called help desk during the day when at work but it would be
easier if he was standing in front of thermostat.
• Respondent 6: yes, this is a good thing. She would love to see a test group for
SmartRate customers.
• Respondent 9: It would depend on the demographics of customers. If we rolled it
out to her dad, it’s not going to happen. If we’re looking at Silicon Valley, gen Y and
maybe some gen X’s, they would do better. The performance of the device and
PG&E’s Emerging Technologies Program
installation are ready for prime time. They went smoothly and the installer did a
good job.
SIXTH QUESTION “If you are a non-employee customer, how can you be marketed to, to say this is available?”
• Respondent 8: Home depot, Lowes, PG&E MyAccount and home market expos.
Coupons or kiosk in thermostat aisle.
• Respondent 6: Find out who you want and go after them. High bill folks? Tech
savvy people? Use a screening survey.
• Respondent 5: For a pilot, plug into existing channels for residential customers to
minimize impact to customers. We have an HVAC QM program. Can we tie into
these customers so don’t have to schedule another appointment – do it when HVAC
vendor is already there. For larger group, Home Depot, social media, people that
already doing service calls for something else. Don’t want to have to wait in a four
hour window to get installation.
• Respondent 1: Random, outbound calls from call center.
• Respondent 3: retail alliances are important but small. A bigger channel will be the
HVAC contractors. Piggy back on existing programs. Information at the Apple Store.
• Respondent 2: Send marketing with high bills notices. People investing in retrofit
programs. Focus in areas that have negatively complained about Smart Meters.
Movie theaters.
• Respondent 4: Should add to other smart phones, but for now should sell through
the app store.
• Respondent 9: Once again, it depends on demographics. Her dad reads the paper
and that would be a better way to reach him. She has an iPad and iPhone and she
gets information from there. Facebook and twitter could work, depending on target
market. If you’re pursuing tech savvy people, she thinks you should tweet it and
push it out that way. She doesn’t think Craig’s list would be good here.
SEVENTH QUESTION “Will rolling this out now enhance PG&E’s image or not?” (Respondent numbers not
recorded) • Respondent: Depends on how we market it. It should help if done well.
• Respondent: It all goes back to what expectation we set up front and do we meet or
exceed that expectation. Are we setting an expectation that they will save energy.
If they don’t exceed it they may be unhappy.
• Respondent: Don’t sell on saving energy, sell on education and remote control. It’s
empowering.
• Respondent: In general it will enhance PG&E’s image but only if done well.
• Respondent: needs to be lowest common denominator technology. Can’t just be for
the techies—need to be simple for everyone (like the color coding of the EV
mentioned).
• Respondent: Focus on control and comfort, not energy savings.
PG&E’s Emerging Technologies Program
• Respondent: What is the goal – energy savings, customer satisfaction, choice?
Does PG&E have to show savings to put the product into play?
• Respondent 9: It will enhance it. Putting information in the hands of customers is
important.
EIGHTH QUESTION Participants were asked whether they liked the messaging they had been receiving. No one remembered any messages.
NINTH QUESTION “What kind of messages would be useful?”
• Respondent (number not recorded): Remind them to use it.
• Respondent 6: Wants to know how she is doing relative to a goal and to alert her if
something looks unusual or wrong so she can check it out. But give people choice
regarding the type and amount of messaging.
• Respondent 5: If you are setting your thermostat out of the norm for the
population, ask whether that’s what you really meant.
• Respondent 1: Messages should only emphasize those that entail significant dollar
amounts. He would like to see a message that gives tips for saving energy and
money and ties behavior change to dollar amount.
TENTH QUESTION
In the tenth question participants were asked whether they would like having the ability to program the thermostat itself rather than having to program it through the iPhone or PC
apps. Participants agreed that that would be an important and useful feature, specifically for times when they cannot access the apps.
LEADER BOARD
Although not part of any particular focus group question, several participants expressed frustration with the leader board function of the iPhone app. This part of the app compares
a customer’s HVAC usage and climate control choices to other customers and places customers in a ranking based on who is most efficient. Participants had trouble understanding the rankings and more than one mentioned frustration at undertaking an
effort to improve their leader board rank only to fall in the rankings.8
DATA ANALYSIS
Based on their analysis of their own operations, Opower provided certain statistics on technical problems that affected participants and on usage of the thermostat and the apps. Major findings were that:
• All participants were affected during a 17-minute outage of the Opower database. During this time, the thermostat would not respond to commands;
8 The leader board function was removed from the app and will not be shown to Beta customers.
PG&E’s Emerging Technologies Program
• Due to other problems between the Opower database and the thermostat operating system, seven users experienced total outages greater than 1 hour in length and two
users experienced total outages greater than 2 hours in length;
• Due to problems between the operating system and the thermostat itself, one user experienced 20 hours of downtime and another experienced 82 hours of downtime. During these periods, the thermostat did not respond to user commands. This was
caused by a poor connection between the Gateway (which houses the operating system) and the thermostat itself;
• Aside from these issues, there was very little downtime for participants;
• Participants viewed the mobile app an average of 0.8 times per day and the PC app
about four times less frequently;
• The mobile app usage was steady over the course of the study period;
• The average time spent on the mobile app was 2-3 minutes per use; and
• The thermostat page was the most frequently viewed page on the mobile app, followed by the schedule page.
EVALUATIONS
The main results of the evaluation are that the technology works well, although there are
certain refinements that would be useful. The new technology appears to be an improvement on existing thermostats in many ways.
The primary marketing barrier is that customers must have iPhones. This limitation is being addressed in the Beta test by expanding the technology scope to include Android smart
phones.
Energy savings due to the technology has not yet been measured, but seems plausible. It will depend completely on participants in the Beta test using the thermostat to set lower
temperatures at home during the winter and higher temperatures during the summer, primarily when no one is at home.
RECOMMENDATIONS Recommendations from the Alpha test are:
• That Opower or Honeywell follow up with respondents who indicated either that the thermostat isn’t working or that one of the apps isn’t working. The most important steps are to identify the problem and communicate back what the problem is, how it
can be fixed and whether that problem is likely to occur often in the broader pilot;
• That Honeywell add a step to the installation process in which the installer fully verifies that the iPhone app controls the thermostat before the installation is
considered complete;
• That PG&E work with Opower, Honeywell and FSC to revise the supporting materials that installers leave behind based on the suggestions below;
• That PG&E adopt the name My Thermostat for the product; and
PG&E’s Emerging Technologies Program
• That the thermostat be modified to work with Android smart phones as well (a step that is being taken for the Beta test).
APPENDIX A SURVEY INSTRUMENTS
First Survey Smart Thermostat Employee Trial
Survey 1: Installation and Initial Impressions
Introduction:
Thank you for taking the time to participate in this trial and completing this survey. This survey should take around 10 minutes to complete. We encourage your candid feedback
and there are many optional open-ended questions to enable you to expand on your impressions. Your responses will be used for research purposes only.
To begin, please click on the button below.
1. We are interested in understanding your overall impression of the web portal
enrollment that you used to sign up for the Smart Thermostat Program. Using the following 10-point scale, where “1” means “very negative” and “10” means “very
positive,” please rate your overall experience with the web portal enrollment process. 1 2 3 4 5 6 7 8 9 10
Very negative Very positive
2. [if Q1 is not 8-10] Why did you give a rating of [insert rating from above] for the online
enrollment process? Please be as specific and detailed as possible. [Open end] 3. We are interested in understanding your overall impression of the call center, which
you used to schedule your installation and may have used if you had any other issues.
Using the following 10-point scale, where “1” means “very negative” and “10” means “very positive,” please rate your overall experience for any interactions with the call center.
1 2 3 4 5 6 7 8 9 10
Very negative Very positive
4. [if Q3 is not 8-10] Why did you give a rating of [insert rating from above] for the call
center experience? Please be as specific and detailed as possible. [Open end and not required]
5. We are interested in understanding your overall impression of the installation of your thermostat. Using the following 10-point scale, where “1” means “very negative” and “10” means “very positive,” please rate your overall experience with the installation
process of the thermostat. 1 2 3 4 5 6 7 8 9 10
Very negative Very positive
6. [if Q5 is not 8-10] Why did you give a rating of [insert rating from above] for the
installation process? Please be as specific and detailed as possible. [Open end and not required]
7. Did the installer arrive at the agreed upon time? (yes/no)
PG&E’s Emerging Technologies Program
8. [If no to 7] You indicated that the installer did not arrive at the agreed upon time. Please provide us some more information about when your installer arrived. [Open end
and not required] 9. Did you feel that the amount of time it took to install the thermostat solution was
reasonable? (yes/no) 10. [If no to 9] You indicated that the amount of time it took to install the thermostat
solution was not reasonable. What do you think would be a more reasonable amount of time? [Open end and not required]
11. We are interested in understanding your overall impression of the installer of your thermostat. Using the following 10-point scale, where “1” means “very negative” and
“10” means “very positive,” please rate your overall experience with the thermostat installer.
1 2 3 4 5 6 7 8 9 10
Very negative Very positive
12. [if Q11 is not 8-10] Why did you give a rating of [insert rating from above] for the installer? Please be as specific and detailed as possible. [Open end and not required]
13. Please indicate your level of agreement with the following statements using a 10-point scale where “1” means you “strongly disagree”, “10” means you “strongly agree”, and
you can use any number between 1 and 10… (Randomize statements)
1 2 3 4 5 6 7 8 9 10
Strongly disagree Strongly agree
The support materials left at your home after installation: Provide useful information
Are easy to understand Do a good job describing how the Smart Thermostat can help you control your energy use
and save money Are missing important information
13b. [If Q13 is not 8-10] What additional information would you suggest could be added to the support materials? Please be as specific and detailed as possible. (open end and not required)
14. We are interested in your initial impression of the thermostat. Again, using a 10-point scale, would you say you…?
1 2 3 4 5 6 7 8 9 10
Do not like this thermostat
Like this thermostat very much
15. About how often have you tried using the iPhone app to control your thermostat? 5 times or more
3-4 times 1-2 times Not at all
16. [If more than not at all to prior question] We are interested in your impression of the iPhone app. Again, using a 10-point scale, would you say you…?
1 2 3 4 5 6 7 8 9 10
Do not like the iPhone app
Like the iPhone app very much
17. [if Q16 is not 8-10] Why did you give a rating of [insert rating from above] for the
iPhone app? Please be as specific and detailed as possible. [Open end and not required] 18. About how often have you tried using the internet app on your PC to control your
thermostat?
PG&E’s Emerging Technologies Program
5 times or more 3-4 times
1-2 times Not at all 19. [if more than not at all to prior question] We are interested in your impression of the
internet app on your PC. Again, using a 10-point scale, would you say you…
1 2 3 4 5 6 7 8 9 10
Do not like the internet app on your PC
Like the internet app your PC
20. [if Q19 is not 8-10] Why did you give a rating of [insert rating from above] for the
internet app on your PC? Please be as specific and detailed as possible. [Open end and not required]
21. Is there anything else you would like to share about your experience (enrollment,
installation, use of devices, etc.) so far with the Smart Thermostat Program? [Open end] 22. If we have follow-up questions about your responses, what is the best phone number
and time for us to reach you? [Open end] We’re almost finished. Our last question is for classification purposes only. D1. How many people, including yourself, live in your household? (Choose only one)
1, just me 2 3 4
5 or more That is the end of our survey.
Thank you very much!
Second Survey Smart Thermostat Employee Trial
Survey 2: Thermostat Usability
Introduction:
Thank you for taking the time to participate in this trial and completing this survey. This
survey should take less than 10 minutes to complete. We encourage your candid feedback and there are many OPTIONAL open-ended questions to enable you to expand on your impressions. Your responses will remain anonymous and will be used for research purposes
only. To begin, please click on the button below.
1. We are interested in your overall impression of the thermostat to date. Using a 10-point scale, would you say you…?
1 2 3 4 5 6 7 8 9 10
Do not like this thermostat
Like this thermostat very much
PG&E’s Emerging Technologies Program
2. About how often have you tried using the iPhone app to control your thermostat? a. More than 16 times
b. 6-15 times c. 2-5 times d. Once e. never
3. [If not e] We are interested in your impression of the iPhone app. Again, using a
10-point scale, would you say you…?
1 2 3 4 5 6 7 8 9 10
Do not like the iPhone app
Like the iPhone app very much
4. [if prior question answered] Why did you give a rating of [insert rating from
above] for the iPhone app? Please be as specific and detailed as possible. [open-
end and not required]
5. About how often have you tried using the internet app on your PC to control your thermostat?
a. More than 16 times b. 6-15 times c. 2-5 times
d. Once e. never
6. [if not e to prior question] We are interested in your impression of the internet
app on your PC. Again, using a 10-point scale, would you say you…
1 2 3 4 5 6 7 8 9 10
Do not like the internet app on your PC
Like the internet app your PC
7. [if prior question answered] Why did you give a rating of [insert rating from
above] for the internet app on your PC? Please be as specific and detailed as possible. [open-end and not required]
8. Please indicate whether you agree or disagree with the following statements using a
10-point scale where “1” means you “strongly disagree”, “10” means you “strongly
agree”, and you can use any number between 1 and 10... [Randomize
statements]
1 2 3 4 5 6 7 8 9 10
Strongly disagree
Strongly agree
a. The thermostat, iPhone, and web applications raise my concerns about data privacy.
b. The thermostat, iPhone, and web applications have met my expectations.
c. The thermostat, iPhone, and web applications give me greater control over my energy usage.
d. The thermostat on the wall is easy to use. e. The iPhone application is easy to use.
f. The web application is easy to use. g. The iPhone app and web applications provide useful information. h. I’m glad I signed up for the trial.
PG&E’s Emerging Technologies Program
i. I make adjustments to my temperature settings when coming home early or late.
j. My initial pre-programmed settings have kept me comfortable. k. I like the messages I receive when I make temperature adjustments. l. The thermostat, iPhone, and web applications have made me more aware of
how much energy I use.
9. If you have any comments about the ratings you just provided, please note them
here. Please be as detailed as possible. [open-end and not required]
10. Have you had any problems connecting with or controlling your thermostat remotely?
a. Yes
b. No
11. [if yes to prior question] Please describe the problems you had. Please be as specific and detailed as possible. [open-end and not required]
12. Did you need to contact customer service for support at any point?
a. Yes
b. No
13. [if yes to prior question] Did you feel that your customer service experience was positive? Using a 10-point scale, would you say you had a very positive customer
experience (10) or a negative customer experience (10)?
1 2 3 4 5 6 7 8 9 10
Negative customer experience
Positive customer experience
14. [if Q14 is not 8-10] Why did you give a rating of [insert rating from above] for
your customer service experience? Please be as specific and detailed as possible (Optional). [Open end and not required]
15. Besides the support materials left at your home after installation, would you like to
receive additional educational communications or information regarding your Smart Thermostat?
a. Yes b. No c. Don’t know
16. [If 15 is yes] What additional communications or information would you like to
receive? Please be as specific and detailed as possible. [open-end and not
required]
17. [If 16 is yes] How would you like to receive the additional information? (Choose all
that apply) [Randomize options]
a. By mail, in the same envelope as my PG&E bill b. By mail, but in a separate envelope from my PG&E bill c. Via email d. Online at pge.com
e. Mobile application on your SmartPhone f. Phone call from PG&E g. Text message on your mobile phone from PG&E
PG&E’s Emerging Technologies Program
h. Other (Specify)
18. Do you believe that you have reduced your energy use as a result of the thermostat
and messaging provided through this pilot? [Yes/no]
19. If no, please provide feedback on why you feel the thermostat and messaging have not helped you reduce your energy use. [open-end and not required]
20. If yes, please rank in order of importance the following reasons why you think you
have been able to reduce your energy use (where 1 is the most important reason and 5 is the least important reason):
Reason Ranking
Setting up my temperature profile upon enrollment _______ Remote access to my thermostat through my iPhone _______ The messages I have received _______ The Leader Board feature of the iPhone app _______
Other _______ Please explain ___________________________________________________
21. If you have any comments about the ratings you just provided, please note them here. Please be as detailed as possible. [open-end and not required]
22. If PG&E decides to roll this out beyond the employee pilot, a name will need to be created to market the program. Based on your experiences with the thermostat solution please rank your top three choices for a name: [Randomize]
a. My Thermostat b. My Therm c. Intelistat
d. Mobilestat e. Remotestat f. Customstat g. Mobile Access Thermostat (MAT)
h. EasySet Thermostat i. OptiTemp Thermostat
23. What other feedback, if any, do you have concerning your participation in this pilot (Optional)? [Open end and not required]
24. Based on your overall experience with the participating in this pilot so far, would you
recommend that PG&E expand the pilot to include participation by up to 750 customers?
a. Yes
b. No (if no, Please indicate the primary reason or reasons why you think PG&E should not expand this to a broader customer population at this time.).
25. If we have follow-up questions about your responses, may we contact you? a. Yes b. No
PG&E’s Emerging Technologies Program
26. [if yes to prior question] What is the best e-mail/phone number and time for us to reach you? (We will contact you only if we have follow-up questions to this survey.)
That is the end of our survey. Thank you.