Key Findings from the 2011 E Source Website Study

www.esource.com

Key Findings from the 2011

E Source Website Study

Stephanie Spalding

Research Manager, E Source

EUCI Web Self-Service Conference

January 2012

www.esource.com || © 2011 E Source 2

The Good!

“a real pleasure”


“clever and fun”


“detailed and easily

accessible

user-friendly

experience.”


“awesome!

great resources

I would feel prepared and

informed”


The Bad …

“Overall it was a

frustrating

experience.”


“This was just too many

steps and kind of

annoying.”


“The site is poorly

organized and a few years

behind the times.”


… and the Ugly

“lost, confused,

and frustrated. Should

be revamped.”


“What a horrible online

account page.”


“This was a pretty awful

experience.”


Updated in 2011: Mobile Web


New in 2011: Social Media

Source: Baltimore Gas & Electric

Source: Pacific Gas and Electric Co.


Which Features Were Reviewed?

General Information

Home Page

Contact Us

Payment and Billing

Options

Moving

Energy-Saving Tips

Programs and

Services

Search

Secure

Registration and Login

My Account

Account Balance

Account History

Online Bill Viewing

Online Payment

My Energy Use

Outage/Gas Leak

Status

Online Account

Management

Work Order Status

Payment

Arrangements

Mobile

Home Page

Contact Us

Outage/Emergency

Reporting*

Outage/Gas Leak

Status


My Account

Account Balance

Online Payment

My Energy Use

Account Alerts*

Payment

Arrangements * = Only reviewed on mobile devices


Which Companies Were Included?

The largest electric and gas companies in the U.S.

and Canada plus a representative set of small and

midsize utilities

100 companies met our criteria, including 94 that

were also reviewed in 2009

72 companies provided us with account information

to review secure options


How Were Reviews Conducted?

Between December 2010 and April 2011

By task in four phases

Tasks by company were randomly assigned

Questions reviewers asked:

Was the feature available on the website?

Was it easy to use?

Reviewers commented on their experiences,

provided ratings, and took screenshots of each

feature


Who Were the Reviewers?

Six people recruited from our local Boulder,

Colorado, community

Three men, three women

Internet and mobile-device savvy

Two reviewer segments:

“Transactional” reviewers view and pay their bills online.

“Informational” reviewers use the web but do not view or

pay their bills online.


How Was Quality Controlled?

We provided extensive training

Supervisors scrutinized reviews for consistency

between the comments and the rating given, and to

ensure our processes were followed accurately

Half of the reviews were conducted a second time

by a different reviewer


Methodology Changes

Tasks were not named

Reviewers were given an unnamed scenario describing

the task they should attempt (“Task A,” not “Payment and

Billing Options”)

This change was intended to remove any bias toward

looking for an exact feature name.

Task A

Scenario: You’ve been receiving and paying your bill through

the same methods for years and are curious what your other

options are. You go to the website to investigate.


Methodology Changes (cont.)

Limited review without account access

If we didn’t have account access, we did not attempt to

review several features that we determined would only be

available behind a login:

My Account

My Account (Mobile)

Account Balance

Account Balance (Mobile)

Account History

Online Account Management

Online Bill Viewing


What Is “Usability”?

A measure of the reviewer’s experience using the

web to accomplish their task.

Usability rating Interpretation

5 (very good) Excellent experience; easy to find and/or use; there are

no suggestions for improvement

4 (good) Good experience; reasonably easy to find and/or use;

has few areas that need improvement

3 (average) Average experience; neither easy nor difficult to find

and/or use; has obvious areas that need improvement

2 (poor) Poor experience; difficult to find and/or use; requires

significant improvements

1 (very poor) Very poor experience; the option is present on the

website, but is extremely difficult to find and/or use;

requires significant improvements to be usable © E Source


What Is “Functionality”?

The percentage of the 29 features that were found on

an electric or gas utility’s website.

Example: If we found 20 of the 29 features we

looked for, the functionality score would be 69

percent.


© E Source

1

2

3

4

5

0 20 40 60 80 100

Us

ab

ilit

y

Ave

rag

e r

atin

g fo

r e

ach

we

bsite

Functionality Average percentage of features found

Account Access No Account Access

Average functionality = 75%

Average usability = 3.1

Overall Performance of Utility Websites


Slight Preference for Apple Devices

1

2

3

4

5

BlackBerry Curve (88) iPad (145) iPod Touch (445) Sanyo Zio [Android](103)

Ave

rag

e u

sa

bil

ity

Mobile device (number of features reviewed on device)

Note: Usability was rated on a scale from 1 to

5, where 1 = very poor and 5 = very good.

© E Source

Overall mobile average usability = 3.0

Total features tested on mobile devices = 781


Usability Ratings Lowest for Online

Payment and My Energy Use

© E Source

Feature Average usability

rating

My Energy Use 2.91

Online Payment 2.87

Online Payment (Mobile) 2.83

My Energy Use (Mobile) 2.78


Not Much Difference Between Mobile

and Desktop Usability

1 2 3 4 5

My Energy Use

Online Payment

My Account


Contact Us

Payment Arrangements

Account Balance

Home Page

Outage/Gas Leak Status

Usability rating

Fe

atu

re

Mobile Desktop

© E Source


1

4

4

5

5

6

8

10

13

0 2 4 6 8 10 12 14

Account Alerts


Online Payment

Contact Us

My Account

Account Balance

Outage/Gas Leak Status

Home Page

Outage/Emergency Reporting

Number of Sites Supporting

Mo

bil

e-O

pti

miz

ed

Featu

re

Most Common Mobile-Optimized

Features

© E Source


Mobile-Optimized is Best

3.1 vs. 3.7


Top Recommendations from 2011


What’s Next?

E Source Customer Experience Conference

May 7-9, 2012

Green Valley Ranch, Henderson, Nevada (close to Las

Vegas)

Highlights:

Utility Chief Customer Officer Keynote

Outside industry experts (Zappos, FedEx, etc.)

Zappos Insights Tour

Great facility, networking, and dining!

www.esource.com/ucec2012

http://www.esource.com/ucec2012


For More Information

Stephanie Spalding

Research Manager, E Source

303-345-9121 [email protected]

@EsourceSteph

Have a question? Ask our experts: www.esource.com/question

mailto:[email protected]

http://www.esource.com/question


Questions?

Key Findings from the 2011 E Source Website Study

Documents

Transcript of Key Findings from the 2011 E Source Website Study