Keynote Benchmark

t’s a promise that seemed almost too good to be true: Overnight, one of the company’s enterprise-critical applications is updated

to the latest version. In the morning, employees — at the home office and field offices across the country — are working along as

usual, noticing only some important new features that help them get their jobs done better. Later, at a management meeting, the

CMO reports that faster responses to opportunities have bumped their share five points. And the CIO reports that the technology

spend is exactly on target, no additional budget needed.I

Forecast: Partly Cloudy and A Chance of Applications

The Promise and Performance Challenges of Software-As-A-Service

2

Elsewhere, in a nondescript glass-and-steel

building in northern Virginia, the CEO of a

two-year-old tech startup reports to his team

that the overnight upgrade to their core

enterprise application offering went flawlessly.

Monthly subscription revenue is up by high

double-digits. Customers are happy. The press

veritably glows about their cloud-based

application’s wonders. And Google, Microsoft,

and a host of VCs and investment bankers are

lighting up the phone lines with offers of

truckloads of cash.

SAAS AT LAST?After years of latency, software-as-a-service is

moving to center stage, swept along with the

bigger concept of cloud computing. On the

software side, applications like Salesforce,

Workday and Freshbooks — and yes,

GoogleApps — have become serious players in

enterprises of all sizes. And in the bigger

cloud, services like Amazon’s EC2 Elastic

Compute Cloud and IBM Smart cloud

services — and yes, Google AppEngine — are

changing the way IT departments approach

their missions and their development tasks,

enabling greater speed and flexibility than

ever before.

Forrester Research counts

“ubiquitous deployment of software-as-a-

service for packaged applications” as one of its

15 tech trends for the next three years.i

Gartner predicts a 23.8 percent compound

annual growth rate for SaaS through 2012,

more than doubling overall market growth of

11.4 percent.ii And according to a Piper Jaffray

report quoted in Information Week, spending

on the cloud will outstrip the overall software

market by a factor of five.iii

We’ve seen this type of technology

trajectory before and can recognize its course;

think back to the Web itself, to early social

networking, to the still-snowballing mobile

data market. The dominant SaaS providers are

still rising to the top; economics, standards

and best practices are being sorted out;

business models are being built and business

cases made. The difference is that these

sweeping technological changes keep

happening faster and faster.

Many point to Salesforce.com as a

success story and example of what’s possible.

Indeed, a Gartner consultant recently put

Salesforce’s share of the total SaaS spend at

14.6 percent.iv But without certainty as to the

potential, nor enough track record to know

and avoid the pitfalls, how willing will

enterprises be to trust their critical

applications to cloud-based solutions that

ultimately are out of their control? A few

stories in the press about outages at Google,

Facebook, and Twitter are enough to give any

business decision-maker pause. Even so, the

SaaS train has apparently left the station. The

question is, at what stop will businesses get on,

and for how long a ride?

CLOUDY TERMINOLOGYTo the average business person, the whole

concept can be a little — cloudy. In the most

general sense, “the cloud” refers to IT

infrastructure and functionality that exists

outside a company’s walls (except in the case of

“internal clouds,” but that’s another story),

whether it’s for data infrastructure

(Infrastructure as a Service), processing power

(Platform as a Service), or business

applications (SaaS). These things collectively

make up “the cloud.”

While all aspects of the cloud can

and do impact businesses as a whole, as well as

individual users, it is software-as-a-service

that is the most visible and has the most direct

effect on how people do their jobs every day.

SaaS either replaces or supplements the

on-premises or desktop applications that

workers use day-in and day-out, whether it’s

simple word processing or complex CRM or

HR functions. In some cases, most notably

Microsoft and its Office suite, hybrid solutions

combine familiar desktop applications with

added functionality from the cloud, such as

enhanced collaboration, online file storage and

anywhere-accessibility.

Such hybrid configurations may

serve as the gateway to cloud applications for

many companies. But by any route, it is likely

that SaaS is going to become business as usual.

Even Bill Gates saw it coming while still

running Microsoft, describing the “services

wave” as “the next sea change.”v

Anshu Agarwal, vice president of

marketing for Keynote Systems, sees users

themselves as important drivers of SaaS

adoption. “What I have seen is that, after years

of being dependent upon internal business

system schemes, users and companies have

become tired of having to deal with upgrades,

installations, and internal bottlenecks,” he

says. “Software-as-a-service is being driven as

much by internal users as it is by software-as-

a-service companies. It comes down to users

saying ‘I’m just tired of having to wait for what

I need. I can turn this on next week. I don’t

have to wait in a queue, I don’t have to be

denied the kinds of automations I want.’ I

think that’s what’s driving it.”

PROMISES, PROMISESIt’s a great idea on paper — replacing big

outlays for software and maintenance with flat

monthly fees, fast implementation, never

having to worry about upgrades, slashing

hardware expenses — but the SaaS cloud may

have to come closer down to earth for these

promises to be realized. Some of them are

more in reach than others.

Lower Cost. It’s a reasonable

conclusion that initial implementation cost

can be significant lower with SaaS than with

on-premises software. Forrester Research says

SaaS implementation can typically run .5x to

1x the first year’s subscription fee, vs. 1x to 5x

for traditional software.vi Total cost of

ownership, however, is not so clear-cut.

Keynote Benchmark

SaaS TCO can be lower if it brings

corresponding reductions in IT hiring,

hardware purchases, and database and

app server licenses.vii The balancing of price

and performance will ultimately determine

whether money is saved, or value added for a

comparable investment. One big advantage

SaaS does have is that costs are knowable,

and predictable, based on fixed monthly

per-user fees.

Rapid Deployment. Speed of

implementation is a real advantage for SaaS.

Companies have no need to gear up with

hardware or devote IT resources to

installation, testing, and rollout. All that is

done on the vendor side. The trade-off,

though, is that implementations are fairly

standard, with limited opportunity for

customization. If it fits as is, great. If not,

business processes need to be adjusted to

accommodate the software, instead of vice

versa. For some, that is a serious trade-off.

Seamless Upgrades and Maintenance.

Another checkmark in the plus column for

SaaS — instead of the highly disruptive and

expensive process of upgrading on-premises

software, SaaS upgrades happen on the

vendor side, often without any interruption of

normal business processes. And with SaaS,

upgrades and fixes can happen more

frequently. The SaaS vendor team is dedicated

to one system that serves many clients — with

no string of legacy products to support — and

therefore can bring innovation more quickly

to client companies with none of the stress

and turmoil of traditional upgrades. As an

example, in the ERP arena, in the five-year

period from 2004 to 2009, SAP rolled out just

two product releases, and Oracle and

PeopleSoft three each. During the same

period, Workday, a SaaS competitor,

implemented nine product releases.viii

And underlying all these benefits is

the potential for a now-smaller corporate IT

staff to focus on gaining competitive

advantage instead of routine housekeeping

and utility work.

THE ELEPHANT IN THE CLOUD: PERFORMANCEFor all of its tremendous potential, there’s one

thing that will make or break the SaaS model

both for vendors and users: Performance.

No longer is technology contained within the

walls of the enterprise, running on its own

proven network, controlled closely by its own

IT department. Now, the nerve center of the

enterprise, the productivity of workers, the

integrity of information assets is controlled by

an outside entity, flows through the various

pipes from a remote data center over the

Internet into the enterprise’s network and

ultimately, into a browser. And that presents

challenges both for performance and

user experience.

“My browser is not like a dedicated

desktop application,” Agarwal says. “Take for

example, Outlook. It’s fired up in the morning

and it synchronizes and voilà! The mail is

there. But I can also use Web mail, right? And

it’s a very different experience.”

“It may not be as rich, it’s not as

responsive. And of course, the performance

will vary. I think what’s likely to happen is,

there’s going to be some amount of buyer’s

remorse. Users are going to feel like

they gave up a lot of rich features and

functionality. It’s not the same, experiencing

an application through a browser. It’s not

going to come close.”

The problem with the browser being

the front-end for SaaS applications is that

users have very clear expectations of a browser

experience, based on their use of the Web.

Users go to a site and expect it to load fast —

in two seconds or less. They don’t like to

wait, and won’t. Google says that for every

additional 500ms of delay, the site loses 20

percent of its traffic. With fast broadband and

wireless connections everywhere, users expect

blazing speed when they fire up their browser.

3

Keynote Benchmark

Once you take an application outside the confines of the enterprise data center, a whole host of factors can have an impact on performance.

“You need to understand the browser,” says

Robert Hughes, director of Global Services &

Solution Consulting at Keynote. “Browsers are

now full-fledged application environments,

rendering, executing, initializing, and

transacting every day.” Even though there are a

whole string of opportunities for performance

to degrade between the SaaS data center and

the enterprise user, what happens in the

browser may have the biggest impact.

“Researchers have found that 75

percent of the performance problems with

Web applications actually happen in the

browser,” Agarwal says. “Not in the data center,

not in the database, not in the network or

anywhere in the back end, but in the browser.

And so the way you architect the user interface

with JavaScript and all the other browser

technologies is really what is going to impact

the performance of the service.”

And then there’s the 25 percent of

opportunities for something to go wrong that

are not browser-related — CDNs, domain

name servers, routers, load balancers,

client-side scripts, browser add-ons, etc.

Understanding where along the line

performance is compromised is the key to

delivering applications that will satisfy and

retain users.

MEASURE YOUR WAY TO SUCCESSIf performance is the key to success,

measurement is the key to performance. For

both SaaS vendors and enterprise customers,

the combination of active performance

monitoring and robust, enforceable service

level agreements is essential for successful,

profitable operations.

SaaS vendors will keep their

customers happy with proactive performance

management and, if written into SLAs as

standard operating procedure, will have a

distinct marketing advantage over those who

do not. Additionally, a clear understanding of

where and when slowdowns occur can help

keep their partners honest — if the bottleneck

is with one of their vendors — and also protect

and maximize their return on investment. For

example, investment in additional servers or

network capacity may not be required, if the

problem can be pinpointed to the application-

browser interaction, a third-party component,

or at some other point along the path.

Performance visibility enables SaaS vendors to

invest their resources where they will reap the

highest return. And to get that kind of

visibility requires stepping outside the walls of

the data center.

“You have to understand how it’s

experienced as an end user,” Agarwal says.

“You have to monitor the true experience, look

at the transaction and look at where the

slowdown is. You have to see whether it’s a

data center problem, whether it’s the network,

or whether it’s just the way you built the

application that is now being accessed by your

customers through their browsers.”

“Your users are not sitting in a

data center,” Hughes adds. “Test from the

Internet, test with a browser, test with what

your users use.”

TURNING INCREDIBLE PROMISE INTO CREDIBLE REALITYNew technology adoption is always a push-pull

dance. Enterprises want the productivity and

competitive advantage, but are understandably

cautious until the risks are understood and

performance proven. New technology vendors

want to get out ahead of their competitors and

gain preemptive market share quickly,

sometimes before all the kinks are worked out.

For both sides, it boils down to credibility.

4

Keynote Benchmark

Performance degradation can happen at any point in the circuitous journey a SaaS application and its data take to get to the user’s desktop. That’s why end-to-end, internal and external testing are so critical to ensure optimum performance.

5

Keynote Benchmark

© 2010 Keynote Systems, Inc. All rights reserved. The trademarks of Keynote Systems, Inc. include Keynote®, DataPulse®, CustomerScope®, Keynote CE Rankings®, Keynote Customer Experience Rankings®, Perspective®, Keynote Red Alert®, Keynote Traffic Perspective®, Keynote WebEffective®, The Internet Performance Authority®, MyKeynote® , SIGOS®, SITE®, keynote™, The Mobile & Internet Performance Authority™ and all related trademarks, trade names, logos, characters, design and trade dress are trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries and may not be used without written permission.

.............................................................................................................................................................................

FOOTNOTES

i. Forrester Blogs, “CIOs: Develop A Technology Watch List,” by Sharyn Leaver, 11/6/09

ii. Gartner, “Market Trends: Software as a Service, Worldwide, 2007-2012,” Sharon A. Mertz and others, 9/12/08

iii. Information Week/Plug Into the Cloud blog, “Piper Jaffray Sees Gold Rush In Cloud Software,” by Charles Babcock, 2/23/10

iv. Gartner webinar, “Ensure Your SaaS, Cloud Investment Will Deliver Results and Save You Money,” 4/2010

v. Trumba, “White Paper: Five Benefits of Software as a Service,” 3/8/07

vi. Forrester Research, “The ROI Of Software-As-A-Service,” by Liz Herbert and Jon Erickson, 7/13/09

vii. Gartner webinar, “Ensure Your SaaS, Cloud Investment Will Deliver Results and Save You Money,” 4/2010

viii. Knowledge Infusion Center of Excellence Research, “The Continuous Innovation Advantage of Software-As-A-Service,” by Jason Corsello, 10/2009

Some SaaS vendors are proactively putting

their performance stake in the ground and

backing it up with guarantees. RightNow, a

CRM application provider, starts giving money

back to clients if up-time drops below 99.9

percent. Intacct, which offers financial

management and accounting applications,

offers a similar guarantee, and publishes their

uptime stats right on their Web site.

“Web performance monitoring helps

you to protect your bottom line, but also to

build your brand and build customer loyalty,”

says Neeraja Rasmussen, Keynote senior

marketing manager for Web Performance,

about SaaS vendors. “You can guarantee that

you can deliver a certain level of service.

Verified third-party data with high levels of

trust is a definite competitive advantage for

SaaS providers.” Such data is the heart of

effective, enforceable SLAs. And in a nascent

SaaS marketplace, solid SLAs are imperative

for credibility.

A KEY COMPONENT OF THE CONNECTED FUTUREThe lines are increasingly blurred between

what happens on our computers, and our

phones, and our TVs, and what happens in the

cloud. What’s important is getting the tasks

done efficiently and effectively. SaaS will no

doubt play a bigger and bigger role in

accomplishing that end, though we eventually

may stop making that distinction. It will all

just be “computing.”