Testing Cloud Services: SaaS, PaaSand IaaS€¦ · Testing Cloud Services: SaaS, PaaSand IaaS...

Testing Cloud Services: SaaS, PaaS and IaaS 5-5-2014

© Polteq 1

Testing Cloud Services: SaaS, PaaS and IaaS

Martin Pol

Jeroen Mengerink

Agenda

• Introduction Cloud computing

• Challenges Risks

• Solutions Test measures


© Polteq 2

Objectives

• Learn how to

– Cope with Cloud services

– Detect the (new) risks

– Mitigate these risks

By using

• Your skills

• Heuristics

• The Book

• Discussions and exchange experiences

In the cloud?


© Polteq 3

What can be done in the cloud?

searching, recording, accounting, paying, writing,reviewing, tracking, calculating, developing, listening,analyzing, transmitting, learning, controlling,purchasing, testing, alarming, changing, updating,deleting, accessing, rejecting, correcting, studying,booking, receiving, tracing, protecting, deciding,managing, teaching, facilitating, identifying, copying,removing, demonstrating, checking, showing,selecting, subscribing, unsubscribing, sharing,mailing, communicating, reading, playing, working,meeting, gambling, shopping, storing, crosschecking, retrieving, configuring, sketching, saving,accelerating, enhancing, creating, growing, checkingin, checking out, finding out, reaching, denying,talking, designing, making, verifying, measuring


© Polteq 4

EmailSurfTransfer

Develop and Test

Operate and Manage Store

EmailSurfTransfer

Develop and Test


storage claim

80% unused

redundancy limitations

environmentally unfriendly

management overheadcosts for innovation


© Polteq 5

EmailSurfTransfer

Develop and Test


standard software bandwidth

internet technologySOA

virtualization

searching, recording, accounting, paying, writing,reviewing, tracking, calculating, developing, listening,analyzing, transmitting, learning, controlling,purchasing, testing, alarming, changing, updating,deleting, accessing, rejecting, correcting, studying,booking, receiving, tracing, protecting, deciding,managing, teaching, facilitating, identifying, copying,removing, demonstrating, checking, showing,selecting, subscribing, unsubscribing, sharing,mailing, communicating, reading, playing, working,meeting, gambling, shopping, storing, crosschecking, retrieving, configuring, sketching, saving,accelerating, enhancing, creating, growing, checkingin, checking out, finding out, reaching, denying,talking, designing, making, verifying, measuring

EmailSurfTransfer

Develop and Test



© Polteq 6

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

On-demand service

Self service provisioning, pay-per-use

No human interaction


© Polteq 7



On-demand service

Broad network access

Standard mechanisms over networks

“Any” client



On-demand service


Resource pooling

Multi-tenant

Storage, processing, memory, virtual machines, …

Location independent


© Polteq 8



On-demand service


Resource pooling

Rapid elasticity

Rapid scale in and out

“Any quantity” at any time



On-demand service


Resource pooling

Rapid elasticity

Measured service

Controlled resource use

Transparency, pay-per-use


© Polteq 9

Service models

• Nocloud

• Infrastructure as a Service

• Platform as a Service

• Software as a Service

Application

Platform

Virtualization

Hardware

CloudInternal

Deployment models

• Public

• Private

• Community

• Hybrid


© Polteq 10



On-demand service


Resource pooling

Rapid elasticity

Measured service

Deployment models

– private cloud

– community cloud

– public cloud

– hybrid cloud

Service Models

Software as a Service

Platform as a Service

Infrastructure as a Service

What is “done” in the cloud?

>500

PrivateHybrideCommunity

IaaS, PaaS, DaaS, SaaS

Taas

*aaS

Data CentreData Management

Business processes

Consumer

Public

SaaS

Surf and mailAppsSocial mediaDropboxGoogle servicesSpotifyPicasaGames……………

<500 employees

Public

*aaS

MailStorage

Infrastructure

CRM

Finance

Business processes


© Polteq 11

ISBN 978-1-937538-38-5

Theoretical Basis

What is in the book?

1. What is cloud computing?2. The changing role of the test manager3. What to test?4. How to test?

Page xx

What do you see as the two main risks of involving the cloud?


© Polteq 12

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards

143143

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

StandardsPerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRis

ks

Ris

ks


© Polteq 13

PerformancePerformance

SecuritySecurity






ks

Ris

ks


SecuritySecurity






ks

Ris

ks


© Polteq 14


SecuritySecurity






ks

Ris

ks

Page 37

Other customers

YOUROperational Profile



PLUS


PLUS


SecuritySecurity






ks

Ris

ks


© Polteq 15


SecuritySecurity






ks

Ris

ks


SecuritySecurity






ks

Ris

ks

Page 40


© Polteq 16

Everything over the web

The idea:“it’s safe”The idea:“it’s safe”

Home ground for hackers

Home ground for hackers


SecuritySecurity



ManageabilityManageability



ks

Ris

ks


SecuritySecurity






ks

Ris

ks

Page 40


© Polteq 17

Bring Your Own Device

No free choice of device.

No free choice of device.

Endlesspossibilities.

Endlesspossibilities.


SecuritySecurity






ks

Ris

ks


SecuritySecurity






ks

Ris

ks


© Polteq 18


SecuritySecurity






ks

Ris

ks

Page 42


SecuritySecurity






ks

Ris

ks

Internet connection lost@ supplier

@ user

@ other systems

‘Off line” does not work

Information is lost


© Polteq 19


SecuritySecurity






ks

Ris

ks


SecuritySecurity






ks

Ris

ks


© Polteq 20


SecuritySecurity






ks

Ris

ks

Page 44


SecuritySecurity






ks

Ris

ks

Mismatchservice <> business process

Functionality is changed

Insufficient usability


© Polteq 21


SecuritySecurity






ks

Ris

ks


SecuritySecurity






ks

Ris

ks

Page 46


© Polteq 22

Backup and recovery

Taken care of.Taken care of.

Who will support me?

Who will support me?


SecuritySecurity






ks

Ris

ks


SecuritySecurity






ks

Ris

ks

Page 46


© Polteq 23

Updates, patches, fixes, …

Planned and controlled

Planned and controlled

Do I have a choice?

Do I have a choice?


SecuritySecurity






ks

Ris

ks


SecuritySecurity




Legislation & regulationsLegislation & regulations


ks

Ris

ks


© Polteq 24


SecuritySecurity






ks

Ris

ks

Where is my data?

And is that OK?

In house.In house.

Somewhere…Somewhere…


SecuritySecurity






ks

Ris

ks


© Polteq 25


SecuritySecurity





Suppliers & outsourcingSuppliers & outsourcingRis

ks

Ris

ks


SecuritySecurity






ks

Ris

ks

Page 49


© Polteq 26


SecuritySecurity






ks

Ris

ks

Vendor lock in

No agreements

Supplier of the supplier of the supplier …

Supplier is taken over


SecuritySecurity






ks

Ris

ks


© Polteq 27

Testing?

Check

Review

Monitor

Interview

Proof of concept

Testing!

Check

Review

Monitor

Interview

Proof of conceptTestenProef

IntakeInterviewProof of concept


© Polteq 28

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity TestingAvailability & Continuity Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due toLegislation & RegulationsTesting due toLegislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during SelectionT

es

t Me

as

ure

sT

es

t Me

as

ure

s

TestenProefIntake

InterviewProof of concept

Selection

Implementation

Production









Testing during SelectionTesting during Selection

Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 29










es

t Me

as

ure

sT

es

t Me

as

ure

s


SecuritySecurity






ks

Ris

ks










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 30


SecuritySecurity






ks

Ris

ks










es

t Me

as

ure

sT

es

t Me

as

ure

s


SecuritySecurity






ks

Ris

ks










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 31










es

t Me

as

ure

sT

es

t Me

as

ure

sPerformancePerformance

SecuritySecurity






ks

Ris

ks

Architecture

From “individual” risksto

“individual” test measures

Architecture

From “individual” risksto

“individual” test measures

Page 47


© Polteq 32










es

t Me

as

ure

sT

es

t Me

as

ure


SecuritySecurity






ks

Ris

ks

Selection Criteria










Te

st M

ea

su

res

Te

st M

ea

su

res

CompletenessControllableFor serviceFor supplier

Spec’s and termsReferences……


© Polteq 33

How do you select the service?

Page 61-64


© Polteq 34

“Inspiration List”

CRITERION PRIOFunctionalDo the service and the specific business processes align?Does the service fit well in the E2E business process?Is the service sufficiently adaptable to specific requirements?Are many adjustments needed?Is customization possibleIs (a lot of) customization needed?Are the required platforms supported?Are “het nieuwe werken” and BYOD supported sufficiently?Is it possible to connect / integrate the service with the other systems?Are sufficient manuals and/or courses available?ImplementationIs the impact on current activities acceptable?Is a feasible route for migration towards the service available?


CRITERION PRIOSupportAre changes in the service announced beforehand?Are sufficient test facilities available around the service (test environment, test tooling, testware, access to infrastructure, …)?Are there sufficient support facilities?Is it clear how incidents can be reported?Are incidents resolved fast enough?PerformanceAre response times low enough?Is the number of possible simultaneous users high enough?Is bandwidth sufficient?Is sufficient potential for growth available?Is the actual use charged correctly?


© Polteq 35


CRITERION PRIOSecurityAre adequate authorization and authentication possibilities in place?Is the physical security of the service locations sufficient?Is the support access security of the service sufficient?Is mutual access security between customers sufficient?Are data changes traceable?Is data storage for the service reliable?Is deleting data in the service reliable?Is security of the connection to the service sufficient?Are security options for the customer sufficient?Does the supplier have security certificates? (for example SAS 70 type II)?AvailabilityIs the level of availability for the service sufficient?Are back-up / fail-over / disaster-recovery provisions sufficient?


CRITERION PRIOLaw and regulationsDoes the data location comply to all legal requirements?Does the data processing comply to all legal requirements?Do the terms contain parts that are conflicting to the duties of the customer?SupplierIs clear what happens when the contract ends, or in case of bankruptcy or conflict?Is a good helpdesk available?Does the supplier have experience in:- Offering this particular service?- Offering services in general?- Developing services?- The customer’s field?- Developing, testing and supporting services (know how)?Do methods used by supplier align with those of the customer (if relevant)?


© Polteq 36


CRITERION PRIOSupplierIs quality assurance arranged?Is the supplier ahead in its field?Is the size of the supplier in accordance with the expectations of the customer?Does the supplier have a good reputation (are there references)?Is providing services the core business of the supplier?Does the supplier have opportunities for future expansion?Does the supplier speak the same language?Is transparency and flexibility of the supplier sufficient?

Proof of Concept










Te

st M

ea

su

res

Te

st M

ea

su

res

Dynamic testingMore suppliersTime boxingRepresentative


© Polteq 37










es

t Me

as

ure

sT

es

t Me

as

ure


SecuritySecurity






ks

Ris

ks

Known measurestuned and tweaked

New measures developed

Use:

– Your skills

– Heuristics

– The Book

– Your peers







Testing due to Legislation & RegulationsTesting due to Legislation & Regulations



Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 38

Suppose your payroll system moves into the Cloud.

What would you do?

Load Testing




PLUSACTUAL MOMENT


PLUSACTUAL MOMENT










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 39

Operational profile

Page 73

Performance testing

• Test cases aimed at specific bottlenecks

• Including cloud aspectsin test cases

• Test setup for a

performance test

• Representative?


© Polteq 40

Stress Testing

Yes, you can!Yes, you can!

Definitely NOT!Definitely NOT!










es

t Me

as

ure

sT

es

t Me

as

ure

s

Elasticity

Load and stress.Load and stress.

Load and elasticity.Load and elasticity.










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 41

How would you test elasticity?

load

load test – ‘up’

extend?

200 charged

100charged

no

yes

path test

99

100

101

boundary values

‘up’tc 1: use=99, pay 100tc 2: use=100, pay 100tc 3: use=101, pay 200

‘down’tc1: use=101, pay 200tc2: use=100, pay 100tc3: use=99, pay 100

boundary values

load test – ‘down’


© Polteq 42

load

load test – ‘up’

extend?

200 charged

100charged

no

yes

path test

99

100

101

boundary values

‘up’tc 1: use=99, pay 100tc 2: use=100, pay 100tc 3: use=101, pay 200

‘down’tc1: use=101, pay 200tc2: use=100, pay 100tc3: use=99, pay 100

boundary values

load test – ‘down’

• (Automatic) scaling up or down does not perform as required

• At scaling moments functional problems emerge

• Insight in use based costs is not sufficient

ISO 27001 aspects:

• Confidentiality of the data and the accompanying risk that unauthorized people can view the data

• Integrity of data and the accompanying risk that data is altered or lost unintentionally

• Availability of data and the accompanying risk that data (and services) is not available when it is required










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 43

ISO 27001 aspects:

• Confidentiality of the data and the accompanying risk that unauthorized people can view the data

• Integrity of data and the accompanying risk that data is altered or lost unintentionally

• Availability of data and the accompanying risk that data (and services) is not available when it is required










es

t Me

as

ure

sT

es

t Me

as

ure

s

• Who has access to the data?

• Can the user trust that the data is correct?

• Can the user gain access to the data at all times?

• Security at:

– Network

– Supplier

– User










Te

st M

ea

su

res

Te

st M

ea

su

resTesting security robustness against Internet

attacks - Directory traversal. Read and/or write in

directories other than those allowed.- XML external entity attack. Include extra

(bad) data in an XML file.- SQL injection. Request and/or change data

by manipulating SQL queries. - Cross-site scripting (XSS). Transfer data to

other websites without the user knowing.- Session manipulation. Skip steps or

validation in a session.


© Polteq 44

• Security at:

– Network

– Supplier

– User

• Encryption

• Authentication and authorisation










es

t Me

as

ure

sT

es

t Me

as

ure

s

IDaaS

• Security at:

– Network

– Supplier

– User

• Encryption

• Authentication and authorisation

• Test logs and audit trails

• Security Audits










Te

st M

ea

su

res

Te

st M

ea

su

res

IDaaS

Experts

Security patch routines


© Polteq 45

• Completeness and correctness of specifications and manuals

– Supplier

– User

• Availability of test environments










es

t Me

as

ure

sT

es

t Me

as

ure

s

Interface specifications

Supported platforms

Business process specs

User manuals

How do you setup your environment including the

external service?


© Polteq 46

Manageablity of test environments

• Everything in the cloud


• Link all current environments to the service


© Polteq 47


• Link Production to the real service

• Link other environments to a MOCK SERVICE(or another instance of the service)


– Supplier

– User


• Management of:

– Defects

– Changes










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 48

Defect Management

Incident

Supplier resolves it

Client resolves it

Incident not resolved

Test

Change work process

Change configuration

Custom solution

Service not selected

Terminate use of service

Workaround work instruction

Test

Test

Test

Test

Migrate and test


– Supplier

– User


• Management of:

– Defects

– Changes

• Maintainability of the software










Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 49

• Role of system architecture

• Monitoring and Logging

• Guarantees and SLA’s

• Test fail-over mechanism

• Test online/offline










es

t Me

as

ure

sT

es

t Me

as

ure

s

Fail-over testing

A: disrupted

B: active

A: active

B: inactive

A is disrupted

B takes over service

A: inactive

B: active

disru

ptio

nin

A e

nded

no ch

ange

A is d

isrupte

d

no ch

ange

A: active

B: disruptedB is disrupted

A takes over service

B is

dis

rupte

dno c

hange

dis

ruptio

nin

B e

nded

no c

hange A: disrupted

B: disrupted


© Polteq 50

Online – Offline

Use case testing.Global testing.











es

t Me

as

ure

sT

es

t Me

as

ure

s

Online – Offline












Te

st M

ea

su

res

Te

st M

ea

su

res

Off line tests focussed on problems:

• Work continues, based on out-of-date information, and

this information could be changed in the cloud during the

offline period.

• The users are not aware that they are working (partly)

online (and are lead to believe differently).*

• Synchronization conflicts arise because data is changed

locally as well as in the cloud.


© Polteq 51

Online – Offline












es

t Me

as

ure

sT

es

t Me

as

ure

s

Off line test cases:

• End the connection and check whether the users can see

that they are working offline.

• Disrupt the connection (for instance, a port or a certain

type of IP traffic) and check whether problems arise.

• Check whether changes that are made offline find their

way to the cloud when online status is regained.

• Check whether conflicts between offline and cloud data

are handled robustly (which is in fact a functional

requirement).







Testing caused byLegislation & RegulationsTesting caused byLegislation & Regulations



Te

st M

ea

su

res

Te

st M

ea

su

res


© Polteq 52

How does functional testing change with services?

Functional test objectives

• Does the service fit the business processes and vv?

• Is the service quality sufficient (number of bugs)?

• Is the service sufficiently user friendly?

• Is the service configuration done correctly?

• Does supplier customization function properly?

• Does customer customization function properly?

• Do interfaces work properly?

• Are platforms properly supported?

• Does everything work after changes (is there no regression)?


© Polteq 53

Any device – any platform

Multiplatform testing.













es

t Me

as

ure

sT

es

t Me

as

ure

s

3997 distinct Android devices

http://opensignal.com/reports/fragmentation.php


© Polteq 54

Internet Explorer 6

Internet Explorer 7

Internet Explorer 8

Firefox 3.5

Firefox 3.6

Firefox 4

Safari 4

Safari 5

Chrome11

Opera11

Windows XP

Windows Vista

Windows 7

Windows 2003 Server

Windows 8

Windows CE

Linux

Unix

Mac OS Lion

Mac OS Snow Leopard

iOS

Android

Operating systems

Browsers

Multi-platform testing

Devices

Computer

Mobile phones

Tablet

PC

Macintosh

SUN

NOKIA …

Samsung …Windows Mobile

iPhone ...

…

MOTOROLA …

Blackberry …

ASUS ...

…

Internet Explorer 6

Internet Explorer 7

Internet Explorer 8

Firefox 3.5

Firefox 3.6

Firefox 4

Safari 4

Safari 5

Chrome11

Opera11

Windows XP

Windows Vista

Windows 7

Windows 2003 Server

Windows 8

Windows CE

Linux

Unix

Mac OS Lion

Mac OS Snow Leopard

iOS

Android

Operating systems

Browsers

Multi-platform testing

Devices

Computer

Mobile phones

Tablet

PC

Macintosh

SUN

NOKIA …

Samsung …Windows Mobile

iPhone ...

…

MOTOROLA …

Blackberry …

ASUS ...

…


© Polteq 55

Any device – any platform














es

t Me

as

ure

sT

es

t Me

as

ure

s










Te

st M

ea

su

res

Te

st M

ea

su

res

Scenarios

• Transfer into the cloud, applications remain the same– data moved to another

location

• Transfer to SaaS– data migrated to new service

• Transfer from one to another SaaS– similar

• Transfer out of the cloud.– similar


© Polteq 56

Data conversion

• Testing conversion rules

• Testing conversion on input data

• Testing if any data is lost

• Testing ongoing transactions

Existingsystems

Extraction Conversion Import

Conversion software

Service

• Rounding (totals incorrect)• Field lengths (truncation)• Totals (information lost)• Date and time conversions

what is 08-09-11?

• Audit trail, check sums

• E2E business scenario’s

Other aspects

• Cleaning data defects

– solved before migration

– no problems during migration

• Testing security aspects

– during and after migration

– not TOO much data migrated

• Testing performance

– speed (how long does it take?)

– volume (capacity sufficient?)

– stability at full volume


© Polteq 57

Example: email to the cloud

• Tools migrate existing emails to the cloud

• Low risk:

– migrating one or some mailboxes and executing a limited testing

– if successful: implementation for all mail boxes

• High risk:

– no emails lost in migration?

– formatting of the emails still correct?

– all attachments still there?

– all attributes migrated (priorities, timestamps, flags, …)?

Legal importance of email

reading, forwarding, replying, check on contents

Legislation + Regulations

=

Test basis

Incidental testing.Incidental testing.

Compliancy testing.Compliancy testing.










Te

st M

ea

su

res

Te

st M

ea

su

res

Sarbanes Oxley


© Polteq 58

Where is my data stored?

– nothing, or hardly anything, to be found on this subject

– service stores data outside the borders of permitted countries additional measures?

– service stores data within the borders of permitted counties okay

data owner is responsible for ensuring that the protection of personal data is at the required level wherever it is held

Checking for legislation and regulations

• List where data that is stored in the cloud

• Find the requirements that are applicable to this data

• Check supplier terms with customer’s requirements

• Perform (external) audit for high risk

• Test manager provides advice, management decides

Legal support needed for high risk

Example. A supplier of a storage service claims to be the owner of the intellectual capital of all data stored at their facilities. It is highly unlikely that this is compatible with the interests of the organization that is the actual owner of the data.


© Polteq 59










es

t Me

as

ure

sT

es

t Me

as

ure

sLegal issues – threats

Example: Dropbox

Compliance with Laws and Law Enforcement Requests; Protection of Dropbox's Rights.

• We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of Dropbox or its users; or (d) to protect Dropbox’s property rights. If we provide your Dropboxfiles to a law enforcement agency as set forth above, we will remove Dropbox’s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.


© Polteq 60

Is testing in production necessary?










Te

st M

ea

su

res

Te

st M

ea

su

res


SecuritySecurity






ks

Ris

ks


© Polteq 61










es

t Me

as

ure

sT

es

t Me

as

ure

s

ContinuousEnd-to-End Testing

Continuous Change

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards

Check

Intake

Monitor

Interview

Proof of concept


© Polteq 62










es

t Me

as

ure

sT

es

t Me

as

ure


SecuritySecurity






ks

Ris

ks

Testing starts early: in selectionScope of testing is widened

Testing continues in production

Testing starts early: in selectionScope of testing is widened

Testing continues in production










Te

st M

ea

su

res

Te

st M

ea

su

res


SecuritySecurity






ks

Ris

ks

Questions?Questions?

www.polteq.com

[email protected]@polteq.com


© Polteq 63










es

t Me

as

ure

sT

es

t Me

as

ure


SecuritySecurity






ks

Ris

ks

Thank you!Thank you!

www.polteq.com

[email protected]@polteq.com

Testing Cloud Services: SaaS, PaaSand IaaS€¦ · Testing Cloud Services: SaaS, PaaSand IaaS...

Documents

Transcript of Testing Cloud Services: SaaS, PaaSand IaaS€¦ · Testing Cloud Services: SaaS, PaaSand IaaS...