NF Risk Assessment Framework Increasing Predictability … Risk Assessment Framework ... o Fault...
18
NF Risk Assessment Framework – Increasing Predictability of Non-Functional Defects 2014
Transcript of NF Risk Assessment Framework Increasing Predictability … Risk Assessment Framework ... o Fault...
NF Risk Assessment Framework –
Increasing Predictability of Non-Functional Defects
2014
Outline
2
Conventional approach to NFT
What is not NFT?
Why Risk Assessment?
NF Risk Assessment Framework
Framework Explained
Applicability & Benefits
Conventional Approach to Non-Functional Testing (NFT)
3
Also Release
Management
wants it
Application
should not
crash in
production !!
Will users get
the response
within 5
seconds?
I have got a
good 3 weeks
before go-live
Do we have
enough CPU
and
Memory?
Somebody’s
gotta do it
OK.
I know what to do.
Let’s do a Load Test
first and then
increase the number
of users to Stress
the system
Let’s gate crash the
Quality Gateway
Non-Functional Testing ??
Pitfalls of the Conventional Approach to NFT
4
Pre-production Production
o Am I focusing too much focus
on response time SLA only?
o Do I know what I am looking
for ?
o Am I ensuring adequate
coverage besides the happy
scenarios?
o Should I test integrated with
other applications or defer it to
the next release?
o May be I have taken a one
size fits all approach!
o Day 1 – so far so good!
o Day 2 – all was well until 3 pm
and then seemed kind of
sluggish
o Day 3 – Nothing serious but
‘system was unavailable’ for 15
min this morning (suspected due
to a Gateway outage the
previous night)
o Day 5 – 4 EoDs were great. But
EoW is a different story
o Day 10 – Not as quick as it was
last week (already aging?)
Well, what constitutes Non-functional Testing?
5
Non-functional Testing Performance Testing
Wikipedia
Baseline testing
Compliance testing
Documentation testing
Endurance testing
Load testing
Localization testing and
Internationalization testing
Performance testing
Recovery testing
Resilience testing
Security testing
Scalability testing
Stress testing
Usability testing
Volume testing
Software Quality characteristics as per
ISO 9126 Standard
Functionality
o Interoperability
o Security
Reliability
o Fault Tolerance
o Recoverability
Usability
Efficiency
o Performance
Maintainability
o Stability
Portability
o Adaptability
o Instability
Source: http://en.wikipedia.org/wiki/Non-functional_testing Source: http://www.sqa.net/iso9126.html
NFT – Where to start and where to end?
6
? How do I know what non-functional quality attributes to
test for?
? How do I know whether my test approach is good enough?
? How do I know how much testing is enough testing?
? How do I know what can possibly go wrong in production?
? How do I know my application is ready for production?
Non-Functional
Risk Assessment
Therefore Assess
7
System Appreciation
& Technical Assessment
Develop Risk Matrix
Create Risk Catalogue
Non – Functional Risk Assessment Framework
Risks – Tests Traceability
Start Risk Assessment
Step 1: System Appreciation and Technical Assessment
Study Product
Architecture
and Design
Understand Future
Deployment
and Workload
Characteristics
Architecture,
Protocols
Transaction Model
Interfaces
QoS
Data retention
Data replication
Overlapping
processes
Infra deployment view
Capacity, Network
HA and DR
Workload pattern
Volumetrics
Customer behavior
Business growth
Co-existing components
8
Historical
Non-Functional
Incidents Analysis
Incident Description
Technical Analysis
Non-Functional? (Y/N)
NF Domain
Class of Issue
Sub-Class
Potential way of
detecting the issue
Applicable NF Test
9
Step 2: Develop NF Risk Matrix
Risk Matrix = Threats Vs Focus Areas
“Threat” – technical attribute or event that can impact
the non-functional quality of the SUT
“Focus Area” – a component or set of functionalities
in the SUT that is critical to the non-functional quality of
the SUT
Dedicated
center of
excellence
providing
full-
fledged
performan
ce testing
solutions
10
Risk Matrix – Indicative Threats
o Processing Overlaps
o Concurrency
o Integration Complexity
o Network Latency
o VM Sharing
o JVM Sharing
o Database Sharing
o Large Volume Workloads
o Co-existence with Maintenance
o Vertical Scalability (lack of)
o Horizontal Scalability (lack of)
o Stress Conditions
o Prolonged Usage
o Large Volume Workloads
o Database Size
o Multi Geo Access
o Offline/Shutdown
o Incorrect Error Handling
o Zone App Crash
o VM Crash
o JVM Crash
o Database Crash
o Shared Resources
o Unconstrained Resource Usage
Dedicated
center of
excellence
providing
full-
fledged
performan
ce testing
solutions
11
Risk Matrix – Threats vs Focus Areas
Online
Transaction
Processing
(Transactional)
EOD
Processing
Adhoc
Reports
Interfaces
Processing
Global App
Behaviour
Zone
Behaviour
Infrastructure
Utilization
NF Domain Threat Infrastructure
Process ing Overlaps r p a r r p a
Concurrency p r g a p r a
Integration Complexi ty a a w p w r g
Network Latency r g a w P a r
Process ing Overlaps a a g g a p w
Stress Conditions a p a r r p a
Concurrency p a g r a a w
Prolonged Usage r a a a p p a
Concurrency a a r a r p a
Large Volume Workloads p a r a g r a
Database Size r a r w w a w
Multi Geo Access p w a w p a r
Offl ine / Shutdown r r a r r p w
Incorrect Error Handl ing a p g a a a w
Zone App Crash r r r r w r w
VM Crash a r g a a r w
JVM Crash a p g a a r w
Platform (Appserver/DB) Crash r r r r r r w
Non-Functional Focus Areas >>
Scalability
NFT Risk Matrix
Resilience &
Recoverability
Reliability
Performance
TI Processing TI Systems
Dedicated
center of
excellence
providing
full-
fledged
performan
ce testing
solutions
12
Risk Matrix – Risk Ranking
Domains of Concern 1 2 3 4 5 Total
Scalability 8 25 28 14 9 84
Reliability 6 4 13 3 2 28
Performance 4 7 11 1 5 28
Resilience and Recoverability 3 18 11 3 7 42
Capacity 0 4 7 3 0 14
Interoperability 1 1 3 0 2 7Compatibility 0 0 14 0 0 14
Total Count of Risks 22 59 87 24 25 217
Take-aways
NF Tests will be designed targeting each of the Rank1 and Rank2 Risks
These tests will also include Test Scenarios covering Rank3 Risks
Non Functional and Technical Risks Summary
Risk Ranking
Dedicated
center of
excellence
providing
full-
fledged
performan
ce testing
solutions
13
Step3: Create Risk Catalog
# 1 2
NF Domain Scalability Performance
Threat Processing Overlap Multi Geo Access
Sub-Threats
Intra Zone Processing -
Risk
Two or more Multi Bank Entities (MBE) within a Zone could be
performing different operations at the
same time, leveraging the same
application/OS/database resources
and processing the same data set or
accessing from the same data source (table/schema/database)
User sites are spread across the
globe, however all user access have
to pass through the Global Single
Sign On (SSO). There will be only one
primary instance of Global App in one
location and all users will be routed through this single Global app.
Impact
There will be intermittent delays in
online transaction processing (OLTP)
or delays in Message transmission into the Transport Client
User accesses from multiple
geographies to the global App and the
response therefore will potentially be
slow, influenced by the bandwidth
congestion over the wide area
network (WAN) between the user sites and the global app
Parameters to Measure
OLTP Response Time Global Dashboard Response Time
Focus Area Zone Behavior Global App (SSO, Dashboard)
Dedicated
center of
excellence
providing
full-
fledged
performan
ce testing
solutions
14
Risks – Tests Traceability
15
Study Product
Architecture and Design Historical Non-Functional
Incidents Analysis
Understand Future
Deployment and
Workload Characteristics
Non-Functional Risk Matrix
Non-Functional Focus/Impact Areas >>
Online Transaction Processing
(Transactional)
EOD Processing
Adhoc Reports
Interfaces Processing
Global App Behaviour
Zone Behaviour
Infrastructure Utilization
NF Domain NF Threat Processing Systems Infrastructure
Scalability Processing Overlaps r p a r r p a Horizontal Scalability (lack of) r a w a a r w
Reliability Processing Overlaps a a g g a p w Prolonged Usage r a a a p p a
NF Risk Catalogue
Risks – Tests Traceability
Non-Functional Risk Assessment Framework
Applicability
16
COTS providers/product development initiatives
Large IT initiatives (e.g. platform revamps)
Large infrastructure consolidations/transformations
X Stable/matured systems
X Periodic releases
X Minor enhancements
Benefits & Advantages
17
Gain a precise understanding of the technically
vulnerable areas of the SUT
Develop an exhaustive repository of non-functional test
scenarios
Ability to design tests to simulate the specific technical
risks
Ensure maximum possible coverage and traceability of
the NF risks in the SUT
Predictability into all probable outcomes in production
in the event of a technical failure or an unexpected workload
situation or projected business growth
Benefits:
Thank you!
Q ? A
Vijayanand Chelliahdhas
18
Benefits:
