Post on 30-Jun-2015
Partners Testing ProductsPre-Sales, Sales, & Post Sales Support
2
Requirement Management Solution
Optimal Trace
Raven Flow
Functional Testing Tools
TestPartner
QA Director
Silk Test
Performance Testing Tools
QALoad
Web Load
Silk Performer
Development and Debugging Tools
DevPartner Studio
DevPartner Java
Bounds checker
Security checker
IT Service Management Products
Vantage
Vantage for Network Performance
Vantage for Java/.NET Performance
Vantage for Server Performance
IonIdea -Testing Services Offerings
3
Ion - Virtual Testing Team
Functional Testing Automation Services
Performance Testing Services
Software Testing Automation Consulting
Scripted Functional Testing Services
Software Testing outsourced
Training – Test Automation
Delivering High Performing ApplicationsWhat’s Needed: Load Testing with Infrastructure Monitoring
4
Virtual Users
Web Servers Application Servers
DBMS
Mainframe
Combining load testing with infrastructure monitoring:
• Provides realistic emulation of transaction and traffic usage patterns to:
• Ensure servers will scale to meet current and future
needs
• Correlate important performance relationships
between all components of the application
infrastructure
Delivering High Performing ApplicationsWhat’s Needed: End-to-End Analysis of All Application Components
5
Virtual Users
Web Servers Application ServersDBMS
Mainframe
JSP-ASP
Method
Method
Method
Method
Method
URL JSP-ASP
Method
Method
Method
Web Service
Web Service
JDBC-ODBC
MQ
End-to-end performance analysis:
• Provides the depth of analysis necessary to pinpoint performance
problems
7
Why Load Testing is Not Enough
NOT ENOUGH INFORMATION
• Only delivers general response time, throughput or server metrics
• Does not identify where bottlenecks are, across environment or inside application
• Doesn’t get to the root cause of the problem
• Leads to finger-pointing
NOT TIMELY
• You have to wait until after load testing to understand whether you have a problem
RESULT…
• Missed delivery dates
• Poor-quality applications
• High-end resources involved in resolving problems and waiting until the end of testing
• Costly/unnecessary infrastructure changes to fix problems
8
PREDICTION:
• Predict performance under varying conditions
• Identify impact of network on application from multiple locations
• Pinpoint bottlenecks across application tiers
• Fix code prior to conducting load testing
TROUBLESHOOTING:
• Deeper analysis during load test
• Pinpoint Java and .NET performance problems
• Perform fewer application retests
What Sets Performance Triage apart?
9
Load Testing only…
PT Prediction & Troubleshooting…
Performance Triage (PT)
10
Re-active Leverage profiling to deal with performance issues before
load testing
Helps to make best use of limited testing timeframes
Performance Triage – Two mixes
Pro-active Well structured approach to performance testing to prevent
the ‘hair on fire’ scenarios
12
Setting the scene• Online banking arm of large corporate finance house
• Urgent requirement to validate existing infrastructure capacity and to investigate capacity to handle further growth
• Limited time to execute
PT in Action - Case StudyIntroduction
13
PT in Action - Case StudyPerformance Goals
• Response Time SLA• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Based on broadband bandwidth
• Concurrency SLA• Support 600 concurrent end users
• Server Utilization SLA• Server utilization < 50% measured as CPU, Memory and Disk I/O
14
PT in Action – Case StudyLoad testing alone won’t identify hidden problems
Bank Data Centre
Web Servers
App Servers
DB Servers
External Users
Internal Users
WAN Sensitivity
Bad SQL
Slow Methods
Insufficient Capacity
Contention Issues
15
• Enrollment
• Signon
• View account summary
• View account details
• Intra FI transfer
• View check images
PT in Action - Case Study
Identify and profile key transactions
Profile
WAN Sensitivity Bad SQL
16
PT in Action – Case Study
Profile – Predicting WAN sensitivity
Increase in response time of 11 seconds when connecting over T1 link with 50ms latency
17
PT in Action – Case Study
Profile – Bad SQL performance
SQL call taking in excess of 13 seconds to complete
18
PT in Action – Case StudyProfiling identifies hidden problems
Bank Data Centre
Web Servers
App Servers
DB Servers
External Users
Internal Users
WAN Sensitivity
Bad SQL
Slow Methods
Insufficient Capacity
Contention Issues
19
PT in Action – Case Study
Load Testing
Load Test
Contention Issues
Insufficient Capacity
• Performance goals• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Support 600 concurrent end users
• Server utilization < 50% measured as CPU, Memory and Disk I/O
Slow Methods
20
PT in Action – Case StudyLoad Testing – Transaction performance
Concurrent users
Elapsed timeTransaction response time
Concurrent SLA: 600 users
Response time SLA: < 20 seconds
Transaction performance exceeds response time SLA
FAIL!! Response Time
21
PT in Action – Case StudyLoad Testing – Server performance
Concurrent users
Elapsed time
Web server CPU %
Concurrent SLA: 600 users
Server CPU SLA: < 50 %
Web server CPU utilization breaches SLA
FAIL!! Insufficient Capacity
22
PT in Action – Case StudyLoad Testing – Application server
Analysis inside the JVM and CLR
Slow performing method impacting response time and web server CPU
FAIL!! Slow Method
23
PT in Action – Case StudyLoad Testing – Application server
Analysis inside the JVM and CLR
Memory not being released
FAIL!! Memory Leak
24
PT in Action – Case StudyLoad Testing – Transaction performance
Concurrent users
Elapsed timeTransaction response time
Concurrent SLA: 600 users
Response time SLA: < 20 seconds
Transaction performance remains below response time SLA
SUCCESS!!
25
PT in Action – Case StudyLoad Testing – Server performance
Concurrent users
Elapsed time
Web server CPU %
Concurrent SLA: 600 users
Server CPU SLA: < 50 %
Web server CPU utilization remains below SLA
SUCCESS!!
26
PT in Action – Case Study
Profiling + Load Testing = All problems identified
Bank Data Centre
Web Servers
App Servers
DB Servers
External Users
Internal Users
WAN Sensitivity
Bad SQL
Slow Methods
Insufficient Capacity
Contention Issues
27
PT in Action – Case StudyDeliverables
PASS!! • Response Time SLA• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Based on broadband bandwidth
• Concurrency SLA• Support 600 concurrent end users
• Server Utilization SLA• Server utilization < 50% measured as CPU, Memory and Disk I/O
PASS!!
PASS!!
28
PT – ‘Pro-active’ remix
PT – Becoming pro-activeAvoiding the ‘Hair on Fire’ scenario
29
PT ‘Pro-active remix’ – Mission Statement
• Performance considerations should be factored into the application lifecycle at the earliest opportunity
31
PT ‘Pro-active remix’ – Requirements capturePerformance goals
• Response Time SLA• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• Transaction response time less than 20 seconds
• Based on broadband bandwidth
• Application Concurrency SLA• 600 concurrent end users
• Server Resource Utilization SLA• Less than 50% measured as CPU, Memory, Network Utilization and Disk I/O
32
PT ‘Pro-active remix’ – Requirements captureTest environment
Compare production to test system and identify differences
33
• Enrollment
• Signon
• View account summary
• View account details
• Intra FI transfer
• View check images
• Copy check
• Bill Payment Transaction
• Check reorder
PT ‘Pro-active remix’ – Requirements captureIdentify key transactions
34
PT ‘Pro-active remix’ – Requirements captureTransaction detail + Data
View Account Summary
• Home Page <- Target URL
• Logon Page <- Login credentials
• Verification Page
• Challenge Page
• Retrieving default account summary
• Retrieving maximum amount of account history (90 days max)
• Logoff
35
PT ‘Pro-active remix’ – Requirements captureIdentify data requirements
• A minimum of 100 sets of unique end user login credentials
• Test database to be a recent cut of live database
• Each consumer banking logon account should contain:
• 2-3 different accounts. (If multiple hosts are accessed during a logon, additional considerations may be warranted).
• Several items of history in one of the accounts – generally a direct draw/checking account. Best results are obtained when this transaction history contains 20 or more items per month distributed over 90 days.
36
PT ‘Pro-active remix’ – Requirements captureVirtual user allocation
Allocation of virtual users to transactions
37
PT ‘Pro-active remix’ – Requirements captureSummary
Critical requirements identified
Performance goals ( SLA’s )
Validation of performance test environment
Data requirements identified
Key transactions identified
Virtual user allocation
© 2008 Compuware Corporation — All Rights Reserved
IonIdea38 – 40 EPIP White Field
Bangalore - 560066
Mohammed Ibrahim, Manager – Testing Practice
Works - +91-80-66581577
Mobile - +91-9986650636
Cell(US)- (703) 268-2920
E-mail: mohammed.ibrahim@ionidea.com
www.ionidea.com
Contact