Essence of Performance and Load Testing
April 30th , 2009Gihan B Madawala
Argus Technologies LtdCREO Products Inc
Chancery Software LtdMeridian Project Systems Ltd
Modular Mining Systems Canada Ltd.
Topics
Why do we need to test performanceMyths surrounding performance testingPerformance / Scalability testingSingle User Performance testingWorst-case scenario Performance bottlenecksPerformance and Scalability TargetsPerformance Testing LabPerformance Team
Topics Contd..
Microsoft Testing ToolsCommercial Performance Testing ToolsProcess of analyzing performance and
Scalability issuesTool selection strategyProblems with manual perf testingTool Evaluation ProcessSome Performance related
issues/solutionsBooks & Reference materials
Bad comments/Questions/Answers
It’s the first release of this feature so of course it is going to be slow !
It only takes couple of seconds so what is the big dealQ: Why is this page too slow ?A: It is because we have TWO users in the system
Q: How many users that we can support with this system?
A: 20 users per server
Good comments/questions/Answers
“ The technology of the product is based upon current solid architecture and easy to work with”
“The system is flexible to build upon and can be managed easily by our personnel.”
Product has the architecture required to function in large projects. It not only performs well but it scales
Q: How many users can we support with our system?
A: 400 users per server
Why do we need to test performance
Studies show performance was one of the most important qualities of a application
Experience shows Performance issues are the most likely to “bite with surprise” released into live operation
Lack of knowledge on Performance & Scalability testing
Myths surrounding Performance testing
We do not need to measure System Performance Our system is incredibly impressive with all the
features, users will not mind if it is little slow We do not really need precise measurement goals Performance testing has to be done late in the
project. Anyone can measure performance; it does not
require any skills or expertise Any test lab/tool is about the same as any other
one and you can mimic real life usage of the app
Performance testing should start early in the project and should be a on going process
Performance and testing practices
Operational Performance was Acceptable (48%)
Operational Performance was NOT Acceptable (52%)
Reviewed or stimulated performance during requirements and design phases
21% 0%
Tested early in development
35% 6%
Tested late in development
38% 26%
Did not do performance or load testing
6% 60%
Performance testing / Scalability Testing
The purpose of performance testing is to measure system’s performance for agreed requirements
Scalability testing (Load Testing, Stress testing) is measurement of performance under heavy load: Peak or worst-case conditions :-
Performance usually is measured as weighted mix of response time, throughput (Network Bandwidth) and availability
Response time – Measure of how long the system takes to complete a task or group of tasks
Single user performance testing
How important is to measure single user performance
Gives an early indication of system design feasibility
Can be measured with existing set of tools available
Cost of single user testing is low and can be done prior to multiple user testing
Worst-case scenario
If the test pass and scale well with worst-case we know for sure that the application will perform better for a lesser load.
Full data population is necessary Project Ex: (worst-case)
-Larger data packet size -More frequency data packets -More client threads
Performance and Scalability Targets
Achieve the specified Response time for one user
Maintain the response time within a specified range when the system is under load
Maximize the number of active users per available bandwidth
Minimize network bandwidth usage CPU usage of machines should be < 80% under heavy load
Performance Bottlenecks
Client CPU Database Server CPU Client, Server Memory Network Bandwidth Disk I/O Transaction processing speed Serialization and de-serialization speed Network latency Limitations of the 2nd/3rd party servers/add-ins
Performance Testing Lab
Simulation of realistic scenarioHaving adequate hardwareHaving feature rich testing toolsTest Automation and Unit testsPerformance testing expertiseCreating real test data
Performance Team
Performance Test EngineerSenior DeveloperDatabase SpecialistHardware EngineerNetwork Specialist
Microsoft Testing Tools
PERFMONSQL ProfilerACT (Application Centre Test)WAST (Web Application Stress Tool)TEAM TEST (Visual Studio Team
System)windebug /ddk
Some Industry Leading Performance and Scalability Tools
Load Runner (Mercury now HP) *Silk Performer (Segue now Boland) *QA LoadANTSOpen STA (Open Source)E-Load (Emprix)SHUNRASPIRENT Test Centre
Process of Analyzing Performance and Scalability Issues
Non Invasive action (Low hanging fruits)1 Analyzing of Performance counters2 Profiling3 Load testing tool analysis. Two steps
- Run concurrent users - Run specific number of users continuously
4 Simulate different network conditions5 Tracing and debugging .NET components6 Use of windebug /ddk
Invasive Action1.Re-factoring 2.Architectural change
Tool selection Strategy
# Areas to Test
GUI Automation (Feature Testing)
Single User performa
nceMultiple User
PerformanceStress/Stability
Testing
1 Web ClientsTestPartner/
TestComplete
Manual/Benchmark Tools/Visual Studio
LoadRunner/SilkPerformer/TestComplete
LoadRunner/SilkPerformer/TestComplete
2 Office clientTestPartner/
TestComplete
Manual/Benchmark Tools/Visual Studio
LoadRunner/SilkPerformer
LoadRunner/SilkPerformer
3
Central server –
DB serverTestPartner/
TestComplete
Manual/Benchmark Tools/Visual Studio
LoadRunner/SilkPerformer/TestComplete
LoadRunner/SilkPerformer/TestComplete
4NetworkInfrastructure SUNRA/SPIRENT
Manual/Benchmark Tools/Visual Studio SUNRA/SPIRENT SUNRA/SPIRENT
5HardwareInterfaces Internal Simulators
Manual/Benchmark Tools/Visual Studio Internal Simulators Internal Simulators
6Mobile Client –Windows CE TestComplete
Manual/Benchmark Tools/Visual Studio Internal Simulators Internal Simulators
Problems with Manual Performance testing in the Lab
Difficult to mimic concurrency operationsCannot be repeated easilyDifficult to setup and organizeLab could be expensiveVery time consumingCannot afford to have large scale
performance labs in every development locations
Full NFRs cannot be tested
Commercial tools Vs Internal tools
Consider the scope of features neededEstimate the ROIHow soon you need the toolDoes the test team has performance
testing expertiseCustomer requirements for benchmarks
and proper test results
Tool Evaluation Process
Phase 1 – Fact finding of requirements, suitable vendors and preparing a proposal for project stakeholders – 5 weeks
Phase 2 - Tool demonstrations in lab – 6 weeks (2 weeks for each vendor)
Phase 3 – Complete Evaluation Process –
1 week Phase 4 – Negotiation and Purchase of a tool –
4 weeks Phase 5 – Tool deployment – 1 week
Issue # 1 – Testing Non Functional
RequirementsUnless it is tested we cannot guarantee
that the application going to work for the stated NFRs. That is if we say that our application can support 500 mobile clients we need to test these scenarios before clamming the limits of our system. We cannot test 50 clients and say that the system should work with 500 clients calculating performance parameters by interpolating them.
Solution #1: Always test/simulate full
NFRsShouldn’t interpolate single user data for
multiple usersTry to test with worst-case scenarioDifferent bottlenecks restrain achieving
better performance/Scalability.Proper testing LabUse tools to simulate and test full NFRs
Issue # 2 – Cost of fixing Performance
DefectsWe need to start testing Performance and scalability early in the development life cycle. More you do this early it is easy to find solutions and less costly to the project.
Issue # 3 – Unable to gauge performance
Performance optimizations shouldn’t be integrated haphazardly. We need to have mechanism and tools to monitor performance and scalability regularly. If it is not with weekly builds we should at least try to monitor this at least once a month with set of test scenarios covering good part of operations of the application. It would be ideal if we can automate this.
Issue # 4- Unrealistic/Untested NFRs
We need to better scrutinize the NFR requirements coming from Marketing to really understand that we can support them with the technology and the platform that we select for the product.
Issue # 5 – Agile Performance Testing
If Agile method has been adopted for the development process try to do performance and scalability testing in each sprint tied down to user stories. This will give developers immediate feedback of the status of performance of the particular area.
Issue # 6 – Testing with unrealistic data
We need to test systems with more realistic data as possible. We should create different sizes/profiles of databases and benchmark results for each category. We can develop these data sets together with the development team as this is important for them as well.
Solution # 6 – Data Generation
We need to programmatically generate application data keeping the data integrity.
Do this for different profiles (standard, heavy and ceiling)
Update and maintain this dataThis is a responsibility of both test and dev teams
Issue # 7 – lack of customer feedback till
end of the project
Many projects we find major issues because our process does not has room to get customer feedback a early in the project.
Solution # 7 – More input from Support
Dept and CustomersProvide Implementation/Support department with working software/system before the application transitioned to collect feedback
Get customer feedback if possible before the release
Expose identified dev and test team members for real customer environments if possible.
Issue # 8: Growing customer demands
Once a customer buys your product with time their requirements overpass the original application requirements. Mostly this is to do with handling increase amount of data. Some customers may even require to keep even 10 years worth of data in the running system. This could be a major challenge if not planned properly.
Solution # 8: Capacity Planning
Collecting data from Customers/fieldConsolidate this data for a 6 months/1
year targetsFocus more on most frequent & most
important featuresGet the Performance Team to start on
these targets earlyDo Capacity planning for both
software and hardware/network
Microsoft Performance Testing Labs
Can use for benchmarkingCan use testing tools with 1000’s of
user licensesCan use Microsoft domain experts
on PerformanceWorth investment towards
improving system performance
Books & Reference materials
MSDN articlesPattern & Practices (Performance
tuning MS .NET Applications) – By MS ACE Team)
Improving .NET Application Performance and Scalability – by Rico Mariani, Scott Barber)
Top Related