Performance and Reliability 101 Brent Cromarty Ping Identity [email protected].
-
Upload
summer-beachley -
Category
Documents
-
view
214 -
download
0
Transcript of Performance and Reliability 101 Brent Cromarty Ping Identity [email protected].
![Page 2: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/2.jpg)
A little about me
• Like– Long walks on the beach– Red wine
• Dislike– Mean people– Early mornings
• Encourage questions throughout presentation– Although I may hold off if I am going to address it with
material in a future slide.– You may have to ask again
![Page 3: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/3.jpg)
OK Seriously…
• Spent bulk of my career (14 years) at SAP– By way of Business Objects acquisition
• By way of Crystal Decisions acquisition– By way of Seagate Software IMG
• 5 years experience in customer support– Discovered impatience, dislike for people
• 9 years of performance and reliability (P&R) testing for Crystal Reports in Crystal/Business Objects Enterprise product
• Currently in my second year with Ping Identity
![Page 4: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/4.jpg)
Why are we here?• Types of testing that make up P&R – Design• What is the goal of each test type?• What does it prove/disprove?
– Execution• How is the test run?
– Results Analysis• How to figure out if the test passed or failed?
• Best Practices (Tips/Tricks/Suggestions/Filler)• Suggestions for root cause analysis
![Page 5: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/5.jpg)
So… What are these test types that I speak of?
• Types of P&R tests– Load– Scalability– Endurance– Stress– Reliability
![Page 6: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/6.jpg)
Load
![Page 7: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/7.jpg)
Load Testing• Performance equivalent of functional “smoke” test• Functional test/workflow executed under “load”– Typically “load” is in the form of concurrent users
• Executed with a Load Generator tool– Load Runner, JMeter, QALoad, Grinder, etc…
– Does the component stand up?• Does the test pass functionally? For all users?• Does it crash? Does the system grind to a halt?
– Metrics to consider• Response time (average, 90th percentile, min, max)• Throughput• CPU and memory utilization on the target system
![Page 8: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/8.jpg)
Scalability
![Page 9: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/9.jpg)
Scalability Testing• Executed as a series of Load Tests– Workload Scalability
• Vary the user load from test to test
– Resource Scalability• Vary the resources from test to test
– Functional success• Error rate “too high”, scalability results are meaningless
– How does performance change from test to test?• Response time (average, 90th percentile, min, max)• Throughput• CPU and memory utilization on the target system
– Do not discount single user performance• A system can exhibit linear scalability, but still perform poorly
![Page 10: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/10.jpg)
Endurance
![Page 11: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/11.jpg)
Endurance Testing• Also know is “Soak” testing• Load test executed over an extended duration– Typically overnight or over the weekend
• Proves “reliability” of the system– Consistency of functional results
• Very first result same as very last and all those in between?• Depending on requirements, error rate > 0 can be acceptable
– Consistency of performance• Does response time or throughput degrade over time?
– Consistency of resource utilization• Are we leaking memory?• How does CPU usage look over the duration?
![Page 12: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/12.jpg)
Stress
![Page 13: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/13.jpg)
Stress Testing• Often mistakenly referred to as “Load” testing• Best thought of as “extreme” load testing– Resiliency of the system when pushed beyond limits
• 150% to 200% of the “nominal” load for the system• Half the system resources suggested for a given load– CPUs, memory, network bandwidth, etc…
• Looking for “graceful failure”– Best: System returns “Too Busy”– Acceptable: System slows down, maybe some requests time out
• Better: effective error messaging so that uses know system is maxed out
– Bad: Crash– Worst: Unpredictable results, misleading error messages
![Page 14: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/14.jpg)
Reliability
![Page 15: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/15.jpg)
Reliability Testing• Negative condition Load Testing• Test resiliency under error conditions– Error condition code paths typically don’t get the same
coverage as the “happy path” – Is the system consistent under constant error conditions?
• Are results consistent and predictable over time?
• Consistency of resource utilization– Error conditions are notorious for resource leaks
• Security tests– i.e.: Denial of Service
![Page 16: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/16.jpg)
Random Suggestions (Time Filler)• Choose workflows that fit the “80/20 rule”
– Some workflows need P&R testing, others don’t. Choose wisely.
• Use sufficient hardware for your Load Generator application– Size your client hardware like you would your target system
• Don’t use “intrusive” validation in your test cases– Heavy test validation will slow down your test and affect concurrency
• Avoid use of “intrusive” monitoring when possible• Beware of logging
– Logging is useful, but can kill performance
• Visualize your results– A picture is worth a thousand words. Who doesn’t like charts?– Include context (resource utilization of the systems under test)
![Page 17: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/17.jpg)
So what do I do if I think there is a problem?
• Too slow?– Is your system tuned?
• Ensure you have not configured a bottleneck in your deployment
– Try a profiling tool• Can show which areas of the code are taking the most time
– Add some lightweight logging to code• Add “timing code” to log out elapsed time in functions/paths
– Use a stack dumping utility• Repeated stack dumps can show where you are “stuck”
• Using too much memory or leaking?– Try a profiling tool
• Can show
– Add “size” logging for container classes• Can show you if your containers are growing unbounded
![Page 18: Performance and Reliability 101 Brent Cromarty Ping Identity bcromarty@pingidentity.com.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649c925503460f9494d7b2/html5/thumbnails/18.jpg)
Questions?