LHC Status Tue Morning 22-Aug Bernhard Holzer, Joerg Wenninger
Joerg Nalik - SAP · PDF fileSAP LABs 2003, Stress Testing, Joerg Nalik / 27 SAP J2EE 6.20...
Transcript of Joerg Nalik - SAP · PDF fileSAP LABs 2003, Stress Testing, Joerg Nalik / 27 SAP J2EE 6.20...
2003 Business Information, Technology & Infrastructure Forum
Stress Testing Component Landscapes for Performance Optimization and Sizing
Joerg NalikSAP LABsSession Code: 1612
SAP LABs 2003, Stress Testing, Joerg Nalik / 2
High level Goals of Stress Testing
• Achieve required transactional throughput
• Achieve good system response times for end users
• Minimize hardware costs
Stress testing helps to achieve and verify these goals!
SAP LABs 2003, Stress Testing, Joerg Nalik / 3
Overview
• Introduction
• Stress Testing Project Management
• Stress Testing Strategy
• Monitoring and Tuning– Abap components– Java/J2EE components– Web-servers– Network– Response times– Business processes
SAP LABs 2003, Stress Testing, Joerg Nalik / 5
Background of this presentation
• Experiences from SAP internal performance testing of the SRM solution and other SAP products.
• SAP standard procedures and recommendations for monitoring, tuning and sizing.
• Managing complex stress testing projects.
• Contacts with customers who do stress testing.
• Helping SAP support with customer system performance issues.
SAP LABs 2003, Stress Testing, Joerg Nalik / 6
Component landscapes
SAP’s business solutions follow a component approach. In general, multiple application components of a solution are installed on multiple hardware servers and bound together via the NetWeaver Infrastructure components.
EBP, Buyer
NetWeaver-XI
SUS, Seller
SAP LABs 2003, Stress Testing, Joerg Nalik / 8
Technical Goals of Stress Testing
• Complement functional testing: Verify system stability under load
• Verify hardware capacity planning
• Test for scalability and performance bottlenecks
• Check end user response time stability under high load
• Optimize performance through monitoring and tuning
SAP LABs 2003, Stress Testing, Joerg Nalik / 9
Challenges of Stress Testing
• Find a suitable test landscape:– Co-use of QA or Production landscapes needs
good coordination. Temporary exclusive access to a test landscape is recommended.
• Tools for producing test load– line up a group of manual testers: only for one-
time tests.– use of QA or hard-beat monitoring tools: might
not be able to generate high load, lack of integration.
– use of load testing tools: license expense.
• Defining test strategy, cases and goals
SAP LABs 2003, Stress Testing, Joerg Nalik / 10
Build your Team
• Environment Specialist(s):– Experienced in hardware and network
infrastructure
• Application Specialist(s):– Trained in application configuration and
administration
• Business Scenario Specialist(s):– Experienced in business processes and their
implementation across application components
• Test Engineer(s):– Trained in using load testing tools
SAP LABs 2003, Stress Testing, Joerg Nalik / 11
Project Milestones
• End of project definition phase for resources, test cases and test goals.
• All application components are installed and interconnected (application integration).
• Test cases are implemented and run functionally correct.
• Test-Tuning cycle is finished.
• Final measurements, documentation of results.
SAP LABs 2003, Stress Testing, Joerg Nalik / 13
Component versus Scenario Stress Tests
• Application integration to Business Solutions leads to solution landscapes consisting out of many individual infrastructure (example NetWeaver) and application components (example SRM).
• Solution business scenarios are executed by users with different roles and might include overall >100 screen changes.
! Reducing complexity of stress testing:" testing of important sub-scenarios only" one user role only in a sub-scenario.
SAP LABs 2003, Stress Testing, Joerg Nalik / 14
Choosing Test cases
• What causes most CPU utilization in a productive environment?
• Which sub-scenarios are response time critical (end-user satisfaction)?
• Example: SAP’s User Management Engine– 90% case: Login and Logout - performance is critical – 10% case: Administer User Accounts, important for
functional correctness, but not critical for performance testing
Tip: Review capacity planning for CPU intensive scenarios.
SAP LABs 2003, Stress Testing, Joerg Nalik / 15
Choose Metrics and Goals
• Metrics: What should be measured? What impacts CPU and other resource utilization?
• Goals: When is it good enough? Examples:– 100 Purchase Orders per hour with– average 5 line items per Purchase Order– average response times below 3 seconds
SAP LABs 2003, Stress Testing, Joerg Nalik / 16
Create and customize test scripts
• The load testing tool should allow:– recording of scenarios, – parameterization and run-time substitution, for
instance for user names and business objects IDs.
• To avoid artificial resource leaks, bracket every scenario with login and logout.
SAP LABs 2003, Stress Testing, Joerg Nalik / 17
Important Features for Stress Test Execution
• programmable load variation over time
• for web-applications: correct simulation of static content download, browser caching, cookie handling and connection handling (keep-alive).
• integration of internal load test tool and external monitors of the operating system and other components (application servers, databases, …)
• automated test result recording and reporting.
SAP LABs 2003, Stress Testing, Joerg Nalik / 18
Test Methodology
Considerations for Concurrent Users versus Transaction throughput:
• How many Concurrent Users can be served by an application system?
– The number of concurrent users is meaningless without defining “think time”. Think time is the time needed by users to send a subsequent request after receiving a response. Think time helps to simulate real user behavior.
• How many Transactions per hour can be handled?– Example: 100 conc. users, 60 sec. think time, 2 sec. resp.
time, 40 pages/transact.:– Transact./h = 100 * 3600 /[( 60 + 2 ) * 40] = 145
TIP: Backend session management load is impacted by number of concurrent users and think time. Consider session time out settings, connection time outs ….
SAP LABs 2003, Stress Testing, Joerg Nalik / 19
Test Strategy I
• Determine maximum sustainable load– “Ramp-up” users slowly over time. Monitor CPU
consumption and transactions/second.
SAP LABs 2003, Stress Testing, Joerg Nalik / 20
Test Strategy II
• Use constant load Test for stability verification, and UI response time.
– Start with a fast ramp-up, long period of constant load (hours), then quick ramp-down (model of SAP benchmarks)
– Each simulated user performs many iterations during the test. Use about 70% of max. load known from ramp-up test.
– Monitor in detail: CPU, memory, network traffic, Java Garbage Collection ….
SAP LABs 2003, Stress Testing, Joerg Nalik / 21
Test Strategy III
Strategy for all load tests:• Start with small and short test runs for detecting and fixing
easy problems and tuning. When small tests are successful continue with next higher load level tests.
• Example:
• Goal: processing of 10,000 Transactions/12 hours, Test time 5 days.
• Big-Bang approach: about 5 test/improvement runs possible.
• Start small approach:– 1st day: 12 runs of 100 Transactions– 2nd day: 6 runs of 500 Transactions each– 3rd day: 3 runs of 2000 Transactions each– 4th+5th day: 1 run with 10,000 Transactions each day– Total: 23 test/improvement cycles
SAP LABs 2003, Stress Testing, Joerg Nalik / 23
What to monitor?
Load Test inherent monitors:• Transactions/time(second,hour) • number of simulated users• Sum of Server+Network request response times• Hits/second• Data volume download and upload• For web-appl.: Breakdown of page content download volumes
and times (#, size and download time of mime files, breakdown of network roundtrips for connect, https verification, data package downloads ….)
Other monitors, preferable integrated in load tool:• OS monitors for: CPU, Memory, Disk I/O, Network, …• Application and infrastructure component monitors (application
servers, databases, ….)
SAP LABs 2003, Stress Testing, Joerg Nalik / 24
What to tune?
• Abap components
• Java/J2EE components
• Web-servers
• Network
• Response times
• Business processes
SAP LABs 2003, Stress Testing, Joerg Nalik / 25
Abap component tuning
Useful Abap monitor transactions:
• ST02: check buffer sizing
• ST04: database monitor, in particular for locking
• ST06: OS monitoring
• ST03: Application monitoring
• ST05: SQL-, RFC-, Enqueue- tracing
SAP LABs 2003, Stress Testing, Joerg Nalik / 26
Java Virtual Machine tuning
Start memory configuration:• for Sun/HP JDK 1.3.x: tune “young” space to be about 1/3rd of
total Java heap. Example:– -Xms256M, -Xmx512M– -XX:NewSize = -XX:MaxNewSize = 192M– see details in OSS note 552522
• Disable programmed forced Garbage collection: -XX:+DisableExplicitGC
• Enable Garbage Collection monitoring: -verbose:gc
Check that time spent in Garbage Collection is <5% of CPU time. Otherwise try to tune Java runtime parameters.
Useful GC trace viewers: HPJtune, SUN GC Portal
SAP LABs 2003, Stress Testing, Joerg Nalik / 27
SAP J2EE 6.20 tuning
Monitor and tune:
• Thread pools
• DB-connection pools
• concurrent session/network socket usage
• …
Reference:
Session 1603: Performance Tuning your Java Engine by Marc Chan
Monday, October 27, 2003; 2:40 PM - 3:50 PM
SAP LABs 2003, Stress Testing, Joerg Nalik / 28
Web-server tuning
• The raw (un-buffered) size of a complete Web-site is in the range of 100KB and it takes 10 to 20 request/response cycles to download all web-objects of one page. Reduce volume to about 10KB/page through 2 to 5 roundtrips:
– If access is via WAN enable compression of certain mime files (.js, .css) and dynamic html content.
– Tag mime files with an expiration date in order to avoid repetitive downloads.
– configure “keep-alive” for connections.
• References: OSS notes: 553311, 350646
SAP LABs 2003, Stress Testing, Joerg Nalik / 29
Networks inner workings
Network tuning can have a major impact on response times through WAN, Internet and modem connections. The following are key parameters:
• Bandwidth: impacts transmission times for a certain data volume.
• “ping time” between sending and receiving end. Good measure for latency time for one TCP/IP network roundtrip
• number of TCP/IP roundtrips for one application roundtrip. TCP/IP roundtrips occur for:
– opening a connection. Tune keep-alive to keep connections open longer. However, watch number of concurrently open sessions in application servers, which might increase.
– https certificate verification– first package download– subsequent package downloads. Might be reduced through use of
compression
SAP LABs 2003, Stress Testing, Joerg Nalik / 30
Estimating network timesExample: • 10KB dynamic website content• 3 application roundtrips, • 1 connect, 1 https, 1 first package TCP/IP roundtrip• 0.1 second ping time• 56 kbps modem connection bandwidth
! Bandwidth time = 10 KB * 8 / 56kbps = 1.4 seconds Accumulative latency time = 3 appl. r. * 3 TCP/IP r. * 0.1 sec. ping time = 0.9 sec
Total maximum time = 2.3 second
Note: application roundtrips might be executed in parallel and actual network times therefore might be shorter.
SAP LABs 2003, Stress Testing, Joerg Nalik / 31
Response time contributions
Browser
Web Server
Application Server
staticcontent
dynamiccontent
• Browser rendering time
• Network bandwidth and latency times
• Web server response times
• Application server response times
Due to use of parallel connections and multi threading of browsers response times might be shorter than sum of individual times.
SAP LABs 2003, Stress Testing, Joerg Nalik / 32
Response time optimization summary
Bad response times are not necessarily caused by the application (server). Response times can be optimized in all tier components and the network. Consider:
– for network times: compression, expiration tagging of mime files, local proxy server.
– for browser rendering times: faster desktop hardware.
– for web and application servers: tuning, keep-alive configuration, faster hardware.
SAP LABs 2003, Stress Testing, Joerg Nalik / 33
Business process tuning
Individual business scenarios might be tuned in their own particular ways. Example:– Reverse auctioning in SRM Live Auctioning
Cockpit (LAC):# Each bidder is polling latest bid price updates every few
seconds.# Bids are done with fixed size bid decrements.
– Server load can be reduced by increasing bid update interval times and by reducing #of bids/auction through setting larger bid decrements.
SAP LABs 2003, Stress Testing, Joerg Nalik / 34
Summary
Key success factors for component landscape stress testing:
• manage stress testing as a project
• define test goals and deliverables at the beginning
• define your test methodology
• consider use and features of load testing tools
• start small and work your way up
• plan for monitoring/tuning improvement cycles
SAP LABs 2003, Stress Testing, Joerg Nalik / 35
Acknowledgement
We like to thank our partners for their support of the effort required to develop input for this presentation:
– HP for hardware and engineering support.– SUN Microsystems for hardware and engineering
support.– Mercury Interactive for load testing software and
engineering support.
SAP LABs 2003, Stress Testing, Joerg Nalik / 36
2003 Business Information, Technology & Infrastructure Forum
Thank You For Attending!
Please remember to complete and return your evaluation form following this session.
Session Code: 1612
SAP LABs 2003, Stress Testing, Joerg Nalik / 37
• No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.
• Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
• Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation.
• IBM®, DB2®, DB2 Universal Database, OS/2®, Parallel Sysplex®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix and Informix® Dynamic ServerTM are trademarks of IBM Corporation in USA and/or other countries.
• ORACLE® is a registered trademark of ORACLE Corporation.• UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group.• Citrix®, the Citrix logo, ICA®, Program Neighborhood®, MetaFrame®, WinFrame®,
VideoFrame®, MultiWin® and other Citrix product names referenced herein are trademarks of Citrix Systems, Inc.
• HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.
• JAVA® is a registered trademark of Sun Microsystems, Inc. • JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under
license for technology invented and implemented by Netscape. • MarketSet and Enterprise Buyer are jointly owned trademarks of SAP AG and
Copyright 2003 SAP AG. All Rights Reserved
SAP LABs 2003, Stress Testing, Joerg Nalik / 38
• Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die aus-drückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankün-digung geändert werden.
• Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten.
• Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® und SQL Server®sind eingetragene Marken der Microsoft Corporation.
• IBM®, DB2®, DB2 Universal Database, OS/2®, Parallel Sysplex®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix und Informix® Dynamic ServerTM sind Marken der IBM Corporation in den USA und/oder anderen Ländern.
• ORACLE® ist eine eingetragene Marke der ORACLE Corporation.• UNIX®, X/Open®, OSF/1® und Motif® sind eingetragene Marken der Open Group.• Citrix®, das Citrix-Logo, ICA®, Program Neighborhood®, MetaFrame®, WinFrame®,
VideoFrame®, MultiWin® und andere hier erwähnte Namen von Citrix-Produkten sind Marken von Citrix Systems, Inc.
• HTML, DHTML, XML, XHTML sind Marken oder eingetragene Marken des W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.
• JAVA® ist eine eingetragene Marke der Sun Microsystems, Inc. • JAVASCRIPT® ist eine eingetragene Marke der Sun Microsystems, Inc., verwendet
unter der Lizenz der von Netscape entwickelten und implementierten Technologie.
Copyright 2003 SAP AG. Alle Rechte vorbehalten