119500 OS Performance Testing Using Apache JMeter

Open Source Performance Testing

Using

Apache JMeter

A Success Story Of

A Leading Financial Company in the U.S

By

Naga, Sravanthi (119500)

Performance Testing CoE

Open Source Performance Testing Using Apache JMeter

White Paper of 12

Table of Contents

Abstract ..................................................................................................................... 3

1.0 Introduction ...................................................................................................... 4

1.1 About the Organization ..................................................................................................... 4

2.0 Performance Testing ....................................................................................... 4

2.1 What is Performance Testing? ......................................................................................... 4

2.2 Need for Performance Testing ......................................................................................... 4

2.3 Costs associated with Performance testing ................................................................... 5

2.4 Open Source Solution ....................................................................................................... 5

3.0 Success Story .................................................................................................. 6

3.1 Application Architecture – An example ........................................................................... 6

3.2 Core Performance Testing Activities ............................................................................... 7

3.3 Implementation: Challenges faced .................................................................................. 7

3.4 Implementation: Open Source Benefits .......................................................................... 8

3.4.1 Implementation: Apache JMeter Features & Overview ................................................... 9

3.4.2 Implementation: Apache JMeter Limitations ..................................................................... 9

3.5 Implementation: Value Additions from Cognizant ....................................................... 10

3.5.1 Introduction to JMeter – GLog Reporting Framework .............................................. 10

3.5.2 Features .................................................................................................................... 10

3.5.3 Benefits ..................................................................................................................... 10

3.6 Implementation: Lessons learnt ..................................................................................... 11

3.7 Client specific benefits .................................................................................................... 12


White Paper of 12

Abstract

In early 2007, Cognizant’s performance testing team had started its journey in the Open Source arena. During this tenure of 1.5 years, we’ve gained considerable mastery in some of the popular Open Source tools backed by a lot of success stories, new innovations, continuous Research & Development of new tools & utilities, new frameworks being built in this exciting Open Source galaxy.

What are the reasons for this growing affinity towards the Open Source tools? The reasons being not just the Cost savings, but Open source solutions have much to offer performance test engineers. It provides an alternative choice to commercial test tools. With proper planning and evaluation, the Open source Performance team can become a cost effective alternative solution for an organization’s performance testing needs. Having Performance testers with the right skill sets and with programming knowledge can go a long way in making the Open source Performance testing beneficial in the true sense.

This white paper explains the success story of a leading financial organization in United States and how performance testing activities were successfully carried out by using the Open Source tool Apache JMeter without compromising on quality. It also depicts the value additions that the Cognizant’s performance testing team has provided to the customer. Without the rigor described in this paper, performance testing could mean lots of invested dollars on the tool and the support from the tool.


White Paper of 12

1.0 Introduction

1.1 About the Organization

The company is a leading financial organization with a clearly defined mission: to help craft a vision for ideal financial future - especially when it comes to retirement - and then help you make it a reality. It is the next generation of one of the most respected financial services companies. With over two million clients and one of the largest advisor networks in the country, it is ushering in a dynamic new era of financial planning - one where we can help you pursue your dreams with greater confidence, increased peace of mind and the benefits of an enduring relationship with an advisor. Over 110 years in financial services, more than 10,000 personal financial advisors in more than 3,500 offices nationwide and a heritage of corporate philanthropy.

2.0 Performance Testing

2.1 What is Performance Testing?

Performance testing of an application / system is basically the process of understanding how the application and its operating environment behave at various user loads. In general, it is performed by simulating virtual users to determine / validate the scalability, availability, robustness, hardware & software resource utilization of the application thus ideally paving the way for effective capacity planning of the system for future usage.

One of the main objectives of performance testing is to help maintain the system with low latency, high throughput, and low utilization.

Various types of Performance Testing include:

Load Testing Stress Testing Volume Testing Spike Testing Endurance Testing

2.2 Need for Performance Testing

Performance Testing sets the ‘best possible’ performance expectation under a given configuration of infrastructure. It also highlights early in the testing process if changes need to be made before application goes into production. Some of the reasons for which Performance Testing has to be mandatorily carried out are:


White Paper of 12

Poor Response Time A critical webpage may take one minute to show up. The expected response time is less than 10 seconds. Scalability When 100 users from new branch were introduced into the system, the application was unable to handle the additional load and response times degraded from 5 seconds to 50 seconds. Batch Window Issues The batch upload that is supposed to happen in a 1 AM to 7 AM window does not complete till 9:30 AM. This impacts the availability of data for online applications. Availability The application server needs to be restarted owing to resource leaks within the application. This causes a downtime of atleast 0.5 hours a day. Resource Capacity Planning An application consumes 90% CPU on a server that is supposed to host another application after 3 months. There is no extra capacity to take this additional load.

2.3 Costs associated with Performance testing

Performance testing usually has huge costs associated with it. The costs associated with Performance testing can be categorized as

The Investment made on the Performance testing tool. Time consumed in Preparation of test data. The Load Testing Infrastructure / environment setup should be as close as possible to the production environments. In case of administrative organizations, acquiring the required permissions is a tedious process, since performance tests typically require many administrative rights.

Bottom-line is while there are many costs associated with Performance testing, the

Investment on the tool is considered to be the most indispensable contributor to the total cost. Most of the commercial tools require separate licenses to be procured based on the user load which contributes to huge Performance testing effort cost itself.

2.4 Open Source Solution

Quest for better alternative The Investment on the tool has lead to search out for better

alternative solutions. This quest has ended in the Open Source Performance Testing tools without sacrificing quality.

What is Open Source? ‘Open Source’ is defined as a software code that is available for users to examine and change freely without violating any patents, copyright laws, or licensing agreements. The idea with open source is that when everyone can work together and build upon existing tools, the ultimate result is much better software. It is a way for


White Paper of 12

many companies and individuals to collaborate and improve software that each person could not do alone.

Why should one go for open source?

No cost – Absolutely FREE!! Evolving Software (Can be customized to users’ needs). High portability. Scalable and offers an effective solution for growing business demands. Huge open source forum technical support which stands in par with any other commercial tool support. In fact it’s more advantageous as there is no cost or logistics involved like other commercial tool technical support. In summary, open source solutions are not just about cost savings. It is more customizable and user friendly, highly portable across various platforms and offers an effective solution for small growing business demands. The leading financial organization had exactly used this success formula to performance test its various kinds of Web & Web-service based applications.

Curious to know how? Its time get to the success story now…

3.0 Success Story

3.1 Application Architecture – An example

Wow!

Browser

Authentication Authorization System and Site

Administrator

Site, Page,

Portlet &

Navigation

Custom

Portlets

Dynamic Portal

Module

Common

Content

Management

System

Database


White Paper of 12

Multi-Tier Architecture. Portlet data is customized according to user queries and data is retrieved from the database. The data is then fed into the Common Content management system after which it is returned to the browser. So thus all the various Web & Web-service based applications which were involved in this Content management system were successfully tested using the Open Source Performance Testing Tool Apache JMeter.

3.2 Core Performance Testing Activities

The following are some of the key activities during the successful implementation of Performance testing for the leading financial organization using JMeter:

Understand and Analyze the Business Requirements. Define Performance Test Goals and create a Performance Test Plan. Create JMeter Scripts for identified critical business flows and execute various load test scenarios. Prepare customized analysis summary reports for every test execution and an overall Engagement Summary Report at the end of the engagement. Based on the performance bottlenecks idenitified, provide suggestions to the application development team for tuning the application. Test executions are thus run in iterative manner until the desired acceptance criteria are met.

3.3 Implementation: Challenges faced

Being the first venture into the world of Open source Performance testing, the following were the challenges.

Newer concepts and various ways of implementing different protocols had to be explored from scratch. For instance Bean Shell Technology, Handling FTP requests and dealing with various types of Java based requests. The reporting aspect of the tool was very limited. The data given by the tool was in a raw form and a lot of manual effort had to be involved in order to derive meaningful results and to generate elegant presentable reports. Lot of other open source JMeter utilities and plug-ins had to be be explored in order to effectively replicate the scenarios. The associates had to be highly flexible – For instance JMeter cannot record the AJAX specific requests. However it will be able to play them back. Hence those missing requests, headers had to be captured from other HTTP profiling tools like HTTP Watch and Packet Sniffing tools such as Wireshark and had to be placed in JMeter. Hence one has to be highly flexible in terms of technical expertise and usage of various tools and technologies in order to yield the maximum benefits out of Open Source.


White Paper of 12

3.4 Implementation: Open Source Benefits

Huge Open Source Support - Since the tool has a huge open source support community, learning new protocols were relatively easy and the queries posted almost had solutions. This is beneficial as most commercial tools include separate cost for providing technical support.

No need of High-tech infrastructure - Doesn’t demand a state of the art infrastructure for load testing.

Distributed Testing - Supports multiple load-injectors managed by a single controller. User-friendly GUI - Most of the popular Performance Testing open source tools

have a very good GUI thereby allowing faster operations and more precise timings.

Hence learning and adapting to the tool becomes easier over a period of time.

No Licenses and Maintenance Issues - Most commercial tools require separate licenses to be procured based on the user load specifications which contribute to huge tool cost. In addition to that, there involves an extra burden of maintaining the licenses. With the use of Open Source tools no such cost / efforts are involved to explicitly maintain the tool.


White Paper of 12

3.4.1 Implementation: Apache JMeter Features & Overview

Apache JMeter is a powerful desktop performance tool from the Apache Jakarta project, written in Java, for load-testing web pages, web applications, and other static and dynamic resources including databases, files, Servlets, Perl scripts, Java Objects, FTP Servers, and more. The main component in JMeter is the ‘Java Swing-based Graphical User Interface’ which can be used both for Scripting and Execution.

Excellent cost saving solution for small projects as it is an open source tool. Robust in handling complex test scenarios that demand ‘n’ number of virtual users. Complete portability and supports 100% all the Java based applications. Less scripting efforts as compared to other tools because of its user effective GUI. Is used to conduct Load, Stress, Volume & Endurance tests on the Web & Web-

service based applications. HTTP & HTTPS load testing can be done without any adding additional pluggable

samplers. Since it is Java based, the tool was highly compatible with most of the Java based

requests i.e. it can be used to directly test the Java requests, JDBC requests and JMS publisher & Subscriber. Hence it is a very good cost effective solution for small development activities as well.

3.4.2 Implementation: Apache JMeter Limitations

Reporting Framework Limitations Does not provide enough statistics: Most of the open source tools (JMeter as

well) does not provide the client side statistics summary reports in a presentable format as available with the other popular commercial tools such as HP Load Runner and Borland Silk Performer.

Huge Log Files: Added to that JMeter provides the log details for every user and every iteration thus in turn causing the log files generated to be very huge.


White Paper of 12

Manual Effort: Analyzing such voluminous log files involves a lot of manual effort in order to create highly sophisticated analysis reports.

Time Consuming: Above all, creating the final customer presentable summary report consumes huge amount of time.

AJAX components cannot be interpreted by JMeter. Cannot collate the Running Vusers data correcly in a Distributed Load Testing

conducted using multiple Load Generators. Cannot monitor the Server related components. There is no feature which allows to pass all the requests concurrently at the same time

unlike in other commercial tools.

3.5 Implementation: Value Additions from Cognizant

3.5.1 Introduction to JMeter – GLog Reporting Framework

‘JMeter – GLog’ is a new age Java reporting tool which is essentially used to plot the graphs and generate reports for the Open Source tool Apache JMeter. It presents a smart solution to customize and generate elegant reports from the log files within fraction of mouse clicks, thus bringing them in par with any other commercial Performance Testing Tool.

3.5.2 Features

The tool can spawn any number of records, transcending Excel’s 65K or 1048K limitation.

Capable of collating the log files from multiple load generators within fraction of mouse clicks.

The tool has a rich palette of features which allows the users to zoom, print, merge and save the graphs.

The observations can be included in the same screen as the report window, thus easing the task for the end user.

3.5.3 Benefits

The major benefit is one can drastically reduce the time consumed in generation of elegant final summary reports.

Automated generation of both PDF and MS-Word which includes elegant and sophisticated reports with all possible client side statistics as any other popular performance testing tool.

No external database is required. No more installations - simply unzip the file and start using it. Above all one can save one’s productive time


White Paper of 12

3.6 Implementation: Lessons learnt

JMeter was an excellent choice as this project's performance test tool. The following lessons learned provide additional detail. Team Skill set: If the group in charge of implementing and executing the performance test effort does not have a member with a solid programming background, the open source initiative will surely fail. There must be a person on the team capable of addressing issues related just not only to usage of the performance tool, but making changes and/or modifications to the open source code. The team must have the appropriate skill set to properly implement as well as use these tools. The first implication was that in order to prepare proper performance test scripts the JMeter users should have sufficient Java technical skills. Training: The Cognizant Performance Testing team had to independently learn on how to use the tool and also offer training to the other team members on their learning. This clearly states that there is the necessity to get the right skills and experience with performance testing and use of performance test tools to make the project a success. Limited Support: The open source community does an excellent job of providing online support. However, depending on the timeline of one’s performance testing effort, the type of support needed may not come as fast as one would like to be. As a result of the unavailability of a dedicated support team from Apache, the Performance Testing team had to overcome the problems faced during their testing activities.

Reporting Framework: JMeter generates a huge output file and a lot of manual effort is needed to create sophisticated analysis report. Analyzing JMeter logs and creating graphs is a tedious and time consuming process. To overcome the analyzing issue and time constraint, a work around was carried out to extract details from voluminous data. Using this method, average, maximum, minimum response time, number of passed and failed transactions, throughput can be extracted with minimum effort.


White Paper of 12

However now a Reporting framework is readily available which had made life easier

3.7 Client specific benefits

The entire Open Source framework provided major cost savings to the client without compromising on the quality of the deliverables. JMeter was easy to install and of medium complexity to learn and hence the effort involved in adapting to the tool was minimal. Being developed on Java, the tool was highly portable that made us explore different options thus inturn helping the client in Performance testing of different technologies. The Application was tested with upto 2400 Active Users volumes thus helping the client identify possible production performance issues beforehand. Since it is stage by stage performance test, the system bottlenecks were ironed out thus reducing the chances of production outages. Cognizant’s Performance Testing Team was highly flexible and adaptive to the various challenges pose by the Open Source tools. The team was able to overcome the reporting limitation by developing a complete reporting framework thus making it as an entire Open source packaged solution to the customer.

119500 OS Performance Testing Using Apache JMeter

Documents

Transcript of 119500 OS Performance Testing Using Apache JMeter