Unleashing the power of JMeter

Suresh Malan, Senior Consultant

Venkata Goday, Director

Capgemini Technology Services

AbstractCan Apache JMeter replace some of the established and standard performance tools in the market? Currently this is the most sought after question by performance testing managers across the Industry, few years ago this was never even thought of but with the change in IT landscape and open source culture this is inevitable. Also, in a typical IT application stack more than 70% are web based hence considering the trade-off with tools offering large number of protocol support, JMeter is economical. However to be able to do that JMeter considered to be open source standard needs to suddenly scale to enterprise level and the high expectations of performance engineers.

In this white paper we want to bring out how to extend JMeter capabilities in areas such as reporting, standardization, scripting, scenario design, executions which can make it feature rich and tool of choice for performance engineers. We will also talk about how JMeter can be effectively used for new and future technologies like Cloud, mobile IoT, Big data and a lot more. This article can help performance managers take the leap towards open source.

IntroductionApache JMeter has been the most popular Open source Performance testing tool for a long time. However off late a lot of organizations have started questioning the fact, if it is possible to use JMeter as the primary Performance testing tool. Reasons may vary from the organizations making a conscious move towards Open source or the impact of controlled overall budgets resulting in reduction of expenditure on performance testing.

Foremost point in favour of JMeter is that it is Open source making it highly economical. Being Java based makes it platform independent and it is easily extensible with the use of plug-ins. Though all this is possible, it has been observed that till recently, Apache JMeter did not have comprehensive reporting framework, making analysis cumbersome. In addition to this, there

are certain activities like script enhancement, scheduled execution and report generation which can be automated which will help in streamlining the process.

JMeter has a large community support with frequent releases and updates making it a sustainable and dependable tool.

Also another point worth considering is the capability of JMeter to handle Performance testing with newer/future technologies like Cloud, Mobile IoT or Big Data to name a few. Given that it has extension possible via plug-ins for instance it has plug-ins for Databases, IoT, and so on, and given that complex tests can be written using the Beanshell and JSR223 samplers make it future-proof. The automation, extensibility and future-readiness will help JMeter scale up to provide a comprehensive enterprise level Performance testing experience.

Before we discuss these possibilities, a brief introduction is in order for Performance testing and the tool under consideration, Apache JMeter.

Performance TestingThis section will only provide a brief introduction for Performance testing. In software engineering, Performance testing1 is in general, a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage. The image below helps visualize Performance testing using JMeter.

Objectives of Performance testing can be to help evaluate:

If the system meets performance criteria under the different identified workloads.

Compare two similar systems as to which one performs better. Identify any bottlenecks / slow components that are causing the system to perform

badly under a certain given workload.

There are a variety of Performance testing tools available ranging from market leading, license-based, commercial tools with lots of capabilities and support to free Open Source tools with good capabilities and certain limitations. Though the commercial tools, with the ease of use and guaranteed customer support are better at testing more complex protocols. However given that a large part of the Performance testing expenditure is spent on the cost of the Commercial tools used, especially with the simpler protocols, the Open Source Performance testing tools can bring a lot of benefits since they do not have any cost involved.

Also the organizations have started making a move towards Cloud, Big Data and IoT so there is a shift from the traditional performance requirements. So there are adjustments that will have to be made to the process of Performance testing and the tools in use have to adapt to these changing requirements.

Apache JMeter is Open Source and it is possible to extend its capabilities. So it will be free for use and extensibility will help it adjust to the changing requirements. The next section briefly describes Apache JMeter, its features and capabilities.

Apache JMeter – a brief introduction

Apache JMeter2 is one of the most popular Open Source Performance testing tools. A few of the features are mentioned below –

It is capable of performance testing many different application/server/protocol types like Web – HTTP/HTTPS, Web services, JMS, Java objects and so on.

It is built in Java and its command-line mode can be used across different operating systems, making it platform independent.

It has a full featured test IDE that allows fast Test Plan recording (from Browsers or native applications), building and debugging.

It is highly extensible with the use of different samplers, especially the scriptable ones allowing for unlimited testing capabilities. Also plug-ins are available to enhance the capabilities

JMeter has all these features but until recently, it lacked a comprehensive reporting functionality. This particular drawback negatively affected the process of analyzing the performance test results. It has overcome the reporting limitations with the HTML report that it generates with the newer versions.

The next section describes automation of script enhancements and report generation along with scheduled test execution, that helps JMeter resolve any issues and improves the existing capabilities.

Extending the JMeter capabilities

JMeter is capable of much more in terms of protocols supported however it has been used to test web services and simple web-HTTP based applications. However off late the clients have been asking if it can act as an alternative to the commercial tools in use. This is majorly because of the cost factor and the tendency to move towards Open source.

In order to cater to this requirement, JMeter has to enhance itself from an Open source tool to an enterprise level where it supports the end-to-end performance testing lifecycle. The enhancements discussed here have been performed taking this into consideration and with minimal changes to the default JMeter features thereby having minimal impact on the existing scripts and results.

The basic performance test execution process follows the generic steps below -

The steps above can be automated and optimized using custom Framework around JMeter. Implementation details of the framework are detailed below.

1) Script Automation:

If you look at the process of script recording, there is little scope for automation. Currently the performance tester has to understand the steps forming part of the transaction, repeat the

Records the script based on the identified transactions

Script Recording

Perform script enhancements like Parameterization, Correlation, Transaction Renaming, Timer addition and Adding of response assertions

Script enhancements

Execution of the various types of testsTest execution

Analysis of the test results to gain insights

Results Analysis

Test report preparation and formatting

Reporting

same during the recording process, and if there is any step missed the entire process has to be repeated, until a successful recording is completed.

Automating the process which is simple yet radical wherein the script can be recorded right when the Client associate / Business architect is showing the transaction, avoiding a lot of cumbersome process and complexity thereby saving a whole lot of time.

2) Script enhancements:

The Framework helps with the script enhancements focused at – Parameterization & Correlation, Transaction Renaming, Timers and Assertions

a. Parameterization & Correlation:The automated script generation process also helps with this, where it identifies the different points where there is a possibility to induce parameterization and also locations where correlation is required for successful script execution.

b. Transaction Renaming:It helps by automating the process of transaction renaming where the project has a typical transaction naming convention and each time a script is recorded the transactions have to be renamed to follow the naming convention.

c. Timers:Framework allows the user to choose the transactions for adding the timers at appropriate locations in the script.

d. Response Assertions:These are text validations placed to validate the response received from the server. The Framework helps to enhance the script by automating the process of inserting response assertions within existing scripts.

3) Scheduled Test Execution:

Two approaches have been used to schedule the execution a. As part of the Continuous Integration (CI) / Continuous Deployment (CD) process

using Jenkins or b. Using the Framework to control the execution.

4) Results Analysis and Report Generation:

Framework saves a lot of time spent by automating the process of analyzing the test results and producing the reports which are available in editable as well as non-editable formats. So the reports can be generated with a pre-formatted structure and that too within seconds of the test execution.

As can be seen the Framework around JMeter helps achieve a sizeable gain in the overall time/effort spent on the end-to-end performance testing lifecycle by automating the processes with respect to script recording, script enhancements, test execution, analysis and reporting. Thus the Framework helps JMeter to improve the productivity of the performance test analyst resulting in a swift approach to performance testing.

Compatibility with new and future technologiesChanging technologies are impacting the way the end-users are accessing the application, the way the organizations are hosting their infrastructure and also how the applications themselves are communicating with the endpoints and end-users. All these changes do have an impact on the way the Performance testing is carried out, and the process needs to be adjusted taking into consideration these changes. A few of the technological changes, their impacts and how JMeter caters to them are discussed below.

- Mobile usersA lot of the end-users are moving towards mobile and this population is constantly increasing. This makes it mandatory for the organizations to include these transactions in performance testing. It is well known that JMeter does cater to Web(HTTP/HTTPS) and Web services performance testing, however using the same approach of proxy based recording, JMeter is capable of recording the traffic that flows from a Mobile end user, be it via a native app or web. These requests will get captured and the same can be replayed with JMeter using the required number of users. End-user network characteristics can be identified and the same can be simulated to understand the end user experience.

- Cloud infrastructureCloud infrastructure provides the flexibility of pay-per-use model. Generally Performance testing is not an ongoing activity in an organization and the pay-per-use model of the cloud can be greatly used to perform the testing. Blazemeter provides a solution in this respect3. It is a cloud based solution which uses JMeter for performance testing.

- IoT – MQTT protocolMQTT4 stands for Message Queue Telemetry Transport. It is publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimize network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

The MQTT Plugin in JMeter5 is used for the injection testing of MQTT server. It permits to completely test many scenarios, which depend on type of messages and type of connections. Thus JMeter is capable of testing IoT scenarios as well.

- Big DataMany organizations are looking at Big Data for faster processing of huge amounts of data that need to be filtered and analyzed for relevant details. Performance testing of Big Data will always be challenging, since it deals with huge amounts of complex, structured/unstructured data with high levels of insertion and retrieval rates. However tools like JMeter help since they already have a set of plug-ins for testing databases like the one available for MongoDB, arguably the most popular NoSQL database.

ConclusionThis paper has discussed that Apache JMeter is an Open source tool with the required features for performance testing across different platforms and technologies; however there are certain areas which can be improved. Further it was discussed how with the Framework around JMeter has proven that it is possible to automate the performance testing cycle performed using JMeter, thereby reducing the effort by 30-40%. Also JMeter has the capability to handle the new and future technologies like Cloud, Mobile, IoT and BigData

With the economy of Open source, the enhancements through automation, the ease of use that follows and the future-readiness, JMeter will be a force to reckon with.

References & AppendixReferences

1. https://en.wikipedia.org/wiki/Software_performance_testing 2. http://JMeter.apache.org/ 3. https://www.blazemeter.com/ 4. http://mqtt.org/ 5. https://github.com/tuanhiep/mqtt-jmeter

Abbreviations used

:Abbreviation Full form

IT Information Technology

IoT Internet of Things

HTTP Hypertext Transfer Protocol

JMS Java Message Service

IDE Integrated Development Environment

MQTT Message Queue Telemetry Transport

CI / CD Continuous Integration / Continuous Deployment

Author BiographySuresh Malan is part of the Performance Testing & Engineering CoE for Capgemini global; he has been in the field of Performance Testing / Engineering for the past decade working on performance testing engagements with several clients and also supporting development of different performance testing assets.

Venkata Goday heads Performance Testing & Engineering CoE for Capgemini global; he has been leading this practice from the last 10 years and is well known to be a thought leader in performance engineering space. He has rich experience in setting up Performance CoE’s for customers and has partnered with several clients for strategic consulting and transformation initiatives. He architected and developed many assets in Non Functional testing space.

THANK YOU!