Software Performance Testing Overview

Rohan Bhattarai

Contents What is Performance Testing? What are the Terminologies I should know? Why Performance Testing? What are the Performance Bottlenecks? Why do we have bottlenecks ? What are the contributors? What are the types of Performance Testing? When is it required? What should be tested? What should be my Performance Testing Process? Why should I use Tool ? What should be my tools combination? What tools are available and what should I use for Performance

Testing? What are the Performance Metrics? What should be my best approach?

What is Performance Testing? Performance testing is

the process of determining the speed or effectiveness of a computer, network, software program or device.

Let’s talk about Software Performance Testing Before going into the details, we should

understand the factors that governs Software Performance Testing:

Throughput Response Time Tuning Benchmarking

What are the Terminologies I should know? Application Architecture HTTP vs HTTPS Request Vs Response GET and POST Methods HTTP Status Codes Throughput Response Time Tuning Benchmarking Transaction Mix User Mix User Think Time Ramp-up Pattern

What are the Terminologies I should know? Application Architecture

Client-Server 2-Tier 3-Tier N-Tier

What are the Terminologies I should know?HTTP vs HTTPS The Hypertext Transfer Protocol (HTTP) is a networking

protocol for distributed, collaborative, hypermedia information systems .

HTTP functions as a request-response protocol in the client-server computing model.

In HTTP, a web browser, for example, acts as a client, while an application running on a computer hosting a web site functions as a server. The client submits an HTTP request message to the server. The server, which stores content, or provides resources, such as HTML files, or performs other functions on behalf of the client, returns a response message to the client.

HTTPS is simply HTTP using Secure Sockets Layer (SSL) to encrypt and decrypt data for secure transactions

What are the Terminologies I should know?Request Vs Response Client Request

GET /index.html HTTP/1.1

Host: www.example.com Server Response

HTTP/1.1 200 OK

Date: Mon, 23 May 2005 22:38:34 GMT

Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)

Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT

Etag: "3f80f-1b6-3e1cb03b"

Accept-Ranges: bytes

Content-Length: 438

Connection: close

Content-Type: text/html; charset=UTF-8

What are the Terminologies I should know?GET and POST Methods GET is the simplest HTTP method, and its main job in life is to

ask the server to get a resource and send it back. That resource might be an HTML page, a JPEG, a PDF, etc. Doesn't matter. The point of GET is to get something back from the server.

POST is a more powerful request. It's like a GET++. With POST, you can request something and at the same time send form data to the server

GET and POST basically allow information to be sent back to the webserver from a browser (or other HTTP client for that matter). Imagine that you have a form on a HTML page and clicking the "submit" button sends the data in the form back to the server, as "name=value" pairs.

What are the Terminologies I should know?GET and POST Methods Choosing GET as the "method" will append all of the data to the

URL and it will show up in the URL bar of your browser. The amount of information you can send back using a GET is restricted as URLs can only be 1024 characters.

http://www.google.co.uk/#hl=en&source=hp&q=Rational+Performance+Tester&oq=Rational+Performance+Tester&aq=f&aqi=g10&aql=&gs_sm=e&gs_upl=1296l6140l0l6328l33l25l5l6l6l0l297l2062l2-8l8l0&bav=on.2,or.r_gc.r_pw.&fp=950be6bfd6a5c219&biw=1024&bih=563

A POST on the other hand will (typically) send the information through a socket back to the webserver and it won't show up in the URL bar. You can send much more information to the server this way - and it's not restricted to textual data either. It is possible to send files and even binary data such as serialized Java objects!

What are the Terminologies I should know?

GET Method: All the name value pairs are submitted as a

query string in URL. It's not secured as it is visible in plain text

format in the Location bar of the web browser.

As the data transfers through address bar (URL) there are some restrictions in using space, some characters like ampersand (&) etc in the GET method of posting data. We have to take special care for encoding data if such special characters are present.

Length of the string is restricted. If method is not mentioned in the Form tag,

this is the default method used. If get method is used and if the page is

refreshed it would not prompt before the request is submitted again.

One can store the name value pairs as bookmark and directly be used while sharing with others - example search results.

Data is always submitted in the form of text

If the response of the page is always same for the posted query then use GET example database searches

POST Method: 1. All the name value pairs are submitted in

the Message Body of the request. 2. Length of the string (amount of data

submitted) is not restricted. 3. Post Method is secured because Name-

Value pairs cannot be seen in location bar of the web browser.

4. If post method is used and if the page is refreshed it would prompt before the request is resubmitted.

5. If the service associated with the processing of a form has side effects (for example, modification of a database or subscription to a service), the method should be POST.

6. Data is submitted in the form as specified in enctype attribute of form tag and thus files can be used in FileUpload input box.

What are the Terminologies I should know?HTTP Status Codes In HTTP/1.0 and since, the first line of the HTTP

response is called the status line and includes a numeric status code (such as "404") and a textual reason phrase (such as "Not Found").

1xx Informational 2xx Success 3xx Redirection 4xx Client Error 5xx Server Error

URL: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

What are the Terminologies I should know?Throughput Capability of a product to handle

multiple transactions in a give period.

Throughput represents the number of requests/business transactions processed by the product in a specified time duration.

As the number of concurrent users increase, the throughput increases almost linearly with the number of requests. As there is very little congestion within the Application Server system queues.

What are the Terminologies I should know?Response Time It is equally

important to find out how much time each of the transactions took to complete.

Response time is defined as the delay between the point of request and the first response from the product.

The response time increases proportionally to the user load.

What are the Terminologies I should know?Tuning Tuning is the procedure

by which product performance is enhanced by setting different values to the parameters of the product, operating system and other components.

Tuning improves the product performance without having to touch the source code of the product.

What are the Terminologies I should know?Tuning It is a process of identifying and eliminating

bottlenecks Performance tuning of Webbased systems is a

multivariable and iterative process It is a collaborative effort between cross functional

teams Performance tester is an enabler since he provides

the necessary information in an efficient manner

What are the Terminologies I should know?Benchmarking A very well-improved

performance of a product makes no business sense if that performance does not match up to the competitive products.

A careful analysis is needed to chalk out the list of transactions to be compared across products so that an apple-apple comparison becomes possible.

What are the Terminologies I should know?Transaction Mix

CRUD Operations Mission Critical Transactions High Frequency Transactions

Search

Buy

Hack

What are the Terminologies I should know? User Mix

User level and their roles No. of users accessing the application

Admin

Power

Normal

What are the Terminologies I should know? User Think Time

Time that user think to perform any event How long do user think while navigating from one page to

another?

5 sec

random

fixed

What are the Terminologies I should know? Ramp-up Pattern

How many user should login to application at a time?

Time

User Load

Defining Performance Testing Again The testing to evaluate the response time (speed),

throughput and utilization of system to execute its required functions in comparison with different versions of the same product or a different competitive product is called Performance Testing.

Performance testing is done to derive benchmark numbers for the system.

Heavy load is not applied to the system

Tuning is performed until the System Under Test (SUT) achieves the expected levels of performance.

Why Performance Testing? The week of Feb 6, 2000:

Hackers delivered over 1-Billion transactions concurrently to each one of these sites.

YahooeBayBuy.comAmazoneTrade

How did you think they performed ?

Why Performance Testing? Is your application scalable ? How reliable is it? How fast is it? How secure is it? What are other performance bottlenecks?

“Only conducting Performance testing at the conclusion of system or functional

testing is like ordering a diagnostic blood test after the patient is dead.”

Scott Barber

Why Performance Testing?

Why Performance Testing? Identifies problems early on before they become

costly to resolve. Reduces development cycles. Produces better quality, more scalable code. Prevents revenue and credibility loss due to poor

Web site performance. Enables intelligent planning for future expansion. To ensure that the system meets performance

expectations such as response time, throughput etc. under given levels of load.

Expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.

What are the Performance Bottlenecks?

Why do we have bottlenecks ? People

Reactive than proactive, Not Thinking About “Load” or “Scalability” or

“Performance” During Des & Dev Limited Expertise in System Design / Development

Process Poor Architecture and System Design Changing requirements Changing project plans

Technology Inadequate Hardware Inadequate Network Capacity Poor Implementation / Tuning of Databases and

components

What are the contributors?

What are the types of Performance Testing? Load Vs Stress Testing Goal based Performance Testing Reactive and Proactive Performance

Engineering Performance Validation, Performance Testing

and Performance Engineering/Tuning Other Types - Soak Testing / Endurance

Testing, Spike Testing, Scalability Vs Stability Testing, Application Speed Vs Application Scalability, Scaling Up Vs Scaling Out

What are the types of Performance Testing? Load Vs Stress Testing

What are the types of Performance Testing?

Load Testing Process of exercising the system

under test by feeding it the largest tasks it can operate with.

Constantly increasing the load on the system via automated tools to simulate real time scenario with virtual users.

Examples: Testing a word processor by

editing a very large document. For Web Application load is

defined in terms of concurrent users or HTTP connections.

What are the types of Performance Testing?

Stress Testing Trying to break the system under

test by overwhelming its resources or by taking resources away from it.

Purpose is to make sure that the

system fails and recovers gracefully.

Example: Double the baseline number for

concurrent users/HTTP connections. Randomly shut down and restart

ports on the network switches/routers that connects servers.

What are the types of Performance Testing? Goal based Performance Testing

Reactive and Proactive Performance Engineering

What are the types of Performance Testing? Performance Validation, Performance Testing

and Performance Engineering/Tuning

DB

Web Server (s)

App Server (s)

What are the types of Performance Testing?

When is it required?Design Phase:

Pages containing lots of images and multimedia for reasonable wait times. Heavy loads are less important than knowing which types of content cause slowdowns.

Development Phase:

To check results of individual pages and processes, looking for breaking points, unnecessary code and bottlenecks.

Deployment Phase:

To identify the minimum hardware and software requirements for the application.

What should be tested? High frequency transactions: The most frequently used

transactions have the potential to impact the performance of all of the other transactions if they are not efficient.

Mission Critical transactions: The more important transactions that facilitate the core objectives of the system should be included, as failure under load of these transactions has, by definition, the greatest impact.

Read Transactions: At least one READ ONLY transaction should be included, so that performance of such transactions can be differentiated from other more complex transactions.

Update Transactions: At least one update transaction should be included so that performance of such transactions can be differentiated from other transactions.

What should be my Performance Testing Process?

What should be my Performance Testing Process?

Determine the performance testing objectives

Describe the application to test using a application model

1. Describe the Hardware environment

2. Create a Benchmark (Agenda) to be recorded in Phase 2.

A. Define what tasks each user will perform B. Define (or estimate) the percentage of users per task.

1.Planning1.Planning

What should be my Performance Testing Process?

Record Record the defined testing activities that will be used as a

foundation for your load test scripts. One activity per task or multiple activities depending on user

task definition

Modify Modify load test scripts defined by recorder to reflect more

realistic Load test simulations. Defining the project, users Randomize parameters (Data, times, environment) Randomize user activities that occur during the load test

3.Modify3.Modify2.Record2.Record1.Planning

1.Planning

What should be my Performance Testing Process?

Virtual Users (VUs): Test Goals

Start: 5 Max Response Time <= 20 Sec

Incremented by: 5

Maximum: 200

Think Time: 5 sec

Test Script:

One typical user from login through completion.

4. Execute4. Execute

What should be my Performance Testing Process?

Monitoring the scenario: We monitor scenario execution using the various online runtime monitors. 

Analysing test results: During scenario execution, the tool records the performance of the application under different loads. We use the graphs and reports to analyse the application’s performance.

6. Analyze6. Analyze5.Monitor5.Monitor

Why should I use Tool ? Manual Testing Limitation

Testers

Load Generation System Under Test

Do you have the testing resources?• Testing personnel• Client machines

How do you coordinate and synchronize users?

How do you collect and analyze results?

How do you achieve test repeatability?

Analysis?

123.20

All of you, click the GO button again

Web server Database server

Coordinator

Why should I use Tool ?

Manual Testing LimitationsExpensive, requiring large amounts of both

personnel and machinery.Complicated, especially co-ordinating and

synchronising multiple testers Involves a high degree of organization,

especially to record and analyse results meaningfully

Repeatability of the manual tests is limited

Why should I use Tool ? Benefit of Tool

Load Generation System Under Test

Web server Database server

Vuser host

• Controller manages the virtual users• Analyze results with graphs and

reports

• Replaces testers with virtual users

Solves the resource limitations

• Runs many Vusers on a few machinesAnalysis

Controller

Why should I use Tool ?Using Automated Tools Reduces personnel requirements by replacing human

users with virtual users or Vusers. These Vusers emulate the behaviour of real users

Because numerous Vusers can run on a single computer, the tool reduces the amount of hardware required for testing.

Monitors the application performance online, enabling you to fine-tune your system during test execution.

It automatically records the performance of the application during a test. You can choose from a wide variety of graphs and reports to view the performance data.

Because the tests are fully automated, you can easily repeat them as often as you need.

What should be my tools combination?

What tools are available and what should I use for Performance Testing?

ToolVendor Test Suite - Companion Tools

QALoadCompuware QACenter Enterprise Edition+

LoadRunnerMercury Quality Center

e-LoadEmpirix e-TEST suite

Rational Performance Tester

IBM Rational Suite

WebLOADRadView TestView Suite

SilkPerformerBorland(Segue)

SilkCentral Test Manager

Apache JMeter™Apache ?

Rational Performance Tester

IBM Rational Suite

What are the Performance Metrics? Client-side V/s Server-side Performance

Metrics Server-side Metrics – Application level and

OS level Client side – Response Time, Pass Fail

Statistics, Throughput, Hits/sec, Customized Functionality Validation Counters, User Abandonment

Server-side – OS – CPU, Memory, Network and Disk I/O

Server-side – Performance related metrics and observations for Web Server (s), Application Server (s) and/or Database Server (s)

What should be my best approach? How do we find and fix the problems? The

framework? PE FRAMEWORK:

Performance engineering is the process by which software is reviewed, tested and tuned with the intent of realizing the required performance. This process aims to optimize the most important application performance trait, user experience.

What should be my best approach?

Let’s Re-cap What is Performance Testing? What are the Terminologies I should know? Why Performance Testing? What are the Performance Bottlenecks? Why do we have bottlenecks ? What are the contributors? What are the types of Performance Testing? When is it required? What should be tested? What should be my Performance Testing Process? Why should I use Tool ? What should be my tools combination? What tools are available and what should I use for Performance

Testing? What are the Performance Metrics? What should be my best approach?

THANK YOU

Rohan Bhattarai