IBM Testing Services © 2008 IBM Corporation @ 2009 IBM Corporation Amitabh Kaushal – Performance...

IBM Testing Services

© 2008 IBM Corporation

TTQ

@ 2009 IBM Corporation

Amitabh Kaushal – Performance Test ManagerVivek Sharma – Performance Test Lead20 November 2010

Performance test automation using IBM Rational BuildForge and IBM Rational Performance Tester


© 2009 IBM Corporation2

Agenda

Case Study

Introduction & Background

Performance Test Life Cycle

Urgence Performance Framework

Automation of Performance Test

Test Execution Sequence

Comparison with Urgence Performance Test Cycle

Major Benefits of using Urgence Framework

Summary



Largest European Bank in UK : Case Study

ClientBusiness

Areas ServedEngagement Details

Largest European

Bank

Trades & Settlement

Lifecycle - Trade Capture & vesting,

Messaging like SWIFT, CREST

Positions & Balance,

Settlements,

Statutory reporting and Reconciliations

Global Markets

Derivatives

Type of Service : Managed Test Services

IBM is providing managed testing services, to assist one of the largest European bank in performing offshore Managed Testing Services for trading and settlement application In financial market service.

IBM responsibility:

Provide expertise in testing transition in strategic level

Responsible for transition planning and execution

Help obtain knowledge transfer and documentation from Customer during transition period

Designs test process and framework

Formulate test strategy, test plan and requirement traceability

Develop test scenarios, functional test cases, test scripts, execute testing, record and compile results

Performance Testing, Test Automation and Regression improvement

Day-to-Day management of testing services

IBM has successfully completed testing of several releases of developed applications

Benefits realised

1. 50% Reduction in Performance Regression Test Execution Timelines

• Indexing/cataloging of Regression Test Pack

• Complete Automated Solution• Effective risk-based approach

2. SLA-Based Performance KPIs

• Test Quality• Schedule Adherence• Test Effectiveness

3. Significant Knowledge Management Improvements

• Reduction in reliance on Customer’s SMEs

• Greater flexibility in test execution

4. On-track for 16% reduction in fixed-costs of Service

5. Service Improvement initiatives around

• Performance Testing effectiveness and methods

• Further Test Automation in Other projects

• Driving Quality Methods further up the SDLC



Introduction and Background

Customer’s application is a ‘Trades Settlement Platform’ which processes ~2.5M trades/day, supports 33 Back Offices in all major financial Geographies, Europe’s biggest Oracle Database

IBM currently delivers a range of Testing Services for this Platform. Customer challenged IBM to provide “… specialised, on-demand, repeatable and cost effective (offshore) performance test service for this application. This will be a different delivery model to SIT, Regression testing etc.”

IBM did deliver a repeatable, on-demand and cost effective Performance Regression Testing solution with the following attributes:

– Cover end to end Performance Test Lifecycle (Test Preparation – Shakedown – Execution – Interim Reporting - Final Reporting), using IBM Rational tooling and IBM Research innovation.

– Test Phase duration got significantly reduced (target was 50%)

– An automated “chassis” with rules-based Environment deployment, performance tests automation capabilities, threshold driven automated report generation.

– Based around the capabilities of IBM Build Forge product which forms the foundation of the Service

– Uses IBM's Rational Performance Test (RPT) tooling for test execution of the online testing functionality

– Combines existing monitoring augmented by Unix level Monitoring tools / integration with Optier-Corefirst / online transaction timing metrics.

– Combined uses of Open Source tool with others to monitoring server side metrics for analysis and scalability - PMRE, SQL Developer, Oracle OEM, NaGIOS etc.



Performance Test Life Cycle

Performance RiskWorkshop

WorkshopExit Report

Test Preparation

C1 ImplementPERFORMANCELC

Detailed PerformanceTest Plan

C3C2 Report

The performance test lifecycle primarily consists of:

1. Performance Risk Workshop to decide the scope of the performance testing and the test mitigation for the risks the project will introduce.

2. Workshop Exit Report

3. Detailed Performance test Plan which consists of the scope, approach, risks, dependencies and estimated schedule for the performance test.



Deploy Test Data

Install Test Tools

Validate Test Environment Readiness

Generating Test Reports

GUI Application

Define and use Build Forge to set up Test environment

Storage of RPT scripts and data

Preparation, Execution and monitoring of Performance Test

BF AgentBF Agent

Rational Build Forge

Test Data

Test preparation, Execution and Monitoring are automated with Rational Build Forge and Rational Performance Tester

Rational Performance Tester

Rational Performance Tester

Performance Test EnvironmentSolaris, Win, Linux boxes with

Database Servers

Performance Test EnvironmentSolaris, Win, Linux boxes with

Database Servers

Urgence Performance Framework

Open Source Tool for

monitoring

Open Source Tool for

monitoring



Automation of Performance Test – Preparation, Execution, Monitoring

Test Preparation

• Prepare test input files using IBM Build Forge• Execute Performance Health Check using IBM RPT, BuildForge, Open

Source tool for monitoring • Environment validation – Automatically raise JIRA/GIMS (incident

management system) if required processes are not running in the test system Test Execution

• Schedule multiple Test runs for various regions – APAC, EUROPE, USA• 12-20 hrs online tests running in-parallel with IBM Rational Performance

Tester. This whole end-to-end execution is fully automated now. Test Monitoring & Reporting

• Continuous monitoring of different databases to assess the trade flow using OEM, SQL Developer, build forge

• Server resource utilization monitoring using PMRE and native Solaris commands - Prstat/Top, IOstat, netstat. Use in-house shell scripts to capture channel throughput.

• Raise JIRA automatically and send notification mail to stakeholders.



T: 0 T: 1 T: 2 T: 3

eSpear Required Process Monitoring ( 10 minute? Loop)

Server Monitoring ( 5 minute ? Loop)

T: 4 T: 5 T: 6

Online RPT Testing

TK Batch

T: 7 T: 8 T: 9 T: 10 T: 11 T: 12 T: 13 T: 14 T: 15 T: 16 T: 17 T: 18 T: 19 T: 20 T: 21 T: 22 T: 23 T: 24

HK Batch

UK Intraday

UK Settlement

UK Monitoring

UK Batch

US Intraday

US Monitoring

US Batch

Batch Results

TK Intraday

TK Monitoring

HK Intraday

HK Monitoring

TK Settlement

Intraday Results

Online Test Results

Urgence Test Execution SequenceFollowing scheduled tasks are driven by Buildforge – comprises of the entire performance test steps

Test Execution Scheduling

Test execution consists of running online tests for 12 hrs in parallel to the intraday tests for various regions at scheduled time.

The servers are monitored the entire duration of the tests to view CPU utilisation.

These tasks are sequenced as shown and executed at the specified time in Build Forge.



StepPlanned

Start Time (BST)

PlannedDuration (HH:MM)

Actual Start Time

(BST)

Actual End Time

(BST)

Prerequisite Steps

Owner JIRA Execution - Online, Intraday (TK/HK/UK/US), Batch, Settlements Status Assumptions/Comments

83 4:20 0:10 3:54 5:11 NA IBM Urgence POPS-23286Raise JIRA to Bounce web Applications & Clusters hosted on eSPEAR App Servers for Performance Test

Completed

84 4:30 0:30 3:54 5:30 83 DSOPS DSOPS will bounce the server and notify us. They will take help from TA team if needed. Completed

85 4:30 0:05 4:51 NA IBM Urgence Check Business dates for all regions Completed86 4:35 0:05 4:52 85 IBM Urgence Check Public Holiday for current week Completed

87 4:40 0:05 86 IBM UrgenceIf today's date (Full-run) is a Public Holiday for any region then raise JIRA to change that day to normal business day.

NA

88 4:45 0:15 87 DSOPS DSOPS will change the day as business day from holiday and notify us. NA

89 5:15 0:05 3:40 3:46 33 IBM Urgence OETA-3919Check status of Channel Space JIRA raised against TA on T+1 day, update it for T+2 day channel status

Completed

90 5:20 1:00 5:20 89 TA OETA-3919 TA will check the channel space and purge the data, if needed. Completed91 5:20 0:15 4:58 5:05 NA IBM Urgence Check BF Agents Status on all boxes in UAT Completed92 5:35 0:30 91 TA Check BF Agents Status - If down, escalate it to TA NA

93 5:25 0:10 5:01 5:03 76 IBM Urgence Check the status of Apache Servers, If server status is down, raise OETA JIRA against TA Completed

94 5:35 0:05 93 TA TA will bring up the Apache server and notify us NA

95 5:30 0:15 4:45 5:13 NA DBAs OEDBA-10315Raise OEDBA JIRA to Run DB invalid Report on UAT env and make sure Database is in good health by confirming with Component Team

Completed

96 5:45 1:00 5:25 95 Components If DB Invalid can cause problem, component will deploy recompile baseline. Completed97 5:00 5:05 4:34 4:40 NA IBM Urgence Check for JAVA Poller process status, raise JIRA if needed Completed

98 5:00 0:05 5:34 5:48 NA IBM UrgenceCheck MM task count in table exception_log, mm_open trade, mm_allege, mm_engine_resource and truncate them if needed.

Completed

99 5:00 0:05 5:57 98 DBAs OEDBA-10317 Raise request to DBA team to truncate those table if don’t have access to do so. Completed

100 5:10 0:20 98 IBM UrgenceCheck Task subscribe and take action to clear it, if needed take support from Settlement team

Completed

101 5:30 1:00 100Urgence+TA+s

ettlementMake sure those Task Channel are cleared. NA

URGENCE Performance Test Execution Plan & Activity Checklist



Process Monitoring Reporting Environment Status



Standard Performance Test Cycle

~2.2 weeks cycle time reduced to 5 days

6 days effort saved on plan per Cycle (typically critical-path)

60 days saved on plan per year (based on 4 releases per year) and being utilised for more run now

Within Test Execution Task there can be many ‘runs’, and therefore more robust performance metrics

ID Task Name Start Finish Duration28 Dec 2008 4 Jan 2009 11 Jan 2009 18 Jan 2009

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

2 5d16/01/200912/01/2009Preparation

5 3d30/01/200928/01/2009Reports

3 5d23/01/200919/01/2009Test Execution

1 5d09/01/200905/01/2009Deploy/Refresh

25 Jan 2009 1 Feb 2009

28 29 30 31 1 2 3 4 5 6

4 7d27/01/200919/01/2009Analysis

ID Task Name Start Finish Duration28 Dec 2008 4 Jan 2009 11 Jan 2009 18 Jan 2009

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

2 2d08/01/200907/01/2009Preparation

4 2d19/01/200916/01/2009Reports

3 5d15/01/200909/01/2009Test Execution &Analysis

1 2d06/01/200905/01/2009Deploy/Refresh

25 Jan 2009 1 Feb 2009

28 29 30 31 1 2 3 4 5 6

Urgence Performance



Major Benefits of using Urgence Performance Framework

Re-usability and Modularity

- Used Build Forge feature to create common projects across environments wherever it is applicable

- Parallel execution possible at a time in 2 or more environments

Reduced / Eliminated dependency

- Reduce Support Team and SME involvement for providing result.

- Usage of licensed software like TeamQuest, Loadrunner etc.

- Dependency on Technical Architect and DBA group to provide data and analyse result

- Performance tester involvement in checking process down and raising JIRA accordingly, reduce tester effort

- Resource Utilization (Server Stats) using external tool like – TeamQuest, Analyzer

Automation – Reduction in manual efforts

- Creation of test data (bulk trade files and online data)

- Trade Monitoring (to collect vesting results)

- Use Multi-threading concept to speed up the execution process

- Can run unattended performance test (using scheduling feature in BF)

Central Repository for all test results



With Urgence Framework, Customer has achieved significant…

Reduction in test timelines

Shorter test execution runs (5 days for 3 run)

Fewer attempts to get to first good run

Get the analysis done and find the bottleneck by comparing different run results

Increases time dedicated to analysis

Automated testing will provide answers for “how does release Y equate to release X”

Opens the door to aggressive tuning activities “tweak a variable, run the test, tweak again, run the test”.

Raise the consistency and confidence in Performance testing.

Reduction in human resource costs

Resources running the tests will have lower rates – completely offshore

SMEs will spend less time on Performance Testing

Reduced dependency on Support Team(TA, DBA) to fetch baseline stats

Eliminating manual, error prone tasks

Repeatable and consistent processes for improved efficiency



Summary

Using Build Forge, RPT and Open source tool we have automated the end to end performance testing for customer’s application.

We have demonstrated reduction in overall test cycle time, ability to execute the tests 'On Demand' in a cost effective way.

We have reduced manual efforts by introducing automation (i.e. creation of test data, Trade Monitoring, JIRA, running unattended performance test with the scheduling feature in Build Forge).

Performance test scope is now decided early in the release lifecycle by conducting a ‘risk workshop’ to discuss the non-functional requirements.

Specific deliverables like risk workshop exit report, test strategy, test plan are delivered in different phases of the performance test life cycle and obtained a sign off before the execution of tests.

In Summary, we were able to realise the customer's vision for an 'On Demand' performance test capability.

IBM Testing Services © 2008 IBM Corporation @ 2009 IBM Corporation Amitabh Kaushal – Performance...

Documents

Transcript of IBM Testing Services © 2008 IBM Corporation @ 2009 IBM Corporation Amitabh Kaushal – Performance...