Sonam Chauhan

Corporate Expresssonam.chauhan@ce.com.au

webMethods Testing at Corporate Express

Overview

•Introduction•Change Process•General Requirements for Testing•Old Testing Process•New Testing Process•Implementation Tips•Future Steps and Trends•References

Introduction

•Constant pressure for change

•Changes need to be tested for bugs

•Testing has business impact–Good testing costs money–Bad testing costs more money!

•Need to find the “Golden Mean”

Introduction

•webMethods issues that relate to testing•Dev/Test/UAT/Live Builds Much

Variation»Due to IS’s multiple packages feature

•Source Code Management Rudimentary»Support planned for interfacing to SCMs

•Complex Change Deployments Risky»Partially addressed by wMDeployer

•Testing Support Absent»wM aren’t adding it

Change Process

Requirements Definition

Requirements Specification

Development

Review and Testing

Change Deployment (Dev Test UAT Live)

Changes

ValidatedChanges

General Requirements for Testing

•Like development, testing is an engineering skill

–Allocate limited resources for max. benefit.•Black-box Integration testing for Regressions

Best return for your testing buck–High level tests simulate live use–Tests can be external to code

•Easy to retrofit to existing codebase–Regression are the costliest bugs

•All changes must “first do no harm”–This is not Unit Testing

•Unit tests can’t assure system functionality•This is integration/system-level testing

General Requirements for Testing

•Developers can be testers!–Commitment to quality

•Make test development part of development process•Make reviews of the test code part of code review process

–Automate testing•Automated Regression Testing Harness

–Without automation, testing is tedious work•Automation increases productivity and efficiency

–Permanently add hard-won insight to corporate memory•And replay it on demand

–Essential for managing change in complex systems–Active testing catches problems early

•Load testing often very important–Some bugs can only be "teased out" under load.

Old Testing Process

•Implemented Circa 2001•Mostly manual•Repository of test cases •Test case input submitted

to system being tested•System output compared

to stored test case outputs and “Diff” generated (image at right)

•“Diff” manually inspected and signed off

Old Testing Process

• Advantages– Thorough Inspection of Changes

• A human eyeballs all differences

• Disadvantages – Expensive

• Due to tedium, testing is only done when required

– Not Scalable• Testing time increases significantly as platform grows more complex

– Limited Coverage • No load testing • Fixed issues are not re-tested

– Errors• Quality of QA suffers as most testing is manual and tedious

New Testing ProcessThe ‘metatest’ Testing Architecture

•Implemented Circa 2004•Drivers:

–Increasing Complexity–Need for Automation

•‘metatest’ Testing Architecture–Perl command line engine–Open source components–Controls test engines that

execute parameterized test scripts–Supports multiple test

engines, test parallelization (in addition to threading within tests), test retargeting and test load variation

–Test scripts execute sequence of steps, with assertions made at end of each step.

–Entire test runtime and test script database in SCM system (CVS)•Effort: 1 man-months (excluding

test scripts) with one developer (me) •Enhanced over a year

New Testing ProcessSample Test Script – JMeter Test Engine

Load testing part of same

test script

Documents retrieved from

TN database to test document

transform assertions

•JMeter [4] test scripts are executed by JMeter instances controlled by metatest

New Testing Process• Advantages

– Cheap• Open-source software, no new hardware cost• Completely test automation - no tedium

– Scalable• No time increase with platform complexity - tests run in parallel

– Coverage • Includes load testing• Can run selective tests, or run regression test an entire system in 5 minutes

– No Human Errors• Quality of QA improves

– New Benefits• Detailed test results, Test Driven Development, Continual testing

• Disadvantages– Typically, only specific assertions are automatically proven

• All differences not examined – this may be mitigated by using ‘full-document’ assertions regular expressions adjusting for variances

•Automatic archival of detailed results

•Data useful for:–Load and stress

performance validation (eg: image on the right) [5]

–Capacity planning–Trend Detection–Analysing software

component failure rates •New analytical

methods can be applied to stored data in future

New Testing ProcessNew Benefit – Detailed Test Results

•While not Unit Testing, our approach has several of the benefits of ‘classical TDD’:

–Tests and code concurrently developed–Bugs replicated as tests and fixed–Bugfix tests added to test database–Simple tests–Brief run time for tests–Immediate testing feedback

New Testing ProcessNew Benefit –Test Driven Development

•Continual Testing –Dev/Test/UAT tested nightly–Test & UAT also load tested

(Load imposed = 4 x production load)–Results emailed to

developer list so they see it first thing in the morning (see image on right)

•Benefits–All platforms fully regression

and load tested each night–Differential testing –

identical tests run on different systems each night. When the results differ, bugs are identified by comparing the two systems. In example at right, failures are only on ‘test’ in ‘load’ testing…

–The load imposed in load testing is updated to reflect growth. Thus, bottlenecks are identified before they hit production.

New Testing ProcessNew Benefit – Continual Testing

•Documentation very important–Documentation crucial to getting developer buy-in–‘metatest’ architecture has 35 pages of user documentation (in addition

to documentation for the tests themselves)•When using open source components, expect bugs

–Interact with developers of the component–Contribute if possible–Document problems, link to Bugzilla DBs etc

•Version control everything to do with testing –Test scripts–Even test engines binaries–Advantages: Branching/ Merging, new developer can be setup in

minutes•Leverage existing infrastructure

–Most software can simulate endpoints. For eg: WM C1 OnRamp can create a CommerceOne envelope from XML•Get a domain expert to write your tests

Implementation Tips

Future Steps and Trends

•Plans to improve the ‘metatest’ architecture–Implementing other types of testing–Integration with WmDeployer –Integration with CECM Verification Tool [6]

•Other Testing Resources in the WM space–Customware’s WmUnit [1]–Wmusers.com article on testing using JUnit [2]–Solstice Software [3]

Future Steps and Trends

•webMethods Testing Support–WM response to group request for testing

support: “Currently no plans exist add unit testing or test service generation services to the developer.”

–However, WM’s IDE is moving to the Eclipse platform. This should open up interesting possibilities with Eclipse plug-ins.

–Server support: Ideally, on throwing an particular exception, a WM server would save the pipeline in a testcase format for bug replication on development server. (WM tool does this already?)

References

[1] Customware website: http://www.customware.net/

[2] wmUsers.com testing article: http://www.wmusers.com/ezine/2003feb_parvanitis_1.shtml

[3] Solstice software website: http://www.solsticesoftware.com/

[4] JMeter website: http://jakarta.apache.org/jmeter

[5] Article on analysis of JMeter logfiles in stress testing:

http://wiki.apache.org/jakarta-jmeter/LogAnalysis

[6] CECM change verification tool described in:

http://www.wmusers.com/software/uploads/2820B2B_Change_Management_Presentation.ppt