1

“As a part of Liberty Mutual Information Systems enable Liberty Mutual Group to exceed their business objectives by delivering high value, market responsive IT solutions”

Testing During The Development Phase Using Open Source Tools

March 2008

Richard Thompson (Verification Technologist)

2

Agenda

IntroductionWhy Open SourceExamples Through the LifecycleLessons in Implementing Open Source SuccessfullyQuestions

3

Overview of Liberty-IT

Belfast based software development & testing house with ~230 staff Our parent company is Liberty Mutual Group

a leading global insurerLiberty Mutual ranks 95th on the Fortune 500 list of largest U.S. corporations (based on 2006 revenue). $85.5 billion in consolidated assets

Technologies includeJava, .Net, Struts, IBM PortalHibernate, Spring, Web Services -> SOAMultiple database platformsDedicated Testing practice focusing on technical testing solutions

4

Disclaimer

The views expressed within this presentation belong solely to Richard Thompson and do not necessarily represent those of Liberty Mutual or Liberty-IT.

5

Why Open Source?

6

Business Demands

FasterStrategic advantage to be first to marketQuarterly -> Monthly -> Fortnightly Releases

BetterLower and lower tolerance for poor qualityExpect higher productivityDemand more visibility into project status

CheaperContinually looking to reduce costsEverything must have a ROI (Return on Investment)

7

Perception of high vendors pricesGrowing number of high quality, mature open source products

Which are highly reliableHave large world wide support

More and more large companies are using Open Source tools/ components

AmazonGoogleIBMConducted by Boston-based Forrester Research Inc. in February 2004, a study revealed that 65 of the 140 companies questioned -- almost 50 percent -- are using open source applications.

Growing Role of Open Source

8

Examples Through the Lifecycle

9

Open Source Solutions

Unit Testing JUnit, HttpUnit, HTMLUnit, MaxQ

Automated Functional/ Regression Testing

Watir, Samie, Pamie, Jiffie

Performance Testing

OpenSTA, JMeter, WebInject, Grinder

Web Service Testing SoapUI, HttpUnit, XMLUnit

Code CoverageEmma, Cobertura

Test Management TestLink,Test Case Web

Static Analysis CheckStyle, PMD, FindMoreBugs

Continuous Integration

Cruise Control

Configuration Management

CVS, Subversion

Build Management ANT

Project Management XPlanner, dotProject

Application Monitoring

Big Brother, Nagios

Defect Tracking Mantis, Bugzilla

10

RequirementsRequirements DevelopmentDevelopment

Unit/ComponentTesting

RequirementsReview & Testability

IS TestingIS Testing ProductionProduction

Performance Testing

Smoke Testing

Application Monitoring

Defect Tracking

Reports

SystemTesting

Code Review & Static Analysis

PostImplementation

Validation

Code Coverage

Business TestingBusiness Testing

Acceptance Testing

RegressionTesting

Testing Through the Lifecycle

11

Automated Unit Test Reporting

JUnit• Simple framework for writing automated tests• Written in Java to test Java, so Developers only need to know 1 language• Most widely used automated unit testing framework for Java• Provides fast/ repeated feedback on quality of code

12

PMD – Static Analysis

PMD scans Java source code and looks for potential problems like:•Possible bugs - empty try/catch/finally/switch statements •Dead code - unused local variables, parameters and private methods •Suboptimal code - wasteful String/StringBuffer usage •Overcomplicated expressions - unnecessary if statements, for loops that could be while loops •Duplicate code - copied/pasted code means copied/pasted bugs

13

Continuous Integration Process

Code Repository

Change Request

Cruise Control Server

(Continuous Build & Test)

Build(Automated Build

Script)

Automated Unit Test

Automated Code Analysis

1. Code Coverage, 2. Static Analysis, 3.Coding Standards

Reporting DB

Developer

14

Code Coverage

15

Cruise Control Dashboard

16

Continuous Integration – Daily builds

17

Pre-commit validationEnsures that comment containsa) BUGID:b) BUGNOTE:c) TEST PLAN/DESCRIPTION

Post-commit validationWrites out the followinginformation to a DB tablea) BUGID:b) BUGNOTE:c) TEST PLAN/DESCRIPTIONd) CVS Committere) Files updated (with full path)

CVS

Comment OK

Comment InvalidDeveloper

Checks In code

MantisBugnote

CVS Check-In Process

18

Watir – Browser AutomationWatir•Web Application Testing in Ruby •Watir is a toolkit used to automate browser-based tests in Internet Explorer using IE’s COM interface

19

OpenSTA•Open Source alternative to LoadRunner/ Rational Performance Manager•Large community, excellent online documentation•Option to purchase guaranteed support (for a price ☺)

Performance Testing

20

Lessons In Implementing Open Source Successfully

21

Factors to consider when selectingMain Places to look

Sourceforge (http://sourceforge.net/ )Freshmeat (http://freshmeat.net/ )

What to look forActive communityRegular releases> 90% ActivityDocumentationDo the technologies used match those used within the company?

Perform a PilotTry on a single project, review lessons learned, move forward

Determining ROIMeasure cost to implement on project verses benefits gainedAlso factor in ongoing maintenance

ImplementationTypically requires a higher level of technical ability than Vendor solutions

22

• Perform a pilot implementation and measure the setup time, learning curve & value added

• For example: Piloting Watir on a particular project resulted in:• 38% less defects identified during Business Testing

• Testing Coverage significantly increased

• Developers free to perform other activities

• Automated tests reused within Regression & Smoke Testing

• Reduction in Defects found in Production

Pilot & Measure The Value

0102030405060708090

Requirements Development Testing Production

%

MayAug

23

Questions?