ALM (Application Lifecycle Management)

Practical Application Life cycle Management Framework

- DRAFT Version

Principle ConsultantByungwook Cho C.

([email protected])

AGENDA

• Realistic of software development• Application Lifecycle Management• Practical Application Lifecycle Management Framework• Delivery Practical ALM Framework• Strategy for delivery practical ALM Framework• Case study – TBD

<Insert Picture Here>

Realistic of Software development

Software development

What we want

justwell defined process, well structured managementIn-depth software”

Software development

But what’s real?

Realistic of developer •Additional Requirement•Changing Requirement •Meeting,reporting,education•Communication•Overtime job•Not tracked work

Realistic of manager •Always 99.999..% complete•Always overtime, but what is he doing? •When can I release?•What are risk items?•Wants know status of issue.•Task Priority

Realistic of software • A house of straw• Defects are discovered after freezing code• No testing for changing• Big bag integration

The three little pigs‘A house of straw’

http://www.delmadang.com/cwb-bin/CrazyWWWBoard.exe?db=dmdres3&mode=download&num=5219&file=m_setup.exe


Many Dr. and genius developed many methodologies RUP,CBD,CMMI

Theories

So…

There are only few genius Theory is theory We need more realistic and practical methodology

But…

• Practical development methodology• Erich Gamma, Joel Spolsky, Kent beck, Andrew Hunt• Iterative & Incremental• Based on agile

• Difference between traditional methodology.• Traditional methodology assumes that there are no change

and no defect in each previous step. (eg. Waterfall model)• It assumes there are many potential defects, and focus on

finding the defects by frequent testing. • It assumes there may be requirement changes, and try to

accept the changes.• Focus on collaboration and communication

What is alternative solution?Practical Development

http://www.threeriversinstitute.org/Kent%202.jpg


ALM Application Lifecycle Management

ALM definition

Application lifecycle management (ALM) is the marriage of business management to software engineering made possible by tools that facilitate and integrate requirements management, architecture, coding, testing, tracking, and release management.[1]

• Definition from Wikipedia

• Scope of ALM• Covers whole SDLC• Complex and getting theatrical

Current ALM market

• All of vendor uses ALM as a marketing message• McCave just has testing solution, but they said “We are ALM

company”• HP also has testing oriented solution, but they said “We

have ALM solution”

• There are few pure ALM players but..• Serena, Polarion etc already has matured ALM product.• But it is complex and hard to apply. (needs genius)

• We need more practical ALM framework.• Depending on teams’ maturity level.• Depending on customer’s mega process (methodology)• Depending on budget and time

Four pillars of ALM

• Process– Cover software life cycle from requirement analysis,implementation to release.

• Quality Control– Provide capabilities to assure software quality– For example : testing, short release

• Tracking– Provide visibilities and traceability of project activity.– Provide traceability between deliverable. For example traceability between

requirement to actual code• Collaboration

– Especially collaborate with Biz dept,.– Requirement management– Task management etc

ProcessQualityControl

TrackingCollaboration


Practical ALM Framework

Practical ALM

• Lightweight, more easier and practical• Composed with open source tools and cheap popular

commercial tools– Open source tools Reduce learning curves– Less investment Free or cheap commercial tools

• Based on practical and agile methodology

Practical ALM use case scenarioDeveloper Perspective

Scenario• Developer goes to office in morning.• Run eclipse IDE.• In IDE, there are task list that are assigned to him.• He checks tasks with priority and pick up one.• Change a status of the task to In Progress• Check out code from source code repository.• After implementation, he implement unit test with testing framework.• Execute test with standard build script. It automatically generate test result, code coverage rate and

run code inspection. Code inspection shows illegal naming convention or potential bugs.• He fix the code and re-run the script.• He checks code coverage rate in eclipse IDE. After confirming the code coverage rate is meet target

rate. He commit the code with the task ID.• CI (Contiguous Integration) system get a notify that main source repository has been changed.• It check out the code and run build script.• Compile and deploy the code to staging machine.• Run a test case. If test is broken. It automatically sends a notification to the developer and project

manager. And then roll back the code in source repository and staging machine.• If the test is passed. it invokes code inspection and coverage analysis. If there is a problem (cannot

match up target test code coverage rate or some static bug.) it will also report the problem.• Developer notify that all of implementation has been done successfully.• He write a his work history in task management system thru eclipse IDE. and change a status

to Resolved• Project Manage get a notification the task is resolved. He check up a note of the task and task result

etc. And then change a status of the task toClosed• Developer come back home.

Practical ALM use case scenarioProject Leader/Manager perspective

Scenario• Project manager attend on requirement meeting with customer.• He summarize meeting minutes and extract tasks from that with customer's

agreement (or confirm)• He creates new tasks with priority.• He plans the task followed by project milestone.• He check up developer's load by using task management system to find who is

under lower load.And assign it to appropriate person.• Project manager estimate task implementation time with assigned developers.• Project manager adjust plan with schedule and estimated time.• During the project, project manager tracking the task status and everyday adjust

the tasks.• Project manager or senior developer checks project health status with unit

testing result and code coverage rate.• In addition senior developer concentrate testing in high code complexity area. It

is founded by code complexity analysis report from CI tool's daily build.• Project manager also manage risk by using project status report from task

management tool and reports from CI tools. And he check up high priorityOpened task every day to control risk.

Practical ALM use case scenarioProject Leader/Manager perspective

Scenario• Customer side PM come to office in the morning.• Execute web browser and go to ALM dash board page.• Each project has one page dash board. It shows how many tasks are

opened and closed for each milestone.How many pending tasks are there. How many critical tasks are opened.

• He just notify potential risk and project healthy status by using the dash board.

• Make a meeting with Project Manager or Project Leader to control the risks.

• If he want to track the critical issue (task). He subscribe the task. All of comment and change are automatically reported. So he doesn't need to make a meeting or make a phone call to check up progress of the critical task.

Modules

Extract Requirement

Task Management

Dashboard

Source code management

Contiguous Build

Standard IDE

Standard Build Script

Testing Process

Testing Framework

Static Testing

Defect Management

Wiki based doc mgmt

Forum based bi-directionalcommunication

Code review

•Project management •Provide development environment

•Assure quality •Leverage collaboration and communication

Module 1. Task Management

• OverviewCovers project management like- Project schedule management- Project resource (human) management- Requirement gathering- Provide KPI dashboard for management group

• Scope

TraditionalDevelopment process

ALM / Task Management Process(Scrum based customized for Enterprise)

•Manage major milestone•Define mega process

• Manage daily task• Delicate resource and schedule management

Module 1. Task ManagementExtract Requirement

• Process1. Extract requirement from

meeting with client2. Gather all of conversation

to meeting minutes 3. Summarize meeting

minutes and post it into internal wiki

4. Extract action items and get a agreement from customer.

5. And put that action items into task list.

• Useful tools– Google site : gather all

meeting minutes into one place with jump start

– Atalssian confluence wiki : cheap and powerful.

CustomerCustomer

PLPL

Meeting Minutes

Extract Action ItemEx)AI 1. Implement Logging with JMS (JIRA-101)AI 2. Evaluate Logging architecture in Test (JIRA-102)

Extract Action ItemEx)AI 1. Implement Logging with JMS (JIRA-101)AI 2. Evaluate Logging architecture in Test (JIRA-102)

Task

1.Log whole of meeting1.Log whole of meeting

2. Extract action item in end of meeting.Action item should be agreed between customer and PL

2. Extract action item in end of meeting.Action item should be agreed between customer and PL

WIKIWIKI

MeetingMeeting

Task List or Task Management System

Task List or Task Management System

Register action item to TaskRegister action item to Task

ProcessProcess

Module 1. Task Management Extract Requirement

• Sample

Action items

Task number in task listIt provide traceability from requirement to task

Module 1. Task ManagementTask Management

• Overview• Managing schedule and task of project team• Agile scrum methodology based.• Subset of project management methodology

– Mega process is up to traditional methodologies or customer’s methodologies.

– Cover delicate daily task management only


• Scrum methodology

• Original scrum methodology is not appropriate Enterprise project management• Hard to manage risk.• Customer usually has their own methodology.

• Need to customize for Enterprise project


• Task Management Process1. Prepare product backlog It means requirement list

2. Release Planning – Define major milestone and release

3. Sprint Planning – Planning task for each iteration

4. Track Sprint – Daily tracking progress. Eg. Burn down chart

5. End Sprint – Simply provide demo to customer and get feed back

6. Update Sprint backlog – Update and re priories requirement

7. Retrospective – Review process and mature it


• Task Management ProcessProcess

Duration 4~6 weeksDuration 4~6 weeks

Requirement Use case

(Sub) RequirementTask

Task

Task

AnalysisDesign

AnalysisDesign

ImplementationImplementation

TestingTestingWorking Item

Documentation

Meeting

TestingInternal ReleaseQA has to test every internal release based on test plan

Internal ReleaseQA has to test every internal release based on test plan

General Available(Release)

General Available(Release)

High RiskMandatory FunctionsNon-Functional RequirementMajor UI

High RiskMandatory FunctionsNon-Functional RequirementMajor UI

Low RiskOptional FunctionsEnhancement

Low RiskOptional FunctionsEnhancement

Scheduling(Before implementation phase)

Scheduling(Before implementation phase)

WikiWiki

Task Management SystemTask Management System

Tracking in WBS (Link to MS Project)Tracking in WBS (Link to MS Project)

Use numbering in title to track of RequirementUse numbering in title to track of Requirement

Task

Task

TaskTask

LINKLINK

ReleaseRelease

Sprint Sprint Sprint

Requirement

Requirement

Requirement

Requirement

Requirement

Requirement

Meeting minutes

OptionalDepends on requirement granularity

OptionalDepends on requirement granularity


• Task Type– Task is real working item to implement requirement– Task has to have traceability to requirement– Task type definition is very important.– Task can have hierarchy.

System Task Sub Task Description

Requirement N/A Detailed functional requirement.Requirement cannot contain any child requirement. It is leaf node of requirement tree. Name of requirement has to have number to provide traceability.Ex) 2.1.1 Logging inbound transaction

WorkingItem Working items for developer. Eg. Coding,debugging etc.

Documentation Write deliverables as document on wiki or document

Test Develop and execute test of “Requirement”. It includes implementing Unit Test,design, execute etc.

Meeting Meeting about specific issue

Bug N/A Bug reported by somebody(not a author) after every internal release phase.

[!] Requirement can be separated into more detail sub requirement


• Workflow– Workflow defines task handling process.– It is very important for task management– Keep it simple and easy to understand– It needs to define task type and hierarchy


• Workflow- Requirement management process

Open Scheduled In Progress Resolved Closed

Customer decide to close the child requirement based following decision factor. 1. Code coverage rate2. Code inspection result3. Deliverables4. Unit Test Result

Customer decide to close the child requirement based following decision factor. 1. Code coverage rate2. Code inspection result3. Deliverables4. Unit Test Result

PLPL

PLPL

PLPL

PLPL

CustomerCustomer

Based on technical use case.Extract Child requirement and

register it into JIRA

Based on technical use case.Extract Child requirement and

register it into JIRA

Scheduling child requirement based on iterative development. It is linked to WBS in this phase

Scheduling child requirement based on iterative development. It is linked to WBS in this phase

In Every iteration, when the requirement related sub-task is started, change status to In Progress

In Every iteration, when the requirement related sub-task is started, change status to In Progress

After finishing all of sub tasks for the child requirement. Change the status to “Resolved”

After finishing all of sub tasks for the child requirement. Change the status to “Resolved”


• Workflow- Task management process

Open Assigned In Progress Resolved Closed

PLPL

Need more

Information

Postponed

PL creates detail task for developer.Task is usually finished 1~4 day. (1~2 days are recommended)

Discuss with developer and estimate time.Assign the time to task

PL creates detail task for developer.Task is usually finished 1~4 day. (1~2 days are recommended)

Discuss with developer and estimate time.Assign the time to task

DeveloperDeveloper

DeveloperDeveloper

DeveloperDeveloper

DeveloperDeveloper

PLPL

PLPL

Developer discuss with PLPL lets developer to postpone the sub task.The postponed task should be finished in same iteration phase. Or closed

Developer discuss with PLPL lets developer to postpone the sub task.The postponed task should be finished in same iteration phase. Or closed


• Mega Process Summary


• Implementation– Jump start with Microsoft Excel

- Product back log

- Sprint back log


• Implementation– Build task management system with Atlassian product

JIRA : for task management (2400 USD for professional version)

Confluence wiki : for requirement management (2200 USD for 50 user)

zAgile portal : Integrate distributed view MS project : Manage schedule by PMO perspective

Reference Architecture

http://www.atlassian.comhttp://www.zagile.com

http://www.atlassian.com/

http://www.zagile.com/


• Implementation– Build task management system with VersionOne product

VersionOne provides hosted service. (No install is required, just use it)

It is based on agile methodology. (Atlassain JIRA is started from bug tracking system)

Sometimes hard to integrated with testing framework, CI tools, SCM etc.

http://www.versionone.com

Module 1. Task ManagementDashboard

• Overview– Provide personalized view for each stakeholder– Show KPI for management group (Very important to convince

manager)

Progress Risk items Velocity of team Workload for resources Escalated issues

– Has to be integrated with other ALM modules Test Result Build Status Test coverage rate

– Just use only meaningful KPI“Patterns in our team velocity and temperature of Seoul are

same..?! “


• View For project manager or customer

– Burn down chart– Task or Bug open/close progress– Risk item list– Escalated task list

For project leader or manager in small project– Build status– Test result and test coverage rate– Workload of team members

For developer– Assigned task– Project status (Burn down chart, build status) / To

understand whole of project status as a project team member


• Implementation

Task Management System

Contiguous IntegrationSystem

Testing Framework

DashboardOPEN APIFILE

PORTALOR

WIKIOR??

•Personalization•Integrated view


• ExampleDashboard with Atlassian JIRA

Number of task by priority

Number of task by status

Progress of each iteration

TASK CREATE/CLOSE TRACKING

Workload of team members


• ExampleDashboard with Inflectra/Spira Team

Module 2. Build EnvironmentSource Code Management

• Source Code Management (SCM)• Share source code• Track change of source• Manage branch


• Branch management strategy

Tagging

Branch

Tagging every build

Every release

Trunk(main branch)

•Merge heuristically from Release Branch.•Merge systemically from patch branch.


• Branch management strategy• Release branch

• This branch is created when software has been released. (Even if it is internal release,manage branch)

• Bug fix and enhancements of this version are applied to this branch.

• Bug fix and enhancement of this version are merged to trunk branch by heuristically. (If released versions are different, logic or architecture can be changed, so not to use system merge)

• Working branch• When some co-work is required and it will not be commited during

a day, make a working branch.

• Enhancement or patch branch• To make temporary patch or evaluate enhancement, make patch

branch before applying fix to trunk branch.• It can be applied by using simple system merging.

• Making a new branch provides additional managing effort.

Module 2. Build EnvironmentStandard IDE

• Standard IDE• Enforce all of developer to develop on same

environment- Reduce error from environment

• Save time to set up the environment• Process

1)Developer download standard development environment from wiki

2)Just unzip the file to personal computer.3)Set up personal preference (server ip, directory

structure etc )4)And followed by instruction in wiki, quick start to

develop

Module 2. Build EnvironmentStandard IDE

• Standard IDE Packaging

IDE

JVM

Test Framework Std build script

JEE Server

Property file

Install Guide

Development Framework

Zipped achieves

Module 2. Build EnvironmentContiguous Build• CI Process

Check out source

Build

Packaging

Deploy

Testing& coverage anaysis

Inspect code

Roll back code

Tagging

Reporting

Send Notification

Build Failed

Test failed

Tagging current version andRoll back SCM to previous tagging version

SCM CommitEvent

Scheduled time

EventTrigger

Send SMS/Email/IMS

Module 2. Build EnvironmentContiguous Build

• System configuration for CI

ContiguousIntegration Build Script

Testing Framework

SCM

Notifier

Check out codeTagging orRollbackwhen build fail

Invoke

Send notification message when build or test failed

Use

Module 2. Build Environment Contiguous Build

• Jobs for CI

Build Task Description

Compile Only compile

Packaging Packaging archives

Deploy Deploy achieves to server

Testing Run Testing

Testing and coverage analysis

Run Testing with code coverage analysis

Code Inspection Run Code Inspection

Full build Compile Packaging Deploy Testing with coverage analysis Code Inspection

Job Description

Full build Compile Packaging Deploy Testing with coverage analysis Code Inspection

Module 3. Testing AutomationOverview• Overview

• One of most important objective in ALM is guarantee quality of software

• Test Automation Module is designed for cover whole test cycle of V-Model.

• Unit/Integration tests are covered by development team and System/UAT are covered by QA team. Followed by your role , you can customize this process and framework

• Theoretical background• Mega process is based on ISTQB testing principals.• Unit testing and automated regression testing is

based on Practical methodology

Module 3. Testing Automation

• Testing Model– V-Model : Enhancement of waterfall model– Verification & Validation

Verification Validation

To verify the artifacts that has been produced in each development cycle.

Valid & estimate the system

Review & Inspectionwith artifacts from each development step

Testing with system

Static testing Dynamic testing

< Verification & Validation >< V-Model >

Module 3. Testing Automation

• Testing LevelLevel Description Testing Type Responsibility

Unit Test Verify software component White box test Lead by Dev Team Use Code coverage

Integration Test Verify integration between component. Verify software flow ,interface & interaction

White box test Lead by Dev Team Use Mock Object (Test Stub ,Driver)

System Test Test system over production environment to verify system itself and production environmentInclude Functional & Non Functional Test (Availibility, Stablity, Extendability, Performance etc)

Black box test Lead by QA Team(Specialized for system test)

Acceptance Test Verify customer’s requirement (End User)Verify customer’s Legal issue (Legal)Verify customer’s maintanance issue (SM)

Black box test Lead by QA Team or Customer

Alpha TestBeta Test

※ Integration strategiesTop down Bottom up Big-bang Backbone

Desc Integrate from top(use test stub)

Integrate from bottom(use test driver)

Integrate all in time Most important & high risk component first

Adv easily find architectural defect

Can test implementation logic

Short time

Ex. BPM SOA svc Small project

Module 3. Testing Automation Overview

• Test Cycle– Each test is consists of test cycle– Test cycle

Pre test - check up testability Main test Conformation test - check up the defect found in

previous “Main test” Regression test - check up the impact from change

Pre Test Main Test

ConformationTest

RegressionTest

TEST CYCLE

Module 3. Testing Automation Testing Process

• Mega process of Testing– Based on ISTQB testing model

Define test scope

objective

Risk anal

Set up strategy

Estimateresource & time

Planning

PRE TEST

TEST

Monitoring

Report


• Step 1. Risk Analysis– Risk = Likelihood * impact– Risk identification

– Recommend < 36 identification.– Likelihood : Complexity, Implementation technical Level,

Size,Developer skill– Impact : Biz impact

Likelihood Impact

factor factor factor factor

Risk item

Risk item

Likelihood

Impact

STA

(Severe Test Area)

SSTA

(Strong Test Area)

FTA(Fundamental

Test Area)

ITA

(Intensive Test

Area)

Erik van neneendaal, Risk Based Testing, STAREAST,2006


• Step 2. Define test strategy– Risk based test strategy

– Other strategies• Analytical approaches• Model based approaches• Methodical approaches• Dynamic and heuristic approaches

Likelihood

Impact

STA

(Severe Test Area)

SSTA

(Strong Test Area)

FTA(Fundamental

Test Area)

ITA

(Intensive Test

Area)

Likelihood

Impact

STA

(Severe Test Area)

SSTA

(Strong Test Area)

FTA(Fundamental

Test Area)

ITA

(Intensive Test

Area)

( Low Level Test) ( High Level Test)

- Unit Test

- Integration Test

- System Test

- Acceptance Test


• Step 3. Estimate resource and time


• Step 4.Test planning & control– Define activity of each testing level– Define milestone ,resource ,schedule.– Make a plan based on analyzed risk factor– Define test strategy based on risk– Define approach & techniques for testing

(testing techniques, coverage, test item, test ware)– Must include time & resource for preparing testware– Define completion condition


• Step 5. Test analysis and design– Test analysis

• Review - test basis, testability• Identify test requirement & test data• Identify test infra and tools

– Test design• Identify test condition• Test case specification• Define test procedure


• Step 5. Test analysis and design (Technique)

Design technique•Specification based

•Equivalence partitioning, Boundary value analysis

•Pairwise, Decision table testing

•State transition testing, Usecase testing

•Structure based

•Control Flow Test

•Basic Path Test – Based on Cyclomatic complexity (Edge – Node + 2P) P = number of connected component

•Elementary comparision test – Based on MC/DC Coverage

* Coverage : Statement, Condition, Decision, Condition Decision Coverage, Modified Condition/Descision Coverage(MC/DC), Multiple Condition Coverage

•Experience based

•Exploratory Testing – Adhoc testing, based on test charter, define timezone

•Classification Tree method

•CheckList


• Step 6. Execute Test– Execution

• Define serverity & priority• Develop TC,Test data,Test stub & driver,Test

script,Test suite

– Reporting• Logging expected result, real result• Current status• Time & resource usage

– Defect tracking• Use Bug(Issue) tracking system

( Mantis,Bugzilla,JIRA,Trac) can be integrated with Task Management module in ALM.

• Define tracking workflow

TESTDRIVER

TESTSTUB

TARGET SYSTEM

TEST CASE

TESTINGTOOL

TEST DATA MODULE

MODULE MODULE MODULE

TEST SCRIPT

TEST SUITE1

TEST SUITE2

TEST

SIMULATOR

Test Execution

Test Report Defects

Issue Tracking System

Progress Report

Test logs


• Step 7. Evaluate exit criteria and reporting• Test exit Criteria

• Coverage rate, Duration, # of unresolved defect

• Report Test Result• Progress report• Release advice• Final report per testing level

• Matrix• Defects per KLOC, Days test effort per requirement,

Unresolved defect impact analysis, Defects / Hour ,Cumulative defects / Day ,Test coverage / Day , Risk vs number of TC


• Step 8. Testing closure activities• Evaluating

• Evaluate test target• Evaluate test process for maturing next text

• Final reporting• Store test-ware

Module 3. Testing Automation Testing Framework

• Testing framework– Tools and environment to support executing testing

Unit Test FrameworkUnit Test Framework

Load TestingTools

Load TestingTools

UI basedAutomated

Testing

UI basedAutomated

Testing

MonitoringMonitoring

Test caseManagement

Test caseManagement

Defect Management ToolDefect Management Tool

Contiguous Integration ToolContiguous Integration Tool

Unit Test Integration Test System Test UAT

RegressionTest

RegressionTest

RegressionTest

Report Defects

TestcaseManagement

Execute Test


• Unit Test framework

User Interface

BusinessLogic

DAO

Interface(RMI,REST,WS*)

xUnitxUnit

for DBTestingxUnit

for Protocol or message Type

xUnitfor UI

In Container Testing Tool

Code Coverage Analysis Tool

Test Test Test Test

Ap

plicati

on

Un

it T

est

fram

ew

ork

Support J2ee incontainerTesting

Validate test itself


• Load Testing• Support non-functional testing.• Generate load.

• Monitoring• Application Performance Monitoring Tools• Used for finding bottleneck in performance. • Eg. Jennifer(Korea),Wily Introscope. (different from Profiler)

• UI Based Automated Tool• Let tester to free from factory job.• Record user activity and replay it.• Validate expected action.• It is used for user acceptance test. (Especially functional testing)※ Test case can be reused in Stress testing.

• Test case management

Module 3. Testing Automation Static Testing

• Static testing techniques• Not running code, review code statically.• Code Review - Manual Review is done by human• Static Analysis - Static review is done by code inspection

tools

Static TestingStatic Testing

By humanBy human

By Automated toolBy Automated tool

Code ReviewCode Review

Static AnalysisStatic Analysis

Automated inspection toolsAutomated inspection tools

• Code Review• Introduced by Fagan as Code Inspection.• Developed to more vary Code Review.• Very effective and low cost action to keep code quality. • Kind of Code Review

• Code Inspection• Intend to find a defect. Executed with formal process and team. (For more information. See Fagan’s Code inspection)• Team has four roles. (Mediator – project manager, Reader – understand system to be inspected, Designer/Coder – Inspect

and generate recommendation, Tester – Test inspected defect and validate) And Six steps in process.• To execute code inspection, you need to have formal inspection team in your organization or bring a consulting who has

highly trained. • Performance of inspection is poor. But effect is very nice. Code Inspection is done after implementation. Recommend to

use code inspection in system testing phase.

• Team Review• Less formal than code inspection. But it has process also.• Mediator manage Team review session, and code author explains about his code.• Find a defects or gather idea for enhancement and write it to code review report. (Recommend to

use wiki)• The ideas and defects are handled as Task in ALM/Task Management • Recommend one time per week. Or after testing, after internal releasing, after sprint.

• Walkthrough• Informal review. It can be used to share idea.• Only author has a rule that explains his code. Nobody has a responsibility.• All things are up to the author

• PeerReview• Review code on a desk. • Usually 2~3 people can attend on review. Review code over a monitor and share idea.• It is nice to educate junior developer or who doesn’t have experience about product or skills.



• Static Analysis• Test code or diagram with “static analysis tool”• Analysis

• Can find syntax error or well known bugs• Validate naming rules• Can find dead code or missing parameter• Can analysis code complexity & dependency

• Quality?• Efficient but not so smart• It need a rule and need a maturing

• Easy to appeal your manager• Tools

• Open source – PMD, Find Bugs,Jdepend etc.• Commercial – Klockworks, Jtest etc.

Module 3. Testing Automation Defect management

• Defect management Process

DefectManagement System

Call coordinator

Validator Project manager

Developer

QA Engineer User

1.Create case

Filter

•Summary•Description•Log and screen shout•Reproducible case

2.Yank& validate

2-1 Request more information

3.Assign 5.Assign

4. Priories and schedule

6.Fix (Workaround or patch)

7. Provide Solution

8. Confirm

Source codeManagement system

9. Commit to main branch with Defect #

Defect Reporter

Implementation Team

Resolved defects list is combined to release note

Module 3. Testing Automation Defect management

• Defect management system• Defect management system is integrated with SCM to track change

of source code to fix the defect.• Web based SCMbrowser shows changes for defect in a web from

Defect management system link• Task Management system can be used as Defect management

system as well.• In Big project or if there are so many defects are reported then

separate defect management system and integrate it into task management system. It needs to separate organization to implementation and defect handling team

DefectManagement System

Source codeManagement system

Web basedSCM browser

Track changes

LinkShows change of code

TaskManagement System

Defect is task


DeliveryPractical ALM Framework

Aapproach to delivery ALM solution

• There is 3 perspective to delivery ALM.

Process

Analysis Design Implementation Testing

UnitTesting

Integration

Testing

SystemTest

UAT

Task Management Process (Traditional Methodology)

Task Management Process (ALM/Scrum based)

Task Management Process (ALM/Scrum basder)

Build Process

Test Process

SCM Process

Test Process

Defect Mgmt Process

Task Management Process (ALM/Scrum based))

Developer

Management(PM/PMO/PL)

QA

DevelopmentLifecycle

Testing Model(Based on V-Model)

Agile/Scrum Base

Practical methodology base

V-Model Based

Reference ArchitectureALM architecture overview

• Reference architecture describes what we need to build up ALM system with tools and software.

• Layout of ALM architecture

Task Mgmt

ContiguousIntegration

Testing Framework

Build Script

Source CodeManagement

IDE

Build Script

Testing Framework

Defect Management

Testing Framework

(System& UAT)

Dashboard

Build Status Test Result

Task Status

Defect Status

Check out

Check in

Management(PM/PMO/PL)

Developer

QAAssign Task

Wiki Provide guide & processManage document.

Reference ArchitectureTask Management.

• Atlassian product is very popular, so easy to get a support and get a resource who can use it.

• Atlassian has CI tools, SCM connector, Coverage analysis tools,,Wiki etc. So it provides strong integrity.

• But, origin of JIRA is for Bug Tracking System not a Agile based task management system. It needs a customization in process and tool.

Atlassian Confluence Wiki

Atlassian JIRA

zAgilePortal

MS Project

Requirement Management Task Management Major Milestone managementfor mgmt org. (eg. PMO)

Personalization, Integrated view

Scrum based task management process

Scrum in JiRA : http://www.slideshare.net/paulrene/adapting-jira-for-scrum-presentation

http://www.slideshare.net/paulrene/adapting-jira-for-scrum-presentation

Reference ArchitectureTask Management – Alternative solution “Spira Team”

• SpriaTeam has great agile approach. It already contains concept of Release,Iteration,Requirement,Task,Test..

• Most excellent thing is it provides traceability from Requirement-Task-Test. It enables track whole things in manner of ALM.

• It also provides some connector to integrate with issue tracking system (JIRA,Bugzilla)

• But it doesn’t support integration with CI tools or coverage analysis tools . It should be implemented with other product.

• Company looks new and small, so they seems like have few reference yet. But cheap..

Atlassian Confluence Wiki

InfrectraSpira Team

Requirement ManagementRequirement Management, Task Management, Schedule mgmt& Test case management

Scrum based task management process

LINK

Reference ArchitectureTask Management – Another alternative

• VersionOne (http://www.versionone.com)- Agile Oriented- They provide hosted task management tool and sell it as package.- Well support scrum methodology. - Free license for one small team for evaluation. Try it…

• Rally Enterprise (http://www.rallydev.com) - Agile Oriendted- They provide hosted (Saas) Task management tool.- It is based on Agile methodology. And they have great agile knowledge.- But it is not inconvenient for my practical ALM approach. (It doesn’t mean their product is

poor)- If you want to get a deep consulting about Task management process, I want to

recommend this one.

• Serena (http://www.serena.com)- It is traditional pure ALM player.- It is very popular and covers big area of ALM.- It needs high level team maturity. - I just want to recommend this one to Enterprise who really willing to invest ALM and

Governance.

• Polarion (http://www.polarion.com)- It is also very powerful ALM vendor. - Same as serena, it covers whole of ALM scope and IT governance. It supports CMMI level

as well and has very rich capabilities.- There is light version- Polarion Track & Wiki (Issue tracking + Wiki + light build mgmt +

Eclipse Plug in)

• Codebeamer

※This evaluation of ALM tools is up to my personal opinion.

http://www.versionone.com/

http://www.rallydev.com/

Reference ArchitectureTest Automation

JUnit

EasyMock Cactus

Selenium DBUnit XMLUnit

Japex

Basic testing framework

Mockup test J2EE In container test

UI Test DB Test XML Test

Junit based stress test

Cobertura

Test Code coverage analysis

SourceForgeGrinder

Stress Test

SOAPUI

Stress Tes t forWebservice,REST

Find Bugs

Code inspection

AtlassianJIRA

Defect tracking

Functional Testing Framework(Unit testing/Integration Testing)

Non functional testing(System Testing)

Static Testing Defect mgmt

V model

Alternative•HP Load Runner (Commercial)•Oracle ATS (Commercial)•Apache Jmeter (OpenSource)

Alternative•PMD (Open Source)

Alternative•Atlassian Clover (Commercial)

Alternative•Mantis (Free)•Bugzilla (OpenSource)

Reference ArchitectureTest Automation

• Design principal of Test framework is to support automated testing and regression test.

• Test framework covers functional testing includes unit test and integration test. (Integration test can be viewed as coarse grained unit test)

• Test framework combines – Functional testing capabilities followed by characteristics of component like DB Testing,UI Testing,XML based messaging and Unit based stress test, coverage analysis and mock up test environment etc.

• This test framework covers broad range of testing. It means it is not easy. (concentrated on Unit testing and Integration testing. Partially covers System test and UAT. I recommend to use more matured tools for System test and UAT)

• ROI of code inspection is very high. How to managing rule set is most important.

• Optionally, I recommend to set up test case management tools from QA team. (ex. Testopia)

Reference ArchitectureBuild Environment

• Apache ant will not be developed anymore, Maven will replace ant. Maven also support unified library management.

• Hudson is now most popular free CI tools.- It supports a lot of plug ins for JIRA,Junit,Find Bugs etc.- It is very easy to learn.

Functional TestingFramework

Test framework

Maven


Subversion

Source code Management

Eclipse

Standard IDE

Hudson

Contiguous Integration

Practical methodology (Joel Spolsky,Kent Beck, Erich Gamma)

Alternative•Atlassian Bamboo

Reference ArchitectureCollaboration

• Just provide minimum collaboration environment.• Atlassian confluence provides a lot of plug in. Like MS-

Office connector etc. It is very useful to sharing your team knowledge.

• If you already have collaboration infrastructure in your company, do not extend product.

AtlassianConfluence Wiki

Document ManagementKnowledge Transfer

JForum

Bi-directional knowledge exchange

Reference ArchitectureDevelopment environment

• In staging environment, highly recommend to use hardware virtualization (CPU,Memory). It is very useful to validate system capacity. (Scalability test result is be basis of capacity planning)

DevelopmentEnvironment

Staging Production

Responsibility Development Team QA Team SM Team

GA version(General Available)

InternalRelease

GA

Unit TestIntegration Test

System Test(Software perspective)

System Test(Include validating hardware infrastructure)UAT

Virtualization is useful

OrganizationRole & Responsibility

PMO(Project Mgmt Office)

PMO(Project Mgmt Office) Implementation Team

(PL)

Implementation Team(PL) Application Architect

Application ArchitectQuality Architect

Quality Architect

ALM MasterALM Master

Extract Requirement

Task Management

Dashboard


Contiguous Build

Standard IDE


Testing Process

Testing Frameworkfor Unit,Integration Testing

Static Testing

Defect ManagementWiki based doc mgmt


Code review

Testing Frameworkfor System,UAT

• Who has a responsibility to build up process and system?

Mentoring (Very important)



Implementation Team(PL)

Implementation Team(PL)

Extract Requirement

Task Management

Dashboard


Contiguous Build

Standard IDE


Testing Frameworkfor Unit,Integration Testing

Static Testing

Defect ManagementWiki based doc mgmt


Code review

• Implementation team uses a lot of things in ALM It needs a strong mentoring to use it. Mentoring is not a

option but a mandatory..!!

ALM MasterALM Master




• ALM Master Required capabilities : Understand whole ALM architecture &

tools and process - May be genius? No. think with customer, act with customer. (Just right understanding of their ALM and help them not to miss a way..)

Responsibility Help customer to build up their ALM process and system. Help customer to adapt ALM. Inspect ALM usage in the team. (Inspect task management

system note, Review Wiki usage and structure. Check up Build script and CI status. ) and mentor them to right way.

Convince customer or project manager that no more free lunch.!! (to do something, let them know it needs a time and resource)

Each stakeholder wants to customize ALM for their perspective. In ALM, all of stakeholder has to share their perspective. (Even if they can see different views for their role) – It is liquid flow. So ALM master lead the stakeholder to have same perspective..

Help customer to change their culture slowly,smooth without revolt, awareness.


Strategy for deliveringPractical ALM Framework

Key Strategy

• LiquidEach process of ALM are integrated as LIQUID

• SeamlessTools and solution that are used to realize ALM are integrated as seamless. Should be looks as one platform.

• Process OrientedTool and Systems are just tool to realize ALM concept. Do not depend on Tool. Process is key.

• Step by step depends on team levelEven if ALM is lightweight. It is hard to apply at once.Apply ALM modules step by step followed by team maturity level.

• Be SimpleStart of ALM is to remove complexity of traditional methodology.Complexity reduce efficiency and requires a lot of learning cost.

• And need a coach. (ALM Master)• Do not enforcing process or tool, but change culture.

Most important point of delivering ALM is changing culture.

• Delivery ALM secretly. Developer should not aware change. It means changes are applied smoothly.

Delivery Step

Overall road map of applying ALM

Build Environment Task Management Test Automation Collaboration

Level 1 SCM,CI, (AA)Standard BuildScript

Level 2 Standard IDE (PL) Excel based Task Management(PL)

Code Review (PL) Wiki (PL)

Level 3 Branch mgmt strategy (AA)

System based Task Mgmt (PM/PMO)Dashboard (PMO)

UnitTest (AA or PL) Regression Test (AA or PL)

Level 4 Extract Requirement (AA) Static Analysis (AA)

Level 5 Testing process (QA)

Level 6 Defect mgmt system (QA)

ALM (Application Lifecycle Management)

Technology

Transcript of ALM (Application Lifecycle Management)