A PMA example from a Norwegian company

Tor StålhaneIDI / NTNU

The PMAWhat follows is a short summary of a one-day session with all the project participants from a large IT-company.1.The project’s time line2.The positive issues

– Affinity diagram– Ishikawa diagram

3.The negative issues– Affinity diagram– Ishikawa diagram

Oct Des Feb Apr Jun Aug Oct Des

Year 1 Year 2

Feb Apr Jun

Project establishedExternal project manger

Test manager

External project manager outExternal architect outExternal functions responsible out

New project managerNew external functions responsible New architectDefined scope

New scopeNew improved companystandard architecture

Demo version

Low intensity ”fumbling phase”

Clearification of several rolesStarted cooperationwith customer

New, internalFunctions responsible

Formal cooperationwith customer

Deliver technical pilot

Four iterations

Use cases are goodUse case driven=>good structure

Use case2

IterativeIterative process gave good progress and controlIncremental deliveriesIterative process positive

Iterative process3

Good methods that can be reused in further projectsNew and improved methodsHas improved us – competence and process

Working process

Evaluation of iterations –useful activityGood process for QA improvement Willing til to try newprocesses / methods

The will to change

Cruise control => tool support in developmentJ-unit test as new method inthe development processTool for performance testing

New tools

3

3

Good mix of resources => banking experience=> technology competenceNot only programmers(Java + Transigo)Interesting to learn about banking from The X personnelResourceful consultants with highcompetence (Java ++)Good mix of competence Team organized according to product

Competence mix and organization

2

Bruk av nye metoder ogteknologi er motiverende(JSF / J-unit)Nytt, bedre utviklingsmiljøCC, J-unit, JSFNy teknologiGodt teknisk grunnlag(moderne)Mye læring – ny teknologiJSF +++

New standard technology

Stable testing environmentTesting and correctionUse of QC for doing follow-up of reported defectsGood use of QC => goodoverview for status

2

Good testing process

Base for change control

QC change project

1

We have deliveredDelivered to productionwithout large delays

Delivery focus 4

Good cooperation with customerBank’s contribution – testNew concept => positive reception in the market place

Customer focus

4

Status meeting in the morningMorning meetings gives common groundMorning meetings during testingRegular status meetingsDaily status meetings duringsystem test period

Status meetings

6

5STG for testingWorkshop /STG for test casesStructured reviews of UCReviews of test cases together with the developersStructures reviews increased thequality of the documents

STG

Competent testersGood cooperation with QAGood project moral Good cooperation and a fine collaborative atmosphere Competent personnelExciting and challenging Good cooperation and communicationGood cooperation in the development teamCooperation with our Swedish subsidiaryStructured / knowledgeable PMGood follow-up of the projectGood cooperation in the project, also during error correction / test phase

Cooperation

5

Status meetings Once a day oronce a week

Start early

Consider the whole project

Problem solving Find info (who knows what)

Improve spec

Use the team

Identify problems(hindrances)

Structure

Be focused

”15 minutes”

Discipline

Do not startlong discussions

Cross disciplinary

Consider the whole project

3

SAD for the system was lackingNo good SAD was madeLacked documentation of architecture decisions Bad architecture / designNeed a functional Lead and Technical Lead. Important roles

High level design

Needed more detailed designToo little focus on designReuse of common components wasstarted too lateSome functionality should have beenbetter specified and designedUse case realization was not doneaccording to intentions

Low level design 7

Some iterations were too largeToo short test period in each iterationLarge time pressure on developers => little unit testingMuch work => time pressure possibly underestimatedPressure at the end of theprojectUnit testing for X was severelyunderestimatedMuch reallocation of requirementsBad requirements estimation

Estimation

5

Little ”paid” for good ideas• ClearCase• STG• automated unit testes Lacked prototype for STG for UC

Immature use of tools

2Unit test Java checklist not finishedwhen turned over to system test not used.Development not finished when testing started

Start of test

New development methods require timeMore focus on unit testingToo little focus on CC – status meetings?Low focus on PMD errors(exploded for common components)

Methods (dev)

4

6

Avoid part-time resourcesin a projectDifficult to usepart-time resourcesToo many part-timerecoursesToo few recourses fortestingLate staffing of test teamneeds dedicated adm.Resources full time

Resources

Overtime to deliver on timeDelivery on time dependedon large amount of overtime

Overtime

Handover f-team => t-teamLacking details in specFunctional description toosmall / not good

Thin functional spec

3

Lacking access toapplication server with the same OS as the production environmentTest new components in application env. earlierDifferent technical platformfor development and test

Development vs. test

Error correction started lateLooong correction time duringthe first part of systems test

Error correction 2

Difficult to involve IT operationsToo little info / resources from KernelThe whole value chain must berepresented Discover lacking functionality in the main system earlierLacking documentation in generaland especially for central partsLacked communication with thosewho implemented functionality in the main system. Too much work planned initially

Stakeholders 2

Problems when reusingcommon components The project mandate was unclear when the project startedNo connection between complexity/ volume and mandate from managementSteering group didn’t understand the complexity

Project scope

2

4

For little team spirit betweenfunctions and development Development had too little focus on UC and system specUpdating the spec during development => unclearresponsibilityChangers in use cases were notwell handledFunctional changes duringdevelopment – test resourceswere not informed Changes were not handled inan optimal way“Request for project changes”didn’t workLacked decisions on questionspertaining to functionality evenwhen the system test had started Functional spec and technical spec where out of phase

Project discipline

High level design

Low level design

Start of test

Estimation

Immature useof tools

Methods (development)

Resources Overtime

Thin functionalspec

Development vs. test Error correction

Stakeholders

Project scope

Project discipline

Low-level designproblems

Thin functional spec

Lack of businessunderstanding

We are not used to produce a low-leveldesign

Complicated to make a low-level design

Lacking a system’sarchitecture

Management / control

Only UC Lacks a process

Lacking documentation

Oral infoLacking documentation

Lack of resources

Where can we find info?

Bad templates

Lacking guidelines

Lacking examples

Needs to gothrough this together

Focus oncode

Time of delivery

The need is identified too latein the process

The need mustbe identifiedearly in the project

Little focuson this

Must be possible todo just a part of it

Lacks competence

Is often not required(balancing item)

A second look

We have been working with this company for a long time. It would thus be interesting to compare this PMA with information from

• An earlier, general PMA in the same company• An analysis of the errors in the company’s

defect reporting system• A company-wide gap analysis

Culture Competence Who is the customer

Working methods Time pressure

Requirements spec problems

Changes over timeFlexibility

Level of details

Competition

Difficult to estimate

Work in the same way as always before

Understand the customer

Work together with the customer

Commitment from those who know the requirements

Application knowledge

Consequences of a small, subordinate clause

Testing Requirements

specToo many temp. solutions

Defect reports   Simple Medium Extensive Cost index Percentage

Functional errors 4 5 2 39 2,41Errors caused by wrong test data and in the system build process 60 17 4 151 9,33

Wrong or unclear specification 63 26 12 261 16,12

Incorrect test data 8 1 1 21 1,30Errors introduced during development. 187 130 57 1147 70,85

Sum 322 179 76 1619 100,00

Development and wrong or unclear specification account for almost 87% of all corrections costs. We will thus have a closer look at these two categories. We focus on errors inserted by the project in question, and have removederrors like user misunderstandings and errors in third party software.

Development defects and Pareto – 1

Acc

0,020,040,060,080,0

100,0120,0

Acc

Development defects and Pareto – 2 Five error categories are needed to account for 80% of all

correction costs.• Logical error• Misc. Error• Wrong / unclear requirements• Wrong interface• Undefined error.

Approximately 95% of the defects can be categorized as• Wrong / missing functionality• Wrong / missing information displayed.

This holds for most of the errors, irrespective of the categories assigned by the developers.

Defect descriptions – examples • Logical error: Fails when opening a new account.

Link is missing in the menu on the left-hand side• Misc. error: Write-command fails. The looking-

glass icon is missing. • Wrong / unclear requirements: Account number

is not reported. The radio button ADD is missing.• Wrong interface: The code for X is not changed.

Spelling-error.• Undefined error: Cannot access screen Z. Missing

values in a menu.

Defect causesWe observed during the first, general PMA thatthe developers were proud of their applicationknowledge and preferred a fuzzy requirementsspecification since this allows them to use theircreativity.

They get problems with requirements and testing when they add a large number of consultants

• with excellent development and coding • without application knowledge

Customerrequirements

Finished system

Coding Testing

Application knowledge

Development knowledge

The role of application knowledge – 1

The role of application knowledge – 2 When analyzing the error reports we saw that when

application knowledge is not available• test quality will suffer• we will get incomplete functionality and information

display

Time pressure due to bad estimation, made the situation even worse for the testers.

• Primary problems: lack of application knowledge, incomplete requirement, bad estimation

• Secondary problems: time pressure, low-quality testing, incomplete functionality in the finished system

PMA, gap analysis and error analysis

Arch.Functional

specManagement

Business understanding

Testing Low-level

designEst.

RCA RCA RCA RCA (PMA) PMA PMA

Gap Gap Gap Gap

Errors Errors Errors Errors

• Problems identified by error analysis are also are identified by PMA or RCA.

• The PMA focuses on low-level design and identified functional specification and business understanding as secondary problem areas in the RCA diagram.

• Estimation is only identified in the affinity diagram• Testing only occurs as an item in the two affinity diagram groups

Estimation and Resources.

PMA problemsProblems with the PMA are that people:• Tend to focus on the start and end of a period

– e.g. a project – and forget most of what went on in between.

• Are good at identifying and collecting important data but are not so good at analyzing these data.

• Put emphasis on single, concrete events over statistical data. Personal experience takes precedence over numerical data.

Improved PMA process • Perform a PMA. This will give us an affinity diagram and one or

more RCAs. • Collect error data, e.g. from the system test.

– where in the process the errors were introduced– why the errors were introduced– the correction costs

• Identify problem causes from the affinity diagram and the RCAs. Prioritize the problems using – the participants’ opinion– priorities based on information from the error reports.

In some sense this can also be seen as triangulation [13] but in our opinion it is something more – it helps us to focus the results of a people process by using concrete and at least partly objective information.