Business Strategy

1

Tsuneo Yamaura

Hitachi Software Engineering Co., Ltd.

[email protected]

T-Shirts vs. Kimono Approachin

Software Quality Assurance

　　　　　　　 Jan. 16, 2006

AOSS Training@Bankgkok

2

Some QA Statistics at Hitachi Software:

Percentage of Bug Detection in Software Phases code unit subsystem system inspection after releaseinspection debugging debugging debugging by QA dept. (at user site)

21.5% 35.1% 28.0% 6.1% 9.3%

Only 2 bugs out of 10,000 emerge at customer’s site.

10 engineers work for 12 months to make 100KLOC and 1,000 bugs.

0.2 bugs emerge at customers ’ sites

0.02%

3

What Do We Do for Quality Assurance ?

(1) Developers design, write test cases, and run test case.

(2) Statistical bug data is collected throughout product life cycle.

(3) A project manager analyzes the data to control the project.

(4) Independent QA engineers test products.

There is no magic. We simply employ traditional methods.

4

(1) Developers design, write, and run test cases

a. When developers have completed coding, they start to design

test cases.

b. Test cases are documented. Advantages of the written test

cases are:

- Designing test cases requires that the specification be examined

from different vew points. Many critical bugs can be detected.

- You can re-execute the same test cases.

- You can « test » the quakity of the test cases.

- The execution of the test cases can be passed on to another

person, perhaps somebody who was added to the project to

recover a delay and does not understand the program very well.

c. Automatic test execution (test suites) is desired.

5

(1) Developers design, write, and run test cases (cont.)

Test case design guideline

a. approximately 1 test case per 5 - 15 LOC

Language processor: 1 test case per 8-12LOC

Online system : 1 test case per 5-10 LOC

Batch system : 1 test case per 10-15 LOC

b. normal cases : 60% or less,

abnormal cases : 15% or more,

boundary cases : 10% or more,

enviromental cases: 15% or more

6

(1) Developers design, write, and run test cases (cont.) Example of matrix-based test cases

test case ID AD001 AD002 AD003AD004 AD005 AD006 AD007 AD008 ZZ X

-1 X0 X

Age 6 X12 X18 X19 X

999 X$0 X X

Admission $5 X Fee $8 X

$10 XErr msg X X X

Category abnrm bound norm norm norm norm norm abnrmCode inspection 5-May 5-May 5-May 5-May 5-May 5-May 5-May 6-MayMachine test 1- J un 1- J un 2- J un 2- J un 2- J un 2- J un 3- J un 3- J un

admission fee: $0: age 6 or under, $5: age 7 - 12, $8: age 13 - 18, $10: age 19 and above

7

　　 From when ?

　　 All the compilation errors are eliminated (from the code

　　 inspection phase).

　　 What must be reported ?

　　 a. when detected

　　 ID number, symptom of the bug, who caught the bug,

　　　　 when, seriousness, etc.

(2) Statistical bug data collection throughout the life cycle

　　 Developers must report all the bugs (Bug Report)

　　 b. when fixed

　　　 Modules in which the bug resides, explanation of the bug,

　　　 who fixed, when fixed and what was tested by whom, code

　　　 before and after.

8

(3) Bug Data Analysis by Project Manager

- A project manager drows the « test progress plans» before

getting into an actual testing.

numb. of untested cases (target)

cumulative bugs expected

time

numb. oftest cases

Dec/30 Jan/5 Jan/12 Jan/19 Jan/26 Feb/2 Feb/9

9


The Project Manager plots the test progress on the chart:

- to determine the best strategies to improve the quality

- to predict the release date.


numb. of untested cases (actual)

cumulative bugs found

cumulative bugs expected

time

numb. oftest cases


10

(3) Bug Data Analysis by Project Manager (continued)

remaining test cases

expectedactual

bugs found

bugs expected

best case


expected actual

bugs found

bugs expected

low quality in previous process


expectedactual

bugs found

bugs expected

inexperienced testers


expected

actual

bugs found

bugs expected

low test case quality

11



numb. of untested cases (actual)

Generation of bugs gets flat

All the test cases are executed.

time

numb. oftest cases


Project Manager thinks it’s time release when:

- All the test cases are executed.

- The growth of bugs gets flat.

- All the bugs are fixed.

- 48-hour continuous operation successfully completed.

12

(4) Independent QA engineers test products

We have QA departments which are cpmpletely independentfrom the development departments.

Q1: How many engineers are assigned to the QA departments ?A1: Approximately 8% of the entire engineers (330 out of 4,000).

Q2: What do they test ?A2: All the developers’ output (requirement specifications,

design documents, programs, test cases, etc).

13

Q4: How do they test ?

A4: From the customer’s vew point. They design, and

write test cases, which are completely independent

from those created by developers.

(4) Independent QA engineers test products (continued)

Q3: How powerful are they ?A3: A product cannot be released without their ‘go-ahead.’

14

The cost of the quality (There is no free lunch !)

Assumption : A 12-month project with 10 developers works on

C-based software with 100,000 LOC.

Months for each phase

- requirement specification : 2 months

- designing : 3 months

- coding : 2 months

- debugging : 3 months

- testing (by QA engineers) : 2 months

Details of the 3-month debugging phase

-100,000 LOC software needs 10,000 test cases (1,000 per developer).

-10 days to design 10,000 test cases (100 a day per developer)

-10 days to check 1,000 test cases in code inspection (10 cs/day-develpr)

-40 days to check 10,000 test cases in machine debugging (25cs/day-dvlp)

15

Q: How did we attain such a high goal (99.98% bug-free) ?

Are we better than others ?

A: Hopefully yes, but I am not sure.

The main reasons are:

- We have not changed the QA methods for more than

30 years.

- Our turnover rate is quite low (an average project

manager, for example, has worked for the company

for more than 15 years).

- Our review process works pretty well.

16

Applicability of Our QA Method to Development Models

[1] Waterfall Model

Our QA approach fits very well, because:

- The development lifecycle spans several months (or years),

- The start and end of each development phase are clear,

- Each development phase goes sequentially.

[2] Object Oriented

We are in the researching stage. We may need some

modification, but perhaps existing method can be used.

[3] Spiral Model

If the development lifecycle is very quick (for example, a

product comes out in every week), our QA approach may

not be suitable.

17

Q: Do you think our QA approach is applicable to the programmers in the US and India ? A: Of course NOT !

18

Conflictions in the Software Development Priority

USA ans India vs. Japan = marketability vs. quality

American/Indian type Japan type

It does not mean a thing if it does not sell well.

Quality is it.

differentreligions

19

magnitude

release timing

impact to market

quality maturity

USA/India Japan

Differences in sales strategies (Japan before 2000)

cost < qulity < release cost < release < qulity

20

Differences in Software Industries in Japan and the US

1. Success as a team is important.

- Honest data can be easily

collected.

- Statistics-based QA is

applicable.

1. Personal achievement is more important than a team success.

- “Making bugs” = “I have failed.”

- The bug data is unintentionally

cooked.

Software Industries in Japan Software Industries in the US

21

Differences in Software Industries in Japan and the US (cont.)

2. Quality is more important than cost and release schedule to avoid huge bug fixing cost.

2. Dominating the market by early release is more important than quality.


22

Differences in Software Industries in Japan and the US (cont.)

3. Looks for “industrialization” in

software.

- Key word = “Play safe”

- Tends to plan riskless products

- Follows the technologies

invented in the US / India.

3. Seeks “novelty” and “technology in

the next generation.”

- Key word = “Challenge”

- Goes for ambitious products

- Delivers epoch-making

technologies.


23

Differences in sales strategies (What Japan faces after 2000: Part 1)

magnitude

release timing

impact to market

quality maturity

USA/India Japancost < qulity < release cost < qulity < release ?

・ The marketing strategy has changed from “heavy/thick/big/long” to “light/thin/small/short.” When you want sell inexpensive goods in a big quantity, sales timing is crusial (e.g., If you cannot release by Christmas season, you will miss a sale chance).・ A strong demand from customers (e.g., Must be released by the beginning of a new fiscal year)

Businessstrategies in

the Era of Web and embedded

system

24


magnitude

release timing

impact to market

quality maturity

USA/India Japancost < qulity < release cost < qulity = release

・ We have to shorten the release timing as well as to retain the high quality.

25

CMM Issue

Qustion: India is the No. 1 country that has most CMM-level-5 companies

in the world. Does that mean they really make high quality software ?

Answer: - CMM checks if a software company furnishes software development

processes to improve quality.

- CMM does not consider if a company makes quality software.

- A little league baseball team will get CMM level 5 if the team

has a pitching coach, a batting coach, health checking stuff.

If, on the other hand, a major league baseball team does not have

any coaches, it will get level 1.

- CMM considers the processes to develop quality software, not

the actual actual quality.


26

- In order to cut the development cost, and to shorten the development

time frame, we have to use off-the-shelf software (like MS Word, Excel)

in stead of develop everything.

　 → Bugs in the off-the-shelf software will not be fixed in a timely fashion.

　 → You have to assumu such bugs as « Specifications. »


- The technology of software tends to be highly specialized, you have to

completely depend on a particular company that has such special

technology, such as the browser of a mobile phone.

　→　 The quality of your software depends on the company.

　→　 You cannot force them your quality processes.

27

One of the promissing solutions is:

- Open Sourece

- Standardized Souftware

Business Strategy

Documents

Transcript of Business Strategy