Software Reliability CMM-DFSS

25
Guy Van Hooveld CTO Office Deployment of DFSS in software – experience within Philips Consumer Lifestyle

description

Presentation given during a software reliability seminar organized by Holland Innovative BV - subject was the combination of CMM and DFSS within Philips Consumer Lifestyle

Transcript of Software Reliability CMM-DFSS

Guy Van HooveldCTO Office

Deployment of DFSS in software – experience within Philips Consumer Lifestyle

Guy Van Hooveld 2March , 2008

Objectives of this presentation

• Give insight in two different approaches to apply DFSS in a software environment with Philips Consumer Lifestyle– Bruges TV experience – link between DFSS and CMM L4– Bangalore experience – link between DFSS and

milestone driven software project management• Conclude on the gains and limitations of those

approaches based on the current experience.

The TV experience in Bruges

Guy Van Hooveld 4March , 2008

CMM model ______ KPA _____ ( Key Process Area )

CMM Level 1

CMM Level 2 RM Requirements Management SLC Software lifecycle PP Project Planning PTO Project Tracking and Oversight SM Subcontract Management QA/SCV Quality Assurance/Software Compliance Verification SCM Software Configuration Management

CMM Level 3 OPF Organisation Process Focus

OPD Organisation Process Definition TP Training Program IM Integrated Management PE Product Engineering IC Intergroup Coordination PR Peer Reviews

CMM Level 4 QPM Quantitative Process Management

SQM Software Quality Management CMM Level 5 DP Defect Prevention

TCM Technology Change Management PCM Process Change Management

Guy Van Hooveld 5March , 2008

CMM model

Level 4 SQM : definition

The purpose of Software Quality Management is to develop a quantitative understanding of the quality of the project's software products and achieve specific quality goals.

Software Quality Management involves defining quality goals for the software products, establishing plans to achieve these goals, and monitoring and adjusting the software plans, software work products, activities, and quality goals to satisfy the needs and desires of the customer and end user for high quality products.

Guy Van Hooveld 6March , 2008

Quality goals to satisfy the needs and desires of the customer and end user for high quality products.• The reference framework in the software is CMM (L4)• Theoretically no actual DFSS deployment as such in the software, but:• Use of VOC to derive requirements• Use of derived CTQs to measure software quality• Focus on non-functional requirements• CMM philosophy applied + classical software reliability engineering applied

– Fault prevention• Reviews• Requirements management• FMEAs / risk management• Architecture

– Fault tolerance• Reboots / watchdogs• …

– Fault detection/correction• Verification/validation• Root cause analysis with feedback loop etc…

• In fact a lot of commonality with DFSS, close to an actual DFSS deployment.

Guy Van Hooveld 7March , 2008

What is SW Quality ? (choices made for a specific project)

– Stability– Critical resources : e.g.

• Memory Budget tracking• CPU load, • latency, • bandwidth

– Performance : e.g. startup time, zapping time, UI

– SDE warnings : Koala, compiler– QAC warnings– Asserts– Debugging– NFR conformity (validation)– Reviews : Defect densities

Guy Van Hooveld 8March , 2008

SQM in a TV project

Step 1. - create plan :

SW_Quality_attribute_baseline

Step 2. - deploy the planmeasurements & reporting ( by product, subsystems, suppliers)

Step 3. - corrective actions on deviations

Guy Van Hooveld 9March , 2008

Plan for a TV project : overview of CTQ’s

CTQ metrics on the Plan Product Platform

Subsystem

Stability x

Critical res. : Memory Budget tracking

x x

Critical res.: CPU load, latency, bandwidth

x x

Performance : startup, zapping, UI

x x

SDE warnings : Koala, compiler x x

QAC warnings x x

Asserts x x

Debugger warnings x

NFR conformity (validation) x

Defect densities x

Guy Van Hooveld 10March , 2008

Example (zoom-in) : Performance (prod), QAC

(subsys)

Guy Van Hooveld 11March , 2008

TV project : quality parameters Reporting

SDE Warnings

0

10

20

30

40

412 413 414 415 416 417 418 419 420 421Koala Warnings Compiler Warnings

Target Koala Target Compiler

Example :

Guy Van Hooveld 12March , 2008

Right design vs design right

• Right design (what does the consumer want ?)– DFSS in requirements process– Requirements process – functionality vs quality– Support from development to marketing

• Design right (how do we implement it correctly ?)– Software processes– DFSS in development

Guy Van Hooveld 13March , 2008

Lessons learned

• The CMM L4/DFSS approach has worked to improve quality• Approach made non-functional requirements more visible• When marketing is not capable of providing exact requirements (logical

for technical constraints for instance), development can help suggesting CTQs to start discussion.

• FMEAs not efficient enough in the software (risks can be listed but more difficult to make things measurable/w tolerance cf mechanical/electrical)

• A number of measurements reflect mainly process quality, not product quality (defects density)

• Supplier management: – improve control over critical parameters from our suppliers : Stability, NFR,

robustness.– for some SW-suppliers this is handled as well as possible through provisions in

requirements (NFR) and work agreements (measurement & reporting)– for others : there are still examples where this is difficult to achieve.

• SW CTQ's (and NFR's) are twofold :– those that contribute mainly to end-product (Stability, performance)– those that contribute mainly to development trajectory (warnings…)– Outside SW, only CTQ's contributing to the end-product are considered

Experience in Bangalore

Guy Van Hooveld 15March , 2008

Approach in Bangalore

• More classical DFSS deployment (cf Bruges example)• Alignment between DFSS DIDOV phases and project

milestones• Translation of DFSS to software tasks• This has been applied in numerous projects (wireless audio,

DVD-recorder, …)

• Clear results with respect to customer satisfaction, reduced

cost of non-quality, first time right design.

Guy Van Hooveld 16March , 2008

DFSS Approach in Software Life cycle CONQ reduction

Listen to Consumer needs (Voice of consumer)

Translate needs into CTQs with targets and specification limits

Flow down the top-level CTQ (Y) into lower level inputs Xs

Select Best design to meet the CTQ

Predict Capability on “Y” based on current Xs

Optimize gap between predicted and desired

Verify CTQs on target, Factory

Monitor CTQs in Field

FMEA &

Mistake proofing

Req.Mgt

Arch,Des

&Imp

Int.

Test

Maint

Sw Life-cycle DD

II

DD

OO

vv

MM

Guy Van Hooveld 17March , 2008

Requirements

Architecture / Top Level Design

Detail Design

Implementation

Module testing

Integration Testing

Alpha testing

DFSS mapping to V-model and SPEED

Feasibility Beta testing Maintenance

VPH CS PRS DR CR MPRProgramming Conception Development Maturity Maintenance

DefineIdentify

DesignOptimize

VerifyMonitor

Guy Van Hooveld 18March , 2008

Quality process - Software Mapping

Quality process based on 4 main principles

1. Identify the Voice of the Customer (VOC)• Obtain & prioritize customer requirements, Derive CTQs

2. Manage Critical parameters• Define targets for CTQs, Flow-down CTQs• Flow-up CTQs through predictions, Establish control plan

3. Establish Robust & Reliable Design• Identify sources of variation, de-sensitize design• Optimize Performance, Verify & Validate robustness

4. Manage risks & problems• Anticipate risk, counter or mitigate them• Detect and solve problems

Guy Van Hooveld 19March , 2008

Quality - Software Mapping

Kano

Risk Benefit

Brand Positioning

Marketing Concept

Consumer Insight

Benefits Reasons to Believe

Discriminator

TargetComp.Environment

“When we watch digital TV or DVD movies, we want a system that can playback and record in excellent audio and video quality.”

Selectives/ Classics 25-49 years old/Married w children or coupled Young Potentials Achieving while balancing family life

Takes time and effort to research on the market or product he is interested in Sensible consumers who value quality, functionality and ease of use.

Consumer InsightTargetCompetitive Environment

• Turn up your experience with Hi-definition Audio & Video

DiscriminatorReasons to BelieveBenefits

•Recording gets simpler, yet the quality of the picture gets better•The picture quality enhancements allow for a cinematic viewing experience, completely immersing us into the movie.

• Ease of recording with EPG•Picture quality improvements happen automatically•User-friendly user-interface

•Best Picture Quality

•There's always my favorite content waiting to be watched on the hard drive

AdvancedEasy To ExperienceDesigned Around You

Marketing Concept: (fixed: the essence of the marketing concept)

ValueHouse

DVDR5xxxH/7xxxH series

Emotional:•A rewarding experience that brings you

closer to the action.•Always know something you want to

watch is waiting for you. • Your recordings bring you closer to the

action than ever before.Functional:•Watch recordings in the best picture quality,

even better than broadcast TV.•Simple operation and use, facilitated by EPG

• Best viewing experience:HDMI , 1080i upsampling, Digital TV (DVB-T )

• Best sound:HDMI , 5 .1 channel recording from STB

• Ease of Experience:More than 400hrs of recordings (250Gb), EPG, user- friendly UI .

C brands entering Q3 at 499 Euros HDD capacity increased substantially HDMI being offered by others (Toshiba,

LG, Samsung) There are more recorder models with

DTT introduced by Sony, Panasonic, Samsung in Europe, especially in UK & Germany. •H

VPH

Extremely important

Very important Less important Not important

Playback music of a disc Adapting the set recording to the time that the program is broadcast. (VPS/PDC)

Playback of a recorded program, while the recorder is still recording the program.

Playback photos of a disc

Changing subtitle language while watching DVD

Recording 1. Timer recording 2. Direct recording 1. Show View 2. EPG (TV

guide)

Copying from hard disk to DVD disc (archiving)

Repeated recording of daily/weekly broadcasted program Changing audio

language while watching DVD

Playback of a recorded program and record another live tv program in the same time

Repeated playback of a recording or a part of a recording

Replay a soundtrack

Using a recording device: 1. Playback a movie 2. Watching a TV program 3. Jumping to the next chapter

Zoom in on an object in a movie/TV programme

For editing, most often used: 1. Deleting recording 2. Change title recording. Playback in slow-

motion Watching TV by switching channels through recorder tuner Pause live TV

(for currently watching channel)

Connecting an extra product: 1. VCR, for copying videocassettes to DVD discs 2. Digital camera, for watching pictures on TV screen and copying photo’s to DVD discs 3. Camcorder, for watching video on TV screen and copying video’s to DVD discs 4. DVD player, for copying DVD discs

Pausing during recording, so that parts of the broadcast (e.g. commercials) are not recorded Changing camera

angle in a movie

CRS

CTQs , CFs

HECE, UIS, FRS

1- Identify the Voice of customer (VOC)

Guy Van Hooveld 20March , 2008

CTQ flow downCTQ – Targets & Limits

CTQ flow-up & predictions

Control Plan - Dashboard

Quality - Software Mapping 2- Manage Critical Parameters

Guy Van Hooveld 21March , 2008

Arch/Des choices

CTQ Trade-offsCTQ optimisation, Noise de-sensitization

Transfer functions

3- Establish robust & reliable design

Quality - Software Mapping

Guy Van Hooveld 22March , 2008

User/CEC tests

Test coverage

Stress TestsField Tests

FMEA

PMI, PR/CR tracking

Verify CTQs

Quality - Software Mapping 4 – Manage Risks & Problems

Guy Van Hooveld 23March , 2008

Gains• Software engineering (addressing the pain areas)

– Better Requirements Management in terms of specifications, prioritization, traceability– Focus on non-functional aspects of usability, interoperability, performance, robustness– Importance of Execution architecture (State-transitions, CPU loading, Memory)– Increase in Design effort, upfront design discussions automatically reduce rework– Early involvement of test team – CTQ measurement method– CTQs as leading indicators of product quality as against PMI

• Cross-functional approach– Reduction in communication barrier – way of working with marketing

• e.g. challenging specifications, Kano, risk-benefit analysis– Common language of CTQs with other disciplines for synergy e.g. hardware, electrical

• End-user perspective– Development community sensitive to VOC E.g. needs versus wants– Measurement focus (variation rather than average), best case-worst case spectrum

• System understanding (Product perspective)– Transfer functions triggers to understand the system better (may expose competency)– Understanding of noise and its effect on the system (usage environment)– Trigger on “what could fail” (FMEA, mistake-proofing) improves robustness

• Soft Aspects – Mindset of “first time right” instead of “build-test-fix”

Guy Van Hooveld 24March , 2008

Issues– Everything is linked to CTQs so there is a chance of completely missing important

ones– Does not compensate for domain competency (CTQ flow down, FMEA etc)– Many requirements in software fall under Yes/No, Pass/Fail category so limit

setting limits, target becomes fuzzy (Critical factors rather than CTQs)– Out of limits need not strictly mean defective as in case of hardware e.g. start-up

time, hence predicting DPMO (defects per million opportunities) may not reflect reality

– Random failures due to only software are rare due to which concept like MTBF for software alone is questionable (however makes sense at product level)

– No concept of samples – the same piece of code is corrected and used, so hard-core statistical concepts cannot be applied at software level only (however can be done at product level)

– Configuration Management has to be addressed in the traditional way– No direct tools available to control regression and side-effects in software– Additional effort needs to be budgeted in the initial phases (confidence of Project)– With more and more projects going the ODM way (Black-box model) – not clear on

how to select/manage suppliers on software CTQs– Marketing community still needs to be educated to be able to specify CTQs in a

quantifiable way and as part of the requirements specification itself