7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 1 Standards & Assessments CMMI, ISO 9000,...

7/31/2007 SE 652 - 2007_7_31_CMMI_Software_Quality.ppt

1

Standards & AssessmentsCMMI, ISO 9000, TL9000

Sources: ASQ CSQE Primer

Introduction to CMMI

CMMI Distilled


2

August 4 Class

CMMI Introduction & Configuration Management Appraisal

ISO 9000/TL-9000

Due today (31-July): Cycle 2 Design & Code, hand off to System Tester

System Test Plan Inspected & Baselined

Project notebook updates including inspection records, meeting minutes, etc.


3

Topics

Audits & Assessments

CMM / CMMI & SCAMPI

ISO 9000: ISO 9001:2000, ISO 9000-3:1997, TickIT

Q9000, TL9000


4

Capability Maturity Model (CMM)

Created in 1987 by Software Engineering Institute (SEI)

5 level model based on proficiency in Key Process Areas (KPAs)

Migrating to Capability Maturity Model Integration (CMMI)Three source models:

– CMM for Software

– Systems Engineering Capability model

– Integrated Product Development CMM

CMMI v1.1


5

What is it?


6

Why Would I want one?

Required– Contractual

– Senior Management Decree (e.g. ROI of 7 to 1)

Sales Tool

Want to improve


7

Schedule Example

Organization 1Project Schedule Performance

0

1

2

3

4

1 2 3 4 5 7

# Pro

jects

Drop Page Fields Here

Count of Months Late

Months Late

Drop Series Fields Here

Organization 2Project Schedule Performance

0

1

2

3

4

5

4 5 6

# Pro

jects

Drop Page Fields Here

Count of Months Late

Months Late

Drop Series Fields Here


8

Process Capability

Ability of a process to produce planned results

• Predictable

• Measureable


9

Process Models

CMMI is model basedModel = structured collection of elements that describes characteristics of

effective processesProcess Area = cluster of related practices that when performed collectively,

satisfy a set of goals considered important for making significant improvement in that area

Processes selected are those proven by experience to be effective (i.e. best practices, practical knowledge from previous endeavors)

Notes: A process area is not a processA model is not a process

models show what to do, not how to do it!

Philosophy“All models are wrong, some are useful” – George Box


10

CMMI Models

Model Options:Software Engineering (SW)

Systems Engineering + Software Engineering (SE/SW)

Systems Engineering + Software Engineering + Integrated Process & Product Development (SE/SW/IPPD)

… + Supplier Sourcing (SE/SW/IPPD/SS)

Representation Options:Staged (Maturity Levels)

Migration from CMM to CMMI

Continuous (Capability Levels)Migration from EIA/IS-731 to CMMI

Recommended order for process improvements, but not prescribed …


11

Levels

Zero – Ad Hoc

One – Doing it (in Continuous, Ad Hoc in Staged)

Two – Process performed for individual projects

Three – Process focus at organizational level

Four – Projects and processes are quantitatively managed

Five – Projects and processes being optimized based on performance data & results


12

Representations Revisited

Continuous Model– 25 Process Areas each assessed at level 0-5

Configuration Mgmt = capability level 3

Risk Mgmt = not done (capability level 0)

Requirements Mgmt = capability level 2

– Result can be presented as a Kiviat chart

Staged Model– 25 Process Areas assigned to each of 4 Maturity Levels (see next slide)

– Result is a grade (1-5)


13

Staged RepresentationMaturity Levels (MLx)

1Initial

Process unpredictable, poorly controlled & reactive

2Managed

Process characterized for project & often reactive

3Defined

Process characterized by organization is proactive

4Quantitatively Managed

Process measured & controlled

5Optimizing

Focus on process improvement

Proj

ect

Org

aniz

atio

n

Project

Organization


15

Staged RepresentationProcess Area Mapping to Maturity Levels

5. Optimizing Continuous Process Improvement

Organizational Innovation & DeploymentCausal Analysis & Resolution

4. Quantitatively Managed

Quantitative Management

Organizational Process PerformanceQuantitative Project Management

3. Defined Process Standardization

Requirements DevelopmentTechnical SolutionProduct Integration

VerificationValidation

Organizational Process FocusOrganizational Process Definition

Organizational TrainingRisk Management

Decision Analysis & Resolution

2. Managed Basic Project Management

Requirements ManagementProject Planning

Project Monitoring & ControlSupplier Agreement Management

Measurement & AnalysisProcess & Product Quality Assurance

Configuration Management

1. Initial None


16

Continuous RepresentationProcess Areas

Process Management– Organizational Process Focus (OPF-3)– Organizational Process Definition (OPD-3)– Organizational Training (OT-3)– Organizational Process Performance (OPP-4)– Organizational Innovation & Deployment

(OID-5)Project Management

– Project Planning (PP-2)– Project Monitoring & Control (PMC-2)– Supplier Agreement Management (SAM-2)– Integrated Project Management (IPM-3)– Risk Management (RSKM-3)– Integrated Teaming (IT-3)– Integrated Supplier Management (ISM-3)– Quantitative Project Management (QPM-4)

Engineering– Requirements Management (REQM-2)– Requirements Development (RD-3)– Technical Solution (TS-3)– Product Integration (PI-3)– Verification (VER-3)– Validation (VAL-3)

Support– Configuration Management (CM-2)– Process & Product Quality Assurance

(PPQA – 2)– Measurement and Analysis (MA-2)– Decision Analysis and Resolution (DAR-3)– Organizational Environment for Integration

(OEI-3)– Causal Analysis and Resolution (CAR-5)


17

CMMI Assessment Cheat Sheet

Institutionalization – Ingrained Way of Doing Business that an organization follows routinely as part of its corporate culture

Specific Goals – Required model component that describes the unique characteristics that must be present to satisfy the process area

Specific Practice – Expected model component that is considered important to achieving the associated specific goal. The specific practices describe the activities expected to result in achievement of the specific goals of a process area. (In continuous representation – every specific practice (SP) is associated with a CL, in staged – all SPs are treated equally)

Generic Goal – Required model component that describes the characteristics that must be present to satisfy the institutionalization of the processes that implement a process area

Generic Practice – Expected model component that is considered important in achieving the associated generic goal. The generic practices describe the activities that are expected to result in achievement of the generic goal and contribute to the institutionalization of the processes associated with a process area.


18

CMMI Assessment Cheat Sheet (continued)

Managed Process:– Performed process planned & executed in accordance with policy– Employs skilled people– Adequate resources– Produces controlled outputs– Involves relevant stake holders– Monitored, controlled & reviewed– Evaluated for adherence to process description

Defined Process:– Managed process tailored from the organizational standard processes– Maintained process description– Contributes work products, measures & other process info to organizational process assets

Performed Process– Accomplishes needed work to produce work products– Specific goals of the process area are satisfied

Establish & Maintain– Includes documentation & usage:

• Planned• Documented &• Used


19

Configuration Management (CM)Assessment


20

DeMarco & Lister on Process

Organizations driving to be SEI Level 5 (at least level N+1)

Standards are good, but …

Most success centered around standard interfaces

Mandating a “best practice” is a bad practice

Process improvement is good, but process improvement programs aren’t

Competent people improve processes all the time (pride, growth, etc.)

Formal process improvement moves responsibility from the individual to the organization

Process improvement programs focus on process rather than product(making a poor product efficiently is often worse than making a good product poorly)

Focus on process “level” tends to make organizations risk averse

“The projects most worth doing are the ones that will move you down one full level on your process scale!”


21

Break


22

Quality Standard Rationale

Customers want & need assessments of supplier quality

Means:Individually audit (i.e. qualify) vendor:

Specific products

Processes (e.g. manufacturing, design & development, support)

Alternative:Common Quality Assurance standards & audits


23

Major Audit Types

First Party Audit

Within own company (aka internal audit)

Used to measure own performance, strengths & weaknesses against internally established procedures & systems

Second Party Audit

Performed by customer on their supplier (aka external audit)

Third Party Audit

Outside, independent auditor contracted to audit on behalf of company or a supplier (e.g. ISO 9000 registration audit)

Assessments (e.g. SCAMPI)

Similar to first party audit, but typically performed by external assessors


24

Other Audit Types

System Audit – examination of bigger picture of organization &/or project

Typical cross organizational, cross process & cross product

Process Audit – verify inputs, actions & outputs in accordance with defined requirements (e.g. software inspections)

Product Audit – final product or service for “fitness for use”

Customer oriented

Compliance Audit

Regulatory – audit to government regulations

Management – audit to organizational rules, effectiveness & conformance

Quality – systematic & independent of quality activities vs. established procedures


25

ISO 9001:2000

Objective

Provide confidence that vendor can produce quality products

Assumptions: good practices will produce good products

Standard for assessing organization’s Quality Management System (QMS)– Processes

– Activities

– Behaviors

– Training

But, ISO focuses on Quality Assurance not Quality Control

ISO-9001 certification does not guarantee quality products!


26

Tenants of ISO 9001

1) Say what you do

2) Do what you say

3) Prove it!


27

ISO 9000 Audits

Customers write requirements for current ISO-9001 certification into purchasing contracts

Organizations apply for 3rd party audit,end result is ISO-9001 certification

ISO International Accreditation Forum (IAF) board

Audits national accreditation boards (i.e. one board each nation)

Who register individual registrars (e.g. Lloyd’s, DNV)

Who audit organization internal auditors (e.g. Lucent Optical Networking) & spot check

Who audit design, development, manufacturing & support teams within the organization


28

ISO Alphabet Soup

ISO 9000:2000

Overall framework, fundamentals of quality management systems & terminology

ISO 9001:2000

Requirements for quality management systems (qms) & what is required to demonstrate compliance

ISO 90003 2004 (previously 9000-3)

Guidelines for the application of ISO 9001:2000 to computer software

ISO 19011

Guidelines for auditing quality and environmental management systems


29

What is wrong with ISO 9001?

Vendors ISO-9001 certified, but quality still elusive!No visibility into supplier quality levels

Not getting quality levels they wanted

Solution:TL9000 (Quest forum, telecommunications)

QS9000 (automotive)


30

TL9000

ISO on steroids

Wholly subsumes ISO 9001-2000

Requires vendors prove they are actually improving

Metrics focused on cost drivers of service providers:Know vendor is measuring

Visibility into quality improvement results


31

TL9000 Top Management Requirements

Monitor & improve customer satisfaction

Set long & short term objectives for organization effectiveness

Set targets for TL9000 product performance metrics

Use an explicit life-cycle model

Establish a quality improvement program

Periodic management review of quality system


32

TL9000 Metrics

Cross-discipline metrics– # of problem reports

– Problem report fix response time

– Overdue problem report fix responsiveness

– On-time delivery

Hardware & Software measurements– System Outages

Hardware measurements– Return rates

Software measurements– Software installation & release application aborts

– Corrective patch quality

– Feature patch quality

– Software update quality


33

TL9000 Common Audit Questions

• Do you know how to find your Quality Policy, QMS and the processes you should be using for your work?

• Do you know your organization’s product delivery & improvement goals and what you must do to support them?

• Do you know what skills you should have?

• Do you know what you have to do to approve/baseline/finalize your documents, designs & code?

• Do you know how to store & find records of reviews, inspections, key decisions, etc.?

• Do you know what to do if a problem is found with the product or process?

• Do you know your organization’s performance with respect to customer satisfaction, quality of delivered products & process execution?


34

TL9000 Sample Requirements

Planning– Must have methods for estimating & tracking

– Determine where you will do reviews & tests

– Risk management plans, customer, user & supplier involvement in reviews & evaluation

Software Outputs– Requires architecture, detailed designs, code & user documentation

– Each design thread must be reviewed at some point prior to integration or system test

Software Testing– All testing must have test plans; test process must be documented

– Plans must include test cases with inputs, output & test success criteria

– Plans must include types of testing, requirements traceability, coverage definition & measurement, test environment, defect handling, et.al.

– Integration testing specifically required


35

Team Project Postmortem

Tracking process improvements during projectProcess Improvement Proposals (PIP)

Port-Mortem

Areas to considerBetter personal practices

Improved tools

Process changes


36

Postmortem process

Team discussion of project data

Review & critique of roles


37

Postmortem process

Review Process DataReview of cycle data including SUMP & SUMQ forms

Examine data on team & team member activities & accomplishments

Identify where process worked & where it didn’t

Quality ReviewAnalysis of team’s defect data

Actual performance vs. plan

Lessons learned

Opportunities for improvement

Problems to be corrected in future

PIP forms for all improvement suggestions

Role EvaluationsWhat worked?

Problems?

Improvement areas?

Improvement goals for next cycle / project?


38

Cycle Report

Table of contents

Summary

Role ReportsLeadership – leadership perspective

Motivational & commitment issues, meeting facilitation, req’d instructor support

DevelopmentEffectiveness of development strategy, design & implementation issues

PlanningTeam’s performance vs. plan, improvements to planning process

Quality / ProcessProcess discipline, adherence, documentation, PIPs & analysis, inspections

Cross-team system testing planning & execution

SupportFacilities, CM & Change Control, change activity data & change handling, ITL

Engineer Reports – individual assessments


39

Role Evaluations & Peer Forms

Consider & fill out PEER forms

Ratings (1-5) on work, team & project performance, roles & team members

Additional role evaluations suggestions

Constructive feedback

Discuss behaviors or product, not person


40

Project Notebook

Updated Requirements & Design documents

Conceptual Design, SRS, SDS, System Test Plan, User Documentation*

Updated Process descriptions

Baseline processes, continuous process improvement, CM

Tracking forms

ITL, LOGD, Inspection forms, LOGTEST

Planning & actual performance

Team Task, Schedule, SUMP, SUMQ, SUMS, SUMTASK, CCR*


41

August 4 Class

CMMI Introduction & Configuration Management Appraisal

ISO 9000/TL-9000

Due July 31: Cycle 2 Design & Code, hand off to System TesterSystem Test Plan Inspected & BaselinedProject notebook updates including inspection records, meeting minutes, etc.

Deliverables for August 7Project Postmortem (cycle report)Cycle 2 presentationsPeer Feedback formsCompleted project notebooks

Cycle ExitCompleted project (source, documents & all quality records)

7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 1 Standards & Assessments CMMI, ISO 9000,...

Documents

Transcript of 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 1 Standards & Assessments CMMI, ISO 9000,...