Leveraging Quality For Competitive Advantage · John Vu Engineering, Operations and Technology...

John VuBOEING is a trademark of Boeing Management Company.Copyright © 2006 Boeing. All rights reserved.

Leveraging Quality For Competitive Advantage

(A Presentation by The Boeing Company)

John D. Vu Ph.DTechnical Fellow

The Boeing Company

John Vu

Engineering, Operations and Technology | Information Technology

Copyright © 2006 Boeing. All rights reserved.

Global Demand For IT Workers

1995 2000 2005 2010 2015 2020

15 M

12 M

9 M

6 M

3 M

1 M

0.5 M

Source: International IT education conference 2005

Supply: number of Bachelor Degrees in IT in global schools

Global Demand for IT workers

3 Million shortage

10 Million shortage

John Vu



Global Shortage Of IT Workers

• Competitive global economy creates more demand for IT skilled workers and demand has exceeded supply. Business must hire workers globally to fill this gap or subcontract to reduce future impact

• Current IT subcontracting market:– 45 percent to India

– 18 percent to Canada and Europe

– 18 percent to China

– 8 percent to Russia and E. Europe

– 6 percent to S. America

– 5 percent to S. E. Asia (Singapore, Malaysia, Philippines)

Source: Gartner Group 2005

John Vu



Shortage Of IT Workers = Business Opportunities

• Projected shortage of 3 million information technology (IT) workers in 2010 and 10 million IT workers in 2020 globally; both India and China are also experiencing shortage of IT workers but have plans to train more – up to 3 million to fill this gap by 2012

• IT Subcontracting is a USD $100 Billion market in 2004 and couldreach USD $200 Billion by 2010

• IT Workers in China and India are pushing their regional economies to a new high level

• India’s IT business continues to grow 20 percent annually with revenueof USD $48B annually


John Vu



Current Opportunity …

• Most major subcontracting deals, estimated at USD $118 billion in annualized contract value (ACV), are set to expire between 2006-2008

• Approximately one quarter of these deals have already been renewed, based on confirmation by key suppliers in 2006, but the majority of these contracts, valued at $88 billion, have not

• New subcontracting deals, estimated at USD $98 billion in contract value are projected in the next ten year 2006 - 2016

• Many countries and suppliers are positioning themselvesaggressively for this business


John Vu



Factors Affecting Client’s Choice

Cost Access to Low-cost Labor Pool, Attrition Levels

Infrastructure Telecom, Power and Office-space cost, Transportation conditions

Quality Quality Commitment, Process maturity, Other Certifications

Skills Education Systems, Training, Communication

Knowledge Experiences, Domain expertise, SpecializationCustomization

Source: CMU subcontracting Benchmarking study 2004

John Vu



Change In Determinant Factor With Time

Low Cost Labor

Infrastructure

Quality Delivery

Skills

Client’s Domain Knowledge

Time

Rev

enue


John Vu



Market Outlook: Low Cost Competition

• The large pool of highly skilled workers in low-wage countries is growing quickly, fueled by a strong increase in information technology graduates with average growth > 5.5 percent annually

• The number of degrees awarded in information technology jumped from 15 percent to 45 percent in Philippines, Malaysia, Indonesia in past five years

• Currently India accounts for nearly 30 percent of IT professionals while China has 11 percent of global workers

• The share of degrees awarded in business management jumped from 18 percent to 31 percent in Russia, and from 16 percent to 36 percent in Poland in just five years


John Vu



Low Cost Subcontracting Suppliers

• India: Low Cost Provider with experiences • Indian salaries are rising > 15 percent per year• High employee turn-over > 21 percent• Shortage of skilled workers

• China: Low Cost Provider with strong infrastructure• Language barrier (English)• Project management needs improvement• Shortage of skilled workers

• Eastern Europe: Low Cost Provider with Western culture• Growing market segment due to young workforce• Good command of English language• Strong government support (Romania, Poland, Czech Republic

and Hungary)• Philippines: English-speaking low-cost provider

• Alternative Asia destination• Good English skills throughout its population


John Vu



What Worked and Did Not Worked

Subcontracting results are mixed:The 2004 - 2005 Gartner/Dataquest survey found that:

1) 82 percent report moderate satisfaction 2) 37 percent report canceling contract3) 55 percent report renegotiating contract

The 2006 Information Week’s study found:1) 57 percent of contracts met obligations2) 25 percent of contracts were cancelled due to poor performance 3) 14 percent of contracts were not fully satisfactory

Study also found many low-cost suppliers failed to perform due to poor quality, not meeting clients’ schedule and functionality

John Vu



Expand Market Beyond Cost …

• Many global companies that subcontracted have gone beyond cost as the main factor and are seeking quality and domain expertise in specific areas that can meet their needs

• Improving quality and meeting client’s business needs in both financial and technical are the key success factors in expanding subcontracting business

• Vietnam’s IT subcontracting companies need to focus on bothquality and understanding of client’s needs to capture this market

• Vietnam’s IT industry needs to improve education and training in higher value skills that are in high demand today and in the future


John Vu



Software Quality

• Software quality is becoming increasingly important. Software is now using in many applications and software defects has caused serious damage and even physical harm.

• Defects in financial applications can be very costly but defects in software that fly airplanes, drive cars, operate machines, or run manufactories can be very dangerous.

• Software suppliers that do not have a good quality management systems are risking their business in a highly competitive world.

• Many global clients are now consider quality as the key factor in selecting software suppliers.

John Vu



Cost of Software Defects

Requirement

Analysis

Requirements Definition $1

Design$5

Coding $20

Testing $50

Maintenance$100

Relative cost factor to find and fix defects at each phase of the Software Development Life Cycle

Source: Boehm, B., Software Engineering Economics, Prentice-Hall, 1981

Relative cost factor to find and fix defects at each phase of the Software Development Life Cycle

Source: Boehm, B., Software Engineering Economics, Prentice-Hall, 1981

BUGS

John Vu



This cycle can take from10-30 hours per defect

Using an average of $15 per hour for Software Engineering time...

Cost per defect is:

30 * $15 = $450

For a product with 100 defects, the cost is:

100 * $450 = $45,000

Find/Fix Cycle

Development includes fix in next baseline

New release to SQA

SQA performs regression testing

Testing found a defect

Defect reported

Defect being investigated and verified

Defect fixed and test

John Vu



Quality Issues

• Low quality product = Higher cost• Low quality product = Take longer time• Low quality product = May not meet schedule• Low quality product = May not have all features• Low quality product = May not meet customer’s needs• Low quality product = Unhappy customer• Low quality product = Bad for business

Almost half of software subcontracting contracts did not meet quality requirements and end up costing subcontracting suppliers their business

Many customers are willing to pay higher prices for higher quality products

Customer satisfaction = key factor to success in global subcontracting business

John Vu



Improve Quality = Focus on Process

The quality of a software product is governed by the quality of the processes used to develop and maintain it

To improve quality of the product, one must improve the quality of theprocesses that create the product

John Vu



Software Process

• A process is a sequence of steps performed for a given purpose• A software process is a set of activities, methods, practices, that

people employ to develop and maintain software and the associated products (e.g. project plans, design documents, code,test cases, and user manuals)

Source: The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, 2005

Note: a process is what people do - not what documentation says they should do

John Vu



Context

Software Development and Maintenance Process: The development and maintenance of products that meet the needs of customers and markets

Software Improvement Process: Application of technology and disciplines to improve software process to meet business goals and objectives

Business Goals, Objectives Implementation

plans and measures

Customer needs and Market trends

Process Improvement

Software Development and Maintenance ProcessSoftwareProduct



Capability Maturity Models Integration (CMMI)

Created at Software Engineering Institute (SEI), a federally funded research lab at Carnegie Mellon University (CMU) as:

• Frameworks for process improvement based on state-of-the-art software and systems engineering practices.

• A benchmarking model to compare capability of software organizations.

• A model tool used to select software and systems contractor by U.S. Department of Defense (DoD).

History:

Software CMM (1989); System CMM (1994); People CMM (1996);CMM Integration (CMMI) 1999; IPD CMM (1999) SA-CMM (2002),CMMI Acquisition (2006)

John Vu



Global CMMI Usage

• Capability Maturity Model Integration (CMMI) is widely used worldwide:

• Over 3500 appraisals conducted (1992- 2004), more than half were outside the U.S.

• CMMI in regions around the world:• Asia/Pacific• Western Europe• Eastern Europe• S. America• Australia

Source: CMU’s Benchmarking Study 2004

Maturity Profile Worldwide

0

5

10

15

20

25

30

35

Level 1 Level 2 Level 3 Level 4 Level 5

John Vu



The CMMI Maturity Levels

1

2

3

4

5

Process unpredictable, poorly controlled, and reactive

Process characterized for projects and is often reactive

Process characterized for the organizationand is proactive

Process characterized for the business where critical processes are measured and controlled

Focus on continuous process improvement of the business, organization, and projects

Optimizing

QuantitativelyManaged

Defined

Initial

Managed

Optimizing

Defined

John Vu



The Capability Maturity Models

The CMMI have become a de-facto standard for assessing and improving software processes

SW-CMM (earlier versions) – 1988

SW-CMM V1.1 – 1991

CMMI V1.0 – 1998

CMMI V 1.1 – 2001

CMMI V 1.2 – 2005

Process Improvement using CMMI is > 16 years

John Vu



Failure To Improve

Industry data found that

• 72 percent of organizations report little or no success in software improvement after an assessment

• 83 percent of organizations abandon their improvement efforts in the first three years

• 57 percent of organizations that abandoned improvement efforts did restart their improvement later

• Less than one percent of organizations claimed success or reported any improvement data

* SEI Software Industry Benchmarking Study 2004

John Vu



Why Do So Many Improvement Efforts Fail ?

1. Over-emphasis on passing an appraisal but not much focus on making quality improvement happen

2. Focus on maturity levels without measurable objectives

3. Lack of a skilled personnel to manage quality improvement activities for business advantages

4. Confusion between terminology and actual practices

5. Implementation of improvement solutions poorly managed

* Software Industry Benchmarking Study 2001Source: SEI Software Industry Benchmarking Study 2004

John Vu



Lessons Learned

• Appraisal consumes a lot of effort– Training– Logistics and Preparation – Artifacts, documentation and reviews– Interviewing– Briefing

• Appraisal is expensive– Hiring qualified lead assessor– Training and tools (templates and documents)– Consulting fees and follow-on activities

• How many appraisals are needed?– Several attempts to achieve a maturity level?– The need to buy tools and templates to pass the appraisal



Some People Believe that...

Level 2 by 2002…Level 3 by 2003…Level 4 by 2004…or else!

CMMI

“The goal of process improvement is the maturity level”

Many organizations confuse maturity level with improvement objectives

Some organizations believe the maturity level is the “miracle” that can make business happen



Business Balanced Scorecard

Time to Market ?

Productivity ? Operational Efficiency ?

Operating Costs ?

Customer Satisfaction ?

Skills and Knowledge ?

Profit and Loss ?

Maturity levels are meaningless …if they cannot be explained in terms of business objectives

However, The Fact Is...

Quality ?Revenue ?

Market Share ?



The Need To Link Improvement To Business Value

Organizations need to understand that the goal of improvement must be based on business needs, NOT maturity levels.

Management must communicate process improvement in terms of…

Business Value:Increase QualityReduce CostReduce Cycle TimeIncrease ProductivityIncrease Customer SatisfactionIncrease Employee Satisfaction

If process improvement is not directly aligned with business goals and objectives,it has no value and should not be allowed to continue.

John Vu




Many organizations believe CMMI is a requirement, and they must document everything according to this book

Many organizations are interested in a maturity level and doing just the minimum to pass an appraisal

This approach will not improve the business or help the organization to succeed in a highly competition world

“Just document all CMMI practices and select few projects to represent the organization for the appraisal then claim

the entire organization has achieved a high level of maturity”

John Vu




Process improvement is about solving critical business problems, NOT achieving meaningless maturity levels

Appraisals must focus on both process compliance and organizational performance

The appraisal method requires organizations to disclose the scope of the appraisal, and all improvements must be measured as part of the business value stream

Appraisal is more than just reviewing documents. Processes must be practiced, used, measured, continuously improved

to achieve business goals and objectives.

John Vu




Many organizations are buying CMMI templates to substitute for process documentation

Having documents according to a CMMI book does not reflect your company’s processes or the way your company does business

Documentation alone will not lead to improvement. All processes must be used, measured, verified and institutionalized

“Just fill in this template -- improvement will come!The CMMI requires documentation

These templates are your document”

John Vu




Managers must identify their workforce’s competencies and provide training to improve the way the organization develops software

Managers must facilitate and coordinate improvement activities and measure improvement benefits

Skills and knowledge are key assets to improving the business

Process improvement is about changing the way people develop software by practicing a disciplined engineering

approach to meet the organization’s business goals.

John Vu



Context

Schedule Improvement

Quality ImprovementQuality Improvement

Cost/Cycle TimeImprovement

BusinessValue

Scope of Improvement

Enterprise

Business Unit/Domain

Organization

Project

Area Of Management

Project Management

Product Management

ProcessManagement

Product LineManagement

1

2

3

4

5

John Vu



John Vu



Context




BusinessValue


Enterprise


Organization

Project

Area Of Management

Project Management

Product Management

ProcessManagement


1

2

3

4

5



Comparison Between Level 1 and 2

Level 1 Level 2

Ad-hoc process Disciplined process

Undocumented practices Documented practices

Schedule is fixed Schedule is negotiable

Guesstimates Project estimates

No measurement Project measurement

Reactive mode Get out of reactive mode

No facts and data Facts and data being collected



Overview Of Level 2

RequirementsManagement

Configuration Management

Approved Requirements

Process and Product

QualityAssurance

Supplier Agreement

ManagementReview Progress

Commitments

ProjectPlanning

Project Monitoringand Control

Commitments

Commitments

Commitments



AllocatedRequirements

Measurement and Analysis

John Vu



What Is Software Project Management?

Definition:The application of knowledge, skills, tools, and techniques to software project activities in order to meet customer’s needs and expectations

Meaning:Complete the software project:• On time• Within budget• According to plan• According to customer’s requirements

John Vu



Software Project Management Phases

1) Initiating• To gain an understanding of all aspects of the project• To define roles and responsibilities• To determine “completion” and success criteria

2) Planning • To determine the project scope• To define actions required to achieve project objectives• To determine resources needed for the project• To define the means to control the execution of the project

3) Executing• To manage the process defined to deliver the product• To effective mitigate project risks• To measure project performance• To manage project performance• To manage project integration with others

John Vu



Software Project Management Phases

4) Controlling • To track progress against the plan• To take corrective actions when there are deviations from

the plan5) Closing

• To deliver software product to customer• To conclude the business relationship in the context of the

project

John Vu



Project Planning

• Project plans are the basis for executing a project. The projectmanager's ability to control a project is highly dependent on the accuracy and completeness of the plan

• The project plan development the most important part of the project

• A good project plan includes estimates, schedules, resources, manpower, skill requirements, and risk management

• The plan should also include how configuration management, quality assurance, progress and earned-value tracking, and requirements management will be accomplished

• Key personnel skills and key technologies should also be identified along with management reserves for high-risk elements of the plan



Project Planning Process

Decomposerequirements

Develop software

yes

EstimatesActuals

Tracking Actuals vs Estimates

Estimation

Initial Requirements

(Scope, FunctionalitySchedule, Budget)

EstimateSize

EstimateResource

EstimateSchedule

Re-Negotiate

Is Schedule Meeting

Customer’s Requirements

No

John Vu



Improve Project Estimates

The utilization of historical data will improve project performance by reducing the variation in project estimates.

• Better estimates will improve project schedules.• Better schedules will improve project management.• Better management will improve project performance in terms of

increase quality and reduce costs.• Better performance will improve customer satisfaction.• Satisfied customers will improve relationships.• Better relationships will improve the business.

Source: Lessons Learned From projects – John Vu - 2004

John Vu



Software Estimates(Actual vs Planned)

Sche

dule

Var

iatio

n

- 148percent

+ 22percent

-125percent

+26percent +20

percent

- 24percent

+12percent

-18percent

+4percent

- 7percent

Utilize Historical data for all estimates

Ove

r est

imat

esU

nder

est

imat

es

Improvement In Project Estimates

Source: CMU Data collected from 180 software projects - 2005

CMMI Level 4 and 5



A Project Plan

A Project Plan should include the following:• The project’s chosen software life cycle• A list of products to be developed (Deliverables)• Schedules• Estimates (cost, size, effort, etc.)• Facilities, support tools, hardware• Project risks -- identify, mitigate

Plans must be based on estimates

In creating estimates for size, effort, cost, schedule, etc:Use historical data, where availableDocument assumptions and estimates

John Vu



Project Monitoring and Control

Provide understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan

Meaning• Monitor progress against project plan:

– Actual vs. planned (cost, effort, schedule, milestones, etc.) – Progress reviews (earned value, return on investment, etc.)

• Monitor risks (data, skills, turnover etc.)• Analyze issues• Take corrective action• Track corrective action to closure

John Vu



Monitoring and Control

Project MonitoringProject manager compares progress of product development againstthe project plan and discusses variances with project team to identify problems and initiate a search for possible solutions

Manage Changes to ScopeProject manager gives visibility to changes in the scope. Changerequests are used to define and justify changes in scope

Control issuesProblems that prevent the project team from working or being successful are controlled by project manager

Project reviewsAllow for the approach and process being followed by the projectteam to be evaluated and changed, if needed

John Vu



Monitoring and Control

Technical reviewsProject manager validates the technical decisions and directionsadopted by the project

Documentation controlPerformed by Configuration Management function

Product delivery controlPerformed by Changed/Release Management function

Lessons Learned ...Use Wrap-Up activities at the end of the project to produce Lessons learned and reassign personnel and equipment

John Vu



Project Requirements

• The key to the success of any project is the establishment of project requirements; goals and outcomes are derived from requirements and communicate expectations between customer and developer

• On successful projects, requirements are clearly understood by user and developer. Additionally, requirements must be complete, correct, precise, consistent, relevant, feasible, manageable, testable, traceable, and free of design detail

• Requirements must be controled. Project managers who run successful projects control requirements and ensure there is a common understanding of requirements between customer and developer

John Vu



Successful Tips:

• Freeze requirements before planning a project

• If requirements are changed or added, replan the project

• For organizations that maintain software and develop periodic software updates, freeze the set of requirements for each software release and provide a mechanism to allow a new high-priority requirement to be added to the current development baseline

John Vu



Requirements Management

Manage the requirements of the project and identify inconsistencies between those requirements and the project’s plans and work products

Meaning

1. Obtain requirements from customer2. Document requirements 3. Control requirements changes4. Maintain bi-directional traceability of requirements5. Identify inconsistencies between project work and requirements

Tips:• Requirements must be documented• If requirements change, the changes must be documented and

all resulting necessary changes in other documents must be tracked and verified

John Vu



Some People Believe That ...

The requirements are not clear, but we can work out the details later …

Most software requirements are ambiguous and not well communicated

Many engineers want to start the project quickly and not take time to work out the detailed requirements

Customer

Engineer

John Vu



The Fact Is ...

Requirements must be:

1. Documented2. Clear and concise3. Understood4. Testable5. Usable

Project Manager must work cooperatively with customerto establish and maintain a requirements baseline

to ensure all requirements are expressed in simple, unambiguous terms, quantified and verifiable

Manager

John Vu




Establish and maintain the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits

Meaning– Identifying configuration items/units– Systematically controlling changes

– Establish baselines– Control changes

– Maintaining integrity and traceability of the configuration throughout the software life cycle

– Reporting accurate status to all stakeholders



The CM Disciplines


(CM)

Configuration Identification

Change Management

Baseline and Integrity Control

Configuration Audits and Review

Status Accounting


Planning




Discipline Question AnswerPlanning What CM activities are All CM activities are identified in

performed? The CM plan and included inthe project schedule

Identification What is my software The software consists of theconfiguration ? following items ....., each of

which has its own versiondesignation .....

Change How are changes The steps are .....Management managed ?

Integrity and How do you baseline The baseline content is ...Traceability and control changes ? The version is ….

Status accounting What changes have The current configuration been made and when ? was changed as follows ....

Audits and Review How do you make sure The CM process is reviewedchanges are correct and for compliance per scheduledthe process is followed? reviews and outside audits



Change Management Flow

Change

Change Board

Analyst

Change Board

Programmer

Tester/QA

CM Librarian

Change Board

Release

Approve/Reject

AnalyzeEstimate Recommend

Authorize

Implement

Control/Storage

Verify

ApproveAuthorize

Baseline



Change Control Board (CCB)

Change Board needs following information to make decision for each change:

Size (How big is the change ?)

Complexity (Is the change within a component or involve others ?)

Schedule/Priority (When do you want it ?)

Impact (Consequences of this change ?)

Cost (Potential cost or cost saving by making this change ?)

Severity (How critical is the change ?)

Relationship to other changes (Superceded or invalidate others ?)

Test (Any additional test required ?)

Resource (How many people needed for this change ?)

Benefits (Advantages by implementing this change ?)



Perform CM Audits

Audit should be performed periodically to ensure the CM practicesand procedures are consistently followed.

Internal audit (performed by CM staff)

External audit (performed by QA)

CM audit must be planned and scheduled (in CM plan)

CM audit must be performed before every major baseline change

CM audit team must be trained and NOT involved in tasks being audited

CM audit must verify that all changes to baseline are implemented properly

CM audit must verify that the CM process is being followed appropriately

CM auditing is continuous throughout the life of a project

John Vu



Process and Product Quality Assurance

Provide staff and management with objective insight into processes and associated work products

Meaning

• Objectively evaluate processes, work products, and services against the applicable process description, standards, and procedures

• Identify non-compliance issues

• Provide feedback to project staff and managers

• Ensure non-compliance issues are addressed

The value of QA is that it provides an independent view of projectactivities, process, and product



Quality Assurance Responsibilities

Review all plans (project plan, CM plan, testing plan, etc.) for completeness

Participate in project and phase reviews as indicated in project plan

Review all test plans for compliance to standards

Review a sample of all test results for compliance to plans

Audit CM performance to determine compliance to standard

Participate in organization establishment of standards, procedures, processes

John Vu



Measurement and Analysis

Develop and sustain a measurement capability that is used to support management information needs

Meaning1. Specify the objectives of measurement and analysis2. Define the measures, data collection and storage, analysis

techniques, reporting, and feedback 3. Implement project measurement4. Provide objective results to support management

decisions and corrective actions

Without data, quality is just talk

John Vu



Measurement Process

Identify ScopeObjectivesPurpose

Measures

Define ProcedureMeasuresCollectionAnalysisFeedback

Collect DataRecord and Store data

Revise and ValidateExtract

Analyze DataPrepare Data

ChartReport

Recommend

Take ActionsImprove Process

Evaluate ProgressEvaluate process

Recommend

John Vu



Example of Project Measures

1. Defect:• Pre-released• Post released

2. Efforts:• Plan vs. actual

3. Cycle time:• Cycle time per development phases• Cycle time per each product release

4. Project estimates:• Size, effort, time (plan vs. actual)

5. Customer satisfaction6. Employee satisfaction

John Vu



Context




BusinessValue


Enterprise


Organization

Project

Area Of Management

Project Management

Product Management

ProcessManagement


1

2

3

4

5



Moving From Level 2 to Level 3

At level 2, the focus is on project management

At level 3, the emphasis shifts to consistent processes across the organization

• Assets are gathered across the organization

• Standard processes are defined based on assets

The organization supports projects by establishing:

• An organization’s standard processes

• An organization’s common measurements

• An organization’s required training program



Comparison Between Level 2 and 3

Level 2 Level 3

Stabilizing the environment Establish standard processes

Project management Integrated project management

Processes are repeatable Processes are shared

Project management discipline Standardize project management

Historical data is collected Historical data is analyzed and used

Project measurements Project and organization measurements

Skills and knowledge exist Skills and knowledge are trained

Some training is available Some training is required

John Vu



CMMI Level 3

At level 1: The organization invests resources primarily in code and testing, but not project management resulting in poor quality and significant rework

At level 2: The organization begins to take control by investing resources in project management functions so projects can be developed more efficiently and predictably

At level 3: The organization organizes better by focus on standardized processes, where all projects share best practices for more consistency and better performance

By level 3, All projects should be able to show measurable improvements in both schedule performance and quality improvement

John Vu



Implement Level 3 Processes

Level 3 focuses on process standardization across the organization; therefore, a sequence of logical steps must be initiated to identify assets, organized into standards and applied to all projects to achieve maximum results:

1) Organization Process Focus (OPF)Set up infrastructure to collect assets for improvement

2) Organization Process Definition (OPD)Define organization standards based on assets

3) Organizational Training (OT)Train people to use organization standards

4) Integrated Project Management (IPM)Apply organization standards in all projects Get all stakeholders working together as a team

John Vu



Implement Level 3 Processes

5) Requirement Development (RD)Establish product requirements

6) Risk Management (RSKM)Identify and mitigate risks

7) Decision Analysis and Resolution (DAR)Evaluate alternatives and select solutions

8) Technical Solution (TS)Apply engineering methods and tools to solve problems

9) Product Integration (PI)Assemble work products from components

10) Verification (VER)Conduct informal and formal reviews to verify work products

11) Validation (VAR)Perform formal testing to validate work product

John Vu



Organizational Process Focus

Plan and improve organizational processes based on a thorough understanding of the current strengths and weaknesses of the organization

Meaning1. Establish goals and objectives2. Appraise the organization’s processes3. Identify the organization’s weaknesses and strengths 4. Establish improvement plan5. Implement improvement plan6. Collect organizational process assets7. Incorporate process-related experiences into the

organizational standard processes

John Vu



CMMI and Business Goals

Task

Process Improvement Plan

Organization’s Business Goals

CMMI Current Process Capability

Training

Reviews

Implementation

Measurements

Improvement Tasks

CMMI is a guide

Business goals are Key drivers

Plan is based on appraisal results

Data are used to verify im

provement results

John Vu



Establish and maintain a usable set of organizational process assets

Meaning1. Establish Standard Processes2. Establish Life Cycle model description3. Establish tailoring criteria and guidelines4. Establish the organization’s measurements repository5. Establish the organization’s Process Asset Library

Organizational Process Definition

John Vu



Organization’s Standard Processes

The organization’s standard processes are the means by which the organization expresses requirements that all projects’ development processes must follow

The objective of the standard is to establish commonality across the organization’s projects, in order to ensure consistency among projects, support common measurements, reduce waste, and redundancy by standardization and continuously improve its process

The standard is the high level of abstraction where it contains all the primary processes and rules that all projects are expected to follow

The standard establishes common terminology to support better communication between managers, customers, users, and developers



Organizational Process Assets

A collection of practices, entities, maintained by an organization, for use by projects in developing, maintaining, implementing,and measuring their practices

Process assets include:

The organization’s standard processes

Descriptions of life cycles approved for use

Guidelines and criteria for tailoring standard processes

Measurement repository (historical database)

Process Assets Library (PAL)



Organization’s Measurement Repository

A central repository for organization’s measurement data

Contains:

Actual measurement data (the numbers) from individual projects

The related information needed to understand the measurementdata and apply it to new projects

This is where planning and replanning data for the organizationas a whole are kept



Process Assets Library (PAL)

A repository where “Assets” used on past projects are kept

It also contains:

Lessons learned reports, example documents

Templates

In general, the PAL contains documents that can be used asmodels or examples for future projects

Assets = Artifacts with data

John Vu



Documentation

The project plan documents all considerations and tailoring aspects of the organization’s standard processes into the project’s defined process where it is used to manage specific project

Managing according to a plan:

The project plan is used to manage the project

Process Management:

Tailoring is a key element in the organization’s ability to develop reasonable and efficient project plans for use in managing and controlling projects according to a standard process

John Vu



Organizational Training

Develop the skills and knowledge of people so they can perform their roles effectively and efficiently

Meaning:

1. Identify training needs for every project2. Establish strategic training plan3. Develop training capability4. Deliver the training5. Establish training record6. Assess training effectiveness



Training Needs and Plan

ProjectTasks

Critical Skills Training Needs

Organization’sTraining Plan

Skills Training

Training Opportunities

Develop/Procure training

TrainingEffectivenessFeedback

John Vu



Providing Training

OrganizationTraining Plan

Individual careerdevelopment plan



TrackingTraining

MeasuringTraining effectiveness

John Vu



Integrated Project Management

Establish and manage the project and the involvement of the relevant stakeholders according to an integrated and defined process that is tailored from the organization’s set of standard processes

Meaning:

1. Tailor the standard process into the project’s defined process2. Plan the project by using organization assets3. Integrate plan4. Manage project according to project plan5. Implement project6. Provide data to measurement repository7. Contribute to Process Asset Library



Organizational Processes and Project Processes

Tailoring GuideLife Cycle

Requirements

Process Assets

Process Architecture

Organization’s Standard Processes

Project Defined Process

Project Plan

Activities

John Vu



Integrated Workflow

Document The

IT PlanEstablish System

Requirements Monitor Work and

Resolve Issues

CoordinateCritical

DependenciesReview

Work productsManage

Intergroup issuesConduct

Technical Interchanges

CustomerRequirements

John Vu



Product Integration

Assemble the product from components and ensure that the integrated product functions properly

Meaning:1. Determine the process for integration2. Define integration procedures and criteria3. Review interface descriptions for completeness4. Manage interfaces5. Assemble product components6. Evaluate the integrated product7. Package and deliver the product

John Vu



Product Integration

Assemble Product Components

and Deliver the Product

IntegrationPlan

Prepare forProduct Integration

Technical Solution

Ensure Interface

Compatibility

Assemblies

Sub-assemblies

John Vu



Risk Management

Identify potential problems before they occur, so that risk mitigation can be implemented across the life of the project to reduce impacts on achieving objectives

Meaning:1. Determine risk sources and categories2. Define risk parameters3. Establish risk mitigation strategy4. Identify and analyze risks5. Evaluate, categorize and prioritize risks6. Develop risk mitigation plans7. Implement risk mitigation plans

John Vu


Copyright © 2006 Boeing. All rights reserved. CC12091081.ppt

Risk or Issue?

???

???

Preventive action is cheaper than corrective actionPreventive action is cheaper than corrective action

Risk is something that has not yet occurredRequires preventive action

Issue is something already occurredRequires corrective action

John Vu



Risk Management Process

How are things going?

Assesshandlingoptions

Assesshandlingoptions

Plan and performmitigation

Plan and performmitigation

Analyzerisk

Analyzerisk

Identify risk

Identify risk

Communicateand

track risks

Communicateand

track risks

• How likely is it?• What would be the

consequences?

What can go wrong?

What can we do about it?

What is needed to reduce the risk?Risk Management is a continuous, disciplined process

for managing uncertaintyRisk Management is a continuous, disciplined process

for managing uncertainty

John Vu



Technical Solution

Design, develop, and implement solutions to satisfy customer’s requirements

Meaning1. Develop solutions and selection criteria2. Define product-component solutions3. Perform Make, buy, or reuse analysis4. Design the product component5. Design the interfaces6. Implement the design7. Develop final product that meet requirements

John Vu



Decision Analysis and Resolution

To analyze decisions using a formal evaluation process that identifies alternatives against established criteria

Meaning:1. Provide a formal process to evaluate alternatives2. Establish guidelines for decision making analysis3. Evaluate alternatives (requirements, functions,

configurations)4. Balance considerations (cost, performance, productivity,

testability,compatibility, supportability, etc)5. Select final solution

John Vu



Engineering = Balanced Trade-Off

Performancerequirements

ReliabilitySchedule

Maintainability

Cost

Technicalparameters Productivity

John Vu



Verification

Ensure that work products meet their specified requirements

Meaning1. Select work products for verification2. Establish verification procedures and criteria3. Conduct Peer reviews4. Analyze Peer review data5. Perform verification6. Analyze verification results and identify corrective action

John Vu



Validation

Demonstrate that a product fulfills its intended use when placedin its intended environment

Meaning1. Select product for validation2. Establish the validation environment3. Establish validation procedures and criteria4. Perform Validation5. Analyze validation results

John Vu



Context




BusinessValue


Enterprise


Organization

Project

Area Of Management

Project Management

Product Management

ProcessManagement


1

2

3

4

5

John Vu



Using data to maximize benefit of software development process

1) Project PerformanceSize variances trend (Plan vs. Actuals) Effort variances trend (Plan vs. Actuals)Defect density trendSchedule variances trend....................

2) Organization PerformanceNumber of projects following the standard processMilestone completionRequirements changesOrganization schedule variance....................

These measurements help the organization determine whereproblems exist to allow for correction as early as possible

1) Project PerformanceSize variances trend (Plan vs. Actuals) Effort variances trend (Plan vs. Actuals)Defect density trendSchedule variances trend....................

2) Organization PerformanceNumber of projects following the standard processMilestone completionRequirements changesOrganization schedule variance....................

These measurements help the organization determine whereproblems exist to allow for correction as early as possible

Quantitative Process Management

John Vu



Software Quality Management

1) Project performance dataDefect detection within limitsOther quality attributes (Reliability, Usability, etc.)Cost/effort within limitsSchedule performance within limits....................

2) Organization performance dataNumber of projects performing within limitsMilestone completion Requirements changes within limits.....................

These measurements help the organization determine whereProduct problems may exist to fine tune the standard process

1) Project performance dataDefect detection within limitsOther quality attributes (Reliability, Usability, etc.)Cost/effort within limitsSchedule performance within limits....................

2) Organization performance dataNumber of projects performing within limitsMilestone completion Requirements changes within limits.....................

These measurements help the organization determine whereProduct problems may exist to fine tune the standard process

Using data to manage products and services to achieve business goals

John Vu



Measurement Program

ProjectPerformance

Measurements

OrganizationPerformance

Measurements

Product-LineMeasurements

ProjectPerformance

Measures

ProjectPerformance

Metrics

DeploymentProgress

Measurements

ProcessPenetration

Metrics

Product/ProcessCorrelation

Metrics

• Cycle time• Effort• Size• Defect

• Normalizedcycle time

• Effort schedulevariance

• Cost of rework• Normalized

defect/phase

• Percent ofproject usingprocesses

DeploymentProgressMeasures

• # ofrecommendations

• Deploymentstatus

DeploymentProgressMetrics

• Percent ofrecommendationsdeployed

• Alignment to Product-line Goalsand Objectives

Core measures are in red

Statistical Process Control

John Vu



Correlation Between QPM and SQM

Analysis of process data and product data help the organization to:

• Determine the effectiveness of the standard process• Forecast trends• Maximize investment• Identify skills needed• Plan budget expenditures• Prepare for problem resolution• Measure customer / user satisfaction• Measure employee satisfaction• Identify conflict between groups• Quantify return on investment (ROI)

Analysis of process data and product data help the organization to:

• Determine the effectiveness of the standard process• Forecast trends• Maximize investment• Identify skills needed• Plan budget expenditures• Prepare for problem resolution• Measure customer / user satisfaction• Measure employee satisfaction• Identify conflict between groups• Quantify return on investment (ROI)

John Vu



Example: Improve Project Estimates

• The utilization of historical data will improve project performance by reducing the variation in project estimates

• Better estimates will improve project schedules

• Better schedules will improve project management

• Better management will improve project performance in terms of increase quality and reduce costs

• Better performance will improve customer satisfaction

• Satisfied customers will improve relationships

• Better relationships will improve the business

John Vu



Software Estimates(Actual vs Planned)

Sche

dule

Var

iatio

n

- 148percent

+ 22percent

+20percent +12

percent+4

percent

-125percent

+26percent

- 24percent

-18percent

- 7percent

Utilize Historical data for all estimates

Ove

r est

imat

esU

nder

est

imat

es

Improvement In Project Estimates

Source: CMU Data collected from 180 projects - 2004

CMMI Level 4 and 5

John Vu



Improvement Benefits


John Vu



Product Lines: Maximize Benefits

• Organization must leverage existing components to produce systems or services that meet the demand of the business

• A product line is a set of components that share a common, managed set of features, satisfying the specific needs of a particular market segment that are developed from a common set of core assets in a prescribed way

• By integrating existing components into new products, the enterprise can speed up time to market, reduce cost, increase productivity, quality, and gain significant advantage in the highly competitive business world



Reduce Cost = Increase Software ReusePe

rcen

t of S

oftw

are

reus

e

80

70

60

50

40

30

20

10

0

10 percent

25 percent

58 percent64 percent

Level 1 Level 2 Level 3 level 4 level 5Level 1 Level 2 Level 3 level 4 level 5

?


John Vu



Context




BusinessValue


Enterprise


Organization

Project

Area Of Management

Project Management

Product Management

ProcessManagement


1

2

3

4

5

John Vu



Behavior Of Higher Levels Organization

1. Organization can demonstrate business benefit from process improvement (improvement trends or results)

2. Organization must know which projects follow (or not) the standard processes

3. Organization’s processes must be verified independently that they are used and controlled at project level

4. Day-to-day decision making must be made based on facts and data (where appropriate)

5. Business goals are prioritized and intergroup conflicts are resolved according to a business strategy and plan

John Vu



Business Scorecard

Internal PerspectiveWhat must we be good at?How well are we meeting our business goals?How satisfied are our employees?

Innovation and Learning Capability PerspectiveWhat do we need to drive future performance?

Financial PerspectiveHow do we look to shareholders?

Customer PerspectiveHow do they perceive us?How satisfied are they?Have their needs been met?

Increase Customer Satisfaction

Speed and EfficiencyGrowth and AcquisitionsEmployee satisfaction

Skills and Knowledges

Reduce Cost, Reduce Cycle timeIncrease Quality

John Vu



Defect Prevention

• Project teams analyze defects and determine their causes

• Project teams evaluate organizational processes to prevent knowntypes of defects from recurring

• Organization uses defect data to forecast defect trends and set quantitative goals

• Organization monitors defect trends to improve product quality

John Vu



Formal Review Benefit Ratio

Req. Design Code Test Post-Release

Rew

ork

Effo

rt

Before Formal ReviewAfter Implemented Formal Review

8 percent

1 percent

12 percent

3 percent

19 percent

4 percent

Reduce 31 percentin rework

Formal Review increased effort by 4 percentdecreased rework by 31 percent

Cost: Benefit ratio is four percent : 31 percent or 1: 7.75

John Vu



Technology Change Management

• Organization identifies new technologies

• Organization follows a process to evaluate the new technologies

• Organization selects new technologies based on a business strategy and cost-benefits data

• Organization adopts new technologies according to a defined technology transfer process

• Organization measures and monitors adopted technologies and continuously improve its technology change management process

John Vu



Process Change Management

• Organization management continuously monitors all improvement activities and makes needed changes to optimize its return-on-investment

• Organization management continues to broaden the scope of improvement across the product line

• Organization management encourages people to be involved in the business and empowers them to make changes as needed

• Improvement occurs in both incremental improving of the process and by technological breakthrough by innovation

John Vu



And Finally ….

John Vu



Has Your Organization Experienced:

John Vu



Industry Data On Improvement

Process improvement can reduce costs an average of 35 percent per year

Some organizations have sustained cost improvements of 55 percent or more per year

Process improvement can improve quality an average of 40 percent per year

Some organization have sustained quality improvements of 70 percent or more per year

Source: James Herbsleb et al, “Benefits of CMM based Software process improvement CMU/SEI 94-TR-13, 1994

John Vu



Industry Data On Improvement

Organization Results

Boeing information systems Estimates within 20 percent , $6 million saved in 1 year

CSC 65 percent reduction in error rates

Hewlett Packard SESD ROI 9: 1

Hughes 2 million annual reduction in cost overruns

IBM Toronto 90 percent reduction in defects, 80 percent reduction in rework

Motorola GED 2.3X productivity increased ROI 6.7: 1

Raytheon ROI 7: 1

Siemens 90 percent reduction in defects

Telcordia Reduce defect 90 percent, ROI 6.5: 1

Organization ResultsOrganization Results

CSC 65 Percent reduction in error rates

Hewlett Packard SESD ROI 9: 1

Hughes $2 Million annual reduction in cost overruns

IBM Toronto 90 Percent reduction in defects, 80 percent reductionin rework

Motorola GED 2.3X Productivity increased ROI 6.7: 1

Raytheon ROI 7: 1

Siemens 90 Percent reduction in product defects

Telcordia Reduce defect 90 percent, ROI 6.5: 1

Source: CMU’s Benchmarking Study 2004

John Vu



CMMI Performance Improvement

CMMI Defect Productivity Rework Time Variance ROI

1 – 2 - 18% + 15% - 28% + 12% 160% - 16%

2 – 3 - 36% + 42% - 32% + 35% 85% + 80%

3 – 4 - 72% + 75% - 78% + 85% 32% + 176%

4 – 5 - 98% + 95% - 96% + 93% 18% + 420%

Source: CMU CMMI Study 2000 – 2005. Defect: # Error/KSLOC; Productivity: KLOC/Man-Month; Rework: Effort of fixing defect/Total project Time; Project Estimate variances: Planned vs. Actuals; Cycle Time: Time response to customer change request;ROI: Labor cost saving/Cost of improvement

John Vu



What’s Next

• You have seen the global market opportunity and how quality can be a key discriminator in getting business

• The Boeing Company would like to help Vietnam improve its information technology industry and be a key player in the global market

People Working Together

Leveraging Quality For Competitive Advantage · John Vu Engineering, Operations and Technology...

Documents

Transcript of Leveraging Quality For Competitive Advantage · John Vu Engineering, Operations and Technology...