Assuring Our

Assessments, Leaders

and Team Member’s

Quality and

Qualification Process

and Quality

Kobi Vider

K.V.P Consulting

Kobi.vider@hotmail.com

kobivp@aol.com

+972522946676

Agenda and Topics

• Opening

• The Internal and External Assessors Community

• The Challenge Statement

• The ‘Project’ Goals / Success Factors

• Tool DEMO And Short Walkthrough

• Results and Benefits

• Conclusions and Recommendations

• Future Plans

• Discussion Points

Opening

• As for today most of major industries mainly defense and

aerospace, as well as armed forces

• Which runs and mange large and complex programs

• Need to comply with more than just one quality standards

in many disciplines (e.g. HW, optics, software)

• They use large groups of internal and external assessors

that perform implementation checks, progress checks,

readiness reviews and formal appraisals.

• These communities are typically composed from groups of

very experienced and professional individuals

• These individuals have the best knowledge in their

professional domain but not necessarily on how to conduct

an efficient and effective appraisal which provide

meaningful results.

The Internal and External

Assessors Community

• ~450 auditors (internal military staff and externals reserves)

• ~80 externals (contractors, that perform audits for and on the XYZ

ORG)

• Few tens that are the contractors internal audits and assessments teams

(quality wise)

• ~200 quality engineers (full time)

• ~150 quality practitioners / trustees (part time)

• ~X00 project / program and acquisitions officers

The Challenge Statement

• The combination of the effort and expected

resources increase the risks on qualification

of auditors, domain knowledge, and

calibration of results and findings

effectiveness.

• When I have initial try to map the risks and

challenges, I have discovered that the main

ones are:

The Challenge Statement

• Large diversity in tailoring the methods for ‘of the

same’ appraisal (such as scope, coverage, rating

understanding)

• Inconsistency in ratings

• Inconsistency of interpretations

• Reports calibration for the same unit / group and

on organizational level is very difficult if not

impassible

• Performing quality assurance of the performed

process is very challenging

The ‘Project’ Goals / Success

Factors

• Role based profile and criteria

• Role development path

• Calibration mechanism and criteria

• Evaluation mechanism and criteria

• Leveling the different quality engineers and

‘auditors’ in the XYZ ORG by specific

performances and graduations

The ‘Project’ Goals / Success

Factors

• Flowing specific trainings (on different levels) as

personal development and qualification criteria

• Listing specific performances as indicators for

leveling justifications

• Structuring the different audits and reporting

guidelines in a single mandatory to follow (based

on MDD and SLA-BOK) process, to increase

calibration and quality of professional conduct and

results (for process and meaningfully

improvements)

3/14/2013

PP

SSD

SCON

CAR

PMC

M&A OT

REQM

SST

DAR

SAM

IRP

OPD

QPM

SD

PPQA

OID

RSKM

OPF

CAM

STSM CM

OPP

IPM

ISO 9000:2008 Correlation Snapshot

Management

Responsibility

Measurement

Analysis &

Improvement

Product

Realization

Resource

Management

Quality

Management

System

20 are covered

PP

SSD

SCON

CAR

PMC

M&A OT

REQM

SST

DAR

SAM

IRP

OPD

QPM

SD

PPQA

OID

RSKM

OPF

CAM

STSM CM

OPP

IPM

PMBOK Correlation Snapshot

Integration

Scope

Time

Cost

Quality

15 PAs are covered

Procurement Risk Communications Human

Resource

PP

SSD

SCON

CAR

PMC

M&A OT

REQM

SST

DAR

SAM

IRP

OPD

QPM

SD

PPQA

OID

RSKM

OPF

CAM

STSM CM

OPP

IPM

ITIL – CMMI Correlation Snapshot Service Delivery

Financial Management

for

IT Services

Capacity

Management

IT Service

Continuity Management

Availability

Management

Service Level

Management

PP

SSD

SCON

CAR

PMC

M&A OT

REQM

SST

DAR

SAM

IRP

OPD

QPM

SD

PPQA

OID

RSKM

OPF

CAM

STSM CM

OPP

IPM

ITIL – CMMI Correlation Snapshot Service Support

The Service Desk

Incident Management

Problem Management

Configuration

Management

Change Management

Release Management

PP

SSD

SCON

CAR

PMC

M&A OT

REQM

SST

DAR

SAM

IRP

OPD

QPM

SD

PPQA

OID

RSKM

OPF

CAM

STSM CM

OPP

IPM

ISO 20000 – CMMI

Correlation Snapshot Control processes

new or changed

services

Service delivery

process

Requirements for a

management system

Relationship processes

Resolution processes Release process Planning and

implementing

service management

Supporting Tools for

High Maturity

HLM ATM Training agenda

High Level Maturity Appraisal

Team Member References

High Level Maturity

Appraisal Team

Member

References

Statistical Methods

Chi-Square ..................................................................................................................................................... 4

Chi-Square Test for Goodness of Fit ......................................................................................................... 4

Chi-Square Test for Independence ........................................................................................................... 5

Assumptions and Accuracy of the Chi-Square Test .................................................................................. 7

Common Cause of Variation ......................................................................................................................... 8

Continuous Probability Distribution ........................................................................................................... 11

Control Chart ............................................................................................................................................... 14

Control Charts for Variables .................................................................................................................... 15

Design of Experiments (DOE) ...................................................................................................................... 19

Description .............................................................................................................................................. 19

When to Use DOE.................................................................................................................................... 19

DOE Procedure ........................................................................................................................................ 20

Discrete Probability Distributions ............................................................................................................... 22

One-sided Test ............................................................................................................................................ 22

Example ................................................................................................................................................... 23

Example ................................................................................................................................................... 24

One Sample t-test ....................................................................................................................................... 25

Two Sample t-test ....................................................................................................................................... 25

P-Value ........................................................................................................................................................ 26

Discrete Data ............................................................................................................................................... 26

Exponential Distribution ............................................................................................................................. 26

Probability density function ................................................................................................................ 26

Hypothesis Testing ...................................................................................................................................... 28

Null Hypothesis ....................................................................................................................................... 29

Alternative Hypothesis ............................................................................................................................ 29

Simple Hypothesis ................................................................................................................................... 29

Composite Hypothesis ............................................................................................................................ 30

High Level Maturity

Appraisal Team

Member

References

Acknowledgments ............................................................................................................................ 1

Table of Content .............................................................................................................................. 1

Abstract ............................................................................................................................................ 1

1.1. Purpose ............................................................................................................................ 2

1.2. Objectives ......................................................................................................................... 2

1.3. CMMI Terminology .......................................................................................................... 2

1.4. Industry Terminology ....................................................................................................... 2

1.5. Terms and definitions ...................................................................................................... 2

1.6. Process Performance Models References in the CMMI Suite ......................................... 2

1.7. Process Performance Models relationships with Measurements in the CMMI Suite ..... 2

1.8. Relationships from PSP and TSP to support CMMI Measurements and high maturity ... 2

1.1 high level listen for in process performance models ....................................................... 4

1.1.1 Overview .................................................................................................................. 4

1.1.2 In Small Setting Organizations ................................................................................. 4

1.1.3 In Agile like development and teams ....................................................................... 4

1.1.4 In different type Organizations ................................................................................ 4

1.1.5 At Level 2 & 3 Organizations .................................................................................... 4

1.1.6 At Level 4 & 5 Organizations .................................................................................... 4

1.2 Business Objectives; quantitative objectives, Needs and Between ................................ 4

1.2.1 Business Objectives .................................................................................................. 4

1.2.2 Quantitative objectives ............................................................................................ 4

1.3 Analyzing the organization lifecycle and process ............................................................ 4

1.3.1 Key players in the process ........................................................................................ 4

1.3.2 Key users of performance models ........................................................................... 4

1.3.3 Key elements (building blocks) of the process ......................................................... 4

1.3.4 Domain Specific Perspective .................................................................................... 4

1.3.5 Process perspective .................................................................................................. 4

1.3.6 Organizational Role Perspective .............................................................................. 4

3.1 Project Management Process Areas ................................................................................ 5

3.1.1 Project Planning (PP) 5

PPM-BOK PROCESS PERFORMANCE MODELS ........................................................................................................... 1

APPLICATION AND APPROPRIATE USAGE (METHODOLOGY) .................................................................... 1

ACKNOWLEDGMENTS ....................................................................................................................................... 1

TABLE OF CONTENT .......................................................................................................................................... 1

ABSTRACT ...................................................................................................................................................... 6

INTRODUCTION ........................................................................................................................................ 7

1.1. Purpose ...................................................................................................................................... 7

1.2. Objectives .................................................................................................................................. 7

1.3. CMMI Terminology .................................................................................................................... 7

1.4. Industry Terminology ................................................................................................................. 7

1.5. Process Performance Models References in the CMMI Suite .................................................... 7

1.6. Process Performance Models relationships with Measurements in the CMMI Suite ................ 7

1.7. Relationships from PSP and TSP to support CMMI Measurements and high maturity ............. 7

INTRODUCTION TO PROCESS PERFORMANCE MODELS ............................................................................ 8

1.1 Why to Measure the process performance ............................................................................... 8

1.2 What is all about? ..................................................................................................................... 8

1.3 Benefits from using process performance models ..................................................................... 9 1.3.1 Overview .............................................................................................................................................. 9 1.3.2 Application in Small Setting Organizations ........................................................................................... 9 1.3.3 Application in Agile like development and teams ................................................................................ 9 1.3.4 Application in different type Organizations ......................................................................................... 9

1.3.4.1 Software ..................................................................................................................................... 9 1.3.4.2 System ........................................................................................................................................ 9 1.3.4.3 Services ....................................................................................................................................... 9 1.3.4.4 Acquisition .................................................................................................................................. 9 1.3.4.5 Maintenance ............................................................................................................................... 9 1.3.4.6 Support ....................................................................................................................................... 9

1.3.5 Application and support low maturity organization ............................................................................. 9 1.3.6 At Level 2 & 3 Organizations ................................................................................................................ 9 1.3.7 At Level 4 & 5 Organizations ................................................................................................................ 9

1.4 The Measurements and Analysis Team ................................................................................... 10 1.4.1 As Organizational Level Functional Team ........................................................................................... 10

1.4.1.1 Position in the Organizational Structure ................................................................................... 10 1.4.1.2 Team Suggested Structure ........................................................................................................ 10 1.4.1.3 Team Roles, Responsibilities and authority .............................................................................. 10

1.4.1.3.1 Roles ..................................................................................................................................... 10 1.4.1.3.2 Responsibilities ..................................................................................................................... 10 1.4.1.3.3 Authority .............................................................................................................................. 10

1.4.1.4 Principles of Professional Conduct............................................................................................ 10 1.4.1.5 Basic Activities .......................................................................................................................... 10 1.4.1.6 Additional Activities .................................................................................................................. 10 1.4.1.7 Knowledge and Skills ................................................................................................................ 10 1.4.1.8 Personal Development Path ..................................................................................................... 10

1.4.2 As Group of Individuals ...................................................................................................................... 11 1.4.2.1 Position in the Business Organization Structure ....................................................................... 11 1.4.2.2 Position in the Quality Organization Structure ......................................................................... 11

High Level Maturity

Lead Appraisal

Guideline

Acknowledgments ............................................................................................................................ 1

Table of Content .............................................................................................................................. 1

Abstract ............................................................................................................................................ 1

1.1. Purpose ............................................................................................................................ 2

1.2. Objectives ......................................................................................................................... 2

1.3. CMMI Terminology .......................................................................................................... 2

1.4. Industry Terminology ....................................................................................................... 2

1.5. Terms and definitions ...................................................................................................... 2

1.6. Process Performance Models References in the CMMI Suite ......................................... 2

1.7. Process Performance Models relationships with Measurements in the CMMI Suite ..... 2

1.8. Relationships from PSP and TSP to support CMMI Measurements and high maturity ... 2

1.1 High level SCAMPI scripts for process performance models ........................................... 4

1.1.1 Overview .................................................................................................................. 4

1.1.2 In Small Setting Organizations ................................................................................. 4

1.1.3 In Agile like development and teams ....................................................................... 4

1.1.4 In different type Organizations ................................................................................ 4

1.1.5 At Level 2 & 3 Organizations .................................................................................... 4

1.1.6 At Level 4 & 5 Organizations .................................................................................... 4

1.2 SCAMPI scripts for Business Objectives; quantitative objectives, Needs and Between .. 4

1.2.1 Business Objectives .................................................................................................. 4

1.2.2 Quantitative objectives ............................................................................................ 4

1.3 SCAMPI scripts for analyzing the organization lifecycle and process .............................. 4

1.3.1 Key players in the process ........................................................................................ 4

1.3.2 Key users of performance models ........................................................................... 4

1.3.3 Key elements (building blocks) of the process ......................................................... 4

1.3.4 Domain Specific Perspective .................................................................................... 4

1.3.5 Process perspective .................................................................................................. 4

1.3.6 Organizational Role Perspective .............................................................................. 4

3.1 Project Management Process Areas ................................................................................ 6

3.1.1 Project Planning (PP) 6

Results and Benefits Methodology conclusions.

• Model formulation is complex, but evaluation is straightforward.

• Standard tools, easy to integrate on domain cases.

• Method is better at capturing SMEs insights and project dynamics.

• Opportunity to further include additional domains

SME related conclusions (preliminary).

• Small units have better acceptance for audits

• Engineering related organizations have a good case for higher maturity levels

• Critical dependency on allowed investment horizon.

• Appraisal costs impacts on overall investment not dominant.

• Benefits for organizations operating at volatile (uncertain) segments.

Conclusions and Recommendations

• Method can be implemented in order to develop a decision

support system in the management of new product

development domain

• This model addresses various aspects of new product

development over multiple stages, and easily can be

adjusted to other domains

• The model can deal with quantitative and qualitative input

and missing data

• Method can be implemented to address our managerial

problems and to support our managers with strong ‘gate

keepers’

• The implementation of a graphical user interface hiding the

complexities of the method and model is significant

Future Plans

• Continue refinement of model.

• Parameter ranges.

• Roadmap involving multiple SCAMPI activities.

• Complete Validation and Verification.

• Integrate work and receive feed from other

initiatives.

• Continue sharing with other domains

Discussion Points

• Performance data

• Cost of poor planning

• Quantifying the operational impact

• Effecting and effected stakeholders

mapping

• Quantifying the impact of support planning

on the development teams

• Appling this model on other domains

Questions ?

Contact

Kobi Vider

K.V.P Consulting

Kobi.Vider@hotmail.com

KobiVP@aol.com

Phone: +972522946676