CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright ©...
-
Upload
morgan-wilkinson -
Category
Documents
-
view
224 -
download
3
Transcript of CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright ©...
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and AssessmentCopyright © 1995-2004, Dennis J. Frailey, All Rights Reserved
Slide 1
CSE7315M37
January 10, 2004
SMU CSE 7315 / NTU SE 584-NPlanning and Managing a
Software Project
Module 37Process Appraisal and
Assessment
Slide # 2 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Objective of This Module
• To examine methods of appraising and assessing process maturity
Slide # 3 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Process Appraisal“If you don’t know where you are, a map
won’t help” (1)
An appraisal is a way to determine where you are
(1) Humphrey (textbook)
Slide # 4 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Why Do Appraisals?
Most organizations suffer because they cannot see their problems, not because they cannot solve them.
Most organizations are focused on defining solutions, not on defining problems.
Most organizations tend to solve symptoms, rather than the fundamental, underlying problems.
Why not just solve
the problems?
Slide # 5 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Responding to Symptoms
Problem
Symptom
Symptom
Symptom
Response LongerTerm Cure
Response
TemporaryCure
Slide # 6 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Types of “Appraisals”• Reviews
• Audits
• Capability Evaluations
• Self Assessments
• Other Appraisals
Slide # 7 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Possible Exam Question
Compare and contrast the different forms of appraisal with respect to:–Cost–Length–Who Performs–Objectives–Advantages–Drawbacks–etc.
Slide # 8 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
ReviewsPurpose:
To learn the status of the project
Performed by:
Managers
Method: Practitioners report on
the status and plans of their projects, following specified formats and reporting on specific metrics
Slide # 9 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Reviews (continued)Typical Duration: A few hours to several daysAchieves: Uncovers problems (or, at least, symptoms)Drawbacks: Does not identify solutions May motivate hiding of problemsAdvantages: Relatively inexpensive Tends to get everyone back on the same
track
Slide # 10 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
AuditsPurpose: To study a project in detail and find problems. Also to keep things on trackPerformed by: Independent technical experts, often outsidersMethod: Experts question practitioners and examine artifacts
of their process to determine what is happeningTypical Duration: Several days to several weeks
Slide # 11 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Audits (continued)
Achieves:
Uncovers real problems more often
Informs staff that management cares about the results
Advantages:
Tends to uncover real problems
Tends to confirm or disprove suspected problems
Slide # 12 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Audits (continued)
Drawbacks: More expensive than a review Does not identify solutions May motivate hiding of problems Can generate hostility and lack of cooperation Can de-motivate Outsiders often do not understand and even if they do, they are not believed
You don’ttrust us!
Slide # 13 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Capability EvaluationsPurpose: To determine if an organization is capable of doing a
job wellPerformed by: Outside experts in all affected areasMethod: -- Many methods have been used for years -- For example, each of the armed services has ways
of doing contractor capability evaluations before selecting contractors
-- SEI has defined the SCE method, based on the CMM
Slide # 14 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Definition
Typical duration: several days
A software capability evaluation is an appraisal by a trained team of
professionals to identify contractors who are qualified to perform the software work or to monitor the
state of the software process used on an existing software effort.
Slide # 15 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Capability Evaluations (continued)
• The SEI SCE (Software Capability Evaluation) is like an organized audit where the CMM is the model against which the program is evaluated
CMM 1.1
Slide # 16 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Capability Evaluations What They Achieve
• They are based on well established best practices for software development
• So the results can be seen in comparison with accepted best practices
• They uncover problems (or, at least, symptoms)
• They note lack of problems• They note capabilities
Slide # 17 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Capability Evaluations Drawbacks
• They do not identify solutions• They may motivate hiding of
problems• They are relatively expensive• Inexperienced evaluators can
misinterpret the findings, resulting in incorrect or unfair results– Example: focusing on practices rather
than goals and commitments
Slide # 18 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Capability Evaluations Drawbacks (continued)
• Inexperienced or immature managers may not understand the conclusions, interpret them correctly, or act appropriately– Example: instituting metrics when the
problem is communication
Slide # 19 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
“Self” AssessmentsPurpose: To help an organization learn where it is, how to
improve, and what to do nextPerformed by: A team of inside and/or outside technical and management expertsMethod: There are many methods of self assessment SEI has defined the two methods of software capability
assessment based on the CMM There have been several versions of the SEI method,
which can cause confusion
Slide # 20 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Definition (from Paulk)
“A software process assessment is an appraisal by a trained team of
software professionals to determine the state of an organization's current software process, to determine the
high-priority software process-related issues facing an organization, and to obtain the organizational support for
software process improvement.”
Slide # 21 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
“Self” Assessments Achieve
• They uncover problems • They identify recommended
solutions• They an be used to motivate– People “buy in” when they participate
in the assessment
Slide # 22 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
“Self” Assessment Drawbacks
• Can mislead if too many internal participants
• Can be misused as a sort of audit or proof of capability
• Can be very expensive• Can do more harm than good if
practitioners are motivated to change and management does not support the changes
Slide # 23 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Assessment vs. Evaluation
CMM 1.0
Evaluations
• Find Problems
• Evaluate Capability
• For External Use
Assessments
• Find Problems
• Find Solutions
• Motivate Improvement
• For Internal Use
CMM 1.1
CMM 1.0
Humphrey5-LevelModel
Slide # 24 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Assessment Methods
• SPA– Software Process Assessment
• CBA-IPI– CMM Based Appraisal for Internal
Process Improvement
With both methods, SEI receives results and records confidentially. Results are published in aggregate form, quarterly.
Slide # 25 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SEI Publishes a Maturity Profile
http://www.sei.cmu.edu/sema/profile.html
64.8
26.7
8.5
0
10
20
30
40
50
60
70
Percent of Reports
Commercial
DefenseContractor
Military
Slide # 26 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Maturity Profile as ofMarch, 2001
32.2
39.3
19.4
5.4 3.7
0
5
10
15
20
25
30
35
40
1 2 3 4 5
Percent of Total
Slide # 27 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SPA Method• Version 1 was based on a questionnaire or
survey, derived from Watts Humphrey’s model• Typical duration: 1 week• Several intermediate versions have been
based on intermediate versions of the CMM• The most recent and current version is based
on CMM 1.1• In each case there have also been several
degrees of “certification” of assessors, leading to many misunderstandings and unverifiable claims
Slide # 28 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Training for SEI Assessments • Initial Assessment Method (1988-93)– Assessors were trained by SEI– Later, assessors were given “certification”
by SEI– An assessment was deemed “more
reliable” if one or more trained and/or certified assessors were involved
– See Humphrey, chapter 3, for more details• Later Assessment Methods– Individuals and/or organizations certified as
assessment team leaders and/or as trainers
Slide # 29 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Confidentiality
• SEI does not release results of any specific assessment or confirm or deny the validity of any assessment
• They only release composite results to show overall industry characteristics and trends
Slide # 30 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
CBA-IPI Method• Typical duration: 1-2 weeks• More comprehensive than SPA• Places increased emphasis on CMM, when
compared with the SPA method• Intended to foster improvement for an
organization, not just focused on software• May be conducted by internal or external
personnel• Must be “led” by an “SEI Certified Lead Appraiser”
Slide # 31 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
CMM-based Appraisals
Maturity LevelsMaturity Levels
Key Process AreasKey Process Areas
Common FeaturesCommon Features
Key PracticesKey Practices
Goals
Implementation
Infrastructureor Activities
ProcessCapability
Address
Describe
Achieve
Contain
Organized by
Contain
Indicate
Questions
Slide # 32 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SPA and CBA-IPI IssuesThese appraisals are intended to help an
organization evaluate itself– But too often, organizations will
publicize their appraisal results in the form of an achieved SEI maturity level
– SEI states that “when the focus is primarily on achieving a maturity level, it can distort the purpose of process improvement by diverting attention from genuine process improvement activities”
Slide # 33 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
SPA and CBA-IPI Issues(continued)
• Contracting agencies, especially in the government, may encourage incorrect use of assessments by insisting on specific SEI maturity levels for contract eligibility
• SEI specifically states that “finding a maturity level is optional” for SPA and CBA-IPI and “not intended or appropriate” for an SCE
Slide # 34 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
More SPA and CBA-IPI IssuesAssessment expense has generated many
lesser approaches– Typical appraisal cost: $80,000 (large org)• Assessment team (external): $45,000• Preparation $15,000• Time for participants $20,000
– Alternatives are OK for individual use, but the less extensive the less reliable the results
– Lack of external participation, while saving money, can also result in too-harsh or too-lenient appraisals
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and AssessmentCopyright © 1995-2004, Dennis J. Frailey, All Rights Reserved
Slide 35
CSE7315M37
January 10, 2004
Low Cost Assessment Approaches
Slide # 36 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
“Temperature Taker”
• Select one or two random questions pertaining to key practices of a given KPA (a total of 6-12 questions for level 2, for example)
• Ask these questions of each project and discuss answers with participants
• Tally results at organization level to find strengths and weaknesses
Slide # 37 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Characteristics of“Temperature Taker”
• Costs about 2 hours per project to collect data plus a few hours to tally
• Spots significant issues• And communicates the
organization’s concern about maturity
• But may miss significant issues
Slide # 38 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
“Continuous Record”
• Organization selects key practices to focus on within each KPA
• Each project maintains a list of key practices and a chart of whether they do each practice
• Display a graph in the work area and report monthly to the organization
• Tally at the organization level
Slide # 39 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Characteristics of“Continuous Record”
• Costs less than 1 hour per project per reporting period
• Spots issues and trends• And communicates the
organization’s concern about maturity
• But relies on project self-assessment, so may still miss significant issues
Slide # 40 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
“Internal Assessment”
• Organization defines an assessment team of internal individuals
• Periodically, this team does an assessment
• Results are reported as in a full assessment
Slide # 41 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Characteristics of“Internal Assessment”
• Costs less than a full assessment– No external participants– Assessors have a smaller “learning curve”
• Spots issues and trends• And communicates the organization’s
concern about maturity• But relies on internal participants, so
may still miss significant issues or blow issues out of proportion
Slide # 42 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
Module Summary• Appraisals of various kinds use
the CMM or other models to evaluate an organization’s maturity
• The more effective appraisal methods identify solutions as well as problems and foster buy-in and acceptance at all levels in the organization
Slide # 43 January 10, 2004
CSE 7315 - SW Project Management / Module 37 - Process Appraisal and
AssessmentCopyright © 1995-2004, Dennis J. Frailey,
All Rights Reserved
CSE7315M37
END OFMODULE 37