Prioritizing Process Improvement And Ignoring The Rating

The leading pioneer in GPS technology

Copyright © 2008 NavCom Technology, Inc.

Prioritizing Process Improvement and Prioritizing Process Improvement and Ignoring the RatingIgnoring the Rating

Bruce Pedersen

12 June 2008

SEPG Europe 2008

Copyright © 2008 NavCom Technology, Inc.2

Case StudyCase Study

This is a case study of a small high technology software company that decided to strategically apply the benefits of CMMI process improvement to meet immediate business goals and ignore the rating. It will be shown that for this company, applying just 3 of the Level 2 process areas (PPQA, M&A, CM) struck the right balance in achieving cultural buy-in, allowing efficient utilization of available resources, all while reaching a significant return on investment.


TopicsTopics

Background What is the problem? Management Speaks Determining the Best Approach Implementation Strategy Did all of this Work Make a Difference? Next Steps


BackgroundBackground

NavCom Technologies Inc. is a John Deere company that supplies the enterprise with high precision Global Positioning System (GPS) navigation solutions for integration into the farm tractor auto-track system

Software releases are transmitted to another John Deere company, Intelligent Vehicle Systems (IVS), who performs system integration of the GPS component with other components on the farm tractor

IVS also operates the customer service center and manages complaints from the farmers about tractor operation

IVS attempts to separate out and communicate the GPS related farmer “bug reports” to NavCom as well as “bug reports” generated during system test

Increasingly high number of software defects are being reported back to IVS from the farmer

The software release quality issue has become visible to farming customers around the world and action has to be taken to correct the problem

NavCom is directed by John Deere corporate to “fix” the software release quality problem


Who is NavCom Technology?Who is NavCom Technology?

125 Employees located at the Torrance, California facility

35 software engineers supporting six concurrent GPS Software Development projects

Average GPS Product development cycle 6 months to 2 years

Process improvement has been on-going for less than two years

Established in 1992, NavCom Technology, Inc, a John Deere Company, is a leading provider of advanced GPS receivers

NavCom provides John Deere with a centimeter level GPS position solution that is integrated into the farming tractor Autotrack system (40,000 units in the field)

Autotrack combined with terrain compensation provides straight line operations on hillsides or rough terrain “hands free”

For more information visit: http://navcomtech.com/

http://navcomtech.com/


GPS at Work for John DeereGPS at Work for John Deere

GPS

StarFireCommunications

Satellite

DGPSCorrections

Galileo

WAAS

EGNOSGlonass

MSAS


Software Bundle Releases Software Bundle Releases

Software releases are tied to farming seasons

Two bundle releases (summer/winter) are made per year

Major activity by both NavCom and IVS in pre-bundle release testing (see right)

NavCom performs an accelerated software development life cycle

After bundle release, reported software bugs from both IVS and farmers fixed and saved for the next bundle release

Software Engineering

Test

Product Verification & Validation

Bug Reports from both IVS and Farmers

Analysis and Implement Bug

Fixes

NavCom IVS

IVSNavCom

Feature requests by the farmer are also saved for a bundle releases

Show stopper bugs found after a bundle release are fixed and bundle updates made


What’s the Problem?What’s the Problem?

Software Defect Visualized

NavCom- Reactive software development culture (i.e. firefighting)

Bug fixes and schedule agreed to over the phone without requirement traceability

Feature requests taken over the phone for next release- Lack of engineering test resources (personnel and tools)

available to apply the necessary pre-release software engineering test rigor

IVS - System Integrator- Lack of GPS expertise to accurately report bug details

necessary for bug environment recreation at NavCom - Lack of system test capabilities to isolate bugs caused by GPS

vs. other integrated tractor component sources General

- Poor communication path between NavCom and the IVS (very little paper trail)

Priority differences based on different business models Trust


Management SpeaksManagement Speaks

Management defines business goal related to software releases- “Significantly reduce the total number of software defects

released to the customer in delivered software” Choose a methodology that will minimize

- Cost to Implement- Business Disruption- Cultural Impact- Time for improvement visibility

Noticeable reduction in customer reported defects Make recommendations for what’s needed to apply additional

software engineering test rigor- Internal infrastructure changes- Additional test tools

There is no corporate or customer pressure for:- A particular methodology- Schedule target for a particular methodology level or rating


Determining the Best ApproachDetermining the Best Approach

What methodology?- What improvement scope? - What representation?

What implementation strategy?- What Organizational infrastructure changes are needed

to support the initiative?- What software engineering infrastructure changes?- What additional test tools are needed?


What Methodology?What Methodology?

The CMMI model selected for implementation- CMMI used by other John Deere divisions- World wide software industry acceptance of the CMMI

model and its best practices- CMMI model flexibility

Tailorable for unique business casesSelectable representation staged vs. continuous to

better fit company objectives- The “I” in CMMI allows phased integration of Systems

and Hardware into the process improvement scope- Management directives can be satisfied using CMMI


What Improvement Scope?What Improvement Scope?

Software only with future integration of systems and hardware engineering

Product development software projects in scope of CMMI process improvement- Research and development software projects out of

scope Test group in scope Hardware only projects out of scope Maintenance software projects to use portions of

the CMMI model that makes sense (e.g. peer reviews, test)


What Representation?What Representation?

Representation selected should allow process improvement focus to be directly tied to the company business goal- “Significantly reduce the total number of software

defects released to the customer in delivered software” Representation selected should also take into

consideration Management directives- Cost to Implement- Business Disruption- Cultural Impact- Time for improvement visibility (noticeable reduction in

customer reported defects)


Representation ComparisonRepresentation Comparison

Provides freedom to select the order of improvement that best meets the organization’s business objectives

Increased visibility into the capability achieved within each individual process area

Quick wins increase buy-in Comparison across the

divisions can only be made on a process area by process area bases (no maturity rating)

A relatively new approach which has limited data to demonstrate its ROI

Provides a predefined sequence of improvement areas based on proven path of successive levels each serving as a foundation for the next

Provides visibility at the maturity with limited visibility at the process area level

Permits easy comparison across divisions due to single maturity rating

Builds on a relatively long history of use that builds on case studies and data that demonstrates proven ROI

Continuous Representation Staged Representation


Representation SelectionRepresentation Selection

The continuous representation chosen- Satisfies Management directives- No customer pressure for CMMI capability

level rating- Organic process improvement growth works

for NavCom

Mapping of process areas to business goals imperative- Process Areas chosen that focuses on the immediate business

goals- 5 year plan created to phase in future process areas to support

projected business goals- Plan updated every year to reflect changing business strategy

CMMIBusiness

Goals


Implementation StrategyImplementation Strategy

Process Area Selection Establish a Capability Level Profile Define the current project processes Establish an organizational infrastructure that will

support continuous process improvement- Involve practitioners early on to foster process ownership

Develop plans and processes that will enable selected process area support of the current project processes

Pilot processes and obtain practitioner feedback for constant improvement

Measure improvement progress using metrics


Process Area SelectionProcess Area Selection

Select Process Areas based on:- Direct contribution to business goal- Provide foundation for future process area integration- Consider dependencies between process area’s and generic practices - Ease of implementation - complexity- Amount of support processes in place already at NavCom

Choose the desired capability level for each of the selected Process Areas (1-5)- The continuous model allows additional capability levels to be achieved

on focused process areas

Construct a capability level profile- Demonstrate process improvement progress to management- Guide process improvement activities

Process improvement can be conducted on individual PA’s at different rates Resources can be focused on lagging PA activities


Process Area Selection (cont)Process Area Selection (cont)

The process areas that meet the selection criteria are a subset of the categorized “Basic Support” process areas:- Process and Product Quality Assurance (PPQA)- Configuration Management (CM)- Measurement & Analysis (MA)

Peer Reviews and Test were also selected from the Verification process area (Specific Goals 2 & 3) to directly support “reducing the number of defects in software releases”


Process Area MappingProcess Area Mapping

Minimize MA PPQA CM

Cost to Implement

Med Med Low

Business Disruption

Med Low Low

Cultural Impact Med Low Low

Time for ROI Visibility

Low Med Med

PP PMC SAM MA PPQA CM

Management Products

Code Product

Practitioners

Project Managers

Level 2 PAs

Management Requirements

VERPeer Reviews and partial implementation of testing

REQM


Process Area DescriptionProcess Area Description

PPQA – provide staff and management with objective insight into processes and associated work products via independent auditing

CM – establish and maintain the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits

MA –develop and sustain a measurement capability that is used to support management information needs

Peer Reviews and Test – ensure that selected work products meet their specified requirements (VER process area)

Note: Test refers to system test prior to Product Verification and Validation (PV&V) testing


Establish a Capability Level Profile Establish a Capability Level Profile

Determine what capability level to target

for each selected process area

Establish a method for overlaying the

current achievement profile to the target- Determination of current % complete report to

Management- Used for appraisal readiness check

Use target staging to establish a path of process improvement to be followed by the organization

Capability Level Profile

0

1

2

3

4

5

PPQA CM MA

Cap

abil

ity

Lev

el


Capability Level ProfileCapability Level Profile

0

1

2

3

4

5

Cap

abili

ty L

evel

PPQA CM MA

Target Profile

AchievementLevel

AchievementLevel

AchievementLevel

Target Profile

Target Profile

GG3 Institutionalize a Defined Process


Define the Current Project ProcessDefine the Current Project Process

The Current Project Process defines the way a project develops software today

Current Project Process is unique to each project (Level II)

Provides awareness to project practitioners of gapsin current unmanaged project software process- Written processes not consistently followed

(e.g. skipped in schedule crunch)- Unwritten processes verbally communicated by peers- Non-Existent processes made-up by individuals to do their job

Starting point for application of selected CMMI Process Area support

Opens up communication path between the process group and project practitioners- Flow chart used to communicate current process at the 50,000 foot level- Allows further drill down in focused process areas using non-CMMI

jargon or concepts


PA Support of Current Project ProcessPA Support of Current Project Process

Configuration items and change requests

Current Project ProcessInformation Needs

Measurements and analysis

Baselines and audit reports

Quality and noncompliance issues

Standards and procedures

Defect Reports

Peer Reviews & Testing

PR + Test PPQA

MACM

L2

L2

L2

L3

Fosters practitioner process ownership in helping the process group define needed processes

Identifies areas were application of process improvement could be focused for near term ROI

Allows groups supporting the project to have a voice

Can be used for training

new project personnel


Establishing an Organizational Process Establishing an Organizational Process Improvement InfrastructureImprovement Infrastructure

SPI Sponsor selected (Director of Systems) Software Process Improvement group formed

- SPI Manager- PPQA Lead Auditor- Process Engineer - Administrator (training records, PAL administrator, etc.)

SEPG formed with representation from each project and functional group (7 members)- SPI Manager Chair- Program Management- Software Development- Test

Technical Working Groups managed by SEPG to accomplish work 3 member reserve audit team established to support the PPQA

Lead- Made up of project practitioners- Reserve auditor training performed by PPQA Lead- Auditor performs cross project audits only (not own project)


Focused Efforts at Cultural Buy-InFocused Efforts at Cultural Buy-In

Early participation by practitioners in defining processes foster process ownership

Cross functional representation in the SEPG

- Membership rotation scheme introduced to expose a greater % of the practitioners

to process improvement activities Tool deployed to facilitate practitioners process

improvement suggestions Monthly published process improvement

newsletter that features SPI topics and progress SCAMPI B and C appraisals conducted every 6 months

- Focus process improvement efforts towards a near term goal- Align culture to the appraisal process

Periodic Workshops held with individual project to discuss new process deployment and lessons learned

Training program initiated to teach new processes to project personnel


Process Creation and DeploymentProcess Creation and Deployment

The SPI Group creates plans and processes for direct support of projects defined process- Project practitioners participate in defining the current project

process- Practitioners participate in content review of project plan/process

documents The SPI Group creates workshops to familiarize project

practitioners on new processes New processes are piloted first and practitioner feedback

used for process improvement updates Metrics are defined to measure process effectiveness Lessons learned are managed and used for process

updates


What Else Do We NeedWhat Else Do We Need

Test – additional rigor focused on finding defects before release to the customer- Formalize testing using

historical test scenarios- Dedicated group formed to

perform software engineering test function

Tools –leverages process improvement advantage- Artifact storage- Automated CM- Bug tracking- Central repository for

process improvement documents (PAL)


Establishing a Test OrganizationEstablishing a Test Organization

A Software Engineering Test Group (SETG) was formed with a charter to formalize the software pre-release engineering test process of the organization- SETG Lead- Three Test Engineers

In parallel, two tasks were conducted to support the SETG- A PC based GUI tool was developed to automate regression

testing against baseline and new releases GUI allows test engineer to select and control the battery of tests to

be run Automatic report generation and plotting used for analyzing test

results- Regression library created to furnish regression test scenarios

that have been created to solve problems during the past seven years

Scenarios generated to recreate conditions that would reproduce reported bugs


What Test Tools are Needed?What Test Tools are Needed?

CM tool (Microsoft Team Foundation Server) selected to help software development teams communicate and collaborate more effectively- Streamlines the implementation of the CM process area across

the organization- Enforces CM on both source code and documents via predefined

rules- Provides a central repository for process documents, templates

(Process Assets Library) and artifacts supporting the appraisal- Integrated CMMI methodology component for “head start” in

process definition and implementation - Integrated work item tracking and reporting (e.g. bug tracking)- Extensive suite of “out of the box” reports for tracking progress

and metrics mining (e.g. defect phase containment, density)


Did All of This Work Make a Difference?Did All of This Work Make a Difference?


Total Software Reported Defects by Bundle

0

10

20

30

40

50

60

70

80

90

100

SSB05 WSB06 SSB06 WSB07 SSB07 WSB08

Def

ects

CustomerReportedTotal Defects

10.3%

17.7%

9.3% 8.1%

7.1%

1.6%

NavComMajor GPS Technology

Release

SSB = Summer SW Bundle, WSB = Winter SW Bundle

NavCom’s SETG On-Line

IVS Test Group

Revamped


Customer Defect Metric ExplanationCustomer Defect Metric Explanation

The total reported defects “green column” are what the system integrator (IVS) reported back to NavCom - Customer reported defects are included in the green column

More feature requests and bug fixes are included in the winter bundles thus generating more defects

With each bundle release comes increased complexity from newly implemented GPS features and capabilities

A major technology upgrade to increase both GPS accuracy and reliability was made in the Winter Software Bundle 07


Bottom LineBottom Line

The drop in customer reported bugs between the Winter and Summer bundle 06 is due to the activation of the SETG and early adoption of peer reviews- SETG able to detect complex navigation related bugs via simulation

The dramatic increase of defect detection by IVS in SSB06 was due to the addition of testing resources and training by NavCom on effective GPS testing techniques

Although NavCom’s defect metric database is still immature, indirect evidence indicates that NavCom’s cultural change has started and defect detection and correction is getting better

NavCom’s cultural change is rippling over to IVS causing them to also get better at detecting defects and reporting back accurate test results

The farmer is benefiting from all of this by reducing the number of defects seen in the field from 17.7% to 1.6% in two years

Shown is a downward trend in total software defects released to the customer


Next StepsNext Steps

Align the process improvement 5 year plan to the newly created 5 year strategic business plan

for NavCom

Based on lessons learned over the past two years,select the next group of process areas for future implementation- Requirements Management- Technical Solution- Project Planning

Continue refining the metrics database to give better visibility to defect detection effectiveness

Continue laying the infrastructure foundation to allow seamless

transition to level 3 process areas

Prioritizing Process Improvement And Ignoring The Rating

Documents

Transcript of Prioritizing Process Improvement And Ignoring The Rating