NASA Langley Research Center Software Process Improvement Initiative (SPII) Findings Presentation...

NASA Langley Research Center

Software Process Improvement Initiative

(SPII)

Findings Presentation

October 27, 1997


CornerStone Findings Presentation

CornerStone Overview Findings Next Steps


CornerStone Overview


CornerStone Goals The CornerStone Phase is the foundation of LaRC’s

overall Software Process Improvement Initiative The goals of the CornerStone Phase are:

Develop a plan to improve LaRC’s software development practices Identify current state of software development at

LaRC Identify current best practices used in software

development at LaRC Develop a High Performance Model for LaRC’s

software development activities (incorporates the appropriate elements of the Capability Maturity Model, ISO 9000-3, Strategic and Quality Framework, and Baldrige Award Criteria)

Obtain management’s support, complete with resources, to implement LaRC’s Software Process Improvement Plan


Software Process Improvement Initiative (SPII) Goals

Improve the work environment for LaRC’s software community, leading to higher morale and increased productivity

Develop sustainable mechanisms for continuous improvement in the productivity and quality of software developed across LaRC

Increase customer satisfaction with LaRC software products


CornerStone Activities

Lay the Foundation Establish Infrastructure

Plan for the Assessment CornerStone Planning and Validation

Baseline Workshop Training Customer Workshops Supplier Workshops Follow-Up Interviews (as needed) Best Practices Documentation Analyze Workshop Results Prepare and Present Findings

Plan Define LaRC SEPG Develop SPI Plan Review SPI Plan with Sponsors Present SPI Plan

Implement Improvements …


CornerStone Team Members

Norma Campbell, RTG/FDCD Victoria Chung, IOG/ISSD Michael Holloway, RTG/FETD Chuck Niles, IOG/FSED Pam Rinsland, IOG/AESDPat Schuler, IOG/ISSDJim Townsend, RTG/FMADJim Watson, OSEMA/OMASue Voigt, SASPG/SSCD

Consultant: Cindy Torpey, ChangeBridge Inc.


CornerStone Scope

Centerwide involvement Organizations involved in software

management, development, and maintenance

Customers of software projects 101 civil servants and contractors

interviewed


CornerStone Principles

Start with a process framework Baseline current state (not audit)

Conduct interviews and discussions Observe strict confidentiality

Involve senior management Work as a Centerwide team Focus on LaRC needs


Capability Maturity Model

Initial

Repeatable

Defined

Process is informal and unpredictable

Project management system in place; performance is repeatable

Software engineering and management processes defined and integrated

Product and process are quantitatively controlled

Time/$/...

Time/$/...

Time/$/...

Optimising Process improvement is institutionalised

Time/$/...

Time/$/...

Level Process Characteristics Predicted Performance

Managed

Pro

bab

ilit

y

Ta

rge

t N

-2

Pro

bab

ilit

y

Ta

rge

t N

-y

Pro

bab

ilit

y

Ta

rge

t N

-x

Pro

bab

ilit

y

Ta

rge

t N

-1-a

Pro

bab

ilit

y

Ta

rge

t N


Capability Maturity Model Repeatable / Level 2 (All Key Process Areas covered in

interviews ) Requirements Management Software Project Planning Software Project Tracking & Oversight Software Quality Assurance Software Subcontractor Management Software Configuration Management

Defined Level (Selected Key Process covered in interviews) Training Program Software Product Engineering Intergroup Coordination Peer Reviews


CornerStone Approach….

Baseline current state (snapshot in time) …some improvements are already initiated

Baseline included all areas which impact the software development group … some are under our control, others will require a

coordinated effort Not all points are at the same detail

…some are very specific, other are more general Processes were assessed, not specific groups

…especially when we discuss SQA and Training Program

Identify areas for improvement …not specify how to


CornerStone Findings

Key Process Area

Purpose and Description

Best Practices

Improvement Opportunities


Requirements Management

Purpose Establish a common understanding of the customer’s

requirements that will be addressed by the software project Description

Establish and maintain an agreement with the customer on the requirements for the software project

The "customer" is a system engineering group, TAG, a project office, another internal organization, or an external customer

Agreement covers both the technical and nontechnical (e.g., delivery dates) requirements

Agreement forms the basis for estimating, planning, performing, and tracking the software project's activities throughout the software life cycle



Best Practices Requirements are prioritized and managed

according to priority Developer works with customer to

interactively draw out requirements Operational Concepts Documents help in

understanding the requirements (some projects put on the web for increased visibility)

Test cases traceable back to requirements Explicit contractor task to capture and

manage requirements Rapid prototyping to validate requirements

with customer



Improvement Opportunities Importance of requirements is not realized Requirements are neither defined early enough, nor refined far enough Inadequate resources dedicated to requirements management Software requirements do not track to system requirements Don’t know how to adequate document requirements Customers are not trained in requirements generation Changes in requirements handled by overtime masks over-commitment Requirements creep is not managed nor controlled (cost is not

understood) No established policy to guide projects in requirements management Role and responsibility of systems analysis is not clearly understood Requirements are not consistently documented and reviewed with the

customers Segmentation of responsibilities between researchers and software

developers leads to uncertainty in product functionality



Consequences Project schedule and cost are significantly

increased when requirements are inadequately defined and documented

Rework is common due to changing requirements

Customer satisfaction is decreased when their requirements are not met


Software Project Planning

Purpose Establish reasonable plans for performing

software engineering and managing the software project

Description Develop estimates for the work to be

performed, establish the necessary commitments, and define the plan to perform the work



Best Practices Project cost, effort and duration estimates are

developed and tracked Software Work Breakdown Structure is generated

at the same level as hardware WBS Scheduling tools (MS Project, REVIC) are used Statements of Work specifies creation of project

plans Software staged release at planned milestones Software manager designated to oversee a

software development project Software personnel participate in project plans SQA involved in project planning


Software Project Planning Improvement Opportunities

Schedules are driven by externally set milestones, instead of actual work required

Software Management Plans are not consistently documented, if at all

No planning metrics are captured across the organization on which to base realistic future estimates

Commitments are not honestly negotiated Inadequate project planning is compensated for by overtime Over-commitment No LaRC project policy for managing software projects - the

approach to software project management is project dependent

The value of software project plans is not fully understood by developers, managers, or researchers

Personnel are not aware of available procedures and guidelines for software estimating

No guidelines are available for document software development plans



Consequences Software projects do not meet schedules or

budgets (could face cancelation) Burn out civil servants and contractors


Software Project Tracking and Oversight

Purpose Provide adequate visibility into actual progress

so that management can take effective actions when the software project's performance deviates significantly from the software plans

Description Track and review the software accomplishments

and results against documented estimates, commitments, and plans

Adjust plans based on the actual accomplishments and results



Best Practices Informal interchange meetings are held

frequently for tracking Automated tools (MS Project, PC Artemis) are

used to track project progress Milestones used as checklist Formal reviews held at selected milestones Dedicated business manager to track

software project status


Software Project Tracking and Oversight Improvement Opportunities

Corrective actions are not taken in a timely manner Software costs are not accurately reflected in project

charge structures No measurements or lessons learned are captured and

used for future projects Software size is not estimated or tracked Managers are not trained in software project

management and tracking Technical issues are not managed and communicated Software Development Plans are not used to manage

the project Software project reviews are ad hoc and ineffective;

results are not elevated to management (guidelines needed)



Consequences Products not completed on expected dates The real software project status is not known

until it is too late to do anything about it


Software Subcontractor Management

Purpose Select qualified software subcontractors and

manage them effectively Description

Select a software subcontractor Establish commitments with the

subcontractor Track and review the subcontractor's

performance and results


Software Subcontractor Management Best Practices

Periodic reviews are conducted with contractors to review progress and communicate status

COTRs and Technical Monitors are designated to establish and manage the software contract

Source Evaluation Boards select contractors based on their ability to perform the work

Success has been demonstrated using Performance Based Contracting

Formal quarterly evaluations are required Software services are acquired using standard

NASA contracting regulations


Software Subcontractor Management Improvement Opportunities

No accountability for poor contractor performance COTRs and Technical Monitors are not trained in software

engineering or managing software development efforts Performance Based Contracting is misunderstood, and

due to poor training it is ineffectively applied Source Evaluation Board frequently recommends lowest

bidder rather than best qualified Required reviews are not consistently conducted Contractor turnover is high and requires frequent re-

orientation resulting in increased costs No written LaRC policy or guidelines for managing

software contracts Contractors over-commit and rely on “free” overtime Skill level and training on the contract does not match

what is required


Software Subcontractor Management

Consequences Frequent dissatisfaction with contractor’s

products PBC task generation is more time-consuming


Software Quality Assurance

Purpose Provide management with appropriate visibility

into the process being used by the software project and of the products being built

Description Review and audit the software products and

activities to verify that they comply with the applicable procedures and standards

Provide the software project and other appropriate managers with the results of these reviews and audits



Best Practices

SQA contractors advise and give consultation to projects on software engineering when requested (on configuration management and software management plan)

Infractions are reported to participating project teams



Improvement Opportunities Need sufficient staff for uniform coverage of SQA

on Langley software projects Increase awareness of added value of SQA

practices Appropriate guidelines for SQA activities (current

LHB not widely accepted) Review SQA activities on a regular basis Track cost and associated return on investments

on SQA tasks


Software Configuration Management

Purpose Establish and maintain the integrity of the products of the

software project throughout the project's software life cycle Description

Identify the configuration of the software (i.e., selected software work products and their descriptions) at given points in time

Systematically control changes to the configuration Maintaining the integrity and traceability of the

configuration throughout the software life cycle Placed under software configuration management both the

software products delivered to the customer (e.g., the software requirements document and the code) and the items identified with or required to create these software products (e.g., the compiler)



Best Practices Configuration Control Boards are established and

used to control changes, assess impacts and communicate status

Automated tools (ClearCase, PVCS, RCS, LibLink, CVS, LabView) are used to control software work products (requirements, change rationale, code and supporting documentation)

Intent of SCM can be met without the use of automated tools for projects

Web-based change request system established (ClearDDTS)

SCM process documented and followed on some projects



Improvement Opportunities Rigor of formal SCM is not always appropriate for project’s

size, phase and purpose Change request status is not adequately communicated across

all project personnel Lack of awareness of existing guidelines and templates results

in significant duplication of effort Insufficient funding for SCM staff, tools and training SCM typically applied too late in project phase SCM plans are not documented and/or followed and changes

are not controlled Lack of understanding of the benefits of SCM Contracts do not always require the appropriate level of SCM

and there is inadequate CM on deliverables



Consequences Lack of SCM results in compromised mission

certainty and validity of data and products


Training Program

Purpose Develop the skills and knowledge of individuals so

they can perform their roles effectively and efficiently Description

Identify the training needed by the organization, projects, and individuals

Develop or procure training to address the identified needs

Evaluate current and future skill needs and determines how these skills will be obtained

Use informal vehicles when appropriate (e.g., on-the-job training and informal mentoring)


Training Program

Best Practices Training Office exists to meet the needs of the

LaRC including the software development community

Some projects and organizations successfully leverage the capabilities of the Training Office to meet their needs

Training Office annually surveys LaRC to establish training needs


Training Program

Improvement Opportunities Increase is needed in training budget due to cross

training and retraining of LaRC personnel Inconsistent management support for training Insufficient time is allocated for attending training Training is not considered a priority Need full implementation of Individual Development

Plan (IDP) which ties to the SQF and the agency strategic plan

No effective mechanism for consistently training contractors and civil servants working on the same project

No project training plan Travel funds shortage limits training opportunities

and technical exchange


Software Product Engineering

Purpose Consistently perform a well-defined

engineering process that integrates all the software engineering activities to produce correct, consistent software products effectively and efficiently

Description Perform the engineering tasks to build and

maintain the software using the project's defined software process and appropriate methods and tools



Best Practices Operational Concept, Users Manual and Test Plans/Cases

developed prior to code Apply techniques such as Object Oriented Design (OOD),

reverse engineering, structured programming, and modularity to reduce code complexity, increase reusability and improve maintainability

Automated software development tools (Purify, Quantify, PureCoverage, Rational Rose, ClearCase, ClearDDTS, Object Manual, LabView GUI) are used to design, test, CM, and document software

Provide realistic operational testing scenario for software Office automation tools used in innovative ways for

software development, testing and documentation (databases, spreadsheets)



Best Practices (cont.) A project has demonstrated LaRC’s ability to meet

FAA standards for airworthy flight software On line Web sites exist that describe software

engineering guidebooks, formal methods and metrics collection database

Third party tests against requirements agreed to by users

Continuous rapid prototyping used to demonstrate feasibility and early progress

Requirements gathering techniques resulted in improved software product

Centralized facility provides access, guidance and expertise for domain-specific tools



Improvement Opportunities Insufficient time allocated for software engineering

tasks (requirements definition, design, testing, integration, configuration management, and documentation)

Software engineering activities often hidden because their value is not recognized, perceived as overhead by projects, and researchers/projects do not want to pay for reuse (no corporate view for long-term investment)

No rewards for good software engineering Single point failures on many projects created by one

person and insufficient documentation Testing address physics only, not software quality No LaRC dissemination guidelines for software exist



Improvement Opportunities (cont.) Lack of in-house software product engineering

expertise Need point of contact where software engineering

expertise and guidance can be obtained Ineffective tool utilization due to poor communication

and awareness Web sites not advertised Software engineering techniques not appropriately

tailored for small projects No incentive to take research code to production

quality or make it reusable by other projects Lack of expertise in project planning and scheduling

leads to insufficient time for fundamental software engineering tasks


Software Product Maintenance

Best Practices Use portable computer to set up, perform experimental

test, and analyze results Improvement Opportunities

Insufficient funding for sustaining and maintenance of software

Ineffective implementation of CM prohibits quick minor changes to software for experimental use

Insufficient verification of software prior to delivery leads to high maintenance cost

Due to insufficient manpower and maintenance procedures, staff is not adequately notified on software changes

Constant changes in COTS upgrade decrease productivity due to perpetual learning curve


Intergroup Coordination

Purpose Establish means for the software engineering group to

participate actively with the other engineering groups so the project is better able to satisfy the customer's needs

Description Software engineering group participates with other project

engineering groups to address system-level requirements, objectives, and issues

Representatives of the project's engineering groups participate in establishing the system-level requirements, objectives, and plans by working with the customer and end users, as appropriate

Requirements, objectives, and plans become the basis for all engineering activities


Intergroup Coordination

Best Practices Workshops and shared facilities bring different groups

together for improved information exchange Interface Control Documents, frequent meetings and

timely meeting notes ensure effective exchange of information

Web-based information and email exchange facilitates technical coordination

Physical co-location of discipline specialists promotes intergroup coordination during a project

Teamwork training is available Communication between software programs used by

different groups achieved by in-house developed interfaces (scripts, GUIs, wrappers, standard features, filters)


Intergroup Coordination Improvement Opportunities

Need to have representatives of all technical disciplines (including software) involved from the start of a project (requirements, cost estimates, operational concepts, requirements allocation)

Due to the lack of systems engineering performed at LaRC, software and hardware staff are forced to fill that role in an ad-hoc manner

Poor communication among groups doing similar work within LaRC results in duplication of work

No effective mechanism to ensure all disciplines are represented on project teams

Software specialists not aware of overall project concept Perception that software can fix anything, including poor hardware

selection, leads wasted funds and staff hours Lack of documented commitments between engineering groups Changes are not effectively communicated across the engineering

groups Contractors resist communication due to proprietary fear


Peer Reviews

Purpose Remove defects from software products early in

the development process where it is more cost effective to remove them

Develop better understanding of software products and defects that might be prevented

Description Peers methodically examine software work

products to identify defects and areas where changes are needed

Identify and schedule specific products that will undergo peer review as part of the defined software process


Peer Reviews

Best Practices Peer reviews identify problems early in the lifecycle resulting

in saved time and decreased costs Post-mortem review is effective mechanism to capture

lessons learned Informal review by an in-group peer works well on small

projects Peer reviews captures problems not otherwise identified Completion of peer reviews is a phase exit requirement and

indicates readiness to proceed Peer reviews are effective mechanism to train staff and

indoctrinate new hires Peer review process is documented (NASA Formal Inspection

Handbook, project documentation) User requirements are basis for code reviews Email and Key Activities are used to document peer review

results


Peer Reviews Improvement Opportunities

Peer reviews are not commonly used (even unknown by some projects) in spite of positive return on investment experienced at LaRC

Current review process frequently omits software issues

Current formal design review system (PDR, CDR) often misses large problems

Post-mortem reviews hardly ever held Limited time, training, and staff are provided for

performing peer reviews No data collected on peer review results and

lessons learned (dormant data base)


Non-CMM Related Concerns

Improper ISO 9000 implementation could impose restrictive procedures on research and significantly impact resources to perform future research

Numerous management issues were identified such as: The lack of management awareness of the pervasiveness and

magnitude of work required to develop software at LaRC Need investment, encouragement, and reward system for

improvements, best practices, positive technical transfer Lack of adequate resources for performing software

development

There is a lack of a central pool of software developers to service LaRC

Relation of LaRC’s Strategic Quality Framework to real work is poorly defined


Next Steps


Critical Point in LaRC’s SPI Program

CornerStone phase is a critical milestone in the software process improvement program

CornerStone phase baselines our current state and develops a plan for future process improvement

Proactive management is required to go forward with successful implementation of the Improvement Plan


Upcoming Activities

Receive feedback on Findings Briefing Develop draft Software Process

Improvement Plan Review SPI Plan with sponsors Establish Senior Management Steering

Group (SMSG) Establish Software Engineering Process

Group (SEPG) Finalize SPI Plan Conduct SPI Plan Briefing Implement Improvement Plan


Management Steering Group Description Comprised of the LaRC CIO, ISO Project Manager, and

Division Chiefs who have a stake in the successful achievement of the Software Process Improvement (SPI) goals

Monitors and evaluates SPI efforts from the perspective of the total organization to ensure that the overall improvement efforts are in concert with LaRC’s mission and goals

Meets regularly with the Software Engineering Process Group (SEPG) to discuss the progress of the SPI program, problems, issues and concerns

Works together with the SEPG to identify and provide the long-term commitments and resources required to coordinate the development and maintenance of software engineering processes for use by current and future software projects


Suggested Management Steering Group Membership

Doug Arbuckle/Associate Director, LaRC Fay Collier/ISO 9000 Project Manager, LaRC Pat Dunnington/CIO, LaRC Luat Nguyen/FDCD Bill Wessel/OSEMA Carl Gray/FSED Milt Holt/FETD Rob Kudlinski/ISSD Lenny McMaster/AESD Bill Smith/SSCD David Stephens/FMAD


Responsibilities of the Management Steering Group (MSG) Ensure alignment of Software Process Improvement (SPI)

initiatives with LaRC mission and goals Approve strategic plan for SPI Provide sponsorship, pro-active commitment, and visible

management support Allocate resources Monitor the progress of the SPI initiative Provide guidance and direction to the Software

Engineering Process Group (SEPG) Conduct regular meetings with the SPI Group Promote cooperation and cross-functional

communications Cultivate an enabling environment for continuous

process improvement Obtain and sustain the support for the SPI initiative


Software Engineering Process Group (SEPG) Description Chartered by the MSG, as the focal point for software

process improvement Coordinate and plan the SPI program activities in

concert with the guidance and direction of the MSG Report the progress of SPI related activities to the MSG

Provide technical support and consultation Staffed by experienced software development

practitioners who facilitate the definition, maintenance, and improvement of the software engineering processes used within the organization

Work collaboratively with the projects to resolve process related problems and assist in the development, training and implementation of processes and procedures


Suggested Software Engineering Process Group (SEPG) Membership Members of the CornerStone team

Volunteers from interview workshops

Appointees from participating divisions


Responsibilities of the Software Engineering Process Group (SEPG)

Define and manage the plan for development and implementation of software process improvements across the LaRC

Provide a pool of software engineering expertise and corporate knowledge (Formal part of LaRC organization with assigned resources and management commitment)

Provide consultation and guidance on appropriate level of software engineering implementation and future directions

Provide and facilitate education on software engineering to management and staff via workshops, seminars, symposiums, and setting up news/user groups and maintain web sight

Provide a repository for reuse code, documents, tool knowledge, procedures, processes, LaRC best practices, templates, lessons learned, metrics, and examples

Facilitate sharing of tools and maintenance of COTS across projects Build and reinforce sponsorship for the SPI program management

support at all levels


Next Steps Establish membership of MSG & SEPG (by Nov. 12)

Post Best Practices on Web

Prioritize Improvement Opportunities

Develop SPI Project Plan

Review and Obtain Approval of Plan (by Dec. 19)

Complete Final Report

Implement Improvement Plan


Closing Thought

“Creative thinking may simply be the realization that there is no particular value in doing things the way we’ve always done them.”

- R. Flesch, Educator


Odd and ends follow:

Global Improvement Opportunities Seminars (maybe during lunch) on SW topics of interest Establish Web site to capture and publicize best practices Establish basic training in software engineering Short-course training on widely-used COTS. Need to provide guidelines and expertise for appropriate level of software

engineeringprocedures based on project size, application and criticality.


Odds and ends cont.:

Some analysis was started on the Closing Q1 data that is captured below(not to be used in findings briefing):

Global Improvement Opportunities>Provide education on requirement gathering and documentation>Do pilot/demonstration projects on selected software engineering activities>Conduct lesson learned review on project complete and post results on the web

Management Related Improvement Opportunities>Need to overcome resistance to positive change>Segmentation of responsibilities across organizations has led to lack of single point of accountability for technical issues>Improve management awareness of the pervasiveness?? importance and magnitude of work required to develop software

Global Concerns>Improper ISO 9000 implementation could impose restrictive procedures on research and significantly impact resources to perform future research>Civil servants should be made technically knowledgeable in software>Sponsor research in appropriate software engineering practices for the research environment

NASA software put in commercial system with no profit to NASA (companies make profits on NASA software)


Responsibilities of the Software Engineering Process Group (SEPG)

Establish a repository for SPI products (best practices, code, and lessons learned)

Establish appropriate organizational metrics and mechanisms for the collection of metrics

Provide desk-side mentoring to project teams Track, monitor and report the status of SPI initiatives to the MSG Promote technical awareness and coordinate training for software

engineering with the LaRC training office Provide consultation and CMM expertise to the MSG and the technical

working groups which implement improvements Compare current practices against the goals and key practices of the

CMM Keep LaRC apprised of SPI activities and progress


Center Sponsors

Kristin Hessenius/Associate Director, LaRC

Fay Collier/ISO 9000 Project Manager, LaRC

Pat Dunnington/CIO, LaRC

Rob Kudlinski/ISSD

NASA Langley Research Center Software Process Improvement Initiative (SPII) Findings Presentation...

Documents

Transcript of NASA Langley Research Center Software Process Improvement Initiative (SPII) Findings Presentation...