PDR.06 PRELIMINARY SOFTWARE ENGINEERING PLANbroekema/papers/SDP-PDR... · To do before CDR Check...

Document No: SKA-TEL-SDP-0000042 Unrestricted

Revision: 01 Author: Kevin Vinsen

Release Date: 2015-02-09 of 27

PDR.06 PRELIMINARY SOFTWARE

ENGINEERING PLAN

Document number SKA-TEL-SDP-0000042

Context MGT

Revision 01

Author Kevin Vinsen

Release Date 2015-02-09

Document Classification Unrestricted

Status Draft




Name Designation Affiliation

Signature & Date:

Name Designation Affiliation

Signature & Date:

Version Date of Issue Prepared by Comments

0.1 Kevin Vinsen

ORGANISATION DETAILS

Name Science Data Processor Consortium

Signature:

Email:

Signature:

Email:

Kevin Vinsen (Feb 10, 2015)Kevin Vinsen

Senior Research Fellow

[email protected]

ICRARKevin Vinsen

Paul Alexander (Feb 10, 2015)Paul Alexander

SDP Lead

[email protected]

University of CambridgePaul Alexander

https://secure.echosign.com/verifier?tx=XJE6UN8W2M6SR3I

https://secure.echosign.com/verifier?tx=XJE6UN8W2M6SR3I




1 Table of Contents 1 Table of Contents .................................................................................................................................. 3

1.1 List of Figures ................................................................................................................................. 5

1.2 List of Tables .................................................................................................................................. 5

1.3 List of Definitions ........................................................................................................................... 5

1.4 List of Abbreviations ...................................................................................................................... 5

2 Introduction ........................................................................................................................................... 5

2.1 Purpose of the document .............................................................................................................. 5

2.2 Scope of the document ................................................................................................................. 6

3 References ............................................................................................................................................. 7

3.1 Applicable Documents ................................................................................................................... 7

3.2 Reference Documents ................................................................................................................... 7

4 Software Categories .............................................................................................................................. 7

5 Configuration Management Plan Guidelines ........................................................................................ 8

5.1 Introduction ................................................................................................................................... 8

5.2 Management ................................................................................................................................. 9

5.2.1 Organisation .......................................................................................................................... 9

5.2.2 Responsibilities .................................................................................................................... 10

5.2.3 Policies, directives and procedures ..................................................................................... 11

5.3 General Configuration Management .......................................................................................... 11

5.3.1 General ................................................................................................................................ 11

5.3.2 Configuration Item Identification ........................................................................................ 12

5.3.3 Configuration Control .......................................................................................................... 13

5.3.4 Configuration Status Accounting ......................................................................................... 14

5.3.5 Configuration Verification ................................................................................................... 14

5.3.6 Audits of CM system ............................................................................................................ 14

5.4 Implementation of information/documentation management .................................................. 14

5.4.1 Issue Management .............................................................................................................. 14

5.4.2 Control ................................................................................................................................. 14

5.5 Software Configuration Management ......................................................................................... 15

5.5.1 Software Version Control Tool ............................................................................................ 15




5.5.2 Software Release ................................................................................................................. 16

6 Software Engineering Plan Guidelines ................................................................................................ 16

6.1 Introduction ................................................................................................................................. 16

6.2 Software related system requirement process ........................................................................... 19

6.3 Software management process ................................................................................................... 20

6.4 Software requirements and architecture engineering process .................................................. 20

6.5 Software design and implementation process ............................................................................ 20

6.6 Software validation process ........................................................................................................ 21

6.7 Software delivery and acceptance process ................................................................................. 21

6.8 Software verification process ...................................................................................................... 22

6.9 Software operation process ........................................................................................................ 22

6.10 Software maintenance process ................................................................................................... 22

7 Agile Software Development For SDP Products .................................................................................. 23

7.1 10.1. The Role of Testing ............................................................................................................. 23

7.2 Granularity of Requirements and Releases ................................................................................. 24

7.3 Continuous Integration and Continuous Delivery ....................................................................... 24

7.4 Large Scale Commercial Distributed Agile Development Example from ThoughtWorks ............ 25

7.5 Role of the Scientific Community ................................................................................................ 26

7.6 Testing Frameworks .................................................................................................................... 26

7.7 Support and Maintenance ........................................................................................................... 27




1.1 List of Figures Figure 1: Example organisation chart ............................................................................................................ 9

Figure 2: Overview of the software life cycle process .................................... Error! Bookmark not defined.

1.2 List of Tables Table 1: The categories of software the SDP element expects to use. ......................................................... 8

Table 2: Item Types ..................................................................................................................................... 12

1.3 List of Definitions Specific to this document:

Term Definition

Customer The body purchasing or acquiring the software system. This could be the SKAO or the SDP consortium on behalf of the SKAO.

Product A product or sub-product from the SDP product tree.

Project The top level SDP project or the project that will be created to develop a product.

A list of SDP specific definitions is provided in [RD01].

1.4 List of Abbreviations A list of abbreviations is provided in [RD01].

2 Introduction 2.1 Purpose of the document This document provides guidelines for the production of the Software Engineering and Configuration

Management Plans CMPs respectively. Each product from the SDP product tree will be required to

produce their own Software Engineering Plan (SEP) and Configuration Management Plan (CMP).

This preliminary plan uses the ECSS standards [RD02, RD03] as the basis for templates and detailed

overviews of the individual processes. The Configuration Management (CM) component is based on

ECSS-M-ST-40C [RD02] and the software component is based on ECSS-E-ST-40C [RD03].

Suppliers/developers will be at liberty to customise their plans, any deviations from the guidelines

provided in this document and the ECSS standards will need to be clearly identified in the product’s

plans. In particular, it is expected that COTS products and modifications to existing systems will have

heavily customised plans.

Therefore, the purpose of this Preliminary Software Engineering Plan is to define the overarching

Software Engineering and Configuration Management processes, tools and techniques to be used for




software that is created or adopted for the SKA.SDP element. This includes procured software, open

source software, custom software and associated systems.

Templates for the CMPs and SEPs generated for each product are provided in the ECSS standards [RD02,

RD03]. Note that this preliminary document is kept very generic by intention as it will need to comply

with the, currently unknown, SKA wide tendering and procurement guidelines and procedures. In the

final version for CDR it is expected to provide more specific, but also more restrictive guidelines for

potential bidders, vendors and implementers of SDP products. Examples of such restrictions and

recommendations are provided in various places throughout this document.

2.2 Scope of the document All software systems designed, developed, modified, procured or configured within the SDP work

package lie within the scope of this document. In the context of this document a ‘software system’

means software and other elements, which operate in tandem (e.g. data, hardware). Some of these

systems may interface with non-software elements which are not within the scope of this document.

Data migration, tagging strategies, data format policies, data repositories, buffers etc., are outside the

scope of this document.

Standards per se are not within the scope of this document. It is expected that standards and

standardisation are considered where appropriate, but how standards are considered is not prescribed.

At this stage the document does not attempt to define what aspects of the CMP and SEP plans will be

mandatory for different types of software procurement (bespoken, COTS, or MOTS).

To do before CDR

As the SDP design matures and the procurement model for the SDP develops, the CMP and SEP requirements will become clearer and we will define our own standards and what customisation will be allowed and what will be mandatory.




3 References 3.1 Applicable Documents The following documents are applicable to the extent stated herein. In the event of conflict between the

contents of the applicable documents and this document, the applicable documents shall take

precedence.

Reference Number

Reference

AD01 Configuration Items List - SKA-TEL-SDP-PDR-009

3.2 Reference Documents The following documents are referenced in this document. In the event of conflict between the contents

of the referenced documents and this document, this document shall take precedence.

Reference Number Reference

RD01 SKA-TEL-SDP-0000007-glossary-PDR16

RD02 ECSS-M-ST-40C

RD03 ECSS-E-ST-40C

RD04 ECSS-M-ST-10

RD05 SKA-TEL-SDP-0000051 PDR12 Prelim Verification & Qualification Plan (V&Q)

4 Software Categories The software systems that will be designed, developed, modified, procured or configured for the SDP

workpackage can be divided into 3 generic categories as shown in table 1.




Category Description

Bespoke Bespoke software will be written expressly for the SDP. This category of software system will require a full CMP and SEP. The plans can be tailored to allow for the development methodology of the supplier (Agile, Waterfall, etc.).

Modified Modified software is software that will be developed from an existing code base. The code base could be for “closed” commercial software or “open source” software. These will need most parts of the full plans, but some areas could be omitted.

COTS This software should work out of the box. This could be commercial software or open source software that is to be used unmodified. The plans will focus primarily on configuration and integration of the software.

TABLE 1: THE CATEGORIES OF SOFTWARE THE SDP ELEMENT EXPECTS TO USE.

Recommendations

Given the nature of the SDP project an Agile development method would be strongly recommended. Section 7 of this document describes how Agile Methods could be applied.

Recommendations

This document is split into two sections, one for the CMP (Configuration Management Plan) and one for the SEP (Software Engineering Plan).

5 Configuration Management Plan Guidelines 5.1 Introduction Configuration management (CM) refers to a discipline for evaluating, coordinating, approving or

rejecting, and implementing changes in artifacts that are used to construct and maintain software

systems. An artifact may be a piece of hardware or software or documentation. CM enables the

management of artifacts from the initial concept through design, implementation, testing, baselining,

building, release, and maintenance.

This document defines the “preliminary” configuration management requirements, for the development

of any product from the SDP product tree. Each product will be required to have a configuration

management plan (CMP) responding to these configuration management requirements. In the case of

modified software, the modifier will be expected to prepare the document and in the case of COTS

products, a member of the customer configuration management team will prepare the document. The




CMP will then be submitted to the customer for approval. Upon customer approval, the supplier will

execute their own CMP and ensure that their lower tier suppliers execute their CMP.

The purpose of the CMP is to define the process and resources for managing the configuration of the

product, in a controlled and traceable manner throughout its life cycle. It also describes the means for an

efficient comparison between the predicted (“as‐designed”) and the actual (“as‐built”) configuration of

the delivered product. It defines the relationship with the project management, system engineering and

quality management process.

The CMP provides all elements necessary to ensure that the implementation of the

information/documentation management meets all customer requirements, and that it is in line with the

programme or project organisation and management structure.

The customer defines the project phase during which the CMP is prepared and approved.

The plan will document a person responsible for implementing configuration management activities and

for implementing information/documentation management activities within the project team. Their

role, responsibilities and authorities will be described in the CMP.

The CMP for each product shall contain the following sections which are described in detail in the

subsequent sections of the document. A template document is given in RD02.

To do before CDR Check the templates for the Configuration Management Plans in RD02 are appropriate for the SDP and remove ECSS elements that are not applicable

5.2 Management 5.2.1 Organisation An organisation chart should be provided and an example is shown in Figure 1. The role of the Manager

is not defined by the CMP, it is usually the project manager. The roles under the manager are briefly

described below.

FIGURE 1: EXAMPLE ORGANISATION CHART




5.2.2 Responsibilities The responsibilities described here are for configuration management. A supplier is at liberty to

customise these CM roles, but these roles are considered the minimum necessary for successful CM. For

small projects these roles could all be performed by one or two individuals. For example for small

projects the responsibilities could be as follows:

● Project Manager and Test Manager roles could be one person. This strategy is a good option for

smaller test teams.

● Test Manager could also the same person who takes the Test Designer role. This would require

strong management and leadership skills as well as strong technical skills and experience.

● Test Manager and Test Analyst roles could be performed by a single person.

5.2.2.1 Configuration Manager The Configuration Manager will be responsible for configuration management activities inside the

product. The following list is not exhaustive, but is intended as a guideline to the CM activities:

● Assure that Configuration Management Plan (CMP) is correctly applied in the product and

provide appropriate reasons in case of non-conformances

● Define which Configuration Items are to be managed in the Configuration Item List

● Define the Product Baseline

● Support changes to Configuration Items within the Change Control Board

● Manage the delivery of software products

● Maintain the Configuration Item List

● Manage the configuration control resources used by the products, in particular the product

specific areas of the source code management system

● Be aware of the relation between the elements of the Product Baseline (in order for instance to

be able to answer the question: “What is the environment and which software is installed?”)

● Check that the CMP procedures are correctly applied when Configuration Items are changed

● Participate in CCB activities

The Configuration Manager will be the secretary of the CCB and work with the support of the Scientific

and Technical leaders of the product, and participate in the CCB monitoring the development and

change control process.

5.2.2.2 Quality Assurance Leader The Quality Assurance leader will be responsible for Product Assurance activities within the project. The

activities are mainly:

● Checking the baselines before delivery

● Being aware of the relations between the elements of the Configuration Baseline (in order for

instance to be able to answer the question: “Which validation tests correspond to this software

release?”)

● Defining and applying the delivery conditions for the software product,

● Participate in CCB activities according to the Software Product Assurance Plan

● Check that the Engineering Guidelines are respected during the development process




5.2.2.3 Test Manager The test manager will be responsible for all test activities in the specific organisation (SDP or product).

The activities are mainly:

• Checking of tests in line with baselines

• Being aware of the relations between the elements of the Configuration baseline (in order for

instance to be able to answer the question: “Which defects correspond to which software

products/releases?”)

• Defining and applying test plans and evaluations for the software product

• Planning and management of test resources

• Maintaining the level of quality through the resolution of important defects

• Updating/maintaining issue tracking of software defects and change requests

• Participating in CCB activities

• Assessing and advocating the quality of the software product

5.2.3 Policies, directives and procedures

To do before CDR Describe any directives from the SKAO that are applicable.

5.3 General Configuration Management 5.3.1 General At its heart, CM is intended to eliminate the confusion and error brought about by the existence of

different versions of artifacts. Artifact change is a fact of life and must be planned for. Changes are made

to correct errors, provide enhancements, or simply reflect the evolutionary refinement of product

definition. CM is about keeping the inevitable change under control. Without a well-enforced CM

process, different team members (possibly at different sites) can use different versions of artifacts

unintentionally; individuals can create versions without the proper authority; and the wrong version of

an artifact can be used inadvertently. Successful CM requires a well-defined and institutionalised set of

policies and standards that clearly define:

● The set of artifacts (configuration items) under the jurisdiction of CM

● How artifacts are named

● How artifacts enter and leave the controlled set

● How an artifact under CM is allowed to change




● How different versions of an artifact under CM are made available and under what conditions

each one can be used

● How CM tools are used to enable and enforce CM

These policies and standards are documented in a CMP that informs everyone in the organisation just

how CM is carried out.

5.3.2 Configuration Item Identification Each component of the SDP that is deemed significant and warrants being placed under CM, or that

could become significant for the project, should be considered a Configuration Item (CI). Each CI will

have a unique Configuration Item Identifier. To avoid confusion and to simplify the identification of all

CIs inside SDP, a simple rule to build this ID as an alphanumeric string is presented. The best option is to

use an ID that can give some information about the CI. Each configuration item shall be identified by a

unique ID. The naming convention to use is:

SDP_Type_Product_X[.Y]

where:

● SDP: is just a prefix to show which consortia the item comes from if it is referred to by the SKAO.

● Type: is the acronym corresponding to the type of CI defined in Table 2.

● Product: is the product name/abbreviation (from the Product tree) or SDP for a CI that covers all

products;

● X: is a sequential number inside SDP or a product, monotonically increasing for the same type

and sub-product; starting value 1;

● Y: is a sequential number for sub-CIs if the product is subdivided in sub-products (optional);

starting value 1

Item Type Acronym

Software deliverable SW

Database DB

Document DOC

Test dataset TD

Hardware HW

Network NW

Operational system OS

Common off the shelf software COTS

System1 SYS

TABLE 2: ITEM TYPES

1 This is an Aggregation CI used to hold a system of systems




The SDP currently only maintains a Product Tree for the whole project. It will become necessary to break

some products into sub-products that feed up into the master product tree.

A database of CIs will be maintained which at a minimum will contain the following:

● Configuration Item ID

● Description of the CI

● Version number of the CI

● Dates created and modified

● Update history, along with change notes

● Version Information

● The product baseline the CI applies to

To do before CDR

Investigate a tool that can do this across the SDP consortium and feedback to the SKAO.

5.3.3 Configuration Control 5.3.3.1 Overview Configuration control is the process for controlling the evolution of, or departures from agreed

baselines. It includes the preparation, justification, evaluation, disposition and implementation of

engineering and contractual changes, deviations and waivers.

5.3.3.2 Change Procedure A change procedure will need to be established to describe the process for changing a configuration

baseline.

5.3.3.3 Configuration Control Board Configuration Control Boards (CCBs) should be established at each project level as the relevant authority

for all changes. The CCB is convened by the configuration manager in agreement with the project

manager. The CCB consists of permanent representatives of all programme or project disciplines

necessary for the review and evaluation of changes. The members of the CCB must have decision‐making

authority.

A change initiated by the customer can only be implemented after examination and approval of the

supplier’s response, e.g. change proposal.

5.3.3.4 Classification of changes and departures The classification of a change or departure, defines the type of approval and release cycle required

according to criteria with regard to impacts on cost, schedule, technical specification and other technical

or contractual characteristics.

According to its effects, a change proposal or a departure request is processed through the different

levels of the organisation. The appropriate level to decide its disposition is the level for which the effects




of the change or departure have no repercussions on the commitments made to the customer. The

disposition is, however, transmitted to the customer for information.

5.3.3.5 Interface Control Interface control is part of the configuration control activity and defines the process necessary to freeze

and implement interface data, and the control of changes affecting the interfaces. The interface control

process is under the responsibility of system engineering supported by configuration management. The

CM activity provides the means to identify, track and report on the status of approved interfaces.

5.3.3.6 Supplier Control For both subcontracted and acquired products, the CMP shall define the process and data to flow down

the CM requirements and the programme monitoring methods to control the supplier.

5.3.4 Configuration Status Accounting The location and use of software must be tracked. A register must be maintained showing locations,

versions, and authorisations relating to each item. This links to Version Control and Security procedures.

5.3.5 Configuration Verification The CMP shall describe the process and data to verify the current configuration status from which the

configuration baselines are established.

5.3.6 Audits of CM system The CMP shall describe the process, data and schedule for configuration audits to ensure that the

configuration management of the programme or project is performed.

5.4 Implementation of information/documentation management 5.4.1 Issue Management An issue tracking system is a package that manages and maintains lists of issues, as needed by a project.

The issue tracking systems will be used to create, update, and resolve reported issues.

An issue can be a task, a new feature request, a bug report, document corrections, etc. Each issue in the

system may have an urgency value assigned to it, based on the overall importance of that issue. Other

details of issues include the person experiencing the issue, date of submission, detailed descriptions of

the problem being experienced, attempted solutions or workarounds, and other relevant information.

Each issue maintains a history of each change.

Recommendations The Jira issue tracking system would be ideal for this purpose. The SDP and SKAO use Jira internally for issue tracking

5.4.2 Control Each CI will be subject to version control. This entails issuing a version number and a schema for

updating the version under explicit conditions. Control of version numbers will be defined, how they are

updated and logged, and who has authority to grant or change version numbers.




The CI’s characteristics may change over the time for a number of reasons e.g. new functionality, bug

fixes, new feature request etc. To easily identify the current status of a CI integrated in an environment

composed of many CIs, it is necessary to define a Product Baseline.

A Product Baseline is identified as follow:

R C.N.m

Where:

● C: cycle number

● N : progressive number inside the cycle identifying major releases; starting value 0;

● m:minor release; starting value 0.

Major (N) and minor (m) releases are to be used depending on needs. A major release implies

substantial changes in terms of functionality and/or interfaces. A minor release implies that changes are

only small and non-breaking changes. That means that the behaviour of the new (minor) version and the

old version are compatible. For example data written with the old version can be manipulated with the

new (minor) version of the software.

If minor releases cannot easily be identified, the suggestion is to use only major releases. Follows

examples indicate some releases for a generic software product.

R 6.0.0: is the first release in cycle 6;

R 6.1.0: is the second release in cycle six; changes from previous release R 6.0.0 are quite

significant;

R 6.1.1: is a minor release in cycle 6, changes from R 6.1.0 are very few and non-breaking.

5.5 Software Configuration Management 5.5.1 Software Version Control Tool The system needs the following features:

● Support a distributed and adaptive workflow

● Have the usual features: branching, tagging, merging, and rebasing

● Strong support for non-linear development

● Have very strong safeguards against corruption, either accidental or malicious

● Efficient handling of large projects

Recommendations GIT is being used throughout the SDP. It is proven distributed version control system.




5.5.2 Software Release Within the SDP each product will usually be released at least once per cycle. The Development Plan of

each product will give details of the planned releases. Additional releases may be done in case of bug

fixing, issues, or in case that the previous one is incomplete.

Each release needs to be identified with:

● Configuration Item

● Documentation: Software User Manual, Software Release Notes, Software Test Report

● Tag in the software repository

● Location of the deliverables

This information links to the product baseline.

The SDP/product leader is in charge of preparing the release. After CCB approval, the release will be

delivered to the SDP.

Recommendations A CI tool will be need for continuous integration and deployment (see section 5.3.2). Bamboo provides the tool set necessary to perform Continuous integration and deployment. It also integrates with Jira directly (both products are from Atlassian).

6 Software Engineering Plan Guidelines 6.1 Introduction As a discipline, software engineering is not as mature as other engineering disciplines, and it still lacks

consensus on a well-recognized set of fundamental principles. Software engineering is defined by the

IEEE as:

The application of a systematic, disciplined, quantifiable approach to the development, operation

and maintenance of software, i.e. the application of engineering to software.

This document defines the core requirements that must be meet by the software engineering process.

This is customisable depending on the type of product being produced, in the same way the CMP can be

customised. The details of the requirements for the Software Engineering Plan and templates for the

various documents are provided in RD03.

A fundamental principle of this document is the ‘customer–supplier’ relationship (taken from RD03),

assumed for all software developments . The customer is, in the general case, the procurer of the

software for a system, subsystem, set, equipment or assembly. The concept of the “customer–supplier”

relationship can be applied recursively, i.e. the customer can be a supplier to a higher level in the SDP

system. The software customer therefore has two important interfaces:

● The software customer interfaces with his software suppliers in order to adequately allocate to

them functional and performance requirements, through the functional analysis.




● The software customer assumes a supplier role to interface in turn with his customer at the next

higher level, ensuring that higher level system requirements are adequately taken into account.

The customer derives the functional and performance requirements for the software, based on system

engineering principles and methods. The customer can also control the interface between the software

and hardware. Software items are defined in the SDP product tree at different levels. Nevertheless, it is

important to manage the software–software interfaces irrespective of the level at which they occur. The

customer’s requirements are specified by this process, and they provide the starting point for the

software engineering.

Reviews are the main interaction points between the customer and the supplier. They also synchronise

software engineering processes. The reviews relevant to the software engineering processes are the

system requirements review (SRR), preliminary design review (PDR), critical design review (CDR),

qualification review (QR), acceptance review (AR), and operational readiness review (ORR) as defined in

ECSS-M-ST-10 [RD04]. This Standard offers in addition the possibility to anticipate the PDR in a software

requirements review (SWRR), and the CDR into a detailed design review (DDR). The SWRR and DDR are

executed as a first part of the ECSS‐M‐ST‐10 reviews, but precede them.

To Do before CDR

Investigate introducing a series of Vendor Design Reviews (VDRs), where the first one would essentially be required to prove that the SDP-CDR design is matched by the Vendor design proposal. VDR0 could simply be the design submitted during tendering. The subsequent VDRs can be aligned with the release cycle and thus made more lightweight and streamlined.

All reviews are applicable to software. The reviews occur at different levels in the customer–supplier

hierarchy and are sequenced according to the overall system level planning.

The notion of engineering processes is fundamental to this document and the design of the SDP. The

processes provide the means to describe the overall constraints and interfaces to the engineering

processes at system level, even when agile methods are used. At the same time, they provide the

possibility for the supplier to implement the individual activities and tasks implied by the processes in

accordance with a selected software life cycle. This document is a process model, and does not prescribe

a particular software life cycle. Although the structure of this document suggest a waterfall model, the

software development plan can implement any life cycle, in particular by the use of the technical

reviews, less formal than the project reviews. Agile methods have demonstrated their effectiveness and

are reviewed in the context of SDP development in section 7.

The software development plan defines and instantiates in details the particular implementation of this

standard in a project.

Figure 2 (taken from RD03) identifies the main concurrent software engineering processes and shows

how and when they are synchronized by the customer/supplier reviews.




FIGURE 2 OVERVIEW OF THE SOFTWARE LIFE CYCLE PROCESS




To Do before CDR

Correct this diagram to have links to our documentation and not ECSS.

6.2 Software related system requirement process The software related system requirement process produces the information for input to the system

requirements review (SRR). This establishes the functional and the performance requirements baseline

(RB) (including the interface requirement specification) of the software development.

According to the recursive customer‐supplier model, at each level, the customer is responsible for the

delivery of a system in which the developed software is integrated. According to the recursive system‐

subsystem model of RD04, they are responsible for the specification of the system requirements at

lower levels and, in particular, for any software comprised in the system. As such, the tasks of this

process are normally performed by the customer, but can be delegated to the supplier under the

responsibility of the customer.

RD04 describes the following system engineering functions:

● Requirement engineering, producing the system and lower level functional and technical

specifications,

● System analysis, consolidating requirements through trade‐off and various analysis (mission,

requirement, functional, physical, performance), producing in particular the functional and

physical architecture,

● System design (producing the physical architecture) and system configuration (producing the

physical system with the software),

● System verification as being conformant to requirements, and

● System engineering integration and control, including in particular system integration, its

relation with production, operation, product assurance and management, and the engineering

database, the interface management and the technical budget management

The system engineering functions giving input to the software are the requirement engineering, the

system verification and the system engineering integration and control. For software, the system

analysis and design only support the system requirement engineering, the system configuration includes

the software development.

The system engineering produces “lower level element functional specification” in phase A before the

preliminary requirement review (PRR), and “technical specification” of lower level elements in the

preliminary system design file, baselined at system design review (SDR) in phase B. For software, this

document is the requirements baseline addressed in this clause, and the software SRR is synchronized

with the system SDR.




6.3 Software management process This process covers the complete life cycle of the software project. The main activity is the production of

a software development plan including the life cycle description, activities description, milestones and

outputs, the techniques to be used, and the risks identification.

The management process also includes the organisation and handling of all the joint reviews, the

definition of procedures for the management of the system interface, and the technical budget and

margin management.

Software requirements or detailed design dedicated reviews (i.e. SWRR and DDR) are defined as

anticipation of the PDR and CDR respectively.

6.4 Software requirements and architecture engineering process The software requirements and architecture engineering process consists of

● The elaboration of the technical specification, including the preliminary definition of the ICD

(TS), which is the supplier’s response to the requirements baseline including the interface

requirement specification,

● The architectural design documented in a preliminary SDD

It is the duty of the supplier to involve all stakeholders in the requirement elicitation.

During this process, the result of all significant trade‐offs, feasibility analyses, make‐or‐buy decisions and

supporting technical assessments are documented in a design justification file (DJF).

The software requirements and architecture engineering process is completed by the preliminary design

review (PDR).

To Do before CDR

A process such as ‘Architecture Centric Engineering’ from the SEI is used to define the architecture and improve the quality of the system.

6.5 Software design and implementation process One of the outputs of this process is the detailed design of the software items identified in the software

product tree. It is provided in response to the technical specification, the ICD and the preliminary DDF.

All elements of the software design are documented in the design definition file (DDF). The DDF contains

all the levels of design engineering results, including software code listings.

The rationale for important design choices, and analysis and test data that show that the design meets

all requirements, is added to the DJF by this process. The results of this process are the input to the

critical design review (CDR).

As part of this process, the code, unit testing and integration testing of the software product are also

produced. All elements of the testing activities are documented in the design justification file (DJF).




6.6 Software validation process In this document, the phrase software validation refers to software product testing against both the

technical specification and the requirements baseline. This process is intended to confirm that the

technical specification and the requirements baseline functions and performances are correctly and

completely implemented in the final product. The results of this process are included in the DJF.

This process is established before the PDR to basically produce a software validation plan (”process

implementation”). The plan includes the validation activity with respect to the technical specification

(which is held before the CDR) and the validation activity with respect to the requirements baseline

(which is held before the QR and possibly repeated at AR).

This process can include a test readiness review (TRR) to verify that all test facilities and test cases and

procedures are available before each significant test campaign, and under configuration control.

The results relevant to the validation against the technical specification (TS) are checked at the CDR, and

those against the requirements baseline (RB) are verified at the QR, using the DJF as input.

This process can be executed with varying degrees of independence. The degree of independence can

range from the same person, or a different person in the same organization, to a person in a different

organization, with varying degrees of separation. Where the software validation process and the

software verification process are executed by an organization independent of the supplier, it is called

Independent Software Verification and Validation (ISVV), or Independent Software Validation (ISV) if

only the validation process is independent.

Recommendations

Defects are commonly injected into software at every cycle of the design and development process, and removed in several remedial phases: individual and team evaluations and reviews (for architecture, high level & detailed design and code), compilation and testing (at least unit, integration and system testing phases). The average cost of removal of a defect increases very dramatically when it is found during later phases, hence early detection in reviews is desirable.

To execute this effectively attention must be paid to design comprehensive reviews.

6.7 Software delivery and acceptance process This process prepares the software product for delivery and testing in its operational environment. It is

completed with an acceptance review (AR), with the DJF as input. The acceptance review is a formal

event in which the software product is evaluated in its operational environment. It is carried out after

the software product is transferred to the customer and installed on an operational basis.




6.8 Software verification process The software verification process is intended to confirm that adequate specifications and inputs exist for

every activity and that the outputs of the activities are correct and consistent with the specifications and

inputs.

This process can be concurrent with all the previous processes.

The customer verifies the requirement baseline for the SRR.

For the supplier, this process is established before the PDR to basically produce a software verification

plan (”process implementation”). This process includes all the verification activities of all the expected

outputs of the development activities

The result of this process is included in the DJF and reported at each reviews.

This process can be executed with varying degrees of independence. The degree of independence can

range from the same person, or a different person in the same organization, to a person in a different

organization, with varying degrees of separation. Where the software verification process and the

software validation process are executed by an organization independent of the supplier, it is called

Independent Software Verification and Validation (ISVV).

6.9 Software operation process The operation process can start after completion of the acceptance review of the software. Since

software products form an integrated part of the SDP, the phasing and management of operations are

determined by the overall system requirements and applied to the software products. The operation

process is not directly connected to the overall mission phase E, but is instead determined by the

requirement at system level to operate the software product at a given time.

The organisational entity responsible for the software operation support (SOS entity) ensures that the

software remains operational for the users. Examples of SOS entities and users for different types of

software are typically the following for a software tool e.g. a distributed database with a human machine

interface:

● The user uses the software through the human machine interface;

● The SOS entity administers the database by supporting user’s requests, rebooting or backing up

the system.

6.10 Software maintenance process This process is activated when the software product undergoes any modification to code or associated

documentation as a result of correcting an error, a problem or implementing an improvement or

adaptation.

The maintenance process contains the activities and tasks of the maintainer. The objective is to modify

an existing software product while preserving its integrity. This process includes the migration and

retirement of the software product. The process ends with the retirement of the software product.

The maintainer manages the maintenance process at the project level following the management

process, which is instantiated for software in this process.




7 Agile Software Development For SDP Products The Agile software development methodology consists of a collection of principles and practices that

influence the way teams organise, the tools they use, and the way individual team members interact

with each other. This overview attempts to condense over a decade of research, insights and evolution

into a few pages. The term ”Agile” to refer to a specific collection of software development

methodologies first appeared in the Agile Manifesto, and included some value statements as well as a

set of guiding principles. These values and principles arose from a discussion about the commonalities

found amongst a large number of software development approaches being used at the time. These

different approaches, including XP, Scrum, and others, address issues of software project planning as

well as specific software development and engineering practices.

The main aspects of Agile software development considered here are the specific practices and

principles most relevant to the SDP project specifically and scientific computing more generally. The

principles of Agile include rapid feedback and complete transparency. While not a principle, Agile relies

heavily on automation of testing, builds, and deployments, following the notion that anything done

frequently should be automated. Focusing on the Agile principles helps when applying Agile to different

situations. While scientific computing shares many characteristics with traditional business software

development, there are differences. These differences manifest themselves in variations to some of the

standard practices.

The practices outlined here include the following: automated unit testing for code correctness,

automated functional testing for behavioural correctness and documentation of expected behaviour,

frequent releases to demonstrate progress of development, more granular requirements to facilitate

tracking of real progress, and continuous integration to facilitate collaboration. While these practices,

and the principles that inspire them, do not make up the totality of what constitutes Agile, the set does

include many critical enablers to Agile software development for scientific computing and particularly for

teams that are geographically distributed. In addition to these practices, Agile software teams organize

themselves differently than many traditional teams and adopt a working style that mirrors that often

adopted by multi-disciplinary research teams.

7.1 10.1. The Role of Testing Two aspects of testing, and more specifically automated testing, that are relevant for scientific

computing are unit testing and automated functional testing. Agile developers often practice test driven

development, often abbreviated to TDD [RD05]. There are two very different aspects to TDD:, the tests

that result from TDD and the impact on the design on software by using TDD. Indeed, some argue that a

more accurate name for the practice is Test Driven Design. The basic idea behind TDD is simple; no code

is ever written without a failing automated test. Once the test fails, the developer writes code to make

the test pass. The result is two-fold: a suite of tests that represent the desired behaviour of the code and

code that is generally smaller, simpler, and easier to understand. The suite of tests provides a safety net

to encourage refactoring and more general updating of the code, since the tests tell us when some piece

of code no longer behaves as expected. Unit test coverage is frequently a metric tracked to monitor the

health of a code base.

The second relevant aspect of testing is the use of automated functional acceptance tests. These test

suites are valuable for communicating the expected behaviour of a system as well as speeding up




regression testing as new functionality is added. These tests should focus on the expected systems

behaviour rather than on the implementation of that behaviour.

7.2 Granularity of Requirements and Releases An important agile principle is rapid feedback and transparency. This principle manifests itself in the

practices of frequent releases and small requirements. Frequent releases provide for earlier feedback

from users of systems. However, achieving releases every month or so means that features must be

small enough to be completed in that time. These two practices thus reinforce each other. Generally,

agile teams like the individual feature requests, often called stories, to be small enough to be completed

in at most two days, and the target is often only one day. If a particular feature is too complex to be

developed that quickly, the feature is decomposed into smaller stories. A release is made up of a

collection of stories. Agile development often uses the notion of an iteration, which is simply a short

time box, one or two weeks. At the end of an iteration, the users of the system have the opportunity to

provide feedback to the developer on the newly implemented features. The end users in the case of

scientific computing may be the scientist developing the system or they may be a community of

scientists. Regardless, the value of frequent releases derives from the transparency into real progress. In

Agile development, there is an unambiguous definition of done. Code can never be 80 percent done. It is

either done, meaning it passes all the tests, or it isn’t finished. Since individual stories are small, this all

or nothing status metric isn’t an onerous burden over the course of a software effort.

One aspect of scientific computing might seem at odds with Agile development. Many development

efforts in scientific computing are exploratory in nature, or possibly even pure research. The feasibility of

requirements of this form for scientific computing is a legitimate question. While a full discussion of this

topic is beyond the scope of this paper, the issue must be addressed. One key is to separate the

correctness of the implementation from the validity of the hypothesis being tested with the code. For

example, the requirements for some pattern matching code should focus on whether the pattern

matching code works. The ability for the pattern matching code to detect planets is not something

appropriate for testing in this context.

7.3 Continuous Integration and Continuous Delivery

A critical aspect of Agile, and one that is particularly important for distributed development, is

continuous integration. The basic tenet of continuous integration is that developers should be checking

their code into source control frequently and ensuring the code passes all the tests before proceeding.

This practice isn’t as important if there is only one developer, since there are limited code integration

issues in that case. As the size and geographic distribution of a team grows, however, continuous

integration is crucial. By integrating different aspects of the code frequently, feedback on incompatible

changes comes in a more timely fashion, making the issues much easier to fix. This practice helps keep

different parts of the development team synchronized, even across time zones.

Continuous Delivery takes continuous integration further by focusing on the automation of deployments

and environment provisioning. The goal of continuous delivery is to make deployments boring by making

them reliable and fast. Automated environment provisioning enables the creation of new environments




quickly to support new development teams or test environments. Deployment automation reduces the

opportunities for mistakes in deployments, as well as speeding them up.

The practices that make up continuous delivery will be important to support the complex software

landscape of the SKA.

7.4 Large Scale Commercial Distributed Agile Development Example from ThoughtWorks

The first example is an e-commerce site that serves a primarily European market. The site has

tremendous performance and load requirements and significant integration challenges. The code base is

several hundred thousand lines of Java code spread across hundreds of classes. The team supporting and

enhancing this system is located in two different countries at four different locations. At present, the

team is about 100 strong; at its peak, the team was in excess of 200 people.

The size of both the team and the system necessitates heavy use of automation in testing and builds.

Continuous integration is critical to keep the large numbers of developers working together. Since so

many features are under development at any point in time, the probability of conflicting changes is high.

Continuous integration, including frequent execution of the functional tests helps spot any conflicts

early, allowing for quick resolution.

Given the size of the code, on-boarding of new team members is challenging. The unit and functional

tests provide reliable documentation of the intended behaviour of the code. Written documentation

generally never gets kept up to date. The value of the test suite as documentation is simple; the

documentation is provably accurate since the tests are all passing.

Another critical success factor was the overall program management function. With the number of

different systems involved and the size of the development effort, keeping the different teams working

smoothly was essential. While program management evokes a sense of “heavyweight” processes, Agile

Program Management is essential. Agile and planning are not mutually exclusive.

The second example highlights the way to work with a community of stakeholders. The system is a point

of sale system for a large retail group that had multiple brands. The system needed to serve the needs of

the individual brands while maintaining sufficient consistency for reporting and financial purposes. A

critical role in this effort was the individual tasked with reconciling the needs of the different brands.

This person accepted requirements from the different stakeholders and rationalized them into a

coherent set for the development team. A critical enabler for the success of this role is the relationship

formed with both the stakeholders and with the development team since this person’s job was to push

back on some stakeholders and on the team at times.

With the software complexity of the SKA systems and the geographic and organizational distribution of

the development team, the Agile practices of test automation and continuous integration will be crucial

in allowing the different teams to progress independently and safely. The practices will also support the

efficient on-boarding of team members in the different locations.

The management of the different stakeholders of the SKA software components will also be crucial.

Frequent releases are very helpful in reassuring stakeholders of progress. Strong stakeholder

management will still be critical, but the frequent showcasing of progress will make that task simpler.




7.5 Role of the Scientific Community A major role in an Agile software project is that of the customer. The customer, or end user, interacts

frequently with the development team, specifies the requirement of the systems in the form of stories

with acceptance criteria, and participates in the frequent showcases of the software, providing feedback

on the implemented features. The stakeholders for the SKA, however, are too numerous to interact with

in this way. The SKA governing bodies will be interested in tracking progress against commitment. The

scientific community will be interested in understanding the capabilities that SKA will make available to

them. The consortia working on the new hardware components necessary for the SKA will need to know

how and when they will integrate with the software components. While there won’t be a specific

customer in this context, the engagement between the different stakeholders and the SKA software

development consortia should still follow the customer model. Showcases, for example, could be made

available to the broader science community through video and webcasts, providing opportunities for

feedback to the software teams. The existence of the automated tests will simplify eventual community

contributions to the SKA suite of analysis tools, as another example.

7.6 Testing Frameworks Testing, and in particular automated testing, plays a prominent role in Agile Software Development. As

such, a number of tools and frameworks have been developed, both proprietary and open source, to

support this level of testing. The arguably best known is jUnit, the unit testing tool that supports Java

applications. The analogue in the C#/.NET world is nUnit. These open source frameworks provide the

mechanism to support unit testing, including integration with the development environments (IDEs).

These tools provide mechanisms to support expression of a variety of assertions, failure of which fails

the build.

Moving up the chain, we come to tools to support automated functional and acceptance testing. Here,

the implementation mechanism is again important. There are web testing frameworks like Selenium,

Sahi, and White. There are frameworks like jMeter that tend to be used more for load and performance

testing. There are tools like Go and Cucumber that allow the expression of tests in a domain specific

language making it easier to involve users in the test creation. Tools at this level can also drive tests of

integrated systems by supplying the “triggering event” that propagates through the various systems.

Checking the success of such a test is more difficult, since the response is likely to be delayed in time.

Tests of this nature will be a major testing component for the SKA.

Specifying the test and expected results is only one aspect of automated testing. A critical component is

the provisioning of test environments. Access to test instances of specialized hardware is a serious issue

for integration testing. Virtualization technology can help some, but it doesn’t address issues like

performance testing in situations like scientific computing where compute time is a significant factor.

One approach to addressing this issue is to utilize stub and mock systems that can mimic the behaviour

of systems that are hard to access in test environments. In general, stub systems simply replay canned

responses to requests, where mocks tend to have more intelligence built into them. A critical issue with

using this approach is to ensure they do not get out of synchronisation with the systems they are

mimicking. To determine the continued validity of the mocks and stubs, we generally have automated

tests that run against the real systems to ensure the behaviour hasn’t changed. This approach to

contract testing is also useful in situations where cooperating systems are being developed by disparate




teams. Each team provides the other with a test suite that shows the assumptions being made. A failure

in one of these tests triggers a conversation between the development teams to address the issue.

Large scale distributed development requires significant investment in testing, to ensure the various

parts of the system work together as needed. The tools that have matured significantly due to the

emphasis on testing from the Agile development community will play a critical role in the success of the

SKA software components.

7.7 Support and Maintenance While the majority of attention to a software system focuses on the initial development of the system,

the support and maintenance costs and effort frequently dwarf the initial investment. As such, planning

for the support phase is critical. Aspects to consider for supporting a system that was developed using

Agile methods include the role of tests, metrics and continuous delivery.

As described previously, tests contribute to the on-boarding of new team members by providing

accurate documentation about both, the behaviour of the system and the expectations about how the

various parts of the system fit together. Over time, these tests are critical in maintaining the

organizational knowledge of the system, since people move on and forget the details.

Software internal quality metrics are increasingly being recognized as important for understanding and

improving the maintainability of a system. By incorporating these metrics into the build process, the

system quality over time can be monitored and corrective action can be taken to address creeping

quality issues.

Continuous delivery provides the foundation for supporting the continued upgrading and release of a

system. Once in a maintenance phase, there might seem to be less pressure to release quickly. However,

the ability to quickly address a critical issue is no less important when system is in maintenance mode.

The staffing approach to support varies depending on ownership of the code. Open source communities

often provide the support resources for systems; these resources include support forums where people

can get help, in addition to the more obvious bug fixing or feature requests. Support systems generally

require a mechanism for getting questions answered, and some way of recording defects and feature

requests. The health of a community is often assessed on how active its forum is. The SKA software

consortia will need to provide a similar set of capabilities.

Agile methods apply equally well during the support phase, although the level of interaction may be

different. The same testing, build, and release approaches should continue through the support phase of

a system. The same attention to test maintenance and code quality must also continue or the software

asset will atrophy. Systems become out-dated even if there is nothing wrong with them, as new

technologies and new approaches enter into the industry. Thus, on-going maintenance of large software

systems needs to be addressed, even though it is often given far less attention than it deserves.

PDR06PreliminarySoftwareEngineeringPlan(3)EchoSign Document History February 10, 2015

Created: February 09, 2015

By: Verity Allan ([email protected])

Status: SIGNED

Transaction ID: XJE6UN8W2M6SR3I

“PDR06PreliminarySoftwareEngineeringPlan (3)” HistoryDocument created by Verity Allan ([email protected])February 09, 2015 - 10:58 AM GMT - IP address: 131.111.185.15

Document emailed to Kevin Vinsen ([email protected]) for signatureFebruary 09, 2015 - 10:59 AM GMT

Document viewed by Kevin Vinsen ([email protected])February 10, 2015 - 2:18 AM GMT - IP address: 180.149.250.159

Document e-signed by Kevin Vinsen ([email protected])Signature Date: February 10, 2015 - 2:37 AM GMT - Time Source: server - IP address: 180.149.250.159

Document emailed to Paul Alexander ([email protected]) for signatureFebruary 10, 2015 - 2:37 AM GMT

Document viewed by Paul Alexander ([email protected])February 10, 2015 - 8:57 AM GMT - IP address: 131.111.185.15

Document e-signed by Paul Alexander ([email protected])Signature Date: February 10, 2015 - 8:57 AM GMT - Time Source: server - IP address: 131.111.185.15

Signed document emailed to Kevin Vinsen ([email protected]), Paul Alexander ([email protected]) andVerity Allan ([email protected])February 10, 2015 - 8:57 AM GMT

PDR.06 PRELIMINARY SOFTWARE ENGINEERING PLANbroekema/papers/SDP-PDR... · To do before CDR Check...

Documents

Transcript of PDR.06 PRELIMINARY SOFTWARE ENGINEERING PLANbroekema/papers/SDP-PDR... · To do before CDR Check...