Agile methods and safety critical software - Peter Gardner

Peter Gardner

Technical Consultant

AGILE METHODS AND SAFETY-CRITICAL SOFTWARE

Are They Compatible ?

SILVER-ATENA’s Business

• Software development principally in the rail and

avionics sectors

• Consultancy services

• Provides skilled software engineering staff to work

on clients’ premises

• Full system development capability through other

partner companies in the same group

• Group offices in Malmesbury, Bangalore, Madrid

and Munich

SILVER-ATENA Agile Club

• Established August 2009

• Purpose : investigate the use of Agile

methods in safety-critical software

development

• “Client” is the Open-DO wider community

SILVER-ATENA Agile Club

• Average attendance 12 people, internal and external

• Total attendance 32 : consultants, software engineers, project managers, quality department

• Videos, Powerpoint, discussion

What the Presentation is About

• (1) Can Agile methods be applied to safety-critical software and the software still be rigorously built and meet certification criteria?

[compatibility].

• (2) What evidence is there for the benefits of Agile methods, especially as regards safety-critical software ?

[cost, quality, time etc]

What the Presentation is About

• For (1), this paper will look at, in particular, DO-178B and EN 50128, for the aviation and rail sectors respectively

(level A/B, SIL 3/4)

• For (2), this paper will summarise the results of a websearch looking for evidence of the benefits of Agile

• I will also look at some opinions from people who work in the sector

Categories of Source Papers

• Papers on Agile methods and non-safety-

critical software (10)

• Papers on Agile methods and safety-critical

software (7)

Different Variants of Agile

• XP

• Adaptive Software Development

• Crystal (family)

• Dynamic Systems Development Method

• Scrum

• Agile Unified Process

• Feature-Driven Development

Features of Agile Methods

• Short release cycles

• Incremental requirements development

• Customer presence in project

• Continuous integration

• Customer determines functions in each release cycle

• Generally for small teams (up to about 12)

• More difficult to apply to larger teams

• Adapt for each project

• Multi-skilled team members (possibly)

• Any team member can change anything


• Pair-programming (XP)• Test-driven development• Retrospectives• Larger teams should have shorter cycles (!)• Refactoring• Less emphasis on design and documentation• Pair-programming (XP)• Self-organising teams• Project lasts as long as customer gives high-value requirements• Departments as service providers

Yes, mostly

• Pass on 15 out of 20

• Maybe on 3 out of 20

• No, but not serious, for 1 out of 20 : anyone can change anything

• But there is a real problem with one item : reduced emphasis on design and documentation


• Short release cycles

• Incremental requirements development

• Customer presence in project

• Continuous integration

• Customer determines functions in each release cycle

• Generally for small teams (up to about 12)

• More difficult to apply to larger teams

• Adapt for each project

• Multi-skilled team members (possibly)

• Any team member can change anything

Features of Agile Methods• Pair-programming (XP)• Test-driven development• Retrospectives• Larger teams should have shorter cycles• Refactoring• Less emphasis on design and documentation• Change company structure (ideally) (departmental to project structure)• Self-organising teams• Project lasts as long as customer gives high-value requirements• Departments as service providers

DO-178B Outputs(1)

• Plan for Software Aspects of Certification

• Software Development Plan

• Software Verification Plan

• Software Configuration Management Plan

• Software Quality Assurance Plan

• Software Requirements Standards

• Software Design Standards

• Software Code Standards

• Software Requirements Data

DO-178B Outputs(2)

• Software Design Description

• Source Code

• Executable Object Code

• Software Verification Cases and Procedures

• Software Verification Results

• Problem Reports

• Software Configuration Management Records

• Software Quality Assurance Records

• Software Accomplishment Summary

EN 50128 Outputs (part 1)

• System safety plan

• Software Quality Assurance Plan

• Software Configuration Management Plan

• Software Verification Plan

• Software Integration Test Plan

• Software /Hardware Integration Test Plan

• Software Validation Plan

• Software Maintenance Plan

• Data Preparation Plan

• Data Test Plan

EN 50128 Outputs (part 2)

• Software Requirements Specification

• System Safety Requirements Specification

• Software Requirements Test Specification

• Software Architecture Specification

• Software Design Specification

• Software Module Design Specification

• Software Module Test Specification

• Software Source Code

EN 50128 Outputs (part 3)• Software Module Test Report• Software Integration Test Report• Software Requirements Verification Report• Software Module Verification Report• Software Architecture and Design Verification Report• Software Source Code Verification Report• Software/Hardware Integration Test Report• Software Validation Report• Software Assessment Report• Software Change Records

Documents and Contents

• Its not just a case of producing documentation

• Much more importantly, one has to produce the

contents of the documents i.e software

development artefacts

• For example, architectural design,

low-level design, test cases etc

Traceability

• Both DO-178B and EN 50128 require traceability through the lifecycle, yet :

• Agile methods never even consider traceability

Independence of Roles

• DO-178B and EN 50128 also require (for higher levels) some of the roles to be independent eg. implementor and tester

• This constraint also has to be factored into Agile methods

Views on Design, Documentation, Traceability

• Stojanovic et. al. : “[According to] their proponents software code is the main deliverable, while the role of system analysis, design and documentation in software development and maintenance is de-emphasised and to some extent ignored”.

• Chenu : “without any adaptation, XP’s set of by-the-book practices does not provide the formalism and the proofs required for certification inspections….documentation must be considered as part of the product”

• Chenu : “we’ve added the continuous traceability practice”

• ITEA Deliverable D.1 :“Agile development methods and practices as applied for classical software development cannot identically be copied for embedded real-time software development”.

• ITEA Deliverable D.1 : in large, complex systems “a predefined architecture is needed…refactoring is not enough, a design is needed”.

• ITEA Deliverable D.1 : in safety-critical systems“emphasis is placed [on] requirements traceability and thus extra documentation is required”.


• ITEA Deliverable D.2.12 : “developers of mission-critical airborne software are heavily constrained by the RTCA DO-178B regulations. These regulations impose strict rules regarding traceability and documentation that make it extremely hard to employ an iterative software development process” [did they really mean this?]• ITEA Deliverable D.2.12 : “the mission-critical nature of this software has lead to stringent procedures and plans that could specifically exclude the use of Agile methods”• Wils and Van Baelen : “Agility potential is inherently lower for DO-178B compliant projects” • Thomas : “beware Agile methods…dangerous where…the system is safety-critical” [says use Z and SPARK etc]• Ambler : would be “leery of applying Agile modelling to life-critical systems”.• Somerville : “critical system engineering needs Agile approaches to development”• Somerville : “XP includes a set of good practices that have the potential to contribute to critical systems engineering” but “some of these need to change and new practices need to be included in the XP process”


• Van Schooenderwoert : “Agile doesn’t treat safety-critical software differently; all is maximum quality”

• Turk et.al. : “formal specification, rigorous test coverage, and other formal analysis and evaluation techniques included in software engineering approaches provide better, but also more expensive, mechanisms to tackle the development of safety- or business-critical software”

• Rakitin : “it seems to me that this is nothing more than an attempt to legitimise hacker behaviour”

Empirical Evidence(1)

• We have seen some opinions

• Is there any actual data ?


• My web search found two papers (which were themselves surveys) :

• Abrahamsson [7] found 36 papers with some evidence

• ITEA (Information Technology for European Advancement) project deliverable D.1 contained three studies


• Abrahamsson : online survey, 36 papers with empirical evidence :

- a reduction in release time of 20% to 75%

- a pre-release reduction in defects of 40% to 65%

- a post-release reduction in defects of 30% to 40%

- a productivity improvement of 15% to 70%

Not known if any of these were safety-critical projects


• ITEA (1) “has the adoption of Agile processes altered the quality of applications?”

significantly better 36%, better 53%, unchanged 11%, somewhat worse 1%

(2) “has the adoption of Agile processes altered the cost of development?”

much less expensive 5%, less expensive 44%, unchanged 46%, more expensive 5%

(3) “has the adoption of Agile processes altered the level of business satisfaction with the software ?”

significantly better 26%, better 57%, unchanged 16%, somewhat worse 0%, much worse 1%.

(4) “what proportion of projects are appropriate for Agile processes?”

all 16%, most 50%, half 22%, some 7%, none 5%

• Only two out of seven specific projects reported in the literature were safety-critical projects

Summary

• I have examined whether Agile methods are compatible with the production of safety-critical software

• I have presented evidence from the literature of the benefits of Agile methods, particularly as regards safety-critical software

Conclusions• Agile methods, in general, need to be adapted for use in the

area of safety-critical software development

• Certification requirements mean that necessary

documentation must be incorporated into any

Agile process selected.

• Traceability and role independence must also be considered

• The logical extension of Agile methods to the safety-critical

world requires continuous certification and a colocated

certification authority

• Release cycles are likely to be longer in safety-critical projects but

there is no reason not to move towards the concept of frequent

release cycles with an onsite customer

• We need to gather more evidence, using reliable, standard metrics

• Silver-Atena’s Agile club would be willing to assist in this work

References (non-safety-critical)

[1] Modeling and Architectural Design in Agile Development Methodologies, Stojanovic, Dahanayake, Sol http://www.emmsad.org/2003/Final%20Copy/27.pdf[2] Agile in Embedded Software Development : State-of the-Art Review in Literature and Practice ITEA Deliverable D.1 http://www.agile-itea.org/public/deliverables.php[3] A Practical Guide to Seven Agile Methodologies, part 1, Rod Coffin and Derek Lane http://www.devx.com/architect/Article/32761/1954[4] A Practical Guide to Seven Agile Methodologies, part 2, Rod Coffin and Derek Lane http://www.devx.com/architect/Article/32836/1954[5] Empirical Findings in Agile Methods, Lindvall et. al. http://www.cs.umd.edu/~mvz/pub/agile.pdf[6] The Agile Unified Process, Scott Ambler http://www.ambysoft.com/unifiedprocess/agileUP.html[7] Pealing the Hype into Pieces : What do we Really Know About Agile in Research and Practice ? Pekka Abrahamsson,VTT Technical Research Centre of Finland http://www.agile-itea.org/public/papers/OLIO_abrahamsson.pdf[8] Limitations of Agile Software Processes, Turk, France, Rump http://www.agilealliance.org/system/article/file/1096/file.pdf[9] The Cockburn Scale http://en.wikipedia.org/wiki/Cockburn_Scale[10] Manifesto Elicits Cynicism, Steven Rakitin Computer, December 2001 http://sunset.usc.edu/events/2002/arr/letters.pdf

References (safety-critical)

[11] Agility and Lean for Avionics, Emmanuel Chenu

http://manu40k.free.fr/AgilityAndLeanForAvionics1.pdf

[12] ITEA Deliverable D.2.12 Towards An Agile Avionics Process, Wils and Van Baelen

http://www.agile-itea.org/public/deliverables/ITEA-AGILE-D2.12_v1.0.pdf

[13] Agility in the Avionics World, Wils and Van Baelen

https://lirias.kuleuven.be/bitstream/123456789/133945/1/2006-Sofia1.pdf

[14] Critical Software : The Need For A Radical Solution, Martyn Thomas

http://www.thomas-associates.co.uk/Critical%20Software.pdf

[15] Get Ready for Agile Methods, With Care, Barry Boehm, IEEE Computer 2002

http://www2.umassd.edu/swpi/xp/papers/r1064.pdf

[16] Extreme Programming for Critical Systems, Ian Sommerville

http://www.cs.st-andrews.ac.uk/~ifs/Talks/XPForCritSys.pdf

[17] Safety-Critical Applications Built via Agile Discipline, Nancy van Schooenderwoert

http://www.boston-spin.org/slides/boston_spin_slides_2008_09.pdf

Thank you for your attention !

Agile methods and safety critical software - Peter Gardner

Technology

Transcript of Agile methods and safety critical software - Peter Gardner