Environment Support for Improving Software Development...

Int J Software Informatics, Volume 5, Issue 3 (2011), pp. 0–0 E-mail: [email protected]

International Journal of Software and Informatics, ISSN 1673-7288 http://www.ijsi.org

c©2011 by ISCAS. All rights reserved. Tel: +86-10-62661040

Environment Support for Improving Software

Development Processes: A Vision Influenced by the

Work of Barry W. Boehm

Lori A. Clarke

(Department of Computer Sciene, University of Massachusetts,

Amherst, Massachusetts 01003, USA)

Abstract Throughout his career, Barry Boehm has advocated the importance of un-

derstanding software development processes, measuring their performance, and using those

measurements to guide the development of improved process models. In this paper, we de-

scribe PIE, a Process Improvement Environment, which supports that vision. PIE supports

the definition of process models that can be analyzed and executed. The analysis is used to

detect errors and vulnerabilities in the process models. Validated process models can then

be simulated to detect inefficiencies and bottlenecks. Future work includes executing these

process models, monitoring their performance, and then using that information to drive

further process improvements.

Key words: software development processes; process improvement; process modeling;

process analysis; requirement properties; model checking; fault-tree analysis; failure modes

and effects analysis; human-intensive systems

Clarke LA. Environment support for improving software development processes: A

vision influenced by the work of Barry W Boehm. Int J Software Informatics, Vol.5, No.3

(2011): 000-000. http://www.ijsi.org/1673-7288/5/i95.htm

1 Introduction

This paper describes environment support for improving software developmentprocesses. Much of this work can be traced back to and has built upon contribu-tions made by Barry Boehm over the course of his career. Even in his early papers,Boehm argued for creating a model of the process under which software would bedeveloped (e.g., [2]). He advocated that the process model to be followed in devel-oping a software system was one of the first artifacts that needed to be explicitlyarticulated and agreed upon by all the stakeholders. Much of Boehm’s career hasbeen devoted to evaluating these process models. From the early waterfall model[25],to the various military standards that encoded this model in doctrine (e.g., [14]), tothe more free-wheeling agile methods (e.g., [11, 19, 26]), and to the more thought-ful, risk-based approaches that Boehm has proposed, particularly the Spiral Model[4]

* This material is based upon work supported by the National Science Foundation under Awards

CCF-0820198, CCF-0905530 and IIS-0705772. Any opinions, findings, and conclusions or recom-

mendations expressed in this publication are those of the author and do not necessarily reflect theviews of the NSF.

Paper received on 2011-03-07; Revised paper received on 2011-03-28; Paper accepted on 2011-03-29;

Published online 2011-00-00.

2 International Journal of Software and Informatics, Volume 5, Issue 3 (2011)

and the Incremental Commitment Model[6], he has undertaken careful assessments ofthese processes and, based on his experimental findings, reported on their strengthsand weaknesses. Working with other pioneers (e.g., [5]), Boehm has argued for a sci-entific basis for evaluating and improving software development processes, and thussoftware development.

In this paper, we describe PIE, a Process Improvement Environment, being devel-oped at the University of Massachusetts. This environment provides capabilities formodeling processes, for analyzing these processes for errors and vulnerabilities, andfor providing execution and simulation support. We believe that this approach is astep toward supporting the vision that Boehm has consistently advocated throughouthis career for explicitly understanding the processes that are being applied, for mea-suring their actual performance, and for using that information to develop improvedprocess models.

2. Overview of the PIE

The Laboratory of Advanced Software Engineering (LASER) has been develop-ing and evaluating technology to support the continuous improvement of processes.Specifically we have developed a modeling language, called Little-JIL[7], and a suite ofanalysis tools for evaluating Little-JIL process definitions. We have applied these tech-nologies to various domains including software development environments[20], govern-ment processes, such as the conduct of elections[28] and on-line dispute resolution[17],and several life-critical medical processes, such as the administration of chemotherapy[9]

and the transfusion of blood[16].Creating an accurate and detailed process model is a labor-intensive activity

that demands considerable time and effort. Our work has demonstrated, however,that this investment can be leveraged by applying a range of analyzers to the processmodels to uncover an assortment of errors, vulnerabilities, and inefficiencies. Figure1 depicts how these analyzers are applied in PIE. The leftmost column in this figureshows the tools used to create a process definition (the Little-JIL editor) and to createproperties (the PROPEL property elicitor[10;29]) that specify the requirements thatare suppose to be upheld by this process definition. The second column shows theresulting process definition and property specifications and other information neededto support the analysis capabilities, which are shown in the third column. The fourthcolumn shows the outputs from these analysis tools.

Figure 1. Architecture of the process improvement environment

Lori A. Clarke: Environment support for improving software development processes ... 3

2.1 Little-JIL process definitions

The Little-JIL process language has been shown to be effective in encoding thedefinition of processes that must handle both normative and exceptional situations,concurrent activities, and the coordination of different types of agents. These agentsinclude human agents with different roles (e.g., developers, testers, managers), dif-ferent kinds of software systems (e.g., compilers, testing frameworks) and hardwaredevices (e.g., PDAs). Little-JIL enables specification of a wide range of process de-tails, while still supporting the flexibility and freedom of choice that human agentsexpect. In addition, it has well-defined semantics so that the resulting process defi-nitions can be rigorously analyzed. Applying Little-JIL to a variety of processes hasproved to be challenging, and one of the lessons we have learned is that even so-calledsimple processes can have considerable complexity. Our experiences have led to im-provements to the Little-JIL language, most recently to improvements in the handlingof exceptions[18] and to resource modeling[35,36].

Here we show a small example to give a sense of the Little-JIL process definitionlanguage. A coordination diagram in Little-JIL specifies the hierarchy of tasks, orsteps, in a process. For example, Figure 2 shows a Little-JIL process definition for aSprint, a subprocess of the Scrum software development process[11,26]. In Little-JIL,the steps are shown as a hierarchical decomposition. The steps to be done are eachshown as black bars, with each step’s immediate substeps shown underneath. Thebadge, on the right hand side of the bar, indicates the order in which the substepsare to be done. The Sprint subprocess is the Scrum activity during which actualdevelopment work gets done. To be more precise, the Sprint process consists of thirtyconsecutive performances of the Daily Sprint subprocess (as indicated by the “30”annotation on the edge connecting the Sprint parent step to its Daily Sprint childstep and by the right arrow step kind badge in the Sprint step). As indicated by theequal sign step kind badge in the Daily Sprint step, this subprocess is carried outas the parallel performance of its three substeps, Daily Scrum, Checked Work, andRevise Sprint Backlog. Both the Daily Scrum and the Revise Sprint Backlog stepsrequire read and update access to the sprint backlog. The sprint backlog channelprovides the concurrency controls needed to coordinates the accesses for these twosteps.

The Daily Scrum step is a 15-minute (as indicated by the specification of thisdeadline by means of the diamond annotation) progress meeting during which theteam meets to assess their progress. Note that the sprint backlog artifact is passedin as a parameter, and then also passed out of this step, after which it is written tothe sprint backlog channel so that it is made available to the Revise Sprint Backlogstep, which may be executing in parallel.

In addition to the execution of the Daily Scrum step, there are multiple per-formances of the Checked Work step (note the + sign on the edge connecting theChecked Work step to its parent). Each instance of the Checked Work step producesa new version of the product artifact, presumably being comprised of more completedwork items after the execution of this step. The agent responsible for performing thisstep is the team. Concurrently with the performances of the Checked Work step,there may be multiple performances of the Revise Sprint Backlog step (as indicated


by the * on the edge connecting this step to its parent). The agent for this step is alsothe team, and the effect of a performance of this step is to update the sprint backlogto reflect the addition or removal of tasks in the backlog.

Any of the leaf steps in a coordination diagram can be further decomposed asa coordination diagram. Figure 3 shows the coordination diagram for the leaf stepChecked Work shown in Fig.2. Checked Work is comprised of the sequential executionof its two child steps, Work and Integrate, each of which would be further elaborated,but are not shown here. Note that the step Integrate has a post requisite, as shown bythe up pointing triangle on the left hand side of the step. If this step’s post requisiteis not satisfied, then the exception Build Failed is raised (not actually shown). Theclosest handler is found by searching up the tree from where the exception is thrown.In this case, the closest exception handler for this type of exception is defined at theimmediate parent, Checked Work step. The handler for this exception is defined bythe step Rework, the step attached to the edge emanating from the X (for exception)on the right hand side of the step. The step Rework would undoubtedly be furtherelaborated as a Little-JIL diagram and might throw and handle additional exceptionsitself.

Figure 2. Little-JIL definition of the sprint subprocess

Although other process languages could be used in PIE instead of Little-JIL,Little-JIL has several advantages. Because its main abstraction is task decomposi-tion, it is easier than in a data flow representation to specify exception handling witha number of different termination semantics. In addition to rich semantics for ex-ception handling, it provides good concurrency control. Importantly, it also supportshierarchical decomposition and abstraction. Any of the steps, such as Rework, can beinvoked from another step, with the appropriate context information being providedthrough parameter passing. Of particular significance is that the language has welldefined semantics and thus processes written in the language can be the subjects ofrigorous analysis. We next discuss some of the analysis capabilities that are currently


supported in PIE.

Figure 3. Elaboration of the checked work step shown in Figure 2

2.2 Process model narrator

The description of the Scrum process given in Section 2.1 is based on the di-agrammatic representation shown in Figs.2 and 3. Alternatively, PIE can take thediagrammatic representation and automatically generate a hyperlinked textual repre-sentation. We have found that non-computer scientists often prefer a natural languagedescription. In fact, even computer scientists have found the textual description tobe a useful alternative representation that helps them find errors in the encodingof the process in Little-JIL. Figure 4 shows the generated textual description of theLittle-JIL process definition shown in Fig.2. Unlike Electronic Process Guides, whichtend to provide a broader range of alternative representations but require considerablemanual input, the Little-JIL narrator automatically generates an English descriptionand addresses many of the limitations pointed out by Boehm and his co-authors intheir evaluations of EPGs[23].

The narrative generation is done using templates for each of the Little-JIL con-structs. There are a few customization rules that allow for the use of synonyms andcontrol over the level of verbosity. It would be interesting indeed to see if descriptionscould be easily generated for other languages in addition to English.

It is also interesting to note that computer scientists went to diagrammatic rep-resentations because natural language descriptions are usually imprecise, incomplete,and ambiguous. The generated textual descriptions, however, are as precise, completeand unambiguous as the diagrammatic representation from which they are derived.They are, however, verbose and not very “natural”, although often easier for non-computer scientists to comprehend than the original diagrams.

2.3 Validating process definitions

Our work on developing process definitions has clearly indicated the importanceof validating those processes. There are three main thrusts here.

• If a process is important and complex enough to warrant being modeled, thenthose models need to be scrutinized for errors and weaknesses before using them.


Such analysis might also uncover ways in which the model deviates from the actualdesired process. All of these discoveries should lead to improved process models.

• Applying analysis approaches to the carefully validated models will help detecterrors or weaknesses in the real processes. This is obviously an important goal thatcould reduce errors and improve efficiency.

• Analyses can be used to carefully vet proposed process improvements to assurethat they do not introduce new errors or inconsistencies. Detecting problems beforethey are actually introduced could reduce errors and avoid inefficiencies. Again, ifprocesses are complex enough to warrant careful modeling, then the implications ofchanges to the process may not be obvious in all their manifestations.

Figure 4. Generated hypertext representation of the Little-JIL diagram shown in Fig.2

As shown in Fig.1, PIE currently supports using model checking to verify safetyproperties. The properties are represented using PROPEL, a tool to support bothdomain experts and computer scientists in creating mathematically precise specifi-cations of desired properties. PROPEL allows domain experts to work with naturallanguage, while assisting the specifiers in accurately capturing the often subtle detailsof such specifications. Specifying the desired properties of complex domains, how-ever, made us aware of the need to extend PROPEL with support for exceptional


conditions[22].After a property is specified, the FLAVERS model checking system[15] determines

if there are any possible process executions that could violate the specified property.If so, a path through the process that causes the violation is shown. To mitigate thewell-known state explosion problem associated with model checking, we have devel-oped a number of effective optimization techniques[30−32]. Providing process languageconstructs that give the human agent considerable flexibility tend to increase the sizeof the model, especially when combined with concurrency and complex exceptionhandling. On the other hand, process models tend to focus on coordination and theflow of control, so many of the problems associated with aliasing and unknown valuesdo not arise. Thus, most of the counter example paths that are found are executableand correspond to real problems, eliminating the abundance of false positives thatusually occur with model checking.

To complement property verification, fault tree analysis (FTA) and failure modeand effects analysis (FMEA) techniques are applied. FTA[33] and FMEA[13] are wellknown safety analysis approaches for identifying process vulnerabilities due to in-correct performance of process activities. In contrast, model checking assumes thesteps are done correctly but looks for violations of temporal properties. For a userspecified hazard, such as an incorrect product being produced by a Scrum, our FTAtool[8] automatically derives the fault tree from the Little-JIL definition and thendetermines the minimal cut sets, the minimal combinations of inaccurately appliedsteps that could cause the hazard to occur. The fault tree for this hazard is shownin Fig.5 for the Sprint process definition given in Figs. 2 and 3. Conversely FMEAcan be automatically applied to the Little-JIL process definition[34] to show how afaulty step in the process could propagate inaccurate results to other process steps,providing insight about possible hazards that should be considered for subsequentFTA analysis.

It usually takes considerable human effort to create a single fault tree or FMEAtable, but with our approach considerable care is taken to create the process definitionthat is then used to automatically create FMEA tables or fault trees for a number ofpotential hazards.

As noted, the safety analysis and model checking approaches complement eachother. The safety representations derived from the process model would be of littlevalue if the process model was not carefully verified using model checking or othervalidation techniques. The minimum cut sets derived from the fault tree, however,describe combinations of events that could lead to a hazard and, thus, could subse-quently be the basis for additional properties to be verified using model checking.

2.4 Discrete event simulation

The model checking and the safety analyses described above focus on errors andvulnerabilities that can be detected statically, without the need for execution. Dis-crete event simulation, on the other hand, is an analysis approach that uses dynamicexecution state to study a complementary range of issues. The Little-JIL simulator,JSim, takes as input a Little-JIL process definition and specifications of how to sim-ulate the behaviors of the agents as well as a model of the resources[24,35,36] We haveused JSim to study how different resource mixes and allocation strategies affect pro-


cess flow. Our experience, although preliminary, suggests that modeling processes andtheir required resources in detail seems to improve the accuracy of the simulations.

Figure 5. Fault tree automatically generated for the hazard in which an incorrect product

is provided from the scrum in the sprint defined in Figs.2 and 3

3 Conclusion

We have observed that the very act of trying to model a process or its propertiesleads to improved understanding of that process, to discovering process defects, andto identifying possible process improvements. The model checking that we appliedsuccessfully found important violations. Even when provided with counterexampletraces, however, the modifications initially proposed by domain experts sometimesdid not fix the problem or introduced new errors. Similarly, our safety analysis toolsuncovered previously unknown errors in these processes. Even when the errors were inthe introduced artifacts (i.e., the process model or property), finding these errors im-proved the accuracy of these artifacts and the subsequent analyses based on them. Wethink it is particularly important to emphasize that this investment enables proposedprocess modifications to be carefully vetted before being introduced.

In our recent work, we are moving to an environment where the validated process


definitions are executed. We plan to develop and evaluate capabilities that supportprocess execution, monitoring, and guidance. Moreover, context information gatheredduring monitoring will be used to inform analysis capabilities, while the analysiscapabilities can provide valuable insight that will enable the guidance technologiesto warn of impending hazards and propose compensating actions. Thus, we planto move beyond static modeling and analysis techniques, to an environment thatsynergistically uses static analysis results to provide process execution guidance whilegathering operational information that can be used to improve the static analysiscapabilities.

In many respects, the work suggested here is consistent with the classical notionof continuous process improvement introduced by Shewhart[27], effectively applied byDeming[12], and strongly advocated for software development by Boehm (e.g., [1; 3-6]). The essence of this approach is to capture the process to be improved, compareits characteristics to those that are desired, identify weaknesses and shortcomings,propose and evaluate improvements, and then incorporate those improvements inthe process to complete the improvement cycle and form the basis for a subsequentimprovement cycle. This cycle relies essentially on the ability to understand theprocess, understand its desired properties, and analyze the ways in which the processdoes or does not adhere to those properties. Typically, these analyses have beenobtained informally. Recent research, including our own, has shown that processesand properties can be defined with precise notations and evaluated for various kindsof consistency using automated reasoning approaches. It is this rigorous approachfor defining properties and properties, along with powerful reasoning techniques, thatwe now propose to apply to processes. Preliminary work applying this approachto software engineering processes, particularly Scrum[20] and the Spiral Model, havedemonstrated some of the insights that can be obtained.

Acknowledgements

The work reported here has been done in close collaboration with Leon J. Os-terweil and George S. Avrunin. A number of former and current students have con-tributed to this project including Aaron Cass, Bin Chen, Stefan Christov, JamiesonCobleigh, Rachel Cobleigh, Heather Conboy, Matt Dwyer, Eric McCall, Gleb Nau-movich, Huong Phan, M.S. Raunak, and Jianbin Tan, as well as staff programmer,Alexander Wise.

References

[1] Boehm BW. Software and its impact: A qualitative assessment. Datamation, 1973. 48–59.

[2] Boehm BW. Software design and structuring. In: Horowitz E, ed. Practical Strategies for

Developing Large Software Systems. Addison-Wesley, 1975. 103–128.

[3] Boehm BW. A spiral model of software development and enhancement. Second International

Workshop on the Software Process and Software Environments. 1985. 22–42.

[4] Boehm BW. A spiral model of software development and enhancement. Computer, 1988, 21(5):

61–72.

[5] Boehm BW, Basili VR. Gaining intellectual control of software development. Computer, 2000,

33(5): 27–33.

[6] Boehm BW, Lane JA. Using the incremental commitment model to integrate system acquisition,

systems engineering, and software engineering. CROSSTALK The Journal of Defense Software


Engineering, 2007, Vol.(October): 4–9.

[7] Cass AG, Lerner BS, McCall EK, Osterweil LJ, Sutton Jr. SM, et al. Little-Jil/Juliette: A pro-

cess definition language and interpreter. Demonstration Paper. 22nd International Conference

on Software Engineering. Limerick, Ireland, 2000. 754–758.

[8] Chen B, Avrunin GS, Clarke LA, Osterweil LJ. Automatic fault tree derivation from Little-Jil

process definitions. Software Process Workshop and Process Simulation Workshop. Shanghai,

China, 2006. 150–158.

[9] Christov S, Chen B, Avrunin GS, Clarke LA, Osterweil LJ, et al. Formally defining medi-

cal processes. Methods of Information in Medicine. Special Topic on Model-Based Design of

Trustworthy Health Information Systems, 2008, 47(5): 392–398.

[10] Cobleigh R L, Avrunin GS, Clarke LA. User guidance for creating precise and accessible property

specifications. 14th SIGSOFT International Symposium on Foundations of Software Engineer-

ing. Portland, OR, 2006. 208–218.

[11] Cohn M. Succeeding with Agile: Software Development Using Scrum. Pearson Education, Inc.,

Boston, MA, 2010.

[12] Deming WE. Out of the Crisis. MIT Press, Cambridge, 1982.

[13] DOD. Procedures for Performing a Failure Mode, Effects and Criticality Analysis. Mil-Std-1629,

Department of Defense, Washington D.C., 1980.

[14] DOD. Dod-Std-2167a, Military Standard: Defense System Software Development, 1988.

[15] Dwyer MB, Clarke LA, Cobleigh JM, Naumovich G. Flow analysis for verifying properties of

concurrent software systems. ACM Trans. on Software Engineering and Methodology, 2004,

13(4): 359–430.

[16] Henneman EA, Avrunin GS, Clarke LA, Osterweil LJ, Andrzejewski CJ, et al. Increasing

patient safety and efficiency in transfusion therapy using formal process definitions. Transfusion

Medicine Reviews, 2007, 21(1): 49–57.

[17] Katsh E, Wing L. Online dispute resolution in the last decade and the future. Toledo Law

Review, 2006, 38: 101–126.

[18] Lerner BS, Christov S, Osterweil LJ, Bendraou R, Kannengiesser U, et al. Exception handling

patterns for process modeling. IEEE Trans. on Software Engineering, 2010, 36(2): 162–183.

[19] Muller MM, Padberg F. On the economic evaluations of XP projects. Joint Ninth European

Software Engineering Conference and the 11th ACM SIGSOFT Symposium on the Foundations

of Software Engineering. Helsinki, Finland, 2003. 168–177.

[20] Osterweil LJ, Wise A. Using process definitions to support reasoning about satisfaction of process

requirements. International Conference on Software Process. Paderborn, Germany, 2010. 2–13.

[21] Osterweil LJ. A process programmer looks at the spiral model: A tribute to the deep insights

of Barry W. Boehm. International Journal of Software and Informatics, 2011, 5(3).

[22] Phan H, Avrunin GS, Clarke LA. Considering the exceptional: Incorporating exceptions into

property specifications. Department of Computer Science, University of Massachusetts, Amherst,

MA 01003, 2008.

[23] Phongpaibul M, Koolmanojwong S, Lam A, Boehm B. Comparative experiences with electronic

process guide generator tools. International Conference on Software Process. Minneapolis, MN,

2007. 61–72.

[24] Raunak MS, Osterweil LJ. Process definition language support for rapid simulation prototyping.

Software Process Workshop. Beijing, China, 2005. 416–432.

[25] Royce WW. Managing the development of large software systems: Concepts and techniques.

IEEE Western Electronic Show and Convention, Los Angeles, CA, 1970. 1–9.

[26] Schwaber K, Beedle M. Agile Software Development with Scrum. Prentice Hall, Upper Saddle

River, New Jersey, 2002.

[27] Shewhart WA. Economic Control of Quality of Manufactured Product. D. Van Nostrand Co.,

1931.

[28] Simidchieva B, Engle SJ, Clifford M, Jones AC, Peisert S, et al. Modeling and analyzing faults to

improve election process robustness. 2010 Electronic Voting Technology Workshop/Workshop

on Trustworthy Elections. Washington, DC. 2010.

[29] Smith RL, Avrunin GS, Clarke LA, Osterweil LJ. Propel: An approach supporting property


elucidation. 24th International Conference on Software Engineering. Orlando, FL, 2002. 11–21.

[30] Tan J, Avrunin GS, Clarke LA. Heuristic-Based model refinement for FLAVERS. 26th Interna-

tional Conference on Software Engineering. Edinburgh, Scotland, 2004. 635–644.

[31] Tan J, Avrunin GS, Clarke LA, Ziberstein S, Leue S. Heuristic-Guided counterexample search in

FLAVERS. 12th International Symposium on the Foundations of Software Engineering. New-

port Beach, CA, 2004. 201–210.

[32] Tan J, Avrunin GS, Clarke LA. Managing space for finite-state verification. 28th International

Conference on Software Engineering. Shanghai, China, 2006. 152–161.

[33] Vesely W, Goldberg F, Roberts N, Haasl D. Fault Tree Handbook U.S. Nuclear Regulatory

Commission, Washington, DC, 1981.

[34] Wang D, Pan J, Avrunin G, Clarke LA, Chen B. An automatic failure mode and effect analysis

technique for processes defined in the little-Jil process definition language. The 22nd Interna-

tional Conference on Software Engineering and Knowledge Engineering, 2010, San Francisco,

CA, 2010. 765–770.

[35] Xiao J, Osterweil LJ, Wang Q. Dynamic scheduling of emergency department resources. First

ACM International Health Informatics Symposium, 2010, Arlington, VA: 2010. 590–599.

[36] Xiao J, Osterweil LJ, Wang Q, Li M. Dynamic resource scheduling in disruption-prone software

development environments. Fundamental Approaches to Software Engineering.Paphos, Cyprus.

2010.

Environment Support for Improving Software Development...

Documents

Transcript of Environment Support for Improving Software Development...