A more rigorous and comprehensive approach to software process assessment

SOFTWARE PROCESS: IMPROVEMENT AND PRACTICESoftw. Process Improve. Pract. 2000; 5:3–30 (2000)

A More Rigorous andComprehensive Approachto Software ProcessAssessment

Research SectionJuan Ares1, Rafael Garcıa1, Natalia Juristo2, MartaLopez1,* and Ana Ma Moreno2

1Corunna University, Computer Science School, Campus of Elvina,15071, Corunna, Spain2Madrid Technical University, Computer Science School, Campus ofMontegancedo, 28660, Madrid, Spain

Software process assessment is an essential activity for improving software development inan organization. It is very difficult to put together an efficient and effective improvementplan unless it is based on the results of a preparatory assessment. This will determine thecurrent status of the organization’s software process and will identify the areas or pointsthat need improvement. There is a need for a rigorous method of assessment that encompassesthe factors that affect software production. By rigorous we mean an evaluation based onEvaluation Theory. This theoretical foundation must assure that the evaluation carried outis comprehensive and reliable. Evaluation Theory, defined by Scriven and other authors,describes the components for each type of evaluation method. Six guidelines can be deducedby generalising these components, according to which any evaluation method should bedeveloped: target; criteria; yardstick; assessment techniques; synthesis techniques, andevaluation process. In this paper, we present a software process assessment method basedon Evaluation Theory. This theoretical foundation was one of the things that compelled usto reflect on the factors to be included in the evaluation. As a result, the proposed methodjointly assesses four essential factors in software production: processes; technology resources;human resources, and organizational design. The aim behind this method is to provide aremedy for the main shortcomings of current software process assessment methods: partialityof the evaluated factors, because most centre on the assessment of management processes;and non-rigorousness of the evaluation processes. The applicability of the proposed methodhas been corroborated by means of an experimentation conducted at small and medium-sized Spanish businesses. Copyright 2000 John Wiley & Sons Ltd

KEY WORDS: software process assessment; software process evaluation; evaluation method; evaluation theory

1. INTRODUCTION

Software process assessment and improvement arestrategies that have been applied over the last

*Correspondence to: Marta Lopez, Corunna University, Com-puter Science School, Campus of Elvina, 15071, Corunna, Spain.

Copyright 2000 John Wiley & Sons, Ltd.

decade for the purpose of improving the qualityof the software developed. This strategy is equival-ent to that implemented in other branches ofengineering and industries where the quality ofthe resulting product is increased by controllingthe process used in its production (Humphrey1989). There are now several software processassessment methods that were developed on the

Research Section J. Ares et al.

basis of this idea and have even adopted theactivities applied in other engineering disciplines.However, the assessments conducted have beenfound not to be as rigorous as would be desirable(Saiedian and Kuzara 1995, Rout et al. 1994), thisbeing a direct consequence of adopting the evalu-ation process used in other fields without satisfac-torily adapting it to the software setting. Theproblems arising as a result of this want ofrigorousness are: questionable results validity;evaluation process non-repeatability; results varia-bility, and non-comparability of the findings ofmore than one assessment, even when applyingone and the same method (Saiedian and Kuzara1995, Ares et al. 1999). A software process assessmentstandard (ISO-15504 1996) has been developed totry to solve these problems. However, none of thecurrent approaches for assessing the software processhas considered Evaluation Theory or the progressmade in other fields concerning evaluation.

Evaluation Theory (Scriven 1991) is a disciplinewhich provides a formal and explicit descriptionof the concept of evaluation, what types of methodsof evaluation there are and, for each method,which components must be developed. By takinginto account this discipline when preparing asoftware process assessment method, the resultingmethod of evaluation will be supported by theoreti-cal concepts and, therefore, be more rigorous andreliable. The ideas promulgated by EvaluationTheory are being applied in fields like Sociologyand Pedagogy, where the need arose to formallydefine evaluation and where most progress hasbeen made in this matter (Shadish et al. 1993,Madaus et al. 1983).

This paper presents a software process assess-ment method that was developed taking intoaccount the basic theoretical concepts providedby Evaluation Theory. This method is termedassessment, rather than evaluation, as it is orientedto software process improvement. The paper hasbeen structured as follows to present this method:Section 2 discusses the basic concepts of EvaluationTheory, which will be used as a framework fordeveloping the method by mapping all the basiccomponents to the software domain; Section 3presents the analysis conducted of the softwareprocess assessment methods currently in use, takinginto account the ideas proposed by EvaluationTheory; Section 4 discusses the proposed assess-ment method, including examples of all the compo-Copyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

4

nents developed; Section 5 describes the experimen-tation conducted at small and medium-sizedSpanish businesses; finally, Section 6 outlines thefinal conclusions of this paper.

It is important to note that the method proposedhere is one possible result of applying the theoreticalfoundation of Evaluation Theory to software pro-cess assessment. Using the same theoretical foun-dation, other authors could come up with variationson this method. However, the main idea that ourresearch seeks to highlight is that any softwareprocess assessment method developed using atheoretical basis will always be a more rigorousand reliable method.

2. APPLICATION OF EVALUATIONTHEORY TO SOFTWARE PROCESSEVALUATION

2.1 Evaluation Theory

Evaluation is an interdisciplinary field which, apartfrom its own concepts and processes, feeds offinput from several fields of knowledge (Scriven1991). No general-purpose studies were availableuntil the 1990s, because evaluation was consideredas a section of other disciplines and not as adiscipline in its own right. However, there isnow a general-purpose theoretical framework thatdefines the different types of evaluation methodsand provides parameters for their classification.The basic methods are classed in Figure 1 accordingto the type of results that will be obtained from theevaluation: objective or subjective (House 1980). Eachindividual method of evaluation can be consideredas an instance of one of these generic methods.

Evaluation Theory also describes the componentsinvolved in an evaluation, although their practicalimplementation and nomenclature will depend onthe selected method and the discipline or field in

Figure 1. Types of evaluation methods

Research Section Rigorous and Comprehensive Approach to Software Process Assessment

Figure 2. Components of an evaluation

which the evaluation is to be conducted. Bygeneralizing the individual components describedfor each type of evaluation method, a set ofelements is obtained which can be classed as basic,because they are common to any type of evaluationmethod; the method of negotiation alone differs inthe number of basic components, as this evaluationdoes not use a standard (or yardstick, accordingto the nomenclature used in this paper). Figure 2shows the evaluation components employing thegeneric nomenclature used in this paper. Eachcomponent is defined as follows (Scriven 1991):

I Target: the object under evaluation.I Criteria: the characteristics of the target which

are to be assessed.I Yardstick: the ideal target against which the

real target is to be compared.I Assessment techniques: the techniques needed

to assess each criterion under analysis.I Synthesis techniques: techniques used to

organize and synthesize the informationobtained with the assessment techniques. Theresult of the synthesis is compared with theyardstick.

I Evaluation process: series of activities and tasksby means of which an assessment is performed.

As shown in Figure 3, all these components

Figure 3. Interrelations between components

Copyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

5

are closely interrelated. The evaluation can becustomized by means of the target, because this isone of the parameters used to select the evaluationmethod. Once the target is known and has beendelimited, its characteristics must be identified forevaluation. All the characteristics and their idealvalues, which indicate what the target should belike under ideal conditions, make up what isknown as the yardstick. Data about the realtarget should be gathered using certain assessmenttechniques, by means of which the value of thetarget under analysis can be obtained for eachcriterion. Once all the data have been collected,they are organized in an appropriate structureand compared against the yardstick by applyingsynthesis techniques. This comparison will outputthe results of the evaluation. Finally, all theabove are linked by the evaluation process, whichindicates when to define the scope and extent ofthe evaluation and when to develop, or adapt ifalready available and when necessary, the criteria,yardstick and techniques. All this is defined by aset of activities and tasks for performance.

All the software process assessment methodsexplicitly describe the evaluation process and theyardstick, which are, in some cases, the onlycomponents taken into account by the teams ofevaluators when performing the evaluation. In thesoftware field, the yardstick usually considered isa standard or maturity model. However, EvaluationTheory establishes that, for each type of methodconsidered, all the components must be explicitlydefined for the evaluation to be conducted rigor-ously and systematically. This means that evalua-tors will know why a task must be performed ata particular time and, generally, is an aid forgaining a detailed understanding of the methodand the components, and, hence, for eradicatingor at least minimizing interpretations, which alwaysarise if the evaluators do not understand whatthey must do to carry out an evaluation.

2.2. Feasibility of applying Evaluation Theoryto software process assessment

The application of Evaluation Theory to softwareprocess assessment involves, firstly, selecting thebest suited type of evaluation method consideringthe target under analysis: the software process.After analysing the methods outlined in Figure 1,the control-oriented method was selected, as its


objective is to control the target to assure that itremains within certain limits defined by the selectedyardsticks. This is the type of evaluation thatis usually performed in engineering fields and,therefore, is the one used in current softwareprocess assessment methods. The selection ofa particular type of method determines somecharacteristics of the evaluation components, butdoes not mean that the method for a particularfield (in this case, the software process) is alreadydeveloped or that there are specific guidelines thatdefine how it should be developed; it merelyallows certain types of techniques to be selectedfor use in designing each evaluation componentand in specifying a set of characteristics for themethod developed. For example, control-orientedevaluation is based on the use of defined criteriaand yardsticks, such as generic standards or models.

A framework has been created in order todevelop the proposed method, by means of whicha software process assessment method can bedesigned by explicitly defining and developingcomponents established in the framework. Thisframework has been designed taking into accountthe basic components shown in Figure 2, the typeof evaluation method selected and the techniquesapplied in several software and non-software fields.Therefore, it is not a framework that can be foundas such in Evaluation Theory. The proposedmethod is an instantiation of the application ofthis framework. However, if this framework isused by other evaluators, the final result (evaluationmethod) will not necessarily be exactly the same,because instantiation involves making a series ofdecisions, which may differ depending on theopinions and the environment of the softwareprocess assessment method developer, for example,considering a particular set of processes for evalu-ation or taking into account other criteria ortechniques. The following subsections outline theguidelines for developing each component; thetechniques that can be used in each component;and the selected techniques, for which purpose themain parameters considered were the type ofevaluation method selected, the target and that themethod under development must be oriented tosoftware process improvement.

2.2.1. Functional analysis of the targetTarget delimitation is the first essential step in anyassessment. In order to identify the criteria it isCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

6

necessary to study in detail the object underassessment and to delimit the factors to be con-sidered. There are not many techniques availablefor performing this step. Indeed, it is the expertsin the software process who very often indicatewhich factors are to be considered. However, thisframework recommends the use of the techniqueof functional analysis, defined as any study of thefunctions, factors, market preferences, models,standards and/or, in general, ideals that might berelevant to software production. It is in fact aperfectly legitimate activity when one is lookingfor all possible guidance (Shadish et al. 1993) indeveloping a method of assessment.

In this context, we consider that software processassessment methods should not only cover the setof processes executed to produce software, andthat other influential factors must be necessarilyconsidered: technological resources (Kuvaja et al.1994); human resources (Herbsleb et al. 1997, Perryet al. 1994, Khodabandeh and Palazzi 1995), which,because they behave in a non-deterministic andsubjective manner, have a decisive impact on theresults of software production, which is basicallyan intellectual activity (Sommerville and Rodden1995, Bach 1994); and organizational design(Cattaneo et al. 1995). However, if an assessmentis to faithfully reflect the current status of theorganization’s software process, these factorsshould be assessed jointly. At present, there isonly one method that focuses on the humanresources factor (Curtis et al. 1995, Hefley andCurtis 1998), but it is performed independently,without taking into account other factors. Work isnow going on (Humphrey 1998) not only on peopleindividually but also on human resources organizedas a work team. The joint evaluation of thesefactors is the most relevant premise of the proposedframework. This premise will be the basis fordeveloping the remaining components of the assess-ment. Section 4.1 details the functional analysiscarried out and the entire set of factors that willbe assessed in the proposed method.

2.2.2. CriteriaCriteria definition is the second essential andcritical step for developing a method of evaluation.Having ascertained and delimited the target, it isnecessary to identify what characteristics of thetarget are of interest for assessment purposes.These characteristics are referred to as evaluation


criteria. In the framework we propose, criteriaelicitation is based on the use of a series of keyquestions: what; why; when; how; where, andwho. The goal is to output a diagrammatic treethat contains all the general and specific criteriathat are to be evaluated. The concept proposed asa starting point for this elicitation is the softwareprocess (Why is the software process executed?How is the software process executed? etc.). Twotypes of responses could be gained from these ques-tions:

I General criteria or issues: characteristics thatcannot be directly assessed and require furtherdecomposition, to which the set of questionswill be applied successively until specific criteriaare obtained.

I Specific criteria: characteristics that can bedirectly assessed using a particular assess-ment technique.

The final result of applying this elicitation processis a criteria tree. This tree is the basis for developingthe software process model that will be used asthe assessment yardstick and for selecting theassessment techniques. Furthermore, this definitionaids accurate understanding of the yardstick,because the evaluator will know exactly whatcharacteristics are to be analysed, without havingto study the yardstick in depth. Section 4.2 outlinesthe criteria tree obtained for the proposed assess-ment method.

2.2.3. YardstickTable 1 outlines the bases proposed by the frame-work for developing the method, taking intoaccount the target, criteria tree and that the methodis oriented to software process improvement.

In this fashion, the software process modeldeveloped for use as a yardstick in assessmentswould not only be able to identify the issues tobe improved by the organization but also topropose prioritized recommendations that indicatewhich processes and, for each one, which criteria areto be implemented first to make the improvementeffective. Section 4.3 shows how this frameworkcomponent was mapped to the proposed softwareprocess assessment method.

2.2.4. Assessment techniquesApart from building the software process modelused as a yardstick, the potentially applicableCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

7

assessment techniques need to be identified, andone or more need to be assigned to each evaluationcriterion. The objective of applying these techniquesis to output pairs, like [criterion,datum/information] or [criterion,datum/information set], whereby the result of theassessment is obtained by comparing this set ofpairs against the specifications of the yardstick,also structured as [criterion, datum/information].The assessment techniques used in current softwareprocess assessment methods can be classed in threegroups, as shown in Table 2.

This framework proposes the use of assignationtechniques. Metrics are not considered, becausemetric production mechanisms are not usuallyimplemented in medium-small organizations, atleast in Spain. Where they are available, the teamof evaluators will analyse the mechanisms andassess whether or not to use the metrics. Neitherare opinion techniques considered, because we arelooking for non-subjective data. One or morecriteria assessment techniques must be assignedby analysing the meaning of each criterion. Onceidentified, each technique must be developed,outputting questionnaires, standard interviews, andlists of documents for inspection. Section 4.4 showsthe assignation of techniques and an example of thequestionnaire developed in the proposed method.

2.2.5. Synthesis techniquesThe synthesis techniques will be used to synthesizeall the data and information obtained after applyingthe assessment techniques and for comparisonagainst the yardstick. Two types of synthesistechniques can be applied in an assessment:

I Single value: a single datum (numerical orotherwise) is obtained as a result of the assess-ment. This group includes methods of combi-nation. When these techniques are applied, ameaningful value scale is required for the datumoutputted. An example of these techniques isthe CMM and its five-level maturity scale.

I Multiple values: these techniques, for example,statistical techniques, criteria grouping anddatum-by-datum comparison with the yard-stick, output more detailed information.

This framework recommends the use of the secondgroup of techniques, because the aim of thesoftware process assessment method is to improvethe software process. We believe criteria grouping


Table 1. Bases for yardstick development

The yardstick used in the assessment should be a software process model developed from the criteria tree obtained in thepreceding step.The general structure of the model should be inferred from the criteria tree.The model must contain the specifications of all defined criteria.For each criterion, the model must define the specifications structured as pairs [criterion, datum/information].In addition, the yardstick must include assessment scales used as guidelines for prioritizing improvement activities. Two scalesshould be included:

I Process assessment scale, required to indicate which processes should be improved or implemented first to assuresatisfactory software process improvement.

I Criteria assessment scale, required to indicate which criteria should be improved first for each process.The assessment scales must be developed on the basis of a dynamic approach; for example, based on the use of a scale of fivevalues that indicate the relative importance of a given process or criterion. The cells below indicate the five values of the processassessment scale and the parameters used to establish the relative importance of the activities covered by the assessment. Thecriteria assessment scale will be developed according to the same strategy. In this case, the values would be: essential criterion,highly important criterion, significant criterion, possibly significant criterion and no evidence that it is required.

Top priority, essential Process that is absolutely essential for software production to be feasible. If this process is notprocess implemented, it will be impossible to develop and output a key product for software

development.Highly important process Process by means of which a product or service that is very important for software production is

outputted. Failure to implement processes of this type can lead to serious deviations inorganizational behaviour and the execution of other processes; for example, failure to control keyproducts, lack of resource and organizational process co-ordination, significant delays inproduction processes etc.

Significant process Process that provides an important product or service for software production. Failure toimplement this process does not cause such significant deviations as outlined under the abovevalue, but the process does contribute to better software production process execution.

Possibly significant Process by means of which a product or service that contributes to more efficient control of anyprocess particular item or concept related to software production is outputted.No evidence that it is Irrelevant process that has no impact on software production. Failure to implement this processrequired does not lead to any change whatsoever.

Table 2. Types of assessment techniques

Measurement It involves the use of the appropriatemeasurement instruments or mechanisms.

Assignation For example, questionnaires, interviews,documentation inspection and simple testsnot involving metric application.

Opinion Techniques for getting subjective assessmentsof the criteria, for example, by observation.

and datum-by-datum comparison with the yard-stick, in particular, to be preferable. In this fashion,the assessment will provide useful information formanagers to guide software process improvementactivities. The criteria grouping technique will beused to synthesize all the information gathered ina document structured in the same manner as theyardstick. The evaluators will be able to use thisdocument as a checklist for assuring that allthe criteria have been assessed. Datum-by-datumcomparison with the yardstick will be applied toCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

8

check the data obtained against the specificationof the yardstick for each criterion. This task willmake it possible to identify: positive factors of theorganization, or criteria whose values match thedefinitions in the yardstick; negative factors of theorganization, or criteria whose values do notcomply with the specifications in the yardstick;and non-applicable factors, or a series of criterianot applicable to the evaluated organization. Forexample, if the projects under analysis are in therequirements analysis phase, neither the design,implementation nor the testing processes can beassessed; all these cases must be explicitly identified.All the positive, negative and non-applicable factorswill be the basis for the preparation of the assessmentresults. Section 4.5 shows the synthesis techniquesthat will be used in the proposed method.

2.2.6. Evaluation processThe evaluation process is a series of specificactivities and tasks that are to be executed to


perform an assessment. This framework describesthree main subprocesses or phases, which matchthe three major points through which an evaluationpasses: prepare evaluation; get data about thetarget, and make a decision on which the finalreport of the evaluation will be based. These threesubprocesses, shown in Figure 4, are a generaliz-ation of the evaluation processes proposed by allthe basic methods of evaluation, which are usuallytailored to particular fields. Specifically, the mainactivities to be addressed are as follows. Section4.6 details the activities proposed for each evalu-ation subprocess.

1. Planning. Activities involving making contactwith the organization, delimiting the assess-ment and planning and managing its execution.This phase ends when all the components havebeen developed or, if necessary, adapted tothe target in question. This is when the teamof assessors is ready to make a visit to orinteract with the organization.

2. Examination. Application of the assessmenttechniques and gathering of the data requiredto assess the organization’s software process.This phase ends when all the pairs [criterion,datum/information] have been obtained for allthe criteria considered in the assessment.

3. Decision making. Judgement of the target bycomparing the data obtained against the yard-stick. The final result is the assessment report.

In sum, the proposed software process assessmentmethod development framework provides guide-lines for developing the components of an evalu-ation: target; criteria; yardstick; assessment tech-niques; synthesis techniques, and evaluationprocess. The explicit definition and full develop-ment of all these elements will provide a rigorousand systematic software process assessmentmethod. Before addressing the proposed method,the above-mentioned components will be used inSection 3 to analyse the rigorousness of currentsoftware process assessment methods.

Figure 4. Phases of an evaluation process


9

3. REVIEWING THE RIGOROUSNESS OFCURRENT SOFTWARE PROCESSASSESSMENT METHODS

Taking into account the assessment componentsmentioned in the preceding section (target, criteria,yardstick, assessment techniques, synthesis tech-niques, and evaluation process), the currentsoftware process assessment methods were ana-lysed to identify how rigorous the methods reallyare and the software process they use. The methodsconsidered were: ISO 9000 (ISO-9001 1994, ISO-9000-3 1994, ISO-10011 1993, Schmauch 1995);TickIT (TickIT 1995); SCE (Byrnes and Phillips1996) and CBA IPI (Dunaway and Masters 1996),based on CMM (Paulk et al. 1995, Caputo 1998);Bootstrap (Kuvaja et al. 1994, Bicego et al. 1998);Trillium (Trillium 1994); STD (Craigmyle andFletcher 1993); and ISO 15504 (ISO-15504 1996,Emam et al. 1998).

This study is based on the idea that unless allthe components of the assessment are explicitlydefined and developed, it will be impossible toconsider a method of evaluation as rigorous. Themeanings of the values assigned to each componentare described in order to give an understandingof the summary table. Table 3 recalls the meaningof each component of the evaluation and alsooutlines what the values of the above componentsshould be for a method that was developedaccording to the framework discussed in thepreceding section (termed ideal method).

The final result of the study is shown in Table 4.Some components could not be analysed, becausethere are confidential methods, for example, STD,Trillium and Bootstrap. In these cases, a dash hasbeen entered in the respective cells. An ideal, fullydeveloped method is included in the last columnfor the purpose of comparison. The entire yardstickcannot be obtained for Bootstrap, although it isavailable a version of the BootCheck tool(BootCheck 1997) that automates evaluation. How-ever, studying this tool, the assessment techniquesused can be analysed.

It follows from Table 4 that the number of factorsconsidered in the evaluation has increased in themore recently developed methods (Bootstrap, STDand ISO 15504). However, none of the currentmethods encompasses the joint assessment ofprocesses (technical, management and strategic),technological resources, human resources and


Table 3. Meaning of the parameters used to review current software process assessment methods

Parameters Ideal method Meaning

Target Processes (all levels), human The method must consider all the factors that influence softwareand technological resources production (processes, technological resources, human resources andand organizational design organizational design) and their interrelations. Table 4 specifies what

factors are considered in the evaluation for each method.Criteria Defined The method must explicitly define all the general and particular

characteristics for evaluation. For example, evaluate technologicalresources by means of the following specific criteria: adequacy,availability and use.Table 4 specifies whether the criteria are implicitly (Implicit) orexplicitly (Defined) defined for the method under consideration.

Yardstick Developed The method must have a defined yardstick (standard or processmodel) for comparison purposes. This yardstick must include thedevelopment of all the evaluation criteria. Table 4 specifies whetherthe yardstick is developed or not developed.

Assessment techniques Developed The method must:I Explicitly identify the assessment techniques to be applied.I Completely develop all these techniques.Table 4 identifies whether the assessment techniques are fullydeveloped (Developed), partially developed (Partially prepared), notdeveloped (Not prepared) or whether the method proposes noparticular technique (Not proposed).

Synthesis techniques Developed The method must:I Explicitly identify the synthesis techniques to be applied.I Completely develop all these techniques.Table 4 specifies for each method whether the synthesis techniquesare developed (Developed) or whether the method merely gives ageneric description of what type of techniques should be used(Generic Description).

Evaluation process Developed The method must define all the activities and tasks that will beperformed during evaluation in full detail. Table 4 specifies whetherthe method provides a developed evaluation process (Developed);whether the method provides no evaluation process (Not Developed),as in the case of ISO 15504, which are requirements; or whether themethod merely gives a generic description of the process withoutgiving any details (Partially Developed), as is the case of Trillium,which merely references other processes, depending on theorientation of the evaluation.

organizational design. Except for ISO 15504, themethods have developed the yardstick, althoughit is not publicly available for all of them. Addition-ally, all the methods have defined the evaluationprocess. However, the other components are notusually rigorously defined. This is especially trueof the evaluation criteria, which are implicit in theyardstick and not explicitly defined in four of themethods analysed. This want of method rigor-ousness could lead to a range of interpretations.The possibility of interpretation can lead to non-repeatability of the evaluation processes with theresulting variability and non-comparability of theassessment results. In order to fill this void, weCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

10

set out to create a new method of assessment thatis presented in this paper. Its development wasbased on the framework proposed in Section 2.All the components developed are presented inthe following section.

4. A RIGOROUS METHOD OFASSESSMENT

The goal of the method developed is to providean extensive set of results and recommendationsserving as the basis for establishing softwareprocess improvement programmes. Moreover, it is


Table 4. Rigorousness of software process assessment methods

ISO 9000 TickIT CMM Bootstrap Trillium STD ISO 15504 Ideal method

Target Quality Quality Management Technical and Telecommun- Technical, Technical, Processes (allsystem system process management ications management management levels), human

process and and strategic and strategic andtechnology processes processes technological

resources andorganizationaldesign

Criteria Implicit Implicit Implicit Defined Implicit – Defined DefinedYardstick Developed Developed Developed – Developed – Not developed DevelopedAssessment Not Not Partially Developed Not – Not proposed Developedtechniques prepared prepared prepared preparedSynthesis Generic Generic Developed Developed – – Developed Developedtechniques description descriptionEvaluation Partially Partially Developed Developed Partially Partially Not developed Developedprocess developed developed developed developed

an independent assessment method; the teamof evaluators does not belong to the assessedorganization. The aim behind this is to make theresults of the assessment as objective as possible.All the components of the method have beenprepared according to the framework specified inthe Section 2.

4.1. Target

The target of a software process assessment methodis the software process. We defined softwareprocess as a series of activities needed to producea software system, performed by a series of humanresources organized according to a particularorganizational structure and equipped withmaterial support, tools and equipment.

As explained in the framework (Section 2), thebest means of defining the target in its entirety isby means of functional analysis. Three types ofgeneric processes are identified from the functionalanalysis of software activities, as shown in Figure 5:

Figure 5. Software process levels


11

technical processes; management processes, andstrategic processes. The technical process(requirements analysis, design, implementation andtesting) is the basic and essential nucleus of theactivities required to produce software. However,other processes are also necessary by means ofwhich to manage the products outputted(requirements specification, design, etc.) and thetechnical process itself: management processes.Figure 5 illustrates these processes as a circleencompassing the technical process, because theyaddress the management of these technical activitiesand, generally, the management of a softwaredevelopment project. Finally, the strategic processwould encompass the management of all thesoftware development projects and all the humanand technological resources of the organization.

Specifically, the management process consideredin this assessment method is composed of thefollowing activities: project management and con-trol; quality management; configuration manage-ment; risk management; subcontracting manage-ment, and customer interaction management.Finally, the strategic process is composed of thefollowing activities: organization management;financial management; personnel and humanresources management; quality system; risk assess-ment; technology management; suppliers manage-ment, and software process management.

According to this process classification, thesoftware process can be divided into three layersor levels: technical, management and strategic. Thisis the basis for analysing the functional structure


of the organization and relating it to the typicalorganic structure of any company. By means ofthe organic structure, it is possible to identify andcontrol the human resources who will perform theprocesses specified in the functional structure ofthe organization. Generally, the organic structureclasses human resources into three categories:senior management, middle management and oper-ational basis. The human and technologicalresources required for executing each of the above-mentioned processes will be provided. A jointassessment of all the factors involves the analysisof the functional and organic structures of theorganization. For this purpose, the processes andresources must be correlated at each level in thehierarchy. This generic relationship is shown inFigure 6. The structure or design of the organizationis inferred from this basic configuration.

The matrix model is used in this method todescribe a software organization. The matrix modelwas chosen because: it was designed to work withall the functional departments that support theprojects and reduce or eliminate the amount ofduplicate effort found in the project team with theoverall organization; and this type of organizationmaintains the functional model, while it establishesa relatively permanent horizontal structure sup-porting the continuous influx of projects. However,as the matrix model contains the functional model,it could be quickly adapted to consider only thefunctional structure. Therefore, the yardstick ismore complete and, at the same, flexible if thematrix model is used. Additionally, a projectdimension can be represented by means of thisstructure: various groups of human resources,specialized in different functions and participating

Figure 6. Basic configuration of the functional andorganic structures of a software development organiza-tion


12

in different software development projects. Figure 7shows the matrix design of an organization andthe positioning of certain processes and humanresources at different levels of the hierarchy. Forsimplicity’s sake, Figure 7 does not illustrate thestrategic process, showing merely Senior Manage-ment (‘General Management’, ‘Senior SystemsEngineer’ and ‘Software Engineering Manager’).The management process is depicted at a lowerlevel of the hierarchy and the technical process atthe bottom. Each of these processes is representedby its manager (quality manager, requirementsanalysis manager, etc.), and all the groups ofpractitioners that execute that process (groups). Theproject dimension is represented by the ‘softwareproject managers’. These managers will direct agroup of practitioners (groups), who will carry outa software development project, with regard toboth the management and the technical processes.

The target considered in the proposed assessmentmethod is composed of the processes (technical,management and strategic) and the human andtechnological resources required for each process.Furthermore, the organizational design and pos-itioning of each activity by hierarchical level willalso be analysed. Thus, it will be possible not onlyto assess the processes but also to check that theyare positioned at the appropriate hierarchical levelfor their actions and results to be correct. None ofthe current assessment methods includes thisassessment. Indeed, when the problem is rootedin the incorrect positioning of a process, this is notdetected by the assessments, and their resultsmight not be correct (Cattaneo et al. 1995). Inthe proposed assessment method, each softwareprocess level (technical, management and strategic)has been linked to a hierarchical level (seniormanagement, middle management and operationalbasis). In this fashion, the organization will executeall the processes directed at managing all theorganizational resources and all the software devel-opment projects carried out by the company at thetop level (senior management); it will execute theprocesses for managing each software productionproject at the intermediate level (middlemanagement); and it will execute all the basicprocesses for software development at the bottomlevel (operational basis). It is important to notethat this operational basis is composed of softwareengineers responsible for performing requirementsanalysis, design, implementation and testing. The


Figure 7. Matrix structure of a software construction organization

other components of the evaluation have beendeveloped taking into account this set of factorsfor analysis.

4.2. Criteria

After applying the criteria elicitation technique,answering the key questions (what, why, when,how, where, and who) described in the framework(Section 2.2.2), the final result of criteria elicitationis the criteria tree shown in Figure 8. This tree isthe basis for developing the software process modelthat will be used as the assessment yardstick. Asshown in Figure 8, the model covers three issues:

I Activities: each process considered will beassessed according to the activities performed(Functional sub-issue), associated human andtechnological resources (Resources sub-issue)and the management control in place (Controlsub-issue). None of the processes related tosoftware process improvement is includedunder this issue, because, owing to their impor-tance, these are addressed under a special issue:software process model issue.

I Software process model: analysis of the softwareprocess improvement activities performed inthe organization; these processes are assessed


13

at the organizational level, not individually foreach project.

I Structure: criteria required to assess the designor structure of the organization.

This diagrammatic tree has been supplementedwith a description of the meaning of each criterionto prevent misinterpretations of the criteriaobtained. An example is given in Table 5, whichcontains the meaning of the criteria of the Activitiesissue, Functional sub-issue, that is, the set ofcharacteristics concerning the execution of activitiesthat make up a process, the products they output,applicable standards, products required for theirexecution and objectives to be met by the process.

Except for Bootstrap, which includes a generictree stating the issues assessed (although themethod for outputting this tree is not stated),no other current assessment method includes anexplicit description of the criteria that are to beevaluated. However, the definition of these criteriais the first and an essential step in any assessment.Furthermore, this definition aids accurate under-standing of the model used as a yardstick.

4.3. Yardstick

The yardstick used in this method is a softwareprocess model obtained from the criteria tree


Figure 8. Diagrammatic tree of evaluation criteria

Table 5. Meaning of the Functional sub-issue criteria

Criterion Meaning

Functional Functional goals Objectives associated with the process under analysis that describe itspurpose, generic requirements for its execution and its mostrepresentative product(s) or service(s).

Functional activities Identification of the tasks involved in the process under analysis.Product developed or Work product Identification and description of the work product or serviceservice provided outputted by process execution.

Characteristics Description of the characteristics proper to or to be met by theproduct obtained or service provided.

Applicable standards Identification of the standards that are to be used in processexecution.

Activities interrelation Input product(s) Identification of the input products required for proper processexecution and their source.

shown in Figure 8. This model is composed of aseries of tables detailing all the criteria of eachissue in the form [criterion, datum/informationset]. The general structure corresponds with thethree general issues: activities, software processmodel and structure. The ‘Activities’ issue isapplied to each process under consideration,classed as a technical, management or strategicprocess. This general structure of the model isCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

14

shown in Table 6. Each production process has anassociated table, which details the criteria properto the ‘Activities’ issue. For example, an extract ofthe model for the ‘Requirements analysis’ process(technical process), detailing the ‘Functional’ sub-issue is shown in Table 7.

As specified in the framework (Section 2.2.3),the model should also include process and criteriaassessment scales to prioritize the improvement


Table 6. Structure of the software process model

1. Structure2. Software process model3. Activities

3.1. Strategic level3.1.1. Quality system3.1.X. Other processes positioned at this level

(technology management, risk assessment etc.)3.2. Management level

3.2.1. Project management and control3.2.X. Processes positioned at this level (qualitymanagement, configuration management etc.)

3.3. Technical level3.3.1. Requirements analysis

3.3.1.1. Functional sub-issueI Functional goalsI Functional activitiesI Product developed/service providedI etc.

3.3.1.2. Resources sub-issue3.3.1.3. Control sub-issue

3.3.X. Processes positioned at this level (design,implementation and testing)

activities. Table 8 shows the five values of theprocess assessment scale and the parameters usedto establish the relative importance of the activitiescovered by the assessment.

As an assessment should cover the set ofmost important and essential software productionactivities, the associated values are top priorityand highly important. Table 9 shows the process-value association for all the processes covered bythis method. The implementation of the differentsoftware production activities can be prioritizedon the basis of this information. Similarly, thesame strategy has been applied to develop thecriteria assessment scale. In this manner, we canprioritize how a given process is to be implemented.

Taking this explicit definition of model construc-tion and its assessment scales, any team of evalua-tors would be able to modify these associationsand thus adapt the model to the characteristics ofthe organization under assessment. The explicitdefinition of model construction would mean thatthe model can evolve to include new activities or,generally, adapt as the software process alters inthe future. Therefore, the yardstick would allowevolutionary strategies to be established so thatthe organization moves from an ad hoc statustowards an optimized status in which the companyis capable of managing and continually improvingCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

15

its software process. In order to provide theseevolutionary strategies, it is necessary to take intoaccount jointly the three process types (technical,management, strategic) and the process and criteriaassessment scales. Thus, for example, Table 9 showsthat it will be necessary for all the processesgrouped under ‘technical process’ (requirementsanalysis, design, implementation, and testing,which are all top priority) to be implemented inthe organization for the company to be able toproduce software. Similarly, a subset of manage-ment processes will be essential for satisfactorilymanaging a software development project. Theevaluators would be able to recommend a possibleevolutionary strategy for software process improve-ment by means of this ‘processes–scales’ relation-ship.

4.4. Assessment techniques

The assessment techniques selected are assignationtechniques, particularly questionnaires, interviewsand documentation inspection. The criteria treeand the software process model will be used todevelop these techniques. One or several criteriaassessment techniques have been assigned byanalysing the meaning of each criterion. Accord-ingly, Table 10 shows an example of the techniquesassociated with the ‘Functional activities’ issuecriteria. Once identified, each technique has beendeveloped, outputting questionnaires, standardinterviews and lists of documents for inspection.As an example of the assessment techniques,Figure 9 shows an extract from the questionnaireapplied to assess the criterion ‘Functional activities’of the ‘Requirements analysis’ process (technicalprocess). The following items, shown in Figure 10,have been included for each questionnaire response:

I The possible responses: No, Yes, Not applicable,Don’t know.

I An observations section for noting down poss-ible explanations or exceptions.

I Three associated sub-questions (Is it docu-mented? Was it performed according to anestablished procedure? Is the proceduredocumented?). These three sub-questions areinterrelated. The first question aims to find outwhether the result of a particular task is setout in a document. For example, consideringquestion 1 in Figure 9, the response to this


Table 7. Extract of the software process model for the Requirements analysis process

Criteria Yardstick

Functional goals The organization must define specific goals for the requirements analysis process:I Develop a set of software requirements based on customer requirements.I Prepare the software requirements document.I Analyse the changes to software requirements throughout the entire software life cycle.

Functional activities DEVELOPMENT1. Develop software requirements.

I Analyse customer requirements and constraints to determine functionality and get a logicmodel of the product from which to:

(i) Identify the main software components and interfaces.(ii) Extract and class requirements as: functional, performance, interfaces, resources,

security, etc.I Analyse all the requirements to check that the set is complete, consistent and

unambiguous.I Prepare a prioritized list of requirements to allow gradual product development.I Prepare the software requirements document.

2. Analyse and approve requirements.I Interact with the customer who is to the analyse the document prepared and approve the

specified requirements:(i) Present the document to the customer using a suitable method that aids

communication and understanding of the requirements (e.g. simulations,prototypes, informative meetings etc.).

(ii) If any needs are found not to be met or faults, inconsistencies andmisunderstandings are identified, the requirements and their specification mustbe modified, repeating this and the preceding activity.

(iii) If the customer accepts the requirements, put down approval in writing.I In any case, record the results of these interactions.I After acceptance and approval, submit the document to the software configuration

management group.

DEVELOPMENT/MAINTENANCE3. Analyse any changes to software requirements during the life cycle:

I Control and record the proposed changes to requirements.I Analyse the changes to assure software requirements completeness and integrity and

identify any side effects or any other possible consequences of the change.I Report the proposed and accepted changes to the project manager for project plan revision

and updating, where necessary, and notify customer and other groups involved.Product Work product The main product outputted by this process is the requirements analysis document. Thisdeveloped/service document is the basis on which the remainder of the technical process will be developed. Alsoprovided to be supplied are:

I Reports on interaction with customer.I Analysis of the proposed changes to software requirements.

Characteristics The requirements document must be prepared according to the structure and guidelinesspecified in IEEE standard 830 (Guide to Software Requirements Specifications).

Applicable standards The life cycle and a specific development method to be applied for software production, whichwill provide detailed guidelines for performing this process, will have been selected during theproject definition phase: structured analysis, prototyping, object-oriented analysis, etc.

Activities Input(s) I Customer requirements document, prepared in project identification phase, including anyinterrelation constraints imposed (standards, protocols, etc.).

I Internal or organization-imposed constraints: programming language, marketing strategies,available technology, etc.


16


Table 8. Process assessment scale

Values Parameters

Top priority, essential process Process that is absolutely essential for software production to be feasible. If this process is notimplemented, it will be impossible to develop and output a key product for softwaredevelopment. For example, requirements analysis, essential for developing the requirementsanalysis document.

Highly important process Process by means of which a product or service that is very important for software productionis outputted. Failure to implement processes of this type can lead to serious deviations inorganizational behaviour and the execution of other processes, for example, failure to controlkey products, lack of resource and organizational process co-ordination, significant delays inproduction processes etc. A highly important process is, for example, configurationmanagement by means of which product versions will be able to be controlled.

Significant process Process that provides an important product or service for software production. Failure toimplement this process does not cause such significant deviations as outlined under the abovevalue, but the process does contribute to better software production process execution, forexample, the documentation control process.

Possibly significant process Process by means of which a product or service that contributes to more efficient control of anyparticular item or concept related to software production is outputted.

No evidence that it is required Irrelevant process that has no impact on software production. Failure to implement this processdoes not lead to any change whatsoever.

Table 9. Process-value association of the process assessmentscale

Value Processes

Top priority, essential Requirements analysisprocess Design

ImplementationTestingProject management and controlQuality managementHuman resources managementTechnology managementQuality systemFinancial management

Highly important process Configuration managementRisk managementCustomer interaction managementSoftware subcontractingmanagementOrganization managementRisk assessmentSupplier managementSoftware process management

sub-question would be ‘Yes’ if the detailedinformation on software requirements isreflected in a document. If the response to thefirst sub-question is yes, the next question willbe whether the task has been executed accordingto an established procedure. Finally, if theresponse to the second sub-question is yes, the


17

next question will be whether the procedurefor executing the task is documented.

For simplicity’s sake, this structure has onlybeen included for the first two questions shownin Figure 9, but these items (responses, observationsand sub-questions) are applied to each question inthe questionnaire. Moreover, help has beenincluded for some questions to prevent misinter-pretations; for example, Figure 9 shows an explana-tory text under question number 8.

The questionnaire is structured on the basis ofthe set of criteria for which this technique will beapplied, but its content is based directly on thespecifications in the yardstick for each processcovered by the assessment. The ease of obtainingthis technique is plain. It also has other features,such as: ease of maintenance, as the softwareprocess, and hence the yardstick used in theassessment, alters in the future; they are supportedquestionnaires, they are objective, not subjective;and it is more adaptable, as the questionnaire canbe modified depending on the company or projectsunder assessment subject to adaptation of theyardstick. For example, if a project is in the earlydevelopment stages, it will not be possible toevaluate the implementation process and lateractivities; in this case, the team of evaluators willhave to adapt the yardstick and questionnaire: thenon-applicable processes are identified and the


Table 10. Assignation of assessment techniques to criteria

Criteria Assessment techniques

Functional goals Documentation inspection; interviewFunctional activities Documentation inspection; interview; questionnaireProduct developed Work product Documentation inspection; interviewService provided Characteristics Documentation inspection; interview; metrics, if applicableApplicable standards Standards not applied: interview; questionnaire

Standards applied: documentation inspection; interviewActivities interrelation Input(s) Documentation inspection; interview; questionnaire

Figure 9. Extract of the questionnaire for the criterion ‘Functional activities’ of the Requirements analysis process


18


Figure 10. Structure of each question in the questionnaire

respective questions are removed. Finally, evenpart assessments could be performed as the scopeof the assessment can be confined to a particulardevelopment project, which may or may not be inthe later stages of development, or to particularprocesses that are to be evaluated.

4.5. Synthesis techniques

The synthesis techniques are used to synthesizeall the data and information obtained after applyingthe assessment techniques and to make the compari-son with the yardstick. The techniques that willbe used in this assessment method are criteriagrouping and datum-by-datum comparison withthe yardstick. Figure 11 shows an extract of theassessment method documentation, reflecting anexample of how to identify the positive andnegative factors for a particular criterion, applicablein this case to the technical process of Requirementsanalysis. A scale of five values, similar to the oneused by ISO 15504, is used for this purpose: (F)Fully adequate; (L) Largely adequate; (P) Partiallyadequate; (N) Not adequate, and (NA) Not Appli-cable.

After applying this technique a document isoutputted that contains all the positive, negativeand non-applicable factors and their values foreach process analysed. This document will bestructured in a similar manner to the yardstick. Allthis information is included in the final assessmentreport and will be supplemented by recommen-dations, narrative descriptions, comparative statisti-cal data of the various projects analysed, compara-tive bar charts etc.


19

4.6. Evaluation process

The evaluation process is composed of the sub-processes and activities shown in Figure 12. In theplanning process, a preliminary meeting is held toget general information about the organization,production, projects under development, appli-cation domains, and the type of software developed.All this information will be used to delimit theassessment, establish its goals, and define thecontract between the two organizations involved:the evaluated company and the evaluating com-pany. This information will also be employed todesign the assessment, which means creating adocument stating the criteria, software processmodel and techniques to be applied in this parti-cular assessment. This document will be ableto be used for comparison with the results offuture assessments.

The next step is to make a more detailed analysisof the target. This will make it possible to performthe following activities: set up the team of evalua-tors; train the team (if necessary); select theprojects to be evaluated; select the members ofthe organization who are to be surveyed andinterviewed; define the infrastructure for gatheringthe information (templates, computer media, tablesetc.), and enter all this information in the ‘Assess-ment Design’ document.

Finally, all the activities and, particularly, the visitto the organization during which the assessmenttechniques will be applied, need to be meticulouslyplanned. A manager of the evaluated companywill have to participate in planning in order todefine and negotiate times, dates and scheduling forinterviews, questionnaires and assure availability ofthe documentation for inspection.


Figure 11. Example of assignation of positive and negative factors for the Work product criterion

The visit to or interaction with the organizationduring which the assessment techniques are appliedcan be organized by means of all this preliminaryplanning. The first techniques to be used arequestionnaires. Their results will be the basis forimplementing the other techniques. Any inconsist-encies, contradictions and omissions in theCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

20

responses will be of assistance in detecting problemareas, which will be analysed in more detail usingother techniques. Then the documentation will beinspected, tests will be conducted (if applicable)and interviews held. The team of evaluators willhave to analyse the data obtained in all cases tocheck for completeness. If any data are missing, the


Figure 12. Overview of the evaluation process

standard interviews and documentation inspectionwill be modified as appropriate in order to accountfor all the information required to assess the target.All these amendments will be recorded in the‘Incidents File’ document, which will be used atthe end of the assessment to evaluate the process.All the data and information gathered will beentered in the ‘Criteria Assessment’ document. Theexamination process ends when all the pairs[criterion, datum/information] have been obtainedfor all the criteria considered in the assessment.

Finally, the decision making process will use the‘Criteria Assessment’ document to prepare theassessment report. The synthesis techniques willbe applied for this purpose. All the pairs will beorganized and synthesized in a similar manner tothe yardstick by means of criteria grouping. Datum-by-datum comparison will be used to class thecriteria as:

I Positive factors of the organization – all thecriteria that comply with the specifications ofthe software process model;

I Negative factors of the organization – all thecriteria that do not comply in part or in wholewith the specifications of the model;

I Non-applicable factors – not all the criteria thatcan be assessed, that is, the criteria that are notapplicable taking into account the characteristicsof the evaluated organization.

The final assessment report is prepared using theseCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

21

factors. Not only must it include the descriptionof the current status of the software process ofthe organization, but also recommendations andguidelines for establishing an improvement pro-gramme. The process and criteria assessment scalesdefined in the proposed software process modelwill be used for this purpose.

The team of evaluators will explain the reportat a meeting with the professionals and managersof the organization involved. The written documentwill be circulated taking into account the technologyused at the company. The team will recommenda strategy; however, it will be a manager of thecompany who circulates the report to its addresses.This activity ends the interaction of the team ofevaluators with the evaluated organization.

Finally, the evaluation process concludes bycollecting all the documentation generated andperforming a self-assessment to analyse the assess-ment performed. The first step will be a useful aidfor comparison with future assessments to thusascertain how the organization’s software processhas evolved. The self-assessment will serve toanalyse all the incidents detected, assess the docu-mentation obtained, the performance of each teammember, and evaluation process follow-up. Thisfeedback will be used to make more accuratefuture plans and refine the evaluation process.

In sum, the proposed assessment method iscomposed of six interrelated components: target;criteria; yardstick; assessment techniques; synthesis


techniques, and evaluation process. The explicitdefinition and full development of all theseelements has made it possible to develop a rigorousmethod according to the foundations of EvaluationTheory. The method is oriented to improving thesoftware process and covers the joint assessmentof the factors of software production, which areprocesses, human resources, technologicalresources and organizational design.

5. EXPERIMENTAL RESULTS

In order to verify the applicability of the proposedmethod, an experiment was conducted at smalland medium-sized Spanish companies. Theseassessments involved the performance of severalevaluations applying CBA IPI, ISO 9000 and theproposed method. Also a series of comparativeparameters were established for the purpose ofanalysing all the evaluation processes and resultsobtained. Considering the shortcomings mentionedin Section 1, the most significant parametersconsidered were: evaluation process repeatability;results invariability, and comparability of the resultsobtained. Additionally, the execution time of thedifferent evaluations, evaluator training and what,for each method, is considered as the object underassessment was also analysed. The experimentationdesign and results obtained are described in moredetail in the following subsections.

5.1. Experimentation design

The experimentation design includes a series ofpreliminary conditions and the activities neededto plan and control the experimentation. Theconditions considered are specified in the following:

1. Establish the hypotheses of the experimen-tation. The objectives pursued are to determinewhether the proposed method improves onthe shortcomings detected in the methods ofassessment shown in Table 4 and verify thatthis method of assessment is applicable forevaluating small and medium-sized businesses.Therefore, to assure that the experimentationwas representative, a series of assessments hadto be executed applying the following methods:CBA IPI, ISO 9000 Certification, and the methoddescribed in this paper. The criteria considered


22

in this selection was the method representa-tiveness and the volume of publicly availableinformation. Specifically, the CBA IPI assess-ment method was selected because, like theproposed method, it is oriented to softwareprocess improvement. For purposes of simpli-fication, this method will be denoted as CMMin this section.

2. Calculate the sample size and select the organi-zations to be evaluated. The experiment inquestion conforms to a binomial distributionwhose parameters are unknown. Small sampletheory has been used in order to establishsample size. This theory proves that for abinomial distribution of unknown variance andan infinite population, nine test cases aresufficient to estimate the value of the parameterp of the distribution with an error of 2.36%and a confidence level of 95%. As the presuppo-sitions of the above proof are identical to thoseconsidered in the experimentation, the designwas based on this result, and only ninecompanies were selected to execute the assess-ments. The companies selected had to meetthree requirements: the organization must havea quality system in place, necessary forexecution of an ISO 9000 Certification; thecompany must accept the execution of severalassessments; and the teams of evaluators mustbe independent of the organization evaluated,in order to perform the assessments as objec-tively as possible.

3. The following conditions were considered toassure that the results were not conditionedby human resources:I The methods are not to be applied simul-taneously within the same period of time inorder to prevent a strong interaction withmembers of the organization leading to dis-torted responses as a result of intervieweesand/or respondents memorizing the answersthey should give.I It is essential that a team of evaluatorsshould not execute two assessments using thesame method at one company. One team mustapply all the methods at different companies.

The experimentation was designed on the basisof these conditions. The activities performed todevelop the design are described below:

1. Establish the parameters of the experimen-


Table 11. Comparative parameters of the experimentation

Repeatable process Analysis of the evaluation processes in order to determine how repeatable each one is. Thisinvolves examining and comparing both the activities and other components of the assessment.

Results invariability Analysis of the results obtained in the different assessments using the same method in order toverify how coincident they are.

Comparable results Results analysis in order to verify their comparability.Target considered Analysis of what, for each method and assessment, is considered as the object under

evaluation.Execution time Analysis of the time for preparing, executing and writing the reports of each assessment.Evaluator training Analysis of the training time required for the different teams of evaluators to be able to execute

an assessment according to a given method.

tation. The entire set of parameters consideredin this experimentation are defined in Table 11.The evaluation processes have been analysedon the basis of Table 12, which shows how thegeneral phases of an assessment match up withthe activities proposed by each method. Allthe references to time will be specified compara-tively, taking the least time identified as areference point.

2. Selection of teams of evaluators. For the pur-pose of assuring the objectivity of the exper-imentation, the following requirements wereestablished for setting up the teams: the evalua-tors must not be acquainted with the softwareprocess assessment methods to be used; noneof the evaluators will be related to the organiza-tion to be assessed; the evaluators must bevery knowledgeable about the software process

Table 12. List of different activities of the evaluation processes analysed

General phases Evaluation method processesISO 9000 CMM Proposed method

Evaluation preparation Audit start up. Plan and prepare the Planning process:Audit preparation. assessment. Establish assessment goals.

Define evaluation components.Analyse target.Plan assessment.

Evaluation execution Opening meeting. Receive presentations. Examination process:Examination Conduct interviews. Apply questionnaire.

Consolidate data. Interviewing.Analyse documentation.Analyse all data obtained.

Report(s) writing and Closing meeting. Deliver draft findings. Decision-making process:submission Audit documents. Make rating judgements. Synthesize and compare data.

Deliver final findings. Prepare, submit and circulateProduce reports. report.

Complete assessmentdocumentation.


23

and potentially applicable technology; all theevaluators will work full time; one of theevaluators will assume the role of assessmentmanager; the teams of evaluators will notinteract. Taking into account these requirementsand the size of the organizations to be evalu-ated, four persons were selected for each group.All the evaluators have received higher edu-cation.

3. Generic description of each organization underassessment. Each team of evaluators was givena generic description of the organization aspreliminary working material. This descriptioncontains data such as: business purpose;environment; size of the company; age; culture;structure; human resources; technologicalresources, and identification of the currentprojects under development. These data were


collected by means of a questionnaire put afterthe managers of the organizations had agreedto be evaluated. The objective was for all theteams to have the same preliminary knowledgeof the companies to be evaluated.

4. Assessment control. For the purpose of compar-ing the evaluation processes, data need to becollected about all the tasks performed by theteams. Therefore, a summary form was definedfor each assessment, as shown in Table 13.The manager of the assessment is the personresponsible for providing the data required andfor delivering all the products developed duringthe assessments such as schedules, reports etc.

This experimentation also planned to control thetraining period in order to assure that all the teamshad the same documentation available for studyingthe different methods of assessment. A series ofbibliographic references and basic documentationon each method were selected for this purpose.This material was delivered at an informativemeeting with the teams. Finally, other informativemeetings were held at all the companies to acquaintthe practitioners and managers of the organizationswith the objectives and design of this experimen-

Table 13. Assessments summary form

General phases Assessment dataAssessment method: Team of evaluators: Manager:

Training Phase duration:Incidents:

Assessment preparation Activities performed: Products outputted: Incidents:Activity X Product associated with Incident activity X

activity X. . . . . . . . .Phase duration:Incidents:

Assessment execution Activities performed: Assessment technique(s) Products outputted: Incidents:applied:

Activity Y Assessment technique A Product associated Incident activity Ywith activity Y

. . . . . . . . .Phase duration:Incidents:

Report(s) writing and Activities performed: Synthesis technique(s) Products outputted: Incidents:submission applied:

Activity Z Synthesis technique B Product associated Incident activity Zwith activity Z

. . . . . . . . . . . .Phase duration:Incidents:


24

tation and the generic description of the assessmenttechniques to be applied.

5.2. Results analysis

After analysing all the data supplied by theassessment managers, a series of tables weredrawn up, which outline the findings of thisexperimentation. For the purpose of briefly describ-ing the approach used to prepare these summarytables, an extract of the results analysis for a particularorganization (hereinafter called ‘organization A’)will be discussed. Finally, the summary tables forthe other 8 companies evaluated are shown inTable 17. For a full analysis see Lopez (1997).

Organization A is a medium-sized company,which is representative of the companies selectedfor this experimentation. The results of the differentassessments were analysed as follows. First, com-parison of the process and results correspondingto the same method using the parameters: repeatableprocess; results invariability, and comparableresults. For example, Table 14 shows very brieflyan extract of these comparisons for the proposedmethod. Similar tables were developed for the


Table 14. Comparison of the results of the assessments as per the proposed method (PM)

Method: proposed method (PM)Parameter Comparison

Repeatable process Analysing the activities performed as part of the two assessments and the components of theassessment, it follows that the process is repeatable. Some insignificant variations were detected.

I Project selection. In the two assessments, four projects were selected of which threecoincided. The analysis of the others shows that, according to the parameters used, theyare practically identical and any of them is representative of the organizational productionprocess. The two teams applied the clustering technique, specified by the method.

I Criteria. The criteria specified by the method are reflected in the design document. Thetwo teams had no problems in identifying and understanding the meaning of the criteriato be assessed.

I Yardstick. Before executing the assessment, the two teams adapted the software process modelto take into account the standards, methodologies and procedures used in the organization.This activity did not take long and the result is the same in both cases. For any processeswhose execution was not governed by a procedure or methodology, the two teams appliedthe specification in the software process model used as a yardstick in these assessments.

I Assessment techniques. The two teams applied the techniques specified by the method. Ittook longer to prepare these techniques than in the CMM and ISO 9000 evaluations. Themain causes were: the scope of the assessment, as it is necessary to analyse the functionaland organic structure of the company, job descriptions and technology applied, as specifiedby the method; and the preparation of the techniques, templates and formats for datacollection. The two teams used the computer as a basic support for data collection. All thedetailed development of the techniques was based on the content of the software processmodel used as a yardstick. Variations were detected in the format of the questions, but thecontent is largely coincident. Therefore, the manner in which the question is posed varies,but what is asked or, ultimately, what criteria are being assessed does not.

I Synthesis techniques. The two teams applied the techniques specified by the method. Theuse of computer support facilitated data organization and comparison with the softwareprocess model. Comparing the synthesis techniques used by the two teams, no majordifference in these techniques was found.

I Activities. The set of activities performed coincide with the specifications of the method.Only small variations were detected in the examination process when the tasks to beperformed were detailed on the basis of the people to be interviewed and/or surveyed andthe documentation to be inspected. However, these are not significant differences. The twoteams adhered to the sequentiality and list of activities defined by the method.

Results invariability The results obtained by the two teams are highly coincident with regard to both structure andcontent. They are not exactly the same as some variability was detected caused by failure todefine threshold values associated with the assessment techniques for each criterion; that is,when to consider that the pairs [criterion, datum/information] meet the specifications in thesoftware process model used as the yardstick in the assessment. With regard to therecommendations, for each coincident criterion, they are the same. The process assessmentscales and criteria defined in the method were used to develop these recommendations.

Comparable results Despite the small variations detected, the results are considered comparable with each other.The two reports have the same structure and are highly coincident. This means that the resultsof various assessments at different levels – entire organization, hierarchical levels, general issuesor more specific criteria – can be directly compared.

CMM and ISO 9000 Certification. Second, generalevaluation of all the assessments, according todifferent methods, executed in the organization; thisinvolves a process of comparison considering allthe parameters shown in Table 11. Table 15 showsa summary of the analysis of each of theseparameters. As an end product, we get the summaryCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

25

table of the experimentation of this organizationA, shown in Table 16.

Similarly, another two tables, structured in thesame manner, were put together: one for the CMMand other for the ISO 9000 Certification. The resultsshown in the first three rows of Table 15 wereobtained by comparing all these tables.


Table 15. Comparison of all the assessments executed in organization A, using different methods

Parameters of the Results obtained from the comparisonexperimentation

Repeatable process From the analysis of the comparative tables of the CMM, ISO 9000 and PM, it follows that thePM and ISO 9000 are the methods with a repeatable process. The CMM assessments havegreater variability.

Results invariability Similarly, these tables show that the PM and ISO 9000 output less variable results. The finalconclusions of the CMM assessments are more variable.

Comparable results The classification of the methods by results comparability (in descending order) is: ISO 9000,PM and CMM.

Execution time The summary forms, plans and design documents were analysed in order to obtain executiontime. The result, expressed in decreasing order of time, is as follows: CMM, PM and ISO 9000.These results are shown comparatively, taking the least time consumed by one method as thereference point. According to the data specified in the bibliography of the CMM and ISO 9000methods, the values obtained do not differ significantly from the average duration of theseassessments.

Target considered The ISO 9000 Certifications are confined to assessing the organization’s quality system.Therefore, it is the most limited method, as it does not cover other factors that have an impacton software production. The results can only be analysed at the organizational and not atproject level.The scope of the CMM assessments was the management process. Neither the technical process,human resources, nor all the technological resources used in the organization were included. Thematurity level is associated with the organization as a whole and not with the projects evaluated.The PM covers a comparatively bigger target: it includes the assessment of human andtechnological resources and a more extensive range of processes at three levels: technical,management and strategic. It is also more versatile with regard to the analysis of the resultsobtained, because they can be classed by various factors: human resources, technologicalresources, processes, process levels, organizational level or by projects.

Evaluator training The comparative training time, in decreasing order, is: CMM, ISO 9000, PM.Initially, the CMM study is relatively quick; however, the in-depth understanding of themethod takes more time than the others. The meetings of members of the teams was a keyelement for facilitating training, as they could discuss and share what each evaluator hadlearnt. From the analysis of the activities executed and the decisions that the teams had tomake, it follows that the evaluators must have quite a lot of experience in and fairly detailedknowledge of the software process and the concept of evaluation. Therefore, the CMM is notan appropriate method for application by an inexperienced team.Similarly, the ISO 9000 Certification study is relatively quick at the beginning. However, all theteams stated that some requirements were difficult to understand and the documents to beinspected were difficult to identify. Like the CMM, it requires more evaluator preparation thandetected for the PM.The training period required for the PM is slightly less than for ISO 9000 Certification. Like theCMM, the meetings of members of the teams were an important component of training.

Table 16. Summary table of the experimentation carried out atorganization A

Parameter Best suited Intermediate Least suitedmethod method method

Repeatable process PM ISO 9000 CMMResults invariability PM ISO 9000 CMMComparable results ISO 9000 PM CMMTarget PM CMM ISO 9000Evaluator training PM ISO 9000 CMMExecution time ISO 9000 PM CMM


26

Table 16 shows a summary of the experimen-tation carried out at organization A, based on theseresults. The rows detail the parameters consideredand the columns contain an assessment for eachparameter of the best suited method, intermediatemethod and least suited method.

This table shows that the proposed methodemerges as the best suited method for mostparameters. For example, it was rated as thebest suited method for the parameter ‘repeatableprocess’. On the other hand, the CMM was foundto be the least suited method. This is mainly due to


the fact that its evaluation process is not very rigorousand that it requires specially trained evaluators.

This analysis process was applied to all thecompanies in the sample to get the summarytable shown in Table 17. All the parameters wereconsidered to be equally important in this analysis.The second column of the table shows, for eachparameter and for each software process assessmentmethod (PM, ISO, CMM), the following data:

I Name of the company (A, B, C etc.) at whichthe method appears as the best suited method.

I Number of times that the method appears asthe best suited method (right-hand side ofthe cell).

For example, for the parameter ‘repeatable pro-cess’, the proposed method was rated as the bestsuited at seven of the organizations (A, B, C, E, F,H, I), whereas ISO 9000 Certification was the bestsuited at two organizations (D and G) and theCMM was not rated as the best suited method atany of the organizations evaluated. The methodmost often classed as best suited is highlighted initalics and the best rated method for each parameterin bold. Column 3 shows the same analysis forthe organizations in which each method appearsas the intermediate method. Finally, the fourthcolumn shows the organizations in which eachmethod appears as the least suited; in this case,the method highlighted in italics is the one whichwas least suited on most occasions. The bottomrow contains a summary, showing the appearancesof each method for each value (best suited, inter-mediate and least suited) in descending order. Forexample, the value of the proposed method as thebest suited method is 34, which is the sum of thevalues specified in the cells of this column, thatis, 71415191910.

Analysing Table 17, the proposed method isfound to be the best suited method for four ofthe parameters considered: repeatable process;comparable results; target, and evaluator training.With regard to results invariability, the proposedmethod and ISO 9000 Certification appear as the bestsuited methods, each on four occasions. However, itis not the best suited method with regard to‘execution time’, because these evaluations are moretime consuming than ISO 9000 Certifications.

With regard to ISO 9000 Certification, its evalu-ation process is found not to be the most repeatableone. However, the invariability of the results ofCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

27

the certification are comparable with those obtainedwith the proposed method. Similarly, the compar-ability of these results is fairly close to that obtainedwith the proposed method. Another positivecharacteristic of these certifications is that theyare less time consuming. However, this methodconsiders a more restricted target and, also, calls formore evaluator training than the proposed method.

Finally, with respect to the CMM, it was foundto be the best suited method, considering theparameters ‘results invariability’ and ‘comparableresults’, at only one organization: I. All the othervalues obtained in this experimentation rate it asan intermediate or least suited method. Thus, itappears as an intermediate method on threeoccasions with regard to evaluation process repeat-ability and on two occasions with regard to thecomparability of the results obtained. This methodalso encompasses a bigger target than ISO 9000Certifications. As far as execution time is concerned,CMM evaluations are executed in a similar timeto those of the proposed method. However, it isthe least suited method for the parameter ‘evaluatortraining’, because it is the method that requiresmost evaluator training.

In conclusion, this experiment has made itpossible to check, considering the parameters underanalysis, that the proposed method is suitablefor evaluating small and medium-sized Spanishcompanies. Even if the company is very small, theyardstick can be adapted by removing the partsthat are not applicable to the organization. It is alsothe only method that covers the joint assessmentof processes, technology, human resources andorganizational design. On the other hand, thisexperimentation revealed a need to refine theassessment and synthesis techniques applied.Specifically, the documentation inspection tech-nique has to be refined, because the application ofthis technique was found to vary in some evalu-ations. Therefore, we plan to develop forms thatinclude the parameters that should be inspected foreach criterion in which documentation inspectionhas to be used. Another recommendation would bethe rigorous definition of threshold values associatedwith the assessment techniques of each criterion.These values will make it possible to systematizethe task of deciding when a value obtained for acriterion is greater or equal to the specifications ofthe yardstick in the synthesis techniques.


Tab

le17

.Su

mm

ary

tabl

eof

the

expe

rim

enta

tion

carr

ied

out

atth

eni

neco

mpa

nies

inth

esa

mpl

e

Par

amet

ers

Bes

tsu

ited

met

hod

Inte

rmed

iate

met

hod

Lea

stsu

ited

met

hod

Rep

eata

ble

proc

ess

PM

:A,B

,C,E

,F,H

,I7

PM

:D,G

2P

M:

0IS

O:D

,G2

ISO

:A,B

,E,H

4IS

O:C

,F,I

3C

MM

:0

CM

M:C

,F,I

3C

MM

:A,B

,D,E

,G,H

6R

esul

tsin

vari

abili

tyP

M:A

,B,G

,H4

PM

:C,D

,E,F

,I5

PM

:0

ISO

:C,D

,E,F

4IS

O:A

,B,G

,H4

ISO

:I1

CM

M:I

1C

MM

:0

CM

M:A

,B,C

,D,E

,F,G

,H8

Com

para

ble

resu

lts

PM

:B,C

,E,F

,H5

PM

:A,I

2P

M:D

,G2

ISO

:A,D

,G3

ISO

:B,C

,E,F

,H5

ISO

:I1

CM

M:I

1C

MM

:D,G

2C

MM

:A,B

,C,E

,F,H

6T

arge

tP

M:A

,B,C

,D,E

,F,G

,H,I

9P

M:

0P

M:

0IS

O:

0IS

O:

0IS

O:A

,B,C

,D,E

,F,G

,H,I

9C

MM

:0

CM

M:A

,B,C

,D,E

,F,G

,H,I

9C

MM

:0

Eva

luat

ortr

aini

ngP

M:A

,B,C

,D,E

,F,G

,H,I

9P

M:

0P

M:

0IS

O:

0IS

O:A

,B,C

,D,E

,F,G

,H,I

9IS

O:

0C

MM

:0

CM

M:

0C

MM

:A,B

,C,D

,E,F

,G,H

,I9

Exe

cuti

onti

me

PM

:0

PM

:A,C

,F,G

,I5

PM

:B,D

,E,H

4IS

O:A

,B,C

,D,E

,F,G

,H,I

9IS

O:

0IS

O:

0C

MM

:0

CM

M:B

,D,E

,H4

CM

M:A

,C,F

,G,I

5Su

mm

ary

PM

:34

ISO

:22

CM

M:3

4IS

O:1

8C

MM

:18

ISO

:14

CM

M:2

PM

:14

PM

:6


28


6. CONCLUSIONS

This paper presents a software process assessmentmethod oriented to software process improvement.Evaluation Theory was used to develop the pro-posed method. For this purpose, firstly, the bestsuited evaluation method for assessing the softwareprocess (control-oriented evaluation) was selectedand, secondly, all the components of the evaluationwere developed explicitly: target; criteria; yardstick;assessment techniques; synthesis techniques, andevaluation process. Thus, we obtained a morerigorous and reliable software process assessmentmethod. None of the current software processassessment methods and software process evalu-ation methods was developed taking into accountEvaluation Theory.

A functional analysis of the software processhad to be performed to determine the factorsincluded in an evaluation in order to develop themethod. As a result of this analysis we selected fourfactors (processes, human resources, technologicalresources, and organizational design), which arejointly evaluated in the proposed method.

However, the line of research is not yet complete.It remains to refine the assessment and synthesistechniques and develop a tool that automates thisassessment method.

Finally, as a supplement to this method, wepropose to develop software process improvementprogrammes based on the assessment results;develop assessment components related to thesoftware maintenance process, and a softwareproduct evaluation method. These lines of researchwould support the organization’s software processand product analysis and software processimprovement, and, more importantly still, theywould bring together two areas which have beeninvestigated separately in the past: software pro-duct evaluation and software process assessment.

REFERENCES

Ares J, Juristo N, Lopez M, Garcıa R. 1999. A formalmethod of software process evaluation. Proceedings ofthe 11th Software Engineering Process Group Conference(SEPG’99), Atlanta, GA, USA, 8–11 March, SoftwareEngineering Institute.

Bach J. 1994. The immaturity of the CMM. AmericanProgrammer 7: 13–18.


29

Bicego A, Khurana M, Kuvaja P. 1998. BOOTSTRAP 3.0-Software Process Assessment Methodology. Proceedingsof the Sixth International Conference on Software QualityManagement, Springer-Verlag, London.

BootCheck. 1997. BootCheck Self-Assessment Tool(Version 3.0). Http://www.sqs.de/pages/welcboot.htm.

Byrnes P, Phillips M. 1996. Software Capability Evalu-ation. Version 3.0. Method Description. Technical ReportCMU/SEI-96-TR-002, Carnegie Mellon University,Software Engineering Institute, Pittsburgh.

Caputo K. 1998. CMM Implementation Guide. Choreograph-ing Software Process Improvement. Addison-Wesley, Read-ing, MA.

Cattaneo F, Fuggetta A, Lavazza L. 1995. An experience inprocess assessment. Proceedings of the 17th InternationalConference on Software Engineering (ICSE’1995). ACMPress Seattle, WA, USA, 23–30 April, 115–121.

Craigmyle M, Fletcher I. 1993. Improving IT effectivenessthrough software process assessment. Software QualityJournal 2: 257–264.

Curtis B, Hefley WE, Miller S. 1995. People CapabilityMaturity Model. Maturity Model CMU/SEI-95-MM-02, Carnegie Mellon University, Software EngineeringInstitute, Pittsburgh.

Dunaway DK, Masters S. 1996. CMMSM-based appraisalfor internal process improvement (CBA IPI): Methoddescription. Technical Report CMU/SEI-96-TR-007, Car-negie Mellon University, Software Engineering Insti-tute, Pittsburgh.

Emam KE, Drouin J-N, Melo W. 1998. SPICE: TheTheory and Practice of Software Process Improvement. IEEEComputer Society: Los Alamitos, CA.

Hefley WE, Curtis B. 1998. People CMM-based assess-ment method description. Technical Report CMU/SEI-98-TR-012, Carnegie Mellon University, Software Engin-eering Institute, Pittsburgh.

Herbsleb J, Zubrow D, Goldenson D, Hayes W, PaulkM. 1997. Software Quality and the Capability MaturityModel. Communications of the ACM 40: 30–40.

House ER. 1980. Evaluating with Validity. Sage Publications.

Humphrey WS. 1989. Managing the Software Process.Addison-Wesley, Reading, MA.

Humphrey WS. 1998. Three dimensions of process improve-ment. Part III: The team process. Crosstalk 11: 14–17.

ISO-9001. 1994. ISO 9001:1994. Quality systems. Modelfor quality assurance in design, development, production,


installation and servicing. International Organisationfor Standardization.

ISO-9000-3. 1994. ISO 9000-3. Quality management andquality assurance standards. Part 3: Guidelines for theapplication of ISO 9001 to the development, supply andmaintenance of software. International Organisationfor Standardization.

ISO-10011, 1993. ISO 10011. Guidelines for auditingquality systems. Parts 1, 2 and 3. International Organis-ation for Standardization.

ISO-15504. 1996. ISO/IEC 15504: Information technology-software process assessment. International Organisationfor Standardization.

Khodabandeh A, Palazzi P. 1995. Software development:people, process, technology. CERN ECP Report 95/5.Http://www.cern.ch/PTGroup/Papers/Sopron94.

Kuvaja P, Simila J, Krzanik L, Bicego A, Koch G,Saukkonen S. 1994. Software Process Assessment andImprovement. The BOOTSTRAP Approach. Blackwell Pub-lishers: Oxford.

Lopez M. 1997. Formal model of software process evalu-ation: PhD Dissertation, Universidade da Coruna, Coruna.

Madaus GF, Scriven M, Stufflebeam D. 1983. EvaluationModels: Viewpoints on Educational and Human ServicesEvaluation. Kluwer-Nijhoff Publishing: Boston.

Paulk MC, Weber CV, Chrissis MB. 1995. The CapabilityMaturity Model: Guidelines for Improving the SoftwareProcess. Addison-Wesley, Reading, MA.

EDITOR’S COMMENT

This substantial paper addresses the need for software process assessments to have a sound theoreticalbasis. The basis proposed is Evaluation Theory, which is shown to provide a set of six criteria by whichassessment models and methods should be judged or developed. Applying those criteria yields aproposed “rigorous and reliable” assessment method, covering all the main dimensions of capability(process, technology, people and organisation). This method was then tested (on the basis of a formalexperimental design - a rarity in our field!) against ISO 9001/9000-3 and the CMM for Software (CBA IPI).

Clearly this is a paper for developers and students of models and methods, not for process practitioners.Equally clearly it is not perfect - the method, for instance, could be said to be too heavyweight. Thereis too much in it to attempt any greater summary or comment in a few lines. What seems mostimportant is that papers of this kind should be not only written and published (which itself happenstoo infrequently) but that they should also be subject to scientific critique. The software process fieldurgently needs to undergo a renaissance, so that scholarly and theoretical contributions are judged onthe basis not of religious affiliation but of open scientific enquiry and debate.

—CTCopyright 2000 John Wiley & Sons, Ltd. Softw. Process Improve. Pract. 2000; 5:3–30

30

Perry DE, Staudenmayer N, Votta LG. 1994. People,organizations and process improvement. IEEE Trans-actions on Software Engineering 7: 36–45.

Rout TP, Neilson MP, Gasston JL. 1994. Experienceswith the use of different approaches to software processassessment. Proceedings of the 2nd International Confer-ence on Software Quality Management. ComputationalMechanics Publications: Edinburgh, Scotland.

Saiedian H, Kuzara R. 1995. SEI Capability MaturityModel’s impact on contractors. IEEE Computer 8: 16–26.

Schmauch CH. 1995. ISO 9000 for Software Developers.ASQC Quality Press: Milwaukee, Wisconsin.

Scriven M. 1991. Evaluation Thesaurus. Sage Publications:Newbury Park, CA.

Shadish WR, Cook TD, Leviton LC. 1993. Foundations ofProgram Evaluation. Theories of Practice. Sage Publi-cations: Newbury Park, CA.

Sommerville I, Rodden T. 1995. Human, social andorganisational influences on the software process. Techni-cal Report CSEG/2/1995, Cooperative Systems Engineer-ing Group, Lancaster University. Web site:Http://www.comp.lancs.ac.uk/computing/research/cseg/95—rep.html

TickIT. 1995. The TickIT Guide. A Guide to SoftwareQuality Management System Construction and Certifi-cation to ISO 9001. British Standards Institution.

Trillium. 1994. Trillium. Model for Telecom ProductDevelopment and Support Process Capability. Bell Canada.

A more rigorous and comprehensive approach to software process assessment

Documents

Transcript of A more rigorous and comprehensive approach to software process assessment