Survey on Requirements Engineering Tools

Survey on Requirements Engineering Tools 1 /26

Survey on Requirements Engineering ToolsREFSQ Industry Track 2011

Juan M. Carrillo de [email protected],Joaquín Nicolás,

José L. Fernández Alemán,Ambrosio Toval

Universidad de Murcia, Spain

Christof EbertVector, Stuttgart, Germany

Aurora VizcaínoUniversidad de Castilla-La

Mancha, Spain

Essen, Germany, March 29, 2011

Contents

1 Introduction

2 Related work

3 Classification framework for RE tools’ capabilities

4 Research methodologyResearch goalsInstrumentationExperimental procedure

5 ResultsParticipantsExperimental resultsDiscussionThreats to validity

6 Conclusions and future workConclusionsFuture work

Introduction

Benefits from automated support to RERE tools and support to the RE processRE tools and developer’s expectationsWhat are the RE tools’ desirable features?How are they supported by current RE tools?Description of the state-of-the-art on RE tools

Current RE tools identificationFramework selection and adaptationSurvey conduction

Updated overview on RE tools’ capabilities and potentials

Introduction

Related work

Surveys on RE

[Liu et al., 2010][Winkler and von Pilgrim, 2010][Carlshamre et al., 2001][Johansson et al., 2001][Benslimane et al., 2007]

Surveys on RE tools

[Zowghi and Coulin, 2005][Alenljung and Persson, 2008][Hall, 2008][Gregoriades and Sutcliffe, 2005][Maiden et al., 2006][Portillo Rodríguez et al., 2010]

Classification framework for RE tools’ capabilities

Regulations and guidelines

ISO/IEC TR 24766:2009 Technical Report (TR) of Type 2Set of capabilities that the RE tools should supportSix major categories

Requirements elicitationRequirements analysisRequirements specificationRequirements V&VRequirements managementOther tool capabilities

Supplements ISO/IEC 14102:2008 International StandardEvaluation of CASE tools

Classification framework for RE tools’ capabilities

RE tools’ capabilities

ISO/IEC TR 24766 tool capabilities

Category AmountRequirements elicitation 37Requirements analysis 36Requirements specification 16Requirements verification and validation 34Requirements management 17Other tool capabilities 17Total 157

Research methodology

Research goals

Goal/Question Metric (GQM) framework

Goal/Question Metric (GQM) framework[Basili and Rombach, 1988]GQM template [Basili et al., 1999]

Goal: To investigate the state-of-the-art on RE tools using aquestionnaire aimed at software vendorsQuestion: Do current RE tools address industry challenges?Metric: Capabilities of RE tools (questionnaire scores)

Instrumentation

Instrumentation I

Databases hosting RE tools lists (July, 2010–August, 2010)

Database AmountIan Alexander 67Alarcos Research Group 7INCOSE 34Ludwig Consulting Services 40Qaguild 7Volere 71@WEBO 41Total sample size after discarding invalid tools 94

Instrumentation

Instrumentation II

DESMET [Kitchenham, 1996] method for evaluating softwareengineering methods and tools

Feature Analysis—Survey

146-items questionnaire6 ISO TR 24766 categories of features plus modelling plustraceability157 ISO TR 24766 features → 126 questions126 technical questions + 20 general, administrative questions

Web-based survey using LimeSurvey

Instrumentation

Instrumentation II

Instrumentation

Instrumentation II

Experimental procedure

Questionnaire and survey system preparation:September, 2010–November, 2010Tool representatives filled in the survey:December 20, 2010–January 15, 2011Follow-up email and deadline extension to February 7, 2011

Results

Participants

Participants I

38 participants out of 94 candidates invited (40.42%)

Acclaro DFSSAligned ElementsAvenqo PEPBlueprintBright Green ProjectsCaliber RMCameoRequirements+

CASE SpecCognition CockpitCradleG-MARCinteGREATIRQAjUCMNav

Leap SEMacA&D/WinA&DMKS IntegrityPACEPolarionRequirementsPsodaQFDcapture

Results

Participants

Participants II

QPackRaQuestRational DOORSReqManReqtifyRequirementsComposer

RTIMERequisiteProRMTrakRommanaScenario PlusSpiraTeam

TestTrack RMTopTeam AnalystTraceCloudTrackStudioVisibleThread On-premise/On-demand

Results

Experimental results

Administrative information I

Year of first release Year of last release

No. of tools

2008 2009 2010 2011

No. of tools

Results

Administrative information II

Platform required Type of license

Windows

Web-based

Windows/ UNIX/Linux

Windows/Mac OS/

UNIX/Linux

Windows/Mac OS

Windows/Linux

Windows/Mac

OS/Linux

Proprietary and not-

Proprietary and free

Open-source and

not-free

Results

Administrative information III

Cost per individual license Amount of licenses in use

No answer Less than 100

100 to 500 501 to 1000

More than 1000

No answer 0 to 100 101 to 1000

1001 to 10000

More than 10000

No. of licenses

Results

Correlation between variables

11 variables—one for each category of features plus global scoreplus cost per individual license plus amount of licenses in useBivariate correlation tests (Pearson’s Correlation Coefficients)Correlation is significant at the (**) 0.01 level/(*) 0.05 level(1-tailed)

Strong direct correlation between each distinct category of featuresCost per individual license–analysis (0.336*), modelling (0.404*),traceability (0.329*), specification (0.545**) and global score (0.358)Number of licenses in use–other tool capabilities (0.513**), cost perindividual license (0.243) and global score (0.183)

Results

Technical information I

Tools’ scores

Very high High Medium Low Very low

Elicitation Analysis Specification Modelling V&V Management Traceability Other tool capabilities

Results

Technical information II

Global score (level of accomplishment of the entire ISO/IEC TR24766)—only calculated for those tools participating in all categories

Very high High Medium Low Very low

Global

Results

Discussion

Scenario 1: Requirements elicitation and V&V

Storing and managing templates for elicitation (57%), elicitationchecklists (60%), prioritization forms (57%)Providing OMG ReqIF/RIF compatibility (26%)Generating exception reports on verification/validation plancases↔ verification/validation procedures (57%/60%)Providing standard format for interfacing to verification/validationtools (57%/57%)Implementing some built-in requirements checks (50%)Both: Cockpit, Cradle, QPack, ReqtifyElicitation: MKS Integrity, Polarion RequirementsV&V: Aligned Elements, CASE Spec, G-MARC, IRQA, PACE,ReqMan, TraceCloud

Results

Discussion

Results

Discussion

Results

Discussion

Scenario 2: Requirements modelling and specification

Providing storage and display of BPMN (44%), goal models(39%), SysML artifacts (31%), DFDs (44%)Checking the document through spell checking, grammarchecking, data dictionaries, and acronym tables (60%)Generating the output of the specification in a finished form (65%)Complete loop between RE tool and formatted document (39%)Both: Cockpit, Cradle, PACEModelling: ReqtifySpecification: G-MARC, inteGREAT, MKS Integrity, QPack,DOORS, TraceCloud, VisibleThread

Results

Discussion

Results

Discussion

Results

Discussion

Scenario 3: Requirements traceability

Generating reports that compare current and previous versionswhen a source document is updated (55%)Tracing across the tools’ boundaries (55%)Tracing text to graphics (52%), graphics to graphics (47%),elements within graphics (42%), tables and cells within a table(36%)Generating reports of traceability attributes (60%)Best: CASE Spec, Cockpit, Cradle, G-MARC, inteGREAT, ReqtifyClose to the best: Avenqo PEP, IRQA, Polarion Requirements,QPack, TopTeam Analyst

Results

Discussion

Scenario 3: Requirements traceability

Generating reports that compare current and previous versionswhen a source document is updated (55%)Tracing across the tools’ boundaries (55%)Tracing text to graphics (52%), graphics to graphics (47%),elements within graphics (42%), tables and cells within a table(36%)Generating reports of traceability attributes (60%)Best: CASE Spec, Cockpit, Cradle, G-MARC, inteGREAT, ReqtifyClose to the best: Avenqo PEP, IRQA, Polarion Requirements,QPack, TopTeam Analyst

Results

Discussion

Scenario 4: Other features

Providing Eclipse support (42%)Providing data federation (28%)Providing an open data model (18%)

Results

Threats to validity

Internal validity (causal relationships)Mortality percentage (7.32%)Commitment of the RE tools representativesTruthfulness of the answers (Hawthorne effect)Triangulation techniqueQuestionnaire planning and design

External validity (generalizations)Participants are representative of the RE tools’ community

Results

Threats to validity

Internal validity (causal relationships)Mortality percentage (7.32%)Commitment of the RE tools representativesTruthfulness of the answers (Hawthorne effect)Triangulation techniqueQuestionnaire planning and design

External validity (generalizations)Participants are representative of the RE tools’ community

Conclusions and future work

Conclusions

The RE process is well-covered by current RE toolsConnection between the scores accomplished by the RE tools ineach category of featuresMore expensive tools offer better requirements analysis,specification, modelling and traceability supportNo association between the cost per individual license and theglobal scoreMore extended tools are stronger in ISO/IEC TR 24766 other toolcapabilitiesThe amount of licenses in use is not associated with the cost perindividual license nor with the global score

Conclusions

Future work

Support to GSD settingsConcrete RE capabilities

ElicitationModellingTraceability...

Dissemination of results

Future work

Thank you for your attention

Thank you!Any questions?

Thank you for your attention

Thank you!Any questions?

Additional information

Score calculation

Participation of the tool t in the category c :

participant (t , c) ={

true, NA(t , c) ≥ 0.5 · NQ(c);false, otherwise.

NA(t , c): number of answers of the tool t in the category cNQ(c): number of questions of the category cScore s of the tool t in the category c :

score(t , c) =(∑NQ(c)

q=1 score(t ,q))·4

score(t ,q) ∈ {0,1}: score of the tool t in the question q

∀s ∈ [0,4], discretisation(s) =

Very low, s ∈ [0,0.5];Low, s ∈ (0.5,1.5];Medium, s ∈ (1.5,2.5];High, s ∈ (2.5,3.5];Very high, s ∈ (3.5,4].

Score calculation

q=1 score(t ,q))·4

Score calculation

q=1 score(t ,q))·4

Bibliography

Bibliography I

Alenljung, B. and Persson, A. (2008).DESCRY: a method for evaluating decision-supportingcapabilities of requirements engineering tools.In Proc. of the 14th Int. Conf. on Requir. Eng.: Foundation forSoftware Quality, pages 52–57. Springer-Verlag.

Basili, V. and Rombach, H. (1988).The TAME project: towards improvement-oriented softwareenvironments.IEEE Trans. Softw. Eng., 14:758–773.

Basili, V. R., Shull, F., and Lanubile, F. (1999).Building knowledge through families of experiments.IEEE Trans. Softw. Eng., 25:456–473.

Bibliography

Bibliography II

Benslimane, Y., Cysneiros, L. M., and Bahli, B. (2007).Assessing critical functional and non-functional requirements forweb-based procurement systems: a comprehensive survey.Requir. Eng., 12:191–198.

Carlshamre, P., Sandahl, K., Lindvall, M., Regnell, B., and Dag,J. N. (2001).An industrial survey of requirements interdependencies insoftware product release planning.In Proc. of the Fifth IEEE Int. Symp. on Requir. Eng., pages84–92, Washington, DC, USA. IEEE Computer Society.

Gregoriades, A. and Sutcliffe, A. (2005).Scenario-based assessment of nonfunctional requirements.IEEE Trans. Softw. Eng., 31:392–409.

Bibliography

Bibliography III

Hall, R. J. (2008).A method and tools for large scale scenarios.Autom. Softw. Eng., 15:113–148.

Johansson, E., Wesslén, A., Bratthall, L., and Höst, M. (2001).The importance of quality requirements in software platformdevelopment - a survey.In Proc. of the 34th Annual Hawaii Int. Conf. on Sys. Sci.,volume 9, Washington, DC, USA. IEEE Computer Society.

Kitchenham, B. (1996).DESMET: a method for evaluating software engineering methodsand tools.Tech. Report TR96-09, Dept. of Computer Science, University ofKeele.

Bibliography

Bibliography IV

Liu, L., Li, T., and Peng, F. (2010).Why requirements engineering fails: a survey report from China.In Proc. of the 2010 18th IEEE Int. Requir. Eng. Conf., pages317–322. IEEE Computer Society.

Maiden, N., Seyff, N., Grunbacher, P., Otojare, O., andMitteregger, K. (2006).Making mobile requirements engineering tools usable and useful.In Proc. of the 14th IEEE Int. Requir. Eng. Conf., pages 26–35.

Portillo Rodríguez, J., Ebert, C., and Vizcaíno, A. (2010).Technologies and tools for distributed teams.IEEE Softw., 27:10–14.Winkler, S. and von Pilgrim, J. (2010).A survey of traceability in requirements engineering andmodel-driven development.Softw. and Sys. Modeling, 9:529–565.

Bibliography

Bibliography V

Zowghi, D. and Coulin, C. (2005).Requirements elicitation: a survey of techniques, approaches,and tools.In Aurum, A. and Wohlin, C., editors, Engineering and ManagingSoftware Requirements, pages 19–46. Springer.

Survey on Requirements Engineering Tools

Business

Transcript of Survey on Requirements Engineering Tools