Quantitative Non-Functional Requirements Publication
-
Upload
mista-ryan -
Category
Documents
-
view
222 -
download
0
Transcript of Quantitative Non-Functional Requirements Publication
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 1/8
An Approach To Qunantitative Non-Functional RequirementsIn Software Development
Andrew J. Ryan
School of Information Technology and EngineeringGeorge Mason UniversityMS 5D3 Fairfax, VA 22030
Abstract. Non-functional requirements are commonly calledthe qualitative aspects of a system -- testability, mobility, andscalability, to name a few. However, when taking a holisticview of a system, non-functional requirements take on aquantitative nature. This paper will describe the RequirementsHierarchy Approach (RHA), a quantifiable method to measureand manipulate the effect non-functional requirements have on
a system by capturing the utility of functional requirements.Through the use of agent-oriented programming, multi-attribute utility analysis (MAUA), and decision science theory,non-functional attributes (of a system) can be used ascontainers in an attempt to capture the broadest cross-sectionof functional requirements, based on stakeholder input. Thefinal result will be an optimal set of requirements that satisfythe stakeholder’s needs and are not in opposition to oneanother.
INTRODUCTION
Requirements drive any systems engineering effort. Most of the focus has been placed in the area of functionalrequirements. According to Alford, “In nearly every softwareproject which fails to meet performance and cost goals,requirements inadequacies play a major and expensive role inproject failure”(Alford 1979). Although written “requirementsinadequacies” it should read “functional requirementsinadequacies.” There has been a tremendous amount of moneyand effort earmarked to solve this issue. With the emphasis onfunctional requirements, it is hard to understand why systemsengineers still struggle to create a complete set of requirementsfor a stakeholder. One reason for this has been the inattentionto non-functional requirements (NFRs) and the affect theyhave on fulfilling functional requirements. This is most
notable in the omission of non-functional requirements by thetwo documents which are considered the guidelines for theEngineering of Systems in the United States: ElectronicsIndustry Association (EIA) 632 and the International StandardsOrganization (ISO) 15228. The choice of this topic is a smallstep in an attempt to give non-functional requirements a moreprominent role in the field of systems engineering.
BACKGROUND
There has been some prior work on non-functionalrequirements (NFRs) in systems/software engineeringKenneth Chung presented one of the earlier attempts to capture
knowledge in this domain (Chung 1993). His work was latefollowed up by Myopoulos et al. (Myopoulos 1999)Myopoulos suggested that all requirements be viewed as goalsEach goal would be an umbrella for related functional and non-functional requirements. As these requirements are satisfied orsatisficed1 the system goals are effectively satisfied. Missingfrom this formula is the impact the stakeholder can have on therequirements elicitation process and the gap that is oftenpresent between stakeholder vision and requirementsrepresentation.
One of the strong points of the RequirementsHierarchy Approach (RHA) is that it creates a visual toolwhich illustrates to stakeholders how their requirements relateto each other and the effect this may have on total systemperformance. Oftentimes, stakeholders have overlappingintentions (not to mention differences in opinion with thedevelopers) and the need for tradeoff becomes necessaryBarber et al. espouses this belief, noting “System stakeholdershave varying notions on the purpose of requirements gatheringand these differences may lead to communication problemsFurther complicating the communication process is the‘requirements gap’ that separates the developer from theclient” (Barber et al. 1997). The requirements gap is one of themain targets of this paper. Moreover, the RequirementHierarchy Approach (RHA) can lead to positive results inrequirements elaboration, requirements tradeoff, requirementsvalidation, and verification.
NON-FUNCTIONAL REQUIREMENTS EXPLORED
Non-functional requirements are best described as qualitybased requirements which can only be realized at the end of thesystem development process. The overall measurement of a
1 To get a result that is good enough. Initially coined by Herbert Simon in his1957 book Models of Man.
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 2/8
non-functional requirement is based on the functionality of thesystem as a whole, and cannot be viewed as independententities. The ISO has identified a set of quality characteristics(or non-functional requirements) which provides a solidtaxonomy for NFRs (see figure 1).
Figure 1. ISO/IEC 9126 Taxonomy of Quality
Requirements (ISO/IEC 1991)
Taking this a step further, non-functional requirements can bequantified by the percentage of functional requirements they
satisfy. That is, by placing non-functional requirements at thetop of the “requirements hierarchy” and allowing theirpresence to filter down into the functional requirements, ameasurable unit can be established. The rest of this paper willbe dedicated to explaining this process in greater detail.
ROLE OF NON-FUNCTIONAL REQUIREMENTS IN A
SYSTEMThere is much debate about the role non-functionalrequirements should play in the requirements elicitation andsystem development process. Systems (and software)engineers fall into two camps when it comes to this question,with the line of demarcation being the “qualitative” nature of
non-functional requirements. A qualitative requirement isanalogous to an immeasurable requirement, which in anydiscipline is something to be avoided at all costs. In recentyears, there has been a push to make non-functionalrequirements more quantitative in nature (Chung 1993, Franch1998, Jacobs 1999). One of the more promising ideas was putforth by a group at the University of Texas (Tran et al. 1999),which includes a CASE tool to view the decision process of their tool.
Stakeholders often describe the system they want builin layman’s terms. Usually citing specifications qualitativelywith accompanying verbiage describing how it should happenFor example, one requirement may be: “The system shall allowanonymous login for unrestricted functions.” A secondrequirement states: “The system shall have protective firewallto enforce varying levels of access.” Implicitly, thestakeholder is contracting two non-functional requirementsusability and security. For the most part, when viewedindependently, non-functional requirements can be addressedroutinely. However, there are times (such as the one abovewhen they present a clash which initiates a ripple affect felt byother requirements – in attempting to satisfy one requirementthe developer is negating another requirement.
Dealing with Requirements Clash. When two or more nonfunctional requirements directly contradict each other, in termof fulfillment, we call this a requirements clash. This type osituation often goes unnoticed unless a visual representationsuch as the Requirements Hierarchy Approach, is utilized as atool to model requirements. There are two approaches that canbe employed when this situation occurs:
1. Decrease the measure of satisficability of one or more ofthe requirements.
2. Use the requirements hierarchy as a basis for requirementstrade-off.
The first approach is the preferred method. Thepower of satisficing goals, comes in the ability to alter thesequence which satisfies a goal. Therefore, if a goal has fivesub-goals, of which three are satisfied, one can then reduce themeasurement for satisficability to three with customer
approval. By decreasing the impetus placed on satisficing agoal, it also decreases the dependency the requirements haveon one other. Doing this can eliminate the clash or at leasdepreciate it to a point that it is manageable.
The second approach involves meeting with thestakeholder and, based on the requirements and the soft goatree, to create a list of trade-off requirements. Theserequirements can then be re-prioritized to alleviate the clashamong the requirements.
The hidden danger is that the conflicting nature of thedesired functionality (impacted by NFRs) of a system is norealized until late in the development life cycle. By managingthese potential conflicts early, the developer protects the
success of the project from being compromised.When bringing a system into being, great measures
are taken to appease the stakeholder – to build a trustingrelationship and confidence that the completion of the projectis in fact a feasible scenario. This action usually appears inconcrete form by a developer insisting they can meet alrequirements. Taken at face value, there is nothingintrinsically wrong with this, until non-functional requirementare brought into the picture.
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 3/8
NON-FUNCTIONAL REQUIREMENTS IGNORED
Late in 1999, NASA’s Mars Climate Orbiter andPolar Lander were lost in Mars’ atmosphere due to inadequateattention paid to a pair of non-functional requirements:interoperability and testability. In terms of interoperability, theOrbiter was developed by separate companies: Lockheed
Martin Aeronautics (main contractor) and the Jet PropulsionLaboratory (JPL). In calculating thrust for the unit, LockheedMartin scientists used English measurements, while JPLscientists used the metric system. This oversight was onlyrealized after the probe was deployed, a definite cause of thefailure
Secondly, the modules that were developed for thePolar Lander were not as succinct as initially thought. Duringsystem testing, one module was excluded (there was a knownproblem with that module) and tests were run. At a later time,the excluded module was tested independently and finaldelivery was made. It was not until the system failed in spacethat that scientists recognized that the ommission of themodule during system testing (they only ran one full test),affected the functionality of the landing gear. These twomistakes cost the United States taxpayers $319M.
RHA METHODOLOGY
A non-believer in the importance on NFRs might question theirimportance by stating that systems have been built withoutmuch attention being paid to NFRs. It has been proven thatomission of non-functional requirements can be detrimental tothe success of a system. Alan Davis states, “Knowing how tospecify behavioral requirements for a software system is onlyhalf the battle. All applications, from the most trivial to themost complex, have additional requirements that define theoverall qualities or attributes to be exhibited by the resultingsoftware system” (Davis 1993). Even if these non-functionalrequirements are not explicitly defined, it is still up to thedeveloper to create a design which takes these into account.Through the creation of a requirements hierarchy, this step isingrained into the system development process.
Linking functional requirements to non-functionalrequirements is not a trivial undertaking. It is worth reiteratingthat this method is not intended to be an elicitation tool, but atool which aids in trade-off and understanding of requirementselaboration, tradeoff, validation, and verification. Moreover,many non-functional requirements will have to be impliedfrom functional requirements or reworded to betterdepartmentalize the requirement within the ISO/IEC 9126
taxonomy. It is the opinion of this author that all requirementscan be reasonably positioned in the hierarchy represented inFigure 1. Although duplication within a category is allowed, itis encouraged to explore why the requirement cannot be placedin one category versus the other. The following section willfurther build on this idea and the process which goes intocreating a quantitative aspect to NFRs. Figure 2 illustrates therecommended cycle when employing this methodology with astakeholder.
Meet with
Stakeholders
Domain Expert
Requirements
Engineer
Document
Results
Run
Simulator
Knowledge
Engineer
Get requirementswith weights
Parse weights amongst top
level requirements; Add
additional sub-characteristics
Create Taxonomical hierarchy
from requirements
Add heuristics;
Look for clashes
Show possible clashes; Maximize strategy;
Risk analysis
Present
results
START
Figure 2. NFR Analysis Cycle
RHA AND MULTI-ATTRIBUTE UTILITY ANALYSIS
Multi-Attribute Utility Assessment (MAUA), firsused by Adelman and Donnel (Adelman et al. 1981), wasinitially used to evaluate knowledge-based systems. MAUAuses scoring and weighting procedures to evaluate the overalutility of a system. Employing the hierarchy presented inFigure 3, it is possible to assign weights to each requiremen(on a scale from .01 [wish list] to 1.0 [mandatory]) and traverseupward until a final number is reached. The weight given toeach requirement can come either directly from the stakeholder
(preferably) or a domain (subject matter) expert.The final number to be evaluated is for the top-level
NFRs. Functionality, reliability, usability, efficiencymaintainability, and portability need to be assessed in relationto each other. That is, the entire system is deemed to be (at100% when all requirements are satisfied. This number is thesummation of the lower six NFRs. It is up to the domainexpert to allocate priority across the six categories, based onthe project requirements. Once this has been done, the processnow switches to the requirements engineer to fill in theskeleton.
The process of linking functional requirements to quality
based attributes is a very objective task. The requirementsengineer should work closely with the domain (subject matter)expert in doing this. It will be shown later in this paper (viaexperiment) that agreement (in a hierarchy) among multipleparties is possible even without communication. The ability tospeak with experts while creating the hierarchy will onlystrengthen the reliability of the hierarchy
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 4/8
Suitability
wt=0.40
R1 (.4)
R2 (.3)
R5 (.1)R20 (.2)
Validity wt=0.3
Accuracy wt=0.7
Accuracy
wt=0.60
Functionality
wt=0.15
Reliability
wt=0.25
Usability
wt=0.05
Efficiency
wt=0.20
Maintainability
wt=0.10
Portability
wt=0.25
SYSTEMD o m a i n e x p e r t s e ts w e ig h t s
D o m a i n e x p e r t c h o o s e c a t e g o r ie s
a n d s e t s w e i g h t s
R e q u i re m e n t s E n g i n e e r c h o o s e c a te g o r ie s
a n d s e t s w e i g h t s
S t a k e h o l d e r (s ) s e t w e i g h t s
Figure 3. Partial Requirements Hierarchy
QUANTITATIVE ANALYSIS OF RESULTS
It would be impractical to believe a staticrepresentation of non-functional requirements can solve anyproblems a requirements engineer may encounter. In using theRHA, subject matter experts and experiencedrequirements/knowledge engineers are invaluable. Without arigid taxonomy of NFRs to act as a baseline, familiarity withthis approach becomes essential.
While only experience can provide the requirementsengineer with the intuition to analyze the results, the followingis a prescribed routine, once the hierarchy has been created:
1. Comparison of each of the top level nodes(functionality, reliability, usability, efficiency,maintainability, portability)
a. A low score does not necessarily meanproduct failure. Portability may not be of relative importance to a stakeholder if thesystem is an electrical heating system for a
skyscraper; however, if it were a solarpowered generator, portability wouldbecome of prime importance to capture asmuch of the sun as possible.
2. Check for uniqueness among requirements:
a. It is the responsibility of the requirementsengineer to parse out requirements in anunbiased manner, regardless of duplication.
An inexperienced requirements engineer maywant to limit their hierarchy to the 28 sub-NFRs in the ISO/IEC model, while aseasoned requirements engineer can be moredetailed and have more categories.
b. While there is nothing intrinsically wrong
with requirements appearing in more thanone category, on a case-by-case instance itmay be necessary to reevaluate therequirement in terms of scope andfulfillment.
i. To avoid bias, if possible, havemore than one knowledge engineerdo this (include something aboutsensitivity analysis).
ii. Another approach can be splittingrequirements into sub-parts,therefore capturing individualfunctionality in each respective sub-
requirement.3. Take a heuristic view of the requirements breakdownand make suggestions to the stakeholder wheredeemed appropriate.
Having the entire team sit in on this evaluation process isalways a good idea so all disciplines and viewpoints arepresent to provide perspective.
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 5/8
PRESENTATION OF RESULTS TO STAKEHOLDERS
The Requirements Hierarchy Approach (RHA) lends itself more to prototype-driven development efforts (spiral,incremental, evolutionary) than iterative approaches (i.e.waterfall). But, it can be looped and used in a waterfall
process too. Regardless of the method used, there will be apoint in time when the stakeholder will reenter the picture.This meeting will give birth to opportunities for tradeoff talks,issues of satisficing versus satisfying, and can serve as abaseline for initial user acceptance. Stakeholders will beimpressed by the simplicity of the requirements hierarchy andwill be able to match their project goals against the aspirationlevels listed for each requirement. Figure 4 demonstrates theactions a stakeholder can take after the first iteration of thecycle.
THE REQUIREMENTS HIERARCHY APPROACH INPRACTICE
To demonstrate the effectiveness of the RHA, a two-part experiment was performed via a web-based questionnaire.The first part consisted of twenty-five requirements whichwere taken from a System Requirements Specification (SRS)used in a graduate level course at George Mason University.The participants were given the system’s Statement of Work (SOW) and then asked to rate the relative importance (on an 11point scale) of each requirement with respect to the SOW. In
the second part, selected respondents were then asked tocategorize each requirement using the first tier (the top six) ofthe ISO 9126 framework. The users were given official ISOdefinitions to fully understand the intended meaning of eachcategory. The results of the survey were then queried byprofession, gender, and education to determine trends (if any).
In the first part of the experiment, respondentscollectively identified the same requirements as moreimportant than others. Among those receiving the highesscores were: “The system shall operate on a universallyaccepted platform” and “Users shall receive 24 hour technicasupport.” Low importance were given to requirements dealingwith the inclusion of a voice recognition interface and theusage of Commercial off the Shelf (COTS) solutions wherepossible. In addition to the average of each relativeimportance, the variance was also calculated as a measure of
uncertainty among thetotal. Requirementswith high/low relativeimportance but high
variance are furtherexamined. Oneinteresting discoverywas that the entirescale (0-10) was noused by many
respondentsespecially those with
non-technicabackgrounds. Thisidea is not a new oneas math/sciencedisciplines use the
total scale much morethan their liberaarts/social science
counterpartsUnderstanding thestakeholder is an
important aspect of using theRHA.The second part of theexperiment tested the
respondents’ ability to place each requirement into their non-functional requirement categories. This exercise is importantas it forces the stakeholder to anticipate how their requirements
will affect the system. Stakeholders will often agree to arequirement in principle without realizing that it has a broaddefinition and may affect various functions of the system. Thisis especially common when the stakeholders represent variousdepartments within the organization. Respondents wereallowed to choose two categories: a first and a second choiceThe first choice was given ten (10) points and the second five(5) points. The results were then measured against a silvestandard that was established prior to conducting the surveyRespondents successfully matched the silver standard in 16 o
M e e t w i t h
S ta ke ho lde rs
D o m a i nE x p e r t
R e q u i r e m e n t s
E n g i n e e r
D o c u m e n t
R e s u l t s
R u nS i m u l a t o r
K n o w l e d g e
E n g i n e e r
R e q u i r e m e n t s
T r a d e o f f
Sat i s fy
R e q u i r e m e n t s
Sat is f ice
R e q u i r e m e n t s
- S t a k e h o l d e r u n d e r s t a n d s
l im i ta t ions on pro pose d se t o f
r e qu i re m e n t s , i s w i l li ng to
sacr i f ice func t ional i ty
- O n c e c o m p l e te a n o t h e r i te r a t io n
o f the c y c le i s r e qu i re d
- S tak e ho lde r w i l ling to m od i f y
p r ior i ti za t ion o f r e q u i re m e n t s in
a t t e mp t inc re ase poss ib le
func t ional i ty
- Se c on d i te ra t ion no t ne c e ssary ,
bu t c ou ld a id in r i sk ana ly s i s
- C u s t o m e r a g r e e s to c u r r e n t
c on f igura t ion
- C h a n g e s ( i f a n y ) s h o u l d b e n o t e d
in de s ign an d in i tia l p ro to ty pe c an
b e g i n
Figure 4. Stakeholder Options after first iteration
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 6/8
21 (76%) requirements. This illustrates that many of therequirements are broadly understood and only those otherrequirements need to be reviewed. Moreover, if face-to-facecommunication were possible, this percentage would be evengreater.
With the experiment complete, the final part was tomap the results. For the sake of exhibiting the full potential of the RHA, each category has been broken down into its secondtier. Furthermore, under usability, the sub-category markedunderstandability has been branched to show an alternateconfiguration (note: only one can be chosen). The overallsystem score shows that trade-off is necessary if fullfunctionality is to be achieved. Finally, the bold on R15 andR24 indicate requirements clash (see Appendix A for fullfigure).
CONCLUSION
Benefits of the Requirements Hierarchy Approach. In using
the Requirements Hierarchy Approach (RHA), clear gains canbe found in the areas of requirements elaboration, tradeoff, andrequirements validation and verification. Involving thestakeholder in the initial concept phase helps clarify theirvision of the system. Furthermore, categorizing requirementsand integrating the level of importance stakholders identifiedcreates a synergistic effect and sets the table for requirementstradeoff and preliminary acceptance metrics. A complimentarybenefit of establishing acceptance levels early in the life cycleis decreasing the risk involved in building a system thestakeholders asked for, but not what they wanted. Moreeloquently put, oftentimes developers appropriate resources tofunctionality that a user may care little about. In using theRHA, developers can be more in tune with the stakeholder’s
aspiration level as it pertains to specific functionality.
Another benefit in using the RHA is requirementsmanagement/project monitoring. With requirements listedcategorically, a project manager is able to track progress back to the hierarchy. Although developers would not be writingcode against the hierarchy, it creates a measurable correlationbetween the functionality they are working on and theunderlying hierarchy in which these requirements are a part.
Areas for Future Study. The methodology explained in thispaper is ripe for further study. The main area of particulainterest is automation. Knowledge-based systems are nowentering their fourth generation – reflecting the usage oautonomous agents to capture data. Moreover, the field ofknowledge engineering is just beginning to appreciate thepower of agents. Organized correctly, there is great promisefor a system that uses agents as the cognitive engine which“assists” in mapping the requirements and then performsanalysis on the hierarchy that was created. In time, throughrefinement, the RHA could ultimately be viewed as anobjective measure (which would be helpful for requirementsverification and validation) as opposed to a subjective one.
Finally, adopting an agreed upon taxonomy for NFRsis of prime importance. It is unrealistic to expect designers anddevelopers to incorporate an entity they cannot readily identifyWhile taxonomies aim to be inclusive of the entire set ofentities in question, a one or two level taxonomy would sufficeinitially as according to Chung et al. (Chung et al. 2000), thereare over 161 identifiable NFRs. Once an endorsed taxonomyis in place, researchers and practitioners would have the abilityto operate around this standard, as opposed to operating from ade facto standard.
REFERENCES
Adelman, Leonard; Donnell, M.L., Evaluating decision
support systems: A General framework and case
study. In S.J. Andriole (Ed.), MicrocomputerDecision Support Systems: Design, Implementationand Evaluation (pp285-310). Wellesley, MA: QEDInformation Science 1986.
Alford, M and Lawson, J., “Software RequirementEngineering Methodology (Development)”, U.S.A.FRome Air Development Center RADC-TR-79-1681979.
Barber, K. Suzanne, et al., “The Systems Engineering ProcessActivities (SEPA – Supporting Early RequirementsAnalysis and Intgration Prior to ImplementationDesign” The Laboratory for Intelligent Processes
and Systems The University of Texas at Austin 1997.Chung, K. L., “Representing and Using Non-functiona
Requirements for Information System DevelopmentA Process Oriented Approach”, Ph.D. Thesis, alsoTech. Rpt. DKBS-TR-93-1, Department of ComputerScience, University of Toronto, June 1993.
Chung, Lawrence et al., Non-Functional Requirements iSoftware Engineering Boston: Kluwer AcademicPress 2000.
Davis, Alan. Software Requirements: Objects, Functions, &
States, New Jersey: Prentice Hall, 1993.Franch, Xavier, “Systematic Formulation of Non-Functiona
Characteristics of Software” Proceedings of the Third
IEEE International Conference on Requiremen
Engineering 1998.
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 7/8
ISO/IEC Information Technology – “Software Productevaluation – Quality Characteristics and guidelinesfor their Use” ISO/IEC 9126 1991 (E).
Jacobs, Stephan, “Introducing Measurable QualityRequirements: A Case Study” Fourth IEEE
International Symposium on Requirements
Engineering 1999.Myopoulos, John, et al., “From Object-Oriented to Goal
Requirements” Transactions of the ACM January1999.
Tran, Quan and Chung, Lawrence, “NFR-Assistant: ToolSupport for Achieving Quality” IEEE Symposium on
Application-Specific Systems and Software
Engineering & Technology 1999.
BIOGRAPHY
Andrew J. Ryan was raised in the Bronx, New York. Heholds a Bachelors degree in Computer Science fromBinghamton University, a Master’s degree in Systems
Engineering and is currently working on a PhD in InformationTechnology – both at George Mason University. His interestsinclude agent programming, knowledge based systems, anddecision science theory. He is currently employed at MetronInc. as a software analyst and also teaches at George MasonUniversity. He is a member of IEEE, ACM, NSBE, INCOSE,and Who’s Who of International Professionals.
8/14/2019 Quantitative Non-Functional Requirements Publication
http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 8/8
Appendix A – Requirements Hierarchy for Proposed System
.400
SYSTEM.883
Functionality
0.4
Portability
.15Maintainability
Efficiency
.1
Usability
.35Reliability
Understandability[.25]
R4 [.230]
R7 [.192]
R8 [.230]
R10 [.115]
R14 [.115]
R15 [.038]
R22 [.076]
Learnability [.1]
R14 [.750]
*R15 [.250]
Attractiveness
[.6]R7 [.192]
R8 [.230]
R9 [.076]
R10 [.115]
R11 [.153]
R12 [.230]
Compliance [.05]
R19 [1]
Time Behavior [1]
R13 [.428]
R14 [.428]
R15 [.142]
Availability [.1]
R21 [1]
Adaptability
[.6]
R17 [1]
Co-existence
[.4]
R24 [1]
Accuracy [.4]
R1 [.257]
R2 [.257]
R3 [.257]
R5b [.228]
Interoperability
[.2]
R6 [.210]
R18 [.368]
R24 [.421]
Suitability [.1]
R5a [1]
Compliance [.2]
R20 [.5]
R23 [.5]
Understandability[.25]
R4 [.272]
R7 [.227]
R9 [.090]
R10 [.136]
R14 [.136]
R15 [.045]
R22 [.090]
.115
.100
.200
.100
.250
.050
.600
.075 .085
0
.090
.0900.085.3410.366
.250