Four Mega-Challenges Facing The Commercial “Software Quality Movement”: Definition, Composition,...
-
Upload
britton-ramsey -
Category
Documents
-
view
214 -
download
0
Transcript of Four Mega-Challenges Facing The Commercial “Software Quality Movement”: Definition, Composition,...
Four Mega-Challenges Facing The Four Mega-Challenges Facing The Commercial “Software Quality Movement”:Commercial “Software Quality Movement”: Definition, Composition, Certification, and Commercialization and Definition, Composition, Certification, and Commercialization and Return-On-InvestmentReturn-On-Investment
Jeffrey M. VoasJeffrey M. VoasChief ScientistChief Scientist
Ask the CIO
A: What do you mean by that?A: What do you mean by that?
Q: Are you interested in quality software?Q: Are you interested in quality software?
Q: Are you interested in software quality?Q: Are you interested in software quality?
A: Yes.A: Yes.
What is Software Quality?
Subjective term that produces confusion among Subjective term that produces confusion among most most software engineering professionalssoftware engineering professionals
IEEE
• ““Totality of features of a software product that Totality of features of a software product that bears on its ability to satisfy given needsbears on its ability to satisfy given needs.” .” [Source: IEEE-STD-729][Source: IEEE-STD-729]
• ““Composite characteristics of software that Composite characteristics of software that determine the degree to which the software in determine the degree to which the software in use will meet the expectations of the use will meet the expectations of the customercustomer.”.” [Source: IEEE-STD-729][Source: IEEE-STD-729]
Three High-Level AttributesThree High-Level Attributes
Quality Software
Reliable/Accurate(integrity)
Secure/private
Timeliness
High-Level AttributesHigh-Level Attributes
Reliable/Accurate(integrity)
Secure/private
Timeliness
Problem: Intuitive, but not formal
Quality Software
Lower-Level AttributesLower-Level Attributes
Reliable/accurate
Secure/private
Timeliness
reliability security performanceavailabilityprivacy
fault tolerance fault tolerance
confidentiality
intrusion tolerancetestability
Non-functional attributes (“ilities”)
Quality Software
Functional attributes
Software Quality (or Quality Software)Software Quality (or Quality Software) must be must be viewed/defined as some combination of: (1) the viewed/defined as some combination of: (1) the degree to which the degree to which the functionalfunctional requirements are met, requirements are met, as well as,as well as, (2) the degree to which the (2) the degree to which the non-functionalnon-functional requirements are met.requirements are met.
Position StatementPosition Statement
reliability security performanceavailabilityprivacy
fault tolerance fault tolerance
confidentiality
intrusion tolerancetestability
Non-functional attributes (“ilities”)
+
Functional attributes
Software QualitySoftware Quality then is then is somesome combination of the following functional and non- combination of the following functional and non-functional attributes:functional attributes:
Reliability [R], Performance [P], Safety [Sa]Reliability [R], Performance [P], Safety [Sa]Fault Tolerance [F], Security [Se], Availability [A]Fault Tolerance [F], Security [Se], Availability [A]Testability [T], and Maintainability [M]Testability [T], and Maintainability [M]
Position StatementPosition Statement
Software QualitySoftware Quality can also be viewed as some combination of the can also be viewed as some combination of the previous attributes PLUS:previous attributes PLUS:
Scalability, Usability, Sustainability, Survivability,Scalability, Usability, Sustainability, Survivability,Interoperability, Extensibility, Reusability, Interoperability, Extensibility, Reusability, Readability, etc.Readability, etc.
However ….However ….
QQ = = aaR + R + bbP + P + ccF + F + ddSa + Sa + eeSe + Se + ffA + A + ggT + T + hhMM
where where a, b, c, d, e, f, g,a, b, c, d, e, f, g, and and hh are units of quantitative are units of quantitative or qualitativeor qualitative measures of a particular attribute. measures of a particular attribute.
Eight in an Equation?Eight in an Equation?
Key ProblemsKey Problems
• The equation cannot be linear, since the The equation cannot be linear, since the
units of measure for each attribute units of measure for each attribute cannot be standardized (the apples and cannot be standardized (the apples and oranges problem).oranges problem).
• ddSa = Sa = QQ – ( – (aaR + R + bbP + P + ccF + F + eeSe + Se + ffA + A + ggT + T + hhM)M)
• Most “ilities” are not quantifiably Most “ilities” are not quantifiably measurable.measurable.
• Reliability, Availability, and Performance are Reliability, Availability, and Performance are measurable (via testing).measurable (via testing).
For Example …MaintainabilityFor Example …Maintainability
• Size, defect density, amount of testing, T, Size, defect density, amount of testing, T,
R, cohesion, coupling, documentation, R, cohesion, coupling, documentation, complexity, depth of inheritance, number complexity, depth of inheritance, number of objects, testing infrastructure, mean-of objects, testing infrastructure, mean-time-to-repair, experience of time-to-repair, experience of maintenance personnel as well as their maintenance personnel as well as their domain knowledge, existence of impact domain knowledge, existence of impact analysis tools, etc., all impact M.analysis tools, etc., all impact M.
• Q: So how can you assign a single Q: So how can you assign a single numerical score for M?numerical score for M?
SecuritySecurity
• The level of security of an information system is a The level of security of an information system is a
function of the partially unknown threat space, that function of the partially unknown threat space, that changes by the minute.changes by the minute.
• Q: So how can you assign a single numerical score for Se?Q: So how can you assign a single numerical score for Se?
• A: You can assess, for a bounded set of anticipated A: You can assess, for a bounded set of anticipated threats, how the system will respond to those, e.g., 100 threats, how the system will respond to those, e.g., 100 known threats, 50 mitigated.known threats, 50 mitigated.
• A: Or you could measure the percentage of patches that A: Or you could measure the percentage of patches that are installed based on the number that need to be, and are installed based on the number that need to be, and then test to make sure those installed work. Such then test to make sure those installed work. Such information could also be used to give security “a score” information could also be used to give security “a score” but once again, that is only a score based on known but once again, that is only a score based on known threats and available patches.threats and available patches.
It is more difficult to directly measure
the quality of software than to achieve quality.
It is more difficult to directly measure
the quality of software than to achieve quality.
The “Culprit” PhenomenonThe “Culprit” Phenomenon
Without a Numerical Quality Equation, But With a Way to
Discuss the Attributes of Quality Software,and Therefore With a Means for Industry to Define
Quality Goals
Without a Numerical Quality Equation, But With a Way to
Discuss the Attributes of Quality Software,and Therefore With a Means for Industry to Define
Quality Goals
So Where Does That Leave Us?So Where Does That Leave Us?
has the following properties:
((aaR, R, bbP, P, ccF, F, ddSa, Sa, eeSe, Se, f fA, A, ggT, T, hhM)M) has the following properties:
(i(iR, R, jjP, P, kkF, F, llSa, Sa, mmSe,Se, n nA, A, ooT, T, ppM)M)
With Attributes
Then f( ) will inherit some level of Quality from the individual components. Is that level of quality an integer? Probability? An n-tuple of values? Color coded (green red yellow)?
Key Point: The Composite Quality must represent something from which predictions of future behavior can be made.
What Have You Got?
Key ProblemsKey Problems
• It is hard enough to know, with any It is hard enough to know, with any preciseprecise accuracy, accuracy,
what the composite what the composite reliabilityreliability score will be as a result of score will be as a result of the the aa and and ii values (let alone for the non-functional values (let alone for the non-functional attributes). attributes).
• But an even greater challenge exists here. For example, But an even greater challenge exists here. For example, the the securitysecurity mechanisms in component mechanisms in component could thwart the could thwart the
performanceperformance that is built into component that is built into component ..
• Attributes are only reasonable to talk about within the Attributes are only reasonable to talk about within the context of a context of a systemsystem, i.e., it is not reasonable to talk about , i.e., it is not reasonable to talk about them and attempt to measure them as standalone them and attempt to measure them as standalone component properties. Their component properties. Their eventual target eventual target environmentsenvironments must weighed into their individual must weighed into their individual assessments.assessments.
EnvironmentEnvironment
Reliable/accurate
Secure/private
Timeliness
reliability security performanceavailabilityprivacy
fault tolerance fault tolerance
confidentiality
intrusion tolerancetestability
Non-functional attributes (“ilities”)
QualityOperational environment
In Search of a Calculus or Calculi for Predicting How a Composite
System Will Behave in the Future in a Specific Environment
In Search of a Calculus or Calculi for Predicting How a Composite
System Will Behave in the Future in a Specific Environment
So Where Does That Leave Us?So Where Does That Leave Us?
Product Certification and Product Certification and Software Engineering StandardsSoftware Engineering StandardsTo Aide the Composition ProblemTo Aide the Composition Problem
Standardized Parts?Standardized Parts?
Ideally, it is a line in the sand from which a certificate of compliance can be written.
Ideally, it is a line in the sand from which a certificate of compliance can be written.
What is a Standard? What is a Standard?
ProsPros
Any bar or hurdle is better than no bar or Any bar or hurdle is better than no bar or hurdlehurdle
ConsCons
Possibly the developers would have done Possibly the developers would have done moremore to improve quality but now feel they to improve quality but now feel they
have a license to do have a license to do lessless..
Premise for SW Product CertificationPremise for SW Product Certification
Commercially built software should be Commercially built software should be tagged tagged
with some guarantee (or with some guarantee (or at leastat least a “warm a “warm fuzzy”) as to how good the software fuzzy”) as to how good the software
should be.should be.
Problem: Software Of Unknown Pedigree Problem: Software Of Unknown Pedigree ((SOUPSOUP))
Goal of Product Certification: Goal of Product Certification: SOSO((KKnown)nown)PP
ProcessesProcesses
ProductsProducts
PeoplePeopleAll SE standards incorporateone or more of these perspectives
All SE standards incorporateone or more of these perspectives
Three Schools of ThoughtThree Schools of Thought
1. Process: Clean Pipes, Dirty Water?1. Process: Clean Pipes, Dirty Water?
Certifying that you know how to do things correctly
does not mean that you do them
correctly!
Certifying that you know how to do things correctly
does not mean that you do them
correctly!
The IEEE Computer Society has developed a The IEEE Computer Society has developed a program to certify software engineering program to certify software engineering
professionals. This program provides professionals. This program provides formal recognition of professionals who formal recognition of professionals who
have successfully achieved a level of have successfully achieved a level of proficiency commonly accepted and valued proficiency commonly accepted and valued
by the industry.by the industry.
2. People2. People
3. Product: The Software Itself3. Product: The Software Itself
Spectrum of possibilities as to what a certificate Spectrum of possibilities as to what a certificate proclaiming that some “quantified” level of quality has proclaiming that some “quantified” level of quality has been built in could state --- it could say anything in the been built in could state --- it could say anything in the range between “Nothing” (range between “Nothing” (e.g.e.g., “here is a piece of , “here is a piece of software”, etc.) to “This software will always work perfectly software”, etc.) to “This software will always work perfectly under all conditions” (i.e., a 100% guarantee of perfection).under all conditions” (i.e., a 100% guarantee of perfection).
0% 0% confidencconfidenc
ee
100% 100% confidenceconfidence
But Problems Exist With StandardsBut Problems Exist With Standards
– Vague: Vague: Develop software that only does "good" thingsDevelop software that only does "good" things• Common sense "dos" and "don'ts" - Very watered done by voting Common sense "dos" and "don'ts" - Very watered done by voting
timetime– Disclaimers by publishing organizationsDisclaimers by publishing organizations
• Profitable to organization that publishes themProfitable to organization that publishes them– Used only if mandatedUsed only if mandated– Return-on-investment is unknownReturn-on-investment is unknown– Thwart intellectual creativity Thwart intellectual creativity
• "Protectionist" legislation"Protectionist" legislation– PaperworkPaperwork
• 2167A: 2167A: ~400~400 English words per Ada code statement English words per Ada code statement– "Old news" before being ratified"Old news" before being ratified– Relating one to another is very hardRelating one to another is very hard
• Hundreds in existenceHundreds in existence– Cannot be easily tested for complianceCannot be easily tested for compliance
• Mis-certifications are possibleMis-certifications are possible
– Different interpretationsDifferent interpretations
– Lack of fairness during certification judgment Lack of fairness during certification judgment
– So much legacy code exists that complies with So much legacy code exists that complies with no standards and therefore get excluded in no standards and therefore get excluded in heterogeneous systems, making it’s impact to heterogeneous systems, making it’s impact to the system unknown.the system unknown.
Suppose you have the following logical expression:
(A and B) or (B and C) or (A and C)
where A, B, and C are Boolean variables
To meet verification requirements for Level A software in RTCA DO178-B, you need to know the number of conditions in this statement
Condition: A Boolean expression containing no Boolean operations
How many conditions are there? 3, 4, 6, or 9
Example of “Standards” ConfusionExample of “Standards” Confusion
[Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]
The FAA Says …The FAA Says …
Distribution of Responses from 39 FAA Certification Authorities
0
5
10
15
20
25
30
35
40
45
3 4 6 9
% ofResponses
35.9
17.9
41.0
5.1
[Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]
And the Answer is …And the Answer is …
6
[Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]
(A and B) or (B and C) or (A and C) has 6 conditions
The full definition for condition is not contained in the glossary entry for that term
Part of the definition is given in the entry for decision
Decision: A Boolean expression composed of conditions and zero or more Boolean operators. A decision without a Boolean operator is a condition. If a condition appears more than once in a decision, each occurrence is a distinct condition.
ExplanationExplanation
[Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]
In Need of More Precise, Less Vague, and Repeatable
Processes, for Grading The Quality of Software
In Need of More Precise, Less Vague, and Repeatable
Processes, for Grading The Quality of Software
So Where Does That Leave Us?So Where Does That Leave Us?
Commercialization IssuesCommercialization Issues
• Proven technology? (empirical vs. anecdotal) Proven technology? (empirical vs. anecdotal)
• Prototypes? Are they Maintainable/Extensible or Prototypes? Are they Maintainable/Extensible or Trashware?Trashware?
• Scalable? Theoretical or Practical? Maturity?Scalable? Theoretical or Practical? Maturity?
• Automated? Is it a solution or standalone?Automated? Is it a solution or standalone?
• What languages/architectures does it support? What languages/architectures does it support? Fad/Lifetime?Fad/Lifetime?
• Difficult to learn? Ease of use? Time-to-market enabler Difficult to learn? Ease of use? Time-to-market enabler or disabler?or disabler?
• Client base: commercial or government? Number of Client base: commercial or government? Number of site?site?
• Evolutionary (leap frog-able) vs. Revolutionary?Evolutionary (leap frog-able) vs. Revolutionary?
• Compatible with existing technology (Compatible with existing technology (e.ge.g., Microsoft)?., Microsoft)?
• Point of Origination: University? Small business?, Large Point of Origination: University? Small business?, Large Corporation?, Government?Corporation?, Government?
• Competing foreign technologies?Competing foreign technologies?
• Process, People, and Product - oriented?Process, People, and Product - oriented?
• Which attribute(s) does it address? Measurement or Which attribute(s) does it address? Measurement or design? If design, how much of that attribute can it design? If design, how much of that attribute can it offer?offer?
• SOUP or SOKP?SOUP or SOKP?
• Does this technology self-certify Quality after use?Does this technology self-certify Quality after use?
1. Quality is a Recipe1. Quality is a Recipe
As with food, ingredients of different types (liquids, As with food, ingredients of different types (liquids, powders, vegetables, meats, etc.) can all be mixed together. powders, vegetables, meats, etc.) can all be mixed together.
What food tastes like is a function of the ingredients and What food tastes like is a function of the ingredients and their proportions.their proportions.
Quality Software can be viewed/defined in a similar manner. Quality Software can be viewed/defined in a similar manner.
And certain ingredients overpower others.And certain ingredients overpower others.
• Software engineering standards are completely necessary Software engineering standards are completely necessary despite their limitations. They usually are a “good rule of despite their limitations. They usually are a “good rule of thumb”, but not an absolute process for achieving thumb”, but not an absolute process for achieving perfection.perfection.
• Virtually any standard beats development chaosVirtually any standard beats development chaos
• What is Missing in SE Standards?What is Missing in SE Standards?• Not technology!Not technology!• How to Implement, How to Implement, • How to gain regulatory approval given uncertain How to gain regulatory approval given uncertain
knowledge as to how judgment will be rendered,knowledge as to how judgment will be rendered,• Fairness in the certification processes,Fairness in the certification processes,• And ROI (most are anecdotes, not statistical studies)And ROI (most are anecdotes, not statistical studies)
2. Standards Beat Chaos2. Standards Beat Chaos
• Until the non-functional attributes of software components Until the non-functional attributes of software components can be graded, and assumptions about the target can be graded, and assumptions about the target environments of those components can be nailed down environments of those components can be nailed down (HUGE PROBLEM), (HUGE PROBLEM), a prioria priori certification of the quality of any certification of the quality of any software component is suspect.software component is suspect.
• Research is needed into how to compose both functional Research is needed into how to compose both functional and non-functional attributes.and non-functional attributes.
3. Only Product Certification Can Address The Composition Problem
3. Only Product Certification Can Address The Composition Problem
4. Attributes Need to Be Pre-Defined4. Attributes Need to Be Pre-Defined
• Requirements should prescribe at some level of Requirements should prescribe at some level of
granularity as to what the weights are for various “ilities”, granularity as to what the weights are for various “ilities”, as well as how much of each “ility” is desired. as well as how much of each “ility” is desired.
• But HOW?But HOW?
• Ignoring the non-functional attributes is not an option for Ignoring the non-functional attributes is not an option for high assurance and trustworthy systems! Make an high assurance and trustworthy systems! Make an attempt to discuss them with the client even if attempt to discuss them with the client even if quantification is not possible. Just get the issue on the quantification is not possible. Just get the issue on the table!table!
ww11R R ww22P P ww33F F ww44Sa Sa
ww55Se Se ww66A A ww77T T ww88MM
in order to not in order to not over-designover-design any attribute into the system. any attribute into the system.
For example, for an e-commerce application, For example, for an e-commerce application, ww44 would would
probably equal 0.0 and probably equal 0.0 and ww7 7 would also be less than something would also be less than something
like like ww22
5. Weighting is Important5. Weighting is Important
How much will you spend for increased reliability knowing that doing so will take needed, financialresources away from security or performance or …?
6. Tradeoffs6. Tradeoffs
• Security vs. PerformanceSecurity vs. Performance
• Fault tolerance vs. TestabilityFault tolerance vs. Testability
• Fault tolerance vs. PerformanceFault tolerance vs. Performance
• etc.etc.
Counterintuitive RealitiesCounterintuitive Realities
• 100% safety and 0% reliability100% safety and 0% reliability
• 100% reliability and 0% safety100% reliability and 0% safety
• 0% functionality/reliability and 100% 0% functionality/reliability and 100% securitysecurity
• 100% availability and 0% reliability100% availability and 0% reliability
• 100% availability and 0% performance100% availability and 0% performance
• 0% performance and 100% safety0% performance and 100% safety
…. is alive and well. There are still many interesting andfascinating software engineering research challenges to pursue that not only will benefit NASA, but industry at-large.
7. The Software Quality Movement …7. The Software Quality Movement …
21351 Ridgetop Circle21351 Ridgetop CircleDulles, VA 20166 USA Dulles, VA 20166 USA www.cigital.comwww.cigital.com
Jeffrey VoasJeffrey Voas
phone: 703.404.9293phone: 703.404.9293
e-mail: [email protected]: [email protected]