Date: November 8, 2017 Researcher Name(s): Lan Lin ... · Date: November 8, 2017 Researcher...
Transcript of Date: November 8, 2017 Researcher Name(s): Lan Lin ... · Date: November 8, 2017 Researcher...
Title: Software Science: How Far Could Mathematics and Rigor Take Us?
Date: November 8, 2017
Researcher Name(s): Lan Lin
University: Ball State University
New Project Proposal Status Report Final Report
Long Term Goal(s)
Modern software development processes for safety- and mission-critical systems rely on
rigorous methods for code development and testing to support dependability claims and
assurance cases [1] that provide the justified and needed confidence. Sequence-based software
specification [2-4] (for requirements analysis and specification development) and Markov chain
usage-based statistical testing (for automated, model-based statistical testing and software
certification) [5-11] are two such methods that since inception have been successfully
combined and used in a variety of industry and government projects (including commercial
products), and have had a large measure of success. Previous collaborators include BNR,
Ericsson, CTI-PET Systems, IBM, Nortel, Oak Ridge National Laboratory [12], Raytheon, Verum
Consultants (in the Netherlands) [13-15], and Fraunhofer Institute for Experimental Software
Engineering (in Germany) [16]. The constructed rigorous specification automatically translates
to a formal model that can be used as a first pass at the state machine for a Markov chain usage
model for statistical testing. This integration defines a workflow and tool chain flowing from the
original, informal, and imperfect requirements, through model-based design and model-based
testing, to the final software certification, enabling a shift of focus up stream – one develops a
precise specification and code that meets the specification, and places in parallel development
and testing activities [5].
With our extensive experience with rigorous software specification and testing over the
last two decades [17-27], and an observation that although software-intensive systems have
become quite large over the years (with systems of 10 million lines of source code now
common), many systems are fielded without benefit of these methods (i.e., exhaustive analysis
of the system’s behavior in all possible scenarios of use, and testing based on a usage profile
that reveals faults in the order of their contribution to reliability, or demonstrates that the
highly likely use paths do not fail), we propose an adapted and augmented process for
developing safety- and mission-critical software-intensive systems. It incorporates our rigorous
specification and testing methods, as well as two other methods for design and testing that
have been widely used and shown great promise. The ultimate goal is to economically produce
zero- or nearly zero-defect safety- and mission-critical software utilizing a combination of these
mathematically sound methods. The resulting specification, design, and testing documentation,
as well as the accompanying quantitative analysis, will provide audit trails of evidence to
support a claim of dependability, and to demonstrate and certify that the system is fit for its
intended use.
Background for Long Term Goals
The proposed research follows the foundations of Cleanroom software engineering originated
with Harlan Mills [28, 29], and the work of Jesse H. Poore and his colleagues at the University of
Tennessee Software Quality Research Laboratory (UTK SQRL) in the past two decades [2-12, 16-
27]. Poore’s work has been focused on developing theory, engineering practices, working
methods and tools to implement Mills’ original concepts, and specifically on two areas: (1)
treating software development from a functional theory point of view, and (2) treating testing
as a problem that can be addressed by statistical science.
As prescribed by Mills [28, 29], every software program implements the mapping rule
for a mathematical function called the black box function. Simply put, the black box function
maps every possible history of system inputs to one and only one system output. Sequence-
based specification [2-5, 21, 22, 24] was developed to facilitate requirements analysis and to
discover this function definition through a systematic and constructive process called sequence
enumeration. The process requires one explicitly consider the software’s behavior in all possible
scenarios of use, and identifies all the control states of the system when it concludes. The result
is a fully documented, complete, consistent, and traceably correct black box specification, from
which one can easily generate a state box specification (or state machine), the code framework,
and much of the code itself [2-5, 24].
Meanwhile, the rigorous specification can also be used to generate a directed graph
that serves as the basis for a usage model for statistical testing [5-12]. One then assigns
probabilities on the arcs that represent the relative frequencies of taking different transitions
from each state, making it a Markov chain usage model. The model depicts the intended use of
the software in the field and represents the population of all possible use cases. Then all kinds
of statistics can be computed routinely from the model, providing a basis for model validation,
revision, and test planning. From the validated model one then generates test cases by walking
the graph, by applying graph algorithms, or by sampling. Test scripts can be associated with arcs
of the usage model which become instructions to manual testers or automated test runners.
Pass and fail data are recorded and analyzed for reliability estimation, coverage analysis, or
stopping decisions. The statistical testing process supports quantitative certification of software
by statistical protocol for standards compliance as well as for the construction and evaluation of
assurance cases for dependable systems.
Public domain tools supporting these rigorous methods are freely available [30-32].
Figure 1 shows the typical workflow combining rigorous specification with automated statistical
testing.
Intermediate Term Objectives
Our goal is to define an augmented process for developing safety- and mission-critical software
that builds on a number of mathematics- and theory-based formal/rigorous methods. In
contrast to the concept of “testing in quality” which is costly and ineffectual, we believe
software quality is achieved in the requirements, specification, architecture, design, code
generation, and coding activities. Our starting point is a precise black box and state box
specification derived from original functional requirements using sequence-based specification.
Then we refine it into a design that can be checked against desirable modularity properties
using conceptual integrity of software systems [33, 34], linear software models and the
modularity matrix [35], and develop code that meets the specification. Finally testing is a
statistical activity to demonstrate that there are no errors, rather than to find errors, and that
the software-intensive product, when released in the field, is fit for its intended use. By model-
based statistical testing, we place the software testing problem in a statistical context and
utilize statistical principles to guide testing strategy and data evaluation. We will also combine
statistical testing with another well-established rigorous testing method, i.e., combinatorial
testing that has proven to have high fault detection rate [36], and generate test suites that not
only target statistical analysis based on the expected software use but also provide good
combinatorial coverage. Figure 2 demonstrates our proposed work flow.
Figure 1. The typical workflow combining rigorous software specification with automated statistical testing
Although the theory for sequence-based specification and statistical testing has been
well established and combining these two rigorous methods was not a new idea, it remains
domain and application specific to integrate them into the existing development process, and
implement full test automation with no human intervention. On the other hand, the integration
of rigorous software specification with linear algebra-based modularity design, and the
integration of statistical testing with combinatorial testing are new research topics to explore
along the path; each could potentially become a major research result in itself. We will follow
the philosophy of theory-practice-tools in our research and exploration: first develop a theory
that has a sound mathematical foundation, then establish engineering practices that reinforce
the theory, and develop tool support that enforces the workflow but hides the mathematics.
Schedule of Major Steps
1. Select a real-world problem as the case study.
2. Derive a state-based specification from functional requirements using sequence-based
specification.
3. Refine the state-based specification into detailed design and implementation taking into
consideration the software architecture.
4. Validate the modular design using linear software models and the modularity matrix for
conceptual integrity.
5. Develop a Markov chain usage model for statistical testing based on the intended use in
the field, and validate the model.
6. Develop a test plan, and an automated test harness.
7. Augment the test suite with combinatorial test cases besides coverage sampling,
random sampling, and weighted sampling.
8. Automatically generate-execute-evaluate a large sample of test cases and record test
results, and project system reliability based on the testing experience.
Figure 2. The proposed workflow for the economical production of high-quality software
Dependencies
Steps 3 – 8 are based on the first two steps. Once a formal specification is derived, design
(Steps 3 – 4) and testing (Steps 5 – 8) could be carried out in parallel.
Major Risks
The selected case study should be small enough to fit the proposed time and budget, and to
make a feasible initial proof-of-concept, but large enough to convince future support in
applying rigorous methods to economically develop safety- and mission-critical software. The
integration of our rigorous methods with other mathematics-based methods might need more
time for the research and implementation. The proposed process needs to be adapted to the
working culture of the sponsoring affiliate.
Budget
Lan Lin, formal methods, mathematical analysis, application: $25,000
Graduate assistant, tool development, application: $25,000
---------------------------------
Total: $50,000
Staffing
Principal investigator: Dr. Lan Lin
One graduate assistant
We will work closely with the software engineers from the sponsoring affiliate and adapt our
methods to their specific process and work culture.
Category of Current Stage
New proposal
Contacts with Affiliates
N/A for the new proposal, however, it is in line with the following previously funded projects
(all through the NSF Security and Software Engineering Research Center):
Combining rigorous specification and testing methodologies to achieve high quality assurance
(single PI), $46,836, Lockheed Martin and Northrop Grumman, 01/01/13 – 12/31/13.
Towards scalable modeling for rigorous software specification and testing (single PI), $50,000,
Rockwell Collins, Air Force Research Laboratory, and Ontario Systems, 11/01/14 – 10/31/15.
Quantifying software quality through rigorous testing and test automation: From theory to
practice (single PI), $30,000, Ontario Systems, 11/01/16 – 10/31/17.
Publications and Other Research Products (actual or potential)
We anticipate 1-2 peer reviewed conference papers and 1-2 peer reviewed journal papers
reporting new research results in suitable venues.
We anticipate the following deliverables:
1. Complete documentation for the case study, including the black box and the state box
specifications, software architecture and modularization, the implemented state box
tables, code, the Markov chain usage model, model analysis, the test oracle, the
automated test harness, test plan, test cases, test scripts, test records, and test case
analysis.
2. Complete documentation for the developed theory that enables the integration of our
methods with other mathematics-based methods, in the form of technical/progress
reports. Released tool that implements the new theory.
References
[1] Jackson D, Thomas M, Millett LI ed., Software for Dependable Systems: Sufficient Evidence?
National Academies Press, 2009.
[2] Prowell SJ, Poore JH, Foundations of sequence-based software specification, IEEE
Transactions on Software Engineering 2003; 29(11): 417 – 429.
[3] Prowell SJ, Trammell CJ, Linger RC, Poore JH, Cleanroom Software Engineering: Technology
and Process, Addison-Wesley, 1999.
[4] Prowell SJ, Poore JH, Sequence-based software specification of deterministic systems,
Software – Practice and Experience 1998; 28(3): 329 – 344.
[5] Poore JH, Theory-practice-tools for automated statistical testing, DoD Software Tech News:
Model-Driven Development January 2010; 12(4): 20 – 24.
[6] Prowell SJ, Poore JH, Computing system reliability using Markov chain usage models, Journal
of Systems and Software 2004; 73(2): 219 – 225.
[7] Poore JH, Trammell CJ, Application of statistical science to testing and evaluating software
intensive systems, Statistics, Testing, and Defense Acquisition: Background Papers (Cohen ML,
Steffey DL, Rolph JE ed.), National Academies Press, 1999.
[8] Poore JH, Trammell CJ, Engineering practices for statistical testing, Crosstalk (DoD Software
Engineering Journal-Newsletter) April 1998; 24 – 28.
[9] Walton GH, Poore JH, Trammell CJ, Statistical testing of software based on a usage model,
Software – Practice and Experience 1995; 25(1): 97 – 108.
[10] Whittaker JA, Thomas MG. A Markov chain model for statistical software testing, IEEE
Transactions on Software Engineering 1994; 30(10): 812 – 824.
[11] Whittaker JA, Poore JH, Markov analysis of software specifications, ACM Transactions on
Software Engineering and Methodologies 1993; 2(1): 93 – 106.
[12] Sayre K, Poore JH, Automated testing of generic computational science libraries,
Proceedings of the 4th Hawaii International Conference on Systems Sciences, Waikoloa, HI,
2007, 277c.
[13] Bouwmeester L, Broadfoot GH, Hopcroft PJ, Compliance test framework, Proceedings of
the 2nd Workshop on Model-Based Testing in Practice, Enscede, The Netherlands, 2009, 97 –
106.
[14] Hopcroft PJ, Broadfoot GH, Combining the box structure development method and CSP for
software development, Electronic Notes in Theoretical Computer Science 2005; 128(6): 127 –
144.
[15] Broadfoot GH, Broadfoot PJ, Academia and industry meet: Some experiences of formal
methods in practice, Proceedings of the 10th Asia-Pacific Software Engineering Conference,
Chiang Mai, Thailand, 2003, 49 – 59.
[16] Bauer T, Beletski T, Boehr F, Eschbach R, Landmann D, Poore JH, From requirements to
statistical testing of embedded systems, Proceedings of the 4th International Workshop on
Software Engineering for Automotive Systems, Minneapolis, MN, 2007, 3 – 9.
[17] Lin L, Xue Y, Song F, A simpler and more direct derivation of system reliability using Markov
chain usage models, Proceedings of the 29th International Conference on Software Engineering
and Knowledge Engineering, Pittsburgh, PA, 2017, 462 – 466.
[18] Lin L, Scalable modeling for rigorous software specification and testing, 2016 Compendium
of Industry-Nominated NSF I/UCRC Technological Breakthroughs, 2016, 107 – 108.
[19] Lin L, Y. Xue, F. Song, An algorithm for forward reduction in sequence-based software
specification, International Journal of Software Engineering and Knowledge Engineering (Special
Issue on Best Papers from SEKE 2016) 2016; 9 & 10: 1431 – 1451.
[20] Lin L, He J, Xue Y, An automated testing framework for statistical testing of GUI
applications, Proceedings of the 27th International Conference on Software Engineering and
Knowledge Engineering, Pittsburgh, PA, 2015, 72 – 79.
[21] Eschbach R, Lin L, Poore, JH, Applying string-rewriting to sequence-based specification,
Formal Methods in System Design 2013; 43(3): 414 – 449.
[22] Lin L, Poore JH, Eschbach R, Hierons RM, Robinson-Mallett C, Augmenting sequence
enumeration with string-rewriting for requirements analysis and behavioral specification,
Proceedings of the 16th International Conference on Fundamental Approaches to Software
Engineering, Rome, Italy, Lecture Notes in Computer Science Volume 7793, 2013, 179 – 193.
[23] Poore JH, Lin L, Eschbach R, Bauer T, Automated statistical testing for embedded systems,
Model-Based Testing for Embedded Systems in the series on Computational Analysis, Synthesis,
and Design of Dynamic Systems (Zander J, Schieferdecker I, Mosterman PJ ed.), CRC Press –
Taylor & Francis, 2011.
[24]Lin L, Prowell SJ, Poore JH, An axiom system for sequence-based specification, Theoretical
Computer Science 2010; 411(2): 360 – 376.
[25] Lin L, Prowell SJ, Poore JH, The impact of requirements changes on specifications and state
machines, Software – Practice and Experience 2009; 39(6): 573 – 610.
[26] Lin L, Carter JM, Poore JH, Using state machines to model and manage requirements
changes and specification changes, Proceedings of the 51st IEEE International Midwest
Symposium on Circuits and Systems, Knoxville, TN, 2008, 523 – 526.
[27] Carter JM, Lin L, Poore JH, Automated functional testing of Simulink control models,
Proceedings of the 1st Workshop on Model-Based Testing in Practice, Berlin, Germany, 2008, 41
– 50.
[28] Mills HD, Dyer M, Linger RC, Cleanroom software engineering, IEEE Software September
1987; 19 – 24.
[29] Mills HD, The new math of computer programming, Communications of the ACM 1975;
18(1): 43 – 48.
[30] Protoseq, Prototype Sequence Enumeration Tool, Software Quality Research Laboratory,
The University of Tennessee, http://sourceforge.net/projects/protoseq, 2017.
[31]REALSBS, Requirements Elicitation and Analysis Sequence-Based Specification Tool,
Software Research Laboratory, The University of Tennessee,
http://sourceforge.net/projects/realsbs, 2017.
[32] J Usage Model Builder Library (JUMBL), Software Quality Research Laboratory, The
University of Tennessee, http://jumbl.sourceforge.net/jumblTop.html, 2017.
[33] Exman I, Conceptual integrity of software systems: Architecture, abstraction and algebra,
Proceedings of the 29th International Conference on Software Engineering and Knowledge
Engineering 2017, Pittsburgh, PA, 2017, 416 – 421.
[34] Exman I, Katz P, Conceptual Software Design: Algebraic axioms for conceptual integrity,
Proceedings of the 29th International Conference on Software Engineering and Knowledge
Engineering 2017, Pittsburgh, PA, 2017, 155 – 160.
[35] Exman I, Linear software models: Standard modularity highlights residual coupling,
International Journal of Software Engineering and Knowledge Engineering 2014; 24(2): 183 –
210.
[36] Kuhn DR, Kacker RN, Lei Y, Practical combinatorial testing, National Institute of Standards
and Technology Special Publication 800-142, October 2010.