A Framework for Computer-aided Validation

September 24, 2007September 24, 2007 NASA IV&V Facility Workshop on ValidationNASA IV&V Facility Workshop on ValidationMorgantown, WV Morgantown, WV

11

A Framework forA Framework forComputer-aided ValidationComputer-aided Validation

Presented by Bret MichaelPresented by Bret Michael

Joint work with Doron Drusinsky and Man-Tak ShingJoint work with Doron Drusinsky and Man-Tak ShingNaval Postgraduate SchoolNaval Postgraduate School

Monterey, CAMonterey, CA


22

DisclaimerDisclaimer

The views and conclusions in this talk are The views and conclusions in this talk are those of the author and should not be those of the author and should not be interpreted as necessarily representing the interpreted as necessarily representing the official policies or endorsements, either official policies or endorsements, either expressed or implied, of the U.S. expressed or implied, of the U.S. GovernmentGovernment


33

Conventional Approach to Conventional Approach to Conducting IV&VConducting IV&V

Relies onRelies on Manual examination of software requirements and design Manual examination of software requirements and design

artifactsartifacts Manual and tool-based code analysisManual and tool-based code analysis Systematic or random independent testing of target codeSystematic or random independent testing of target code

Poses seemingly insurmountable challengesPoses seemingly insurmountable challenges Most of these techniques are ineffective for validating the Most of these techniques are ineffective for validating the

correctness of the developer’s cognitive understanding of correctness of the developer’s cognitive understanding of the requirementsthe requirements

For complex software-intensive systems, manual IV&V For complex software-intensive systems, manual IV&V techniques are inadequate for locating the subtle errors techniques are inadequate for locating the subtle errors in the softwarein the software

For example, sequencing behaviors only observable at runtime For example, sequencing behaviors only observable at runtime and at such a fine level of granularity of time make human and at such a fine level of granularity of time make human intervention at runtime impracticalintervention at runtime impractical


44

Software AutomationSoftware Automation Holds the key to the validation and verification of Holds the key to the validation and verification of

the behaviors of complex software-intensive the behaviors of complex software-intensive systemssystems

Relies on formal specification of system Relies on formal specification of system behaviorsbehaviors

Requires breaking from time-honored rules of Requires breaking from time-honored rules of thumb about how to conduct IV&Vthumb about how to conduct IV&V

Enables IV&V teams toEnables IV&V teams to Accelerate their productivityAccelerate their productivity Cope with the impacts of accelerating technological Cope with the impacts of accelerating technological

change, or what Alan Greenspan refers to as the change, or what Alan Greenspan refers to as the “revolution in information technology”“revolution in information technology”


55

IEEE DefinitionsIEEE Definitions

ValidationValidation ““The process of evaluating a system or component The process of evaluating a system or component

during or at the end of the development process to during or at the end of the development process to determine whether a system or component satisfies determine whether a system or component satisfies specified requirements”specified requirements”

VerificationVerification ““The process of evaluating a system or component to The process of evaluating a system or component to

determine whether a system of a given development determine whether a system of a given development phase satisfies the conditions imposed at the start of phase satisfies the conditions imposed at the start of that phase”that phase”


66

Current IEEE Standards View of Current IEEE Standards View of Validation and Verification (V&V)Validation and Verification (V&V)

Checking theChecking theCorrectness of a target system or component Correctness of a target system or component

against a formal model that is derived from against a formal model that is derived from the natural language requirementsthe natural language requirements

Consistency and completeness of the formal Consistency and completeness of the formal models without ensuring that the developer models without ensuring that the developer understands the requirements and that the understands the requirements and that the formal models correctly match the developer’s formal models correctly match the developer’s cognitive intent of the requirementscognitive intent of the requirements


77

IV&V Team’s Independent IV&V Team’s Independent Requirements EffortRequirements Effort

Describe the necessary attributes, Describe the necessary attributes, characteristics, and qualities of characteristics, and qualities of anyany system system developed to solve the problem and satisfy the developed to solve the problem and satisfy the intended use and user needsintended use and user needs

Ensure that its cognitive understanding of the Ensure that its cognitive understanding of the problem and the requirements for any system problem and the requirements for any system solving the problem are correct before solving the problem are correct before performing IV&V on developer-produced performing IV&V on developer-produced systemssystems


88

Proposed FrameworkProposed Framework

Incorporates advanced computer-aided Incorporates advanced computer-aided validation techniques to the IV&V of validation techniques to the IV&V of software systemssoftware systems

Allows the IV&V team to capture bothAllows the IV&V team to capture both Its own understanding of the problemIts own understanding of the problemThe expected behavior of any proposed The expected behavior of any proposed

system for solving the problem via an system for solving the problem via an executable system reference modelexecutable system reference model


99

Terminology as UsedTerminology as Usedin the Frameworkin the Framework

Developer-generated requirementsDeveloper-generated requirementsThe requirements artifacts produced by the The requirements artifacts produced by the

developer of a systemdeveloper of a systemSystem reference model (SRM)System reference model (SRM)

The artifacts developed by the IV&V team’s The artifacts developed by the IV&V team’s own requirements effortown requirements effort


1010

Contents of a SRMContents of a SRM

Use cases and UML artifactsUse cases and UML artifactsFormal assertions to describe precisely Formal assertions to describe precisely

the necessary behaviors to satisfy system the necessary behaviors to satisfy system goals (i.e., to solve the problem) with goals (i.e., to solve the problem) with respect torespect toWhat the system should doWhat the system should doWhat the should not doWhat the should not doHow the system should respond under non-How the system should respond under non-

nominal circumstancesnominal circumstances


1111

Prerequisites for Using Computer-Prerequisites for Using Computer-Based V&V TechnologyBased V&V Technology

Development of formal, executable Development of formal, executable representations of a system’s properties, representations of a system’s properties, expressed as a set of desired system expressed as a set of desired system behaviorsbehaviors


1212

Classes of System BehaviorsClasses of System Behaviors

Logical behaviorLogical behaviorDescribes the cause and effect of a Describes the cause and effect of a

computation, typically represented as computation, typically represented as functional requirements of a systemfunctional requirements of a system

Sequencing behaviorSequencing behaviorDescribes the behaviors that consist of Describes the behaviors that consist of

sequences of events, conditions and sequences of events, conditions and constraints on data values, and timingconstraints on data values, and timingIn its vanilla form specifies sets of legal (or illegal) In its vanilla form specifies sets of legal (or illegal)

sequencessequences


1313

Beyond Pure SequencingBeyond Pure Sequencing

Timing constraintsTiming constraintsDescribe the timely start and/or termination of Describe the timely start and/or termination of

successful computations at a specific point of successful computations at a specific point of timetimeExample: Deadline of a periodic computation or Example: Deadline of a periodic computation or

the maximum response time of an event handlerthe maximum response time of an event handler

Time-series constraintsTime-series constraintsDescribe the timely execution of a sequence Describe the timely execution of a sequence

of data values within a specific duration of of data values within a specific duration of timetime


1414

Use Cases and UML ArtifactsUse Cases and UML Artifactsof the SRMof the SRM

Stakeholder’s Input(mission statements, operation concepts

documents, user expectations, etc.)

Use Case Scenarios

Dynamic UML Models(Message Sequence Charts, Activity

Diagrams etc.)

Static UML Models(Object Class Diagrams)


1515

Categories of FormalCategories of FormalSpecifications of BehaviorSpecifications of Behavior

Assertion-oriented specificationsAssertion-oriented specifications High-level requirements are decomposed into more High-level requirements are decomposed into more

precise requirements that are mapped one-to-one to precise requirements that are mapped one-to-one to formal assertionsformal assertions

Model-oriented specificationsModel-oriented specifications A single monolithic formal model (either as a state- or A single monolithic formal model (either as a state- or

an algebraic-based system) captures the combined an algebraic-based system) captures the combined expected behavior described by the lower level expected behavior described by the lower level specifications of behaviorspecifications of behavior

Describes the expected behavior of a conceptualized Describes the expected behavior of a conceptualized system from the IV&V team’s understanding of the system from the IV&V team’s understanding of the problem spaceproblem space

May differ significantly from the system design models May differ significantly from the system design models created by the developers in their design spacecreated by the developers in their design space


1616

Example of ConductingExample of ConductingAssertion-oriented SpecificationAssertion-oriented Specification

Start with high-level requirementStart with high-level requirement R1.R1. The track processing system can only handle a The track processing system can only handle a

workload not exceeding 80% of its maximum load workload not exceeding 80% of its maximum load capacity at runtimecapacity at runtime

Reify R1 into lower level requirementReify R1 into lower level requirement R1.1R1.1 Whenever the track count (cnt) Average Arrival Whenever the track count (cnt) Average Arrival

Rate (ART) exceeds 80% of the Rate (ART) exceeds 80% of the MAX_COUNT_PER_MIN, cnt ART must be reduced MAX_COUNT_PER_MIN, cnt ART must be reduced back to 50% of the MAX_COUNT_PER_MIN within 2 back to 50% of the MAX_COUNT_PER_MIN within 2 minutes and cnt ART must remain below 60% of the minutes and cnt ART must remain below 60% of the MAX_COUNT_PER_MIN for at least 10 minutesMAX_COUNT_PER_MIN for at least 10 minutes


1717

Continuation of ExampleContinuation of Example

Map R1.1 to a formal assertion expressed Map R1.1 to a formal assertion expressed as a Statechart assertionas a Statechart assertion

On-Entry/timer120.restart(); cnt = 0;

On-Entry/timer600.restart(); cnt = 0;

On-Entry/nTime = primary.getTime(); cnt = 0;


1818

Advantages of Using an Assertion-Advantages of Using an Assertion-Oriented Specification ApproachOriented Specification Approach

Requirements are traceable because they are Requirements are traceable because they are represented, one-to-one, by assertions (acting represented, one-to-one, by assertions (acting as watchdogs for the requirements)as watchdogs for the requirements) A monolithic model is the sum of all concerns: on A monolithic model is the sum of all concerns: on

detecting a violation of the formal specification, it is detecting a violation of the formal specification, it is difficult to map that violation to a specific human-difficult to map that violation to a specific human-driven requirementdriven requirement

Assertion-oriented specifications have a lower Assertion-oriented specifications have a lower maintenance cost than the model-oriented maintenance cost than the model-oriented counterpart when requirements change (i.e., counterpart when requirements change (i.e., ability to adjust the model)ability to adjust the model)


1919

Continuation of AdvantagesContinuation of Advantages

Assertions can be constructed to represent Assertions can be constructed to represent illegal behaviors, whereas the monolithic model illegal behaviors, whereas the monolithic model typically only represents “good behavior”typically only represents “good behavior”

It is much easier to trace the expected and It is much easier to trace the expected and actual behaviors of the target system to the actual behaviors of the target system to the required behaviors in the requirements space required behaviors in the requirements space and the formal assertions can be used directly and the formal assertions can be used directly as input to the verifiers in the verification as input to the verifiers in the verification dimensiondimension


2020

Continuation of AdvantagesContinuation of Advantages

Conjunction of all the assertions becomes Conjunction of all the assertions becomes a “single” formal model of a a “single” formal model of a conceptualized system from the conceptualized system from the requirement spacerequirement spaceCan be used to check for inconsistencies and Can be used to check for inconsistencies and

other gaps in the specifications with the help other gaps in the specifications with the help of computer-aided toolsof computer-aided tools


2121

Validation of Formal AssertionsValidation of Formal AssertionsFormal assertions must be executable to Formal assertions must be executable to

allow the modelers to visualize the true allow the modelers to visualize the true meaning of the assertions via scenario meaning of the assertions via scenario simulationssimulations

One way to do this is to use an iterative One way to do this is to use an iterative process that allows the modeler toprocess that allows the modeler toWrite formal specifications using Statechart Write formal specifications using Statechart

assertionsassertionsValidate the correctness of the assertions via Validate the correctness of the assertions via

simulated test scenarios within the JUnit test-simulated test scenarios within the JUnit test-frameworkframework


2222

Validation of Statechart Assertion Validation of Statechart Assertion via Scenario-based Testing via Scenario-based Testing

Statechart model with embedded statechart assertions

JUnit test suite

Scenario-based test cases

isSuccess()

Assertion Thread

Assertion


2323

Process for Validating Assertions Process for Validating Assertions (Utilizing the Executable SRM)(Utilizing the Executable SRM)

Start by testing individual assertions using the Start by testing individual assertions using the scenario-based test cases to validate the scenario-based test cases to validate the correctness of the logical and temporal meaning correctness of the logical and temporal meaning of the assertionsof the assertions

Next test the assertions using the scenario-Next test the assertions using the scenario-based test cases subjected to the constraints based test cases subjected to the constraints imposed by the objects in the SRM conceptual imposed by the objects in the SRM conceptual modelmodel

Then use an automated tool to exercise all Then use an automated tool to exercise all assertions together to detect any conflicts in the assertions together to detect any conflicts in the formal specificationformal specification


2424

A process for formal specification A process for formal specification and computer-aided validationand computer-aided validation

Stakeholder’s Input(mission statements, operation concepts

documents, user expectations, etc.)

Use Case Scenarios

Dynamic UML Models(Message Sequence Charts, Activity

Diagrams etc.)

Static UML Models(Object Class Diagrams)

Executable Assertions

JUnit TestFramework

White-box Automatic Tester

(1) Tests driven by use case scenarios

without the application

context

(2) Tests driven by use case scenarioswith the application context

(3) Tests drivenby white-box tester

for detecting assertion (and requirement) conflicts


2525

Runtime Verification (RV)Runtime Verification (RV)Uses executable SRMsUses executable SRMsMonitors the runtime execution of a system Monitors the runtime execution of a system

and checks the observed runtime behavior and checks the observed runtime behavior against the system’s formal specificationagainst the system’s formal specification It serves as an automated observer of the It serves as an automated observer of the

program’s behavior and compares it with the program’s behavior and compares it with the expected behavior per the formal specificationexpected behavior per the formal specification

Requires that the software artifacts Requires that the software artifacts produced by the developer be instrumentedproduced by the developer be instrumented


2626

Execution-based Model Execution-based Model Checking (EMC)Checking (EMC)

Can be used if state-based design models are Can be used if state-based design models are availableavailable

A combination of RV and Automatic Test A combination of RV and Automatic Test Generation (ATG)Generation (ATG) Large volumes of automatically generated tests are Large volumes of automatically generated tests are

used to exercise the program or system under test, used to exercise the program or system under test, using RV on the other end to check the SUT’s using RV on the other end to check the SUT’s conformance to the formal specificationconformance to the formal specification

Examples of ATG tools that can be used in Examples of ATG tools that can be used in combination with RV to conduct EMCcombination with RV to conduct EMC StateRover’s white-box automatic test-generator StateRover’s white-box automatic test-generator

(WBATG)(WBATG) NASA’s Java Path Finder (JPF)NASA’s Java Path Finder (JPF)


2727

Execution-based Model Checking Execution-based Model Checking of State-Based Design Modelsof State-Based Design Models

Statechart model with embedded statechart assertions

JUnit test suite

Auto-generated(white box)test cases

isSuccess()

StateRover Statechart Model

Primary Thread

Prmary Statechart

Assertion Thread

Assertion


2828

Three Ways in Which to Use Three Ways in Which to Use the Auto-generated Teststhe Auto-generated Tests

To search for severe programming errors, To search for severe programming errors, of the kind that induces a JUnit error of the kind that induces a JUnit error status, such as NullPointerExceptionstatus, such as NullPointerException

To identify test cases which violate To identify test cases which violate temporal assertionstemporal assertions

To identify input sequences that lead the To identify input sequences that lead the statechart under test to particular states of statechart under test to particular states of interestinterest


2929

ExampleExample

StateRover generated WBTestCase StateRover generated WBTestCase creates sequences of events and creates sequences of events and conditions for the state chart under testconditions for the state chart under testOnly sequences consisting of events that the Only sequences consisting of events that the

SUT or some assertion is sensitive to, by SUT or some assertion is sensitive to, by repeatedly observing all events that repeatedly observing all events that potentially affect the SUT when it is in a given potentially affect the SUT when it is in a given configuration state, selects one of those configuration state, selects one of those events and fires the SUT using this eventevents and fires the SUT using this event


3030

Hybrid Model- and Hybrid Model- and Specification-based WBATGSpecification-based WBATG

StateRover’s WBTestCase auto-generatesStateRover’s WBTestCase auto-generatesEventsEventsTime-advance increments, for the correct Time-advance increments, for the correct

generation of timeoutFire eventsgeneration of timeoutFire eventsExternal data objects of the type that the External data objects of the type that the

statechart prototype refers tostatechart prototype refers toWBATG observes all entities, namely, the WBATG observes all entities, namely, the

SUT and all embedded assertionsSUT and all embedded assertions It collects all possible events from all of those It collects all possible events from all of those

entitiesentities


3131

Verification of Target CodeVerification of Target Code

If only executable code is available, the If only executable code is available, the IV&V team can use the StateRover white-IV&V team can use the StateRover white-box tester in tandem with the executable box tester in tandem with the executable assertions of the SRM to automate the assertions of the SRM to automate the testing of the target code produced by the testing of the target code produced by the developerdeveloperExecutable assertions of the SRMExecutable assertions of the SRM

Keep track of the set of possible next events to Keep track of the set of possible next events to drive the SUTdrive the SUT

Serve as the observer for the RV during the testServe as the observer for the RV during the test


3232

Automated testing using the Automated testing using the system reference modelsystem reference model

SUT -

(instance of class model)Assertions

ExternalAssertionChecker

WBATG

1. Observe events,data, time delays

3. Dispatch input event and data

4. Output events

Timer

SUT - model

(instance of class model)

Implement time delays

5. isSuccess()

2. incrTime()


3333

Manual Examination of the Developer-Manual Examination of the Developer-Generated RequirementsGenerated Requirements

IV&V team can use the SRM to validate the textual IV&V team can use the SRM to validate the textual descriptions of the requirements produced by the descriptions of the requirements produced by the developerdeveloper Start by associating the developer-generated requirements with Start by associating the developer-generated requirements with

the use cases to obtain the context for assessing the the use cases to obtain the context for assessing the requirementsrequirements

Next, trace the developer-generated requirements to the other Next, trace the developer-generated requirements to the other artifacts, for example trace the requirements to theartifacts, for example trace the requirements to the

Activity and sequence diagrams to help identify the subsystems or Activity and sequence diagrams to help identify the subsystems or components responsible for the system requirements components responsible for the system requirements

Domain model to identify the correct naming of the objects and Domain model to identify the correct naming of the objects and eventsevents

Then use the traces to identify the critical components of the Then use the traces to identify the critical components of the target system for more thorough testingtarget system for more thorough testing


3434

RecapRecap The IV&V team needs to The IV&V team needs to

capture its own capture its own understanding of the understanding of the problem to be solved and problem to be solved and the expected behavior of the expected behavior of any system for solving the any system for solving the problem, using SRMsproblem, using SRMs

Complex system Complex system sequencing behaviors can sequencing behaviors can mainly be understood and mainly be understood and their formal specifications their formal specifications can most effectively be can most effectively be validated via execution-validated via execution-based techniques based techniques We advocate the use of We advocate the use of

assertion-oriented assertion-oriented specificationspecification

We presented a We presented a framework for framework for incorporating computer-incorporating computer-aided validation into the aided validation into the IV&V of complex reactive IV&V of complex reactive systemssystems

We described how the We described how the SRM can be used to SRM can be used to automate the testing of automate the testing of the software artifacts the software artifacts produced by the produced by the developer of the systemdeveloper of the system


3535

Challenge for the NASA’s Software Challenge for the NASA’s Software Engineering CommunityEngineering Community

Taking the proposed exotic validation framework Taking the proposed exotic validation framework from being exotic to being ubiquitous while from being exotic to being ubiquitous while harnessingharnessing ““Creative destructionCreative destruction,” coined by the late Joseph ,” coined by the late Joseph

SchumpeterSchumpeter Reallocate resources to new, productive business practices Reallocate resources to new, productive business practices

(antithesis of catering to the human need for stability and (antithesis of catering to the human need for stability and permanence)permanence)

““Disruptive innovationDisruptive innovation,” coined by Clayton ,” coined by Clayton ChristensenChristensen

Cause a technological innovation, product, or service to Cause a technological innovation, product, or service to overturn the existing dominant technology or status quo overturn the existing dominant technology or status quo product in the marketproduct in the market

A Framework for Computer-aided Validation

Documents

Transcript of A Framework for Computer-aided Validation