SE300 Software Engineering Practicesmercury.pr.erau.edu/~siewerts/se300/documents/Lectures/... ·...

April 7, 2015 Sam Siewert

SE300

Software Engineering Practices

Lecture 13 – Traditional Testing,

Planning and Re-planning, Formal

Methods of V&V and Standards

Minute Paper #7

What is your Status (Individual View) on Assignment #5,

Completion of Analysis and Design?

What is your Status (Individual View) on Readiness to

Test your Design and Prototype, Mock-up, or Build a

PoC for key Algorithms?

Sam Siewert 2

Testing Strategies

Sam Siewert 3

Domain Models – Use Case Details Complete a Design

Provide Code Re-Use Candidates and PoC

Sam Siewert 4

Start Here!

https://www.modelio.org/

Helpful Validation and Verification Features for

Design

• Integrated Models

• Checklists – Completeness

• CPP and Java Code Generation

USE Modelio as your DESIGN TOOL

https://www.modelio.org/

Testing Levels Acceptance Tests – Requirements, Deployment Diagram – Test for each Requirement

– Gating Tests for Shipment, Release Notes

System Test – Analysis Use Cases, Domain Model, OIM Sequence Message Passing, CFD/DFD, Block Diagrams – End-to-End (Fully Integrated)

– Performance

– Major Features

– Fault-Injection

Integration and Test - OM, OIM Methods and Collaboration, Activity Diagrams, Package Diagrams, Subsystems – Components, Subsystems

– Interfaces, Protocols

– Verification of Public Interfaces (Methods and Attributes)

Unit Tests - State Machines, Method Details – Functional, Basic Block, Test Cases (Vectors)

– Test Drivers to Instantiate and Exercise Methods (Functions)

Sam Siewert 5

Copyright {c} 2014 by the McGraw-Hill Companies, Inc. All rights Reserved.

20-6

What Is Software Testing?

• Software testing is a dynamic validation

activity to detect defects and undesirable

behavior in the software, and demonstrate that

the software satisfies its requirements and

constraints.


20-7

Objectives of Software Testing

• Testing is a process of executing a program

with the intent of finding an error.

• A good test case is one that has a high

probability of finding an undetected error.

• A successful test is one that uncovers an

undiscovered error.


20-8

What is Testing and What is Not?

• Testing is a process for finding semantic or

logical errors.

• Testing is not aimed to find syntax errors.

• Testing can reveal the presence of errors, NOT

their absence.

• Class exercise: what is the difference between

testing and debugging?


20-9

Why Software Testing?

• Review and inspection are not adequate because they

do not execute the program. Can you test a car

without driving it?

• Why need testing:

– Instrumentation systems & appliances

• coded in assembly/machine language ===> costly to repair

– Process control systems

• failure is politically unacceptable & economically undesirable.

– Military systems

• involve risk of human life

– Safety - Critical Health Care Systems.


20-10

Software Testing in Practice

• Testing amounts to 40% -- 80% of total development

costs

– ~ 40% for information systems

– more than 80% for real-time embedded systems

• On one occasion the average costs of a field detected

bug was US$15,000 (1991).

• Testing receives the least attention, and often not

enough time and resources.

• Testers are often forced to abandon test efforts

because changes have been made.


20-11

Software Testing in Practice

• Most companies’ new hires are testers.

• Most testing work is manually testing the

software system; help from tools is limited.

• In many cases, testing is not performed by

using systematic testing methods or

techniques. It is not effective.

• There are “conflict of interest” between testers

and developers, and testers and management.


20-12

Benefit of Software Testing

• Benefits: – Conscious of need for error-free software.

– Ensure the more flagrant kinds of errors will be detected.

– Improving software quality.

– Adding a layer of quality assurance to inspection, walkthrough, and peer review.

– Helping in data collection for root cause analysis and process improvement.

– Providing a framework for applying SQA techniques.

• Systematic software testing is more effective than

programmers’ casual testing.


20-13

Test Categories

• Unit Testing: testing at the individual module level, functions, modules, classes, etc.

• Integration Testing: testing the interfaces among the modules or components.

• Validation Testing: test according to customer’s functional requirements.

• System Testing

– Recovery Testing

– Security Testing

– Stress Testing

– Performance Testing


20-14

Test Driven Development

Y

Write/edit tests for

features to be

implemented next

Run new tests

All new

tests fail?

Write/edit

production

code

Run all tests All tests

pass?

Refactor code

More features

to implement?

Halt N

N

N

Y

Y

Test coverage

accomplished? N

Add tests to

increase coverage

Y

Prepare for

TDD


20-15

Software Testing Budget

• Size of testing budget:

• expressed in terms of x% of total SW

development costs.

• in terms of system size: $x per statement.

• compute in terms of productivity of the test

team in terms of time: planned & spent.

SQA – SE420

Test Methods at Each Level for Concurrent Process

– Acceptance Tests – Requirements, Deployment Diagram

– System Test – Analysis Use Cases, Domain Model, OIM

Sequence Message Passing, CFD/DFD, Block Diagrams

– Integration and Test - OM, OIM Methods and Collaboration,

Activity Diagrams, Package Diagrams, Subsystems

– Unit Tests - State Machines, Method Details

Details of Composing Test Drivers, Cases, and

Automation Verification and Validation

Regression Testing

Methods of Black-box and White-box Test Development

Sam Siewert 16

Testing Levels

Acceptance Tests – Requirements, Deployment Diagram

System Test – Analysis Use Cases, Domain Model, OIM Sequence Message Passing, CFD/DFD, Block Diagrams – End-to-End (Fully Integrated)

– Performance

– Major Features

– Fault-Injection

Integration and Test - OM, OIM Methods and Collaboration, Activity Diagrams, Package Diagrams, Subsystems

Unit Tests - State Machines, Method Details

Sam Siewert 17

Take Away Test Definition and Development Should Start With Requirements

Testing Concurrent with Analysis, Design, Implementation

Define Test Cases, Methods, and Use Tools at All Levels of Analysis, Design, Implementation

Define Exit Criteria, So Test Completion is not Arbitrary – E.g. Code Path Coverage

Gating Tests Prevent Shipment of Defective Software

Some Defects and Deficiencies May Be Documented as Limitations in Release Notes

Can’t Prove Code is Bug Free - SWE Formal Methods – Advanced Concepts to Prove Correctness [Research Topic]

– Pre-condition, Post-condition (Axiomatic Proofs)

– Prove Ability to Meet Deadlines (Rate Monotonic)

– Termination and Halting Problem and Undecidable Problems

Sam Siewert 18

en.wikipedia.org/wiki/Halting_problem

Project Planning and Cost Estimation

Assume Cost of Testing and Support

Ramp-Up Time [E.g. Hiring, Training, Tool Acquisition

and Configuration]

Re-Use is not Zero Cost, Even for Open Source

Need to At Least Have Scope Defined through

Requirements and Analysis

Refine and Update over Time

Sam Siewert 19

20

These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e

(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman.

Project Planning Task Set-I

Establish project scope

Determine feasibility

Analyze risks

Risk analysis is considered in detail in Chapter 25.

Define required resources Determine require human resources

Define reusable software resources

Identify environmental resources

21



Project Planning Task Set-II

Estimate cost and effort Decompose the problem

Develop two or more estimates using size, function

points, process tasks or use-cases

Reconcile the estimates

Develop a project schedule

Scheduling is considered in detail in Chapter 34.

• Establish a meaningful task set

• Define a task network

• Use scheduling tools to develop a timeline chart

• Define schedule tracking mechanisms

22



Estimation

Estimation of resources, cost, and schedule for

a software engineering effort requires experience

access to good historical information (metrics)

the courage to commit to quantitative predictions

when qualitative information is all that exists

Estimation carries inherent risk and this risk

leads to uncertainty

23



Write it Down!

Software

Project

Plan

Project Scope

Estimates

Risks

Schedule

Control strategy

24



To Understand Scope ...

Understand the customers needs

understand the business context

understand the project boundaries

understand the customer’s motivation

understand the likely paths for change

understand that ...

Even when you understand,

nothing is guaranteed!

25



What is Scope?

Software scope describes the functions and features that are to be delivered to

end-users

the data that are input and output

the “content” that is presented to users as a

consequence of using the software

the performance, constraints, interfaces, and

reliability that bound the system.

Scope is defined using one of two techniques: • A narrative description of software scope is developed

after communication with all stakeholders.

• A set of use-cases is developed by end-users.

26



Resources

project

people

skills

number

location

reusable

softwareOTS

components

full-experience

components

new

components

part.-experience

components

environment

hardware

software

tools

network

resources

27



Project Estimation

Project scope must be

understood

Elaboration (decomposition) is

necessary

Historical metrics are very helpful

At least two different techniques

should be used

Uncertainty is inherent in the

process

28



Estimation Techniques

Past (similar) project experience

Conventional estimation techniques

task breakdown and effort estimates

size (e.g., FP) estimates

Empirical models

Automated tools

29



Estimation Accuracy

Predicated on …

the degree to which the planner has properly

estimated the size of the product to be built

the ability to translate the size estimate into human

effort, calendar time, and dollars (a function of the

availability of reliable software metrics from past

projects)

the degree to which the project plan reflects the

abilities of the software team

the stability of product requirements and the

environment that supports the software engineering

effort.

30



Functional Decomposition

functional

decomposition

Statement of

Scope Perform a

Grammatical “parse”

31



Example: LOC Approach

Average productivity for systems of this type = 620 LOC/pm.

Burdened labor rate =$8000 per month, the cost per line of code is approximately $13.

Based on the LOC estimate and the historical productivity data, the total estimated project cost is $431,000 and the estimated effort is 54 person-months.

Works Best for Product A’ Based

On A – Next Generation

32



Example: FP Approach

The estimated number of FP is derived:

FPestimated = count-total 3 [0.65 + 0.01 3 S (Fi)]

FPestimated = 375

organizational average productivity = 6.5 FP/pm.

burdened labor rate = $8000 per month, approximately $1230/FP.

Based on the FP estimate and the historical productivity data, total estimated

project cost is $461,000 and estimated effort is 58 person-months.

Feature Point Good Approach for

Green Field or Upgrade Projects

33



Process-Based Estimation Example

Activity

Task

Function

UICF

2DGA

3DGA

DSM

PCF

CGDF

DAM

Totals

% effort

CC PlanningRisk

Analysis EngineeringConstruction

Release TotalsCE

analysis design code test

0.25 0.25 0.25 3.50 20.50 4.50 16.50 46.00

1% 1% 1% 8% 45% 10% 36%

CC = customer communication CE = customer evaluation

0.50

0.750.50

0.50

0.500.25

2.50

4.004.00

3.00

3.002.00

0.40

0.601.00

1.00

0.750.50

5.002.00

3.00

1.50

1.501.50

8.40

7.358.50

6.00

5.754.25

0.50 2.00 0.50 2.00 5.00

n/an/a

n/a

n/a

n/an/a

n/a

Based on an average burdened labor rate of $8,000 per month, the total estimated project cost is $368,000 and the estimated effort is 46 person-months.

Estimate Time in Each Cycle for

Evolutionary Process for Each Major

Function

Mission Critical Systems Systems where FAILURE and MAJOR Defects Cause Loss of Life, Private Data, Property or Significant Financial Impact – Avionics

– Enterprise Systems for Business, Banking, Finance

– Medical

– Industrial Automation and Infrastructure

– Personal Records, Business IP, National Security

Design and Implementation Assurance

Ideally Provably Correct, but Low to Zero Defect Assurance (Standards) Acceptable Due to Cost of Formal Proofs

Sam Siewert 34

Standards and Formal Methods Standards and Guidelines for Systems and Software Testing – Mission Critical

Enterprise Data Systems – Commercial Standards and IV&V Labs (E.g. Medusa Labs Test Tools), Standards for Protocol (SCSI T10 and SAS, Fiber Channel T11, FCoE, iSCSI) and Performance Test Standards (TPC, SPC), SNIA RAID

Aerospace – NASA Standards (https://standards.nasa.gov/documents/nasa ), Military Standards (MIL-STD-498), FAA Launch and Reentry, FAA

Medical Systems – IEC 62304, E.g. Commercial Support

Energy – NIST Testing, DoE STD-1172

Security – TCG, NIST ITL, NIST FISMA, NIST FIPS, NIST TIRs – Overview

Transportation – FAA DO-178B, Software Assurance, V&V, NHTSA Policies on Automated Vehicles

Sam Siewert 35

http://www.jdsu.com/Downloads/TestToolsGuide.pdf

http://www.t10.org/

http://www.t11.org/index.html

https://tools.ietf.org/html/rfc3720

http://www.tpc.org/

http://www.storageperformance.org/home/

http://www.snia.org/tech_activities/

http://www.snia.org/tech_activities/standards/curr_standards/ddf

https://standards.nasa.gov/documents/nasa

http://en.wikipedia.org/wiki/MIL-STD-498





https://www.faa.gov/about/office_org/headquarters_offices/ast/media/AST_Guide_to_Software_Safety_final_070706.pdf

https://www.faa.gov/about/office_org/headquarters_offices/ast/reports_studies/media/DMurray_SW REQTS_IAASS07_FINAL.pdf

http://en.wikipedia.org/wiki/IEC_62304

http://www.vectorcast.com/testing-solutions/software-testing-embedded-medical-devices-fda-iec-62304

http://www.nist.gov/software-testing-metrics-portal.cfm

http://energy.gov/sites/prod/files/2013/06/f1/DOE-STD-1172-2011.pdf



http://www.trustedcomputinggroup.org/

http://www.nist.gov/itl/

http://csrc.nist.gov/groups/SMA/fisma/overview.html

http://csrc.nist.gov/publications/PubsFIPS.html

http://csrc.nist.gov/publications/PubsNISTIRs.html

https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdf

http://en.wikipedia.org/wiki/DO-178B



http://www.faa.gov/documentLibrary/media/Order/1370.109.pdf

https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/cast/cast_papers/media/cast-11A.pdf

http://www.nhtsa.gov/About+NHTSA/Press+Releases/U.S.+Department+of+Transportation+Releases+Policy+on+Automated+Vehicle+Development

http://www.nhtsa.gov/About+NHTSA/Press+Releases/U.S.+Department+of+Transportation+Releases+Policy+on+Automated+Vehicle+Development

36



Chapter 28

Formal Modeling and Verification

Slide Set to accompany

Software Engineering: A Practitioner’s Approach, 8/e by Roger S. Pressman and Bruce R. Maxim

Slides copyright © 1996, 2001, 2005, 2009, 2014 by Roger S. Pressman

For non-profit educational use only May be reproduced ONLY for student use at the university level when used in conjunction

with Software Engineering: A Practitioner's Approach, 8/e. Any other reproduction or use is

prohibited without the express written permission of the author.

All copyright information MUST appear if these slides are posted on a website for student

use.

37



Formal Modeling and Verification

Cleanroom software engineering and formal

methods

Both demand a specialized specification approach and

each applies a unique verification method.

Both are quite rigorous and neither is used widely by the

software engineering community.

If you must build “bullet-proof” software, these

methods can help immeasurably.

38



The Cleanroom Process Model

39



The Cleanroom Strategy-I Increment Planning—adopts the incremental strategy

Requirements Gathering—defines a description of customer level requirements (for each increment)

Box Structure Specification—describes the functional specification

Formal Design—specifications (called “black boxes”) are iteratively refined (with an increment) to become analogous to architectural and procedural designs (called “state boxes” and “clear boxes,” respectively).

Correctness Verification—verification begins with the highest level box structure (specification) and moves toward design detail and code using a set of “correctness questions.” If these do not demonstrate that the specification is correct, more formal (mathematical) methods for verification are used.

Code Generation, Inspection and Verification—the box structure specifications, represented in a specialized language, are transmitted into the appropriate programming language.

40



The Cleanroom Strategy-II

Statistical Test Planning—a suite of test cases that exercise of “probability distribution” of usage are planned and designed

Statistical Usage Testing—execute a series of tests derived from a statistical sample (the probability distribution noted above) of all possible program executions by all users from a targeted population

Certification—once verification, inspection and usage testing have been completed (and all errors are corrected) the increment is certified as ready for integration.

41



Box Structure Specification

black box

state box

clear box

42



Box Structures

black box state box

clear box

43



Design Refinement & Verification

If a function f is expanded into a sequence g and h, the correctness condition for all input to f is:

• Does g followed by h do f?

When a function f is refined into a conditional (if-then-else), the correctness condition for all input to f is:

• Whenever condition <c> is true does g do f and whenever <c> is false, does h do f?

When function f is refined as a loop, the correctness conditions for all input to f is:

• Is termination guaranteed?

• Whenever <c> is true does g followed by f do f, and whenever <c> is false, does skipping the loop still do f?

44



Advantages of Design Verification

It reduces verification to a finite process.

It lets cleanroom teams verify every line of

design and code.

It results in a near zero defect level.

It scales up.

It produces better code than unit testing.

45



Cleanroom Testing

statistical use testing

tests the actual usage of the program

determine a “usage probability distribution”

analyze the specification to identify a set of stimuli

stimuli cause software to change behavior

create usage scenarios

assign probability of use to each stimuli

test cases are generated for each stimuli according

to the usage probability distribution

46



Certification

1. Usage scenarios must be created.

2. A usage profile is specified.

3. Test cases are generated from the profile.

4. Tests are executed and failure data are recorded and analyzed.

5. Reliability is computed and certified.

47



Certification Models

Sampling model. Software testing executes m random test cases and is certified if no failures or a specified numbers of failures occur. The value of m is derived mathematically to ensure that required reliability is achieved.

Component model. A system composed of n components is to be certified. The component model enables the analyst to determine the probability that component i will fail prior to completion.

Certification model. The overall reliability of the system is projected and certified.

48



Formal Methods

“Formal methods used in developing computer systems are mathematically based techniques for describing system properties. Such formal methods provide frameworks within which people can specify, develop, and verify systems in a systematic, rather than ad hoc manner.”

The Encyclopedia of Software Engineering [Mar01]

The Problem with conventional specs:

contradictions

ambiguities

vagueness

incompleteness

mixed levels of abstraction

49



Formal Specification

Desired properties—consistency, completeness, and lack of

ambiguity—are the objectives of all specification methods

The formal syntax of a specification language enables

requirements or design to be interpreted in only one way,

eliminating ambiguity that often occurs when a natural

language (e.g., English) or a graphical notation must be

interpreted

The descriptive facilities of set theory and logic notation enable

clear statement of facts (requirements).

Consistency is ensured by mathematically proving that initial

facts can be formally mapped (using inference rules) into later

statements within the specification.

50



Formal Methods Concepts data invariant—a condition that is true throughout the

execution of the system that contains a collection of data

state

Many formal languages, such as OCL (Section 28.5) , use the notion of states as they were discussed in Chapters 7 and 8, that is, a system can be in one of several states, each representing an externally observable mode of behavior.

The Z language (Section 28.6)defines a state as the stored data which a system accesses and alters

operation—an action that takes place in a system and reads or writes data to a state

precondition defines the circumstances in which a particular operation is valid

postcondition defines what happens when an operation has completed its action

51



States and Data Invariant The state of the spooler is represented by the four components Queues, OutputDevices, Limits, and Sizes.

The data invariant has five components:

• Each output device is associated with an upper limit of print lines

• Each output device is associated with a possibly nonempty queue of files awaiting printing

• Each file is associated with a size

• Each queue associated with an output device contains files that have a size less than the upper limit of the output device

• There will be no more than MaxDevs output devices administered by the spooler

52



Operations

An operation which adds a new output device to the

spooler together with its associated print limit

An operation which removes a file from the queue

associated with a particular output device

An operation which adds a file to the queue associated

with a particular output device

An operation which alters the upper limit of print lines for

a particular output device

An operation which moves a file from a queue associated

with an output device to another queue associated with a

second output device

53



Pre- & Postconditions

For the first operation (adds a new output device to the spooler together with its associated print limit):

Precondition: the output device name does not already exist and that there are currently less than MaxDevs output devices known to the spooler

Postcondition: the name of the new device is added to the collection of existing device names, a new entry is formed for the device with no files being associated with its queue, and the device is associated with its print limit.

Minute Paper #8

Can you Test a Design? Beyond Inspections (Walk-

throughs and Formal Inspection)

If not, Must Test Wait for Code Complete to Start?

Sam Siewert 54

SE300 Software Engineering Practicesmercury.pr.erau.edu/~siewerts/se300/documents/Lectures/... ·...

Documents

Transcript of SE300 Software Engineering Practicesmercury.pr.erau.edu/~siewerts/se300/documents/Lectures/... ·...