Injection of realistic software faults for experimental software

risk assessment

Henrique MadeiraCritical Software, SA

andUniversity of Coimbra, DEI-CISUC

Coimbra, Portugal

Universidade de Coimbra

NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

What’s the point of injecting software faults?

• Software faults are most probably the major cause of computer system outages

• Goals: Experimental risk assessment in component-based

software development Dependability evaluation of COTS components Robustness testing Fault tolerance layer evaluation Dependability benchmarking

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Two possible injection points

1. Injection of interface faults in software components (classical robustness testing)

Interface faults

SW component under test

OutputInput

Software faults

Target SW component

OutputInput

2. Injection of realistic software faults inside software components (new approach)

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Example of results obtained with interface faults (robustness testing)

requestedSize1 = 4294967295;returnStatus = rtems_region_get_segment (regionId, requestedSize1, option, timeout, ptsegment1);

Memory exception at fffffffc (illegal address)Unexpected trap (0x09) at address 0x0200aaacData access exception at 0xfffffffc

Excerpt of application code:

Result:

This software fault was discovered automatically by the Xception robustness testing tool!

Robustness failure in RTEMS 4.5.0

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Example of results obtained with injection of software faultsWhat happens if a software bug in a device driver becomes active?

Availability Feedback Stability

0

2

4

6

8

WorstBest

0

2

4

6

8

BestWorst

0

1

2

3

4

5

6

7

Best

Windows XPWindows 2000Windows NT

Worst

• Software faults are injected in a device driver using the G-SWFIT technique.

• The device is heavily used by programs running during test.

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Component-based software development

• Vision: development of systems using pre-fabricated components. Reuse custom components or buy software components available from software manufactures (Commercial-Off-The-Shelf: COTS).

• Potential advantages: Reduce development effort since the components are already

developed, tested, and matured by execution in different contexts Improve system quality Achieve of shorter time-to-market Improve management of increased complexity of software

• Trend → use general-purpose COTS components and develop domain specific components.

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Some potential problems

• COTS In general, functionality description is not fully provided. No guarantee of adequate testing. COTS must be assessed in relation to their intended use. The source code is normally not available (makes it impossible

white box verification & validation of COTS).

• Reuse of custom components in a different context may expose components faults.

Using COTS (or reusing custom components) represent a risk! How to assess (and reduce) that risk?

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Case-study: I-don’t-care-about software architecture diagram

Software components

Different sizes

Different levels of granularity

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Question 1This is a COTS!

What’s the risk of using it in my

system?

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Question 2

This is custom component previously built!

What’s the risk of reusing it in my system?

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Question 3

This is a new custom component!

What’s the risk of using it without further testing?

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Experimental risk assessment

Risk = prob. of bug * prob. of bug activation * impact of bug activation

Component 1Custom

Component 3COTS

Component 4Custom

Exception handler

Component 2COTS

Example of question:

What’s the risk of using Component 3 in my system?

Software complexity

metrics

Injection of software

faults

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Again, two possible injection points

Component 1Custom

Component 3COTS

Component 4Custom

Exception handler

Component 2COTS

Injection of realistic SW faults

Injection of interface faults

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Why injection or real software faults?

Component 1Custom

Component 3COTS

Component 4Custom

Exception handler

Component 2COTS

Injection of SW faults

Injection of SW faults

• Error propagation through non conventional channels is a reality.

• Faults injected inside components are more representative.

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

How to inject software faults?

• Use G-SWFIT (ISSRE 2002, DSN 2003, DSN 2004)

Injects the top N most common software faults. This top N is based on field data (our study + ODC data from

IBM) and corresponds to ~65% of the bugs found in field data. Injects faults in executable code. Largely independent on the programming language, compiler,

etc that have generated the executable code.

• G-SWFIT is now a reasonably mature technique.

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

G-SWFITGeneric software fault injection technique

010110001001001

Target executable

code

Low-level code mutation engine

Low level mutated versions

. . .Library of software fault injection operators

01X110001001001

010110X01001001

010110001X01001

01011000100X001

Emulate common programmer mistakes

The technique can be applied to binary files prior to execution or to in-memory running processes

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Experimental risk assessment (again)

Risk = prob. of bug * prob. of bug activation * impact of bug activation

Component 1Custom

Component 3COTS

Component 4Custom

Exception handler

Component 2COTS

Example of question:

What’s the risk of using Component 3 in my system?

Software complexity

metrics

Injection of software

faults

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Estimation of the probability of residual bugs

Component 1Custom

Component 3COTS

Component 4Custom

Exception handler

Component 2COTS

Target code

• Many studies indicate that fault probability correlates with the software module complexity

• Metrics of software complexity base on:

• Static feature of the code;

• Dynamic features;

• Possible information on the development process (type of tests, etc);

• ...

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Estimation of bug activation probability and bug impact

Component 1Custom

Component 3COTS

Component 4Custom

Exception handler

Component 2COTS

Software faults

• Test campaigns to evaluate the activation probability and the impact of software faults (bugs) inside the component in the rest of the system.

• Use software metrics to choose the modules to inject faults and define trigger locations accordingly.

Target code

Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Conclusions and current work on experimental risk assessment

• Experimental software risk assessment seems to be viable.

• Risk is a multi-dimensional measure. Many software risks can be assessed, depending on the property I’m interested in.

• Current work: Improve the G-SWFIT technique:

– Improving current tool.– Expansion of the mutation operator library– Construction of a field-usable tool for software fault emulation in Java

environments Study of software metrics and available tools. Composeability measures. Real case-studies to demonstrate the methodology.