IV&V Facility 1 FY2002 Initiative: Software Architecture Metrics Hany Ammar, Mark Shereshevsky,...

1

IV&V Facility

FY2002 Initiative:

Software Architecture Metrics

Hany Ammar, Mark Shereshevsky,Nicholay Gradetsky, Diaa Eldin Nassar, Walid AbdelMoez, J. Shriver, A. Jalan

LANE Department of Computer Science and Electrical EngineeringWest Virginia University

Ali Mili, Bo Yu, Yan Wang, Krupa Doshi

College of Computing ScienceNew Jersey Institute of Technology

The OSMA Software Assurance Symposium Berkley Springs, WV, September, 2002

The more information I give The more information I give

the more errors I may pass–the more errors I may pass– Couched by TimCouched by Tim

WVU UI: Software Architecture MetricsWVU UI: Software Architecture Metrics

2

IV&V Facility

Outline

• Core ideaThe amount of information Transferred

• What we can do

• Benefits

• Project Overview

• Research Objectives

• Quantitative Factors

• Error Propagation

• Change Propagation

• Future Work

3

IV&V Facility

Core ideaThe amount of information Transferred

• If component A talks to components B– It is important how much information can be transferred– In Information theoretic terms this is measured by the entropy of data and

control information

• Standard view– Call graphs – control-flow and data-flow information between components– “This talks to that”

• What is the amount of information transferred?– example1:

• Binary information- entropy is 1-bit– example2:

• 1000 possible commands – entropy is 10-bits

• Intuition:– Error propagation in example2 >

Error propagation in example1

4

IV&V Facility

What We Can Do Develop error propagation matrix (details follow)

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10C1 1 0.11 0.15 0.3 0.76 0.73 0 0 0 0C2 0.37 1 0 0 0 0 0 0.52 0 0C3 0.03 0.32 1 0 0 0 0 0 0 0C4 0.03 0.19 0 1 0 0 0 0 0 0C5 0 0.67 0 0 1 0 0 0 0 0C6 0 0.28 0 0 0 1 0 0 0 0C7 0.32 0 0 0 0 0 1 0 0 0C8 0 0 0 0 0 0 0 1 0 0C9 0 0 0 0 0 0 0 0 1 0

C10 0 0 0 0 0 0 0 0 0 1

B

A

Component B (listens)

alarm!!!

alarm!!!

Component A (talks)

5

IV&V Facility

Benefits

• Use metrics to identify and focus on problem areas in the architecture

• There is a need to analyze the quality of reference architectures in product-line architectural development (NASA Reference Architecture – the SEEDS project at GFSC, Workshop was held in June 17 - 19, 2002 in San Diego, CA)

http://lennier.gsfc.nasa.gov/seeds/• The developments of techniques for measuring

Error/Change Propagation help analysts identify trouble spots in the architecture (advances the state of the art)

• The development of automated tools help the analyst apply these techniques on large architectures (advances the state of the practice)

6

IV&V Facility

Project Overview

FY01• Defined quantitative factors that are relevant to qualitative

attributes of the architecture

– Error Propagation – Change Propagation– Requirements Propagation

• Defined information theoretic (Entropy-based) architecture metrics

7

IV&V Facility

Project Overview (cont.)

FY02

• Establish analytical/and empirical relationships between the Entropy-based metrics and Error/Change propagation factors

• Automate computation of the metrics, and apply to a NASA case study, – Architecture Metrics Tool to be presented in the tools

session, www.csee.wvu.edu/swarch/

8

IV&V Facility

Project Overview (cont.)

FY03

• Apply the process and metrics to several NASA case studies

- Use sanitized Architecture information to estimate the quantitative factors, (links to SIAT tool project with Mark Suder)

- Compare Entropy-based metrics to metrics produced from existing metrics tools such as McCabe, (links to Tim Menzies machine learning tool)

• Conduct empirical Studies to validate the analytical and empirical relationships

9

IV&V Facility

Research Objectives

• Define information metrics for software architectures based on information theoretic measures

• Automate the process of computing these metrics.

• Apply the process and metrics to NASA projects

• Validate the correctness and usefulness of these metrics.

10

IV&V Facility

Quantitative Factors

Selected and defined the following factors• Error Propagation, Significance for Reliability• Change Propagation, Significance for

Maintainability• Requirements Propagation, Significance for

Evolvability

IV&V Facility

11

Definition of Quantitative Factors: Error Propagation

The error propagation (EP) is a measure of the probability that an error in the message sent by A will cause an error in B.

S

X

A B

xx` x` xOR

Error

12

IV&V Facility

Error Propagation

A B

VABv1

v3v2

S1 S2

S3

)(1

),(1

BA

BBA

VR

SVTR

EP(A B) =

Observed Message Redundancy (OMR)

TR(VAB, SB) measures the average amount of redundancy in

messages that B receives from A as seen by component B

Intrinsic Message Redundancy (IMR)

R(VAB) measures the intrinsic amount of redundancy in the

messages that B receives from A

13

IV&V Facility

Static Error Propagation Matrix


C10 0.08 0 0 0 0 0 0 0 0 1

B

A

Error Propagation Coefficient (static) = 0.0503

14

IV&V Facility

Dynamic Error Propagation Matrix


C10 0 0 0 0 0 0 0 0 0 1

B

A

Error Propagation Coefficient (dynamic) = 0.0534

15

IV&V Facility

Controlled Experiments Empirical Validation of Error Propagation Analysis

ExtractModel Info.

.rtmdl DesignExperiments

CorruptModel

Simulateand Log

Normal Log

Error Logs

AnalyzeLogs

Empirical Error Propagation

Components,Connectors

Connector,Message

MATLAB

SummitBASIC

FaultModel

16

IV&V Facility

Error PropagationControlled Experiments Results

The Analytical/Empircal Error Propagation

0.0000

0.2000

0.4000

0.6000

0.8000

1.0000

1.2000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Connetors

EP Analytical

Empirical

Fault Injection

0

200

400

600

800

1000

1200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Connector

# F

au

lt I

nje

cti

on

Correlation results

-1.5000

-1.0000

-0.5000

0.0000

0.5000

1.0000

1.5000

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Connector

Co

rrel

atio

n C

oef

fici

ent

•The following correlation results are based on 3562 experiments

•The correlation goes down as the number of injected faults in the experimental side goes down

•The over all correlation coefficient between the two matrices = 0.8335.

17

IV&V Facility

Change Propagation

We define Change Propagation from component A to component B as the probability that a change in A due to corrective/ perfective maintenance requires a change in B to maintain the overall function of the system.

We propose an analytical formula that approximate change propagation, and present conducted empirical study to measure change propagation.

18

IV&V Facility

Change PropagationAnalytical study

1||

12||),( ||

)(DIR||||

A

BA

IA

BIIA

O

OBACP

•Where |IX| and |OX| stand for the number of inputs and

outputs, respectively of component X.•DIR(Deterministic injectivity rate) of X is defined as

|)(|

||log

||

|)(| DIR(X)

1

1

vX

I

I

vX X

Ov XX

An upper bound on Change Propagation FactorCP between two components A and B

19

IV&V Facility

Change PropagationEmpirical Study

• In our experiment, we randomly select a large amount of changes within each component, then go into the source code level to see, if the change will propagate to other components.

• By computing for each pair of components (A,B), the number for which change in A cause a change in B, we are able to derive the change propagation matrix.

20

IV&V Facility

Change Propagation Matrix (Empirical study)

C1 C2 C3 C4 C5 C6 C7C1 1 0 0 0 0 0 0.6

C2 0.6 1 0.6 0 0.8 0 0.8

C3 0 0 1 0 0 0 0.2

C4 0.6 0 0 1 1 0 0

C5 0 0.2 0 0 1 0 0.2

C6 0 0 0 0 0.2 1 0.8

C7 0.2 0.2 0 0 0 0.2 1

A

21

IV&V Facility

Future Work

• Investigate correlation between Error Propagation estimates and data measured in several NASA software (SIAT using X34, ITOS case studies)

• Investigate correlation between Change Propagation estimates and data measured from several applications, for more examples of changes.

• Investigate the correlation between static and dynamic metrics

• Develop a methodology for estimating Requirements Propagation

IV&V Facility 1 FY2002 Initiative: Software Architecture Metrics Hany Ammar, Mark Shereshevsky,...

Documents

Transcript of IV&V Facility 1 FY2002 Initiative: Software Architecture Metrics Hany Ammar, Mark Shereshevsky,...