West Virginia

University

Architectural-Level Risk Analysis for UML Dynamic Specifications

Dr. Sherif M. YacoubDr. Sherif M. Yacoubsherif_yacoub@hp.com

Hewlett-Packard LaboratoriesPalo Alto, CA

Alaa Ibrahim, and Hany H. Alaa Ibrahim, and Hany H. AmmarAmmar

{ibrahim,ammar}@csee.wvu.eduDepartment of Computer Science

and Electrical Engineering

West Virginia University99thth International Conference on Software Quality International Conference on Software Quality Management, SQM2001Management, SQM2001

1818thth-20-20thth April, 2001 April, 2001Loughborough University, Loughborough, EnglandLoughborough University, Loughborough, England

West Virginia

University

Outline

Research Objectives

Methodology

Towards an Automated Methodology

Process

Case Study: The Pacemaker example

Conclusions

West Virginia

University

Architectural-Level Risk Assessment

Methodology at the early stages of

development(S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October,

2000)

Automated Environment

Automated Risk Assessment

Research Objectives

West Virginia

University

Automated Risk Assessment(continued) Architectural-Level Risk

Assessment Methodology (S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October, 2000)

Utilizes:

• Dynamic Metrics: Component Complexity cpxi Connector

Complexity cpxij (S. Yacoub, H. Ammar, and T. Robinson. Metrics'99,

November 1999)

• Failure Mode Effect Analysis FMEA (MIL_STD 1629A to define

Component Severity svrtyi Connector Severity svrtyij)

• Component Dependency Graphs CDG (adopted from: S. Yacoub, B.

Cukic, and H. Ammar. ISSRE'99 November 1999)

Defines:

• Heuristic Component Risk Factor hrfi = cpxi x svrtyi

• Heuristic Connector Risk Factor hrfij = cpxij x svrtyij

• Risk Aggregation Algorithm that produces HRFappl

West Virginia

University

• Model the architecture of the system using simulation models (UML-RT).

• Perform complexity analysis using simulation traces.

• Perform severity analysis using FMEA and simulation runs.

• Develop heuristic risk factors for components and connectors.

• Develop Components Dependency Graph for risk assessment purposes. (System/Subsystems)

• Aggregate the risk factors using the graph traversal algorithm.

Automated Risk Assessment Architectural-Level Risk Assessment

Methodology (continued) 6 Steps

West Virginia

University

Component Complexity Factors

Connector complexity Factors

CDG “hrfi and hrfij unidentified”

Formatted Excel charts

Violation Tables

UML Simulation Environment

SimulationSettings

SimulationLog andViolationReport

AnalysisTool

Timing Diag.

Violation Table

Analyst

Rose Real Time toolText File

MS Excel

ProcessingMacro

Inspection

Viewing Macro

UML Model

Observer

Sub RunSettings

AnalysisTool HRF

MS Excel

RiskMacro

Excel sheets

SeverityRanking

Severity Analysis (Failure/Effect analysis)

Automated Risk Assessment(continued) Automated Environment

CARA Tool

West Virginia

University

Model the architecture of the system together with the risk logging capability using Rose RealTime.

Adjust the simulation runs in the observer as desired.

Run the simulation and get two log files containing:

• Component complexities.

• Component Execution Time.

• A log of all the messages exchanged.

Automated Risk Assessment Automated Environment (continued)

Process

West Virginia

University

Process the log with Excel Risk Macro and get:

• Transition Probabilities.

• Connector complexities.

• CDG “where Risk Factors = Severity Factors * Complexity Factors (hrfi = cpxi x svrtyi )”

Perform severity analysis using FMEA and simulation runs.

Traverse the CDG using the Excel traversal macro.

Automated Risk Assessment Automated Environment

Process (continued)

West Virginia

University

Example: Pacemaker Main Use Case Diagram

Programming Mode

Operational Modes

DoctorsProgramer

Operating_in_AVIOperating_in_ AAT

Operating_in_ VVIOperating_in_ VVT

PatientsHeart

1

1

1 1

1

1

1

1

1

111

1

Operating_in_ AAI

1

1

1

1

Programming

1

1

1

1

«extend»«extend»«extend»«extend»

«extend»«extend»«extend»«extend»

«extend»«extend»

West Virginia

University

1) Develop a Simulation Model Capsule Diagram

Example: Pacemaker

West Virginia

UniversityAtrial statechartCase Study: Pacemaker (continued)

Idle

A_AVI

A_Self_inhibited

A_Self_triggered

ToOn ToOff

ToTriggered

ToInhibited

ToAVI

ToOnToOnToOn ToOffToOff

ToTriggered

ToInhibited

ToAVI

ToOn

West Virginia

UniversityAtrial statechartCase Study: Pacemaker (continued)

Refractory

Wait

Pacing

ToAVI

A_Pace_Pulse_Done

V_Refract_Done_Received

ini tial ize

V_Sense_Received

Time_Out

ToAVIToAVI

A_Pace_Pulse_Done

V_Refract_Done_Received

ini tial ize

V_Sense_Received

Time_Out

A sequence diagram for the AVI scenarioCommunication

GnomeAtrial Ventricular Heart

ToON

ToON

ToAVI

ToAVI

Refactoring

V SenseGot V Sense

RefTimeOut

Waiting

V Refract Done

Waiting

SensTimeOut

A Pace Start

A Pace Start

Pacing

Pacing

PaceTimeOut

A Pace Done

Pace

Refactoring

Refactoring Refactoring

A sequence diagram for the Programming scenarioProgrammer ReedSwitch CoilDriver Communication

GnomeAtrial Ventricular

ApplyMagnetEnableComm

EnableComm

Pulse

Receiving

IDLE

Pulse

Count++,ResetTimer

BitTimeoutDecode(Count)Store Bit in Byte

Byte Full?enqueue(byte)Yes

ByteTimeOut

Waiting for Bit

IDLE

Count = 1, SetTimer

PulseCount =0

Receiving

ByteTimeOut

OR

IDLE

Waiting For Byte

Validating

IsValid?

Processing

ToAVI

ToON

ToON

ToAVI

YesHerezaByte(ACK)

No

HerezaByte(NAK)

Waiting to Transmit

Waiting to Send Next Byte

West Virginia

University2) Perform Complexity

Analysis

s21

s22

I

init

initI

s11t11

t12

t13

s1

s2

VGx(s11) + VGa(t11) + VGx(s1)+ VGa(t12) + VGe(s2) + VGa(t13) +VGe(s22)

||

1

)()(X

x

ixxi oocpxPSoOCPX

A Transition between Composite States in a component’s Statechart

Operational Complexity of a component using the scenario profile and its complexity per scenario.

West Virginia

University2) Perform Complexity Analysis (cont’d)

A) Quantify Component Complexity Factors using dynamic complexity metrics.

RS CD CG AR VTProgramming ( 0.01) 8.3 67.4 24.3AVI (0.29) 53.2 46.8AAT (0.15) 100AAI (0.20) 100VVI (0.15) 100VVT (0.20) 100% of architecture complexity .083 0.674 0.243 50.428 48.572Normalized to max. complexity 0.002 0.013 0.005 1 0.963

West Virginia

University2) Perform Complexity

Analysis (cont’d)

100|},|),({|

),(

x

jijijix

ji

MT

ooOooooMooEOCx

||

1

)()(X

x

ixxi oOQFSPSoOQFS

||

1

),(),(X

x

jixxji ooEOCPSooEOC

Export Object Coupling

Export Object Coupling(EOC)

EOC with scenario profiles

OQFS with scenario profiles

the export coupling for component Ci with respect to component

Cj, is the percentage of the number of messages sent from Ci to

Cj with respect to the total number of messages exchanged

during the execution of the scenario x

West Virginia

University2) Perform Complexity Analysis (cont’d)

B) Quantify Connector Complexity Factors using dynamic coupling metrics.

RS CD CG AR VT Programmer HeartRS 0.0014 0.0014CD 0.003 0.011CG 0.002 0.0014 0.0014AR 0.25 1VT 0.27 0.873Programmer 0.0014 0.006Heart 0.123 0.307

West Virginia

University3) Perform Severity Analysis

In performing severity analysis, each potential failure mode is ranked according to the consequences of that failure mode.

Steps:

• Identifying Failure Modes Failure modes of individual components.

(Functional faults and state-based faults) Failure modes of individual connectors.

(Interface fault analysis)

West Virginia

University3) Perform Severity Analysis (cont’d)

Steps (cont’d):

• Conducting Effect Analysis Inject the fault. Simulate the faulty model. Monitor output and compare to expected output. Identify the effect of the fault.

• Rank Severity Identify category: Minor, Marginal, Critical, or

Catastrophic. Assign severity index to each component i as (svrtyi),

which takes a value of 0.25, 0.50, 0.75, and 0.95

West Virginia

University

Worst case severity found for the RS, CD, CG, VT, and AR are Minor(0.25), Minor(0.25), Marginal(0.50), Catastrophic(0.95) and Catastrophic (0.95), respectively

Connector Name Failure Mode Cause of Failure Effect of Failure Criticality of effectsRS Failed to enable

communicationError in translatingmagnet command

Unable to program thepacemaker, schedulemaintenance task.

Minor

CD Failed to generategood command

Fault in developingthe command

Unable to program thepacemaker, schedulemaintenance task.

Minor

CG Failed to validatecommand

Fault in thevalidationprocedure

Cannot program thepacemaker, schedulemaintenance task.

Minor

Mis-interpreting aVVT command forVVI

Fault in processingcommand routine

Heart is continuously triggeredbut device is still monitored byphysician, need immediate fixor disable.

Marginal

VT No heart pluses aresensed though heart isworking fine.

Heart sensor ismalfunctioning.

Heart is incorrectly paced,patient could be harmed bycontinuous pulses.

Critical

Refract timer does notgenerate a timeout inan AVI mode

Timer not setcorrectly.

AR and VT are in refactoringstate, no pace is generated forthe heart, patient could die.

Catastrophic

AR Wait timer does notgenerate a timeout inAAI mode

Timer not setcorrectly.

AR stuck at the wait state, nopacing is done to the heart

Catastrophic

FMEA table for the Pacemaker components

West Virginia

UniversityFMEA table for the Pacemaker connectors

Connector Name Failure Mode Cause of Failure Effect of Failure Criticality of effectsRS-CG Failure to enable

communication of theCG

Magnet malfunctioning.RS failed to generatemessage.

Pacemaker is not programmed,schedule maintenance task

Minor

RS-CD Unable to disablecommunication of theCD with theprogrammer

Magnet malfunctioning.RS failed to generatecorrect disable message.

Pacemaker receive bits accidentallyfrom hazards but device is neverprogrammed because CG is disabled,schedule maintenance task.

Minor

CD-Programmer Failed to acknowledgeprogramming

Fault in coding thesending message

Pacemaker is not programmed,schedule maintenance task.

Minor

CD-CG Failed to send bytes ofprogram data to CG

Inappropriate count ofnumber of bits in a byte.

Pacemaker is not programmed,schedule maintenance task.

Minor

CG-AR Send incorrectcommand (ex ToOffinstead of ToIdle)

Incorrect interpretationof program bytes

Incorrect operation mode andincorrect rate of pacing the heart.Device is still monitored by thephysician, immediate maintenance ordisable is required.

Marginal

CG-VT Send incorrectcommand (ex ToOffinstead of ToIdle

Incorrect interpretationof program bytes

Incorrect operation mode andincorrect rate of pacing the heart.Device is still monitored by thephysician, immediate maintenance ordisable is required.

Marginal

AR-Heart Failed to sense heart inAAI mode

Sensor error. Heart is always paced while patientcondition requires only pacing theheart when no pulse is detected

Critical

Failed to pace the heartin AVI mode

Pacing hardware devicemalfunctioning

Heart could be in serious problembecause of no pacing.

Catastrophic

VT-AR VT failed to informAR of finishingrefractoring in AVImode

Timing mismatchesbetween AR and VToperation.

Failure to pace the heart. Catastrophic

West Virginia

University4) Develop Risk Factorshrfi = cpxi x svrtyi

where:

0 <= cpxi <= 1, is the normalized complexity level (dynamic complexity for components or dynamic coupling for connectors), and

0<= svrtyi < 1 , is the severity level for the architecture element.

RS CD CG AR VTDynamicComplexity

0.002 0.013 0.005 1 0.963

Severity 0.25 0.25 0.5 0.95 0.95Risk Factors 0.0005 0.00325 0.0025 0.95 0.91485

Risk Factors for the components in the example

West Virginia

University4) Develop Risk Factors (cont’d)

Comparison between risk factors based on static and dynamic metrics

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RS CD CG AR VT

Ris

k F

acto

rs

Dynamic

CBO

NAS

Connector Risk Factors RS CD CG AR VT Programmer HeartRS 0.00035 0.00035CD 0.00075 0.00275CG 0.0005 0.0007 0.0007AR 0.2375 0.95VT 0.2565 0.82935Programmer 0.00035 .0015Heart 0.11685 0.29165

Risk Factors for the connectors in the pacemaker example

West Virginia

University5) Constructing the CDG

<Prog., 0,5>

<RS,5x10-4,5>

<CD, 3x10-3,5>

<AR,0.95,40>

<VT,0.9,40>

<Heart,0,5><CG, 2.5x10-2,5>

s

t

t

t

<, 0, .01>

<, 0, .64>

<, 0, .35>

<, 0, .01><, 0, .99>

<, 0, .99>

<, 0, .99>

<, 0, .99>

<, 0, .34><, 0, .36>

<,3.5x10-4, .002>

<,1.5x10-3,.008>

<,2.7x10-3,.008>

<,7.5x10-4,.002>

<,3.5x10-4,.005>

<,3.5x10-3,.005><,7x10-4,.0025>

<,5x10-4,.005>

<,7x10-4,.0025>

<,.12,.35><,.29,.64>

<,.26,.29>

<,.95,.47>

<,.24,.19>

<,.26,.29>

West Virginia

University6) Risk Aggregation Algorithm

The algorithm expands all branches of the CDG starting from the start node.

The breadth expansions of the graph represent logical "OR" paths.

• translated as the summation of aggregated risk factors weighted by the transition probability along each path.

The depth of each path represents the sequential execution of components:

• is given by the aggregate: HRF = 1 - i(1-hrfi)

West Virginia

UniversityRisk Aggregation AlgorithmProcedure AssessRiskParameters

consumes CDG, AEappl,(average execution time for the application)produces Riskappl

Initialization:Rappl = Rtemp = 1 (temporary variables for (1-RiskFactor) )Time = 0Algorithmpush tuple <C1, hrf1, EC1 >, Time, Rtemp

while Stack not EMPTY dopop < Ci, hrfi , ECi >, Time, Rtemp

if Time > AEappl or Ci = t; (terminating node)Rappl += Rtemp ;(an OR path)

else < Cj ,hrfj , ECj > children(Ci)

push (<Cj, hrfj ,ECj>, Time += ECi , Rtemp = Rtemp*(1-hrfi)*(1-hrfij )*PTij ) ( AND path)

endend while

Riskappl = 1- Rappl

end Procedure AssessRisk

West Virginia

UniversityRisk Aggregation Algorithm

The algorithm can be used for

• System-level Risk Assessment The risk of the pacemaker that is found to be ~ 0.9

• Subsystem-level Risk Comparison Complex systems are composed of many subsystems. The algorithm can be used to obtain a risk factor for a

subsystem using risk factors of its individual components. Compare risk factors of individual subsystems.

• Sensitivity Analysis Sensitivity to Uncertainties in Component Risk Factors Sensitivity to Uncertainties in Connector Risk Factors

West Virginia

UniversitySensitivity Analysis

0.0

0.2

0.4

0.6

0.8

1.0

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Risk Factor of Individual Components

Ove

rall

Ris

k F

acto

r o

f th

e S

yste

m

R(AR)

R(VT)

R(CG)

R(CD)

R(RS)

0.0

0.2

0.4

0.6

0.8

1.0

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Risk Factor of Individual Connectors

Ove

rall

Sys

tem

Ris

k V

alu

e

R(RS-CD)

R(CG-CD)

R(AR-Heart)

R(VT-AR)

R(VT-Heart)

The Pacemaker risk factor as function of connector risk factors (one at a time)

The Pacemaker risk factor as function of component risk factors (one at a time)

West Virginia

UniversityBenefits

The approach helps in:

• Deciding which components in the architecture require more development resources.

• Deciding which connectors in the architecture are of highest risk. A high risk connector indicates that the interfaces between the corresponding components and the messaging protocol should be carefully designed.

• Studying how uncertainties in component risk factors affect the overall risk value of the system.

• Studying how uncertainties in connector risk factors affect the overall risk value of the system.

West Virginia

UniversityConclusion : Benefits The methodology is applicable early at the

architectural level.

The methodology is based on dynamic metrics. We use dynamic metrics to account for the fact that a fault in a frequently executed component will frequently manifest itself into a failure.

The methodology is based on simulation of architecture models. Simulation helps in:

• Performing FMEA procedures .

• Calculating the CDG parameters such as probability of transitions.

• Obtaining dynamic metrics.

West Virginia

UniversityConclusion : Issues

Using ordinal scale for measuring severity.

Effect of uncertainties in the scenario probabilities and the estimated average execution times.

Scalability issues, applying the methodology to a larger case study.

Methodology is limited to systems with statechart and sequence diagram specifications.

Questions...

West Virginia

UniversityMain Use Case Diagram

Programming Mode

Operational Modes

DoctorsProgramer

Operating_in_AVIOperating_in_ AAT

Operating_in_ VVIOperating_in_ VVT

PatientsHeart

1

1

1 1

1

1

1

1

1

111

1

Operating_in_ AAI

1

1

1

1

Programming

1

1

1

1

«extend»«extend»«extend»«extend»

«extend»«extend»«extend»«extend»

«extend»«extend»