Application Level Fault Tolerance and Detection

Application Level Fault Tolerance and Detection

Principal Investigators:C. Mani Krishna Israel Koren

Graduate Students:Diganta Eric Janhavi Osman Vijay

Architecture and Real-Time Systems (ARTS) Lab.Department of Electrical and Computer Engineering

University of Massachusetts Amherst MA 01003


What is ALFTD?

Application Level Fault Tolerance and Detection ALFTD complements existing system or algorithm

level fault tolerance by leveraging information available only at the application level Using such application level semantic information

significantly reduces the overall cost providing fault tolerance

ALFTD may be used alone or to supplement other fault detection schemes

ALFTD is scalable Error overhead can be traded off with invested time

overhead for fault tolerance


ALFTD Overview

Application Level Fault Tolerance and Detection allows for system survival of both data and system (instruction/hardware) faults. System faults cause a process to eventually cease

functioning Data faults cause a process to continue running with

incorrect results ALFTD has been implemented into OTIS to

determine its feasibility as a fault detection and tolerance method for REE applications

OTIS has two sets of related output data, the temperature and emissivity Experiments have focused mostly on the temperature

output


3. Master sends tasks

OTIS Structure

M

S

S

S

2. MPI Starts Slave and master processes

4. Slave Calculations

5. Slave Output to File

OUTPUT

MPI1. MPI Starts


OTIS’ Work Distribution

OTIS’ dynamic workload distribution allows it to compensate for system faults Work originally partitioned for a failed processor is

instead taken by the remaining processes OTIS does not compensate for data faults

As long as the work is completed, there is no measure of correctness

OTIS does not consider deadline repercussions


OTIS Fault Cases


3. Master sends tasks

ALFTD OTIS Structure

4. Slave Calculations

2. MPI Starts Slave and master processes, primary and secondary

M

S2P1

S1P3

S3P2

MPI 1. MPI Starts

OUTPUT

5. Slave Output to File?

?


Secondaries in OTIS

The secondary required for ALFTD is implemented to be functionally similar to the primary

Secondary scaling occurs through resolution reduction OTIS’ “natural” data input exhibits spatial locality Points not directly calculated can be approximately

estimated using interpolation between calculated points

Secondary processes have been tested at 20%-50% of the primary calculation overhead While 50% affords better quality, 20% has less

overhead


Example of Secondary Resolution

(ALFTD Compensation for 10 rows in a sample dataset)

100% Secondary Resolution





ALFTD Benefit


ALFTD Benefit (cont’d)


Fault Detection

When to run the secondary, and when to use the secondary output, is determined by output filters

Output filters are created to check for application-specific trends in data Aberrations from normal data characteristics can be

considered to be the product of potentially faulty processes

OTIS relies on natural temperature characteristics to detect potentially faulty data Spatial Locality: temperature changes gradually over small

areas Absolute Bounds: temperature should not exceed certain

values


Data Sets

Three data sets were chosen for their interesting characteristics

“Blob” “Stripe” “Spots”Broad,

unchanging areas with dark

spots

Relatively undynamic

except for one “stripe”

Turbulent spots may defy

“spatial locality” predictions


Data Frequency (Values)


Data Frequency (Spatial Locality)


Validation Through Secondaries

When the primary deadline is hit, rows are re-delegated to the secondaries if (and only if): The primary has returned results for that row

suspected to be faulty The secondary results can be used to decide whether the

results are indeed faulty A particular row was never successfully calculated

The secondary results can be immediately used in place of the missing primary results


Validation Through Secondaries (cont’d)

After the secondary has been run to verify a primary’s results, the “better” data is chosen according to the following logic grid:

PrimaryFaultles

sAmbiguo

us Faulty

Faultless Primary Secondary

Secondary

Ambiguous Primary Primary Secondar

yFaulty Primary Primary Primary*

Seco

ndar

y


Fault Tolerance Results: “Spots”

Fault Tolerance with injected faults in “Spots”


Fault Tolerance Results: “Spots” (cont’d)

Faulty Output

33% ALFTD Computation Overhead 50% ALFTD Computation Overhead

Fault-Free Output

ALFTD-corrected faulty output25% ALFTD Computation Overhead


Fault Tolerance Results: “Spots” (cont’d)

No ALFTD 25% ALFTD Computation Overhead 33% ALFTD Computation Overhead 50% ALFTD Computation Overhead

No Error Max Error

Difference Plots – faulty output versus faultless output


Fault Tolerance Results: “Blob”

Fault Tolerance with injected faults in “Blob”


Fault Tolerance Results: “Blob” (cont’d)

Faulty Output


Fault-Free Output



Fault Tolerance Results: “Blob” (cont’d)


No Error Max Error



Fault Tolerance Results: “Stripe”

Fault Tolerance with injected faults in “Stripe”


Fault Tolerance Results: “Stripe”(cont’d)

Faulty Output


Fault-Free Output



Fault Tolerance Results: “Stripe”(cont’d)


No Error Max Error



Emissivity Data

Emissivity is loosely proportional to temperature data Emissivity exhibits spatial locality Emissivity has natural bounds of expected data

Natural Metal~0.5

Vegetatation, Water~1.0

<0.5 - Faulty >1.0 - Faulty

Rock~0.8 - ~0.95


Emissivity Data (cont’d)

Emissivity does not exhibit the same data “closeness” as temperature output This makes it very difficult to distinguish faulty from

non-faulty data Luckily, faults present in temperature output are easily

detected, and reflect faults in emissivity output. Emissivity does not have per-pixel

independence of calculation Dependence on the correctness of neighboring pixels

makes resolution reduction a viable, but not the best, method for secondary reduction


Data Frequency (Emissivity Values)


Conclusion

ALFTD has already shown to be a worthwhile alternative to full redundancy

Improvements on the scheme will increase fault coverage and decrease secondary calculation overhead in both the emissivity and temperature outputs

OTIS, as a general matrix-based, master/slave program is a springboard to other, similar programs (e.g., NGST)

ALFTD as a fault-detection scheme will continue to be effective in programs which exhibit “natural” output


Thank You!


Relative Error Calculation

Error in OTIS output is calculated relative to a faultless “template”

The average relative error is the average of all relative errors of the entire output Faulty value = f(x,y) Faultless value = F(x,y)

Error =),(

)],(),([yxF

yxFyxfabs

Application Level Fault Tolerance and Detection

Documents

Transcript of Application Level Fault Tolerance and Detection