Motivation Background Adaptive Clock Sync. Evaluation Summary 1 of 25 slides

Adaptive Internal Clock SynchronizationSRDS 2008

Zbigniew Jerzak, Robert Fach, and Christof Fetzer

October 6, 2008

{Zbigniew.Jerzak, Robert.Fach, Christof.Fetzer}@inf.tu-dresden.de

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 2 of 25 slides

Introduction

Build safer distributed systems using weaker assumptions [FC03]

In this paper/presentation:

I Relaxed specification for internal clock synchronization

I New clock synchronization algorithm

I New way to deal with crash failures

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides

Current Approach

1. Assume a bounded deviation ∆max

2. However, in timed asynchronous applications:I messages get dropped or are arbitrarily delayedI different nominal and observed oscillator frequenciesI clocks’ speeds fluctuate with timeI processes crash

3. ...thus 1 cannot be guaranteed

Result:

An inherently inflexible and a failure-prone system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides

Current Approach

1. Assume a bounded deviation ∆max

2. However, in timed asynchronous applications:I messages get dropped or are arbitrarily delayedI different nominal and observed oscillator frequenciesI clocks’ speeds fluctuate with timeI processes crash

3. ...thus 1 cannot be guaranteed

Result:

An inherently inflexible and a failure-prone system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides

Current Approach

1. Assume a bounded deviation ∆max

2. However, in timed asynchronous applications:I messages get dropped or are arbitrarily delayedI different nominal and observed oscillator frequenciesI clocks’ speeds fluctuate with timeI processes crash

3. ...thus 1 cannot be guaranteed

Result:

An inherently inflexible and a failure-prone system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 3 of 25 slides

Current Approach

1. Assume a bounded deviation ∆max

2. However, in timed asynchronous applications:I messages get dropped or are arbitrarily delayedI different nominal and observed oscillator frequenciesI clocks’ speeds fluctuate with timeI processes crash

3. ...thus 1 cannot be guaranteed

Result:

An inherently inflexible and a failure-prone system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 4 of 25 slides

Our Approach

I No upper bound for the deviation (∆max)I compute deviation between processes in real-timeI use computed deviation to adapt system behavior

I No bound on the maximum number of crashed processesI mask crashes using clock readings extrapolation

Result:

An adaptive, safer system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 4 of 25 slides

Our Approach

I No upper bound for the deviation (∆max)I compute deviation between processes in real-timeI use computed deviation to adapt system behavior

I No bound on the maximum number of crashed processesI mask crashes using clock readings extrapolation

Result:

An adaptive, safer system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 4 of 25 slides

Our Approach

I No upper bound for the deviation (∆max)I compute deviation between processes in real-timeI use computed deviation to adapt system behavior

I No bound on the maximum number of crashed processesI mask crashes using clock readings extrapolation

Result:

An adaptive, safer system

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 5 of 25 slides

Use Case

I p is assigned a time slot [S ,T ] for access to R

I [S ,T ] defined w.r.t. some abstract clock

I p is not synchronized with a priori known ∆max, instead...I Use Ep – maximum local clock synchronization error

I an upper bound on the deviation from some abstract clockI calculated by every processI propagated to application layer for error handling

I p uses R only within [S + Ep,T − Ep]I p knows how well it is synchronizedI p knows when it is allowed to access R

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 5 of 25 slides

Use Case

I p is assigned a time slot [S ,T ] for access to R

I [S ,T ] defined w.r.t. some abstract clock

I p is not synchronized with a priori known ∆max, instead...I Use Ep – maximum local clock synchronization error

I an upper bound on the deviation from some abstract clockI calculated by every processI propagated to application layer for error handling

I p uses R only within [S + Ep,T − Ep]I p knows how well it is synchronizedI p knows when it is allowed to access R

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 5 of 25 slides

Use Case

I p is assigned a time slot [S ,T ] for access to R

I [S ,T ] defined w.r.t. some abstract clock

I p is not synchronized with a priori known ∆max, instead...I Use Ep – maximum local clock synchronization error

I an upper bound on the deviation from some abstract clockI calculated by every processI propagated to application layer for error handling

I p uses R only within [S + Ep,T − Ep]I p knows how well it is synchronizedI p knows when it is allowed to access R

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 6 of 25 slides

Processes and Hardware Clocksbased on TADSM [CF99]

I Set of N = {p0, p1, . . . , pn} processes

I Connected via communication busI Hardware clocks – Hp(t):

I bounded drift:∀p∈N∀t : |ρp(t)| ≤ ρmax

I within linear envelope of real time:(t − s)(1− ρmax) ≤ Hp(t)− Hp(s) ≤ (t − s)(1 + ρmax)

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 8 of 25 slides

Generic Internal Clock Synchronization Algorithm

1 ClockVa l Ap ; // c u r r e n t ad jus tment2 ClockVa l T; // end o f c u r r e n t round

4 vo id i n i t ( ) {5 (Ap ,T) = i n i t i a l A d j u s t m e n t ( ) ;6 // e v e r y P s t a r t i n g at T7 s c h edu l e ( s ynch ron i z e r , P , T) ;8 }

10 vo id s yn ch r on i z e r ( ) {11 // N − number o f p r o c e s s e s12 ClockVa l c l k [N] ,13 ClockVa l e r r [N ] ;14 // remote c l o c k r e a d i n g15 r e adC l o ck s ( c l k , e r r ) ;16 // ad jus tment f o r the next round17 Ap = ad j u s t (Ap , T, c l k , e r r ) ;18 // s e t T to next round19 T = T + P;20 }

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 9 of 25 slides

Software Clocks and Remote Clock Reading

I We do not apply Ap directly to the Hp(t)

I Instead we use software clocks:Sp(t) = Hp(t) + ap(t)

I Ap is calculated based on the remote clock readingsI probabilistic remote clock reading [FC99b]I provides values of the remote clocks clk[]I and accompanying remote clock reading errors err[]

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 10 of 25 slides

Failure Model

I Messages are supposed to be delivered within δI might be delayedI delivered out of orderI get dropped

I Arbitrary hardware clock failures: |ρp(t)| > ρmax

I converted to stop failures [FC99a]

I Processes’ value (byzantine) failuresI handled by Software Encoded Processing [WF07]

Specifically:

No upper boundon the frequency of communication and process failures

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 10 of 25 slides

Failure Model

I Messages are supposed to be delivered within δI might be delayedI delivered out of orderI get dropped

I Arbitrary hardware clock failures: |ρp(t)| > ρmax

I converted to stop failures [FC99a]

I Processes’ value (byzantine) failuresI handled by Software Encoded Processing [WF07]

Specifically:

No upper boundon the frequency of communication and process failures

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 11 of 25 slides

Problem Statement

∀p,q∈N : Sp(t) = Sq(t) (1)

I However, achieving 1 is not possibleI Minimize & bound difference between software clocks:

I ∀p,q∈N:p,q ¬crashed(t)∀t : |Sp(t)− Sq(t)| ≤ Ep(t) + Eq(t)

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 11 of 25 slides

Problem Statement

∀p,q∈N : Sp(t) = Sq(t) (1)

I However, achieving 1 is not possibleI Minimize & bound difference between software clocks:

I ∀p,q∈N:p,q ¬crashed(t)∀t : |Sp(t)− Sq(t)| ≤ Ep(t) + Eq(t)

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 11 of 25 slides

Problem Statement

∀p,q∈N : Sp(t) = Sq(t) (1)

I However, achieving 1 is not possibleI Minimize & bound difference between software clocks:

I ∀p,q∈N:p,q ¬crashed(t)∀t : |Sp(t)− Sq(t)| ≤ Ep(t) + Eq(t)

Abstract Clock pq

Eq Ep

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 12 of 25 slides

Ep – Maximum Clock Synchronization Error

Ep(T ) = |Cp(T , p)− cfn()|+ Ep(T )

I cfn() = mid([

Lip,U

ip

])I determines the Ap

I Ep(T ) = max{

cfn()−←−Mp(T ),

−→Mp(T )− cfn()

}I determines the midpoint estimation error

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 12 of 25 slides

Ep – Maximum Clock Synchronization Error

Ep(T ) = |Cp(T , p)− cfn()|+ Ep(T )

I cfn() = mid([

Lip,U

ip

])I determines the Ap

I Ep(T ) = max{

cfn()−←−Mp(T ),

−→Mp(T )− cfn()

}I determines the midpoint estimation error

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 13 of 25 slides

Convergence Functions: cfn() = mid([

Lip, U

ip

])Ep(T ) = |Cp(T , p) − cfn()| + Ep(T )

I A convergence function:I determines an abstract clock value which...I ...should be reached at the end of the next round

I Specifically:I cfnV 0() — based on [WL88]I cfnV 1() — based on [CF94]

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 13 of 25 slides

Convergence Functions: cfn() = mid([

Lip, U

ip

])Ep(T ) = |Cp(T , p) − cfn()| + Ep(T )

I A convergence function:I determines an abstract clock value which...I ...should be reached at the end of the next round

I Specifically:I cfnV 0() — based on [WL88]I cfnV 1() — based on [CF94]

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 14 of 25 slides

Convergence Functions: cfn() = mid([

Lip, U

ip

])Ep(T ) = |Cp(T , p) − cfn()| + Ep(T )

I Perfect World (no transmission delays, no drift):

p Cq(T,p) Cr(T,p)

cfn()

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 14 of 25 slides

Convergence Functions: cfn() = mid([

Lip, U

ip

])Ep(T ) = |Cp(T , p) − cfn()| + Ep(T )

I Real Life (transmission delays, drift):

p Cq(T,p) Cr(T,p)

εr(T,p)εq(T,p)

cfn()

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 15 of 25 slides

Midpoint Estimation Error:−→M p(T ),

←−M p(T )

Ep(T ) = |Cp(T , p) − cfn()| + Ep(T )

I Possible values a midpoint can take:

←−Mp(T ) = mid

([←−L p(T ),

←−U p(T )

])−→Mp(T ) = mid

([−→L p(T ),

−→U p(T )

])cfn()

Mp()←

Mp()→

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 16 of 25 slides

Extrapolation & Intersection

I Re-use clock readings from previous round

I Mask transient clock reading failures (extrapolation)

I Improve readings with large error (intersection)I Extrapolation:

I in round i use the reading from round i − 1I shift clock value by P, extend error by (k + 2)Pρmax

I Intersection:I intersect readings from round i with extrapolation from i − 1I both correct → result also correct

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 16 of 25 slides

Extrapolation & Intersection

I Re-use clock readings from previous round

I Mask transient clock reading failures (extrapolation)

I Improve readings with large error (intersection)I Extrapolation:

I in round i use the reading from round i − 1I shift clock value by P, extend error by (k + 2)Pρmax

I Intersection:I intersect readings from round i with extrapolation from i − 1I both correct → result also correct

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 16 of 25 slides

Extrapolation & Intersection

I Re-use clock readings from previous round

I Mask transient clock reading failures (extrapolation)

I Improve readings with large error (intersection)I Extrapolation:

I in round i use the reading from round i − 1I shift clock value by P, extend error by (k + 2)Pρmax

I Intersection:I intersect readings from round i with extrapolation from i − 1I both correct → result also correct

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 17 of 25 slides

Evaluation Environment

I OMNeT++ discrete event based simulatorI global observer view only possible in simulation

I TDMA

I 10 Mbit, half-duplex, shared Ethernet

I 2 · 105 km/s signal propagation speed

I 4 hosts:

A B C D10 [m] 10 [m] 10 [m]

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 18 of 25 slides

Imprecision – cfnV 0() & cfnV 1()

0

10

20

30

40

50

60

25 30 35 40 45 50 55 60

imp

reci

sio

n [

µs]

real-time [s]

V0 - 0%V0 - 1%

V0 - 25%V1 - 0%

V1 - 1%V1 - 25%

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 19 of 25 slides

Intersection Gain

0

0.1

0.2

0.3

0.4

0.5

0.6

5 10 15 20 25 30

mea

n i

mp

rov

emen

t [%

]

real-time [s]

0% -host A0% - host D1% - host A

1% - host D25% - host A25% - host D

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 19 of 25 slides

Intersection Gain

0

1

2

3

4

5

6

7

8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 9

impr

ovem

ent [

%]

real-time [s]

0% -host A0% - host D1% - host A

1% - host D25% - host A25% - host D

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 20 of 25 slides

Extrapolation and Omissions

0.001

0.01

0.1

1

10

100

1000

10000

21.6 21.8 22 22.2 22.4

erro

r [µ

s]

real-time [s]

infinity

max. sync. errorreal sync. error

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 20 of 25 slides

Extrapolation and Omissions

0.001

0.01

0.1

1

10

100

1000

10000

21.6 21.8 22 22.2 22.4

erro

r [µ

s]

real-time [s]

no infinity(extrapolation)

max. sync. errorreal sync. error

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 21 of 25 slides

Asymmetric Permanent Omissions

0

20

40

60

80

100

120

4 5 6 7 8 9 10

max

. sy

nc.

err

or

[µs]

real-time [s]

host Ahost Bhost Chost D

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 21 of 25 slides

Asymmetric Permanent Omissions

0

20

40

60

80

100

120

4 5 6 7 8 9 10

real

/max

sy

nc

err

[µs]

real-time [s]

host A - max sync errhost B - max sync errhost C - max sync errhost D - max sync errhost A - real sync errhost B - real sync errhost C - real sync errhost D - real sync err

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 21 of 25 slides

Asymmetric Permanent Omissions

0

20

40

60

80

100

120

4 5 6 7 8 9 10

real

/max

sync

err

[µs]

real-time [s]

host C - max sync errhost D - max sync errhost C - real sync errhost D - real sync err

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 22 of 25 slides

Crash Failures

0

20

40

60

80

100

120

34 36 38 40 42 44 46 48 50

erro

r [µ

s]

real-time [s]

host A - max. sync. errorhost A - real sync. error

host B - max. sync. errorhost B - real sync. error

host D - max. sync. errorhost D - real sync. error

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 23 of 25 slides

Summary

I Need to build critical systems, however:I underlying components are not synchronousI need to use COTS to cut costI need to use wireless to cut weight

I We propose Adaptive Internal Clock SynchronizationI copes with message delays and process failuresI bounds deviation Ep(T ) from correct clocks in the systemI propagates the deviation Ep(T ) to upper level applications

I Applications use the Ep(T ) to adapt their behaviorI no more synchronization layer crashesI the system is safer

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 23 of 25 slides

Summary

I Need to build critical systems, however:I underlying components are not synchronousI need to use COTS to cut costI need to use wireless to cut weight

I We propose Adaptive Internal Clock SynchronizationI copes with message delays and process failuresI bounds deviation Ep(T ) from correct clocks in the systemI propagates the deviation Ep(T ) to upper level applications

I Applications use the Ep(T ) to adapt their behaviorI no more synchronization layer crashesI the system is safer

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 23 of 25 slides

Summary

I Need to build critical systems, however:I underlying components are not synchronousI need to use COTS to cut costI need to use wireless to cut weight

I We propose Adaptive Internal Clock SynchronizationI copes with message delays and process failuresI bounds deviation Ep(T ) from correct clocks in the systemI propagates the deviation Ep(T ) to upper level applications

I Applications use the Ep(T ) to adapt their behaviorI no more synchronization layer crashesI the system is safer

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 24 of 25 slides

Thank You!http://wwwse.inf.tu-dresden.de/

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer

Motivation Background Adaptive Clock Sync. Evaluation Summary 25 of 25 slides

References

Flaviu Cristian and Christof Fetzer.

Probabilistic internal clock synchronization.In Proceedings of the Thirteenth Symposium on Reliable Distributed Systems (SRDS1994), pages 22–31,October 1994.

Flaviu Cristian and Christof Fetzer.

The timed asynchronous distributed system model.IEEE Trans. on Parallel and Distr. Systems, 10(6):642–657, June 1999.

Christof Fetzer and Flaviu Cristian.

Building fault-tolerant hardware clocks.In Proceedings of the Seventh IFIP Int. Working Conf. on Dependable Computing for Critical Applications,pages 59–78, San Jose, USA, Jan 1999.

Christof Fetzer and Flaviu Cristian.

A fail-aware datagram service.In Iain Bate and Alan Burns, editors, IEE Proceedings - Softw. Eng., volume 146, pages 58–74. IEE, April1999.

Christof Fetzer and Flaviu Cristian.

Fail-awareness: An approach to construct fail-safe systems.Journal of Real-Time Systems, 24(2):203–238, March 2003.

Ute Wappler and Christof Fetzer.

Hardware failure virtualization via software encoded processing.In 5th IEEE Int. Conf. on Industrial Inf. (INDIN 2007), volume 2, pages 977–982, June 2007.

Jennifer Lundelius Welch and Nancy Lynch.

A new fault-tolerant algorithm for clock synchronization.Information and Computing, 77(1):1–36, 1988.

Adaptive Internal Clock Synchronization Zbigniew Jerzak, Robert Fach, and Christof Fetzer