Clock Synchronization in Distributed Systems

Introduction Basic Concepts Algorithms NTP Summary 1 of 45 slides

Clock Synchronization in Distributed SystemsWissenschaftlicher Vortrag

Zbigniew Jerzak

Technisches Universitat Dresden, Fakultat Informatik

Monday 28th September, 2009

Clock Synchronization in Distributed Systems Zbigniew Jerzak

Motivation

Outline

IntroductionWhat is clock synchronization?The challenges of clock synchronization

Basic ConceptsSoftware and hardware clocksBasic clock synchronization algorithm

AlgorithmsDeep dive into landmark papers

NTPInternet scale time synchronization

Summary

Problem Definitionbased on: [LMS85]

1. At any time, the values of all the nonfaulty clocks must beapproximately equal (within ∆max).

2. There is a small bound on the amount by which a nonfaultyprocesss clock is changed.

System Definition

I A set of N distributed processes

I Every process has a local physical clock

I No direct access to a shared global clock

I Communication between processes is message-based

Clock Synchronization Application Areasbased on: [Lis93]

I At most once message delivery [LSW90]

I Cache consistency [GC89]

I Active replication [HCZ08]

I Medium access control [KG94]

I Global Positioning System

I Global System for Mobile communications (second generation)

Clock Synchronization Application Areasbased on: [Lis93]

I At most once message delivery [LSW90]

I Cache consistency [GC89]

I Active replication [HCZ08]

I Medium access control [KG94]

I Global Positioning System

I Global System for Mobile communications (second generation)

Time Division Multiple Access

I Requirement: real-timecommunication using a sharedmedium

I Problem: collisions can arbitrarilydelay messages

I Solution: synchronize clocks todetermine access slots [Joc07]

I frame based data flowI divide frames into slotsI scheduler assigns processes to

slotsI clock synchronization: collision

free schedule execution

Problem: Unstable Clocks

5 6 7 8 9 10 11

time [h]

(in the box)(out of the box)

offset [us]

Problem: Unstable Clocks

12 12.5 13 13.5 14 21

time [h]

clock offset [us]temperature [C]

Problem: Varying Delays

0 20000 40000 60000 80000 100000

round trip time [µs]

LAN: [se09 - sedell06].inf.tu-dresden.deMAN: sews11.inf.tu-dresden.de - mindfab.net

MAN: itias.homeip.net - rg4.polsl.plWAN: sedell06.inf.tu-dresden.de - rg4.polsl.pl

Problem: Omissions and Crashes

I The probability of failureincreases with the increasingnumber of system elements

I Byzantine [LSP82] errors

A system consisting of 280 nodespartitions on average once aday [MPHD06]

Problem: Omissions and Crashes

I The probability of failureincreases with the increasingnumber of system elements

I Byzantine [LSP82] errors

,,Two-faced clocks” presentdifferent values to differentprocesses [LMS85]

Clocks

I The timekeeping element –an oscillator:

I pendulumI quartz crystalI microwave (133Cs)

I In computer science

Clocks

I The timekeeping element –an oscillator:

I pendulumI quartz crystalI microwave (133Cs)

I In computer science

Hardware Clocks

I Hardware clock: H(t)

I Rate: f (t) =dH(t)

I Drift: ρ(t) = f (t)− 1

Clock Drift of Different PlanetLab Hosts

uba.arssvl.kth.seiit-tech.net

Correctness of the Hardware Clock

I |ρ(t)| ≤ ρmax

I ρmax ≤ 500 for HPET [Cor04]

I H(t)− H(s) ≥ (t − s)(1− ρmax)H(t)− H(s) ≤ (t − s)(1 + ρmax)

I linear envelope of real-time

Correctness of the Hardware Clock

I |ρ(t)| ≤ ρmax

I ρmax ≤ 500 for HPET [Cor04]

I H(t)− H(s) ≥ (t − s)(1− ρmax)H(t)− H(s) ≤ (t − s)(1 + ρmax)

I linear envelope of real-time

Software Clocks

I Hardware clocks in general are notsynchronized:

I Hp(t)− Hq(t) is not bounded

I Software clocks are used insteadI Sp(t) = Hp(t) + ap(t)

A process will have a physical clock that ,,ticks” continually and alogical clock whose value equals the value of the physical clockplus some offset [LMS85].

Software Clocks

I Hardware clocks in general are notsynchronized:

I Hp(t)− Hq(t) is not bounded

I Software clocks are used insteadI Sp(t) = Hp(t) + ap(t)

A process will have a physical clock that ,,ticks” continually and alogical clock whose value equals the value of the physical clockplus some offset [LMS85].

ap(t): Continuous and Discrete Software Clocks

Main Components of Clock Synchronization Algorithm

Basic Clock Synchronization Algorithm

1 ClockVa lue Ap ; // c u r r e n t ad jus tment2 ClockVa lue T; // end o f c u r r e n t round3 ClockVa lue P ; // re−sync p e r i o d4

5 vo id i n i t ( ) {6 Ap ,T = i n i t i a l A d j ( ) ;7 s c h edu l e ( synchronizat ionRound , P , T) ;8 }9

10 vo id synchronizat ionRound ( ) {11 ClockVa lue c l k [ |N| ] ; // remote c l o c k r e a d i n g s12 ClockVa lue e r r [ |N| ] ; // remote r e a d i n g e r r o r s13

14 r e adC l o ck s ( c l k , e r r ) ;15 Ap = ad j u s t (Ap , T, c l k , e r r ) ;16 T = T + P;17 }

Algorithms Classification

External vs Internal Clock Synchronization

I External [Cri89, Mil91, CF95]:I time reference external to the

systemI maintain ∆max wrt. external time

reference

I Internal [LMS85, WL88, CF95,FL06]:

I maintain ∆max wrt. other systemmembers

Externally synchronized clocks are also internally synchronized.The converse is not true. [Cri89]

reference

Software vs Hardware Clock Synchronization

I Hardware (assisted) clocksynchronization [KSB85, SR88, KKMS95]

I Very precise (e.g. phase locking)I Very expensive (additional hardware)

I Software clock synchronization [WL88, Mil91, FL06]I Less preciseI More flexibleI Cheap

Software vs Hardware Clock Synchronization

I Hardware (assisted) clocksynchronization [KSB85, SR88, KKMS95]

I Very precise (e.g. phase locking)I Very expensive (additional hardware)

I Software clock synchronization [WL88, Mil91, FL06]I Less preciseI More flexibleI Cheap

Deterministic vs Probabilistic Clock Synchronization

I Deterministic [WL88, FC95, WS07]:I ∃ ub(td)I ∆max holds

I Probabilistic [Cri89, OS94]:I @ ub(td)I ∆max does not holdI indication when ∆max is reached

Deterministic vs Probabilistic Clock Synchronization

I Deterministic [WL88, FC95, WS07]:I ∃ ub(td)I ∆max holds

I Probabilistic [Cri89, OS94]:I @ ub(td)I ∆max does not holdI indication when ∆max is reached

Clock Synchronization Algorithms

I Fault Tolerant Clock Synchronization (FTCS) [WL88]I SoftwareI InternalI Deterministic

I Probabilistic Clock Synchronization (PCS) [Cri89]I SoftwareI ExternalI Probabilistic

I Gossip-based Synchronization [BPQS08]I SoftwareI Internal

Fault Tolerant Clock Synchronizationbased on: [LL84, WL88]

I td ∈ [δmin, δmax]

I ∀p∈N : |ρp(t)| ≤ ρmax

I Sp(t) = Hp(t) + ap(t),I ap(t) - discrete function of time

I Initial synchronization: ∀p,q∈N : |Sp(0)− Sq(0)| < γ

I |N|2 messages per roundI [CF94]: |N|+ 1 for crash-stop failures

I |N| ≥ 3|F|+ 1

FTCS – Algorithm Outline

1. Broadcast Sp(T i )

2. Wait for other broadcasts for γ + δmax

3. Use convergence function to calculate midpoint4. ai+1

p = aip + midpoint

5. Use ai+1p to ,,switch” to new software clock

6. Wait until T i+1 = T i + P

7. Loop

p = aip + midpoint

7. Loop

p = aip + midpoint

7. Loop

p = aip + midpoint

7. Loop

p = aip + midpoint

7. Loop

p = aip + midpoint

7. Loop

p = aip + midpoint

7. Loop

FTCS – Fault Tolerant Convergence Function

1 ClockVa lue cfn ( c l k [ |N| ] , |F|)2 {3 ClockVa lue midpo in t ;4 ClockVa lue tmp [ |N| ] ;5

6 midpo in t = 0 ;7 tmp [ |N| ] = so r t ( c l k [ |N| ] ) ;8

9 f o r ( i=|F| ; i <2|F|+1; ++i )10 {11 midpo in t = midpo in t + tmp [ i ] ;12 }13 midpo in t = midpo in t / |F|+1;14

15 re tu rn midpo in t ;16 }

Probabilistic Clock Synchronizationbased on: [Cri89]

I p (td ∈ [δmin, δmax]) 6= 1

I Remote clocks cannot be read with a priori specified precision

I Timeout delay, which divides messages into slow and fast

I Processes suffer only timing failures

PCS – Remote Clock Reading I

ub(m2) = (D − A)− (C − B)− δmin(m1)

PCS – Remote Clock Reading II

Cp(T , q) = (T − D) + C +ub(m2) + δmin(m1)

Ep(T , q) =ub(m2)− δmin(m1)

PCS – Adjusting Local Clock

I Recall: Sq(t) = Hq(t) + aq(t)I aq(t) = αHq(t) + βI Sq(t) = Hq(t)(1 + α) + β

I Local time: Sq(T ), remote time: Cp(T , q)I Sq(T ) = Hq(T )(1 + α) + β

I Goal: after P local time shows Cp(T , q) + PI Sq(T + P) = Cp(T , q) + P = (Hq(T ) + P)(1 + α) + β

I Solution:

I α =Cp(T , q)− Sq(T )

PI β = Sq(T )− Hq(T )(1 + α)

I Solution:

PI β = Sq(T )− Hq(T )(1 + α)

I Solution:

PI β = Sq(T )− Hq(T )(1 + α)

I Solution:

PI β = Sq(T )− Hq(T )(1 + α)

PCS – Specifying Precision

I Lower ub(m2) implies lower error Ep(T , q)

I Achieving a given error requires a bound ubmax

I Trade-off between Ep(T , q) and probability p(ub(m) > ubmax)I Using k readings and knowing p:

I p(success) = 1− pk

Gossip-based Synchronizationbased on: [BPQS08]

I Problem: scale to thousands of nodes

I Solution: gossip-based algorithms (partial view)

I Remote clock reading: Cristian approach [Cri89]

I Digital signatures

I Discrete clock adjustment

Gossip-based Synchronization – The Algorithm

1. Obtain a random list of neighbors

2. Use the remote clock reading to calculate offsets O3. Sort the offsets

4. Adjustment:1

U − L

U∑i=L

I L = α|N|I U = |N| − LI 0 ≤ α < 0.5

5. Update local clock

6. Loop

Network Time Protocol – Goal & Definitionsbased on: [Mil91, Mil03]

Goal: accurate and precise time on a statistical basis withacceptable network overheads and instabilities in a large, diverseinternet (interconnected) system. [Mil91]

I Offset: |Hp(t)− Hq(t)|

I Skew:

∣∣∣∣dHp(t)

dt− dHq(t)

∣∣∣∣I Clock Synchronization:

I time synchronization: bounding offsetI frequency synchronization: bounding skew

I Skew:

∣∣∣∣dHp(t)

dt− dHq(t)

I Skew:

∣∣∣∣dHp(t)

dt− dHq(t)

I Skew:

∣∣∣∣dHp(t)

dt− dHq(t)

NTP – Configuration

I Servers ordered into strataI Redundant paths

I tolerate link failuresI SP algorithm

NTP – Reading Remote Clock

I Round trip delay: (D − A)− (C − B)

I Clock offset of q wrt. p: θ = (C+B)2 − (D+A)

I Error: (D−A)−(C−B)2

NTP – Data Filtering

I Problem: accurate offset from a sample populationI Solution: minimum filter

I order m readings according to round trip delayI select the lowest round trip (first) reading

NTP – Data Filtering

I Problem: accurate offset from a sample populationI Solution: minimum filter

I order m readings according to round trip delayI select the lowest round trip (first) reading

NTP – Peer Selection

I Problem: select and combine best peersI Solution: calculate per peer statistics

1. order peers by stratum and round trip delay

2. filter dispersion: χ =i=m−1∑

|θi − θ0| 0.5i

3. peer dispersion: ∀j=|N|−1j=0 : χj =

k=|N|−1∑k=0

∣∣θ0j − θ0

∣∣ 0.75k

4. eliminate the peer with highest dispersion5. terminate if one peer left6. terminate if peer dispersion < minimum filter dispersion

NTP – Peer Selection

I Problem: select and combine best peersI Solution: calculate per peer statistics

1. order peers by stratum and round trip delay

2. filter dispersion: χ =i=m−1∑

|θi − θ0| 0.5i

3. peer dispersion: ∀j=|N|−1j=0 : χj =

k=|N|−1∑k=0

∣∣θ0j − θ0

∣∣ 0.75k

4. eliminate the peer with highest dispersion5. terminate if one peer left6. terminate if peer dispersion < minimum filter dispersion

NTP – Clock Correctionbased on: [Mil92]

I Only one peer: directly apply offset

1 ClockVa lue cfn ( o f f s e t [ |N| ] , s t ra tum [ |N| ] , d i s t a n c e [ |N| ] )2 {3 ClockVa lue tmp1 ;4 ClockVa lue tmp2=0;5 ClockVa lue tmp3=0;6

7 f o r ( i =0; i<|N| ; ++i ) {8 tmp1 = 1/( s t ra tum [ i ]∗MAXDISPERS+d i s t a n c e [ i ] ) ;9 tmp2 += tmp1 ;

10 tmp3 += tmp1∗ o f f s e t [ i ] ;11 }12 re tu rn ( tmp3/tmp2 ) ;13 }

NTP – Clock Correctionbased on: [Mil92]

I Only one peer: directly apply offset

1 ClockVa lue cfn ( o f f s e t [ |N| ] , s t ra tum [ |N| ] , d i s t a n c e [ |N| ] )2 {3 ClockVa lue tmp1 ;4 ClockVa lue tmp2=0;5 ClockVa lue tmp3=0;6

7 f o r ( i =0; i<|N| ; ++i ) {8 tmp1 = 1/( s t ra tum [ i ]∗MAXDISPERS+d i s t a n c e [ i ] ) ;9 tmp2 += tmp1 ;

10 tmp3 += tmp1∗ o f f s e t [ i ] ;11 }12 re tu rn ( tmp3/tmp2 ) ;13 }

Summary

I Clock synchronization is a difficult problemI External clock synchronization has lower overheadsI Internal clock synchronization is more robust

I Clock synchronization is an important problemI For hard-real time applicationsI For wireless networks

I Clock synchronization is practicalI GPSI GSM (2G)

Thank You!

References I

Roberto Baldoni, Marco Platania, Leonardo Querzoni, and Sirio Scipioni.

A peer-to-peer filter-based algorithm for internal clock synchronization in presence of corrupted processes.In PRDC 2008: 14th IEEE Pacific Rim International Symposium on Dependable Computing, pages 64–72.IEEE Computer Society, 2008.

Flaviu Cristian and Christof Fetzer.

Probabilistic internal clock synchronization.In Proceedings of the Thirteenth Symposium on Reliable Distributed Systems (SRDS1994), pages 22–31,October 1994.

F. Cristian and C. Fetzer.

Fault-tolerant external clock synchronization.In ICDCS ’95: Proceedings of the 15th International Conference on Distributed Computing Systems,page 70, Washington, DC, USA, 1995. IEEE Computer Society.

Intel Corporation.

Ia-pc hpet (high precision event timers) specification.Online, October 2004.

Flaviu Cristian.

Probabilistic clock synchronization.Distributed Computing, 3(3):146–158, September 1989.

Christof Fetzer and Flaviu Cristian.

An optimal internal clock synchronization algorithm.In Proceedings of the 10th Annual IEEE Conference on Computer Assurance (COMPASS1995), pages187–196, June 1995.

References II

Rui Fan and Nancy A. Lynch.

Gradient clock synchronization.Distributed Computing, 18(4):255–266, 2006.

Cary G. Gray and David R. Cheriton.

Leases: An efficient fault-tolerant mechanism for distributed file cache consistency.In SOSP 1989: Proceedings of the twelfth ACM Symposium on Operating Systems Principles, pages202–210, 1989.

Jeong-Hyon Hwang, Ugur Cetintemel, and Stan Zdonik.

Fast and highly-available stream processing over wide area networks.In ICDE ’08: Proceedings of the 2008 IEEE 24th International Conference on Data Engineering, pages804–813, Washington, DC, USA, 2008. IEEE Computer Society.

M. Jochim.

Zeitig steuern - sichere datenubertragung im automobil.c’t Magazin fur Computertechnik, 2(1):190–195, January 2007.

Hermann Kopetz and Gunter Grunsteidl.

Ttp-a protocol for fault-tolerant real-time systems.Computer, 27(1):14–23, 1994.

H. Kopetz, A. Kruger, D. Millinger, and A. Schedl.

A synchronization strategy for a time-triggered multi-cluster real-time system.In 14th Symposium on Reliable Distributed Systems, 1995. Proceedings, pages 154–161, Bad Neuenahr,Germany, September 1995.

References III

C. M. Krishna, Kang G. Shin, and Ricky W. Butler.

Ensuring fault tolerance of phase-locked clocks.IEEE Trans. Comput., 34(8):752–756, 1985.

Barbara Liskov.

Practical uses of synchronized clocks in distributed systems.Distributed Computing, 6(4):211–219, 1993.

Jennifer Lundelius and Nancy A. Lynch.

An upper and lower bound for clock synchronization.Information and Control, 62(2/3):190–204, 1984.

Leslie Lamport and P. M. Melliar-Smith.

Synchronizing clocks in the presence of faults.J. ACM, 32(1):52–78, 1985.

Leslie Lamport, Robert Shostak, and Marshall Pease.

The byzantine generals problem.ACM Trans. Program. Lang. Syst., 4(3):382–401, 1982.

B. Liskov, L. Shrira, and J. Wroclawski.

Efficient at-most-once messages based on synchronized clocks.In SIGCOMM ’90: Proceedings of the ACM symposium on Communications architectures & protocols,pages 41–49, New York, NY, USA, 1990. ACM.

David L. Mills.

Internet time synchronization: the network time protocol.IEEE Transactions on Communications, 39(10):1482–1493, October 1991.

References IV

David L. Mills.

Network time protocol (version 3) specification, implementation and analysis, March 1992.

David L. Mills.

A brief history of ntp time: memoirs of an internet timekeeper.SIGCOMM Comput. Commun. Rev., 33(2):9–21, 2003.

Alan Mislove, Ansley Post, Andreas Haeberlen, and Peter Druschely.

Experiences in building and operating a reliable peer-to-peer application.In Yolande Berbers and Willy Zwaenepoel, editors, EuroSys, pages 147–159, Leuven, Belgium, April 2006.ACM.

A. Olson and K.G. Shin.

Probabilistic clock synchronization in large distributed systems.IEEE Transactions on Computers, 43(9):1106–1112, September 1994.

K. G. Shin and P. Ramanathan.

Transmission delays in hardware clock synchronization.IEEE Trans. Comput., 37(11):1465–1467, 1988.

Jennifer Lundelius Welch and Nancy Lynch.

A new fault-tolerant algorithm for clock synchronization.Information and Computing, 77(1):1–36, 1988.

Josef Widder and Ulrich Schmid.

Booting clock synchronization in partially synchronous systems with hybrid process and link failures.Distributed Computing, 20(2):115–140, May 2007.

Clock Synchronization in Distributed Systems

Technology

Transcript of Clock Synchronization in Distributed Systems