Consensus andReliable Broadcast

BroadcastIf a process sends a message , then every process eventually delivers m

m

BroadcastIf a process sends a message , then every process eventually delivers

p0

p1

p2

p3

m

m

BroadcastIf a process sends a message , then every process eventually delivers

How can we adapt the spec for an environment where processes can fail? And what does “fail” mean?

p0

p1

p2

p3

m

m

A hierarchy of failure models

Crash

A hierarchy of failure models

CrashFail-stop

A hierarchy of failure models

Crash

Send Omission Receive Omission

Fail-stop

A hierarchy of failure models

Crash

Send Omission

General Omission

Receive Omission

Fail-stop

A hierarchy of failure models

Crash

Send Omission

General Omission

Receive Omission

benign failures

Fail-stop

A hierarchy of failure models

Crash

Arbitrary failures withmessage authentication

Send Omission

General Omission

Receive Omission

benign failures

Fail-stop

A hierarchy of failure models

Crash

Arbitrary failures withmessage authentication

Arbitrary (Byzantine) failures

Send Omission

General Omission

Receive Omission

benign failures

Fail-stop

Reliable Broadcast Validity! ! If the sender is correct and broadcasts a ! ! message , then all correct processes ! ! eventually deliver Agreement!! If a correct process delivers a message ,! ! then all correct processes eventually ! ! deliverIntegrity! ! Every correct process delivers at most one ! ! message, and if it delivers , then some ! ! process must have broadcast

m

m

m

m

m

m

TerminatingReliable Broadcast

Validity! ! If the sender is correct and broadcasts a ! ! message , then all correct processes ! ! eventually deliver Agreement!! If a correct process delivers a message ,! ! then all correct processes eventually ! ! deliverIntegrity! ! Every correct process delivers at most one ! ! message, and if it delivers ≠ SF, then ! ! some process must have broadcast Termination !Every correct process eventually delivers ! ! some message

m

m

m

m

m

m

Consensus Validity! ! If all processes that propose a value ! ! propose , then all correct processes ! ! eventually decide Agreement!! If a correct process decides , then all ! ! correct processes eventually !decide Integrity! ! Every correct process decides at most one ! ! value, and if it decides , then some ! ! process must have proposed Termination !Every correct process eventually decides ! ! some value

v

v

v

v

v

v

Properties of send(m) and receive(m)Benign failures:

Validity If sends to , and , , and the link between them are correct, then eventually receives

Uniform* Integrity For any message , receives at most once from , and only if sent to

* A property is uniform if it applies to both correct and faulty processes

m

m

m

m

m

p p

q

qq

q

q

pp

Properties of send( ) and receive( )

Arbitrary failures:

Integrity For any message , if and are correct then receives at most once from , and only if sent to

m qp

q

q m

mpp

mm

Questions, Questions…Are these problems solvable at all?Can they be solved independent of the failure model?Does solvability depend on the ratio between faulty and correct processes?Does solvability depend on assumptions about the reliability of the network?Are the problems solvable in both synchronous and asynchronous systems?If a solution exists, how expensive is it?

PlanSynchronous Systems

Consensus for synchronous systems with crash failuresLower bound on the number of roundsReliable Broadcast for arbitrary failures with message authenticationLower bound on the ratio of faulty processes for Consensus with arbitrary failuresReliable Broadcast for arbitrary failures

Asynchronous SystemsImpossibility of Consensus for crash failuresFailure detectorsPAXOS

Model

Synchronous Message PassingExecution is a sequence of roundsIn each round every process takes a step

sends messages to neighborsreceives messages sent in that roundchanges its state

Network is fully connected (an -clique)

No communication failures

n

A simple Consensus algorithm

Initially To execute propose( )1:!! send { } to all decide( ) occurs as follows:2: ! for all do3:!!! receive from 4:!!! := 5:!! decide min( )

Process :pi

V = {vi}

pj

vi

vi

x

j, 0!j!n"1, j #= i

Sj

V ! SjV

V

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

v2

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

v2

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

v2

v1 v1

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

v2

v1 v1

v4

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

An execution

p1 p2 p3 p4

p1 p2 p3 p4

v1

v2

v3

v4

v1

v4

v2

v1 v1

v4

v3v3

Suppose at the end of round 1Can decide?

v1 = v3 = v4

p3

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Suppose hasn’t heard from at the end of round 2. Can decide?

p3 p2

p3

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Suppose hasn’t heard from at the end of round 2. Can decide?

p3 p2

p3

p1 p2 p3 p4

p1 p2 p3 p4

p1 p2 p3 p4

round 1

round 2

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Suppose hasn’t heard from at the end of round 2. Can decide?

p3 p2

p3

p1 p2 p3 p4

p1 p2 p3 p4

p1 p2 p3 p4

round 1

round 2

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Suppose hasn’t heard from at the end of round 2. Can decide?

p3 p2

p3

p1 p2 p3 p4

p1 p2 p3 p4

p1 p2 p3 p4

round 1

round 2

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Suppose hasn’t heard from at the end of round 2. Can decide?

p3 p2

p3

p1 p2 p3 p4

p1 p2 p3 p4

p1 p2 p3 p4

round 1

round 2

Echoing values

A process that receives a proposal in round 1, relays it to others during round 2.

Suppose hasn’t heard from at the end of round 2. Can decide?

p3 p2

p3

p1 p2 p3 p4

p1 p2 p3 p4

p1 p2 p3 p4

round 1

round 2

What is going on

A correct process has not received all proposals by the end of round . Can decide?

Another process may have received the missing proposal at the end of round and be ready to relay it in round

p!

p!

i

i + 1

i

Dangerous Chains

Dangerous chain The last process in the chain is correct, all others are faulty

round 1

round 2

rounds

round

p!

p!

p!

p!

p0

p1

p2

pi!1

pi

3...i ! 1

i

Living dangerously

How many rounds can a dangerous chain span?

faulty processes

at most nodes in the chain

spans at most rounds

It is safe to decide by the end of round !

f

f+1

f

f+1

The Algorithm

Initially To execute propose( )! round 1:!!send { has not already sent } to all 2:!!for all do3:!!! receive from 4:!!! := decide( ) occurs as follows:5: if then6: decide min( )

Code for process :pi

k=f+1

j, 0!j!n"1, j #= i

k, 1!k!f+1

V ={vi}

v!V : pi v

V

V ! Sj

Sj pj

vi

x

V

Termination and Integrity

Termination

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

Termination and Integrity

TerminationEvery correct process

reaches round f + 1Decides on min(V) --- which is well defined

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

Termination and Integrity

TerminationEvery correct process

reaches round f + 1Decides on min(V) --- which is well defined

IntegrityAt most one value:

Only if it was proposed:

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

Termination and Integrity

TerminationEvery correct process

reaches round f + 1Decides on min(V) --- which is well defined

IntegrityAt most one value: – one decide, and min(V) is unique

Only if it was proposed:

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

Termination and Integrity

TerminationEvery correct process

reaches round f + 1Decides on min(V) --- which is well defined

IntegrityAt most one value: – one decide, and min(V) is unique

Only if it was proposed:

– To be decided upon, must be in V at round f+1 – if value = vi, then it is proposed in round 1 – else, suppose received in round k. By induction: – k = 1: • by Uniform Integrity of underlying send and receive, it must have been sent in round 1 • by the protocol and because only crash failures, it must have been proposed – Induction Hypothesis: all values received up to round k = j have been proposed – k = j+1 • sent in round j+1 (Uniform Integrity of send and synchronous model) • must have been part of V of sender at end of round j • by protocol, must have been received by sender by end of round j • by induction hypothesis, must have been proposed

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

ValidityInitially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

ValiditySuppose every process proposes

Since only crash model, only can be sent

By Uniform Integrity of send and receive, only can be received

By protocol, = { }

min( ) =

decide( )

v!

v!

v!

v!

v!

v!

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

V

V

AgreementLemma 1 For any , if a process receives a value in round , then there exists a sequence of processes ! ! such that , is ! .’s proponent, and in each round ! sends and receives it. Furthermore, all processes in the sequence are distinct.

Proof By induction on the length of the sequence

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

r!1 p

v r

p0, p1, . . . , pr pr =p p0

v

pk!1 pkv

Agreement

Lemma 2: !In every execution, at the end of round , ! ! for every correct processes and

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

f+1

Vi =Vj pi pj

Agreement

Lemma 2: !In every execution, at the end of round , ! ! for every correct processes and

Agreement follows from Lemma 2, since min is a deterministic function

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

f+1

Vi =Vj pi pj

Agreement

Lemma 2: !In every execution, at the end of round , ! ! for every correct processes and

Proof:• Show that if a correct has in its at ! the end of round , then every correct ! has in its at the end of round

Agreement follows from Lemma 2, since min is a deterministic function

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

p x V

f+1 p

x V f+1

f+1

Vi =Vj pi pj

Agreement

Lemma 2: !In every execution, at the end of round , ! ! for every correct processes and

Proof:• Show that if a correct has in its at ! the end of round , then every correct ! has in its at the end of round • Let be earliest round is added to the ! of a correct . Let that process be

• If , then sends in round ; ! every correct process receives and adds ! to its in round

Agreement follows from Lemma 2, since min is a deterministic function

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

p x V

f+1 p

x V f+1

r x V

p p!

r!f p!

x r+1!f+1

x x

V r+1

f+1

Vi =Vj pi pj

Agreement

Lemma 2: !In every execution, at the end of round , ! ! for every correct processes and

Proof:• Show that if a correct has in its at ! the end of round , then every correct ! has in its at the end of round • Let be earliest round is added to the ! of a correct . Let that process be

• If , then sends in round ; ! every correct process receives and adds ! to its in round • What if ?

Agreement follows from Lemma 2, since min is a deterministic function

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

p x V

f+1 p

x V f+1

r

r=f+1

x V

p p!

r!f p!

x r+1!f+1

x x

V r+1

f+1

Vi =Vj pi pj

Agreement

Lemma 2: !In every execution, at the end of round , ! ! for every correct processes and

Proof:• Show that if a correct has in its at ! the end of round , then every correct ! has in its at the end of round • Let be earliest round is added to the ! of a correct . Let that process be

• If , then sends in round ; ! every correct process receives and adds ! to its in round • What if ?• By Lemma 1, there exists a sequence of ! distinct processes• Consider processes • processes; only faulty• one of is correct, and adds to ! its before does it in round CONTRADICTION!Agreement follows from Lemma 2, since

min is a deterministic function

p0, . . . , pf

p0, . . . , pf

Initially

To execute propose( )! round !1:! ! send { has not already sent } to all 2:! ! for all do3:! ! ! receive from 4:! ! ! :=

decide(x) occurs as follows:5:! if then6:! ! decide min( )

V ={vi}

vi

Sj pj

V ! SjV

k=f+1

V

k, 1!k!f+1

j, 0!j!n"1, j #= i

v!V : pi v

p x V

f+1 p

x V f+1

r

r=f+1

x V

p p!

r!f p!

x r+1!f+1

x x

V r+1

p0, . . . , pf+1 = p!

f+1 f

p!

r

x

V

f+1

Vi =Vj pi pj

TerminatingReliable Broadcast

Validity! ! If the sender is correct and broadcasts a ! ! message , then all correct processes ! ! eventually deliver Agreement!! If a correct process delivers a message ,! ! then all correct processes eventually ! ! deliverIntegrity! ! Every correct process delivers at most one ! ! message, and if it delivers ≠ SF, then ! ! some process must have broadcast Termination !Every correct process eventually delivers ! ! some message

m

m

m

m

m

m

TRB for benign failures

Sender in round 1:1:! send m to all

Process p in round ! k, 1 ≤ k ≤ f+1! !1:! if delivered m in round k-1 and p ≠ sender then2:! ! send m to all 3:! ! halt4:! receive round k messages5:! if received m then6:! ! deliver(m)7:! ! if k = f+1 then halt8:! else if k = f+19:! ! deliver(SF)10:!! halt

Terminates in rounds

How can we do better?find a protocol whose round complexity is proportional to ! –the number of failures that actually occurred–rather than to ..–the max number of failures that may occur

f

f+1

t

Early stopping: the idea

Suppose processes can detect the set of processes that have failed by the end of round

Call that set

If there can be no active dangerous chains, and can safely deliver SF

faulty(p, i)

|faulty(p, i)| < i

p

i

Early Stopping:The Protocol

Let be the set of processes that have failed to send a message to ! ! in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ? and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6: if received value ≠ ? then7:!! value := 8:!! deliver value9:! else if or then10:! ! value := SF11:!! deliver value12:! ! if then halt

|faulty(p, k)| < k

1, . . . , k

k, 1!k!f+1

p

p

k

p

v

k=f+1

k=f+1

v

k!1

m

m

pq q k

faulty(p, k)

faulty(p, k) := faulty(p, k ! 1)"

Termination

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Termination

If in any round a process receives a value, then it delivers the value in that round

If a process has received only “?” for rounds, then it delivers SF in round

f+1

f+1

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Validity

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Validity

If the sender is correct then it sends to all in round 1

By Validity of the underlying send and receive, every correct process will receive by the end of round 1

By the protocol, every correct process will deliver by the end of round 1

m

m

m

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Agreement - 1Lemma 1:

! For any , if a process delivers! ≠ SF in round r, then there exists a sequence of processes such that = sender, , and in each round , sent and received it. Furthermore, all processes in the sequence are distinct, unless and sender

Lemma 2: ! For any , if a process sets value

to SF in round , then there exist some and a sequence of distinct processes !

! such that only receives “?” in rounds 1 to , , and in each round , sends SF to and receives SF

p0, p1, . . . , pr

p0 pr = p

pk!1 pk

p0 = p1 =

m

m

qj , qj+1, . . . , qr = p

qj

qk qk

qk!1

|faulty(qj , j)| < j

k, j+1!k!r

j!r

k, 1!k!r

r!1 p

r=1

r!1 p

r

j

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Agreement - 2

Lemma 3: ! It is impossible for and , not necessarily

correct or distinct, to set value in the same round r to and SF, respectively

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

qp

m

Agreement - 2Proof

By contradictionSuppose sets value = and sets value = SF

By Lemmas 1 and 2 there exist

with the appropriate characteristicsSince did not receive from process in round must conclude that are all faulty processesBut then,

CONTRADICTION

p0, . . . , pr

qj , . . . , qr

|faulty(qj , j)| ! j

p0, . . . , pj!1

pk!1

qj

qj

mp q

Lemma 3: ! It is impossible for and , not necessarily

correct or distinct, to set value in the same round r to and SF, respectively

qp

m

m

1!k!j k

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Agreement - 3

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Agreement - 3Let r be the earliest round in which a correct process delivers value ≠ SF

r ≤ f By Lemma 3, no (correct) process can set value differently in round rIn round r + 1 ≤ f + 1, that correct process sends its value to allEvery correct process receives and delivers the value in round r + 1 ≤ f + 1

r = f + 1By Lemma 1, there exists a sequence p0, …, pf+1

= pr of distinct processes

Consider processes p0, …, pf

f + 1 processes; only f faultyone of p0, …, pf is correct-- let it be pcTo send v in round c + 1, pc must have set

its value to v and delivered v in round c < rCONTRADICTION

ProofIf no correct process ever receives m, then every

correct process delivers SF in round f + 1

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

Integrity

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

IntegrityAt most one

Failures are benign, and a process executes at most one deliver event before halting

If ≠ SF, only if was broadcast

From Lemma 1 in the proof of Agreement

m

mm

k!1

Let be the set of processes that have failed to send a message to in any round

1:! if = sender then value := else value:= ?

Process in round !

2:! send value to all 3:! if value ≠ ?! and delivered in round then halt4:! receive round values from all5:! { | received no value from in round }6:! if received value ≠ ? then7:! ! value := 8:! ! deliver value 9:! else if or then10:! ! value := SF11:! ! deliver value12:! ! if then halt

1, . . . , k

p m

p k, 1!k!f+1

k=f+1 |faulty(p, k)|<k

k=f+1

v

v

k

q p

q k

p

m

faulty(p, k) := faulty(p, k ! 1)"

faulty(p, k)

A Lower Bound

TheoremThere is no algorithm that solves the consensus problem in fewer than rounds in the presence of crash failures, if

We consider a special case to study !! the proof technique

n ! f+2

f+1

f

(f =1)

ViewsLet α be an execution. The view of process in ., denoted by , is the subsequence of computation and message receive events that occur in together with the state of in the initial configuration of

p1 p2 p3 p4

p1 p2 p3 p4

!|pi

pipi

pi

!

!

ViewsLet α be an execution. The view of process in ., denoted by , is the subsequence of computation and message receive events that occur in together with the state of in the initial configuration of

p1 p2 p3 p4

p1 p2 p3 p4

from . from .

!|p3

!|pi

pipi

pi

!

!

p1 p4

SimilarityDefinition Let and be two executions of consensus and let ! be a correct process in both ! and . is similar to with respect to , denoted if

!1 !2

pi

!1 !2

!1 !2

pi !1 !pi!2

!1|pi = !2|pi

SimilarityDefinition Let and be two executions of consensus and let ! be a correct process in both ! and . is similar to with respect to , denoted if

!1 !2

pi

!1 !2

!1 !2

pi !1 !pi!2

!1|pi = !2|pi

Note If then decides the same value in both executions

!1 !pi!2 pi

SimilarityDefinition Let and be two executions of consensus and let ! be a correct process in both ! and . is similar to with respect to , denoted if

!1 !2

pi

!1 !2

!1 !2

pi !1 !pi!2

!1|pi = !2|pi

Note If then decides the same value in both executions

!1 !pi!2 pi

Lemma If and is correct, then dec( ) = dec( )

!1 !pi!2 pi

!1 !2

SimilarityDefinition Let and be two executions of consensus and let ! be a correct process in both ! and . is similar to with respect to , denoted if

!1 !2

pi

!1 !2

!1 !2

pi !1 !pi!2

!1|pi = !2|pi

Note If then decides the same value in both executions

!1 !pi!2 pi

Lemma If and is correct, then dec( ) = dec( )

!1 !pi!2 pi

The transitive closure of ! is denoted! .

We say that ! if there exist executions ! ! such that

!1 !pi!2

!1 ! !2

!1 ! !2

!1, !2, . . . , !k+1

!1 = "1 !pi1"2 !pi2

. . . ,!pik"k+1 = !2

!1 !2

SimilarityDefinition Let and be two executions of consensus and let ! be a correct process in both ! and . is similar to with respect to , denoted if

!1 !2

pi

!1 !2

!1 !2

pi !1 !pi!2

!1|pi = !2|pi

Note If then decides the same value in both executions

!1 !pi!2 pi

Lemma If and is correct, then dec( ) = dec( )

!1 !pi!2 pi

The transitive closure of ! is denoted! .

We say that ! if there exist executions ! ! such that

!1 !pi!2

!1 ! !2

!1 ! !2

!1, !2, . . . , !k+1

!1 = "1 !pi1"2 !pi2

. . . ,!pik"k+1 = !2

Lemma If then ! dec( ) = dec( )

!1 ! !2

!1 !2

!1 !2

Single-Failure Case

There is no algorithm that solves consensus in fewer than two rounds in the presence of one crash failure, if n!3

The IdeaBy contradiction

Consider a one-round execution in which each process proposes 0. What is the decision value?

Consider another one-round execution in which each process proposes 1. What is the decision value?

Show that there is a chain of similar executions that relate the two executions.

So what?

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i+1

p0

pi!1

pi+1

pi

pn!1

1

1

0

0

1

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

!i

sDefinition is the execution of the algorithm in

whichno failures occuronly processes propose 1

!i

p0, . . . , pi!1

1

!n

1

1

1

1

p0

pi!1

pi+1

pi

pn!1

!0

0

0

0

0

0

p0

pi!1

pi+1

pi

pn!1

!i+1

p0

pi!1

pi+1

pi

pn!1

1

1

0

0

1

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

!i

Adjacent s are similar!

Starting from , we build a set of executions ! where as follows:

is obtained from after removing the messages that sends to the j-th highest

numbered processors (excluding itself)

!i

!i

!ij 0 ! j ! n"1

!ij !

i

pi

The executions

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

…

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

…

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

n!1

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

…

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

n!1

The executionsp0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

0

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

1

…

p0

pi!1

pi+1

pi

pn!1

1

0

0

0

1

!i

n!1

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

0

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

1

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

1

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

2

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

2

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

!i

n!1

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

!i

n!1!

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

!

!

!i

n!2

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

!

!

!i

n!3

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

!

!

!i

0

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

!i

n!1

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

!

!

!i

0

!i+1

!

Indistinguishability

p0

pi!1

pi+1

pi

pn!1

1

00

0

1

!i

p0

pi!1

pi+1

pi

pn!1

1

10

0

1

! !i+1

Arbitrary failures with message authentication

Crash

Arbitrary failures withmessage authentication

Arbitrary (Byzantine) failures

Send Omission

General Omission

Receive Omission

Fail-stop

Process can send conflicting messages to different receiversMessages are signed with unforgeable signatures

Valid messages

A valid message has the following form:

in round 1: . ( is signed by the sender)

in round > 1, if received by from : where

= sender; are distinct from each other and from message has not been tampered withp1, . . . , pr

p1 pr = q

m

m

r p q

p

m : sid

m : p1 : p2 : . . . : pr

AFMA: The Idea

A correct process discards all non-valid messages it receivesIf a message is valid,

it “extracts” the value from the messageit relays the message, with its own signature appended

At round :if it extracted exactly one message, delivers itotherwise, delivers SF

p

p

p

f+1

AFMA: The ProtocolInitialization for process :! if = sender and wishes to broadcast then! !extracted := relay :=

Process in round! for each relay! ! send to all! receive round messages from all processes! relay := ! for each valid message received! ! if extracted then! ! ! extracted := extracted ! ! ! relay := relay

At the end of round ! ! if such that extracted = then! ! ! deliver ! !else deliver SF

p

{m}

p

p p m

k, 1!k!f+1

s !

k

!

s = m : p1 : p2 : . . . : pk

m !"

∪ {m}

! {s}

f+1

!m {m}

m

s : p

Termination

In round , every correct process delivers either or SF and then halts

m

f+1

Initialization for process :! if = sender and wishes to broadcast then! ! extracted := relay :=

Process in round! for each relay! ! send to all! receive round messages from all processes! relay := ! for each valid message received! ! if extracted then! ! ! extracted := extracted ! ! ! relay := relay

At the end of round ! ! if such that extracted = then! ! ! deliver ! ! else deliver SF

p p m

{m}

p k, 1!k!f+1

s !

k

!

s = m : p1 : p2 : . . . : pk

m !"

! {m}

! {s}

f+1

!m {m}

m

p

s : p

Lemma. If a correct process extracts , then every correct process eventually extracts

AgreementProofLet be the earliest round in which some correct process extracts . Let that process be .• if is the sender, then in round 1 sends a valid message to all. All correct processes extract that message in round 1• otherwise, has received in round a message! !

• Claim: are all faulty– true for – Suppose , were correct• signed and relayed message in round • extracted message in round

CONTRADICTION• If will send a valid message ! ! in round and every correct process will

extract it in round • If , by Claim above, faulty– At most faulty processes – CONTRADICTiONm

m

r

m p

pp

p r

m : p1 : p2 : . . . : pr

p1, p2, . . . , pr

p1 = s

pj , 1!j!r

pj j

pj j!1

r!f, p

m : p1 : p2 : . . . : pr : p

r+1!f+1

r+1!f+1

r =f+1 p1, p2, . . . , pf+1

f

Initialization for process :! if = sender and wishes to broadcast then! ! extracted := relay :=

Process in round! for each relay! ! send to all! receive round messages from all processes! relay := ! for each valid message received! ! if extracted then! ! ! extracted := extracted ! ! ! relay := relay

At the end of round ! ! if such that extracted = then! ! ! deliver ! ! else deliver SF

p p m

{m}

p k, 1!k!f+1

s !

k

!

s = m : p1 : p2 : . . . : pk

m !"

! {m}

! {s}

f+1

!m {m}

m

p

s : p

ValidityInitialization for process :! if = sender and wishes to broadcast then! ! extracted := relay :=

Process in round! for each relay! ! send to all! receive round messages from all processes! relay := ! for each valid message received! ! if extracted then! ! ! extracted := extracted ! ! ! relay := relay

At the end of round ! ! if such that extracted = then! ! ! deliver ! ! else deliver SF

p p m

{m}

p k, 1!k!f+1

s !

k

!

s = m : p1 : p2 : . . . : pk

m !"

! {m}

! {s}

f+1

!m {m}

m

p

s : p

Validity

From Agreement and the observation that the sender, if correct, delivers its own message.

Initialization for process :! if = sender and wishes to broadcast then! ! extracted := relay :=

Process in round! for each relay! ! send to all! receive round messages from all processes! relay := ! for each valid message received! ! if extracted then! ! ! extracted := extracted ! ! ! relay := relay

At the end of round ! ! if such that extracted = then! ! ! deliver ! ! else deliver SF

p p m

{m}

p k, 1!k!f+1

s !

k

!

s = m : p1 : p2 : . . . : pk

m !"

! {m}

! {s}

f+1

!m {m}

m

p

s : p

TRB for arbitrary failures

Crash

Arbitrary failures withmessage authentication

Arbitrary (Byzantine) failures

Send Omission

General Omission

Receive Omission

Fail-stop

Srikanth, T.K., Toueg S.Simulating Authenticated

Broadcasts to Derive Simple Fault-Tolerant AlgorithmsDistributed Computing 2 (2),

80-94

AF: The Idea

Identify the essential properties of message authentication that made AFMA work

Implement these properties without using message authentication

AF: The Approach

Introduce two primitivesbroadcast (executed by in round )accept !! (executed by in round )

Give axiomatic definitions of broadcast and acceptDerive an algorithm that solves TRB for AF using these primitivesShow an implementation of these primitives that does not use message authentication

q

p

j! i

i(p, m, i)(p, m, i)

Properties ofbroadcast and accept

Correctness If a correct process executes broadcast in round , then all correct processes will execute accept in round

Unforgeability If a correct process executes accept in round , and is correct, then did in fact execute broadcast in round

Relay If a correct process executes accept in round , then all correct processes will execute accept by round

p

p

i

i

p

i

(p, m, i) j! i

(p, m, i)

(p, m, i)

(p, m, i)

(p, m, i) j+1

(p, m, i)j! i

q

q

AF: The Protocol - 1sender in round 0:0:!extract

sender in round 1:1:!broadcast Process in round 2:!if extracted in round and ≠ sender then4:!! broadcast5:!if has executed at least accept in rounds 1 through

! (where (i) distinct from each other and from , (ii) one is , and (iii) ) and has not previously extracted then

6:!! extract 7:!if then8:!! if in the entire execution has extracted exactly one then9:!! deliver10:! else deliver SF11:! halt

(p,m, k)

m

k=f+1

(s,m, 1)

m

s

s

k, 1!k!f+1

1! i!k

1!ji!k

p

p

p

pm k!1

m

m

p

(qi,m, ji)

mp

k k

qi qip s

Termination

In round , every correct process delivers either or SF and then halts

f+1

m

sender in round 0:0:! extract sender in round 1:1:! broadcast

Process in round !! !2:! if extracted in round and ≠ sender then4:! ! broadcast 5:! if has executed at least accept in ! ! rounds 1 through ! ! ! (where (i) distinct from each other and from ! ! ! , (ii) one is , and (iii) )! and has not previously extracted then!6:! ! ! extract 7:! if then8:! ! if in the entire execution has extracted exactly ! ! ! ! ! one then9:! ! ! deliver10:! ! else deliver SF11:! ! halt

(s, m, 1)

s

m

s

k, 1!k!f+1

k!1m p

(p, m, k)

p

p k (qi, m, ji) 1! i!k

k

qi

p qi s 1!ji!k

p

m

m

m

m

p

k = f+1

p

Agreement - 1

LemmaIf a correct process extracts m, then

every correct process eventually extracts m

sender in round 0:0:! extract sender in round 1:1:! broadcast

Process in round !! !2:! if extracted in round and ≠ sender then4:! ! broadcast 5:! if has executed at least accept in ! ! rounds 1 through ! ! ! (where (i) distinct from each other and from ! ! ! , (ii) one is , and (iii) )! and has not previously extracted then!6:! ! ! extract 7:! if then8:! ! if in the entire execution has extracted exactly ! ! ! ! ! one then9:! ! ! deliver10:! ! else deliver SF11:! ! halt

(s, m, 1)

s

m

s

k, 1!k!f+1

k!1m p

(p, m, k)

p

p k (qi, m, ji) 1! i!k

k

qi

p qi s 1!ji!k

p

m

m

m

m

p

k = f+1

p

Agreement - 1

LemmaIf a correct process extracts m, then

every correct process eventually extracts m

sender in round 0:0:! extract sender in round 1:1:! broadcast

Process in round !! !2:! if extracted in round and ≠ sender then4:! ! broadcast 5:! if has executed at least accept in ! ! rounds 1 through ! ! ! (where (i) distinct from each other and from ! ! ! , (ii) one is , and (iii) )! and has not previously extracted then!6:! ! ! extract 7:! if then8:! ! if in the entire execution has extracted exactly ! ! ! ! ! one then9:! ! ! deliver10:! ! else deliver SF11:! ! halt

(s, m, 1)

s

m

s

k, 1!k!f+1

k!1m p

(p, m, k)

p

p k (qi, m, ji) 1! i!k

k

qi

p qi s 1!ji!k

p

m

m

m

m

p

k = f+1

p

Agreement - 1Proof

Let r be the earliest round in which some correct process extracts m. Let that process be p.

if r = 0, then p = s and p will execute broadcast(s,m,1) ! in round 1. By CORRECTNESS, all correct processes ! will execute accept (s,m,1) in round 1 and extract m

if r > 0, the sender is faulty. Since p has extracted ! m in round r, p has accepted at least r triples with ! properties (i), (ii), and (iii) by round r

r ≤ f By RELAY, all correct processes will have ! accepted those r triples by round r + 1p will execute broadcast(p,m,r + 1) in round r + 1By CORRECTNESS, any correct process other than

! p, q1, q2,…,qr will have accepted r + 1 triples ! (qk,m,jk), 1 ≤ jk ≤ r + 1, by round r + 1 q1, q2,…,qr,p are all distinct

every correct process other than q1, q2,…,qr,p will ! extract mp has already extracted m; what about q1, q2,…,qr?

LemmaIf a correct process extracts m, then

every correct process eventually extracts m

sender in round 0:0:! extract sender in round 1:1:! broadcast

Process in round !! !2:! if extracted in round and ≠ sender then4:! ! broadcast 5:! if has executed at least accept in ! ! rounds 1 through ! ! ! (where (i) distinct from each other and from ! ! ! , (ii) one is , and (iii) )! and has not previously extracted then!6:! ! ! extract 7:! if then8:! ! if in the entire execution has extracted exactly ! ! ! ! ! one then9:! ! ! deliver10:! ! else deliver SF11:! ! halt

(s, m, 1)

s

m

s

k, 1!k!f+1

k!1m p

(p, m, k)

p

p k (qi, m, ji) 1! i!k

k

qi

p qi s 1!ji!k

p

m

m

m

m

p

k = f+1

p

Agreement - 2 Claim: are all faulty

Suppose were correct

p has accepted in round

By UNFORGEABILITY, executed !broadcast in round

extracted m in round

CONTRADICTION

Case 2:Since there are at most f faulty processes, some process in is correct

By UNFORGEABILITY, executed broadcast in round

has extracted m in round

CONTRADICTION

ql q1, q2, . . . , qf+1

(ql,m, jl) jl ! r

ql jl!1 < f + 1

jk!1 < rqk

jk

ql

(qk,m, jk)

qk

qk

(qk,m, jk) jk ! r

q1, q2, . . . , qr

r = f+1

sender in round 0:0:! extract sender in round 1:1:! broadcast

Process in round !! !2:! if extracted in round and ≠ sender then4:! ! broadcast 5:! if has executed at least accept in ! ! rounds 1 through ! ! ! (where (i) distinct from each other and from ! ! ! , (ii) one is , and (iii) )! and has not previously extracted then!6:! ! ! extract 7:! if then8:! ! if in the entire execution has extracted exactly ! ! ! ! ! one then9:! ! ! deliver10:! ! else deliver SF11:! ! halt

(s, m, 1)

s

m

s

k, 1!k!f+1

k!1m p

(p, m, k)

p

p k (qi, m, ji) 1! i!k

k

qi

p qi s 1!ji!k

p

m

m

m

m

p

k = f+1

p

ValidityA correct sender executes ! broadcast in round 1

By CORRECTNESS, all correct processes execute accept in round 1 and extract

In order to extract a different message ! , a process must execute accept in some round

By UNFORGEABILITY, and because s is correct, no correct process can extract .

All correct processes will deliver m

m! != m

i ! f + 1

(s,m, 1)m

(s,m!, 1)

(s,m, 1)

m!

sender in round 0:0:! extract sender in round 1:1:! broadcast

Process in round !! !2:! if extracted in round and ≠ sender then4:! ! broadcast 5:! if has executed at least accept in ! ! rounds 1 through ! ! ! (where (i) distinct from each other and from ! ! ! , (ii) one is , and (iii) )! and has not previously extracted then!6:! ! ! extract 7:! if then8:! ! if in the entire execution has extracted exactly ! ! ! ! ! one then9:! ! ! deliver10:! ! else deliver SF11:! ! halt

(s, m, 1)

s

m

s

k, 1!k!f+1

k!1m p

(p, m, k)

p

p k (qi, m, ji) 1! i!k

k

qi

p qi s 1!ji!k

p

m

m

m

m

p

k = f+1

p

Implementing broadcast and accept

A process that wants to broadcast , does so through a series of witnesses

Sends to all Each correct process becomes a witness by relaying to all

If a process receives enough witness confirmations, it accepts

m

m

m

m

Can we rely on witnesses?

Only if not too many faulty processes!

Otherwise, a set of faulty processes could fool a correct process by acting as witnesses of a message that was never broadcast

How large can be with respect to ?f n

Byzantine Generals

One General G, a set of Lieutenants Li

General can order Attack (A) or Retreat (R)General may be a traitor; so may be some of the Lieutenants

* * *I. If G is trustworthy, every trustworthy Li must

follow G’s ordersII. Every trustworthy Li must follow same battleplan

The plot thickens...

G

L1 L2

One traitor

The plot thickens...

G

L1 L2

One traitor

The plot thickens...

G

L1 L2

One traitor

The plot thickens...

G

L1 L2

One traitor

The plot thickens...

G

L1 L2

One traitor

The plot thickens...

G

L1 L2

One traitor

The plot thickens...

G

L1

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

The plot thickens...

G

L1

G

L1 L2

G

L1L2L2

One traitor

A Lower Bound

TheoremThere is no algorithm that solves TRB for Byzantine failures if (Lamport, Shostak, and Pease, The Byzantine Generals Problem, ACM TOPLAS, 4 (3), 382-401, 1982)

n ! 3f

Back to the protocol...To broadcast a message in round , sends to all

A confirmation has the form

A witness sends if either:it receives from directly! orit receives confirmations for from at least ! ! processes (at least one correct witness)

A process accepts if it has received confirmations (as many as possible…)

Protocol proceeds in rounds. Each round has 2 phases

f + 1

(p, m, r)

(p, m, r) n ! f

(echo, p, m, r)

(echo, p, m, r)

(init, p, m, r)

(init, p, m, r) p

pr

Implementation of broadcast and accept

Phase 1:! sends to allPhase2:!if received in phase then3:!! sends to all /* becomes a witness */4:!if receives from at least distinct processes in phase then5:!! accepts Phase 6:!if has received from at least distinct processes in ! phases . then7:!! sends to all processes! /* becomes a witness */8:!if has received from at least processes in !! phases . then9:!! accepts

(2r, 2r + 1, . . . , j)

(2r, 2r + 1, . . . , j ! 1)

(init, p,m, r)

2r!1

2r

j >2r

(p,m, r)q

p

2r!1

(init, p,m, r)

q

q q

q

q

q

q

q

q

(p,m, r)

(echo, p,m, r)

(echo, p,m, r) n!f 2r

(echo, p,m, r) f+1

(echo, p,m, r)

(echo, p,m, r) n!f

Implementation of broadcast and accept

Phase 1:! sends to allPhase2:!if received in phase then3:!! sends to all /* becomes a witness */4:!if receives from at least distinct processes in phase then5:!! accepts Phase 6:!if has received from at least distinct processes in ! phases . then7:!! sends to all processes! /* becomes a witness */8:!if has received from at least processes in !! phases . then9:!! accepts

Is termination a problem?

(2r, 2r + 1, . . . , j)

(2r, 2r + 1, . . . , j ! 1)

(init, p,m, r)

2r!1

2r

j >2r

(p,m, r)q

p

2r!1

(init, p,m, r)

q

q q

q

q

q

q

q

q

(p,m, r)

(echo, p,m, r)

(echo, p,m, r) n!f 2r

(echo, p,m, r) f+1

(echo, p,m, r)

(echo, p,m, r) n!f

The implementation is correct

Theorem

If , the given implementation of broadcast and accept satisfies Unforgeability, Correctness, and Relay

AssumptionChannels are authenticated

n > 3f

(p, m, r) (p, m, r)

Correctness

If a correct process executes broadcast in round , then all correct processes will execute accept in round

(p, m, r)

(p, m, r)

r

r

p

CorrectnessIf is correct then

sends to all in round (phase )by Validity of the underlying send and receive, every correct process receives ! ! ! in phase every correct process becomes a witnessevery correct process sends in phasesince there are at least correct processes, every correct process receives at least echoes in phaseevery correct process executes accept! ! in phase (in round )

If a correct process executes broadcast in round , then all correct processes will execute accept in round

(p, m, r)

(p, m, r)

r

r

p

(echo, p, m, r)

(init, p, m, r)

(init, p, m, r) r

r(p, m, r)

2r!1

2r

2rn!f

2r

n!f

p

p

Unforgeability - 1If a correct process executes accept in round , and is correct, then did in fact execute broadcast in round

• Suppose executes accept in round • received from at least distinct processes by phase , where or • Let be the earliest phase in which some correct process becomes a witness to

k = 2j ! 1

k = 2j

(echo, p, m, r)

(p, m, r)

k!

q!

n−f

k

q

q (p, m, r)

j

(p, m, r) r

p

pj!r

(p, m, r)

q

Unforgeability - 1Case 1:

received from since is correct, it follows that ! did execute broadcast ! in round

Case 2: has become a witness by receiving from distinct processesat most are faulty; one is correctthis process was a witness to ! ! ! before phase

CONTRADICTIONThe first correct process receives ! from !

If a correct process executes accept in round , and is correct, then did in fact execute broadcast in round

• Suppose executes accept in round • received from at least distinct processes by phase , where or • Let be the earliest phase in which some correct process becomes a witness to

k! = 2r ! 1

k! > 2r ! 1

k = 2j ! 1

k = 2j

(echo, p, m, r) f+1

f

(p, m, r) k!

q! (init, p, m, r) p

p

p (p, m, r)r

p(init, p, m, r)

(echo, p, m, r)

(p, m, r)

k!

q!

n−f

k

q

q (p, m, r)

j

(p, m, r) r

p

pj!r

(p, m, r)

q

q!

Unforgeability -2

For to accept, some correct process must become witness.Earliest correct witness becomes so in phase . , and only if did indeed executed broadcastAny correct process that becomes a witness later can only do so if a correct process is already a witness.For any correct process to become a witness, must have executed broadcast

q

q!

2r ! 1

p

(p, m, r)

(p, m, r)

p

Relay

If a correct process executes accept in round , then all correct processes will execute accept by round

q

(p, m, r)

j + 1

(p, m, r)

j ! r

RelaySuppose correct q executes accept in round (phase or )

received at least from distinct processes by phase

At least of them are correct.

All correct procs received from at least correct processes by phase

From , it follows that . Then, all correct processes become witnesses by phase

All correct processes send by phase .

Since there are at least correct processes, all correct processes will accept by phase (round or )

If a correct process executes accept in round , then all correct processes will execute accept by round

q

(p, m, r)

j + 1

(p, m, r)

j ! r

n ! 2f

k = 2j ! 1 k = 2j

n ! 2f k

k

k + 1

2j 2j + 1

(p,m, r)

k + 1

n ! 2f " f + 1

n ! f

k

n > 3f

(p,m, r)

(echo, p,m, r)

(echo, p,m, r)

(echo, p,m, r)

q

j

n!f

Taking a step back...Specified Consensus and TRBIn the synchronous model :

solved Consensus and TRB for General Omission failuresproved lower bound on rounds required by TRBsolved TRB for AFMAproved lower bound on replication for solving TRB with AFsolved TRB with AF

Ordered Broadcastsfor Benign Failures

FIFO Order

If a process broadcasts a message before it broadcasts a message , then no correct process delivers unless it has previously delivered

If a process broadcasts a message before it broadcasts a message , then no process (correct or faulty) delivers unless it has previously delivered

Uniform FIFO Order

m

m

m

m

m!

m!

m!

m!

Causal Order

If the broadcast of a message causally precedes the broadcast of a message , then no correct process delivers unless it has previously delivered

If the broadcast of a message causally precedes the broadcast of a message , then no process (correct or faulty) delivers unless it has previously delivered .

Uniform Causal Order

m!

m!

m!

m

m

m!

m

m

From FIFO to Causal

If a process broadcasts a message m and a process delivers m before broadcasting m’, then no correct process delivers m’ unless it previously delivered m

Local Order

Causal Order = FIFO Order + Local Order

Total Order

If correct processes p and q both deliver messages m and m’, then p delivers m before m’ if and only if q delivers m before m’

If correct or faulty processes p and q both deliver messages m and m’, then p delivers m before m’ if and only if q delivers m before m’

Uniform Total Order

A Modular Approach to Broadcast Protocols

(Hadzilakos & Toueg)

Reliable Broadcast

FIFO Broadcast

Causal Broadcast

AtomicBroadcast

FIFO AtomicBroadcast

Causal AtomicBroadcast

Total Order

Total Order

Total Order

FIFO Order

FIFO Order

Causal Order

Causal Order