An Optimal Algorithm for Mutual Exclusion in Computer Networks

8/7/2019 An Optimal Algorithm for Mutual Exclusion in Computer Networks

http://slidepdf.com/reader/full/an-optimal-algorithm-for-mutual-exclusion-in-computer-networks 1/9

Operating Systems

R. Stockton Gaines

Editor

An Optimal Algorithmfor Mutual Exclusion

in Computer Networks

Glenn RicartNational Institutes of Health

Ashok K. AgrawalaUniversity of Marylan d

An algorithm is proposed that creates mutual exclu-

sion in a computer network whose nodes communicate

only by messages and do not share memory. The algo-

rithm sends only 2* (N - 1) messag es, where N is the

number of nodes in the network per critical section

invocation. This number o f messages is at a minimum if

parallel, distributed, symmetric con trol is used; hence,

the algorithm is optimal in this respect. The time needed

to achieve mutual exclusion is also minimal under some

general assumptions.

As in Lamport's "bakery algorithm," unbounded se-

quence numbers are used to provide first-come first-

served priority into the critical section. It is shown that

the number can be contained in a fixed amount of memory

by storing it as the residue of a modulus. T he number of

messages required to implement the exclusion can be

reduced by using sequential node-by-node processin g, by

using broadcast message techniques, or by sending infor-

mation through timing channels. The "readers and

writers" problem is solved by a simple modification of

the algorithm and the modifications necessary to make

the algorithm robust are described.

Key Words and Phrases: concurrent programming,

critical section, distributed algorithm, mutual exclusion,

network, synchronization

CR Categories: 4.32, 4.33, 4.35

Permission to copy without fee all or part of this material is

granted provided that the copies are not m ade or distributed for directcommercial advantage, the AC M copyright notice and the title of thepublication and its date appear, and notice is given that copying is bypermission of the Association for Computing Machinery. To copyotherwise, or to republish, requires a fee and/or specific permission.

Authors' addresses: G. Ricart, Division of Computer R esearch andTechnolo gy, Nationa l Institutes of Health, Bethesda, M D 20205; A.K.Agrawala, Dep artment of Computer Science, U niversity of Maryland,College Park, M D 20742.

This research was supported in part by the Air Force Office ofScientific Research un der grant AF OS R 78-3654A.© 1981 ACM 0001-0782/81/0100-0009500.75.

1. Introduction

An algorithm is proposed that creates mutua l exclu-

sion in a computer network whose nodes communicate

only by messages and do not share memory. It is assumed

that there is an error-free underlying communications

network in which transit times may vary and messages

may not be delivered in the order sent. Nodes are

assumed to operate correctly; the consequences of node

failure are discussed later. The algorithm is symmetrical,

exhibits fully distributed control, and is insensitive to the

relative speeds of nodes and communication links.The algorithm uses only 2* (N - 1) messages between

nodes, where N is the number of nodes and is optimal in

the sense that a symmetrical, distributed algorithm can-

not use fewer messages if requests are processed by each

node concurrently. In addition, the time required to

obtain the mutual exclusion is minimal if it is assumed

that the nodes do not have access to timing-derived

information and that they act symmetrically.

While many writers have considered implementation

of mutual exclusion [2,3,4,5,6,7,8,9], the only earlier al-

gorithm for mutual exclusion in a computer network was

proposed by Lamport [10,11]. It requires approximately

3* (N - 1) messages to be exchanged per critical section

invocation. The algorithm presented here requires fewer

messages (2* (N - 1)).

2. Algorithm

2.1 Description

A node enters its critical section after all other nodes

have been notified of the request and have sent a reply

granting their permission. A node making an attempt to

invoke mutual exclusion sends a REQUE ST message to

all other nodes. Upon receipt of the REQU EST message,

the other node either sends a REPLY immediately ordefers a response until after it leaves its own critical

section.

The algorithm is based on the fact that a node

receiving a REQ UEST message can immediately deter-

mine whether the requesting node or itself should be

allowed to enter its critical section first. The node origi-

nating the REQUEST message is never told the result of

the comparison. A REPLY message is returned imme-

diately if the originator of the REQUEST message has

priority; otherwise, the REPLY is delayed.

The priority order decision is made by comparing a

sequence number present in each REQUEST message.

If the sequence numbers are equal, the node numbers

are compared to determine which will enter first.

2.2 Specification

The network consists of N nodes. Each node executes

an identical algorithm but refers to its own unique node

number as ME. 1

ME is a pun on "mutual exclusion."

Comm unications January 1981of Volume 24th e ACM Number 1



Fig . 1 . Thr ee-no de mutu al exclusion example . (See Sect. 2 .3 .)

© G{o)

1

{c)

{.)

( ,)

{')

{, )

10 Co mmu n i ca t i o n s

o f

t h e A CM

© {b)

2 t

{d)

{h)

(J)

January 1981

V o l u me 2 4

N u m b e r 1



E ach n o d e h as

mutual exclus ion :

(l )

(2 )

(3)

th ree p rocesses to implement the

One i s awake ned wh en mu tual exclus ion i s invoked

on beha l f o f th i s node.

Another r eceives and p rocesses REQUEST mes-

sages.

The las t r ece ives and p rocesses R EPL Y messages .

The th ree p rocesses run asynchron ous ly bu t opera te

on a se t o f comm on var iab les . A sem aphore i s used to

ser ia l ize access to the com mon var iab les w hen necessary .

I f a node can ge nera te mul t ip le in ternal r eques ts fo r

mut ual exclusion , it mus t have a me thod fo r seria l iz ing

those requests .

The algor i thm is expressed below in an Algol- l ike

language.

S H A R E D DATABASEC O N S T A N T

me, ! Th i s n o d e ' s u n i q u e n u mb er

N ; ! Th e n u mb er o f n o d es i n t h e n e t w o rk

I N T E G E R O u r _ S e q u e n c e N u m b e r ,

! Th e seq u en ce n u mb er ch o sen b y a req u es t! orig inat ing a t th is node

H i g h e s t S e q u e n c e N u m b e r i n i t i a l ( 0 ) ,

! T h e h i g h es t s eq u en ce n u mb e r seen i n an y

! RE Q U E ST messag e sen t o r re ce iv edO u t s t an d i n g _ Rep l y _ Co u n t ;

! Th e n u mb er o f REPLY messag es s ti ll

! expected

BO O LEA N Req u es t i n g Cr i ti c a l_ Sec ti o n i n it i al (FA LSE) ,

! TRUE when th is node i s request ing access

! to its crit ical section

Reply_Deferred [I :N] in i t ia l (FALSE);

! Rep l y _ D efe r red [ j ] i s TRU E w h en t h i s n o d e

! i s d e fe r r i n g a REPLY t o j ' s R EQ U EST messag e

B I N A R Y S E M A P H O R EShared vars initial (1);

! In terlock access to the above sha red

! variab les when necessary

P R O C E S S W H I C H I N V O K E S M U T U A L E X C L U S I O N F O R

T H I S N O D ECo mm en t Req u es t En t ry t o o u r C r i ti c a l Sec ti o n ;

P (Shared_vats)

Co m men t C h o o se a seq u en ce n u mb er ;

Req u es t i n g Cr i t i c a l _ Sec t i o n : = TRU E;

O u r_ Seq u en ce_ N u mb er : = H i g h es t _ Seq u en ce_ N u mb er + l ;V (Shared_vars);

O u t s t a n d i n g _ Rep l y C o u n t : = N - l ;

F O R j : = I S T E P l U N T I L N D O I F j # m e T H E N

Sen d _ M essag e (REQ U EST(O u r_ Seq u en ce_ N u mb er , me ) , j ) ;

Co m men t sen t a REQ U EST messag e co n t a i n in g o u r seq u en ce n u m-b e r an d o u r n o d e n u mb er t o a l l o t h e r no d es ;

Co m men t N o w w a i t fo r a REP LY f ro m each o f t h e o t h e r no d es ;W A ITFO R (O u t s t an d i n g _ Rep l y _ Co u n t = 0 ) ;

Com men t Cri t ica l Sect ion Processing can be perfo rme d a t th is po in t ;

Comment Release the Cri t ica l Sect ion;

Req u es t i n g Cr i t i c a l _ Sec t i o n : = FA LSE;

FO R j : = l STEP 1 U N TI L N D OIF Rep l y _ D efe r red [ j ] TH EN

B E G I N

Rep l y _ D efe r red [ j ] : = FA LSE;

Sen d _ M essag e (REPLY , j ) ;

Co m men t sen d a REP LY t o n o d e j ;

EN D ;

P R O C E S S W H I C H R E C E I V E S R E Q U E S T ( k, j ) M E S S A G E SCo m men t k i s t h e seq u en ce n u mb e r b eg i n req u es t ed ,

j i s the node number making the request ;

B O O L E A N D e f e r i t ;

! TRU E w h en w e can n o t rep l y i mmed i a t e l y

H i g h es t _ Seq u en ce_ N u mb er : ~

M ax i mu m (H i g h es t _ Seq u en ce_ N u mb er , k ) ;

P (Shared_vars);Defer i t :=

Request ing_Cri t ica l_Sect ion

A N D ( (k > O u r_ seq u en ce_ N u mb er )

O R (k = O u r_ Seq u en ce_ N u mb er ANDj > me));

V (Shared_vars);

Co m men t D e fe r_ it w i l l b e TRU E i f w e h av e p r io r i ty o v e r

n o d e j ' s r eq u es t ;

I F D e f e r i t T H E N R e p l y _ D ef e r r e d [ j] : = T R U E E L S E

Sen d _ M essag e (REPLY , j ) ;

P R O C E S S W H I C H R E C E I V E S R EP L Y M E S S A G E S

O u t s t an d i n g _ Rep l y _ Co u n t := O u t s t an d i n g _ Rep l y _ Co u n t - 1;

The R EP LY processing can be represen ted by a decis ion

table:

Condi t ion and act ion en t ries

Receiv ing node i s a lso request -

ing the resource

Received message 's sequence

n u mb er co mp ared t o o u rs

Rece i v ed messag e 's n o d e n u m b er

co mp ared t o o u rs

S e n d R E P L Y b a ck

D e f e r th e R E Q U E S T

R u l e n u m b e r

1 2 3 4

N Y Y Y

I- < > =

II1

x x i x

x !

>

k

2.3 E x am p l e

Imagine a th ree-node network us ing th is a lgor i thm.

In i t ia l ly the h ighes t sequence numb er a t each nod e i s

zero . So l id l ines show REQ UE ST messages ; the numb er

is the sequence num ber o f the r eques t. The dashed l ines

s h o w R E P L Y m es sag es .

In Figu re 1 a), node 3 is the f irs t to at tem pt to invoke

mutual exclus ion . I t chooses sequence number l andsends RE QU ES T messages to nodes 1 and 2 .

Before ei ther message can arr ive, node 2 wishes to

enter i ts cr i t ical section. I t also chooses sequence n um ber

1 and sends RE QU ES T messages to the o ther nodes

(Figure l(b)) .

In Figure l(c) node 2 's messages have arr ived. At

node l , which has no t ye t made a r eques t i t se l f, a RE PLY

is immedia te ly genera ted . At node 3 , 2 ' s r eques t i s found

to have an iden t ica l sequence number to 3 ' s r eques t ;

node 2 wins on the node number t ie -b reak ing ru le . A

REPLY is sent . But at node 2, 3 's request is found to

have an iden t ica l sequence number bu t loses the t ie -

breaker . A reply is deferred.

Figure l (d ) shows node l making a r eques t to en ter

i ts cr i t ical section. I t uses sequence num be r 2 s ince i t has

received a REQUEST message wi th a sequence number

of 1 ( f rom node 2 ) . Owing to an anomaly in the com-

m u n i ca t i o n s s ys tem , t h e R E Q U E ST m es s age t o n o d e 2

over takes the REP LY that i s on i ts way there . No r ep ly

message is sent s ince the message's sequence number is

h igher than node 2 ' s sequence number .

11 Co mmu n i ca t i o n s Jan u a ry 1 98 1

o f V o l u me 2 4

t h e A C M N u m b e r !



In Figure l(e) , node 2 can now enter i ts cr i t ical

section since i t has received bot h o f the necessary replies.

Node l ' s REQUEST has a l so ar r ived a t node 3 bu t has

been defer r ed s ince the r eques t ' s sequence number i s

h igher than th a t se lec ted by node 3 .

W he n no de 2 has f inished i ts cr i t ical section process-

ing , i t sends REP LY messages back to bo th node s 1 and

3 (Figure l( f ) ) .

In Figu re l(g) , nodes 1 and 3 have received their

R E PL Y m es s ag es f r o m n o d e 2 b u t n o t y e t f r o m each

other . Node 3 's request has ar r ived at node 1. Since i tbear s a smal ler sequence number , a REPLY is immedi-

a te ly genera ted .

Figure l(h) shows node 3 enter ing i ts cr i t ical section

af ter i t received both replies .

In Figure l( i) , node 3 has f inished i ts cr i t ical section

process ing and i s r e tu rn ing the defer red RE PL Y m essage

to node 1.

Final ly in Figure l ( j ) , n ode 1 beg ins cri t ica l sect ion

processing. At the conclu sion of i ts cr i tical section, node

1 does no th in g s ince i t knows of no o ther node wish ing

to invoke m utual exclus ion .

2 .4 Di scu ss io n

The sequence numbers are s imi lar to the numbers

used by Lampor t ' s "bakery a lgor i thm." [9 ] The node

wi th the lowes t number i s the nex t one to en ter the

cr i t ical section. Ties are broken by comparing node

numbers . A RE PL Y is genera ted whe n i t s sender agrees

to a l low the node send ing a REQUEST to en ter i t s

critical section first.

T h e s eq u en ce n u m b er s p r ev en t h i g h n u m b er ed n o d es

f r o m b e i n g " s h u t - o u t " b y l o w er n u m b er ed n o d es. O n ce

n o d e A ' s R E Q U E ST m es s ag es h av e b een p r o ces s ed b y

al l o ther nodes , no o ther node may en ter i t s c r i t ica l

section twice before node A has entered i ts cr i t ical sec-

t ion.

T h e s eq u en ce n u m b er s an d n o d e n u m b er s f o r m a

v i r tua l o rder ing among r eques t ing nodes . No one o f the

nodes has a ny more in fo rmat ion than a l is t o f some or

a l l o f the o ther nodes fo l lowing i t in the v i r tua l o rder .

Yet the sys tem as a whole def ines a un ique v i r tua l

order ing based on a f irs t-come-f irs t-served discipline.

3 . Assert io n s

3 .1 Mu tu a l Ex c lu s io n

Mutual exclus ion i s ach ieved when no pai r o f nodes

is ever s imulta neou sly in i ts cr it ical section. For any p air

of nodes, one must leave i ts cr i t ical section before the

o ther may en ter .

ASSERTION.Mutual exclusion is achieved.

PROOF. Assume the con t r ary , tha t a t some t ime two

nodes (A an d B) are bo th in the i r c r it ica l sec t ions a t the

12

same t ime. E xamin e the message t r aff ic associa ted wi th

the cur ren t cycle o f the a lgor i thm tha t occur red in each

node jus t p r io r to th is cond i t ion . Each node sen t a

R E Q U E S T t o th e o t h e r an d r ece iv ed a R E PL Y .

CASE 1: Node A sent a REPLY to Node B's RE-

QUEST before choos ing i t s own sequence number .

Therefore A wi l l choose a sequence numb er h igher tha n

B ' s s eq u en ce n u m b er . W h en B r ece iv ed A ' s R E Q U E ST

with a h igher number , i t mus t have found i t s own

Reques t ing_Cr i t ica l_Sect ion = TR UE s ince th is i s se t to

b e T R U E b e f o re sen d i n g R E Q U E S T an d A h adreceived th is r eques t before send ing i ts own RE QU EST .

T h e a l g o r i t h m t h en d i r ec t s B t o d e f e r t h e R E Q U E ST

and not reply unti l i t has lef t i ts cr i t ical section. Then

node A cou ld no t ye t be in i t s c r i tica l sec t ion con t r ary to

assumpt ion .

CASE 2 : Node B sen t a RE PL Y to A 's RE QU ES T

before choos ing i t s own sequence number . Th is i s the

mir ro r image o f Case 1.

CASE 3 : Both nodes sen t a R EP LY to the o ther ' s

R E Q U E ST a f t e r ch o o s i n g t h e i r o w n s eq u en ce n u m -

ber s . Bo th nodes mus t have found the i r own Re-

questing__Crit ical_Section to be TRUE when receivingthe o ther ' s REQUEST message . Both nodes wi l l com-

p a r e t h e s eq u en ce n u m b er an d n o d e n u m b er i n t h e

R E Q U E ST m es s ag e t o t h e i r o w n s eq u en ce an d n o d e

numbers . The compar isons wi l l develop oppos i te senses

a t each n o d e an d ex ac tl y o n e w il l d e f e r th e R E Q U E ST

unti l i t has lef t i ts own cr i t ical section contradict ing the

assumpt ion .

Therefore , in a l l cases the a lgor i thm wi l l p reven t bo th

nodes f rom en ter ing the i r c r i t ica l sec tions s imul taneous ly

and mu tual exclus ion i s ach ieved .

3 .2 Dea d lo ck

The sys tem of nodes i s sa id to be dead locked w henno n ode i s in i t s c ri t ica l sec t ion and no r eques t ing no de

can ever proceed to i ts own cr i t ical section.

ASSERTION. Deadlock is impossible.

PROOF. Assum e the contra ry, tha t de adlo ck is possi-

b le . The n a l l r eques t ing nodes mus t be unab le to p roceed

to the i r c r i t ica l sec t ions because one o r more REPLYs

are ou ts tand ing . Af ter a su f f ic ien t per iod o f t ime, the

o n l y r ea s o n t h a t t h e R E PL Y co u l d n o t h av e b een r e -

ce i v ed i s t h a t t h e R E Q U E S T i s d e f e r red b y an o t h e r n o d e

which i t se l f i s wai t ing fo r REP LYs and canno t p roceed .

Therefore , there m us t ex is t a c i rcu i t o f nodes , each o fwhich has sen t a RE QU ES T to i ts successor bu t has no t

r eceived a REPLY.

Since each node in the loop has defer r ed the RE-

QUEST sen t to i t , i t mus t be r eques t ing the cr i t ica l

sec t ion i t se l f and have foun d tha t the sequence nu mb er /

n o d e n u m b er p a i r i n t h a t R E Q U E ST w as g r ea t e r t h an

i t s own. However , th i s canno t ho ld fo r a l l nodes in the

supposed c i r cu i t, and thus the asser t ion mus t be t rue .

C o m m u n i c a t i o n s J a n u a r y 1 98 1o f V o l u m e 2 4

t h e A C M N u m b e r 1



3.3 Starvation

Starvation occurs when one node must wait indefi-

nitely to enter its critical section even though other nodes

are entering and exiting their own critical sections.

ASSERTXON.Starvation is impossible.

PROOF. Assume the contrary, that starvation is pos-

sible. Nodes receiving RE QUES T messages will process

them within finite time since the process which handles

them does not block. After processing the REQUEST

sent by the starving node, a receiving node cannot issue

any new requests of its own with the same or lower

sequence number. After some period of time the se-

quence number o f the starving node will be the lowest of

any requesting node. Any REQUESTs received by the

starving node will be deferred, preventing any other

node from entering its critical section. By the previous

assertion, deadlock cannot occur and some process must

be able to enter its critical section. Since it canno t be any

other process, the starving process must be the one to

enter its critica l section.

out of a node must double as the message into some

other node.

If the algorithm presented here is modified so that

messages are sent from node to node sequentially, it

achieves the theoretical minimum number of messages

in this case also. Parallel operation is necessarily sacri-

ficed. The modifications required are considered in Sec-

tion 6.3.

5. Delay in Granting Critical Se ctions

The algorithm also grants mutual exclusion with

minimum delay if some general assumptions are made.

5.1 Definition of Delay

The delay involved in granting the critical section

resource is the stretch of time beginning with the request-

ing node asking for the critical section and ending when

that node enters its critical section. The execution time

of the instructions in the algorithm is assumed to be

negligible compared to the message transmission times.

4. Message Traffic

This algorithm requires one message to (REQUEST)

and one message from (REPLY) each other node for

each entry to a critical section. I f the network consists of

N nodes, 2* (N - l) messages are exchanged. It will be

shown that this number is the minimum required when

nodes act independently and concurrently. Hence, the

algorithm is optimal with regard to the number o f mes-

sages exchanged.

4.1 Concurrent Processing

For a symmetrical, fully distributed algorithm there

must be at least one message into and one message outof each node. If no message enters/leaves some node,

that node must not have been necessary to the algorithm;

then the algorithm is not symmetrical or is not fully

distributed. Furthermore, to allow the algorithm to op-

erate concurrently at all nodes, the messages entering

nodes must not wait for the message generated at the

conclusion of processing at other nodes. This would

indicate that two separate messages per node are re-

quired. The requesting node does not need to send and

receive messages to itself, however, and so a total of

2* (N - 1) messages are needed. This number must be

a m inimum for any parallel, symmetric, distributed al-

gorithm.

4.2 Serial Processing

If the nodes do not act independently of each other,

it is possible to reduce the number of messages by using

serial node-by-node processing. The first condit ion dis-

cussed earlier (one message into and out of each node)

still holds so a minim um of N messages are required. No

parallelism can exist in such a structure since a message

13

5.2 Assumptions

The following assumptions prevent the use o f centralcontrol or extra informat ion derived from timing:

Assumption 1. No information is available bounding

transmission time delays or giving actual transit times.

Because of this assumption, it takes one round-t rip time

to determine the state of another node. By adopting this

assumption, sending information through timing chan-

nels becomes impossible.

Assumption 2. No node possesses the critical section

resource when it has not been requested. This assumption

prevents a node or series of nodes from acting as a

central control because it retained the critical section

re source .Assumption 3. Nodes do not anticipate requests.

5.3 Bounds

Three conditions that put a lower bound on delay

times are developed and the m utua l exclusion algorithm

is shown to achieve these bounds.

5.3.1 Bound 1: Minimum delay time per request.

Before a node enters its critical section, it must make

sure that no other node is entering. To do this it must

determine the current status of any other node that could

take precedence if there is a time overlap and both nodes

are said to be requesting concurrently [10]. By assump-tion 1 this will take at least one round-trip transmission

time. By assumptions 2 and 3 this process cannot start

before the request arrives. Therefore, no request can be

serviced in less than one round-trip time.

5.3.2 Bound 2: Minimum delay time with conflict.

When two nodes are requesting concurrently, they do

not kn ow which of them made their request first because

Communic ations Januar y 1981of Volume 24the ACM Number 1



of the absence of timing inform at ion . A t i e -breaking

scheme, represent ing a to ta l order ing among reques t ing

nodes , mu s t be used . S ince the t i e -breaking ru le does not

know which node ac tua l ly made the ea r l i e r reques t , ha l f

of the t ime a c r i t i ca l s ec tion grant canno t be ma de unt i l

a f t e r the node making the l a t e r reques t has rece ived i t s

roun d- t r ip rep li es. Conf l i c t may a l so occur wi th more

than two nodes . One of them m us t be se lec ted by the ti e -

breaker to be granted access to i ts cr i t ical sect ion fi rs t .

5 .3 .3 Bound 3 : Sys tem throughput . Once a node hasre leased the c r i t i ca l s ec t ion resource , no o ther node can

ente r i t s c r i t i ca l s ec t ion in l e s s than a one-way t r ip

t r a n sm i s s io n t i m e . T h i s i s t h e m i n i m u m a m o u n t o f t im e

needed to not i fy o ther nodes tha t c r i t i ca l s ec t ion pro-

ces s ing has been comple ted and to t ransmi t the new

values of ne twork -wide informat ion .

5.4 Compliance

The a lgor i thm achieves these bounds :

CASE A: If wh en a cri t ical sect ion is released at leas t

one node is e l igible to enter i ts cr i t ical sect ion based on

Boun ds 1 and 2 wi th in a one-way t r ip t ime in the fu ture ,the a lgor i thm wi l l achieve the more ambi t ious Bound 3 .

If the nex t nod e to en ter i ts cr i tical sect ion is e l igible

und er Bo unds 1 and 2 wi th in a one-way t r ip t ime in the

fu ture , then a t l eas t one one-way t r ip t ime has e lapsed

a l ready s ince tha t nod e m ade i ts reques t. S ince i t is next,

only the node cur rent ly re leas ing the c r i t i ca l s ec t ion

c o u l d b e d e l a y i n g a R E P L Y m e s s a g e a n d t h i s R E P L Y

wi l l be t r iggered b y the re lease of the c r i t i ca l s ec tion .

This f ina l rep ly wi l l reach the next node in a one-way

t r ip t ime sa t i s fy ing Bound 3 .

CASE B: Case A does n ot hold. Th e algo ri thm

a c h i ev e s B o u n d 1 o r B o u n d 2 d e p e n d i n g u p o n i n t e r f er -

e n c e. T h e n o d e w i t h lo w e s t s e q u e n c e n u m b e r / n o d e n u m -ber pa i r a mo ng reques t ing no des wi ll have none of it s

reques t s queue d by o ther nodes and , hence , wi ll en te r it s

c r it i ca l s ec t io n i n t h e m i n i m u m a m o u n t o f t im e g i v e n b y

B o u n d s 1 a n d 2 .

In shor t , t he a lgor i thm achieves Bound B whenever

i t can do so wi thout v io la t ing Bound s 1 and 2 . The

a lgor i thm ther e fore has minim al de lay t imes und er as -

sumptions 1, 2, and 3.

T h e d e l a y t i m e e n v e l o p e w h e n p l o t t ed a g a i n st a r r iv a l

rate is discussed fu rthe r in [ 12].

W h e n a p a r t i c u l a r n e t w o r k h a s c lo s e ly b o u n d e d m e s -

sage de lay t imes and e i the r synchronized docks orknow ledge o f t rans i t times , th i s t iming info rmat io n can

be used to re duce d e lay t imes s t i ll fur the r [13].

6. Modifications

Severa l in te res t ing modi f i ca t ions can be made to the

a l g o r i t h m t o t a k e a d v a n t a g e o f d i f f e re n t e n v i r o n m e n t s.

6.1 Implicit Reply

The REPLY message ca r r i es only a s ingle b i t of

informat ion . When the message t ransmiss ion t ime be-

tween nodes has an upp er boun d, the sense of the

response can be changed so tha t no reply wi th in tha t

t ime per io d indica tes an impl ic i t rep ly . An expl ic it mes -

s ag e , c a ll e d " D E F E R R E D " , i s s e nt w h e n R E P L Y w o u l d

ordinar i ly not be se re .

T h e n u m b e r o f m e ss a g es r e q u i r e d b y t h e i m p l ic i t

r e p l y s c h e m e v a r ie s b e t w e e n l * ( N - l ) a n d 3 * ( N - l )

d e p e n d in g o n t h e n u m b e r o f D E F E R R E D m e ss ag e ssent. W hen the re i s l it t le conten t ion for the c r i t ica l s ec -

t i o n r e s o u rc e , t h e n u m b e r o f m e s sa g e s a p p r o a c h e s

I * ( N - 1 ) .

S ince a reques t ing node mus t usua l ly wa i t for the

maximum round- t r ip t ime before en te r ing i t s c r i t i ca l

sec tion , the use fu lness of th i s modi f i ca t ion d epends on

a n u p p e r b o u n d f o r tr a n s m i ss i o n t im e w h i c h i s n o t m u c h

la rger than the average .

6 .2 B r o a d c a s t M e s s a g e s

W h e n t h e c o m m u n i c a t i o n s s t r u c t u r e b e t w e e n n o d e s

permi t s broadcas t messages , the in i t i a l REQUEST mes -

sage can be sent us ing tha t mechanism. The message

t ra f f i c i s reduce d to N messages , one broadcas t RE-

Q U E S T a n d ( N - 1 ) R E P L Y s . I f c o m b i n e d w i t h t h e

impl ic i t rep ly modi f i ca t ion d i scussed above , the message

count can be as low as one .

6.2.1 Communications medium sequenc ing . Broad-

c a s t R E Q U E S T m e s s a g e s n e e d n o t c o n t a i n t h e u s u a l

sequence n um ber i f the i r t ime of successfu l t ransmiss ion

c a n b e m o n i t o r e d . T h e b r o a d c a s t m e d i u m e n f o r c es se ri -

a l i za t i o n o f t h e R E Q U E S T s a n d a q u e u e i n g o r d e r e q u i v-

a l e n t t o t h e s e q u e n c e n u m b e r s m a y b e o b t a i n e d b y

o b s e r v i n g t h e o r d e r o f R E Q U E S T m e s s ag e s a p p e a r in g

o n t h e b r o a d c a s t m e d i u m .

T h e R E P L Y m e s s a g e s c a n a l s o b e b r o a d c a s t , a n donly two messages pe r c r i t i ca l s ec t ion invoca t ion a re

r e q u i r e d . R E P L Y s a r e o n l y n e e d e d f r o m t h o s e o t h e r

nodes which have themse lves success fu l ly broadcas t a

p r i o r R E Q U E S T b u t r e c e i v e d n o c o r r e s p o n d i n g R E -

P L Y .

6.2.2 No communications medium sequencing. E v e n

i f t h e o r d e r o f s u c ce s sf u l R E Q U E S T b r o a d c a s ts c a n n o t

be moni tored , i t i s use fu l to broadcas t the REPLY mes -

sages fol lowing cri t ical sect ion process ing. The s ize of

t h e a u d i e n c e d e p e n d s o n t h e d e g r e e o f c o n te n t i o n . A

broadca s t RE PL Y m essage mus t conta in a l is t of in-

t ended rec ip ient s because i t i s not suf f i c ien t for nodes

wai t ing for a R EP LY to as sume i t appl ies to them. 2

2 Examp le: While no de 1 is perform ing critical section processingrelated to its reque st with sequence numb er 1, node 2 decid es o issuea REQUEST message with sequ encenumber 2. Before he REQUESTmessagearrives at node 1, node I com pletes ts critical section process-ing and broadcasts the REPLY it owes some other node(s). Without alist of intended recipients, node 2 m ight think that th e RE PLY appliesto its REQU EST message and continue. In fact, node 1 may mak e anew requ est with sequence number 2 and be entitled to enter its criticalsection first due to the tie-breaking rule.

14 Communications January 1981of Volume 24the ACM Number 1



6.3 Ring Structure

The numb er of messages can be cut to N by process-

ing the re quests serially through a logical circuit consist-

ing of all nodes instea d o f allowing processing to proceed

concurrent ly. N is the minimum numb er o f messages

required for any distributed symmetric algori thm when

broadcast ing is not avai lable and information is not sent

via timing channels. 3

The algori thm must be modified by replacing the

REP LY message with an echo of the REQ UE ST mes-

sage. As the REQUEST message t ravels around thecircuit of nodes, it may be deferred at several stops.

When it is received at the initiating node, mutual exclu-

sion has been achieved and critical section processing

may begin.

A further possible modificat ion sends the REQ UE ST

message from node to node around the circui t without

pause bu t the no ta t ion "D EF ER RE D by node j " i s

added by each node j that is copying and deferring the

request . The Outstanding_Reply_Count is then set ac-

cording to the notations when it arrives back at the

ini t iat ing node. The nodes which have marked the RE-

QUEST as deferred generate individual REPLYs in the

usual way. This technique comes close to N messageswhile eliminating the cumulative delays at each stop.

6.4 Bounding Sequence Numbers

The sequence numbers in the algori thm increase at

each critical section invocation and are theoretically

unboun ded. The t icket numbers of the "baker y algo-

rithm" [9] suffer from the same problem.

A technique for l imit ing the a mount of s torage nec-

essary to hold these unbounded numbers can be bor-

rowed from computer communicat ions protocols . Al-

though the numbers themselves are unbounded, their

range is bounded . The sequence numbers increase by no

more than one each t ime a node requests entry to i tscritical section. That request cannot be granted as long

as a lower sequence number request is outstanding.

Therefore the numbers must fal l within the range from

x t o x + N - 1 .

The sequence numbers can be s tored modulo M

where M _> 2N - 1. W hen m aking a compar ison, the

smaller number should be increased by M if the differ-

ence is N or more. Thus only l o g 2 ( 2 N - 1) bits o f storage

are need ed regardless of the num ber of t imes the crit ical

section is entered.

6.5 Sequence Number Incrementation

Aside from this method for limiting the storage re-quired to hold sequence numbers, there is no reason for

incrementing sequence numb ers in uni t s teps. Two si tu-

ations make larger increments attractive:

(1) The algori thm tends to favor lower numb ered

nodes slightly, owing to the tie-breaking rule. This

a To involve all nodes, at least one mes sage must be received an d

one sent per node. The minimum number of messages that meet this

requirement is N.

favori t ism can be reduce d by incrementing the sequence

num ber b y a random integer. The t ie-breaking node

num ber is s ti ll required in case the random integers used

were equal .

(2) Deliberate priori ty can be introduced by in-

structing high priori ty nodes to use small increments and

low priority nodes to use large increments. In addition,

high priori ty nodes m ay be al lowed to monopolize cri ti -

cal section processing until forced to increment their

sequence numbe rs past the one chosen by a lower priori ty

node. In doing so, the process at a high priority nodewhich receives and handles messages may choose to

delay act ing on those received from low priori ty nodes

in order to keep the Highest_Sequence_Number from

being prematurely incremented past the one chosen by

the low priori ty node.

6.6 Readers and Writers

The algori thm is easi ly modified to solve the "R ead-

ers and Writers" problem [1] where wri ters are given

priori ty. The modificat ion is s imply that "readers" never

defer a REQUEST for ano ther " reader" ; ins tead they

always REP LY immediately. "Writers" fol low the orig-

inal algorithm.

7. Considerations for Practical Networks

7.1 No de Numbers

It is more convenient to draw node numbers from a

larger range than 1 . . . N. The algori thm may be changed

to map the integers 1 . . . N into the actual node numbers

by indexing a t ab le NAMES [1 . . . N] . The compar i son

of node numbers should then be performed by compar-

ing the values contained in NAMES.

7.2 Insertion of New Nodes

New nodes may be added to the group part icipat ing

in the mutual exclusion algori thm. They must be as-

signed unique node numbers, obtain a l is t of part icipat-

ing nodes, be placed on every other node's l is t of part ic-

ipants , and acquire an ap propriate value for their High-

est_Sequence_Number variable.

7.2.1 Res tart interval. I f the node could have been

previously operational in the group (e.g., it failed and is

now restarting), it should first notify other nodes that it

fai led and then wait long enough to be sure i ts old

messages were del ivered and the network processed i ts

removal . Usual ly the network wil l already be aware ofthe nod e's failure, but this cannot be assum ed. If this

step was not fol lowed, the fai lure may be detected at

approximately the same t ime as the node rejoins the

group. This would result in conflicting bookkeeping at

different nodes.

7.2.2 Reconcile participant lists. A new node mus t

obtain a l ist of other part icipat ing nodes and have i tself

15 Communicat ions Januar y 1981

of Volume 24

the ACM Number i



added to the others' lists. A new node should contact a

"sponsor" node which is already participating in the

group. The sponsor should then invoke mutual exclusion,

initialize the new node's participant list from its own,

and broadcast the new node's identity before releasing

mutual exclusion. Each node receiving this notification

adds the new node number to its NAMES array and

increments N, the num ber o f active nodes.

An alternative is possible if the communications net-

work can deliver a message to all other nodes without

the sender naming all the other nodes in the network. In

this case a new node obtains a list of participants from

a nearby node and then sends a broadcast message

asking all other nodes to include it on their list of

participating nodes.

7.2.3 Set highest sequence number. The High-

est_Sequence_Number variable of a new node must not

be set to any value lower than the sequence number of

any R EQUEST message which would already have been

received had the new node been continuously active.

Until an appropriate value of Highest_Se-

quence_Number is obtained, mutual exclusion cannot

be requested and incoming REQUEST messages areprocessed normally.

A new node can determine that its Highest_

Sequence_Number is high enough by several methods.

(1) Ask all other nodes for their Highest_

Sequence_Number and use the largest.

(2) Wait until one REQUES T message has been re-

ceived from every other node.

(3) Wait until the sequence numbers on REQUES T

messages have increased by N - 1.

(4) Wait until all (N - 1) nodes would have time to

enter and leave their critical sections even if they

all had outstand ing requests. This requires the abil-

ity to bound message transmission times and critical

section times. If no REQUEST message is received

during this time, the value of Highest_

Sequence_Number from any nearby node can be

used.

(5) Wait until the fourth REQUEST message is re-

ceived from a single node. This method requires

that messages are sent and delivered in the same

order. [See Appendix.]

The new node may request access to its critical section

after any of the above methods has been used to verify

that its Highest_Sequence_Number variable is suffi-

ciently high.

7.3 Removal of Nodes

A node wishing to leave the group may do so by

notifying all other nodes of its intention. The othe r nodes

should acknowledge this message. While waiting for

acknowledgement, the departing node may not request

mutual exclusion and must continue to send REPLY

messages to any REQUEST messages it receives. Each

node checks to see if the departing node is listed in its

NAMES array, and if so, removes it and decrements the

value of N, the number of active nodes, by one. If

messages may be delivered out of order, a node awaiting

a REPLY message from a departing node should pretend

the REP LY was received.

7.4 Node Failures

In practice some nodes fail and will not respond to

messages directed at them. To prevent this situation from

stopping the proposed mutual exclusion algorithm, a

timeout-recovery mechanism may be added. The

timeout detection of a failed node relies on knowledge

of an upper bound on the time which may elapse before

a working node responds to a message and an estimate

of the maximum processing time within a critical section.

The only message in the original algorithm which de-

mands a response is the REQUE ST message.

A requesting node should start a timer when the

REQUEST messages are sent. The timer should be

restarted when a REPLY is received and cancelled when

the critical section processing begins.

A bit map, Awaiting_Reply [1... N], can be used to

identify which nodes have not yet sent a REPLY mes-sage. The Awaiting_Reply array is set to all TRUE

values before a REQUEST message is issued. Individual

bits are turned o ff when REP LY messages are received.

If the timer expires, 4 all nodes for which Awaiting_

Reply is TRUE are suspected of having failed. A probing

message, ARE_YOU_THERE(me), should be sent to

each suspect node. If no answer is received during a

second timeout period, ~ the suspect node has failed.

When an ARE_YOU_THERE(j) message is re-

ceived, Reply_Deferred[j] should be examined. If it is

FALSE, it must be that the REQUES T was not received,

the REPLY was lost, or the node has restarted; the

correct response is REPLY(me). If Reply_Deferred[ j] isTRUE, a YES_I_AM_HERE message should be sent to

confirm that the node is alive.

The timeout does not impose an upper limit on the

dura tion of a critical section. If critical section processing

exceeds the timeout, all nodes will respond with YES I

AM_HERE messages and a new timeout period may

begin.

When it has been determined that node j has failed,

this can be broadcast by the node detecting the failure.

Any node which is awaiting a REPLY message from the

failed node should pretend that a REPLY was received.

In addition the node should be erased from the NAMES

array if present and N, the number of active nodes,decremented by one.

If the failed node recognizes that it has failed and

ha s been restarted, it may return to the group through

4 The appropria te value is worst-case round-tr ip message trans-

mission t ime plus worst-case processing t ime a t the d is tant node p lusa reasonable est imate of maxi mum cri t ica l section t ime.

In th is case just round-tr ip message t ime plus worst-case process-

ing t ime a t the d is tant node.

16 Comm unications Januar y 1981of Volume 24t h e A C M N u m b e r 1



the mechanism for adding a new node. If it does not

know that it has failed and issues new R EQ UE ST mes-

sages, any node w hich receives the REQU EST message

and does not fred the node's name in its NAMES array

may return a special message notifying the node that it

should restart itself and use the insertion protocol.

8. Conclusion

An algor i thm is p resen ted tha t implements mutual

exclus ion in a computer ne twork . No a lgor i thm uses

fewer messages, operates faster , and exhibits concurrent ,

symmetr ic , and d is t r ibu ted con t rol . The a lgor i thm is safe

and l ive and mechan isms ex is t to hand le nod e inser t ion ,

r emoval , and fa i lu re .

M o d i f i ca t i o n s can b e m ad e t o r ed u ce t h e n u m b er o f

messages by tak ing advan tage o f ser ia l p rocess ing ,

b roadcas t messages , and t r ansmi t t ing in fo rmat ion

th rough omi t ted r esponses . The sequence numbers can

be s to red in l imi ted memo ry by keep ing them as r esidues

of a mod ulus tha t i s a t leas t tw ice as la rge as the num ber

of nodes . The r eader s and wr i ter s p rob lem is so lved by

the same a lgor i thm wi th a s imple modif ica t ion .

Appendix. The Effect of M es s ag e O r d e ri n g

The a lgor i thm presen ted in th is paper does no t de-

pend on messages being del ivered o r ac ted upon in the

order in which they are sen t . I f such a co nd i t ion does

exist, there is a s tronger l imit to the nu mb er o f times

other nodes can enter their cr i t ical sections before a

reques t ing node A can .

Wi thou t de l ivery in o rder o f t r ansmiss ion , the wors t

case anMysis shows that N ( N + 1)/2 - 1 nodes can enter

the i r c r i tica l sec tion before Node A may .

To determine th is bound, assume that A has the h ighest node

num ber and the refore the least p riori ty in breaking t ies. A 's sequence

n u mb er may b e (N - 1 ) h i g h e r t h an t h e l o w es t o u t s t an d i n g seq u en ce

numbe r. (See Sect ion 6 .4 .) I t i s possib le , by judic iously order ing the

del ivery of messages, for each o ther node to en ter i t s cri t ica l sect ion

w i t h i t s s eq u en ce n u mb er t ak i n g o n each v a l u e b e t w een i t s cu r ren t

value and A 's value . To get the worst case , assume that a l l nodes have

ch o sen a d i s t i n c t seq u en ce n u mb er w i t h A ' s n u mb er t h e h i g h es t .

Therefore , one nod e can en ter i t s cri tica l sect ion N t imes before A may,

an o t h e r (N - 1 ), an o t h e r (N - 2 ) an d so o n d o w n t o th e n o d e w h o se

RE Q U E ST messag e cau sed A ' s seq u en ce n u mb er se l ec t io n . Th i s t ak es

two cri tica l sect ion en t ries a t most . This sum, N + (N - 1) + ( N - 2)

+ . . . + 3 + 2 , i s t h e n u mb er o f t i mes o t h e r no d es may en t e r t he i r

cri t ica l sect ion after A has m ade a request in the worst case .

I f de l ivery i s guaran tee d to be in the o rder o f tr ans-

mission, no othe r node m ay ente r i ts cr it ical section more

than twice between the t ime that A selects a sequence

number and A is permitted to enter i ts cr i t ical section.

No m ore tha n 2*(N - 1) cr i t ical sections are possiblebefore A m ay en ter .

To get th is bound observ e that after node .4 has done i t s "No de

Reque sts Cri t ica l Sect ion" processing , i t cannot receive mor e than o ne

REQ U EST f ro m an o t h e r n o d e ( j ) w h i ch co n t a i n s a l o w er o r eq u a l

seq u en ce n u mb er . By t h e t i me i t g e t s t h e REPL Y f ro m t h i s REQ U ES T,

i t mu s t a l so h av e rece i v ed A ' s RE Q U ES T; i t c an n o t t h e rea f t e r sel ec t a

l o w er o r eq u a l seq u en ce n u mb er . Each o t h e r n o d e j c an en t e r i t s c r i ti c a l

sec t i o n a t mo s t o n ce b ecau se o f an a l read y ap p ro v ed REQ U ES T an d

o n ce w i t h t h e o n e REQ U EST w h i ch co n t a in s a l o w er or eq u a l seq u en ce

number. If every o ther node fo l lows th is worst case pat tern , a t most

2*(N - 1) cri t ica l sect ion en t ries may preceed A 's

When del ivery in o rder i s used , a new node may

assume i t s Highes t_Sequence_Number i s synchron ized

w h en i t h a s h ea r d t h e fo u r t h R E Q U E S T m es s ag e f r om

the same node.

A ssu me t h a t a n o d e j sen t it s REQ U ES T messag es b e fo re th e n ew

node came on-l ine . The new node i s no t synchronized unt i l i t ho lds a

h i g h e r n u mb er i n H i g h es t _ Seq u en ceN u mb er t h an t h e seq u en ce n u m-

ber used by j . Th e refer ence node B (which i s generat ing the four

requests) can en ter i t s cri t ica l sect ion a t most twice before nod e j en ters

i t s cri t ica l sect ion . Therefore , by the t ime B enters i t s cri t ica l sect ion

t h e t h i rd t i me , n o n o d es l i k e j ex i s t w h i ch d id n o t k n o w ab o u t t h e n ew

n o d e w h en t h ey mad e t h e i r r eq u es t s . Re fe ren ce n o d e B may h av e

i s su ed t h ree REQ U ES T messag es seen b y t h e n ew n o d e b e fo re en te r i n g

i t s cri t ica l sect ion for the th i rd t ime. The fourth REQUEST message

guarantees that the cri t ica l sect ion was en tered for the th i rd t ime.

Acknowledgment. T h e au t h o r s w i s h t o t h an k R .Stock ton Gaines fo r h is de ta i led and help fu l comments

on the p resen ta t ion o f th i s mater ia l .

Received 3 /78; rev ised 3 /80; accepted 9 /80

References

1. Courto is , P.J . , Heym ans, F. , and Parnas, D.L. Concu rrent cont ro l

w i t h " read e rs" an d "w r i t e r s. " Comm. ACM 14, 10 (Oct. 1971), 667-

668.

2 . d eBru i jn , N .G . A d d i t i o n a l co mmen t s o n a p ro b l em in co n cu r ren t

p ro g rammi n g an d co n t ro l . Comm. ACM 10, 3 (March 1967), 137-

138.3 . Di jkst ra , E.W. Hierarchical ordering of sequent ia l p rocesses. Acta

lnformatica 1, 2 (1971), 115- 138.

4 . D i j k st ra , E .W . So l ut i o n o f a p ro b l em i n co n cu r ren t p ro g rammi n gco n t ro l Comm. ACM 8, 9 (Sept. 1965), 569.

5 . D i j k s tra , E .W . Th e s t ru c t u re o f t h e TH E mu l t i p ro g rammi n g

system. Comm. ACM 11, 5 (May 1968), 34!-346.

6 . E i sen b erg , M .A . , an d M cG u i re , M .R . Fu r t h e r co mmen t s o n

D i j k s tra ' s co n cu r ren t p ro g ram mi n g co n t ro l p ro b l em. Comm. A CM15, 11 (Nov. 1972), 999.

7 . Hi l l , J . Carver. Synchroniz ing processors wi th mem ory- conte nts-

generated in terrupts . Comm. ACM 16, 6 (June 1973), 350-351.

8 . K n u t h , D .E . A d d i t i o n a l co mm en t s o n a p ro b l em i n co n cu r ren t

p ro g ram mi n g co n t ro l. Comm. ACM 9, 5 (May 1966), 321-322.9 . Lampo rt , U A new solu t ion of Di jkst ra ' s concurr ent

p ro g rammi n g p ro b l em. Comm. ACM 17, 8 (Aug. 1974), 453-455.

10. Lamp ort , U Time, c locks and the ordering of events in ad is t r ibu ted system. Comm. A CM 21, 7 (July 1978), 558-565.

11 . L ampor t , L. Time, c locks and the ordering of events in a

d is t r ibu ted system. Rep. CA-7603-2911, Mass. Comptr. Assoc. ,

Wakefie ld , Mass. March 1976.12. Ricart , G. , and Agrawala , A.K. Perfo rma nce of a d is t r ibu ted

network mutual exclusion a lgori thm. Tech . Rept . TR-774, Dept .

Comp tr. Sci . , Univ . of Maryland , Col lege Park , Md., Ma rch 1979.

13 . Ricart , G. , and Agrawala , A.K. Using exact t iming to implem ent

mutual exclusion in a d is t r ibu ted network . Tech . Rept . TR-742, Dept .

Com ptr. Sci . , Univ . of Maryland , Col lege Park , Md. M arch 1979.

17 Co mmu n i ca t i o n s Jan u a ry 1 98 1

o f V o l u me 2 4

t h e A C M N u m b e r 1

An Optimal Algorithm for Mutual Exclusion in Computer Networks

Documents

Transcript of An Optimal Algorithm for Mutual Exclusion in Computer Networks