Parallel and Distributed Processing CSE 8380

Computer Science and Engineering

Parallel and Distributed Processing

CSE 8380

February 10, 2005February 10, 2005

Session 9Session 9


Contents

Message Passing Model

Complexity Analysis

Summation

Leader Election


Introduction

Recent migrations to distributed systems have increased he need for a better understanding of message passing computational model and algorithms

A processing unit in such systems is an autonomous computer, which may be engaged in its own private computation while at the same time cooperating with other units in the context of some computational task.


Message Passing Computing Models Algorithm – collection of local programs running

concurrently on different processing units. Each program performs a sequence of computation and message passing operations

Message passing can be represented using a communication graph

Processor vs. Process


Degree of Synchrony

Synchronous – Computation and communication are dome in a lockstep manner. (rounds of send, receive, compute)

Asynchronous – processing units take steps at arbitrary speeds and communication delay is unpredictable

Partially synchronous – restrictions on the relative timing events


Synchronous Model n processors & communication graph G(V,E) Modeled as a state machine System is initialized and set to an arbitrary initial state For each process i in V repeat in synchronized rounds

Send messages to outgoing neighbors by applying some message generation function to current state

Obtain the new state by applying a state transition function to the current state and the received messages


Asynchronous Model

n processors & communication graph G(V,E)

Communication does not happen in synchronized rounds

Messages incur an unbounded and unpredictable delay

I/O automata (simple type of state machine in which transitions are associated with actions) is used to model asynchronous systems


Synchronous model as a state machine

M, a fixed message alphabet A process i can be modeled as

Qi, a (possibly infinite) set of states. The system state can be represented using a set of variables.

q0,i, the initial state in the state set Qi. The state variables have initial values in the initial state.

GenMsgi, a message generation function. It is applied to the current system state to generate messages to the outgoing neighbors from elements in M.

Transi, a state transition function that maps the current state and the incoming messages into a new state.


Algorithm Template

Algorithm S_Template

Qi

<state variables used by process i>

q0,i

<state variables> ← <initial values>

GenMsgi

<Send one message to each of a (possibly empty) subset of outgoing neighbors>

Transi

<update the state variables based on the incoming messages>


Complexity Analysis

Message Complexity

Number of messages sent between neighbors during the execution of the algorithm in the worst case

Time Complexity

Synchronous – Number of rounds


Summation on a hypercubeAlgorithm S_Sum_HypercubeQi

buff, an integerdim, a value in {1, 2, ..., log n}

q0,i

buff ← xidim ← log n

GenMsgi

If the current value of dim = 0, do nothing. Otherwise, send the current value of buff to the neighbor along the dimension dim.

Transi

if the incoming message is v & dim > 0, then buff ← buff + v, dim ← dim - 1


Group Work

Work the following example with your neighbor


Leader Election Problem

A leader among n processors is the one that is recognized by all other processors as distinguished to perform a special task

The problem occur when the processors must choose one of them as a leader.

Each processor should eventually decide whether or not it is a leader (each processor is only aware of its identification)

Most important when coordination among processors becomes necessary to recover from a failure or topological change.


A solution

Given a communication graph G = (V,E)

Two steps Each node in the graph would broadcast its unique identifier to

all other nodes After receiving the identifier of all nodes, the node with the

highest identifier declares itself as the leader

Complexity Time Message


Leader election in synchronous rings

By Chang and Roberts

Assumptions Communication is unidirectional

(clockwise) The size of the ring is not known The ID of each node is unique


Leader election in synchronous rings

Algorithm

1. Each process sends its ID to its outgoing neighbor

2. When a process receives an ID from an incoming neighbor, then:

The process sends null to its outgoing neighbor, if the received ID is less than its own ID

The process sends the received ID to its outgoing neighbor, if the received ID is greater than its own ID

The process declares itself as a leader, if the received identifier is equal to its own identifier


Complexity Analysis

Given n processors connected via a ring

Time Complexity – O(n)

Message Complexity – O(n2)

Why?


Group Work

1- Work the following example with your neighbor

2- Study the improved leader election algorithm

Parallel and Distributed Processing CSE 8380

Documents

Transcript of Parallel and Distributed Processing CSE 8380