3. Communication in distributed system

3.1 Introduction

Interprocess communication is at the heart of distributed systems

Distributed Application

Communication middleware

Low-level message passing underlying network

IP addressport, ...

Name, ...

Middleware Protocols

An adapted reference model for networked communication.

2-5

P69

Distributed Middleware

• Communication middleware: RPC, RMI, MPI– name DCE, CORBA

– message-oriented– various communication forms

• Authentication, authorization protocols• Distributed commit protocols

Client-Server TCP

Normal operation of TCP. Transactional TCP.

2-4

P65

Two issues for communication mechanism:• How indicate destination?• How many methods (forms)?

S

TQ

P

Indicate sending and receiving processes

1) <IP address, port no.> ex, <202.119.32.6, 80>

2) Unique name

ex, www.nju.edu.cn; procedure name( )

S

TQ

P

1. Addressing

…...

Domain name IP address Type

www.nju.edu.cn 202.119.32.6 Acs.nju.edu.cn 202.119.36.5 A …… …...

DNS

FTPd webserver

ftp://nju.edu.cnwww.nju.edu.cn

202.119.32.6

80 21

<202.119.32.6, 21><202.119.32.6, 80>

3) Mailbox (queue, indirect)

multi-in, multi-out

2. Methods 1) Synchronous/asynchronous

sender

sender

receiver

receivermailbox

P1 P2

send

receivewaiting

receivewaiting

send

Synchronous: sender will be blocked if receiver is not ready

send( ), recv( ); RPC

P1 P2

send

send

receive

receive

Internet?

Asynchronous UDP, Asynchronous RPC

2) Transient/persistent• Persistent communication:

A message is stored by communication system as long as it takes to deliver it. Neither the sender nor the receiver need to be up.

Ex, email

Synchronous wait, no broadcast safe, simple

Asynchronous buffer full,not safe parallelism, broadcast

Defects merits

Comparison

P100-1

Persistence in Communication P100

• Transient communication:

A message is stored by communication system as long as the sending and receiving applications are executing

Ex, send, recv in Socket

Question: difference (relation) between 1) and 2)

transient synchronous

persistent asynchronous

As to: 1) sender/receiver; 2) message

send, recv; RPC

UDP; asyn RPC

email

Two successive messages have a temporal

relationship.

Ex, movie, audio stream

3) Stream-oriented communication

P1 P2 Proc G(u,v)

Call P2.G(x,y)

node 1 node 2

……

…..

3.2 RPC P68

2003.7 Worm.msBlast

1. Conventional procedure call

Node 1

process Procedure Q(u,box,v)

{ … { …

Q(i,buf,j) …

…

} }

return addr.i

bufj

sp

stack

result

2. How does RPC work?

node 1 node 2

process Procedure Q(u,box,v)

{ … { …

Q(i,buf,j) …

… }

}

?

stack stack

return

sp

return addr 2i

bufjsp

result

return addr.1

bufj

i

proc Qint i ? bufint j resultresult

Original procedure call: Now

push return addr.1

push i

Q(i,buf,j) push buf

push j

goto Q goto c_stub

take result

C_stub

• pack parameters & name• send them to S_stub• receive result• unpack result• return

S_stub

• receive message• unpack message• push return addr.• goto Q( )• take & pack result• send it to C_stub

Q( ){ }

transform

Steps of a Remote Procedure Call

• Client procedure calls client stub in normal way• Client stub builds message, calls local OS• Client's OS sends message to remote OS• Remote OS gives message to server stub• Server stub unpacks parameters, calls server• Server does work, returns result to the stub• Server stub packs it in message, calls local OS• Server's OS sends message to client's OS• Client's OS gives message to client stub• Stub unpacks result, returns to client

P72

Passing Value Parameters

Steps involved in doing remote computation through RPC

2-8

P74

Issues to be handledMachine may be not identicalExecute in different spacesMachine may crash

3. Parameter passingImplementation should consider:

Different types of computers Pointer or call by reference

1) Deal with different types of computers

5 Jill

(a) Original message on the Pentium(b) The message after receipt on the SPARC(c) The message after being inverted. The little numbers in boxes indicate the address of each byte

P75

How represent parameters? – In intermediate form– Indicating in type i

typei typej

Message should include parameters and their type

2) Deal with call by reference

proc Qint i

char bufint j

Node1 Node2Main( )

{ int B[100] proc write(name,buf,len)

… { int len, buf[ ];

write(f1,B,100) …

… }

proc read(name, buf, len)

{ int len, buf[ ];

read(f2,B,100) …

}Write

f1B=5200

100 message

B5200

5200

Writefile f1

Array 100B[1]B[2]……

B[100]int 100

message

Readfile f2

Array 100int 100

message

read

write

proc write(in char name, in int buf[], in int len);

proc read(in char name, out int buf[], in int len);

– Input parameter deliver the array– Output parameter needn’t

4. Where stubs come from?Write server specification (interfaces) including:

Server name, version, list of procs

Stub compiler generates c_stub & s_stub

Specification of file server

Specification of file_server, version 3.1:

long read(in char name[MAX_PATH], out char buf[BUF_SIZE], in long bytes, in lon

g position);

long write(in char name[MAX_PATH], in char buf[BUF_SIZE], in long bytes, in lon

g position);

int create(in char name[MAX_PATH], in int mode);

int delete(in char name[MAX_PATH]);

end; returnIDL

5. Steps for using RPC• write specification• compile• write server code and link with s_stub• write client code and link with c_stub

specificationC_stub

S_stub Stubcompiler

The steps in writing a client and a server in DCE RPC.

P82

user.c file_server.c

proc read( )

read( ) proc write( )

c_stub s_stub

user.exe file_server.exe

read( )

{…… }

{…… }

6. Dynamic binding --- How client find server

1) Register: name, version, handle(IP,port), unique id

2) Look up: name, version (first time)

3) Reply: handle, unique id

4) Request: unique id, parameters

5) Deregister: name, version, unique id

binder

client server

12

3

4

• Advantage: flexible– Server move won’t affect client

– Binder can even load if multiple servers support the same interface

• Disadvantage– Overhead at running time

– Binder is bottleneck

Addressing: name

Method: synchronous, transient

RPC

NFS

XDR

Socket/TLITCP|UDP

IPARP

Driver

ICMP

FTPTelnet

AddressingMethod

7. Extended RPC --- asynchronous RPC

(a) The interconnection between client and server in a traditional RPC

(b) The interaction using asynchronous RPC

A B100 P79

A client and server interacting through two asynchronous RPCs

2-13

A client wants to prefetch network addresses of a set of hosts

P80

3.3 RMI (or Remote Object Invocation)

Common organization of a remote object with client-side proxy.

P86

1. Distributed object

RMI Compares to RPC:• Same:

– the steps of call is similar– separating interface from implementation

• Difference– procedure is a segment of program.– object is a variable of a data type, method is in

object.

Object reference compare to pointer. System provides

2. Object references Object references are similar to pointer, but not the same.

object

object reference

remote

point

local object

distributed obj

…

employee

promote() {…}dismiss() {… }

hire() {… }

Corp server

Sale server

salesman

bonus() {… }

advertise

cost() {… }

personnel

1. IP address

2.3.

?

Object servers

P152

personnel.hire( )

Information object reference contains

• network address of the machine • server id or endpoint (i.e.port number)• object id

P89

• May use location server

locationserver

client server

12

3

4

Server can move

• Server id instead of server’s endpoint. A daemon listens to a well-known endpoint and knows every server.

3. Implementation of object references

server1

server2keep a table

daemon

4. Several remote object invocation

1) CORBA

Open, heterogeneous environment

2) DCOM

Microsoft world

Object Object

C++ JavaCfunc P( )

How introduce object?

chapter 9

3) Java RMI

• Java environment • Distributed objects have been integrated into the

language

Summary

Addressing: name

Method: synchronous, transient

3.4 Message-oriented communication

1. Message-oriented transient communication 1) RPC and RMI

advantages: hide communication; message-orienteddisadvantage: synchronous; both need to be up

2) Recall socket P105

addressing: IP address, portforms: synchronous/asynchronous, transientshortage: low-level, byte-oriented/TCP-IP

P99

Process0

Process1

Process5Process4

Process3

Process2

Process1

Process0

Process2

3) MPI P107

JupiterWorld

designed for parallel applications; offer efficient

communication primitives; for high-speed nets.

communicator

• Addressing (groupID, processID)

• Forms support most transient communication forms

? A B; email

Need for various communication forms• Synchronous/asynchronous• Transient/persistent

100

Various combinations: P103

Why list so many forms? Where to use?

For efficiency.

(a) email

(c) A B

(e) send recv

(f) RPC

100

(persistent, asynchronous)

(transient, asynchronous)

(transient, synchronous)

(transient, synchronous/asyn)

2. Message-oriented persistent communication

1) Message-queuing model

Addressing: name of destination queue

Method: persistent, asynchronous

2) Basic idea• Application inserts messages in specific queue • the messages are forwarded over a series of

communication servers• eventually the messages are delivered to destination

P109

Message-Queuing System

The general organization of a message-queuing system with routers.

2-29

P112

Four combinations for loosely-coupled communications using queues.

2-26

P110

3) Basic interface to a queue

Install a handler to be called when a message is put into the specified queue.

Notify

Check a specified queue for messages, and remove the first. Never block.Poll

Block until the specified queue is nonempty, and remove the first messageGet

Append a message to a specified queuePut

MeaningPrimitive

callback

P110

4) Implementation

All machines in message-queuing system should have a queue manager which manages:

• queues: source queue, destination queue (name)• transmission of messages from S to D• mapping queues to network locations

Two kinds of implementations

a) Create its own forwarding servers (routers), like multicast gateway + Mbone

b) with the aid of Internet routers, like email

a) Architecture of IBM MQSeries

For IBM mainframes, access large scale database

Addressing: (destQM, destQ),

Example: IBM MQSeries

General organization of IBM's MQSeries message-queuing system.

2-31

P116

Message Transfer

The general organization of an MQSeries queuing network using routing tables and aliases.

(QA,SQ1)

Why do message-queue systems send messages to

queue, instead of directly to receivers?

Some applications need this kind of communication

form. For example, tasks are sent to a set of processes

who perform these tasks.

Two ways: • a dispatcher + workers• workers

P143

3.5 Stream-oriented communication

1. Requirements• Real life

– audio stream (simple stream)– movie (complex stream)

• Features– time-dependent information.

– data stream

Stream media?

P119

Addressing: <IP address, port> or

destination name

Method: continuous data stream

• Requirement for transmission:– timing for continuous data stream– synchronization for complex stream

Ex, – Samples in audio stream are played at intervals

of 1/44100 sec.– Video and audio sub-streams need synchronize.

Data Stream

Setting up a stream between two processes across a network.

P122

MP3

An example of multicasting a stream to several receivers.

Differentiated services

4. Stream Synchronization

• Meaning

Maintain temporal relation between streams which may be continuous or discrete data stream.

• Essence

Synchronize at the level of data units.

What is a data unit?

a single audio stream a series of samples

a video stream a series of frames

Data unit

Ex1, synchronization for stereo audio stream takesplace every 1/44100 sec

a data unit = a sample

Ex2, what about synchronization between an audio stream and a video stream?

video frame 30Hz

audio sample 44100Hz

sample/frame 44100/30=1470

an audio data unit = 1470 samples

1/30 sec

• Synchronization mechanism

A process executes read (write) operations on several simple streams, ensuring timing and synchronization.

Ex, alternating between reading an image and reading a block of audio samples every 33ms

sender receiver

audio streamvideo stream

• Does sender send two separate streams or a complex stream?

A complex stream. • Where synchronization should take place, sending

or receiving side?

Sender: multiplex different streams into a single stream containing all data units, i

ncluding those for synchronization(timestamp)

Receiver: demultiplexes the stream, using the timestamps of each packet

Synchronization Mechanisms (2)

The principle of synchronization as supported by high-level interfaces.

2-41

Offer interface to

specify the rates

Summary

1. Key issues

for designing communication mechanism• Addressing: how indicate sender and receiver,

LAN, WAN• Methods : synchronous/asynchronous

persistent/transient

stream-oriented communication• Middleware

Examples?

• How does it work?

• Issues to be handle: type, reference (object?)

• IDL: is used to write server’ specification C_stub

• Steps for using RPC S_stub

• Binder

3. RMI• Difference between RPC and RMI

• Object reference and its features

2. RPC

– address, serverID, objectID• CORBA etc

4. Message-oriented communication• Why need it?• Transient communication: MPI• Persistent communication: Message-queuing system

• Requirement for transmission• Features• Synchronization

5. Stream-oriented communication