Post on 19-Dec-2015
Lecture 3Page 1CS 239, Spring 2001
Interprocess Communications in Distributed Operating Systems
CS 239Distributed Operating Systems
April 9, 2001
Lecture 3Page 2CS 239, Spring 2001
Introduction
• Message passing communications
– Message passing paradigms
– Pipes and sockets
– Multicast and broadcast
• Request/reply communications
– RPC
– Secure RPC
Lecture 3Page 3CS 239, Spring 2001
Message Passing Communications
• By definition, distributed systems require IPC
• Most distributed systems connected only by network
• So most communications must be by message
– Other IPC abstractions built on messages
Lecture 3Page 4CS 239, Spring 2001
Levels of Abstraction of Communication
Interprocess Communication
Network OS
CommunicationsNetwork
Transactions
Request/Reply (RPC)
Message Passing
Transport Connection
Packet Switching
Lecture 3Page 5CS 239, Spring 2001
Basics of Messages
• Messages are collections of data objects– Header– Payload
• At message level, sender and receiver regarded as peers– Neither is in complete control
Lecture 3Page 6CS 239, Spring 2001
Steps in Sending a Message
• Sending process formats message• Invokes local OS to send it• Local OS (usually) copies it into kernel
space• Local OS transfers message to local
network interface• Network interface passes message to
network hardware
Lecture 3Page 7CS 239, Spring 2001
Steps in Receiving a Message
• Message delivered from network to receiver’s network interface
• Interface generates interrupt for OS• OS copies message to buffer • OS either interrupts receiver or waits for
receive system call• Message (usually) copied from OS buffer to
receiver process
Lecture 3Page 8CS 239, Spring 2001
Moving the Message
OS OS
Sender Receiver
2. Copy message into OS buffer3. Put the message in the network interface1. Create the message at the sender
send()
5. Transfer message to network interface6. Transfer to OS buffer
receive()
7. Transfer the message to receiver process4. Transfer message over network
Lecture 3Page 9CS 239, Spring 2001
Basic Message Primitives
• send(destination, message)
– Transfer contents of message to location specified by destination
• receive(source,message)
– Accept a message from source and put it in location specified by message
Lecture 3Page 10CS 239, Spring 2001
What is a Source/Destination?
• Complete specification of identity of a remote process
• Typically, a process name
– Often including a machine name
• Often with some specification of an internal address within the process
Lecture 3Page 11CS 239, Spring 2001
Common Source/Destination Naming Options
• Process name
• Link
• Mailbox
• Port
Lecture 3Page 12CS 239, Spring 2001
Process Name
• Varies depending on system specifics• Possible to have global, transparent
process namespace• More commonly, specify hosting node• Is this a good idea if processes move?• If only component of name, all
messages to a process share one queue
Lecture 3Page 13CS 239, Spring 2001
Symmetric Process Naming
A B
send(B,message1) receive(A,&message)
send(A,message)receive(B,&message)
Lecture 3Page 14CS 239, Spring 2001
Assymmetry in Naming
• Sender assumed to know receiver’s address
• Sometimes, receivers don’t know sender’s address ahead of time
– E.g., server handling multiple clients
• Sometimes receive() fills source into variable, rather than specifying value
Lecture 3Page 15CS 239, Spring 2001
Asymmetric Process Naming
A
B
C
send(C,message)
receive(&sender,&message)
send(C,message)
receive(&sender,&message)
Lecture 3Page 16CS 239, Spring 2001
Source/Destination Naming by Links
• Sometimes we want multiple communications paths between pairs of processes– Connections
• We can name these connections as sources and destinations– Links
Lecture 3Page 17CS 239, Spring 2001
Characteristics of Links
• Unidirectional• Multiple links allowed between two
processes• Each link independently nameable • Similar to multiple process entry points• Dynamically created and destroyed• Direct communications between peers
Lecture 3Page 18CS 239, Spring 2001
Links
A Blink1
link2link3
send(link1,message)create_link(B)create_link(B) create_link(A)
send(link2,message) send(link3,message)
Lecture 3Page 19CS 239, Spring 2001
Source/Destination Naming by Mailboxes
• Links aren’t appropriate for all cases• Sometimes sender doesn’t care about
identity of receiver– E.g., obtaining service from one of pool
of servers• Or receiver doesn’t care about identity of
sender• Mailboxes don’t require matching one
sender to one receiver
Lecture 3Page 20CS 239, Spring 2001
Mailbox Characteristics
• Multiple senders can put messages into the same mailbox
• Multiple receivers can retrieve messages from the same mailbox
• Varying semantics about how each of these works
Lecture 3Page 21CS 239, Spring 2001
Mailboxes
Mailbox
send(mailbox,message)
receive(mailbox,&message)
send(mailbox,message)
receive(mailbox,&message)
Lecture 3Page 22CS 239, Spring 2001
Message Synchronization and Buffering
• Sending and receiving a message requires some synchronization– When is it OK to send?– When is it OK to receive?– When is the process complete?– When can the system free various
buffers?
Lecture 3Page 23CS 239, Spring 2001
Blocking and Non-blocking Send and Receiver
• Blocking send blocks the sender
• Blocking receive blocks the receiver
• For how long?
– Some options here
• Non-blocking versions allow send and receive to go ahead immediately
Lecture 3Page 24CS 239, Spring 2001
Blocking at the Receiver
• In most systems, receiver blocks• Must wait for message to arrive before
proceeding• Sometimes, only blocks from moment of
message arrival until message is in receiver buffer
• Sometimes from point when receiver needs message
Lecture 3Page 27CS 239, Spring 2001
Common Options for Send Blocking
• Non-blocking send• Blocking send• Reliable blocking send• Explicit blocking send• Request and reply
Lecture 3Page 33CS 239, Spring 2001
Buffering
• Any delay in duration of blocking implies a buffering capability– In the sending kernel– In the network– In the receiving kernel
• Buffering allows multiple simultaneous sends– Also provides reliability
Lecture 3Page 34CS 239, Spring 2001
Pipes and Sockets
• A common OS features providing message communications
• The interface between applications and the network
• Hides many details of the network
Lecture 3Page 35CS 239, Spring 2001
Pipes
• Originally built as IPC for single processor system
• Basically a FIFO byte stream buffer
• Bytes fed in one end
• Bytes read out the other end
• In the same order
Lecture 3Page 36CS 239, Spring 2001
Pipe Details
• Once set up, treated like a file• Typically, one process writes, another reads
– Usually parent and child processes• Send is non-blocking
– Assuming local buffer space available• Receive is blocking
– Receiver explicitly asks to receive bytes – And waits for them to arrive
Lecture 3Page 37CS 239, Spring 2001
Pipes, Buffering, and Blocking
• Pipes are implemented in the kernel by setting up a buffer– Of fixed size
• When writer fills the buffer, further writes block
• When reader asks for bytes from empty buffer, reads block
Lecture 3Page 38CS 239, Spring 2001
Named Pipes
• Normal pipes only usable by processes that share file descriptors– Usually parents and children
• Pipes can have names attached– In file system namespace
• Allows any process to name either end of pipe
• Implemented with kernel buffers or file system
Lecture 3Page 39CS 239, Spring 2001
Pipes and Distributed Processing
• Pipe implementations assume a shared buffer space
• Named pipes assume a shared namespace– And shared FS or buffer space
• Generally, distributed systems might not have these
• Can we provide a similar mechanism for that environment?
Lecture 3Page 40CS 239, Spring 2001
Sockets
• A general mechanism for distributed systems IPC
• Subsumes pipes
• Sets up communication channel named by two unique endpoints
Lecture 3Page 41CS 239, Spring 2001
Creating Sockets
• Both communicating processes must issue socket() calls
• Creating a logical communications endpoint (LCE)– Local to creating process
• Must bind LCE to a physical communications endpoint (PCE)
Lecture 3Page 42CS 239, Spring 2001
Example of Creating Sockets
A BsA = socket(domain, type, protocol)
domain describes protocol family to usetype describes high-level characteristics of communicationsprotocol specifies which protocol in the family to use
sB = socket(domain, type, protocol)
Lecture 3Page 43CS 239, Spring 2001
Binding Sockets
• A PCE is a network host address plus a transport port pair– Network host address space is global– Transport port pair numbers
generated locally• The bind() system call performs the
bindings
Lecture 3Page 44CS 239, Spring 2001
Example of Binding Sockets
A Bbind(sA,name,namelen) bind(sB,name,namelen)
Now A can send a message to B using the socket
Lecture 3Page 45CS 239, Spring 2001
Connected Sockets
• Bound sockets can be used for connectionless message sending
• Sending process must know remote PCE, though
• Connecting sockets allows send calls without specifying remote PCE
• Typically, for servers with well-known ports
Lecture 3Page 46CS 239, Spring 2001
Broadcast and Multicast
• Methods for sending the same message to more than one destination
• Broadcast typically sends to everyone on the network
• Multicast typically sends to chosen set of recipients
Lecture 3Page 47CS 239, Spring 2001
Best Efforts Multicast
• Guarantees delivery to at least one member of the group
• Useful when trying to obtain service from set of interchangeable servers
Lecture 3Page 48CS 239, Spring 2001
Reliable Multicast
• Guarantees delivery to all servers– Or none of them
• Useful for distributing shared information to cooperating processes
• If several reliable multicasts going on simultaneously, participants may see messages in different orders
Lecture 3Page 49CS 239, Spring 2001
Handling Failures in Multicast
• Sender expects some feedback from each recipient– Perhaps explicit acknowledgement
• If not all received, re-multicast– Or send to just those who didn’t
acknowledge• Failure of sender harder to deal with
Lecture 3Page 50CS 239, Spring 2001
Multicast Message Ordering
• FIFO: Messages from single source delivered in same order as sent
• Causal order: Causally related messages from different sources delivered in causal order
• Total order: All messages from all sources delivered in single order
Lecture 3Page 51CS 239, Spring 2001
Achieving FIFO Ordering
• Sender keeps local sequence counter
• Each multicast stamped with incremented counter
• Each receiver keeps counter for each sender
• Only deliver messages with expected timestamp
– Queue or reject others
Lecture 3Page 52CS 239, Spring 2001
Achieving Causal Order
• Use what amounts to vector clocks– One element for each sender
• Sender increments local element of vector each time it multicasts– Attaching new vector to message
• Each receiver stores vector of last message it accepted
Lecture 3Page 53CS 239, Spring 2001
Receiving Causally Ordered Messages Properly
• Compare incoming message’s vector to stored vector
• If one element of incoming vector is 1 greater than stored vector, accept it
• Delay it if more than one element greater, or any element greater by more than 1
• Reject if vector is less than stored one
Lecture 3Page 54CS 239, Spring 2001
Example of Receiving Causally Ordered Messages
0 0 0
0 0 0
0 0 01 0 0
1 0 0
1 0 0A
B
C
Should C accept this message?
Yes! 1 = 0+1, 0 = 0, and 0 = 0
But B’s message is lost
1 0 1
1 0 1
Should B accept this message?No! 1 = 0+1, 0 = 0, and 1 = 0+1
B has missed a causal message
1 0 0
1 0 0
1 0 0
1 0 1