Message Passing .

• Message Passing

https://store.theartofservice.com/the-message-passing-toolkit.html

Remote procedure call - Message passing

1 An RPC is initiated by the client, which sends a request message to a known remote server to execute a specified procedure with supplied

parameters


Remote procedure call - Message passing

1 An important difference between remote procedure calls and local calls is that remote

calls can fail because of unpredictable network problems. Also, callers generally must deal with

such failures without knowing whether the remote procedure was actually invoked.

Idempotent procedures (those that have no additional effects if called more than once) are easily handled, but enough difficulties remain that code to call remote procedures is often

confined to carefully written low-level subsystems.


Message passing

1 Message passing in computer science is a form of communication used in concurrent

computing, parallel computing, object-oriented programming, and interprocess

communication. In this model, processes or objects can send and receive messages (comprising zero or more bytes, complex

data structures, or even segments of code) to other processes. By waiting for

messages, processes can also synchronize.


Message passing - Overview

1 Message passing is the paradigm of communication where messages are sent from a sender to one or more

recipients. Forms of messages include (remote) method invocation,

signals, and data packets. When designing a message passing system

several choices are made:



1 Whether messages are guaranteed to be delivered in

order



1 Whether messages are passed one-to-one (unicast), one-to-many

(multicast or broadcast), many-to-one (client–server), or many-to-many

(All-to-All).



1 Whether communication is synchronous or

asynchronous.



1 Examples of the latter include Microkernel operating systems that pass messages between one kernel and one or more server blocks, and the Message Passing Interface used

in high-performance computing.


Message passing - Message passing systems

1 Distributed object and remote method invocation systems like ONC

RPC, CORBA, Java RMI, DCOM, SOAP, .NET Remoting, CTOS, QNX

Neutrino RTOS, OpenBinder, D-Bus, Unison RTOS and similar are message passing systems.



1 Message passing systems have been called "shared nothing" systems because the message passing

abstraction hides underlying state changes that may be used in the

implementation of sending messages.



1 Message passing model based programming languages typically define messaging as the (usually asynchronous) sending (usually by copy) of a data item to a communication endpoint (Actor, process, thread, socket,

etc.). Such messaging is used in Web Services by SOAP. This concept is the higher-

level version of a datagram except that messages can be larger than a packet and can optionally be made reliable, durable,

secure, and/or transacted.



1 Messages are also commonly used in the same sense as a means of

interprocess communication; the other common technique being

streams or pipes, in which data are sent as a sequence of elementary

data items instead (the higher-level version of a virtual circuit).


Message passing - Synchronous versus asynchronous message passing

1 Synchronous message passing systems require the sender and receiver to wait for each other to transfer the message. That is, the sender will not continue until the

receiver has received the message.



1 Synchronous communication has two advantages. The first advantage is that

reasoning about the program can be simplified in that there is a

synchronisation point between sender and receiver on message transfer. The second advantage is that no buffering is required. The message can always be stored on the receiving side, because the sender will not

continue until the receiver is ready.



1 Asynchronous message passing systems deliver a message from

sender to receiver, without waiting for the receiver to be ready. The

advantage of asynchronous communication is that the sender

and receiver can overlap their computation because they do not

wait for each other.



1 Synchronous communication can be built on top of asynchronous

communication by using a so-called Synchronizer. For example, the α-

Synchronizer works by ensuring that the sender always waits for an

acknowledgement message from the receiver. The sender only sends the

next message after the acknowledgement has been

received.https://store.theartofservice.com/the-message-passing-toolkit.html


1 The buffer required in asynchronous communication can cause problems when it is full. A decision has to be

made whether to block the sender or whether to discard future messages. If the sender is blocked, it may lead

to an unexpected deadlock. If messages are dropped, then

communication is no longer reliable.


Message passing - Message passing versus calling

1 Message passing should be contrasted with the alternative communication method for passing information between programs –

the Call. In a traditional Call, arguments are passed to the "callee" (the receiver)

typically by one or more general purpose registers or in a parameter list containing the addresses of each of the arguments. This form of communication differs from message passing in at least three crucial

areas:https://store.theartofservice.com/the-message-passing-toolkit.html


1 In message passing, each of the arguments has to have sufficient available extra memory

for copying the existing argument into a portion of the new message. This applies

irrespective of the size of the original arguments – so if one of the arguments is

(say) an HTML string of 31,000 octets describing a web page (similar to the size of this article), it has to be copied in its entirety

(and perhaps even transmitted) to the receiving program (if not a local program).



1 Web browsers and web servers are examples of processes that communicate by message

passing



1 A subroutine call or method invocation will not exit until the

invoked computation has terminated. Asynchronous message passing, by contrast, can result in a response

arriving a significant time after the request message was sent.



1 A message handler will, in general, process messages from more than one sender. This

means its state can change for reasons unrelated to the behaviour of a single sender

or client process. This is in contrast to the typical behaviour of an object upon which methods are being invoked: the latter is expected to remain in the same state

between method invocations. In other words, the message handler behaves analogously

to a volatile object.


Message passing - Message passing and locks

1 Message passing can be used as a way of controlling access to resources in a concurrent or

asynchronous system. One of the main alternatives is mutual exclusion

or locking. Examples of resources include shared memory, a disk file or region thereof, a database table or

set of rows.



1 In locking, a resource is essentially shared, and processes wishing to

access it (or a sector of it) must first obtain a lock. Once the lock is acquired, other processes are

blocked out, ensuring that corruption from simultaneous writes does not

occur. After the process with the lock is finished with the resource, the lock

is then released.https://store.theartofservice.com/the-message-passing-toolkit.html


1 With the message-passing solution, it is assumed that the resource is not exposed, and all changes to it are made by an associated process, so that the resource is encapsulated


Message passing - Mathematical models

1 The prominent mathematical models of message passing are the Actor model and Pi

calculus.



1 In the terminology of some object-oriented programming languages, a message is the single means to pass

control to an object. If the object "responds" to the message, it has a method for that message. In pure

object-oriented programming, message passing is performed exclusively through a dynamic

dispatch strategy.https://store.theartofservice.com/the-message-passing-toolkit.html


1 Message passing enables extreme late binding in systems



1 Alan Kay has argued that message passing is more important than objects in OOP, and

that objects themselves are often over-emphasized. The live distributed objects

programming model builds upon this observation; it uses the concept of a

distributed data flow to characterize the behavior of a complex distributed system in terms of message patterns, using high-

level, functional-style specifications.


Message passing - Further reading

1 Ramachandran, U.; M. Solomon, M. Vernon (1987). "Hardware support for

interprocess communication". Proceedings of the 14th annual

international symposium on Computer architecture. ACM Press.



1 Chao, Linda (1987), "Architectural features of a message-driven

processor", Thesis (B.S.) Massachusetts Institute of

Technology, Dept. of Electrical Engineering and Computer Science



1 McQuillan, John M.; David C. Walden (1975). "Some considerations for a high performance message-based

interprocess communication system". Proceedings of the 1975 ACM

SIGCOMM/SIGOPS workshop on Interprocess communications. ACM

Press.


Computer cluster - Message passing and communication

1 Two widely used approaches for communication between cluster

nodes are MPI, the Message Passing Interface and PVM, the Parallel

Virtual Machine.



1 PVM was developed at the Oak Ridge National Laboratory around 1989

before MPI was available. PVM must be directly installed on every cluster node and provides a set of software

libraries that paint the node as a "parallel virtual machine". PVM

provides a run-time environment for message-passing, task and resource management, and fault notification. PVM can be used by user programs written in C, C++, or Fortran, etc.



1 MPI emerged in the early 1990s out of discussions among 40 organizations


Message Passing Interface - History

1 Out of that discussion came a Workshop on Standards for Message

Passing in a Distributed Memory Environment held on April 29–30,

1992 in Williamsburg, Virginia



1 The MPI effort involved about 80 people from 40 organizations, mainly

in the United States and Europe. Most of the major vendors of

concurrent computers were involved in MPI along with researchers from

universities, government laboratories, and industry.



1 The MPI standard defines the syntax and semantics of a core of library routines useful to a wide range of users writing portable message

passing programs in Fortran and C.



1 MPI provides a simple-to-use portable interface for the basic user, yet

powerful enough to allow programmers to use the high-performance message passing

operations available on advanced machines.



1 The message passing paradigm is attractive because of wide portability and can be used in communication for distributed-memory and shared-

memory multiprocessors, networks of workstations, and a combination of

these elements



1 Support for MPI meetings came in part from ARPA and US National

Science Foundation under grant ASC-9310330, NSF Science and

Technology Center Cooperative agreement number CCR-8809615,

and the Commission of the European Community through Esprit Project

P6643. The University of Tennessee also made financial contributions to

the MPI Forum.https://store.theartofservice.com/the-message-passing-toolkit.html

Message Passing Interface - Overview

1 MPI is a language-independent communications protocol used to program parallel computers. Both

point-to-point and collective communication are supported. MPI "is a message-passing application

programmer interface, together with protocol and semantic specifications for how its features must behave in

any implementation." MPI's goals are high performance, scalability, and

portability. MPI remains the dominant model used in high-performance

computing today.



1 MPI is not sanctioned by any major standards body; nevertheless, it has

become a de facto standard for communication among processes

that model a parallel program running on a distributed memory

system



1 Although MPI belongs in layers 5 and higher of the OSI Reference Model, implementations may cover most

layers, with sockets and Transmission Control Protocol (TCP) used in the

transport layer.



1 Most MPI implementations consist of a specific set of routines (i.e., an API) directly callable from C, C++, Fortran and any language able to interface

with such libraries, including C#, Java or Python. The advantages of MPI

over older message passing libraries are portability (because MPI has been

implemented for almost every distributed memory architecture) and speed (because each implementation

is in principle optimized for the hardware on which it runs).



1 MPI uses Language Independent Specifications (LIS) for calls and language bindings. The first MPI standard specified ANSI C and

Fortran-77 bindings together with the LIS. The draft was presented at

Supercomputing 1994 (November 1994) and finalized soon thereafter. About 128 functions constitute the

MPI-1.3 standard which was released as the final end of the MPI-1 series in

2008.



1 Object interoperability was also added to allow easier mixed-language message

passing programming



1 MPI-2 is mostly a superset of MPI-1, although some functions have been deprecated. MPI-1.3 programs still work under MPI implementations

compliant with the MPI-2 standard.



1 Threaded shared memory programming models (such as

Pthreads and OpenMP) and message passing programming (MPI/PVM) can

be considered as complementary programming approaches, and can

occasionally be seen together in applications, e.g


Message Passing Interface - Functionality

1 The MPI interface is meant to provide essential virtual topology,

synchronization, and communication functionality between a set of

processes (that have been mapped to nodes/servers/computer instances) in a language-

independent way, with language-specific syntax (bindings), plus a few

language-specific featureshttps://store.theartofservice.com/the-message-passing-toolkit.html


1 MPI library functions include, but are not limited to, point-to-point rendezvous-type send/receive

operations, choosing between a Cartesian or graph-like logical process topology, exchanging data

between process pairs (send/receive operations), combining partial results of computations (gather

and reduce operations), synchronizing nodes (barrier operation) as well as obtaining network-

related information such as the number of processes in the computing session, current processor identity that a process is mapped to, neighboring processes

accessible in a logical topology, and so on



1 MPI-1 and MPI-2 both enable implementations that overlap

communication and computation, but practice and theory differ. MPI also

specifies thread safe interfaces, which have cohesion and coupling strategies that help avoid hidden

state within the interface. It is relatively easy to write multithreaded

point-to-point MPI code, and some implementations support such code.

Multithreaded collective communication is best accomplished

with multiple copies of Communicators, as described below.


Message Passing Interface - Concepts

1 MPI provides a rich range of abilities. The following concepts help in

understanding and providing context for all of those abilities and help the

programmer to decide what functionality to use in their

application programs. Four of MPI's eight basic concepts are unique to

MPI-2.


Message Passing Interface - Communicator

1 Communicator objects connect groups of

processes in the MPI session



1 Communicators can be partitioned using several MPI commands. These commands include MPI_COMM_SPLIT,

where each process joins one of several colored sub-communicator by

declaring itself to have that color.


Message Passing Interface - Point-to-point basics

1 A number of important MPI functions involve communication between two specific

processes



1 MPI-1 specifies mechanisms for both blocking and non-blocking point-to-point communication mechanisms, as well as the so-called 'ready-send' mechanism whereby a send request

can be made only when the matching receive request has

already been made.


Message Passing Interface - Collective basics

1 Collective functions involve communication among all processes in a process group (which can mean the entire process pool or a program-

defined subset)



1 Other operations perform more sophisticated tasks, such as

MPI_Alltoall which rearranges n items of data processor such that the nth node gets the nth item of data from

each.


Message Passing Interface - Derived datatypes

1 Many MPI functions require that you specify the type of data which is sent between processors. This is because these functions pass variables, not defined types. If the data type is a

standard one, such as int, char, double, etc., you can use predefined

MPI datatypes such as MPI_INT, MPI_CHAR, MPI_DOUBLE.



1 However, you may instead wish to send data as one block as opposed to

100 ints. To do this define a "contiguous block" derived data type.



1 Passing a class or a data structure cannot use a predefined data type. MPI_Type_create_struct creates an

MPI derived data type from MPI_predefined data types, as

follows:



1 int MPI_Type_create_struct(int count, int blocklen[],

MPI_Aint disp[],



1 where count is a number of blocks, also number of entries in blocklen[], disp[], and

type[]:



1 blocklen[] — number of elements in each block (array

of integer)



1 The disp[] array is needed because processors require the variables to

be aligned a specific way on the memory



1 Given the following data structures:



1 Here's the C code for building an MPI-derived

data type:



1 //The first and last elements mark the beg

and end of data structure



1 MPI_Datatype type={MPI_LB,

MPI_INT, MPI_SHORT, MPI_INT, MPI_INT,

MPI_UB};https://store.theartofservice.com/the-message-passing-toolkit.html


1 //You need an array to establish the upper bound of

the data structure



1 error=MPI_Type_create_s

truct(6, blocklen, disp, type, &newtype);


Message Passing Interface - One-sided communication

1 MPI-2 defines three one-sided communications operations, Put, Get,

and Accumulate, being a write to remote memory, a read from remote memory, and a reduction operation

on the same memory across a number of tasks. Also defined are

three different methods to synchronize this communication

(global, pairwise, and remote locks) as the specification does not

guarantee that these operations have taken place until a synchronization point.


Message Passing Interface - One-sided communication

1 These types of call can often be useful for algorithms in which

synchronization would be inconvenient (e.g. distributed matrix

multiplication), or where it is desirable for tasks to be able to balance their load while other

processors are operating on data.


Message Passing Interface - Dynamic process management

1 The key aspect is "the ability of an MPI process to participate in the

creation of new MPI processes or to establish communication with MPI processes that have been started

separately." The MPI-2 specification describes three main interfaces by

which MPI processes can dynamically establish communications,

MPI_Comm_spawn, MPI_Comm_accept/MPI_Comm_conne

ct and MPI_Comm_join


Message Passing Interface - I/O

1 The parallel I/O feature is sometimes called MPI-IO, and refers to a set of functions designed to abstract I/O

management on distributed systems to MPI, and allow files to be easily accessed in a patterned way using

the existing derived datatype functionality.


Message Passing Interface - I/O

1 The little research that has been done on this feature indicates the

difficulty for good performance. For example, some implementations of sparse matrix-vector multiplications

using the MPI I/O library are disastrously inefficient.


Message Passing Interface - 'Classical' cluster and supercomputer implementations

1 The MPI implementation language is not constrained to match the

language or languages it seeks to support at runtime. Most

implementations combine C, C++ and assembly language, and target C, C++, and Fortran programmers.

Bindings are available for many other languages, including Perl, Python, R,

Ruby, Java, CL.https://store.theartofservice.com/the-message-passing-toolkit.html

Message Passing Interface - 'Classical' cluster and supercomputer implementations

1 The initial implementation of the MPI 1.x standard was MPICH, from

Argonne National Laboratory (ANL) and Mississippi State University


Message Passing Interface - Python

1 MPI Python implementations include: pyMPI, mpi4py, pypar, MYMPI, and

the MPI submodule in ScientificPython. pyMPI is notable

because it is a variant python interpreter, while pypar, MYMPI, and ScientificPython's module are import modules. They make it the coder's

job to decide where the call to MPI_Init belongs. Recently the well

known Boost C++ Libraries acquired Boost:MPI which included the MPI

Python Bindings. This is of particular help for mixing C++ and Python.


Message Passing Interface - OCaml

1 The OCamlMPI Module implements a large subset of MPI functions and is in active use in scientific computing.

An eleven thousand line OCaml program was "MPI-ified" using the

module, with an additional 500 lines of code and slight restructuring and ran with excellent results on up to

170 nodes in a supercomputer.


Message Passing Interface - Java

1 Although Java does not have an official MPI binding, several groups

attempt to bridge the two, with different degrees of success and

compatibility. One of the first attempts was Bryan Carpenter's mpiJava, essentially a set of Java

Native Interface (JNI) wrappers to a local C MPI library, resulting in a

hybrid implementation with limited portability, which also has to be

compiled against the specific MPI library being used.



1 Beyond the API, Java MPI libraries can be either dependent on a local

MPI library, or implement the message passing functions in Java,

while some like P2P-MPI also provide peer-to-peer functionality and allow

mixed platform operation.



1 Some of the most challenging parts of Java/MPI arise from Java

characteristics such as the lack of explicit pointers and the linear memory address space for its

objects, which make transferring multidimensional arrays and complex

objects inefficient



1 Another Java message passing

system is MPJ Express


Message Passing Interface - Matlab

1 There are a few academic implementations of MPI using Matlab.

Matlab has their own parallel extension library implemented using

MPI and PVM


Message Passing Interface - R

1 MPI R implementations include: Rmpi and pbdMPI where Rmpi focuses on manager-workers parallelism while

pbdMPI focuses on SPMD parallelism. Both implementations fully support

Open MPI or MPICH2.


Message Passing Interface - Common Language Infrastructure

1 The two managed Common Language Infrastructure (CLI) .NET implementations are Pure Mpi.NET and MPI.NET, a research effort at

Indiana University licensed under a BSD-style license. It is compatible

with Mono, and can make full use of underlying low-latency MPI network

fabrics.


Message Passing Interface - Hardware implementations

1 MPI hardware research focuses on implementing MPI directly in

hardware, for example via processor-in-memory, building MPI operations into the microcircuitry of the RAM chips in each node. By implication, this approach is independent of the language, OS or CPU, but cannot be

readily updated or removed.


Message Passing Interface - Hardware implementations

1 Another approach has been to add hardware acceleration to one or

more parts of the operation, including hardware processing of MPI queues and using RDMA to directly transfer data between memory and

the network interface without CPU or OS kernel intervention.


Message Passing Interface - Example program

1 Here is a "Hello World" program in MPI written in C. In this example, we

send a "hello" message to each processor, manipulate it trivially,

return the results to the main process, and print the messages.



1 /* At this point, all programs are running equivalently, the

rank



1 distinguishes the roles of the programs in the SPMD

model, with



1 rank 0 often used specially... */



1 printf("%d: We have %d processors\n", myid,

numprocs);



1 MPI_Send(buff, BUFSIZE, MPI_CHAR, i, TAG,

MPI_COMM_WORLD);



1 MPI_Recv(buff, BUFSIZE, MPI_CHAR, i, TAG,

MPI_COMM_WORLD, &stat);



1 strncat(buff, "reporting for duty\

n", BUFSIZE-1);



1 The runtime environment for the MPI implementation used (often called

mpirun or mpiexec) spawns multiple copies of the program, with the total

number of copies determining the number of process ranks in

MPI_COMM_WORLD, which is an opaque descriptor for communication

between the set of processes



1 MPI uses the notion of process rather than processor


Message Passing Interface - MPI-2 adoption

1 Adoption of MPI-1.2 has been universal, particularly in cluster

computing, but acceptance of MPI-2.1 has been more limited. Issues

include:



1 MPI-2 implementations include I/O and dynamic process management,

and the size of the middleware is substantially larger. Most sites that

use batch scheduling systems cannot support dynamic process

management. MPI-2's parallel I/O is well accepted.



1 Many MPI-1.2 programs were developed before MPI-2. Portability concerns initially slowed, although wider support has lessened this.



1 Many MPI-1.2 applications use only a subset of that standard (16-25

functions) with no real need for MPI-2 functionality.


Message Passing Interface - Future

1 Some aspects of MPI's future appear solid; others less so. The MPI Forum reconvened in 2007, to clarify some

MPI-2 issues and explore developments for a possible MPI-3.



1 Like Fortran, MPI is ubiquitous in technical computing, and it is taught and used widely.



1 Architectures are changing, with greater internal concurrency (multi-core), better fine-grain concurrency

control (threading, affinity), and more levels of memory hierarchy



1 Improved fault tolerance within MPI would have clear benefits for the growing trend of grid computing.


Message Passing Interface - Notes

1 Jump up ^ High-performance and scalable MPI over InfiniBand with reduced memory

usage



1 Jump up ^ Sparse matrix-vector

multiplications using the MPI I/O library



1 Jump up ^ OCamlMPI Module



1 Jump up ^ Yu, H. (2002). "Rmpi: Parallel Statistical Computing in R". R News.



1 Jump up ^ Chen, W.-C., Ostrouchov, G., Schmidt, D., Patel, P., and Yu, H. (2012). "pbdMPI: Programming with

Big Data -- Interface to MPI".



1 Jump up ^ Using OpenMPI, compiled with gcc -g -v

-I/usr/lib/openmpi/include/ -L/usr/lib/openmpi/include/

wiki_mpi_example.c -lmpi and run with mpirun -np 2 ./a.out.


Parallel programming model - Message passing

1 In a message passing model, parallel tasks exchange data through passing

messages to one another. These communications can be

asynchronous or synchronous. The Communicating Sequential Processes

(CSP) formalisation of message-passing employed communication

channels to 'connect' processes, and led to a number of important

languages such as Joyce, occam and Erlang.


Supercomputer - Software tools and message passing

1 The parallel architectures of supercomputers often dictate the use of special programming techniques

to exploit their speed. Software tools for distributed processing include

standard APIs such as MPI and PVM, VTL, and open source-based software

solutions such as Beowulf.



1 In the most common scenario, environments such as PVM and MPI for loosely connected clusters and

OpenMP for tightly coordinated shared memory machines are used.

Significant effort is required to optimize an algorithm for the

interconnect characteristics of the machine it will be run on; the aim is

to prevent any of the CPUs from wasting time waiting on data from

other nodes. GPGPUs have hundreds of processor cores and are

programmed using programming models such as CUDA.



1 Moreover, it is quite difficult to debug and test parallel programs. Special

techniques need to be used for testing and debugging such

applications.


High-performance computing - Software tools and message passing

1 The parallel architectures of supercomputers often dictate the use of special programming techniques

to exploit their speed. Software tools for distributed processing include

standard Application programming interface|APIs such as Message

Passing Interface|MPI and Parallel Virtual Machine|PVM, Virtual tape

library|VTL, and open source-based software solutions such as Beowulf

(computing)|Beowulf.



1 In the most common scenario, environments such as Parallel Virtual Machine|PVM and Message Passing Interface|MPI for loosely connected

clusters and OpenMP for tightly coordinated shared memory

machines are used



1 Moreover, it is quite difficult to debug and test parallel programs. Testing

high-performance computing applications|Special techniques need to be used for testing and debugging

such applications.


Message Passing Interface

1 'Message Passing Interface' ('MPI') is a standardized and portable

message-passing system designed by a group of researchers from

academia and industry to function on a wide variety of parallel computers



1 Support for MPI meetings came in part from DARPA|ARPA and US

National Science Foundation under grant ASC-9310330, NSF Science and

Technology Center Cooperative agreement number CCR-8809615,

and the Commission of the European Community through Esprit Project

P6643. The University of Tennessee also made financial contributions to

the MPI Forum.https://store.theartofservice.com/the-message-passing-toolkit.html


1 MPI is a language-independent communications protocol used to program parallel computers. Both

point-to-point and collective communication are supported. MPI is

a message-passing application programmer interface, together with protocol and semantic specifications for how its features must behave in

any implementation. MPI's goals are high performance, scalability, and

portability. MPI remains the dominant model used in high-performance computing today.



1 [http://portal.acm.org/citation.cfm?id=1188565 High-performance and scalable MPI over InfiniBand with

reduced memory usage]



1 MPI is not sanctioned by any major standards body; nevertheless, it has become a de facto standardization|standard for communication among

processes that model a parallel programming|parallel program

running on a distributed memory system



1 The advantages of MPI over older message passing libraries are

portability (because MPI has been implemented for almost every

distributed memory architecture) and speed (because each implementation

is in principle optimized for the hardware on which it runs).



1 MPI uses Language Independent Specifications (LIS) for calls and language bindings. The first MPI standard specified ANSI C and

Fortran-77 bindings together with the LIS. The draft was presented at



1 Supercomputing 1994 (November 1994)[http://hpc.sagepub.com/conte

nt/8/3-4.toc] and finalized soon thereafter. About 128 functions constitute the MPI-1.3 standard

which was released as the final end of the MPI-1 series in

2008.[http://www.mpi-forum.org/docs/]



1 MPI library functions include, but are not limited to, point-to-point

rendezvous-type send/receive operations, choosing between a

Cartesian tree|Cartesian or Graph (data structure)|graph-like logical

process topology, exchanging data between process pairs (send/receive operations), combining partial results of computations (gather and reduce

operations), synchronizing nodes (barrier operation) as well as

obtaining network-related information such as the number of

processes in the computing session, current processor identity that a

process is mapped to, neighboring processes accessible in a logical

topology, and so on



1 MPI-1 and MPI-2 both enable implementations that overlap

communication and computation, but practice and theory differ



1 Communicator objects connect

groups of processes in the MPI session



1 MPI-1 specifies mechanisms for both blocking (computing)|blocking and

non-blocking point-to-point communication mechanisms, as well

as the so-called 'ready-send' mechanism whereby a send request

can be made only when the matching receive request has

already been made.



1 Collective operation|Collective functions involve communication among all processes in a process group (which can mean the entire process pool or a program-defined

subset)



1 * blocklen[] — number of elements in each block (array

of integer)



1 The disp[] array is needed because processors require the variables to

be aligned a specific way on the memory


Message Passing Interface - if(myid

1 Each process has its own rank, the total number of processes in the

world, and the ability to communicate between them either with point-to-point (send/receive) communication, or by collective

communication among the group


Message Passing Interface - if(myid

1 MPI uses the notion of process rather than processor



1 # MPI-2 implementations include I/O and dynamic process management,

and the size of the middleware is substantially larger. Most sites that

use batch scheduling systems cannot support dynamic process

management. MPI-2's parallel I/O is well accepted.



1 # Many MPI-1.2 programs were developed before MPI-2. Portability concerns initially slowed, although wider support has lessened this.



1 # Many MPI-1.2 applications use only a subset of that standard (16-25

functions) with no real need for MPI-2 functionality.



1 Some aspects of MPI's future appear solid; others less so. The MPI Forum reconvened in 2007, to clarify some

MPI-2 issues and explore developments for a possible MPI-3.



1 Architectures are changing, with greater internal concurrency (multi-core), better fine-grain concurrency

control (threading, affinity), and more levels of memory hierarchy


Computer clusters - Message passing and communication

1 Two widely used approaches for communication between cluster

nodes are MPI, the Message Passing Interface and PVM, the Parallel

Virtual Machine.Distributed services with OpenAFS: for enterprise and

education by Franco Milicchio, Wolfgang Alexander Gehrke 2007,

ISBN pages 339-341 [http://books.google.it/books?

id=bKf4NBaIJI8Cpg=PA339dq=%22message+passing

%22+computer+cluster+MPI+PVMhl=ensa=Xei=-

dD7ToZCj_bhBOOxvI0Iredir_esc=y#v=onepageq=%22message

%20passing%22%20computer%20cluster%20MPI%20PVMf=false]


Computer clusters - Message passing and communication

1 MPI emerged in the early 1990s out of discussions among 40 organizations


Real-time operating system - Message passing

1 The other approach to resource sharing is for tasks to send messages

in an organized message passing scheme


Goto - Message passing

1 One of the main alternatives is message passing, which is of

particular importance in concurrent computing, interprocess

communication, and object oriented programming


SIMPL - Advantages of Synchronized Message Passing

1 * Inherent thread synchronization coordinates the execution of communicating programs


SIMPL - Advantages of Synchronized Message Passing

1 * No data buffering is required


Multi-processor - Message passing

1 processors communicate via message passing.


Multi-processor - Message passing

1 focus attention on costly non-local

operations.


Event loop - Message passing

1 The event loop is a specific implementation technique of systems that use message

passing.


Message (computer science) - Synchronous versus asynchronous message passing

1 One of the most important distinctions among message passing

systems is whether they use synchronous or asynchronous

message passing. Synchronous message passing occurs between

objects that are running at the same time. With asynchronous message

passing it is possible for the receiving object to be busy or not running

when the requesting object sends the message.



1 Synchronous message passing is what typical object-oriented

programming languages such as Java and Smalltalk use. Asynchronous

message passing requires additional capabilities for storing and

retransmitting data for systems that may not run concurrently.



1 For example, if synchronous message passing would be used exclusively, large, distributed systems generally would not perform well enough to be

usable



1 Imagine a busy business office having 100 desktop computers that

send emails to each other using synchronous message passing exclusively. Because the office

system does not use asynchronous message passing, one worker turning

off their computer can cause the other 99 computers to freeze until

the worker turns their computer back on to process a single email.



1 Asynchronous message passing is generally implemented so that all the

complexities that naturally occur when trying to synchronize systems

and data are handled by an intermediary level of software.

Commercial vendors who develop software products to support these

intermediate levels usually call their software middleware. One of the

most common types of middleware to support asynchronous messaging

is called Message-oriented middleware|Message Oriented

Middleware (MOM)



1 Asynchronous message passing simply sends the message to the message

bus


Message (computer science) - Message passing versus calling

1 Distributed or asynchronous message passing has some overhead

associated with it compared to the simpler way of simply calling a

procedure. In a traditional Calling convention|procedure call,

arguments are passed to the receiver typically by one or more general

purpose registers or in a Parameter (computer science)|parameter list

containing the addresses of each of the arguments. This form of

communication differs from message passing in at least three crucial

areas:



1 In message passing, each of the arguments has to copy the existing argument into a portion of the new message. This applies regardless of

the size of the argument and in some cases the arguments can be as large

as a document which can be megabytes worth of data. The

argument has to be copied in its entirety and transmitted to the

receiving object.https://store.theartofservice.com/the-message-passing-toolkit.html


1 By contrast, for a standard procedure call, only an address (a few bits)

needs to be passed for each argument and may even be passed

in a general purpose register requiring zero additional storage and

zero transfer time.



1 Web web browsing|browsers and web servers are examples of processes

that communicate by message passing



1 A subroutine call or method (computer programming)|method invocation will not exit until the

invoked computation has terminated. Asynchronous message passing, by contrast, can result in a response

arriving a significant time after the request message was sent.


Message passing (disambiguation)

1 * Message passing, a mechanism for

interprocess communication


Message passing (disambiguation)

1 * Message passing algorithms for probabilistic inference, such as Belief propagation or variational message

passing.


Message passing in computer clusters

1 Beowulf Cluster Computing With Windows by Thomas Lawrence

Sterling 2001 ISBN 0262692759 MIT Press pages 7–9 Message passing in

computer clusters built with commodity Server (computing)|

servers and Network switch|switches is used by virtually every internet

service.Computer Organization and Design by David Patterson (scientist)|

David Ahttps://store.theartofservice.com/the-message-passing-toolkit.html


1 As the number of nodes in a cluster increases, the rapid growth in the complexity of the communication

subsystem makes message passing delays over the Switched fabric|

interconnect a serious performance issue in the execution of parallel programs.Recent Advances in the

Message Passing Interface by Yiannis Cotronis, Anthony Danalis, Dimitris

Nikolopoulos and Jack Dongarra 2011 ISBN 3642244483 pages 160–162



1 Before a large computer cluster is assembled, a trace-based simulator can use a small number of nodes to

help predict the performance of message passing on larger

configurations


Message passing in computer clusters - Approaches to message passing

1 Historically, the two typical approaches to communication

between cluster nodes have been PVM, the Parallel Virtual Machine and

MPI, the Message Passing Interface.Distributed services with

OpenAFS: for enterprise and education by Franco Milicchio,

Wolfgang Alexander Gehrke 2007, ISBN pages 339-341 However, MPI has now emerged as the de facto standard for message passing on

computer clusters.Recent Advances in Parallel Virtual Machine and

Message Passing Interface by Matti Ropo, Jan Westerholm and Jack

Dongarra 2009 ISBN 3642037690 page 231



1 PVM predates MPI and was developed at the Oak Ridge National Laboratory around 1989. It provides

a set of software libraries that allow a computing node to act as a parallel virtual machine. It provides run-time environment for message-passing,

task and resource management, and fault notification and must be directly installed on every cluster node. PVM

can be used by user programs written in C language|C, C++, or

Fortran, etc.



1 Unlike PVM, which has a concrete implementation, MPI is a

specification rather than a specific set of libraries


Message passing in computer clusters - Testing, evaluation and optimization

1 Tiahnhe-I uses over two thousand FeiTeng-1000 processors to enhance

the operation of its proprietary message passing system, while

computations are performed by Xeon and Nvidia Tesla processors.The TianHe-1A Supercomputer: Its

Hardware and Software by Xue-Jun Yang, Xiang-Ke Liao, et al in the Journal of Computer Science and

Technology, Volume 26, Number 3, May 2011, pages

344–351[http://jcst.ict.ac.cn:8080/jcst/EN/article/downloadArticleFile.do?

attachType=PDFid=9388]U.S



1 One approach to reducing communication overhead is the use of local neighborhoods (also called

Locale (computer hardware)|locales) for specific tasks



1 Given that MPI has now emerged as the de facto standard on computer

clusters, the increase in the number of cluster nodes has resulted in

continued research to improve the efficiency and scalability of MPI

libraries. These efforts have included research to reduce the memory

footprint of MPI libraries.



1 This is achieved by registering callbacks with Peruse, and then

invoking them as triggers as message events take place.Recent

Advances in Parallel Virtual Machine and Message Passing Interface by Bernd Mohr, Jesper Larsson Träff,

Joachim Worringen and Jack Dongarra 2006 ISBN 354039110X

page 347 Peruse can work with the PARAVER visualization system



1 PARAVER: A Tool to Visualize and Analyze Parallel Code by Vincent

Pillet et al, Proceedings of the conference on Transputer and Occam

Developments, 1995, pages 17–31 PARAVER may use trace formats from

other systems, or perform its own tracing


Message passing in computer clusters - Performance analysis

1 Systems such as BIGSIM provide these facilities by allowing the

simulation of performance on various network topology|node topologies, message passing and scheduling strategies.Petascale Computing:

Algorithms and Applications by David A


Message passing in computer clusters - Analytical approaches

1 A well knowm model is Hockney's model which simply relies on point to point communication, using T = L + (M / R) where M is the message size, L is the startup latency and R is the

asymptotic bandwidth in MB/s.Modeling Message Passing

Overhead by C.Y Chou et al


Message passing in computer clusters - Analytical approaches

1 Xu and Hwang generalized Hockney's model to include the number of processors, so that both the latency and the asymptotic bandwidth are functions of the number of processors.High-

Performance Computing and Networking edited by Peter Sloot, Marian Bubak and Bob Hertzberge 1998 ISBN 3540644431 page 935 Gunawan and Cai then generalized this further by introducing

Cache (computing)|cache size, and separated the messages based on their sizes, obtaining two

separate models, one for messages below cache size, and one for those above.


Message passing in computer clusters - Performance simulation

1 The communication overhead for Message Passing Interface|MPI message passing can thus be

simulated and better understood in the context of large-scale parallel job

execution.High Performance Computational Science and

Engineering edited by Michael K



1 Other simulation tools include MPI-sim and BIGSIM.Advances in

Computer Science, Environment, Ecoinformatics, and Education edited by Song Lin and Xiong Huang 2011

ISBN 3642233236 page 16 MPI-Sim is an execution-driven simulator that

requires C or C++ programs to operate. ClusterSim, on the other hand uses a hybrid higher-level

modeling system independent of the programming language used for

program execution.



1 Unlike MPI-Sim, BIGSIM is a trace-driven system that simulates based on the logs of executions saved in

files by a separate emulator program.Languages and Compilers

for Parallel Computing edited by Keith Cooper, John Mellor-Crummey

and Vivek Sarkar 2011 ISBN 3642195946 pages 202–203 BIGSIM

includes an emulator, and a simulator


Transient (computer programming) - Message passing

1 At the level of message passing, transient communication means the way by which the messages are not

saved into a buffer to wait for its deliver at the message receiver. The

messages will be delivered only if both the systems (sender and

receiver) are running. If the receiver is not running at the send time, the message will be discarded, because

it has not been stored into intermediate buffers.


For More Information, Visit:

• https://store.theartofservice.com/the-message-passing-toolkit.html

The Art of Servicehttps://store.theartofservice.com






Message Passing .

Documents

Transcript of Message Passing .