Parallel Programming with MPI Parallel programming overview Brief overview of MPI General Program...

Parallel Programming with MPISARDAR USMAN & EMAD ALAMOUDI

SUPERVISOR: PROF. RASHID MEHMOOD

[email protected]

MARCH 14, 2018

mailto:[email protected]

SourcesThe presentation is compiled using following sources.▪ http://mpi-forum.org/docs/

▪ https://computing.llnl.gov/tutorials/mpi/

▪ http://open-mpi.org

▪ http://mpi.deino.net/mpi_functions/index.htm

▪ http://mpitutorial.com/tutorials

▪ https://bioinfomagician.wordpress.com/category/parallel-computing/

▪ https://lsi.ugr.es/jmantas/ppr/ayuda/mpi_ayuda.php?ayuda=MPI_Barrier&idioma=en

PARALLEL PROGRAMMGIN WITH MPI 23/18/2018

http://mpi-forum.org/docs/

https://computing.llnl.gov/tutorials/mpi/

http://open-mpi.org/

http://mpi.deino.net/mpi_functions/index.htm

http://mpitutorial.com/tutorials

https://bioinfomagician.wordpress.com/category/parallel-computing/

https://lsi.ugr.es/jmantas/ppr/ayuda/mpi_ayuda.php?ayuda=MPI_Barrier&idioma=en

Outline▪ Parallel programming overview

▪ Brief overview of MPI▪ General Program Structure

▪ Basic MPI Functions

▪ Point to point Communication

▪ Communication modes

▪ Blocking and non-blocking communication

▪ MPI Collective Communication

▪ MPI Groups and Communicators


Parallel Computing

4PARALLEL PROGRAMMGIN WITH MPI3/18/2018

Why Parallel ComputingSave Time and Money


Why Parallel Computing▪ Solve Large/More Complex problems e.g. Web Search engines/databases processing millions of instructions/second.

▪ Provide Concurrency

▪ Take advantage of non-local resources

▪ Make better use of underlying parallel hardware

▪ Parallelism is future of computing and race is already on for Exascale computing.


Parallel Programming Models▪ There are several parallel programming models in common use: ▪ Shared Memory (without threads)

▪ Threads

▪ Distributed Memory / Message Passing

▪ Hybrid

▪ Single Program Multiple Data (SPMD)

▪ Multiple Program Multiple Data (MPMD)

▪ Parallel programming models exist as an abstraction above hardware and memory architectures.


Two Memory Models▪ Shared memory: all processors share the same address space▪ OpenMP: directive-based programming

▪ PGAD languages (UPC, Titanium, X10)

▪ Distributed memory: every processor has its own address space▪ MPI: Message Passing Interface

3/18/2018 PARALLEL PROGRAMMGIN WITH MPI 8

Shared Memory Model On A Distributed Memory Machine▪ Kendall Square Research (KSR) ALLCACHE approach. Machine memory was physically distributed across networked machines, but appeared to the user as a single shared memory global address space. Generically, this approach is referred to as "virtual shared memory".


Distributed Memory Model On A Shared Memory Machine▪ Message Passing Interface (MPI) on SGI Origin 2000. The SGI Origin 2000 employed the CC-NUMA type of shared memory architecture, where every task has direct access to global address space spread across all machines. However, the ability to send and receive messages using MPI, as is commonly done over a network of distributed memory machines, was implemented and commonly used.


Parallel Programming Models▪ This model demonstrates the following characteristics: ▪ A set of tasks that use their own

local memory during computation. Multiple tasks can reside on the same physical machine and/or across an arbitrary number of machines.

▪ Tasks exchange data through communications by sending and receiving messages.

▪ Data transfer usually requires cooperative operations to be performed by each process. For example, a send operation must have a matching receive operation.


Message Passing Interface (MPI)▪ MPI is a specification for message passing library that is standardized by MPI Forum

▪ Multiple vendor-specific implementations: MPICH, OpenMPI, Intel MPI

▪ MPI implementations are used for programming systems with distributed memory▪ Each process has a different address space

▪ Processes need to communicate with each other

▪ Can also be used for shared memory and hybrid architectures

▪ MPI specifications have been defined for C, C++ and Fortran programs

▪ Goal of MPI is to establish portable, efficient and flexible standard for writing message passing programs


Message Passing Interface (MPI)▪ Reasons for using MPI

▪ Standardization

▪ Portability

▪ Functionality

▪ Availability

▪ Multiple vendor-specific implementations: MPICH, OpenMPI, Intel MPI


Applications (Science and Engineering) ▪ MPI is widely used in large scale parallel applications in science and engineering ▪ Atmosphere, Earth, Environment

▪ Physics - applied, nuclear, particle, condensed matter, high pressure,

▪ fusion, photonics

▪ Bioscience, Biotechnology, Genetics

▪ Chemistry, Molecular Sciences

▪ Geology, Seismology

▪ Mechanical Engineering - from prosthetics to spacecraft

▪ Electrical Engineering, Circuit Design, Microelectronics

▪ Computer Science, Mathematics


Important considerations while using MPI ▪ All parallelism is explicit: the programmer is responsible for correctly identifying parallelism and implementing parallel algorithms using MPI constructs


MPI Application Structure


MPI Basics▪ mpirun starts the required number of processes.

▪ All processes which have been started by mpirun are organized in a process group (Communicator) called MPI_COMM_WORLD

▪ Every process has unique identifier(rank) which is between 0 and n-1.


Compiling and Running MPI applications▪MPI is a library▪ Applications can be written in C, C++ or Fortran and appropriate calls to MPI

can be added where required

▪Compilation:▪ Regular applications:

▪ gcc test.c -o test

▪ MPI applications▪ mpicc test.c -o test

▪Execution:▪ Regular applications:

▪ ./test

▪ MPI applications (running with 16 processes):▪ mpiexec –n 16 ./test


MPI Basics▪ MPI_Init (&argc,&argv) : Initializes the MPI execution environment.

▪ MPI_Comm_size (comm,&size) : Returns the total number of MPI processes in the specified communicator

▪ MPI_Comm_rank (comm,&rank) : Returns the rank of the calling MPI process within the specified communicator

▪ MPI_Finalize () : Terminates the MPI execution environment


Simple MPI Program Identifying Processes


Data Communication ▪Data communication in MPI is like email exchange

▪One process sends a copy of the data to another process (or a group of processes), and the other process receives it

▪Communication requires the following information: ▪ Sender has to know:

▪ Whom to send the data to (receiver’s process rank)

▪ What kind of data to send (100 integers or 200 characters, etc)

▪ A user-defined “tag” for the message (think of it as an email subject; allows the receiver to understand what type of data is being received)

▪ Receiver “might” have to know: ▪ Who is sending the data (OK if the receiver does not know; in this case

▪ sender rank will be MPI_ANY_SOURCE, meaning anyone can send)

▪ What kind of data is being received (partial information is OK: I might

▪ receive up to 1000 integers)

▪ What the user-defined “tag” of the message is (OK if the receiver does

▪ not know; in this case tag will be MPI_ANY_TAG)


Ranks for communication▪ When sending data, the sender has to specify the destination process’ rank

▪ Tells where the message should go▪ The receiver has to specify the source process’ rank

▪ Tells where the message will come from▪ MPI_ANY_SOURCE is a special “wild-card” source that can be

▪ Used by the receiver to match any source


Point-to-Point Communication▪ Communication between two processes

▪ Source process sends message to destination process

▪ Communication takes place within a communicator

▪ Destination process is identified by its rank in the communicator


3

1 4

0 2

destination

source

Definitions▪ “Completion” means that memory locations used in the message transfer can be safely accessed▪ send: variable sent can be reused after completion

▪ receive: variable received can now be used

▪ MPI communication modes differ in what conditions on the receiving end are needed for completion

▪ Communication modes can be blocking or non-blocking▪ Blocking: return from function call implies completion

▪ Non-blocking: routine returns immediately


Simple Communication in MPI


Sending data

MPI_Send(&S_buf, count, MPI_Datatype , dest, tag,MPI_COMM_WORLD)

Data to send

Number of Elements to send Data Type of Element Process rank where data

need to be sent

User defined Unique message identifier

Process group for all Process started by mpirun


MPI_Send(&S_buf, count, MPI_Datatype , source, tag,MPI_COMM_WORLD, &status)

Data to send

Number of Elements to send Data Type of Element Process rank from where

data is sent

User defined Unique message identifier

Process group for all Process started by mpirun

Receiving data

Status Information


Array decomposition and addition▪The master task first initializes an array and then distributes an equal portion that array to the other tasks.

▪Other tasks perform an addition operation to each array element

▪As each of the non-master tasks finish, they send their updated portion of the array to the master.

▪Finally, the master task displays selected parts of the final array and the global sum of all array elements


Array Decomposition and Addition

1 2 3 4 5 6 7 8

1 2

2+1 3

3 4

3+4 7

7 8

7+8 15

5 6

5+6 11

Process 0 Process 3Process 2Process 1

Process 0

3 7 11 15

3+7+11+15= 36

Process 0


Array Decomposition and Addition


Serial Vs Parallel

0

0.01

0.02

0.03

0.04

0.05

0.06

Serial Parallel

Execution Time

Execution Time


Communication Modes


BufferingIn a perfect world, every send operation would be perfectly synchronized with its matching receive.

MPI implementation must be able to deal with storing data when the two tasks are out of sync.

A send operation occurs 5 seconds before the receive is ready - where is the message while the receive is pending?

Multiple sends arrive at the same receiving task which can only accept one send at a time - what happens to the messages that are "backing up"?


Buffering


System buffer space ▪ Opaque to the programmer and managed entirely by the MPI library

▪ A finite resource that can be easy to exhaust

▪ Able to exist on the sending side, the receiving side, or both.

▪ Something that may improve program performance because it allows send - receive operations to be asynchronous.

▪ MPI also provides for a user managed send buffer.


Blocking and Non-blocking▪ Send and receive can be blocking or non-blocking

▪ A blocking send can be used with a non-blocking receive, and vice-versa

▪ Non-blocking sends can use any mode – synchronous, buffered, standard, or ready.

▪ Non-blocking send and receive routines return almost immediately.

▪ Non-blocking operations simply "request" the MPI library to perform the operation when it is able.

▪ It is unsafe to modify the application buffer (your variable space) until you know for a fact the requested non-blocking operation was actually performed by the library. There are "wait" routines used to do this.


Non-blocking▪ Non-blocking communications are primarily used to overlap computation with communication and exploit possible performance gains.

▪ Characteristics of non-blocking communications▪No possibility of deadlocks

▪Decrease in synchronization overhead

▪Extra computation and code to test and wait for completion

▪Must not access buffer before completion


Avoiding Race conditions▪ Following code is not safe

▪ int i=123;

▪ MPI_Request myRequest;

▪ MPI_Isend(&i, 1, MPI_INT, 1, MY_LITTLE_TAG, MPI_COMM_WORLD, &myRequest);

▪ i=234;

▪ Backend MPI routines will read the value of i after we changed it to 234, so 234 will be sent instead of 123.

▪ This is a race condition, which can be very difficult to debug.


Avoiding Race conditions▪ int i=123;

▪ MPI_Request myRequest;

▪ MPI_Isend(&i, 1, MPI_INT, 1, MY_LITTLE_TAG, MPI_COMM_WORLD, &myRequest);

▪ // do some calculations here

▪ // Before we re-use variable i, we need to wait until the asynchronous function call is complete

▪ MPI_Status myStatus;

▪ MPI_Wait(&myRequest, &myStatus);

▪ i=234;


Non-blocking Communication Functions


Non-blocking Communication▪ Simple hello world program that uses nonblocking send/receive routines.

▪ Request: non-blocking operations may return before the requested system buffer space is obtained, the system issues a unique "request number".


Blocking▪ A blocking send routine will only "return" after it is safe to modify the application buffer (your send data) for reuse.

▪ The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer.

▪ A blocking send can be synchronous which means there is handshaking occurring with the receive task to confirm a safe send.

▪ A blocking send can be asynchronous if a system buffer is used to hold the data for eventual delivery to the receive.

▪ A blocking receive only "returns" after the data has arrived and is ready for use by the program.


Standard and Synchronous Send▪ Standard send▪Completes once message has been sent▪May or may not imply that message arrived▪Don’t make any assumptions (implementation

dependent)

▪ Buffered send▪Data to be sent is copied to a user-specified buffer▪Higher system overhead of copying data to and from

buffer▪ Lower synchronization overhead for sender


Ready and Buffered Send▪ Ready send▪ Ready to receive notification must be posted; otherwise it exits with an

error

▪ Should not be used unless user is certain that corresponding receive is posted before the send

▪ Lower synchronization overhead for sender as compared to synchronous send

▪ Synchronous send▪ Use if need to know that message has been received

▪ Sending and receiving process synchronize regardless of who is faster. Thus, processor idle time is possible

▪ Large synchronization overhead

▪ Safest communication method


Blocking Communication Functions


MPI_Bsend▪ This is a simple program that tests MPI_bsend.


Blocking Buffered Communication


Non-blocking Non-buffered Communication


For a Communication to Succeed▪ Sender must specify a valid destination rank

▪ Receiver must specify a valid source rank

▪ The communicator must be the same

▪ Tags must match

▪ Receiver’s buffer must be large enough

▪ User-specified buffer should be large enough (buffered send only)

▪ Receive posted before send (ready send only)


Completion Tests▪ Waiting and Testing for completion▪ Wait: function does not return until completion finished

▪ Test: function returns a TRUE or FALSE value depending on whether or not the communication has completed

▪ int MPI_Wait(MPI_Request *request, MPI_Status *status)

▪ int MPI_Test(MPI_Request *request, int *flag, MPI_Status*status)


Testing Multiple Communications▪ Test or wait for completion of one (and only one) message

▪ MPI_Waitany & MPI_Testany

▪ Test or wait for completion of all messages▪ MPI_Waitall & MPI_Testall

▪ Test or wait for completion of as many messages as possible

▪ MPI_Waitsome & MPI_Testsome


Wildcarding▪ Receiver can wildcard

▪ To receive from any source specify MPI_ANY_SOURCE as rank of source

▪ To receive with any tag specify MPI_ANY_TAG as tag

▪ Actual source and tag are returned in the receiver’s statusparameter


Receive Information▪ Information of data is returned from MPI_Recv (or MPI_Irecv) as status

▪ Information includes:▪ Source: status.MPI_SOURCE

▪ Tag: status.MPI_TAG

▪ Error: status.MPI_ERROR

▪ Count: message received may not fill receive buffer. Use following function to find number of elements actually received:

▪ int MPI_Get_count(MPI_Status status, MPI_Datatype datatype, int *count)

▪ Message order preservation: messages do not overtake each other. Messages are received in the order sent.


Timers▪ double MPI_Wtime(void)

▪ Time is measured in seconds

▪ Time to perform a task is measured by consulting the timer before and after


Deadlocks▪ A deadlock occurs when two or more processors try to access the same set of resources

▪ Deadlocks are possible in blocking communication

▪ Example: Two processors initiate a blocking send to each other without posting a receive

58

…

…

MPI_Send(P1)

MPI_Recv(P1)

…

…

…

MPI_Send(P0)

MPI_Recv (P0)

…

Process 0 Process 1

PARALLEL PROGRAMMGIN WITH MPI3/18/2018

Avoiding Deadlocks▪ Different ordering of send and receive: one processor post the send while the other posts the receive

▪ Use non-blocking functions: Post non-blocking receives early and test for completion

▪ Use buffered mode: Use buffered sends so that execution continues after copying to user-specified buffer


Matrix Multiplication ▪ In this code, the master task distributes a matrix to numtasks-1 worker tasks.

▪ Each worker task performs the multiplication on their chunk of matrices.

▪ And send the result back to Master.


Matrix Multiplication


Serial vs. Parallel

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Serial Parallel

Execution Time

Execution Time


MPI Collective Communication


MPI Collective Communication▪ All processes in the group have to participate in the same operation.

▪ Process group is defined by the communicator.

▪ For each communicator, one can have one collective operation ongoing at a time.

▪ Eases programming

▪ Enables low-level optimizations and adaptations to the hardware infrastructure.


Characteristics of Collective Communication▪ Collective communication will not interfere with point-to-point communication

▪ All processes must call the collective function

▪ Substitute for a sequence of point-to-point function calls

▪ Synchronization not guaranteed (except for barrier)

▪No tags are needed


Types of Collective Communication▪ Synchronization

• barrier

▪ Data exchange

• broadcast

• gather, scatter, all-gather, and all-to-all exchange

• Variable-size-location versions of above

▪ Global reduction (collective operations)

• sum, minimum, maximum, etc


SynchronizationCOLLECTIVE COMMUNICATION


Barrier Synchronization


Barrier Synchronization▪ Red light for each processor: turns green when all processors have arrived

▪ A process calling it will be blocked until all processes in the group (communicator) have called it

MPI_ Barrier(MPI_Comm comm)

• comm: communicator whose processes need to be synchronized


Data exchangeCOLLECTIVE COMMUNICATION


MPI_Bcast▪ The process with the rank root distributes the data stored in buffer to all other processes in the communicator comm.

▪ Data in buffer is identical on all other processes after the broadcast.


Traditional Send and Receive


0

12

3

4

5

7

6

Collective communication using MPI_Bcast


0

21

3 4 5

7

6

Broadcast▪ One-to-all communication: same data sent from root process to all others in communicator

▪ All processes must call the function specifying the same root and communicator

MPI_Bcast (&buf, count, datatype, root, comm).

• buf: starting address of buffer (sending and receiving)

• count: number of elements to be sent/received

• datatype: MPI datatype of elements

• root: rank of sending process

• comm: MPI communicator of processors involved


MPI_Bcast

PARALLEL PROGRAMMGIN WITH MPI 75

5

Process 1 Process 2 Process 3 Process 4

5

Process 1

5

Process 2

5

Process 3

5

Process 4

After MPI_Bcast

Before MPI_Bcast

3/18/2018

MPI_Bcast▪ A simple example that synchronize before and after sending data, then calculated the time taken.


MPI_Scatter▪ The process with the rank root distribute data stored in sendbuf to all other processes in communicator comm.

▪ Every process gets different segments of the original data at the root process.


Scatter▪ Example: partitioning an array equally among the processes

MPI_Scatter(&sbuf, scount, stype,&rbuf, rcount, rtype, root, comm)

• sbuf and rbuf: starting address of send and receive buffers

• scount and rcount: number of elements sent and received to/from each process

• stype and rtype: MPI datatype of sent/received data




MPI_Scatter


3

4

5

6


3

Process 1

4

Process 2

5

Process 3

6

Process 4

Before MPI_Scatter

After MPI_Scatter

3/18/2018

MPI_Scatter▪ A simple program that distribute a table to different processors that each one takes a row.


MPI_Gather▪ Reverse operation of MPI_Scatter.

▪ The root process receives the data stored in send buffer on all other process in the communicator comm into the receive buffer.


MPI_GatherMPI_Gather(&sbuf, scount, stype, &rbuf, rcount, rtype, root, comm)

• sbuf and rbuf: starting address of send and receive buffers

• scount and rcount: number of elements sent and received to/from each process

• stype and rtype: MPI datatype of sent/received data




MPI_Gather


3

4

5

6


After MPI_Gather

3

Process 1

4

Process 2

5

Process 3

6

Process 4

Before MPI_Gather

3/18/2018

MPI_Gather▪ A simple program the collect integer values from different processor. The master (Processor 0) is the one that will have all the values.


All-Gather and All-to-All (1)▪ All-gather

• All processes, rather than just the root, gather data from the group

▪ All-to-all

• mpi_alltoall is an extension of mpi_allgather to the case where each process sends distinct data to each of the receivers.

• All processes receive data from all processes in rank order

▪ No root process specified


MPI AllGather Example


MPI_Allgather

3/18/2018

MPI Alltoall Example


1 1 1

2 2 2

3 3 3

1 2 3

1 2 3

1 2 3

MPI_Alltoall

3/18/2018

All-Gather and All-to-All (2)

MPI_Allgather(&sbuf, scount, stype, &rbuf, rcount, rtype, comm)

MPI_Alltoall(&sbuf, scount, stype, &rbuf, rcount, rtype, comm)

• scount: number of elements sent to each process; for all-to-all communication, size of sbuf should be scount*p (p = # of processes)

• rcount: number of elements received from any process; size of rbuf should be rcount*p (p = # of processes)


MPI_Allgather▪ A program that computes the average of an array of elements in parallel using MPI_Scatter and MPI_Allgather


Variable-Size-Location Collective Functions

▪ Allows varying size and relative locations of messages in buffer

▪ Examples: MPI_Scatterv, MPI_Gatherv, MPI_Allgatherv, MPI_Alltoallv

▪ Advantages:• More flexibility in writing code

• More compact code

▪ Disadvantage:• may be less efficient than fixed size/location functions


Scatterv and GathervMPI_Scatterv(&sbuf, &scount, &displs, stype, &rbuf, rcount, rtype, root, comm)

MPI_Gatherv(&sbuf, scount, stype, &rbuf, &rcount, &displs, rtype, root, comm)

▪&scount and &rcount: integer array containing number of elements sent/received to/from each process

▪&displs: integer array specifying the displacements relative to start of buffer at which to send/place data to corresponding process


MPI_Gatherv


P 0

P4

P3

P2

P1

8421

Count

7310

displs

3/18/2018

0137

MPI_Scatterv


Root

P4

P3

P2

P1

8421

Count

7310

displs

3/18/2018

MPI_Scatterv▪ A program the distribute data to several processors. However, the data have variant size.


Global reductionCOLLECTIVE COMMUNICATION


Global Reduction Operations (1)▪ Used to compute a result involving data distributed over a group of processes

▪ Result placed in specified process or all processes

▪ Examples

• Global sum or product

• Global maximum or minimum


Global Reduction Operations (2)▪ MPI_Reduce returns results to a single process (root)

▪ MPI_Allreduce returns results to all processes in the group

▪ MPI_Reduce_scatter scatters a vector, which results from a reduce operation, across all processes


Global Reduction Operations (3)

MPI_Reduce(&sbuf, &rbuf, count, stype, op, root, comm)

MPI_Allreduce(&sbuf, &rbuf, count, stype, op, comm)

MPI_Reduce_scatter(&sbuf, &rbuf, &rcounts, stype, op, comm)

• sbuf: address of send buffer

• rbuf: address of receive buffer

• rcounts: integer array that has counts of elements received from each process

• op: reduce operation, which may be MPI predefined or user-defined (by using MPI_Op_create)


Predefined Reduction Operations


MPI name Function

MPI_MAX Maximum

MPI_MIN Minimum

MPI_SUM Sum

MPI_PROD Product

MPI_LAND Logical AND

MPI_BAND Bitwise AND

MPI_LOR Logical OR

MPI_BOR Bitwise OR

MPI_LXOR Logical exclusive OR

MPI_BXOR Bitwise exclusive OR

MPI_MAXLOC Maximum and location

MPI_MINLOC Minimum and location

3/18/2018

MPI Reduce


MPI Reduce Example


MPI_Op


MPI_Reduce▪ Program that computes the average of an array of elements in parallel using MPI_Reduce.


MPI AllReduce


MPI AllReduce Example


MPI_AllReduce▪ Program that computes the standard deviation of an array of elements in parallel using MPI_AllReduce.


MPI Reduce_scatter


MPI_Reduce_scatter▪ A program that calculate a summation of a vector, then distribute the local results to each processors using MPI_Reduce_scatter.


MPI Scan


MPI_Scan▪ We have a histogram distributed across nodes in exp_pdf_i, and then calculate the cumulative frequency histogram (exp_cdf_i) across all nodes.


Minloc and Maxloc▪ Designed to compute a global minimum/maximum and an index associated with the extreme value

• index is processor rank that held the extreme value

▪ If more than one extreme exists, index returned is for the first

▪ Designed to work on operands that consist of a value and index pair. MPI defines such special data types:

• MPI_FLOAT_INT, MPI_DOUBLE_INT, MPI_LONG_INT, MPI_2INT, MPI_SHORT_INT, MPI_LONG_DOUBLE_INT


MPI_Minloc▪ A program that locate a minimum value and its location form an arrays of integer.


MPI Groups and Communicators


MPI Groups and Communicators▪A group is an ordered set of processes• Each process in a group is associated with a unique integer rank between 0

and P-1, with P the number of processes in the group

▪A communicator encompasses a group of processes that may communicate with each other• Communicators can be created for specific groups

• Processes may be in more than one group/communicator

▪Groups/communicators are dynamic and can be setup and removed at any time

▪From the programmer’s perspective, a group and a communicator are the same


MPI Groups and Communicators


1

2 3

4

5

6

7

8

9

5

6

7

891

23

4

12

3

4

12

3

4

5

6

7

89

5

6

7

89

Group1Group2

MPI_COMM_WORLD

comm1 comm2

Communication

3/18/2018

MPI Group Operations▪MPI_Comm_group• Returns the group associated with a communicator

▪MPI_Group_union• Creates a group by combining two groups

▪MPI_Group_intersection• Creates a group from the intersection of two groups

▪MPI_Group_difference• Creates a group from the difference between two groups

▪MPI_Group_incl• Creates a group from listed members of an existing group

▪MOI_Group_excl• Creates a group excluding listed members of an existing group

▪MPI_Group_free• Marks a group for deallocation

PARALLEL PROGRAMMGIN WITH MPI3/18/2018 116

MPI Group Operations

PARALLEL PROGRAMMGIN WITH MPI

0 2

1 3

2 4

3 5

0 2

1 3

2 4

3 5

0 2

1 3

4

5

2

3

Union

Intersection

3/18/2018 117

MPI Communicator Operations

▪MPI_Comm_size• Returns number of processes in communicator’s group

▪MPI_Comm_rank• Returns rank of calling process in communicator’s group

▪MPI_Comm_compare• Compares two communicators

▪MPI_Comm_dup• Duplicates a communicator

▪MPI_Comm_create• Creates a new communicator for a group

▪MPI_Comm_split• Splits a communicator into multiple, non-overlapping communicators

▪MPI_Comm_free• Marks a communicator for deallocation


MPI_Comm_Split

PARALLEL PROGRAMMGIN WITH MPI

0 1

4 5

8 9

2 3

6 7

14 151312

1110

0 1 2 3

0 1 2 3

0 1 2 3

0 1 2 3

3/18/2018 119

MPI_Comm_split▪ Example using MPI_Comm_split to divide a communicator into subcommunicators


MPI_Group▪ Example using MPI_Comm_group to divide a communicator into subcommunicators


Thanks


Contact


Prof. Rashid MehmoodDirector of Research, Training, and Consultancy

HPC Center, King Abdul Aziz University, Jeddah, Saudi Arabia,

[email protected]

mailto:[email protected]

Parallel Programming with MPI Parallel programming overview Brief overview of MPI General Program...

Documents

Transcript of Parallel Programming with MPI Parallel programming overview Brief overview of MPI General Program...