Mpi 0802 Mod Collective
Transcript of Mpi 0802 Mod Collective
-
7/30/2019 Mpi 0802 Mod Collective
1/33
1
Collective Communication
Collective communication
Barrier synchronization
Broadcast*
Scatter*
Gather
Gather/Scatter variations
Summary illustration
Global Reductionoperations
Predefined reductionoperations
MPI _Reduce
Minloc and Maxloc*
User-Defined reduction
operations
Reduction operator functions
Registering a user-defined
reduction operator* Variants ofMPI _Reduce
Class Exercise: Last Ring
*includes sample C and Fortranprograms
-
7/30/2019 Mpi 0802 Mod Collective
2/33
2
Collective communication
Communications involving a group of processes
Called by all processes in a communicator
Examples: Barrier (synchronization)
Broadcast, scatter, gather (data distribution)
Global sum, global maximum, etc. (collectiveoperations)
-
7/30/2019 Mpi 0802 Mod Collective
3/33
3
Characteristics of collective communication
Collective communication will not interfere withpoint-to-point communication and vice-versa
All processes must call the collective routine Synchronization not guaranteed (except forbarrier)
No non-blocking collective communication No tags
Receive buffers must be exactly the right size
-
7/30/2019 Mpi 0802 Mod Collective
4/334
Barrier synchronization
Red light for each processor: turns green whenall processors have arrived
C: i nt MPI _Bar r i er ( MPI _Comm comm)
Fortran:I NTEGER COMM, I ERROR
CALL MPI _BARRI ER ( COMM, I ERROR)
-
7/30/2019 Mpi 0802 Mod Collective
5/335
Broadcast
One-to-all communication: same data sent from rootprocess to all the others in the communicator
C:i nt MPI _Bcast ( voi d *buf f er , i nt , count ,
MPI _Dat at ype dat at ype, i nt r oot , MPI _Comm comm)
Fortran:
BUFFER ( *)I NTEGER COUNT, DATATYPE, ROOT, COMM, I ERRORCALL MPI _BCAST( BUFFER, COUNT, DATATYPE, ROOT, COMM,
I ERROR)
All processes must specify same root rank andcommunicator
-
7/30/2019 Mpi 0802 Mod Collective
6/336
Sample Program #5 - C#i ncl udei nt mai n ( i nt ar gc, char *ar gv[ ] ) {
i nt r ank;
doubl e par am;
MPI _I ni t ( &ar gc, &ar gv) ;
MPI _Comm_r ank( MPI _COMM_WORLD, &r ank) ;
i f ( r ank==5) par am=23. 0;
MPI_Bcast(¶m,1,MPI_DOUBLE,5,MPI_COMM_WORLD);
pr i nt f ( "P: %d af t er br oadcast par amet er i s %f \ n" , r ank, par am) ;
MPI _Fi nal i ze( ) ;
}
Progr am Out putP: 0 af t er br oadcast par amet er i s 23. 000000P: 6 af t er br oadcast par amet er i s 23. 000000P: 5 af t er br oadcast par amet er i s 23. 000000P: 2 af t er br oadcast par amet er i s 23. 000000P: 3 af t er br oadcast par amet er i s 23. 000000P: 7 af t er br oadcast par amet er i s 23. 000000P: 1 af t er br oadcast par amet er i s 23. 000000P: 4 af t er br oadcast par amet er i s 23. 000000
-
7/30/2019 Mpi 0802 Mod Collective
7/337
Sample Program #5 - FortranPROGRAM br oadcastI NCLUDE ' mpi f . h'
I NTEGER er r , r ank, si ze
r eal par am
CALL MPI _I NI T( er r )
CALL MPI _COMM_RANK( MPI _COMM_WORLD, r ank, er r )
CALL MPI _COMM_SI ZE( MPI _COMM_WORLD, si ze, er r )
i f ( r ank. eq. 5) par am=23. 0
call MPI_BCAST(param,1,MPI_REAL,5,MPI_COMM_WORLD,err)
pr i nt *, "P: ", r ank, " af t er br oadcast par am i s ", par am
CALL MPI _FI NALI ZE( er r )
ENDProgr am Out put
P: 1 af t er br oadcast par amet er i s 23.P: 3 af t er br oadcast par amet er i s 23.P: 4 af t er br oadcast par amet er i s 23P: 0 af t er br oadcast par amet er i s 23
P: 5 af t er br oadcast par amet er i s 23.P: 6 af t er br oadcast par amet er i s 23.P: 7 af t er br oadcast par amet er i s 23.P: 2 af t er br oadcast par amet er i s 23.
-
7/30/2019 Mpi 0802 Mod Collective
8/338
Scatter
One-to-all communication: different data sent to eachprocess in the communicator (in rank order)
C:i nt MPI _Scat t er ( voi d* sendbuf , i nt sendcount ,
MPI _Dat at ype sendt ype, voi d* r ecvbuf , i nt r ecvcount ,MPI _Dat at ype r ecvt ype, i nt r oot , MPI _Comm comm)
Fortran:CALL MPI _SCATTER( SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF,RECVCOUNT, RECVTYPE, ROOT, COMM, I ERROR)
SENDBUF( *) , RECVBUF( *)
sendcount is the number of elements sent to eachprocess, not the total number sent send arguments are significant only at the root process
-
7/30/2019 Mpi 0802 Mod Collective
9/33
9
Scatter example
A B C D E
A B C D E
BA C D E
-
7/30/2019 Mpi 0802 Mod Collective
10/33
10
#i ncl ude i nt mai n ( i nt ar gc, char *ar gv[ ] ) {
i nt r ank, si ze, i , j ;
doubl e param[ 4] , mi ne;
i nt sndcnt , r evcnt ;
MPI _I ni t ( &ar gc, &ar gv) ;
MPI _Comm_r ank( MPI _COMM_WORLD, &r ank) ;
MPI _Comm_si ze( MPI _COMM_WORLD, &si ze) ;
r evcnt =1;
i f ( r ank==3) {
f or ( i =0; i
-
7/30/2019 Mpi 0802 Mod Collective
11/33
11
Sample Program #6 - FortranPROGRAM scat t erI NCLUDE ' mpi f . h'
I NTEGER er r , r ank, si ze
r eal par am( 4) , mi ne
i nt eger sndcnt , r cvcnt
CALL MPI _I NI T( er r )
CALL MPI _COMM_RANK( MPI _WORLD_COMM, r ank, er r )CALL MPI _COMM_SI ZE( MPI _WORLD_COMM, si ze, er r )
r cvcnt =1
i f ( r ank. eq. 3) t hen
do i =1, 4
par am( i ) =23. 0+i
end do
sndcnt =1
end i f
call MPI_SCATTER(param,sndcnt,MPI_REAL,mine,rcvcnt,MPI_REAL,
& 3,MPI_COMM_WORLD,err)
pr i nt *, "P: ", r ank, " mi ne i s ", mi ne
CALL MPI _FI NALI ZE( er r )
END Progr amOut putP: 1 mi ne i s 25.P: 3 mi ne i s 27.P: 0 mi ne i s 24.P: 2 mi ne i s 26.
-
7/30/2019 Mpi 0802 Mod Collective
12/33
12
Gather
All-to-one communication: different data collectedby root process Collection done in rank order
MPI _GATHER &MPI _Gat her have samearguments as matching scatter routines
Receive arguments meaningful only at the rootprocess
-
7/30/2019 Mpi 0802 Mod Collective
13/33
13
Gather example
A B C D E
BA C D E
BA C D E
-
7/30/2019 Mpi 0802 Mod Collective
14/33
14
Gather/Scatter variations
MPI _Al l gat her MPI _Al l t oal l
No root process specified: all processes getgathered or scattered data
Send and receive arguments significant for all
processes
-
7/30/2019 Mpi 0802 Mod Collective
15/33
15
Summary
-
7/30/2019 Mpi 0802 Mod Collective
16/33
16
Global Reduction Operations
Used to compute a result involving datadistributed over a group of processes
Examples: Global sum or product Global maximum or minimum
Global user-defined operation
-
7/30/2019 Mpi 0802 Mod Collective
17/33
17
Predefined reduction operations
MPI Name Function
MPI_MAX Maximum
MPI_MIN Minimum
MPI_SUM SumMPI_PROD Product
MPI_LAND Logical AND
MPI_BAND Bitwise AND
MPI_LOR Logical OR
MPI_BOR Bitwise OR
MPI_LXOR Logical exclusive OR
MPI_BXOR Bitwise exclusive OR
MPI_MAXLOC Maximum and location
MPI_MINLOC Minimum and location
-
7/30/2019 Mpi 0802 Mod Collective
18/33
18
Global Reduction Operations
C:i nt MPI _Reduce( voi d* sendbuf , voi d* r ecvbuf , i nt count ,
MPI _Dat at ype dat at ype, MPI _Op op, i nt r oot ,MPI _Comm comm)
Fortran: SENDBUF( *) , RECVBUF( *)I NTEGER COUNT, DATATYPE, OP, ROOT, COMM, I ERRORCALL MPI _REDUCE( SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM,
I ERROR)
op is an associative operator that takes two operands oftype dat at ype and returns a result of the same type
count is the number of ops done on consecutiveelements ofsendbuf (it is also size ofr ecvbuf)
-
7/30/2019 Mpi 0802 Mod Collective
19/33
19
MPI_Reduce
-
7/30/2019 Mpi 0802 Mod Collective
20/33
20
MPI_MINLOC,MPI_MAXLOC
Designed to compute a global minimum/maximumand an index associated with the extreme value Common application: index is the processor rank (see
sample program)
If more than one extreme, get the first
Designed to work on operands that consist of a value
and index pair MPI _Dat at ypes include:
C:MPI _FLOAT_I NT, MPI _DOUBLE_I NT, MPI _LONG_I NT, MPI _2I NT,
MPI _SHORT_I NT, MPI _LONG_DOUBLE_I NT
Fortran:MPI _2REAL, MPI _2DOUBLEPRECI SI ON, MPI _2I NTEGER
-
7/30/2019 Mpi 0802 Mod Collective
21/33
21
Sample Program #7 - C#i ncl ude
/ * Run wi t h 16 pr ocesses * /
i nt mai n ( i nt ar gc, char *ar gv[ ] ) {
i nt r ank;
struct {
doubl e val ue;
i nt r ank;
} i n, out ;
i nt r oot ;
MPI _I ni t ( &ar gc, &ar gv) ;
MPI _Comm_r ank( MPI _COMM_WORLD, &r ank) ;
i n. val ue=r ank+1;
i n. r ank=r ank;
r oot =7;
MPI_Reduce(&in,&out,1,MPI_DOUBLE_INT,MPI_MAXLOC,root,MPI_COMM_WORLD);
i f ( r ank==r oot ) pr i nt f ( "PE: %d max=%l f at r ank %d\ n" , r ank, out . val ue, out . r ank) ;MPI_Reduce(&in,&out,1,MPI_DOUBLE_INT,MPI_MINLOC,root,MPI_COMM_WORLD);
i f ( r ank==r oot ) pr i nt f ( "PE: %d mi n=%l f at r ank %d\ n" , r ank, out . val ue, out . r ank) ;
MPI _Fi nal i ze( ) ;
}
Progr am Out putP: 7 max = 16. 000000 at r ank 15P: 7 mi n = 1. 000000 at r ank 0
-
7/30/2019 Mpi 0802 Mod Collective
22/33
22
PROGRAM MaxMi nC
C Run wi t h 8 processes
C
I NCLUDE ' mpi f . h'
I NTEGER er r , r ank, si ze
i nt eger i n( 2) , out ( 2)
CALL MPI _I NI T( er r )
CALL MPI _COMM_RANK( MPI _WORLD_COMM, r ank, er r )
CALL MPI _COMM_SI ZE( MPI _WORLD_COMM, si ze, er r )
i n( 1) =r ank+1
i n( 2) =r ank
call MPI_REDUCE(in,out,1,MPI_2INTEGER,MPI_MAXLOC,
& 7,MPI_COMM_WORLD,err)
i f ( r ank. eq. 7) pr i nt *, "P: ", r ank, " max=", out ( 1) , " at r ank ", out ( 2)
call MPI_REDUCE(in,out,1,MPI_2INTEGER,MPI_MINLOC,
& 2,MPI_COMM_WORLD,err)
i f ( r ank. eq. 2) pr i nt *, "P: ", r ank, " mi n=", out ( 1) , " at r ank ", out ( 2)
CALL MPI _FI NALI ZE( er r )
END
Sample Program #7 - Fortran
Progr am Out putP: 2 mi n=1 at r ank 0
P: 7 max=8 at r ank 7
-
7/30/2019 Mpi 0802 Mod Collective
23/33
23
User-Defined Reduction Operators
Reducing using an arbitrary operator op
C function of type MPI _User _f unct i on:voi d my_oper at or ( voi d *i nvec, voi d *i nout vec, i nt *l en,
MPI _Dat at ype *dat at ype)
Fortran function of type:FUNCTI ON MY_OPERATOR ( I NVEC( *) , I NOUTVEC( *) , LEN, DATATYPE)
I NVEC( LEN) , I NOUTVEC( LEN)
I NTEGER LEN, DATATYPE
-
7/30/2019 Mpi 0802 Mod Collective
24/33
24
Reduction Operator Functions
Operator function for op must act as:f or ( i =1 t o l en)
i nout vec( i ) = i nout vec( i ) op i nvec( i )
Operator op need not commute
i nout vec argument acts as both a second input
operand as well as the output of the function
R i i U D fi d R d i O
-
7/30/2019 Mpi 0802 Mod Collective
25/33
25
Registering a User-Defined Reduction Operator
Operator handles have type MPI _Op orI NTEGER
Ifcommut e isTRUE, reduction may beperformed faster
C:i nt MPI _Op_cr eat e( MPI _User _f unct i on *f unct i on,i nt commut e, MPI _Op *op)
Fortran:EXTERNAL FUNCI NTEGER OP, I ERRORLOGI CAL COMMUTEMPI _OP_CREATE ( FUNC, COMMUTE, OP, I ERROR)
S l P #8 C
-
7/30/2019 Mpi 0802 Mod Collective
26/33
26
Sample Program #8 - C#i ncl ude t ypedef st r uct {
doubl e r eal , i mag;
} compl ex;
voi d cpr od( compl ex *i n, compl ex *i nout , i nt *l en, MPI _Dat at ype *dpt r ) {
i nt i ;compl ex c;
f or ( i =0; i
-
7/30/2019 Mpi 0802 Mod Collective
27/33
27
Sample Program #8 - C (cont.)MPI _Op myop;MPI _Datat ype ct ype;
MPI _I ni t ( &ar gc, &ar gv) ;
MPI _Comm_r ank( MPI _COMM_WORLD, &r ank) ;
MPI _Type_cont i guous( 2, MPI _DOUBLE, &ct ype) ;
MPI _Type_commi t ( &ct ype) ;MPI _Op_cr eat e( cpr od, TRUE, &myop) ;
r oot =2;
sour ce. r eal =r ank+1;
sour ce. i mag=r ank+2;
MPI _Reduce( &sour ce, &r esul t , 1, ct ype, myop, r oot , MPI _COMM_WORLD) ;i f ( r ank==r oot ) pr i nt f ( "PE: %d r esul t i s %l f + %l f i \ n", r ank, r esul t . r eal ,
r esul t . i mag) ;
MPI _Fi nal i ze( ) ;
}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
P: 2 r esul t i s - 185. 000000 + - 180. 000000i
S l P #8 F
-
7/30/2019 Mpi 0802 Mod Collective
28/33
28
Sample Program #8 - FortranPROGRAM User OP
I NCLUDE ' mpi f . h'
I NTEGER er r , r ank, si ze
i nt eger sour ce, r esl t
ext er nal di gi t
l ogi cal commut e
i nt eger myop
CALL MPI _I NI T( er r )
CALL MPI _COMM_RANK( MPI _WORLD_COMM, r ank, er r )
CALL MPI _COMM_SI ZE( MPI _WORLD_COMM, si ze, er r )
commut e=. t r ue.
cal l MPI _OP_CREATE( di gi t , commut e, myop, er r )
source=( r ank+1) **2
cal l MPI _BARRI ER( MPI _COM_WORLD, er r )cal l MPI _SCAN( sour ce, r esl t , 1, MPI _I NTEGER, myop, MPI _COMM_WORLD, er r )
pr i nt * , "P: " , r ank, " my r esul t i s " , r es l t
CALL MPI _FI NALI ZE( er r )
END
i nt eger f unct i on di gi t ( i n, i nout , l en, t ype)
i nt eger i n( l en) , i nout ( l en)i nt eger l en, t ype
do i =1, l en
i nout ( i ) =mod( ( i n( i ) +i nout ( i ) ) , 10)
end do
di gi t = 5
end
Progr am Out putP: 6 my r esul t i s 0P: 5 my r esul t i s 1P: 7 my r esul t i s 4P: 1 my r esul t i s 5P: 3 my r esul t i s 0P: 2 my r esul t i s 4P: 4 my r esul t i s 5P: 0 my r esul t i s 1
V ri nt f MPI REDUCE
-
7/30/2019 Mpi 0802 Mod Collective
29/33
29
Variants ofMPI_REDUCE
MPI _ALLREDUCE - no root process (all get results) MPI _REDUCE_SCATTER - multiple results are scattered
MPI _SCAN - parallel prefix
MPI ALLREDUCE
-
7/30/2019 Mpi 0802 Mod Collective
30/33
30
MPI_ALLREDUCE
MPI REDUCE SCATTER
-
7/30/2019 Mpi 0802 Mod Collective
31/33
31
MPI_REDUCE_SCATTER
MPI SCAN
-
7/30/2019 Mpi 0802 Mod Collective
32/33
32
MPI_SCAN
Class Exercise: Collective Ring
-
7/30/2019 Mpi 0802 Mod Collective
33/33
33
Class Exercise: Collective Ring
Rewrite the Structured Ring program to use MPIglobal reduction to perform its global sums
Extra credit: Rewrite it so that each processcomputes a partial sum
Extra extra credit: Rewrite this so that each
process prints out its partial result in rank order