FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 1

FIT5174 Distributed & Parallel Systems

Lecture 5

Message Passing and MPI

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 2

Acknowledgement

These slides are based on slides and material by:

C Evangelinos,

Scott Baden,

Zhiliang Xu

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 3

Shared Memory Approach

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 4

Message Passing Model

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 5

Distributed Computing using Message Passing

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 6

Single Program / Multiple Program - Multiple Data

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 7

Programming with message passing

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 8

Message passing concept

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 9

Send and Receive concept

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 10

Message completion

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 11

Buffering concept

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 12

Causality concept

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 13

Asynchronous versus Synchronous

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 14

Overlapping operations

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 15

What is MPI?

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 16

MPI websites

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 17

MPI: Message Passing Interface

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 18

MPI history

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 19

MPI in context

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 20

MPI fundamentals

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 21

Minimal MPI subset

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 22

Initializing MPI

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 23

Getting started with MPI

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 24

MPI program details

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 25

MPI program more details

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 26

Simple MPI C Program

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 27

MPI Communicators and handles

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 28

MPI Communicator size

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 29

MPI Communicator size

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 30

MPI Process rank

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 31

MPI Process rank

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 32

MPI exiting

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 33

MPI communications

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 34

MPI basic datatypes for C

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 35

MPI C data types

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 36

MPI messages

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 37

MPI point-to-point communications

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 38

MPI point-to-point messages

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 39

MPI point-to-point messages requirements

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 40

Synchronous versus Asynchronous communications

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 41

MPI blocking standard Send

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 42

Send and Receive code fragment

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 43

Other MPI Send options

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 44

Considerations for MPI Send

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 45

Performance of MPI Send

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 46

MPI blocking Receive

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 47

MPI Receive considerations

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 48

MPI message passing restrictions

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 49

Simple MPI ‘ping-pong’

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 50

MPI deadlock scenario

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 51

MPI correctness and fairness

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 52

MPI bidirectional communication

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 53

MPI non-blocking communication

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 54

MPI non-blocking standard send

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 55

MPI non-blocking receive

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 56

Avoiding Deadlocks

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 57

MPI non-blocking sends

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 58

MPI non-blocking Synchronous Send

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 59

MPI testing instead of waiting

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 60

MPI with many outstanding calls

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 61

Other MPI communications calls

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 62

MPI persistent communication

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 63

MPI persistent communications

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 64

Useful MPI wildcards and constants

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 65

MPI collective communications

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 66

MPI synchronization

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 67

MPI broadcast

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 68

MPI gather

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 69

MPI scatter

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 70

MPI gather to all

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 71

Using vectors

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 72

Using vectors non-contiguously

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 73

Using binary trees

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 74

Using other kinds of trees

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 75

All-to-All personalized communication

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 76

MPI program structure summary

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 77

MPI program summary

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 78

MPI Simple C program

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 79

MPI Safe C program

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 80

MPI Deadlocking C program

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 81

MPI buffering dependent C program

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 82

Summarizing Distributed versus Shared Memory

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 83

Hybrid programming

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 84

Why use hybrid programming?

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.