Parallel Programming in C with MPI and OpenMP
31
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn Michael J. Quinn
description
Parallel Programming in C with MPI and OpenMP. Michael J. Quinn. Chapter 2. Parallel Architectures. Outline. Interconnection networks Processor arrays Multiprocessors Multicomputers Flynn’s taxonomy. Interconnection Networks. Uses of interconnection networks - PowerPoint PPT Presentation
Transcript of Parallel Programming in C with MPI and OpenMP
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Parallel Programmingin C with MPI and OpenMP
Michael J. QuinnMichael J. Quinn
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Chapter 2
Parallel ArchitecturesParallel Architectures
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Outline
Interconnection networksInterconnection networks Processor arraysProcessor arrays MultiprocessorsMultiprocessors MulticomputersMulticomputers Flynn’s taxonomyFlynn’s taxonomy
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Interconnection Networks
Uses of interconnection networksUses of interconnection networks Connect processors to shared memoryConnect processors to shared memory Connect processors to each otherConnect processors to each other
Interconnection media typesInterconnection media types Shared mediumShared medium Switched mediumSwitched medium
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Shared versus Switched Media
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Shared Medium
Allows only message at a timeAllows only message at a time Messages are broadcastMessages are broadcast Each processor “listens” to every messageEach processor “listens” to every message Arbitration is decentralizedArbitration is decentralized Collisions require resending of messagesCollisions require resending of messages Ethernet is an exampleEthernet is an example
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Switched Medium
Supports point-to-point messages between Supports point-to-point messages between pairs of processorspairs of processors
Each processor has its own path to switchEach processor has its own path to switch Advantages over shared mediaAdvantages over shared media
Allows multiple messages to be sent Allows multiple messages to be sent simultaneouslysimultaneously
Allows scaling of network to Allows scaling of network to accommodate increase in processorsaccommodate increase in processors
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Switch Network Topologies
View switched network as a graphView switched network as a graph Vertices = processors or switchesVertices = processors or switches Edges = communication pathsEdges = communication paths
Two kinds of topologiesTwo kinds of topologies DirectDirect IndirectIndirect
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Direct Topology
Ratio of switch nodes to processor nodes is Ratio of switch nodes to processor nodes is 1:11:1
Every switch node is connected toEvery switch node is connected to 1 processor node1 processor node At least 1 other switch nodeAt least 1 other switch node
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Indirect Topology
Ratio of switch nodes to processor nodes is Ratio of switch nodes to processor nodes is greater than 1:1greater than 1:1
Some switches simply connect other Some switches simply connect other switchesswitches
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating Switch Topologies
DiameterDiameter Bisection widthBisection width Number of edges / nodeNumber of edges / node Constant edge length? (yes/no)Constant edge length? (yes/no)
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
2-D Mesh Network
Direct topologyDirect topology Switches arranged into a 2-D latticeSwitches arranged into a 2-D lattice Communication allowed only between Communication allowed only between
neighboring switchesneighboring switches Variants allow wraparound connections Variants allow wraparound connections
between switches on edge of meshbetween switches on edge of mesh
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
2-D Meshes
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating 2-D Meshes
Diameter: Diameter: ((nn1/21/2))
Bisection width: Bisection width: ((nn1/21/2))
Number of edges per switch: 4Number of edges per switch: 4
Constant edge length? YesConstant edge length? Yes
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Binary Tree Network
Indirect topologyIndirect topology nn = 2 = 2dd processor nodes, processor nodes, nn-1 switches-1 switches
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating Binary Tree Network
Diameter: 2 log nDiameter: 2 log n
Bisection width: 1Bisection width: 1
Edges / node: 3Edges / node: 3
Constant edge length? NoConstant edge length? No
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Hypertree Network
Indirect topologyIndirect topology Shares low diameter of binary treeShares low diameter of binary tree Greatly improves bisection widthGreatly improves bisection width From “front” looks like From “front” looks like kk-ary tree of height -ary tree of height
dd From “side” looks like upside down binary From “side” looks like upside down binary
tree of height tree of height dd
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Hypertree Network
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating 4-ary Hypertree
Diameter: logDiameter: log n n
Bisection width: Bisection width: nn / 2 / 2
Edges / node: 6Edges / node: 6
Constant edge length? NoConstant edge length? No
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Butterfly Network
Indirect topologyIndirect topology nn = 2 = 2dd processor processor
nodes connectednodes connectedby by nn(log (log nn + 1) + 1)switching nodesswitching nodes
0 1 2 3 4 5 6 7
3 ,0 3 ,1 3 ,2 3 ,3 3 ,4 3 ,5 3 ,6 3 ,7
2 ,0 2 ,1 2 ,2 2 ,3 2 ,4 2 ,5 2 ,6 2 ,7
1 ,0 1 ,1 1 ,2 1 ,3 1 ,4 1 ,5 1 ,6 1 ,7
0 ,0 0 ,1 0 ,2 0 ,3 0 ,4 0 ,5 0 ,6 0 ,7R ank 0
R ank 1
R ank 2
R ank 3
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Butterfly Network Routing
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating Butterfly Network
Diameter: log Diameter: log nn
Bisection width: Bisection width: nn / 2 / 2
Edges per node: 4Edges per node: 4
Constant edge length? NoConstant edge length? No
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Hypercube
Directory topologyDirectory topology 2 2 xx 2 2 xx … … xx 2 mesh 2 mesh Number of nodes a power of 2Number of nodes a power of 2 Node addresses 0, 1, …, 2Node addresses 0, 1, …, 2kk-1-1 Node Node ii connected to connected to kk nodes whose nodes whose
addresses differ from addresses differ from ii in exactly one bit in exactly one bit positionposition
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Hypercube Addressing
0010
0000
0100
0110 0111
1110
0001
0101
1000 1001
0011
1010
1111
1011
11011100
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Hypercubes Illustrated
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating Hypercube Network
Diameter: log Diameter: log nn
Bisection width: Bisection width: nn / 2 / 2
Edges per node: log Edges per node: log nn
Constant edge length? NoConstant edge length? No
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Shuffle-exchange
Direct topologyDirect topology Number of nodes a power of 2Number of nodes a power of 2 Nodes have addresses 0, 1, …, 2Nodes have addresses 0, 1, …, 2kk-1-1 Two outgoing links from node Two outgoing links from node ii
Shuffle link to node Shuffle link to node LeftCycle(i)LeftCycle(i) Exchange link to node [xor (Exchange link to node [xor (ii, 1)], 1)]
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Shuffle-exchange Illustrated
0 1 2 3 4 5 6 7
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Shuffle-exchange Addressing
0 0 0 0 0 0 0 1 0 0 1 0 0 0 11 0 1 0 0 0 1 0 1
11 1 0 11 1 11 0 0 0 1 0 0 1 1 0 1 0 1 0 11 11 0 0 11 0 1
0 11 0 0 11 1
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Evaluating Shuffle-exchange
Diameter: 2log Diameter: 2log nn - 1 - 1
Bisection width: Bisection width: n n / log / log nn
Edges per node: 2Edges per node: 2
Constant edge length? NoConstant edge length? No
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Comparing Networks
All have logarithmic diameterAll have logarithmic diameterexcept 2-D meshexcept 2-D mesh
Hypertree, butterfly, and hypercube have Hypertree, butterfly, and hypercube have bisection width bisection width nn / 2 / 2
All have constant edges per node except All have constant edges per node except hypercubehypercube
Only 2-D mesh keeps edge lengths constant Only 2-D mesh keeps edge lengths constant as network size increasesas network size increases
