Asynchronous Interconnection Network and Communication

Chapter 3 of Casanova, et. al.

Interconnection NetworkTopologies

• The processors in a distributed memory parallel system are connected using an interconnection network.

• All computers have specialized coprocessors that route messages and place date in local memories – Nodes consist of a (computing) processor, a memory,

and a communications coprocessor– Nodes are often called processors, when not

ambigious.

Network Topology Types

• Static Topologies– A fixed network that cannot be changed– Nodes connected directly to each other by

point-to-point communications links

• Dynamic Topologies– Topology can change at runtime– One or more nodes can request direct

communication be established between them.• Done using switches

Some Static Topologies

• Fully connected network (or clique)

• Ring

• Two-Dimensional grid

• Torus

• Hypercube

• Fat tree

Examples of Interconnection Topologies

Static Topologies Features• Fixed number of nodes• Degree:

– Nr of nodes incident to edges

• Distance between nodes: – Length of shortest path between two nodes

• Diameter: – Largest distance between two nodes

• Number of links: – Total number of Edges

• Bisection Width: – Minimum nr. of edges that must be removed to

partition the network into two disconnected networks of the same size.

Classical Interconnection Networks Features

• Clique (or Fully Connected)– All processors are connected – p(p-1)/2 edges

• Ring– Very simple and very useful topology

• 2D Grid– Degree of interior processors is 4– Not symmetric, as edge processors have different

properties– Very useful when computations are local and

communications are between neighbors– Has been heavily used previously

Classical Network • 2D Torus

– Easily formed from 2D mesh by connecting matching end points.

• Hypercube– Has been extensively used– Using recursive defn, can design simple but very

efficient algorithms– Has small diameter that is logarithmic in nr of edges– Degree and total number of edges grows too quickly

to be useful with massively parallel machines.

Dynamic Topologies• Fat tree is different than other networks included

– The compute nodes are only at the leaves.– Nodes at higher level do not perform computation– Topology is a binary tree – both in 2D front view and in

side view. – Provides extra bandwidth near root.– Used by Thinking Machine Corp. on the CM-5

• Crossbar Switch– Has p2 switches, which is very expensive for large p– Can connect n processors to combination of n

processors– .Cost rises with the number of switches, which is

quadratic with number of processors.

Dynamic Topologies (cont)

• Benes Network & Omega Networks– Use smaller size crossbars arranged in stages– Only crossbars in adjacent stages are connected

together.– Called multi-stage networks and are cheaper to build

that full crossbar.– Configuring multi-stage networks is more difficult than

crossbar.– Dynamic networks are now the most common used

topologies.

A Simple Communications Performance Model

• Assume a processor Pi sends a message to Pj or

length m.– Cost to transfer message along a network link is

roughly linear in message length. – Results in cost to transfer message along a particular

route to be roughly linear in m.

• Let ci,j(m) denote the time to transfer this

message.

Hockney Performance Model for Communications

• The time ci,j(m) to transfer this message can be

modeled by

ci,j(m) = Li,j + m/Bi,j = Li,j + mbi,j – m is the size of the message

– Li,j is the startup time, also called latency

– Bi,j is the bandwidth, in bytes per second

– bi,j is 1/Bi,j, the inverse of the bandwidth

• Proposed by Hockney in 1994 to evaluate the performance of the Intel Paragon.

• Probably the most commonly used model.

Hockney Performance Model (cont.)

• Factors that Li,j and Bi,j depend on

– Length of route– Communication protocol used– Communications software overhead– Ability to use links in parallel– Whether links are half or full duplex– Etc.

Store and Forward Protocol

• SF is a point-to-point protocol• Each intermediate node receives and stores the

entire message before retransmitting it• Implemented in earliest parallel machines in

which nodes did not have communications coprocessors.

• Intermediate nodes are interrupted to handle messages and route them towards their destination.

Store and Forward Protocol (cont)

• If d(i,j) is the number of links between Pi & Pj,

the formula for ci,j(m) can be re-written as

ci,j(m) = d(i,j) {L+ m/B} = d(i,j)L + d(i,j)mb

where– L is the initial latency & b is the reciprocal for the

broadcast bandwidth for one link.

• This protocol produces a poor latency & bandwith

• The communication cost can be reduced using pipelining.

Store and Forward Protocol using Pipelining

• The message is split into r packets of size m/r.

• The packets are sent one after another from P i to Pj.

• The first packet reaches node j after ci,j(m/r) time units.

• The remaining r-1 packets arrive in

(r-1) (L+ mb/r) time units• Simplifying, total communication time reduces to

[d(i,j) -1+r][L+ mb/r] • Casanova, et.al. finds optimal value for r above.

Two Cut-Through Protocols

• Common performance model:

ci,j(m) = L + d(i,j)* + m/B

where– L is the one-time cost of creating a message. is the routing management overhead– Generally << L as routing management is

performed by hardware while L involve software overhead

– m/B is the time required to transmit the message through entire route

Circuit-Switching Protocol

• First cut-through protocol

• Route created before first message is sent

• Message sent directly to destination through this route– The nodes used in this transmission can not

be used during this transmission for any other communication

Wormhole (WH) Protocol

• A second cut-through protocol

• The destination address is stored in the header of the message.

• Routing is performed dynamically at each node.

• Message is split into small packets called flits

• If two flits arrive at the same time, flits are stored in intermediate nodes’ internal registers

Point-to-Point Communication Comparisons

• Store and Forward is not used in physical networks but only at applications level

• Cut-through protocols are more efficient– Hide distance between nodes– Avoid large buffer requirement for intermediate nodes– Almost no message loss– For small networks, flow-control mechanism not needed

• Wormhole generally preferred to circuit switching– Latency is normally much lower

LogP Model

• Models based on the LogP model are more precise than the Hockney model

• Involves three components of communication – the sender, the network, and the receiver– At times, some of these components may be busy while

others are not.

• Some parameters for LogP– m is the message size (in bytes)– w is the size of packets message is split into– L is an upper bound on the latency– o is the overhead,

• Defined to be the time that the a node is engaged in the transmission or reception of a packet

LogP Model (cont)• Parameters for LogP (cont)

– g or gap is the minimal time interval between consecutive packet transmission or packet reception

• During this time, a node may not use the communication coprocessor (i.e., network card)

– 1/g the communication bandwidth available per node– P the number of nodes in the platform

• Cost of sending m bytes with packet size w

• Processor occupational time on sender/receiver

1( ) 2

mc m o L g

w

1( 1m

o gw

Other LogP Related Models

• LogP attempts to capture in a few parameters the characteristics of parallel platforms.

• Platforms are fine-tuned and may use different protocols for short & long messages

• LogGP is an extension of LogP where G captures the bandwidth for long messages

• pLogP is an extension of LogP where L, o, and g depend on the message size m.

– Also seperates sender overhead os and receiver

overhead or.

Affine Models• The use of the floor functions in LogP models causes

them to be nonlinear.– Causes many problems in analytic & theoretical

studies.– Has resulted in proposal of many fully linear models

• The time that Pi is busy sending a message is expressed as an affine function of the message size – An affine function of m has form f(m) = a*m + b where

a and b are constants. If b=0, then f is linear function

• Similarly, the time Pj is busy receiving the message is expressed as an affine function of the message size

• We will postpone further coverage of the topic of affine models for the present

Modeling Concurrent Communications

• Multi-port model– Assumes that communications are contention-free

and do not interfere with each other.– A consequence is that a node may communicate with

an unlimited number of nodes without any degradation in performance.

– Would require a clique interconnection network to fully support.

– May simplify proofs that certain problems are hard• If hard under ideal communications conditions, then

hard in general.

– Assumption not realistic - communication resources are always limited.

– See Casanova text for additional information.

• Bounded Multi-port model– Proposed by Hong and Prasanna– For applications that uses threads (e.g., on a multi-

core technology), the network link can be shared by several incoming and outgoing communications.

– The sum of bandwidths allocated by operating system to all communications can not exceed bandwidth of network card.

– An unbounded nr of communications can take place if they share the total available bandwidth.

– The bandwidth defines the bandwidth allotted to each communication

– Bandwidth sharing by application is unusual, as is usually handled by operating system.

Concurrent Communications Models (2/5)

Concurrent Communications Models (3/5)

• 1 port (unidirectional or half-duplex) model– Avoids unrealistically optimistic assumptions– Forbids concurrent communication at a node.– A node can either send data or receive it, but not

simultaneously.– This model is very pessimistic, as real world platforms

can achieve some concurrent computations. – Model is simple and is easy to design algorithms that

follow this model.

Concurrent Communications Models (4/5)

• 1 port (bidirectional or full-duplex) model– Currently, most network cards are full-duplex.– Allows a single emission and single reception

simultaneously.– Introduced by Blat, et. al.

• Current hardware does not easily enable multiple messages to be transmitted simultaneous.

• Multiple sends and receives are claimed to be eventually serialized by a single hardware port to the next.

• Saif & Parashar did experimental work that suggests asynchronous sends become serialized when message sizes exceed a few megabytes.

Concurrent Communications Models (5/5)

• k-ports model– A node may have k>1 network cards– This model allows a node to be involved in a

maximum of one emission and one reception on each network card.

– This model is used in Chapters 4 & 5.

Bandwidth Sharing• The previous concurrent communication models

only consider contention on nodes

• Other parts of the network can also limit performance

• It may be useful to determine constraints on each network link

• This type of network model are useful for performance evaluation purposes, but are too complicated for algorithm design purposes.

• Casanova text evalutes algorithms using 2 models:– Hockney model or even simplified versions (e.g.

assuming no latency)– Multi-port (ignoring contention) or the 1 port model.

Case Study: Unidirectional Ring

• We first consider the platform of p processors arranged in a unidirectional ring.

• Processors are denoted Pk for k = 0, 1, … , p-1.

• Each PE can find its logical index by calling My_Num().

Unidirectional Ring Basics• The processor can determine the number of PEs

by calling NumProcs()– Both preceding commands are supported in MPI, a

language implemented on most asychronous systems.

• Each processor has its own memory• All processors execute the same program, which

acts on data in their local memories– Single Program, Multiple Data or SPMD

• Processors communicate by message passing– explicitly sending and receiving messages.

Unidirectional Ring Basics (cont – 2/5)• A processor sends a message using the function

send(addr,m)– addr is the memory address (in the sender process) of

first data item to be sent.– m is the message length (i.e., nr of items to be sent)

• A processor receives a message using function

receive(addr,m)– The addr is the local address in receiving processor

where first data item is to be stored.

– If processor Pi executes a receive, then its predecessor (P(i-1)mod p) must execute a send.

– Since each processor has a unique predecessor and successor, they do not have to be specified

Unidirectional Ring Basics (cont – 3/5)• A restrictive assumption is to assume that both

the send and receive is blocking. – Then the participating processes can not continue

until the communication is complete.– The blocking assumption is typical of 1st generation

platforms

• A classical assumption is keep the receive blocking but to allow the send is non-blocking– The processor executing a send can continue while

the data transfer takes place.– To implement, one function is used to initiate the send

and another function is used to determine when communication is finished.

Unidirectional Ring Basics (cont – 4/5)• In algorithms, we simply indicate the blocking and

non-blocking operations• More recent proposed assumption is that a single

processor can send data, receive data, and compute simultaneously.– All three can occur only if no race condition exists.

– Convenient to think of three logical threads of control running on a processor

• One for computing

• One for sending data

• One for receiving data

• We will usually use the less restrictive third assumption

Unidirectional Ring Basics (cont – 4/5)

• Timings for Send/Receive– We use a simplified version of the Hockney model– The time to send or receive over one link is

c(m) = L+ mb• m is the length of the message• L is the startup cost in seconds due to the physical

latency and the software overhead• b is the inverse of the data transfer rate.

The Broadcast Operation• The broadcast operation allows an processor Pk

to send the same message of length m to all other processors.

• At the beginning of the broadcast operation, the message is stored at the address addr in the memory of the sending process, Pk.

• At the end of the broadcast, the message will be stored at address addr in the memory of all processors.

• All processors must call the following function

Broadcast(k, addr, m)

Broadcast Algorithm Overview

• The message will go around the ring from processor - from Pk to Pk+1 to Pk+2 to … to Pk-1.

• We will assume the processor numbers are modulo p, where p is the number of processors. For example, if k=0 and p=8, then k-1 = p-1 = 7.

• Note there is no parallelism in this algorithm, since the message advances around ring only one processor per round.

• The predecessor of Pk (i.e, Pk-1) does not send the message to Pk.

Analysis of Broadcast Algorithm

• For algorithm to be correct, the “receive” in Step 10 will execute before Step 11.

• Running Time:– Since we have a sequence of p-1 communications, the

time to broadcast a message of length m is

(p-1)(L+mb)

• MPI does not typically use ring topology for creating communication primitives– Instead use various tree topologies that are more

efficient on modern parallel computer platforms.– However, these primitives are simpler on a ring.– Prepares readers to implement primitives, when more

efficient than using MPI primitives.

Scatter Algorithm• Scatter operation allows Pk to send a different

message of length m to each processor.

• Initially, Pk holds a message of length m to be sent

to Pq at location “addr [q]”.

• To keep the array of addresses uniform, space for a message to Pk is also provided.

• At the end of the algorithm, each processor stores its message from Pk at location msg.

• The efficient way to implement this algorithm is to pipeline the messages.– Message to most distant processor (i.e,, Pk-1) is

followed by message to processor Pk-2.

Discussion of Scatter Algorithm• In Steps 5-6, Pk successively send messages to

the other p-1processors in the order of their distance from Pk .

• In Step 7, Pk stores its message to itself.

• The other processors concurrently move messages along as they arrive in steps 9-12.

• Each processor uses two buffers with addresses tempS and tempR.– This allows processors to send a message

and to receive the next message in parallel in Step12.

Discussion of Scatter Algorithm (cont)

• In step 11, tempS tempR means two addresses are switched so received value can be sent to next processor.

• When a processor receives its message from Pk, the processor stops forwarding (Step 10).

• Whatever is in the receive buffer, tempR, at the end is stored as its message from Pk (Step 13).

• The running time of the scatter algorithm is the same as for the broadcast, namely

(p-1)(L+mb)

Example for Scatter Algorithm • Example: In Figure 3.7, let p=6 and k=4.• Steps 5-6: For i = 1 to p-1 do

send(addr[(k+p-i) mod p], m)– Let PE = (k+p-i) mod p = (10 – i) mod 6– For i=1, PE = 9 mod 6 = 3– For i=2, PE = 8 mod 6 = 2– For i=3, PE = 7 mod 6 = 1– For i=4, PE = 6 mod 6 = 0– For i=5, PE = 5 mod 6 = 5

• Note messages are sent to processors in the order 3, 2, 1, 0, 5 – That is, messages to most distant processors sent

first.

Example for Scatter Algorithm (cont)

• Example: In Figure 3.7, let p=6 and k=4.• Steps 10: For i = 1 to (k-1-q) mod p do • Compute: (k-1-q) mod p = (3-q) mod 6 for all q.• Note: q≠ k, which is 4

– q = 5 i = 1 to 4 since (3-5) mod 6 = 4 • PE 5 forwards values in loop from i = 1 to 4

– q = 0 i = 1 to 3 since (3-0) mod 6 = 3• PE 0 forwards values from i = 1 to 3

– q = 1 i = 1 to 2 since (3-1) mod 6 = 2• PE 1 forwards values from i = 1 to 2

– q = 2 i = 1 to 1 since (3-2) mod 6 = 1• PE 2 is active in loop when i = 1

Example for Scatter Algorithm (cont)

– q = 3 i = 1 to 0 since (3-3) mod 6 = 0• PE 3 precedes PE k, so it never forwards a value• However, it receives and stores a message at #9

• Note that in step 9, all processors store the first message they receive.– That means even the processor k-1 receives a value

to store.

All-to-All Algorithm • This command allows all p processors to

simultaneously broadcast a message (to all PEs)• Again, it is assumed all messages have length m• At the beginning, each processor holds the

message it wishes to broadcast at address my_message.

• At the end, each processor will hold an array addr of p messages, where addr[k] holds message from Pk.

• Using pipelining, the running time is the same as for a single broadcast, namely

(p-1)(L+mb)

Gossip Algorithm • Last of the classical collection of communication

operations• Each processor sends a different message to

each processor.• Gossip algorithm is Problem 3.7 in textbook.

– Note it will take 1 step for each PE to send a message to its closest neighbor using all links.

– It will take 2 steps for each PE to send a message to its 2nd closest neighbor, using all links.

– In general, it will take 1*2* …*(p-1) = (p-1)! steps for each PE to send messages to all other nodes, using all links of the network during each step.

– Complexity is (p-1)! = (p-1)(p-2)/2 = O(p2).

Pipelined Broadcast by kth processor• Longer messages can be broadcast faster if they are

broken into smaller pieces

• Suppose they are broken into r pieces of the same length.

• The sender sends the pieces out in order, and has them travel simultaneously on the ring.

• Initially, the pieces are stored at addresses addr[0], addr[1], … , addr[r-1].

• At the end, all pieces are stored in all processors.

• At each step, when a processor receives a message piece, it also forwards the piece it previously received, if any, to its successor.

Pipelined Broadcast (cont)

• There must be p-1 communication steps for first piece to reach the last processor, Pk-1.

• Then it takes r-1 time for the rest of the pieces to reach Pk-1.

• The required time is (p+r-2) ( L + mb/r)• The value of r that minimizes this expression can

be found by setting its derivative (with respect to r) to zero and solving for r.

• For large m, the time required tends to mb– Does not depend on p.

• Compares well to broadcast time: (p-1)(L+mb)

Hypercube

• Defn: A 0-cube consists of a single vertex. For n>0, an n-cube consists of two identical (n-1)-cubes with edges added to join matching pairs of vertices in the two (n-1) cubes.

Hypercubes (cont)• Equivalent defn: An n-cube is a graph consisting

of 2n vertices from 0 to 2n -1 such that two vertices are connected if and only if their binary representation differs by one single bit.

• Property: The diameter and degree of an n-cube are equal to n.

• Proof is left for reader. Easy if use recursion.• Hamming Distance: Let A and B be two points in

an n-cube. H(A,B) is the number of bits that differ between the binary labels of A and B.

• Notation: If b is a binary bit, then let b = 1-b, the bit complement of b.

Hypercube Paths• Using binary representation, let

– A = an-1 an-2 … a2 a1 a0

– B = bn-1 bn-2 … b2 b1 b0

• WLOG, assume that A and B have different bits exactly in their last k bits. – Differing bits at end makes numbering them easier

• Then a pathway from A to B can be created by the following sequence of nodes:– A = an-1 an-2 … a2 a1 a0

– Vertex 1 = an-1 an-2 … a2 a1 a0

– Vertex 2 = an-1 an-2 … a2 a1 a0

– ........

– B = Vertex k = an-1 an-2 … ak ak-1 … a2 a1 a0

Hypercube Paths (cont)• Independent of which bits of A and B agree,

there are k choices for first bit to flip, (k-1) choices for next bit to flip, etc.– This gives k! different paths from A to B.

• How many independent paths exist from A to B?– I.e., paths with only A and B as common vertices.

Theorem: If A and B are n-cube vertices that differ in k bits, then there exist exactly k independent paths from A to B.

Proof: First, we show k independent paths exist.• We build an independent path for each j with

0j<k

Hypercube Paths (cont)• Let P(j, j-1, j-2, … , 0, k-1, k-2, … , j+1) denote the path

from A to B with following sequence of nodes

– A = an-1 an-2 … ak ak-1 ak-2 … aj aj-1 aj-2 … a2 a1 a0

– V(1) = an-1 an-2 … ak ak-1 ak-2 … aj aj-1 aj-2 … a2 a1 a0

– V(2) = an-1 an-2 … ak ak-1 ak-2 … aj aj-1 aj-2 … a2 a1 a0

– ….........

– V(j+1) = an-1 an-2 … ak ak-1 ak-2 … aj aj-1 aj-2 … a2 a1 a0

– V(j+2) = an-1 an-2 … ak ak-1 ak-2 … aj aj-1 aj-2 … a2 a1 a0

– .............

– V(k-1) = an-1 an-2 … ak ak-1 ak+1 … aj+1aj aj-1 aj-2 … a2 a1 a0

– B =V(k)=an-1 an-2 … ak ak-1 ak+1 … aj+1aj aj-1 aj-2 … a2 a1 a0

Hypercube Pathways (cont)• Suppose the following two path have a common vertex X

other than A and B.

P(j, j-1, j-2, … , 0, k-1, k-2, … , j+1)

P(t, t-1, t-2, … , 0, k-1, k-2, … , t+1)

– Since paths are different, and A and B differ by k bits, we may assume 0≤t<j<k

• Let A and X differ in q bits

• To travel from A to X along either path, exactly q bits in circular sequence have been flipped, in a left to right order for each – 1st path: j, j-1 … t, t-1 … 0, k-1, k-2 … j-1

– 2nd path t, t-1 … 0, k-1, k-2 … j-1 … t-1

• This is impossible, as the q bits are flipped along each path can not be exactly the same bits.

Hypercube Paths (cont)

• Finally, there can not be another independent path Q from A to B (i.e., without another common vertex)– If so, the first node in path following A would have to flipped one

bit, say bit q, to agree with B.– But then, the path described earlier that flipped bit q first would

have a common interior vertex with this path Q.

Hypercube Routing• XOR is an exclusive OR. Exactly one input must be a 1.

• To design a route from A to B in the n-cube, we will use the algorithm that will always flip the rightmost bit that disagrees with B.

• The NOR of the binary representation of A and B indicates by “1’s” the bits that have to be flipped.

• For the 5-cube, if A is 10111 and B is 01110,then

– A NOR B is 11001, and the routing is as follows:

A = 10111 10110 11110 01110 = B

• This algorithm can be executed as follows: – A NOR B = 10111 NOR 01110 = 11001, so A routes the message

along link 1 (the digits link) to Node A1 ≡ 10110

– A1 NOR B = 10110 NOR 01110 = 11000, so Node A1 routes

message along link 4 to Node A2 =10110

– A2 NOR B = 10110 NOR 01110 = 10000, so A2 routes message along link 5 to B

Hypercube Routing (cont)• This routing algorithm can be used to implement a

wormhole or cut-through protocol in hardware.

• Problem: If another pair of processors have already reserved one link on the desired path, the message may stall until the end of the other communication.

• Solution: Since there are multiple paths, the routers select links to use based on a link reservation table and message labels.

– In our example, if link 1 in Node A is busy, then instead use link 4 to forward the message to new node 11111 that is on a new path to B.

– If at some point, the current vertex determines there is no useful links available, then current vertex will have to wait for a useful link to become available

– If at some vertex, the desired links are not available, then algorithm could use a link that extends path length

Gray Code • Recursive construction of Gray code:

– G1 = (0,1) and has 21 = 2 elements

– G2 = (00,01,11, 10) and had 22elements

– G3 = (000, 001, 011, 010, 110, 111, 101, 100) & has 23 elements

– etc.

• Gray code for dimension n 1 is denoted Gn, and is defined recursively. G1 = (0,1) and for n>1, Gn is the sequence 0Gn-1 followed by the sequence 1Gn-1

rev (i.e., ) where

– xGn-1 is the sequence obtained by prefixing every element of G with x

– Grev is the sequence obtained by listing the elements of G in the reverse order

• .

11 revnG

Gray Code (cont.)– Since we assume Gn-1 has 2n-1

elements and G n has exactly two copies of Gn-1 , it has 2n elements.

• Summary: The Gray code Gn is an ordered sequence of all the 2n binary codes with n digits whose successive values differ from each other by exactly one bit.

• Notation: Let gi(r) denote the ith element of the Gray

code of dimension r.

• Observation: Since the Gray code Gn = {g1(n), g2

(n),

…, g(n) } form a ordered sequence of names for all of the nodes in a 2n

ring.

Embeddings• Defn: An embedding of a topology (e.g., ring, 2D

mesh, etc) into an n-cube is a 1-1 function f from the vertices of the topology into the n-cube. – An embedding is said to preserve locality if the

image of any two neighbors are also neighbors in the n-cube

– If embeddings do not preserve locality, then we try to minimize the distance of neighbors in the hypercube.

– An embedding is said to be onto if the embedding function f has its range to be the entire n-cube.

A 2n Ring embedding onto n-cubeTheorem: There is an embedding of a ring with 2n

vertices onto an n-cube that preserves locality.

Proof: • Our construction of the Gray code provides an ordered

sequence of binary values with n digits that can be used as names of the nodes of the ring with 2n vertices– The first name (e.g., 00…0) can be used to name any

node in ring.– The sequence of Gray code names are given names

to ring nodes successively, in a clockwise or counter-clockwise order

Embedding Ring onto n-cube (cont)• The gray code binary numbers are identical with

the names assigned to hypercube nodes. – Two successive n-cube nodes are connected since

they only differ by one binary digi

• So the embedding is a result of the Gray code providing an ordered sequence of n-cube names that is used to provide successive names the 2n

ring nodes. • This concludes the proof that this embedding

follows “from construction of Gray code”.• However, the following formal proof provides

more details and an “after construction” argument..

A 2n Ring Embedding onto the n-cube • The following optional formal proof is included for those

who do not find the preceding argument convincing:

Theorem: There is an embedding of a ring with 2n vertices onto an n-cube that preserves locality.

Proof:• We establish the following claim: The mapping f(i) = gi

(n) is an embedding of the ring onto the n-cube.

• This claim is true for n = 1 as G1 = (0,1) and nodes 0 and 1 are connected on both the ring and 1-cube.

• We assume above claim is true for a fixed n-1 with n1. • We use the neighbor-preserving embedding f of the

vertices of a ring with 2n-1 nodes onto the (n-1)-cube to build a similar embedding f* of a ring with 2n vertices onto the n-cube.

Ring Embedding for n-cube (cont)• Recall that gray code for Gn is the sequence 0Gn-1 followed by

the sequence 1Gn-1rev

• The n-cube consists of one copy of an (n-1)-cube with a 0 prefix and a second copy of an (n-1) cube with a 1 prefix.

• By assumption that claim is true for n-1, the gray code sequence 0Gn provides the binary code for a ring of elements in the (n-1)-cube, with each successive element differing by 1 digit from its previous element.

• Likewise, the gray code sequence 1Gn-1rev provides the binary

code for a ring of elements in the second copy of an (n-1)-cube, with each successive element differing by on digit from its previous element..

• The last element of 0Gn-1 is identical to the first element of 1Gn-1

rev except for the added digit, so these two elements also differ by one bit.

2D Torus Embedding for n-cube • We embed a 2r 2s torus onto an n-cube with n=r+s by

using the cartesian product Gr Gs of two Gray codes.

• A processor with coordinates (i,j) on the grid is mapped to the processor f(i,j) = (gi

(r), gi(s) ) in the n-cube.

• Recall the map f1(i) = gi(r), is an embedding of a 2r ring onto a

r-cube “row” ring and f2(j) = gi(s) is an embedding of a 2s ring

onto a s-cube “column” ring.

• We must identify (gi(r), gi

(s) ) with the n=r+s cube where the first r bits are given by gi

(r) and the next s bits are given by gi

(s) .

• Then for a fixed j, f(i1,j) are neighbors of f(i,j) since f1 is an embedding of a 2r ring onto a r-cube.

• Likewise, for a fixed i, f(i, j 1) are neighbors of f(i,j) since f2 is an embedding of a 2s ring onto a s-cube.

Collective Communications in Hypercube• Purpose: Gain overview of complexity of collective

communications for hypercube.– Will focus on broadcast on hypercube.

• Assume processor 0 wants to broadcast. And consider the naïve algorithm:– Processor 0 sends the message to all of its neighbors– Next, every neighbor sends a message to all of its

neighbors.– Etc.

• Redundancy in naïve algorithm– The same processor receives the same message

many times.– E.g., processor 0 receives all its neighbors.– Mismatched SENDS and RECEIVES may happen.

Improved Hypercube Broadcast

• We seek a strategy where – Each processor receives the message only once– The number of steps is minimal.– Will use one or more spanning trees.

• Send and Receive will need parameter of which dimension the communication takes place– SEND( cube_link, send_addr, m )– RECEIVE(cube_link, send_addr, m )

Hypercube Broadcast Algorithm

• There are n steps, numbered from n-1 to 0• All processors will receive their msg on the link

corresponding to their rightmost 1• Processors receiving a msg will forward it on

links whose index is smaller than its rightmost 1.• At step i, all processors whose rightmost 1 is

strictly larger than i forwards the msg on link i.• Let broadcast originates with processor 0

– Assume 0 has a fictitious 1 at position n.

– This adds an additional digit, as has binary digits for positions 0,1, …, n-1.

Trace of Hypercube Broadcast for n=4

• Since broadcast originates with processor 0, we assume its index is 10000– Processor 0 sends the broadcast msg on link

3 to 1000• Next, both processor 0000 and 1000 have their

rightmost 1 in position at least three, so both send a message along link 2 – Msg goes to 0100 and 1100, respectively.

• This process continues until the last step at which every even-numbered processor sends its msg along its link 0.

Broadcasting using Spanning Tree

Broadcast Algorithm

• Let BIT(A,b) denote the value of the bth bit of processor A.

• The algorithm for the broadcast of a message of length m by processor k is given in Algorithm 3.5

• Since there are n steps, the execution time with the store-and-forward model is n(L+mb).

• This algorithm is valid for the 1-port model.– At each step, a processor communicates with at most

one other processor.

Broadcast Algorithm in Hypercube (cont)Observations about Algorithm Steps

2. Specifies that the algorithm action is for processor q

3. n is the number of binary digits used to label processors.

4. pos = q XOR k – uses “exclusive OR”.

– Broadcast is from Pk .

– Updates pos to work as if P0 is root of broadcast ???

5. Steps 5-7 sets “first-1” to the location of first “1” in pos.

Note: Steps 8-10 are the core of algorithm

8. Phase steps through link dimensions, higher ones first

9. If link dimension = first-1, q gets message on first-1 link– Seems to be moving message up to q ???

10.Q sends message along each smaller dimension than first-1 to processor below it.