Post on 13-Oct-2020
Fat-trees and Dragonflies – A perspective
on topologies
Sven-Arne Reinemo
svenar@simula.no
Simula Research Laboratory
HPC Advisory Council Switzerland Workshop
March 15, 2012
o Frank Olaf Sem-Jacobsen
o Mathias Myrland
o Professor Tor Skeie
o Eitan Zahavi, Mellanox
o Gilad Shainer, Mellanox
o Motivation
o The fat-tree topology
o The dragonfly topology
o Group topologies for the dragonfly
o Blocking, cost, and scalability
o Conclusion
o The fat-tree is the dominating topology for InfiniBand networks, but the recently proposed dragonfly topology has been suggested as an alternative.
o In that context we would like to answer the following questions:
o Is the dragonfly a viable alternative to the fat-tree?
o How do they compare in fundamental properties such as blocking and scalability?
o What about cost?
o What dragonfly configurations are attractive?
oMost common topology for InfiniBand clusters.
oCan be routed deadlock free without additional
resources such as virtual lanes.
oFault-tolerant through its path diversity.
oFull bisection bandwidth for arbitrary
permutations.
oScalable, but costly when going beyond to
levels.
oPerformance suffer slightly due to static
routing.
m-port n-tree k-ary n-tree hybrid tree
o Recently proposed by John Kim et al. in [1].
o Caused discussion in the HPC community about is suitability for IB and as an exascale topology.
o The dragonfly is a hierarchical topology with the following properties:
o Several groups are connected together using all to all links, i.e. each group has at least one link directly to each other group.
o The topology inside each group can be any topology. The recommendation in [1] is the flattened butterfly.
o Focus on reducing the number of long links and network diameter.
o Requires non-minimal global adaptive routing and advanced congestion look ahead for efficient operation.
[1] John Kim et al. “Technology-Driven, Highly-Scalable Dragonfly Topology” in proceedings of the
35th
International Symposium on Computer Architecture, 2008.
The recommendation in [1] is to keep:
Inter group
topology
Group0
Groupg-1
Group1
Switch
0
Switch
1
Switcha-1
0 …. p-1
0 …. h-1
a * h links to other groups
Intra group
topology
a ³ 2p ³ 2h
Dragonfly conforming to versus an unrestricted
dragonfly. a ³ 2p ³ 2h
In our studies we will:
oconsider the flattened butterfly as a group
topology.
oconsider the 2d torus as a group
topology.
oconsider a fat-tree as a group topology.
orelax the requirement to
compare it on equal terms with the fat-
tree.
a ³ 2p ³ 2h
A single flattened butterfly group with 8 terminals and 8 external
connections.
Fully connected, but requires non-minimal routing for path
diversity and load balancing.
S0 S1 S2 S3
Statistics for 100 random permutations on a 25 760 ports
dragonfly with flattened butterfly groups, buildt using 36-port
switches.
Good for nearest neighbour traffic, but bisection bandwidth does
not scale for arbitrary traffic.
Statistics for 100 random permutations on a 25 760 ports
dragonfly with 2d torus groups, built using 36-port switches.
Rather than building a complete fat-tree we use the fat-tree as
the group topology in the dragonfly.
oAs shown, the viable dragonfly groups are
the a flattened butterfly and the fat-tree.
oWe will compare these dragonfly
configurations with a 3- and 4-level fat-tree.
oWe look at how the blocking and cost of
these topologies develop as the size
increases.
Comparing the dragonflies with a 3- and 4-tier fat-tree.
Scaling of the 4-tier fat-tree and two hybrid dragonflies.
Key results:
o The topology with best cost/performance depends on the size.
o The fat-tree gives the best cost/worst-case-blocking
performance.
o The dragonfly is almost as good as the fat-tree when
considering cost/average-blocking performance.
Remember:
o The dragonfly require support for adaptive routing and
congestion look ahead for optimal behaviour, this is not
supported by any existing hardware.
o The dragonfly requires multiple virtual channels for deadlock
avoidance.