Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks

Felix Zahn, Pedro Yebenes, Steffen Lammel, Pedro J. Garcia, Holger Fröning

2nd IEEE International Workshop on High-Performance

Interconnection Networks in the Exascale and Big-Data Era

Barcelona, 12th March, 2016

Does  network  power  ma.er  at  all?  

§ Pitfall: don’t make assumptions based on maximum power ratings •  At TDP, processors

outshine anything •  But are processors always

operating at 100% load? •  Energy-proportional: at

x% load, a component consumes only x% power

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 1

It  does!  

§ System power •  Scalable energy-efficient network •  Direct network, integrated switches

§ Dynamic range of components § Many memory-bound applications

•  E.g., emerging integer applications (R. Murphy, Sandia) & graph computations •  DFS & BFS •  Connected Components •  Isomorphism •  Shortest Path •  Graph Partitioning •  BLAST (alignment search) •  zChaff (satisfiability)

§ Exception: compute-bound applications with perfect overlap

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 2

Mo5va5on  

§ Serialization technology dominates power consumption •  Clock recovery, high frequency, equalization, pre-emphasis, …

§ It is link width that matters, not frequency •  CML = Current Mode Logic •  Linear scaling for 10GHz case •  Frequency dependent part is CMOS only

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 3

71%$

14%$

15%$

Power&share&for&NIC&with&integrated&switch&

Links$(6)$

PCIe$

Core$

0.0#

0.5#

1.0#

1.5#

2.0#

2.5#

3.0#

3.5#

4#Lanes# 8#Lanes# 12#Lanes#

Power&con

sump-

on&[n

ormalized

]&

Link&power&scaling&

2.5#GHz#

5#GHz#

10#GHz#

Energy-­‐propor5onality?  

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 4

Energy-­‐propor5onality  vs.  today’s  interconnects  

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 5

Energy-­‐propor5onality  –  today’s  possibili5es  

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 6

§  Energy/data: 34.4 pJ/bit (65nm TSMC-produced Serializers only)

Concept  

§ Application-dependent potential for energy saving • Best case: no transition time, links switched on/off depending on whether they are busy or idling

• Worst case (common case today): all links run with full power

§ Setup: OMNeT++-based simulator

• 3D Torus topology • 64 nodes • XYZ-dimension-order routing

7 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

Workloads  -­‐  NAMD  

§  Satellite Tobacco Mosaic Virus (STMV), 64 MPI tasks

8 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

Workloads  –  Graph500  

§ scale factor = 12, edge factor = 16, 64 MPI tasks

9 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

Results  –  Link  inac5vity  

10 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

§ Execution time: NAMD 3449 ms, Graph500 95.2 ms

Results  –  U5liza5on  

§ Link utilization is highly volatile due to: • Application • Dimension • Routing algorithm

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 11

Results  –  Idle  5me  

12 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

6%   2%  

Results  –  Power  saving  poten5als  

13 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

20%  24%  

Discussion  

§ Network power matters • Today’s interconnection networks contribute up to 20% to the total power consumption of large computing systems

§ Interconnects are highly energy dis-proportional • But they provide the opportunity to implement power saving strategies

§ Huge potential power savings • An optimal and fully energy-proportional NIC would save about 75% of the network power

14 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

Outlook  

§ Understanding energy consumption • Power saving possibilities depend highly on workloads • Communication for HPC applications is complex to model => trace based network simulation

§ Analysis of different hardware parameter • Transition time, granularity of link width, etc. => Useful input on design decisions for future hardware § Network energy model

• Predicting energy consumed by the network based on communication characteristics

Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016 15

Thank  you!  Questions?

16 Analyzing the Energy (Dis-)Proportionality of Scalable Interconnection Networks - Felix Zahn - HiPINEB workshop - 12th March, 2016

Credits  Discussions:  Maximilian  Thürmer,  Markus  Müller,  Benjamin  Klenk,  Alexander  Matz,  Daniel  Schlegel  

(Heidelberg  University),  Sébas5en  Rumley  (Columbia  University),  Francisco  Andujar,  Jesus  Escudero,  Juan  

Villar  (Universidad  de  Cas5lla-­‐La  Mancha)    

Current  main  interac4ons