Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid...
Transcript of Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid...
by Peter Hopton, Fabien Chaix, e.a.; Iceotope, UK; and FORTH, Heraklion, Crete.
Compact Packaging and Liquid Cooling Technology for Exascale
For the Purpose of Dissemination Funding Allocation; “The work described in this ppt has been conducted within the project EuroEXA. This project has received funding from the European Union’s Horizon 2020 (H2020) research and innovation programme under the Grant Agreement no 754337. This ppt and the content included in it do not represent the opinion of the European Union, and the European Union is not responsible for any use that might be made of its content.”
Including Achievements from ExaNeSt and Concepts and Achievements from EuroEXA
M.2(NVMe)512GByte
4PS-GTR
FPGAXCZU9EG-ffvc900
16GByteDDR4
SODIMM
64
64MBQSPI
‘Network’ FPGAXCZU9EG-ffvc900
Connector
10GTH
FPGAXCZU9EG-ffvc900
12+12LVDS
‘Storage’ FPGAXCZU9EG-ffvc900
2GTH
12+12LVDS
2GTH
12+12LVDS
2GTH
12+12LVDS
2GTH
12+12LVDS
2GTH
16GByteDDR4
SODIMM
64
16GByteDDR4
SODIMM
64
16GByteDDR4
SODIMM
64
RGMII2xUART
64MBQSPI
64MBQSPI
64MBQSPI
TheQFDBarchitecture
The QFDB implementation
Rev. A versionàDDR issue (erroneous
swap)à48Và12V regulator issue
Rev. B versionà48Và12V regulator issueàSSD (configuration) issue
Key factsDimension 130 x 120 mmHeight < 30 mmPCB 16 layers, Megtron-6Estimated TDP 100~120W
M.2(NVMe)512GByte
QFDBI2CSensorsnetwork
ExaNeStConfidential4
FPGAXCZU9EG-ffvc900
Connector
FPGAXCZU9EG-ffvc900
‘Storage’ FPGAXCZU9EG-ffvc900
16GByteDDR4
SODIMMMTA18ASF2G72HZ-2G3x4
12VSens.
VCCINTLPSens.
VCCAUX18Sens.
Clockgen.Si5341
VCCINTFPSens.VCCINTSens.
PSDDR504Sens.
16GByteDDR4
SODIMMMTA18ASF2G72HZ-2G3x4
16GByteDDR4
SODIMMMTA18ASF2G72HZ-2G3x4
16GByteDDR4
SODIMMMTA18ASF2G72HZ-2G3x4
‘Network’ FPGAXCZU9EG-ffvc900
VCCINTFPSens.VCCINTSens.
PSDDR504Sens.
VCCINTFPSens.VCCINTSens.
PSDDR504Sens.
VCCINTFPSens.VCCINTSens.
PSDDR504Sens.
PS_I2C0+PL0
PL2
PL1PL1
PL1PL1
PS_I2C1
PS_I2C0+PL0
PS_I2C0+PL0
PS_I2C0+PL0
ThermalSensor
2.5V
3.3V
1.8V
Top-levelmodule
Bottom-levelFPGAmodule
Current QFDB AchievementsFeature StatusZynq functions Validated individually,DDR-RAM Validated
Linux runs with DDR4-2133Bare-metal test pass with DDR4-2400
QSPI memories Validated on LinuxMinor issues with bare-metal
Gigabit Ethernet ValidatedLVDS links Validated, ~5% of traces are problematicI2C busses Validated all but the M.2 deviceUART links ValidatedClock generators ValidatedM.2 Solid State Disk Validated on Rev. A,
Remote RAM Linux setup• We need Linux to validate board sub-systems:
• PCIe, QSPI, Gigabit Ethernet• Successful test of external RAM to boot.è Accessed memory from another boardè Forward memory access through Network FPGA
ExaNeSt Tracks
Track 1• 4 QFDBs• 800W Max.• 1.5 per u (b2b)• 45C Inlet• Passive I/O
Track 2• 16 QFDBs• 3.2kW Max• 2 per u (b2b)• 45C Inlet• Onboard Switch
QFDB x Track-1 Mezzanine
EuroEXA Concepts
Aisle
Aisle
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
Aisle
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
CoolantandDCPower
Aisle
Aisle
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
Aisle
2.5m ICEOTOPE
CONFIDENTIAL
BatteriesandRectifiersfor250kWofLoad
BatteriesandRectifiersfor250kWofLoad
CoolantandDCPower
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
600x60
0mm
cabine
twith
6
chassis&2xCD
Us
CoolantandDCPower
BatteriesandRectifiersfor250kWofLoad
BatteriesandRectifiersfor250kWofLoad
CoolantandDCPower10.6m
Wall
Wall
2MW In 40ft ISO or 10mx2.4m
• Shipping Container Optimised Cabinets (2 rows)• 2MW per shipping container• Enables 3D stacking to minimise distances
Advancing to Track 2
• >4x Thermal Improvement Required• 7x Thermal Improvement Targetted• Addition of an integrated high speed switch• Large thermal and mechanical challenge
What This Means For Track 2Key facts
Dimension 21” Rack x 762mm Deep
Nodes 16 x QFDB (or Next Gen)
Height 1 O-uPCB Megtron-6Estimated TDP 3.2kWI/O (Uplink) 800Gb/sI/O (Total) 2Tb/s
Tackling the 7x Cooling Challenge
Proving it will work!Scope• Track 2 horizontal chassis• Thermal proxy QFDBs x16, nominal
TDP 200W (3.2kW overall)
• Critical conditions• Junction temperature < 100oC• Maximum PC temperature < 70oC
• Three tests defined• Critical SC inlet temperature• Critical TDP duty• Critical SC flow-rate
Proving it will work!
Results at 45C Inlet
Proving it will work!
Results at 45C Inlet
Thermal Validation SummarySummary
• Based on stated nominal conditions The following critical conditions were determined
• SC feed temperature is 47oC• TDP load factor is 1.1 (1,084W)• SC flow-rate is 1.6
• In each case the Primary Coolant max temperature was found to be the limiting factor
• These findings still need further verification experimentally
ManyThanksPleaseConnectatthecoolingguy.me
Presenter– PeterHoptonDisseminationLead,EuroEXA,ExaNeSt
Founder,Iceotopehttp://thecoolingguy.me
© 2017 EuroEXA and ExaNeSt Consortia Member Rights Holders Project ID: 754337