Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study...
Transcript of Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study...
![Page 1: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/1.jpg)
![Page 2: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/2.jpg)
Direct Contact Liquid Cooling on a Project Olympus Server
Cam Turner, P.Eng.CoolIT SystemsSr Program Manager, Hyperscale Solutions
Advanced Cooling Solutions
![Page 3: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/3.jpg)
• Case study objective
• Overview of the Project Olympus Server
• Specification of the components needed to implement a liquid cooling solution on the Project Olympus Server
• Test setup and constants
• Results from testing
• Changes required to support the harmonization effort
RACK & POWER
Agenda
Case Studies
![Page 4: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/4.jpg)
Determine the performance and efficiency improvements achieved by using Direct Contact Liquid Cooling on a Project Olympus 1U Universal Server as compared to air. RACK & POWER
Objective
Case Studies
Rack DCLC™
Project Olympus
![Page 5: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/5.jpg)
https://www.opencompute.org/wiki/Server/ProjectOlympus RACK & POWER
Project Olympus Server
Case Studies
Configuration:• 2x Intel(R) Xeon(R) Platinum 8168
CPU @ 2.70GHz• 24x 32GB DIMM• 1x PCIe
![Page 6: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/6.jpg)
RACK & POWER
Why The Project Olympus Server?
Case Studies
The Project Olympus Server was chosen for this case study because:
• It is a versatile server designed around modular building blocks
• It is an OCP Server Project
• It has an efficient air cooling solution
![Page 7: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/7.jpg)
Components Required for DCLC
coolitsystems.com
Passive Coldplate
Loop (PCL)
Rack Manifold
Coolant Distribution
Unit
![Page 8: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/8.jpg)
coolitsystems.com
Pre-filled, sealed and shipped
direct from CoolIT’s factory,
Passive Coldplate Loops are
mounted entirely within server and include:
▪ Passive Coldplate(s)
▪ Tubing
▪ Quick Disconnects
The Olympus Server is cooled using
several different coldplate
technologies…
• Microchannel Coldplate - CoolIT RX1 Server Modules for Intel Skylake CPU’s (x2)
• Direct Contact Memory Cooling - CoolIT MX4 DIMM Liquid Cooling Array
PCL Components
![Page 9: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/9.jpg)
coolitsystems.com
Pre-filled, sealed and shipped
direct from CoolIT’s factory,
Passive Coldplate Loops are
mounted entirely within server and include:
▪ Passive Coldplate(s)
▪ Tubing
▪ Quick Disconnects
The Olympus Server is cooled using
several different coldplate
technologies…
• Embedded Tube Coldplate - CoolIT designed custom low heat density coldplates
• Chassis blind-mate quick disconnects
PCL Components
![Page 10: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/10.jpg)
coolitsystems.com
Pre-filled, sealed and shipped
direct from CoolIT’s factory,
Passive Coldplate Loops are
mounted entirely within server and include:
▪ Passive Coldplate(s)
▪ Tubing
▪ Quick Disconnects
Treating the PCL as a black box, the
specification is as follows:
• Nominal Flow Rate: 1.0 L/min• Nominal Pressure drop: 4.5 PSI• Quick Disconnect Diameter: 3mm• Complies with TCS/DECS wetted material list
PCL Specification
![Page 11: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/11.jpg)
RACK & POWER
PCL Direction of Flow
Case Studies
![Page 12: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/12.jpg)
Manifold Specification
coolitsystems.com
Blind-mate ManifoldProject Olympus has a blind-mate PMDU, so blind-mate liquid connection is also required.
The manifold specification is as follows:• Nominal Flow Rate: 42.0 L/min• Nominal Pressure drop: 3 PSI• Quick Disconnect Diameter = 20mm• 25.4mm Nitrile Tubing• Complies with TCS/DECS wetted material list• Dimensions:
• 38mm square tube• 1905mm tall by 83mm wide
![Page 13: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/13.jpg)
coolitsystems.com
Blind-mate power and cooling
![Page 14: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/14.jpg)
CDU Specification
coolitsystems.com
CHx80 Rack-mount liquid-to-liquid CDU4U Rack-mount CDU capable of handling up to 42 Olympus servers. 2U is left open unoccupied for networking.
The CDU specification for this setup is as follows:• Nominal Flow Rate: 42.0 L/min• CDU External Pressure Differential: 7.5 PSI• Quick Disconnect Diameter = 20mm• Total power dissipated in Liquid = ~25kW• Approach Temperature = 5C @ W4, Qprimary=65 L/min• Complies with TCS/DECS wetted material list
![Page 15: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/15.jpg)
Constants:• Ambient air temperature = 35°C• Liquid flowrate = 1.0 L/min• Intel PTU Software recording CPU and Memory Temperature• Thermocouples recording liquid and air temperatures
Variables:Secondary fluid temperate ASHRAE W1, W2, W3, W4• W1 = 17 °C• W2 = 27 °C• W3 = 32 °C• W4 = 45 °C
RACK & POWER
Test Setup and Constants
Case Studies
![Page 16: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/16.jpg)
RACK & POWER
Results – CPU Core Temperature
Case Studies
![Page 17: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/17.jpg)
RACK & POWER
Results – Memory Temperature
Case Studies
![Page 18: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/18.jpg)
RACK & POWER
Margin to Max Operating Temp
Case Studies
Max Temp Margin to Max (°C)
CPU 28.5
Mem 26.2
Low Density 12.0
![Page 19: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/19.jpg)
RACK & POWER
Component Thermal Resistance
Case Studies
CPU = 0.09 C/W
CPU = X C/W
MEM = 0.117 C/W
Low Density = 0.415 C/W
![Page 20: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/20.jpg)
RACK & POWER
Server Power Savings
Case Studies
![Page 21: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/21.jpg)
RACK & POWER
Conclusion
Case Studies
Using the Project Olympus server as an example, CoolIT Systems Direct Contact Liquid Cooling solution was able to:
▪ Utilize W4 facility water
▪ Capture over 20 kW of heat in liquid per rack
▪ Reduce server power consumption by up to 4%, including pumping power
▪ Maintain a margin of at least 12 °C to maximum operating temperature
![Page 22: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/22.jpg)
RACK & POWER
Changes Required for Harmonization
Case Studies
Once the harmonization specification is complete, we anticipate the following changes will need to be made:
▪ Adopt the OCP Universal Quick Disconnects for both the PCL and the Manifold to CDU interface
▪ Validate the wetted material list against the OCP specification
▪ Validate performance with OCP specified transfer fluids
▪ Ensure operability within the PQ envelope specification
![Page 23: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/23.jpg)
RACK & POWER
Call to Action
Case Studies
This case study presents specifications required for CoolIT Systems to enable liquid cooling on a Project Olympus server.
For liquid cooling to reach wide scale adoption in hyperscale data centers, a harmonization effort is underway to enable interoperability between various liquid cooling solution vendors and technologies. That is the goal of OCP’s Advanced Cooling Solutions Coldplate workgroup.
Project Wiki with latest specification: https://www.opencompute.org/wiki/Rack_%26_Power/Advanced_Cooling_Solutions
Mailing list: https://ocp-all.groups.io/g/OCP-ACS-Cold-Plate
Please join the group and help develop the harmonization standards that will enable advanced cooling solutions for OpenCompute solutions.
![Page 24: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/24.jpg)
RACK & POWER
Questions?
Case Studies
Stay connected with us:
www.coolitsystems.com
www.linkedin.com/company/coolit-systems-inc.
@coolitsystems
www.facebook.com/coolitsystems
www.youtube.com/user/coolitsystems
![Page 25: Direct Contact Liquid Cooling on a Project Olympus Server.… · 2019-04-03 · • Case study objective • Overview of the Project Olympus Server • Specification of the components](https://reader035.fdocuments.in/reader035/viewer/2022081521/5e8511cb90da694c246c015c/html5/thumbnails/25.jpg)