Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM...
Transcript of Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM...
![Page 1: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/1.jpg)
IBM Research – Zurich
© 2011 IBM Corporation
Hotwater Cooling for Energy-Hungry Datacenters
Towards a Zero – Emission Datacenter
Dr. G. Ingmar MeijerIBM Research – [email protected]
![Page 2: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/2.jpg)
2 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Outline
1. IntroductionA. Datacenter Market Drivers and Trends B. Energy Consumption Issues
2. Hotwater Cooling; Concept3. Prototype High-Performance Computer (“First-of-a-Kind”)4. IBM System x iDataPlex Water Cooled dx360 M4
A. Direct Water CoolingB. .Power & Performance
5. Summary
![Page 3: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/3.jpg)
3 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
1.A. Datacenter Market Drivers and Trends
Total cost of ownership and environmental footprint
Servers used 330 TWh of electrical energy: €25bn or 2% of the electricity production (2009).
ICT industries carbon emission as aviation.
![Page 4: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/4.jpg)
4 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
1.B. Energy-Consumption Issues (and Thermal Issues): propagate through hardware levels
![Page 5: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/5.jpg)
5 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Energy Consumption of Transistor (Leakage Current)
![Page 6: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/6.jpg)
6 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Energy Consumption of Datacenter
Air-cooled datacenters are inefficient. Typical cooling needs as much energy as IT equipment and both are thrown-away.
Provocative: datacenter is a huge “heater with integrated logic.”
For a 10 MW datacenter €2m–€3m is wasted per year.
APC, Whitepaper #113 (2010)
![Page 7: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/7.jpg)
7 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
2. Zero – Emission Datacenter Concept
CMOS 80ºC
a. Micro-channel liquid coolersb. Heat exchangerc. Waste-heat utilization
Water 60ºC
![Page 8: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/8.jpg)
8 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Energy and Emission of Conventional Datacenter
Power Station
$$CO2
Low-grade heat Chiller Heat wasted
![Page 9: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/9.jpg)
9 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
$CO2
Zero – Emission Datacenter (CO2 < 15 % net)
Heat re-used
CO2
Medium-grade heat
$
Power Station
![Page 10: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/10.jpg)
10 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Comparison: Water Cooling vs. Air CoolingWater Cooling:
T ~ 50 – 100 KT ~ 5 – 10 K
Air Cooling:
![Page 11: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/11.jpg)
11 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Energy consumptionreduced by
up to 40%
Direct use of waste heatcuts CO2 emissions by
up to 85%World record in facility-level
efficiency in TFLOP/gCO2
7.9
3. Zero – Emission Datacenter Prototype HPC Cluster at ETH Zurich, “Aquasar”
![Page 12: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/12.jpg)
12 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Water-Cooled IBM BladeCenter QS22
![Page 13: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/13.jpg)
13 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Effective Computing Power 1.4 GFLOPS/W
![Page 14: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/14.jpg)
14 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
14
4. Leibniz Rechenzentrum, Garching, GermanySuperMUC: Warm-Water Cooled 3 PFLOPS System
1Q12—2Q12: ~10000 IBM System x iDataPlex Water Cooled dx360 M4
![Page 15: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/15.jpg)
15 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Hybrid Datacenter w/ Direct Water Cooled Nodes
Highly energy-efficient hybrid-cooling solution:– Compute racks
– 90% Heat flux to warm water– 10% Heat flux to CRAH
– Switch / Storage racks– Rear door heat exchangers
Compute node power consumption reduced ~ 10% due to lower component temperatures and no fans.
Power Usage Effectiveness PTotal / PIT: PUE ~ 1.1
Heat recovery is enabled by the compute node design: Energy Reuse Effectiveness (PTotal – PReuse) / PIT: ERE ~ 0.3
![Page 16: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/16.jpg)
16 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
IBM System x iDataPlex Water Cooled dx360 M4
Heat flux > 90% to water; very low chilled water requirement.
Power advantage over air-cooled node: warm water cooled ~10% (cold water cooled ~15%) due to lower Tcomponents and no fans.
Typical operating conditions: Tair = 25 – 35°C, Twater = 18 – 45°C.
![Page 17: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/17.jpg)
17 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
5. Main Messages / Outlook
Technology exists to design Zero – Emission Datacenter:– Hot-water cooling & Waste heat utilization
Business value / opportunity – Energy cost reduction– Performance increase – Low-carbon pioneers “world-record TFLOP/gCO2”
What is next?IBM System x iDataPlex Water Cooled dx360 M4
Supercomputing SC11 Seattle, November 12-18, 2011:– Node at IBM booth
![Page 18: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/18.jpg)
18 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
IBM Team
Aquasar– IBM Research – Zurich
– IBM Germany Research & Development
IBM System x iDataPlex Water Cooled dx360 M4 and SuperMUC– IBM Systems & Technology Group
– IBM Research – Zurich
– IBM Germany
![Page 19: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/19.jpg)
19 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
Roadmap: 3D CMOS integration and Heat Removal
3D CMOS integration requires (scalable) interlayer liquid cooling
Challenge: isolate electrical interconnects from liquid Experimental heat removal on test vehicle 180 W/cm2 per layer
(extrapolated 7.2 kW from 10 layers with 4 cm2)
![Page 20: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/20.jpg)
20 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich
![Page 21: Hotwater Cooling for Energy-Hungry Datacenters Workshop · IBM Research – Zurich © 2011 IBM Corporation Hotwater Cooling for Energy-Hungry Datacenters Towards a Zero – Emission](https://reader030.fdocuments.in/reader030/viewer/2022011902/5f0e24397e708231d43dcf6f/html5/thumbnails/21.jpg)
21 Hotwater Cooling for Energy-Hungry Datacenters | Towards a Zero – Emission Datacenter © 2011 IBM Corporation
IBM Research – Zurich