Sharing experiences from the EU

Code of Conduct

Alan Real and Jon Summers

22nd November 2012

Team project looks at EU CoC

DESIGN AND OPERATION OF A GREEN

DATA CENTRE

Tristan Owen, Gavin Waite, Fei Hou Lim,

Wee Yeh Tai

Project ran in 2010-2011

1 MW facility.

Dimension 85 analysis

Aims to improve efficiency and aid organisation

Attractive front end

Provides an EU Code of Conduct breakdown

However, could be carried out in house

£3,500 possibly overpriced

The five best practices required are:

3.1.1 Group Involvement

9.3.1 Written report

• 5.2.4 Review of cooling strategy

• 5.3.1 Review and if possible raise target

IT equipment intake air temperature

• 5.3.4 Review set points of air and water

temperatures

Unravelling the EU CoC

Conclusion and

Recommendations

Hot Aisle Containment

Inlet Temperature

– Min, 23 °C, Ideally, 26/27 °C

Bull’s updated compliance with EU Code of Conduct:

– 3.1.1 Group Involvement (Managers, Technicians, HR, etc.)

– 9.3.1 Written report (Annual)

Future work

– CFD, Bull, Economisers

Dense computing@Leeds

– Would give us headroom to accommodate additional equipment

from grants

– Also room to accommodate second phase.

Not just about most efficient estate usage:

– Share as many power supplies as possible

– Aim to operate the supplies at their most efficient loading

– Need to cool as close to the source of heat as possible

»On chip/within rack

– Cannot let hot and cold air mix.

Satisfies green-drivers/datacentre good practice also

– An upgrade of the A/C would not achieve this.

– About managing hot air, rather than providing enough cold!

Rear Door Cooling

There are currently two primary types of rack door

cooling systems available - passive and active.

‘Passive’ Rear Door Cooling – Phase 1

– Reliance on the server fans to provide air movement

– Proprietary CHW cooling doors specific to racks

– Removal of heat at source

– Low maintenance

– Servers/chassis aware of door

Active Rear Door Cooling – Phase 2

– Integral fans within rack door to provide air movement

– Door mating frame enables cooling doors fitting to various rack types

– Interface facility – BMS monitoring

– No integration with servers/chassis

Both are designed

to cool 32kW/rack.

@3,400CFM

3rd November 2010

8

High Density racks with back

door coolers

Convective Heat Transfer in door

)( airrefbdc TTAUhmQ

71.0UUh

CFD by Ali Almoli, PhD Student

Experiments by Adam Thompson, PhD student

Tang (2009)

The Solution

CHW Solution

13/18 F&R Temps

Free Cooling

Packaged duty/standby chiller(s)

Resilient pumping

Rack door cooling

coils

Site Photographs cont.

Data Centre – Phase 1 Works

Site Photographs cont.

Chiller Plant Installations - Roof

Performance

Due to installation of new HPC

»300kW of compute and associated cooling have been

decommissioned

»Saving around 200k per anum.

»Reduction in compute capacity of: 5 Tflops

– Initial system 20Tflops in 3 racks (96 servers per rack)

– Upgraded system delivers 45Tflops in 6 racks

– Upgraded system consumes

– 137kW (104kW compute, 33kW cooling)

– Annual running cost £90k (>50k savings over traditional cooling)

Phase 3 expansion 2012

N8 HPC centre of excellence

▫ Expanded to accommodate

12 33kW racks

▫ 6-week accelerated build

▫ 110Tflop expansion, 5 racks,

153kW

▫ Total capacity:

▫ 155Tflops in 11 racks

▫ 3 generations of equipment

▫ 360kW total load

▫ (260kW compute 100kW

cooling)

Alan – PUE of 1.38!

Site Photographs cont.

N8

installation

3M™ Novec™

High Thermal

Expansivity

Electrical

Insulator

Fire

Extinguishant

Environment

Clean, Safe &

Non- Toxic

Iceotope system

Iceotope system

• Low Pressure System

• Tier 4 Ready Design

• 2N Coolant to the Cabinet

• Hot Swap Clean and Dry

• Water to the Cabinet: 45C In 50C Out

Iceotope system layout

(We can take a look today!)