Energy Efficient Server Rooms at the

University of Cambridge

David Green

dsg1000@eng.cam.ac.uk

Department of Engineering

Presentation Overview

2

• Electricity Incentivisation Scheme (EIS) at the University

of Cambridge

• Design of Engineering’s Data Centre cooling system

• Energy use from 2010 onwards

• Next steps

The Electricity Incentivisation Scheme (EIS)

3

• Financial incentives to use

electricity more efficiently

• Annual allowances at departmental

level

• Financial reward if use less than

allowance

• Financial penalty if exceed

allowance

• Implemented 1 August 2008

• Energy & Carbon Reduction Project

In 2010/11 electricity

usage was 4.4% below

target, saving:

• £0.51 million

• 4,950 MWh

• 2,678 tonnes CO2

Department of Engineering Overview

4

• Accounts for around 10% of

university.

• Activities based in 7 buildings.

• Around 600 members of staff

• Four year M.Eng course – around

1,200 students.

• Postgraduate students numbers:

• 2011 (792) - 2012 (830)

5

Server Room Cooling Project - Introduction

• The Problem

• Increase cooling capacity to support future purchases

• Minimise all aspects of running costs and carbon footprint

• The Solution

• Review cooling arrangements, expand and consider options

• Alternative approach to cooling

• The Results

• PUE of 1.1

6

The Problem

Background

• Initially a distributed arrangement.

• Centralised computing resources in two computer rooms (34 racks,12 racks)

Pre 2010 Cooling Arrangement

• Refrigerant based CRAC system, full recirculation via under floor plenum

• 63kW plug-load

Key Project Drivers

• University Energy Incentivisation Scheme (EIS)

• Further server purchases planned

• IT electricity consumption is a significant part of the Department’s energy base load

Approach

• KJ Tait feasibility study

• Support from the University’s Estate Management

• Computing Staff

• Salix Funding

7

The Problem - Server Room Cooling Project

� Drive to reduce energy costs and carbon footprint

� Consolidation of server rooms

� Power management

� Existing DX cooling equipment could not cope with future plans

� To implement a solution in a live data centre

8

The Problem - Server Room Cooling Project

9

The Solution – Options Considered

• Cold Aisle

containment

• Increasing existing

CRAC capacity

• In-rack cooling with

chilled water

• Evaporative Cooling2

6 10 14 18 22 26 30 34 38

0

25

50

750

5

10

15

20

25

30

air

temp

air RH

10

The Solution - Air Flow

11

The Solution – Temperature

Maximum

Temperature 24C

12

The Solution – Temperature and Flow

Data Rack

Damper

Constant flow and

temperature

Evaporative Cooling

13

The Solution – Temperature

Ventilation plus

attemperation

Evaporative cooling

plus attemperation

Evaporative Cooling

14

The Solution – Design Running Cost

15

The Solution - Installation

Cold aisle

containment

6 EcoCooling

CREC’s giving

150kW N+1

EC Extract Fans

Ambient air

through louver

Self contained

plant room No raised floor

16

The Solution - Installation

The Solution - Installation

17

Mechanical

Cooling Plant

Data Centre (34

racks – 150kW)

Electrical supply

distribution and

metering

18

The Results – Key Points

• System has been operational

since December 2010

• IT load has risen from 63kW to

95kW

• Mix of low and medium density

servers

• Update of air filtration and

humidity control.

• Ambient conditions exceeded

30C with high RH

• Cold aisle did not exceed 25C

• Max RH 70%

• PUE 1.1 over 2 1/2 years

• Annual savings 200 tonnes

carbon and ~£40K

• Some fan and equipment failures

• Some visible dust

19

June 2010 - kWh used per day, per consumer unit

79

1

81

0 86

6

84

0

83

7

83

7

82

3 85

2

86

6

82

6

85

1

85

0

85

0

83

1

78

1 84

8 90

8

86

3

63

3

62

4

77

2

14

92 1

56

5

15

87

14

93

14

57

14

74

14

58

15

68

15

74

15

06

15

25

15

25

15

25

15

46

12

84

16

01

15

84

15

28

74

5

83

0

10

69

0

200

400

600

800

1000

1200

1400

1600

1800

01

/06

/20

10

02

/06

/20

10

03

/06

/20

10

04

/06

/20

10

05

/06

/20

10

06

/06

/20

10

07

/06

/20

10

08

/06

/20

10

09

/06

/20

10

10

/06

/20

10

11

/06

/20

10

12

/06

/20

10

13

/06

/20

10

14

/06

/20

10

15

/06

/20

10

16

/06

/20

10

17

/06

/20

10

18

/06

/20

10

19

/06

/20

10

20

/06

/20

10

21

/06

/20

10

22

/06

/20

10

23

/06

/20

10

24

/06

/20

10

25

/06

/20

10

26

/06

/20

10

27

/06

/20

10

28

/06

/20

10

29

/06

/20

10

30

/06

/20

10

kW

hr

Air-con Units kWh used

Racks Units kWh used

The Results – Energy Use 2010

IT Load

~63kW

Cooling and

Lighting

~35kW

• `

20

June 2010 - kWh used per day, per consumer unit

79

1

81

0 86

6

84

0

83

7

83

7

82

3 85

2

86

6

82

6

85

1

85

0

85

0

83

1

78

1 84

8 90

8

86

3

63

3

62

4

77

2

14

92 1

56

5

15

87

14

93

14

57

14

74

14

58

15

68

15

74

15

06

15

25

15

25

15

25

15

46

12

84

16

01

15

84

15

28

74

5

83

0

10

69

0

200

400

600

800

1000

1200

1400

1600

1800

01

/06

/20

10

02

/06

/20

10

03

/06

/20

10

04

/06

/20

10

05

/06

/20

10

06

/06

/20

10

07

/06

/20

10

08

/06

/20

10

09

/06

/20

10

10

/06

/20

10

11

/06

/20

10

12

/06

/20

10

13

/06

/20

10

14

/06

/20

10

15

/06

/20

10

16

/06

/20

10

17

/06

/20

10

18

/06

/20

10

19

/06

/20

10

20

/06

/20

10

21

/06

/20

10

22

/06

/20

10

23

/06

/20

10

24

/06

/20

10

25

/06

/20

10

26

/06

/20

10

27

/06

/20

10

28

/06

/20

10

29

/06

/20

10

30

/06

/20

10

kW

hr

Air-con Units kWh used

Racks Units kWh used

The Results – Energy Use 2011

June 2011 - kWh used per day, per consumer unit

98

96

96

96 10

9

10

3

10

6

96

96

97

98

10

3

12

8

15

9

16

8

18

0

17

8

17

5 26

0

17

5

17

0

23

15

22

94

22

32

21

90

22

54

22

98

23

56

23

25

22

21

23

00 23

93

22

55

22

37

21

91

22

03

22

20

22

39

22

79

22

10

20

58 2

18

4

0

500

1000

1500

2000

2500

3000

01

/06

/20

11

02

/06

/20

11

03

/06

/20

11

04

/06

/20

11

05

/06

/20

11

06

/06

/20

11

07

/06

/20

11

08

/06

/20

11

09

/06

/20

11

10

/06

/20

11

11

/06

/20

11

12

/06

/20

11

13

/06

/20

11

14

/06

/20

11

15

/06

/20

11

16

/06

/20

11

17

/06

/20

11

18

/06

/20

11

19

/06

/20

11

20

/06

/20

11

21

/06

/20

11

22

/06

/20

11

23

/06

/20

11

24

/06

/20

11

25

/06

/20

11

26

/06

/20

11

27

/06

/20

11

28

/06

/20

11

29

/06

/20

11

30

/06

/20

11

kW

hr

Air-con Units kWh used

Racks Units kWh used

PUE of 1.1

PUE of 1.65

21

Design Development 2012 onwards – temperature,

humidity & air quality monitoring

• Enhanced filtration and air

quality monitoring

• Humidity limiting control

algorithm and web interface

• Fan updates and flow

dampers

• Low levels of equipment

failure

• Hosting from other

university departments

• Fire suppression

The Results – 2013 energy use

22

23

The Results – 2013 temperature & humidity logs

24

Initial Results – Contamination and Server Failure

• Initially limited filtration, now extensive and multi staged.

• Some visible dust and black particulates.

• Basic analysis showed the particulates to consist of dust, possibly pollen particles and diesel engine exhaust particulates.

• There has been a small number of fan failures on servers but this is difficult to directly attribute to the cooling system.

25

The Results – Reliability and Maintenance

• Initially maintenance was not

comprehensively scheduled

• Location - surprising amount of

large fibres caught by insect

screen in the Spring

• 3 monthly maintenance of the

equipment is required

• Routine ‘deep’ cleaning of facility

to ISO 7

• With internal installation room

cleanliness needs to be

maintained

Visibility of Building Performance - Energy Dashboard

26

• Visibility of actual building performance.

• Digital signage.

• Encourage individuals to ‘own’ and take responsibility.

• ‘Buy-in’ now apparent in some equipment purchases.

• Individual racks are metered

Engineering’s Data Centre electrical loads

• 300 MWh electrical base load

• Pre 2010 – 35% = Server rooms

• Now 2 x Data Centres and 23% of

base load

• Purchasing vs energy performance

27

28

Summary

• Evaporative cooling has resulted in

significant energy and carbon savings

• Second Data Centre in Engineering is

now also based on this technology

• Interest from academic and commercial

sectors

• Catalyst for good practice in terms of

energy and carbon reduction

• Option for hot air exhaust use in natural

ventilation strategy – purge/enhance

stack ventilation strategy.