Energy Efficient Data Centers - IBM · - Design flexibility: Sensible heat transport to locations...

IBM Systems and Technology Group Lab Services

Keeping our clients in the race. © 2006 IBM Corporation© 2006 IBM Corporation

Energy Efficient Data Centers

Steven AhladasIBM [email protected]


© 2006 IBM Corporation2 Keeping our clients in the race.

Trends and Best Practices

© 2006 IBM Corporation

All trends going the same way ….IT Servers were an asset, tend to be a commodityEnergy was a commodity for servers, tend to be a cost

But…..Data Centers energy cost now showsAppear as a critical limitation to server extension in the data center

Heat increase can be a growth killer by stopping introduction of new servers (DataCenter rooms limitations)

Energy economics around IT servers must be rebalanced with new energy-conscious criteria to stop cost increase

– 1) Stop the trend– 2) Reduce power consumption

$0

$20

$40

$60

$80

$100

$120

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Installed Base(M Units)

Spending(US$B)

New Server SpendingPower and Cooling

0

5

10

15

20

25

30

35

40

45

50

IDC Presentation, The Impact of Power and Cooling on Data Center Infrastructure, Doc #201722, May 2006





Energy/Performance Metrics

Perf(capacity for storage) Power Perf/Space Perf/Watt0

10

20

30

40

50

60

70

80

Ann

ual C

hang

e(%

)

High End z xSeries 1U p550 High End Storage

Server/Storage Performance/Energy Trends

Server performance increasing much faster than server power

Source: Roger Schmidt





Comparison of typical server utilization rates

UNIX x86Mainframe

Used

Wasted





System 1

APP

1

APP

2

10% busy1KW

System 2

AP

P 3

APP

4

10% busy1KW

Advanced Virtualization

APP

5

APP

6

70% busy4KW

APP

1

APP

2

AP

P 3

APP

4

Energy Management Policy ExampleServer Consolidation Conserves Energy

System 8

APP

7

APP

8

10% busy1KW

APP

7

APP

8…

Server consolidation exploiting virtualization is the most effective tool in reducing energy costs

Total Power 8KW Total Power 4KW





• Virtualization was first introduced by IBM in the 1960s to allow the partitioning of large mainframe environments. System z virtualization remains the gold standard today.

• IBM SAN Volume Controller (SVC). for storage virtualization increases storage utilization rates by 30% while reducing the growth rate of storage capacity by 20%.

• VMware, Microsoft® Virtual Server and Xen offerings are available for IBM System x™ and IBM BladeCenter®

Cool Blue: Virtualization on IBM Systems and Storage

Annual Power Cost per Logical Server0 $100 $200 $300 $400 $500 $600 $700 $800 $900 $1,000

$920.70

$353.14

$550.06

$207.57

$127.23

$73.06

$32.80

Dell PowerEdge 1950

Dell PowerEdge 1950 virtualized

Dell PowerEdge 1955

Dell PowerEdge 1955 virtualized

IBM System z™BC S07 Small

IBM System z BC S07 Expanded

IBM System z EC S18

Source: “Avoiding the $25 Million Server Data Center Power, Cooling, and Growth” report by Sine Nomine Associates, 4/6/07

“This shift has eliminated one floor of servers, cut power and cooling costs by 80 percent, and reduced the administrative staff (and costs) by 10 administrators. No other platform has clearly demonstrated the vertical scaling (scale-up) savings as the mainframe.”





Cool Blue: Storage virtualization

Pool and share I/O Resources for:– Improved utilization

– Reduced energy needsCombined utilization

50%

EMCIBM HPIBM IBMHitachi

SAN Volume Controller Combined utilization

90%





Cool Blue: Deploy efficient IBM System Storage

10 year TCO example using blended tape and disk best practices—Deploy more power efficient storage—Utilize storage more efficiently

250TB of storage25% growthover 10 years





Reduce Consumption Through ProvisioningDe-provision Standby Servers

Standby Servers

125 Servers

Active Servers

875 Servers

Provisioned

De-provisioned

Power SavingsIn Standby Mode





Energy Efficient Data Centers

Data Center Power Usage Effectiveness

0

2

4

6

8

10

12

1.53 1.81 2.10 2.38 MoreBuilding / IT Eqt Power

Num

ber o

f Dat

a C

ente

rs

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

http://www-03.ibm.com/systems/optimizeit/





Computer Loads38%

UPS Losses6%

Lighting2%

HVAC54%

Total Power = 580 kW

UPS Losses13%

HVAC - Air Movement

9%HVAC -

Chilled Water Plant14%

Lighting1%

Computer Loads63%

Total Power = 1700 kW

Data Center A: 54 % HVAC ; Data Center B: 23 % HVACData Center A could possibly save ~ 180 kW “ Those that did many things well still had room to improve…”

DATA CENTER ENERGY EFFICIENCIES

From LBNL DC energy study, W. Tschudi

DATA CENTER A DATA CENTER B





TYPICAL RAISED FLOOR DATA CENTERS

GOAL: provide specified inlet temperatures to every IT Rack as efficiently as possible





cold aislehot aisle z=0.5 feetz=1.5 feetz=2.5 feetz=3.5 feetz=4.5 feetz=5.5 feet

Hot spot at long aisle

z=6.5 feetz=7.5 feetz=8.5 feet

3D Temperature distributions of IBMsupercomputer

13.01oC

54.47oC

33.74oC

yz

x

Measured 3D temperature maps of a datacenter





0 1 2 3 4 5 6 7 8 9

15

20

25

30

35

40

45

inle

t ai

r te

mpe

ratu

re T

inle

t [o C]

height [feet]

1 2 3 4 5 6 7 8 9 10 11 12

upper specs.

Verticle Temperature Gradients





COOLING EFFICIENCY

#1: INTERMIXING between hot and cold air increases locally Inlet Air Temperatures( excessive low CRAC discharge temperatures; affects Cooling Production Efficiency)#2: RECIRCULATION causes small differences in Return and CRAC Discharge Temperatures( CRAC units are not fully utilized; affects Cooling Delivery Efficiency)





- in many places perf. tiles do not supportfor the required cooling capacity

- get high throughput tiles (50 % open)- remove dampers to enhance air flow

(300 cfm ~ 2 kW)

PERFORATED TILES

56% open,with damper

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0 500 1000 1500 2000 2500 3000

Airflow, CFM

Stat

ic P

ress

ure,

in H

2O

Tate GrateAire 56%

Tate GrateAire 56% w/100% Open Damper

56% open,no damper





UNDER-FLOOR BLOCKAGE

- avoid any blockage in the critical path to perforated tiles- put cabling etc. under hot aisles- possibly deploy under-floor boundaries

- limit underfloor airflow to the boundaries of the zone to be cooled- escaping air results in lower underfloor air pressure and room airflow- use under-floor barriers to limit the airflow to the needed places

cables water pipesExamples for blockage





- block cable cut-outs using the methods shown below- add perforated tiles & use higher throughput tiles- try to have good sealing between the tiles

MatsBrushes Bags Pillows

LEAKS AND CABLE CUTOUTS





LEAKAGE in Sample Raised Floors#

CRACs

total flow/cooling

capacity1

#

perf. tiles

total perf. tile

flow/cooling capacity1

total ceiling tile

flow/cooling capacity1

Not-targeted

flow/cooling2

Total

HW / Room Power

Margin

29500 cfm

200 kW

58 tons

11000 cfm

73 kW

21 tons

#3 17 84000 cfm

560 kW

162 tons

24 6000 cfm

40 kW

12 tons

NA 78000 cfm

520 kW

151 tons

(93 %)

175/230 kW

51/68 tons

38000 cfm

250 kW

72 tons

(57 %)

330 kW

96 tons

(143 %)

46.5 kcfm

310 kW

90 tons

31700 cfm

211 kW

61 tons

(74 %)

90 kW

26 tons

(25 %)

300/360 kW

87/104 tons

110/140 kW

32/41 tons

#4 35 194 kcfm

1294 kW

375 tons

185 NA 137.5 kcfm

918 kW

266 tons

(71 %)

585/730 kW

170/212 tons

145 kW

42 tons

(103 %)

564 kW

164 tons

(77%)

1900 cfm

12 kW

3 tons

2000 cfm

13 kW

4 tons

#1 11 67500 cfm

450 kW

130 tons

118

#2 7 42700 cfm

285 kW

82 tons

43

1.) @ 60F (16oC) CRAC discharge temperature2.) Cable cutouts, other losses

H.Hamman 2/2007





Understanding Where Costs Hide is Critical Rear Door Heat eXchanger can remove Rear Door Heat eXchanger can remove over 50% of a rackover 50% of a rack’’s heat outputs heat output

– No new fans or electricity needed.

– Attaches to back of rack (adds 5”) No rearrangement of datacenter

– Cost effective; 1KW cooling = $286

The Cool Blue Heat eXchanger adds The Cool Blue Heat eXchanger adds cooling capacity at ~1/4 the cost of cooling capacity at ~1/4 the cost of traditional methodstraditional methods

Rear Door Heat Exchange

Front Cold

BackHot

Cable Opening

Subfloor

Underfloor Chilled Air

Air flow

Perf tile Tile floor

water lines

Rear Door Heat Xchanger

IBM Enterprise Rack

Front Cold

BackHot

Cable Opening

Subfloor Underfloor Chilled Air

Air flow

Perf tile Tile floor

Normal Operation of Server

Improved Operation of Server





Motivation for Liquid Cooling- Increase in heat removal performance:

Superior thermal properties of liquids compared to air

- Design flexibility: Sensible heat transport to locations with available space

- Centralized secondary heat exchanger

- Efficient water-water heat exchanger

Limited heat transport due to fin efficiencyLong distance transport possible

Thermal conductivity[W/(m*K)]

Volumetricheat capacity[kJ/(m3*K)]

Air 0.0245 1.27H2O 0.6 4176

Transport

Primary

Secondary

Disadvantage: Increased complexity





Data Center Power Distribution Systems & Efficiency

Feeder Transformer

UPS System Raised FloorTransformer

Distribution(all combined)

Server Bulk PS

VRM Load

-

480V Servers

11.2KV(AC) to 480VAC

480VAC to 400VDC to 480VAC

NA - 480VAC to 400VDC to 12VDC

12VDC to 1VDC

1VDC to heat

480V Watts 1.70 1.67 1.50 1.50 1.47 1.25 1400V DC Servers


480VAC to 400VDC

NA - 400VDC to 12VDC

12VDC to 1VDC

1VDC to heat

Std Raised Floor


480VAC to 400VDC to 480VAC

480VAC to 208VAC - 208VAC to

400VDC to 12VDC

12VDC to 1VDC

1VDC to heat

Std Watts 1.79 1.75 1.58 1.52 1.47 1.25 1

400V DC Watts 1.57 1.54 1.46 1.46 1.40 1.25 1

5%

12%

Capital/Cost of Acquisition Savings

One WattOf Chip Power

Requires this much power at the facility level

Power Conversion & Distribution Losses





Thank You!Questions?

Steve Ahladas,Data Center Services:[email protected]

Energy Efficient Data Centers - IBM · - Design flexibility: Sensible heat transport to locations...

Documents

Transcript of Energy Efficient Data Centers - IBM · - Design flexibility: Sensible heat transport to locations...