PUSHING THE ENVELOPE · ‐10 (14) 0 (32) 10 (50) 20 (68) 30 (86) 0. dlb associates | confidential...

$: PUSHING THE ENVELOPE · ‐10 (14) 0 (32) 10 (50) 20 (68) 30 (86) 0. dlb associates | confidential and proprietary 23 Printed: 1/16/2013 4:51 PM s:\dlb 1\a presentation\archive public\2013\bcs$
This image cannot currently be displayed.

Don Beaty, PE, ASHRAE FellowPresident, DLB Associates Consulting Engineers

PUSHING THE ENVELOPE: The impact of increasing the environmental range on IT equipment reliability

2dlb associates | confidential and proprietaryPrinted: 1/16/2013 4:51 PM

s:\dlb 1\a presentation\archive public\2013\bcs ‐ 2013 london ‐ beaty ‐ published version.pptx

Opening Comments

Before ASHRAE first published the Thermal Guidelines Book in 2004, there were no vendor neutral standards for data center temperature and humidity

Common temperatures were 18 – 20 °C.

ASHRAE’s Thermal Guidelines recommended temperatures up to 27 °C with allowable temperatures even higher.

Although there are a number of data centers operating above 20 °C, often those operating temperatures are in the range of 22 – 24 °C

A minority of the data centers are operating above 24 °C

Today’s presentation shares some case studies and experiences about operating at higher temperatures and / or using chiller‐less approaches

We hope this will encourage more companies to take advantage of the ASHRAE Recommended and Allowable Envelopes.



Agenda

PART 1: ASHRAE 2011 THERMAL GUIDELINES OVERVIEW

PART 2: CASE STUDIES

1) Greenfield Mega Internet Data Center (Europe)

2) Existing Colocation Data Center (Europe)

3) Greenfield Colocation Data Center (Asia)

4) Greenfield Mega Internet Data Center (Europe)

CLOSING COMMENTS

GOAL: Demonstrate capital & operational savings through design aligned with ASHRAE Thermal Guidelines



Part 1: ASHRAE 2011 Thermal Guidelines Overview



New Thermal Guidelines – Introduction

Reluctance to operate at higher temperature / humidity ranges include a perceived higher risks of outages, lower resilience, voiding IT equipment warranties, etc.

In 2004, IT OEMs from ASHRAE TC 9.9 published the first VENDOR NEUTRAL temperature and humidity ranges that did NOT void legacy IT equipment warranties.

In 2008, the ranges were further expanded without voiding warranties and in 2011, new groundbreaking data was provided by IT OEMs to help guide greater energy efficiency.

THE OPPORTUNITIES FOR COMPRESSORLESS COOLING (NO REFRIGERATION) ARE HIGH



2011 Equipment Environment Specifications Table (Partial)



Thermal Guidelines – ALLOWABLE Environmental Envelopes

New Classes A1 and A2 are EXACTLY the SAME as previous Classes 1 & 2

Classes A1 and A2 apply to new and legacy equipment.

New Classes A3 and A4 do NOT include legacy equipment.



2008 Recommended Envelope Operating Points

2008 ASHRAE Thermal Guidelines Recommended Envelope – Many Operating Points (171)Dry Bulb

% Relative Humidity

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

18 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =17

19 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =19

20 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =21

21 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =21

22 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =21

23 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =21

24 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =18

25 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =14

26 1 1 1 1 1 1 1 1 1 1 1 =11

27 1 1 1 1 1 1 1 1 =8

=8 =8 =9 =9 =10 =10 =10 =10 =9 =9 =9 =8 =8 =8 =7 =7 =7 =7 =6 =6 =6 =171



London Time at Temperature Histogram

Average yearly dry bulb temperatures for airside economizer for London with air mixing to maintain 15°C (59°F) minimum and mechanical cooling to maintain 35°C (95°F) maximum.

ASHRAE Weather Data Viewer provides statistical averages based on the most recent 25 years of data



Volume Server Failure Rate versus Ambient Temperature

Volume Server Failure Rate Versus Ambient Temperature

Dry Bulb Temperature

Average Relative Failure Rate (Notes 2 and 3)

Annual Server Failure Rate

CommentsBaseline (Note 1)

Adjusted Due To Temperature

A B C D = B x C

15°C (59°F) 0.72 4% 3% Less Failures

20°C (68°F) 1.00 4% 4% Baseline

40°C (104°F) 1.66 4% 7% More Failures

NOTES

1) Table uses the conservative upper bound of typical server failure rates of 2 to 4%.2) Assumes continuous (7 x 24 x 365) operation with Dry Bulb Temperature at ITE inlet. 3) Based on years of Server Manufacturers’ actual failures, Component Manufacturers’

failure data, and modeling (published by ASHRAE TC 9.9).

CONCLUSION: Temperature has minimal impact on Server Failure



Volume Server Failure Rate versus Ambient Temperature

Lower bound, average, and upper bound are included since there is variation in server configuration and utilization



London Net X‐Factor Across The Whole Year

IT hardware failure in London with a variable data center temperature is actually LOWER than if the data center was operating at a tightly controlled temperature of 20°C (68°F).




Average Net Failure Rate Projections for Air‐side Economization (EU Cities)




Chiller Hours Per Year for Air‐side Economization (Europe Cities)

* Assumes 1.5°C (2.7°F) approach between outdoor air dry bulb and supply air due to fan friction




Use & Application Guidance for New ASHRAE Data Center Classes



Part 2: Case Studies



Case Study Overview

Case Study

DetailsEmphasis

Site Location Type

1 Greenfield Europe Mega IDCIncrease Setpoint, Increase Free Cooling Hours, Reduced Mechanical Cooling Plant, Climate Analysis, X‐factor Analysis

2 Existing DC Europe Colocation Increase Setpoint, Increase Free Cooling Hours

3 Greenfield Asia ColocationIncrease Setpoint, Increase Free Cooling Hours, Improve Chiller Efficiency, Reduced Mechanical Cooling Plant, Climate Analysis, X‐factor Analysis

4 Greenfield Europe Mega IDCIncrease Setpoint, Increase Free Cooling Hours,Reduced Mechanical Cooling Plant, Climate Analysis



Case Study 1:Greenfield Mega Internet Data Center (Europe)



Case Study 1: Overview

A European greenfield site was designed without chillers.

This was achieved through:

1) Climate and cooling analysis

2) Application of the ASHRAE Thermal Guidelines

3) Alignment with the client’s value system

By raising the temperature and operating in the upper end of the Recommended range AND to accept excursions beyond that range on occasion, a CHILLERLESS design was achieved.



Raised Floor Area: 20,000m2 (215,000 ft2)

Raised Floor Height: 122cm (48”)

Cooling System: Waterside Economizer feeding in‐row cooling units

Total IT load: UNDISCLOSED

Case Study 1: Floor Plan & Metrics

Cooling Cooling

Power Power

Server Space Server Space Support Space



Case Study 1: Energy Efficiency Measures (EEM) Implemented (Airflow Mgmt.)

The best practice of isolating the hot aisle from the cold aisle ensures mixing does not occur and was key to pushing the temperature limits of energy efficiency in the facilities.

The servers exhaust heat to an enclosed hot‐aisle where air is drawn by fans across a chilled water coil and back into the room. The room acts as one big cold‐aisle.

Room as a Cold Aisle

Room as a Cold Aisle (27°C [80°F])

Plan View Elevation

Enclosed Hot Aisle(>41oCF [105°])

Servers

Servers

Enclosed Hot Aisle (>41oC [105°F])

Servers

Servers

Enclosed

Hot Aisle

Servers

Servers

FansCooling Coil



Case Study 1: Details (Specific European City Wet Bulb Temp. Information)

Annual Wet Bulb Temperature Frequency

Freq

uency (Percentage)

35

30

15

10

5

25

20

Wet Bulb Temperature, °C (°F)

‐30(‐22)

‐20(‐4)

‐10(14)

0(32)

10(50)

20(68)

30(86)

0



Case Study 2:Existing Colocation Data Center (Europe)



An existing European Data center was examined for opportunities to reduce PUE and decrease energy costs.

The data center consists of two data halls (plus one future hall).

The cooling system consists of air‐cooled chillers which supply chilled water to the CRAC units in the server area.

A total PUE of 1.70 was calculated with current PUE contributions from the various systems as follows:

Case Study 2: Overview

Component(s) PUE Contribution

Chiller (with Primary Pumps) 0.40

Secondary Pumps & CRACs 0.15

Electrical (Transformers, UPS) 0.15



Case Study 2: Energy Efficiency Measures (EEM) Implemented

The following best practices were implemented and their impact on PUE and energy costs were analyzed.

1) Partial containment of cold aisle & installation of blanking panels

Prevents mixing of air streams

2) Increase temperature in data halls

Reduces fan energy in CRAC units (utilize their VFDs)

Increases number of hours of both partial and full free cooling in integrated free cooling mode

Reduces or eliminates cooling load in compressors

3) Control CRAC units on cold aisle temperature (instead of return air temperature)

Required installation of cold aisle temperature sensors

4) Control CRAC units as a whole instead of individually

Prevents “fighting” units

Reduces pump energy

5) Balance floor tiles to equalize loads on CRAC units

Reduces fan energy in CRAC units



Case Study 2: Energy Efficiency Measures (EEM) Implemented

Cold Aisle Containment at End of Aisle

CRAC Unit Control Scheme Before (left) & After (right)

Reduced Compressor Load on Air‐Cooled Chiller Plant



7(44.6)

Case Study 2: Details (Supply Water Temperature Impact on Chiller Efficiency)

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

Chiller PUE

Chilled Water Supply Temperature, °C (°F)

Chiller PUE vs. Chilled Water Supply Temperature At 8.5°C (47.3°F) OAT

Chiller PUE Decreases With Increased Supply Water Temperature Due To Increased Partial Free Cooling

16(60.8)

8(46.4)

9(48.2)

10(50)

11(51.8)

12(53.6)

13(55.4)

14(57.2)

15(59)



Case Study 2: Overall Results (PUE Energy Savings Projection)

The changes were expected to reduce the PUE by 0.10 purely from MECHANICAL changes to the existing system.

The cost savings from the reduced PUE was projected over time (assumes increased load over time):

Post‐implementation PUE Reduction

System Component Post‐implementation PUE Reduction

Chillers 0.05CRAC Units 0.04Pumps 0.01Total 0.10

Per Annum Cost Savings

Year Per Annum Saved Cumulative Saved

2012 $250k $250k

2013 $325k $575k

2014 $450k $1.0M

2015+ $550k $1.6M



Case Study 3:Greenfield Colocation Data Center (Asia)



Raised Floor Area: 13,000m² (140,000 ft2)

Raised Floor Height: 91cm (36”)

Cooling System: Modular Chilled Water Plants with Waterside Economizer Feeding CRAH Units

Total IT Load: 10MW

Case Study 3 – Floor Plan & Metrics

Server Area

Power Plant

Office

Cooling Plant (Above)



Case Study 3: Energy Efficiency Measures (EEM) Implemented (Cooling Analysis)Hou

rs Per Year B

in

0

50

100

150

200

250

300

‐20 0 20 40 60 80 1000

50

100

150

200

250

300

‐20 0 20 40 60 80 100 120

A Cooling System Analysis was performed for a Greenfield data center located in Asia to understand the tradeoffs in terms of TCO & reliability.

The analysis was applied to 3 different cooling system types to understand:

1) Amount of available free cooling at differing supply air temperature(higher supply air temperatures mean MORE free cooling)

2) Magnitude of MECHANICAL cooling capacity needed (when outdoor conditions mean that free cooling is NOT suitable)

3) Annual average relative PUE for each cooling system (directly related to relative operating cost)

Wet Bulb Temperature, °C (°F)

‐29(‐20)

Dry Bulb Temperature, °C (°F)

38(100)

27(80)

16(60)

4(40)

‐7(20)

‐18(0)

‐29(‐20)

49(120)

27(80)

16(60)

4(40)

‐7(20)

‐18(0)

38(100)



Case Study 3: Details (Climate Analysis)

Climate Analysis Results

ScenarioNumber System Type Air Flow Approach IT Envelope

Free CoolingPUE

Hours % Of Year

1 Direct Air‐Side Economizer Open 1.7°C(3°F)

Recommended 2,223 25.4 1.275

A1 Allowable 5,632 64.3 1.207

2 Chillers And CRAHs(Water‐Side Economizer) Closed 8.3°C

(15°F)Recommended 5,950 67.9 1.167

A1 Allowable 8,298 94.7 1.153

3 Indirect Adiabatic Cooling Closed 1.7°C (3°F) 70% WBDE

Recommended 7,318 83.5 1.152

A1 Allowable 8,758 100 1.145

The Climate Analysis maps the statistical weather data and the desired operating temperature range inside the data center (IT Envelope) to the defined Cooling System type to build a theoretical ANNUAL performance in terms of economizer hours and mechanical system PUE.

1) For each system type the approach temperature represents the differential between outdoor air conditions and IT inlet conditions.



If the Maximum MECHANICAL cooling capacity (e.g. chiller tonnage) is UNDERSIZED, then EXCURSIONS occur. The occurrences of the excursions are directly related to the Outdoor conditions.

The EXCURSIONS can be considered in terms of annual DURATION (number of hours per year), or essentially how often they will occur and MAGNITUDE (what IT inlet temperature can be expected during the excursion).

0

100

200

300

400

500

600

Case Study 3: Details (Intentional Undersizing and Excursions)

Excursion Temperature, °C (°F)

Cumulative Excursion Freq

uency

(Hou

rs per Year)

At full IT load, sizing a chiller plant at 65% of the max tonnage (e.g., 0.65 x 325‐tons per 1MW) will result in ~520‐hours ABOVE 24°C (75.2°F) and a MAX temp of 29°C (84.2°F).

~150‐hours above 25.5°C (78°F)

~10‐hours above 27°C (80.6°F)

24(75.2)

30(86)

65% Chiller Capacity (Max. Temp. 29°C [84.2°F])

25(77)

26(78.8)

27(80.6)

28(82.4)

29(84.2)



Mechanical Cooling Capacity

20 ≤ T ≤ 25°C68 ≤ T ≤ 77°F

25 < T ≤ 30°C77 < T ≤ 86°F

30 < T ≤ 35°C86 < T ≤ 95°F Net X‐

factor% hours Avg. X‐

Factor % hours Avg. X‐Factor % hours Avg. X‐

Factor

Chillers & CRAHs (Waterside Economizer)

100% 325 Tons 100 1.13 0.0 1.34 0.0 1.48 1.13

65% 210 Tons 98.3 1.13 1.7 1.34 0.0 1.48 1.13

Indirect Adiabatic Cooling

100% 200 Tons 100 1.13 0.0 1.34 0.0 1.48 1.13

‐ Chillerless 83.5 1.13 15.6 1.34 0.9 1.48 1.29

Case Study 3: Details (Excursions and Impact on Failure Rate)

From an IT Equipment perspective, provided that the Maximum Temperature remains within the ALLOWABLE envelope then the IMPACT of each excursion is NOT significant.

The impact of the DURATION of the excursions can be mapped to annual X‐factors to understand the impact on server failures.

The table below identifies that having a 65% Capacity Chiller Plant translates to a NEGLIGIBLE increase in temperature related IT equipment failures per year.



Case Study 3: Overall Results (Summary of Cooling Options)

Allowing elevated temperatures within a data center provides the opportunity to REDUCE or ELIMINATE the amount of MECHANICAL COOLING that is installed.This example shows that even in a location that may experience fairly high ambient temperatures and a fairly humid environment for a significant portion of the year, SIGNIFICANT first cost savings are achievable which will result in a NEGLIGIBLE impact on ITE performance or longevity.

1) Direct Airside Cooling

An efficient solution BUT will require supplement mechanical cooling if supply air temperatures are to be maintained below the ASHRAE allowable thresholds.

In order to control HIGH HUMIDITY a fully‐sized mechanical cooling plant is required –otherwise high humidity conditions would occur (problematic for the facility).

2) Chilled Water CRAHs with Waterside Economizer

Reasonable efficiency especially if the operated at elevated supply air temperatures

Chillers may be intentionally undersized to achieve good Capex savings (~$500,000 / MW) and negligible impact to ITE reliability.

3) Indirect Adiabatic Cooling

MOST efficient solution / provides the MOST free‐cooling hours and the lowest PUE.

Mechanical cooling can be ELIMINATED if supply air temperatures are allowed to rise up to the ASHRAE A1 Allowable threshold for the warmest days of the year.



Case Study 4: Greenfield Mega Internet Data Center (Europe)



A European greenfield site was designed with a unique cooling system.

Some unique features include:

1) Seawater as the primary method of cooling

2) Tempering the outlet seawater to minimize environmental impact

3) Chillerless design

4) Modular design with multiple failsafes

Case Study 4 Introduction



Raised Floor Area: 2 x 100,000+ ft2 (2 x 9,290 m2)

Raised Floor Height: 48” (122cm)

Cooling System: Chillerless Seawater Cooling

Total IT Load: UNDISCLOSED

Case Study 4 Floor Plan & Metrics



The seawater intake temperature was investigated using data taken over several years.

The temperature for an existing intake tunnel was compared with a new proposed location at a lower elevation.

This elevation had significantly less excursion time (Baseline = 20°C [68°F]) than the existing intake.

Case Study 4 Seawater Intake Temperature Investigation

Year 1

Year 2

Year 3



Case Study 4 Overall Results

A closed water‐side cooling system was designed to minimize humidification and contamination concerns.

This was a chillerless design with a theoretical PUE significantly below 1.1.

Seawater provided an economical and environmentally conscious cooling solution that was reliable and exceed industry trends for cutting edge PUE.



Case Study Overview – Techniques

TechniqueCase Studies

1 2 3 4

General

CFD modeling to predict outcomes X X X

Increase supply air temperature X X X X

CHILLERLESS design X X

Reduced fan speed (VFDs) X

Optimize Controls

CRAH Units & Chiller Plants X

Temp. / Humidity sensor locations X X

Improve Air Distribution

Balance tiles / increase floor pressure X

CFD model to review results X

Improve Airstream Separation

Hot / Cold Aisle Containment X X X X

Blanking Panels X X X X



Closing CommentsThere are many ways to use the information presented in the ASHRAE Thermal Guidelines to save energy, capex and opex on your data center.

The examples presented today are all operational data centers that are benefiting from implementing higher operating temperatures & / or accepting excursions outside of their setpoints.

Having design professionals that have ASHRAE expertise eliminate the risk of balancing capex, opexand energy efficiency with mission critical operation.

We hope this will encourage more companies to take advantage of the ASHRAE Recommended and Allowable Envelopes.

This image cannot currently be displayed.

END OF SLIDES

PUSHING THE ENVELOPE · ‐10 (14) 0 (32) 10 (50) 20 (68) 30 (86) 0. dlb associates | confidential...

Documents

Transcript of PUSHING THE ENVELOPE · ‐10 (14) 0 (32) 10 (50) 20 (68) 30 (86) 0. dlb associates | confidential...