Designing Critical Power Chainsfor Higher Density Loads

SURVEY QUESTION

What Is the Average Power Density

(in kW) Per Rack In Your Datacenter?

(If you’re a consultant, “… in datacenters you advise?”)

• < 2kW• 2 – 4kW• 4 – 8kW• 8 – 12kW

• 12 – 16kW• 16kW +• Unsure

Common Misnomer: “Datacenter Average Density Increasing”

Datacenter average density increases have been predicted formany years now, and as a general statement has not happened.

Same Survey Question … ~5 years ago!

Spring 2006 Spring 2007 Spring 2008 Spring 2009 2011 Prediction 2019 Prediction0

4

8

12

16

20

6 6.17.3 7.4

12

16.5

Source: Data Center Users Group (DCUG) – Emerson Network Power

Common Misnomer: “Datacenter Average Density Increasing”

Datacenter average density increases have been predicted formany years now, and as a general statement has not happened.

Same Survey Question … LAST MONTH

Spring 2009 Fall 20130

4

8

12

16

20

7.46.3

Source: Data Center Users Group (DCUG) – Emerson Network Power

And again, users predict average power densities ~12kW

in less than 3 years…

… what is going on?

The Real Story / Problem: Bifurcated Hi + Low Density Cabinets

Higher Density (15kW++)Lower Density (<8kW)

• Storage

• 1U / 2U Servers

• IDF/MDF (Networking)

Two different application profiles in critical computing environments,

each with correspondingly variant energy densities.

• Blade Server

• Compute Clusters (HPC)

1U / 2U Server Power Consumption Is Actually Decreasing

Year Watts / U

2004 (DL360G3)

360 W

2008 (DL360G5)

238 W

2011 (DL360G7)

157 W

2013 (DL360eG8)

170 W

HP DL360 @ 100% Utilization

Source: Data Center Pulse. Wiersma, Jan. “Where is the Rack Density Trend Going?”

Combining…

• major CPU improvements in power/MIPS

• better underlying h/w architecture

– e.g. High-efficiency PSUs

– e.g. Move to SSDs

– e.g. Improved front-side bus architectures (RAM)

… standard 1U / 2U servers are actually less power-hungry per U.

NET:~6 kW for 35 servers

Real Challenge = Deploying High, ALONGSIDE Lower Densities

HP C7000

• 16 blades

• 256 cores

• 10U

Cisco UCS 5108

• 8 blades

• 128 cores

• 6U

4551W 2636W

Source: SPECpower ssj_2008 @ 90% Loads

HP DL360G7

• 1 server

• 12 cores

• 1U

192W

~13.5kW (3 chassis)~18kW (4 chassis)

~15.8kW (6 chassis)~18.5kW (7 chassis)

~6.8kW (35 servers)~8.1kW (42 servers)

…But Rewards of Server Consolidation / Virtualization Worth It

8:1

Typical (Minimal)Physical Server Ratio

Post-Virtualization

– Dramatic Reduction in New Server Provisioning Time

– Increased Redundancy Characteristics

– Reduced Server Administration Workload

– Typical Investment Payback ~1 year or Less

• The compelling business case has driven significant IT virtualization initiatives for almost ten years.

• Scaling these projects upward now affect the power chain.

Sources: Gartner, IDC

Some EXAMPLE Design Challenges

– Cooling Mixed Densities• In-Row cooling?

• Actively cooled cabinets?

• Containment?

• Placement of High vs Low Density Cabinets?

– Correctly Sizing UPS

– RPP-to-Cabinet Topology (busway vs whip)

Two very different problemsa) Design for 8kW Average Rack Power Density

b) 80% of my racks are ~5kW or 6kW… but 20% are 18kW;

– Practical Cabling Concerns• # Whips per cabinet?

• 3-phase as standard?

• Investigate 400V?

– Impact on Co-Location Environments

– Hot Aisle temperature impact

– etc., etc.

The latter raises all sorts of questions. To name a few...

Agenda: Unique Design Challenges Faced by Peers >12kVA

• Safely deploy 400v/3ph distribution to the cabinet;

• Reduce # whips required per cabinet for high density;

• Six L6-30R [208V / 1ph] from one 208V / 3ph whip;

• Required breadth of 17.3kVA configurations[208V / 3ph / 60A input on IEC 460R]

Major FinancialServices Provider

• Increase Rack Density (Servers / Cabinet)

• Reduce high-density (17kVA+) costs with UL Listed power strips utilizing conduit input;

• Develop solution for 50kW cabinet;

• Enforce best-practices for load-balancing:

o blade chassis + other high-power devices;

o on high-power PDUs (208V / 3ph / 60a);

Major InternetSite

• Place five c7000 chassis in a single cabinet;

• While still using economizer for cooling;

Major International

Bank

Agenda: Unique Design Challenges Faced by Peers >12kVA

MAJOR INTERNATIONAL BANK

Free Cooling With 5x HP c7000 per cabinet

Challenge

• Appropriate overcurrent protection breaker count & placement;

• Extremely high ambient temperatures;

RPP / Breaker Sizing

• 4551kW per blade chassis x 5 blades per cabinet

• ÷ 97.5% power factor

• = 23.3kVA per cabinet

• @ 208v / 3phase = 23338VA ÷ 208V ÷ √3 = 64.8A continuous / line;

• x 125% (NFPA 70, Article 645) = 81A per line;

• Rounding up …

• 100A breaker

• 80A rated continuous (28.8kVA)

• @ 208v / 3ph;

MAJOR INTERNATIONAL BANK

Free Cooling With 5x HP c7000 per cabinetDesign 1 - NEC Minimum

• Each breaker (1-phase, 20A, 208V) rated to 4160kVA – so in principle, 9 breakers is sufficient;

A A A

B B B

C C C

D D D

E E E

MAJOR INTERNATIONAL BANK

Free Cooling With 5x HP c7000 per cabinetDesign 2 - Three-Pole Breakers

A

A

A

B

B

B

C

C

C

D

D

D

E

E

E

MAJOR INTERNATIONAL BANK

Free Cooling With 5x HP c7000 per cabinetDesign 3 - Individual 2-pole Breakers

A A AB B BC C C

D D DE E E

Reduced transmission loss (Ohm’s Law V=IR)

Reduced step-down transformation loss

Less copper, more power

Already in widespread deployment worldwide

18kW @400V

4P 5W, 8# (24A)27.3mm (1.08”) OD

17.3kW@208V

Source: General Cable SOOW Spec Sheet

3P 4W, 4# (48A)32mm (1.26”) OD

AC Power Cable Cross-Section @ ~17.5kW

Disney, et al: 400V / 3ph to Cabinet >12.6kVA

Disney: Deploy 400v/3ph Power to Rack Safely

Challenge

• New datacenter in Kings Mountain, NC;

• Client wished to harness cost savings / electrical benefits of 400v/3ph to the rack;

• Still relatively new in North American market;

Raritan Solution

• Leading pioneer of 400v/3ph rack PDUs in North America;

• Largest install base of relatively “exotic” emerging standard [“European”];

• At client’s request, fabricated chassis to match UPS color scheme [blue, green, magenta, and yellow];

208V vs. 400V Intelligent PDUs – Power Density Review

208v / 1ph / 30a 208v / 3ph / 30a 400v / 3ph / 30a

Phase Single phase Three phase Delta Three phase Wye

Input Voltage 208V 208V, 3-phase 400V, 3-phase

Output Voltage 208V 208V 230V

Apparent power 5.0 kVA 8.6 kVA 16.6 kVA

1U servers supported* 24 41 80

2u servers supported* 24 41 80

Blade chassis supported* 1 1-2 3-4

*HP Proliant DL360G7 1U server, 192 watts at 100% load on SPECpower_ssj2008HP Proliant DL380G6 2U server, 208 watts at 80% load on SPECpower_ssj2008HP Proliant C7000, ProLiant BL2x220c G6, 4551 watts at 80% load on SPECpower_ssj2008

…Delivering More Power at Same Ampacity

…but 400v/3ph Power Raises Unique ConsiderationsUntil very recently, 400v/3ph power distribution considered “European”.True, practical expertise at this standard is rare in North America

Arc Flash Hazard. Unlike status quo, 400v/3ph technically poses an arc flash hazard. Special care required by electricians, as well as connected hardware;

Input Plug Preference. Relatively new to North America, confusion abounds regarding the correct input plug to utilize:

• NEMA L22? or NEMA L7?

• IEC 60309 (“Pin-and-Sleeve”)? If so…

- Ground pin at 6HR (415v max) : red plug? typically considered a European plug, can it pass UL?

- Ground pin at 7HR (480v max) : blue plug? typically considered a North American plug, but 480v?

Harmonics. Line-to-Neutral wiring can induce harmonics / current on the neutral line, if load is not properly balanced.

Product Selection (UPS, RPP, Rack PDU). Other vendors lack the install base, and therefore the breadth of product selection, in the 400v/3ph electrical specification.

Cisco: Reduce #Whips for High-Density Cabinet

Challenge

• Common problem for certain blade servers, and many Cisco devices [e.g. Nexus 7000 series]

• Power supplies hardwired with L6-30P power supplies;

• Customers assume rack PDUs cannot be used, run 6x whips;

Raritan Solution

• Design rack PDU with 30A breakers and NEMA L6-30R outputs;

Floor PDU / RPP

Cisco Example: Exadata Cabinets

12 x Single phase 30A Circuits

Floor PDU / RPP

BEFORE

Floor PDU / RPP

Cisco Example: Exadata Cabinets

2x Three-Phase, 50A circuits

Floor PDU / RPP

AFTER

Cisco: Electrical Equivalence with Fewer WhipsIn concept, a remote “Branch Panel”… metered + monitored for fault prevention

MAJOR FINANCIAL SERVICES COMPANY:

Maximize Deployment Flexibility @ 60A/3ph

Challenge

• Relatively small source of vendors specializing in configurations above 12.6kVA;

• Need a 3ph / 60a solution flexible to accommodate blades OR 1U/2U/storage;

• Most vendors assume blades; or 1U/2U/storage – not flexible;

Raritan Solution

• 17.3kVA (208v / 3ph / 60a)

• 12x C19’s allow deployment of up to 4x blade servers;

• 24x C13’s allow deployment of ~20x 2U servers + switches;

• Six circuit breakers protect load against bad power supplies;

• Cable exits on end of PDU to simplify cable management of very large SO cord;

• Each “bank” of outlets alternates phases to enable easier blade server deployment (see “Major Internet Company” case study, later in this slide deck)

IEC 60309 3-Pole 4-Wire

“Pin-and-Sleeve / CEEform”IP67 (“Watertight”)

“460C9W or 460R9W”

MAJOR FINANCIAL SERVICES COMPANY:

Maximize Deployment Flexibility @60A/3ph• 208v / 60A / 3ph power strip requires 4-conductor / #4 AWG input cable;

• SOOW UL rated cable is 1.26” in diameter, and weighs 11.5 pounds*!!

• Appropriate plug selection required to ensure reliable connection;

* assumes 10ft input cable** photo shows 2P3W

IEC 60309 3-Pole 4-Wire**

“Pin-and-Sleeve / CEEform”IP44 (“Splashproof”)“460C9 or 460R9”

NEMA 15-60P

IP44 (“Splashproof”)

DeFACTO STANDARD RARITAN STANDARD(secure locking ring)

CLIENT’S CUSTOM CHOICE(physically smallest)

Twitter: Increase Rack Capacity with Confidence

Challenge

• Client standard = ~35 servers / rack. Power believed stranded / under-utilized;

• Required solution to:

• provide enough outlets to accommodate more servers (42+)

• ensure no power faults due to overloading [monitoring accuracy];

• fits in same exact form factor as incumbent;`

Raritan Solution

NEMA L21-30P208v / 3ph / 30A

8.6kVA

48xC13 and 3x NEMA 5-20R in 2.1” wide chassis

Twitter: Increase Rack Capacity with Confidence

+/- 1% Metering per Breaker Assures Fault Prevention & Detection…

… at same price as competitor’s 36-outlet model

eBay: Reduce High-Density Cost w/ Conduit

Challenge

• Client regularly deploys cabinets exceeding 20kW;

• Above 16.2kVA (208v / 3ph / 45a), SOOW cord and plug becomes very expensive;

Raritan Solution

• Design rack PDU option for “hard-wiring” to user-accessible terminal block;

• Worked with certification authorities to attain UL Listing;

• Worked with StarLine (Universal Electric) to provide appropriate tap box;

• Deployed SealTite flexible conduit at 1” Trade Size;

eBay: Reduce High-Density Cost w/ Conduit

SealTiteflexibleconduit@ 1” Trade Size

Standardconduitfittings mateto punchouton PDU

Electrician-accessibleterminal block

Savings (at ~17kVA)

Part Spec Price

PDU Cord10’ SOOW #4 AWG

$88

PDU Plug IEC 60309 $173

Tap Box Connector

IEC 60309 $205

Tap Box Cord2’ SOOW #4 AWG

$17

Tap Box Strain Relief

Kellems Mesh Grip, 1.5” O.D.

$48

~$1000 per rack

New design far cheaper:

• ~1hr of electrician time (max);

• “Bare” wire conductors;

• Inexpensive conduit;

• “Permanent” installation;

eBay: World’s Highest Density Power Strip*

Challenge

• New Dell container datacenter (“MDC”) to be deployed next month;

• Can consume nearly 50kW per cabinet at peak;

Solution

• Dual inputs: top and bottom, each 27kVA;

* (I think)

MAJOR INTERNET COMPANY

Eliminate Stranded Capacity by DesignChallenge

• Due to weight and exhaust heat, users typically place high-power equipment (e.g. blade servers) toward the bottom of the rack;

• Client deploys 3-phase power;

• Results in overutilization of one electrical phase [outlets closest to bottom of rack]; and underutilization of other two phases = Stranded capacity.

Raritan Solution

Phase 3(L3/L1)

Phase 1(L1/L2)

Phase 2(L2/L3)

Phase 3(L3/L1)

Phase 1(L1/L2)

Phase 2(L2/L3)

Every OutletEvery BreakerClearly labeled

Key Discussion Items

1. “Average Power Density” vs. Select Cabinets @ Very High Density.

2. Traditional Line of Advisory Demarcation Has Moved. At very high densities, clients require your expertise to advise on operational implications of the electrical distribution inside the rack (not just at the RPP).

3. Genuine Strategy for High Ambient Temperatures Required. BMS monitoring of set points, return temperatures, and a few sensors per row may not be sufficient.

• True UL ratings for all equipment;

• Cost-effective cabinet-level monitoring;

• Consider qualifying vendor equipment with load testing + temperature gun;

4. Consider Physical Limitations of Copper. Above 12.6kVA, whips themselves can become operational constraints;

• SO vs. Conduit vs. Overhead Busway;

• 400v vs 208v;

• Don’t forget cabinet openings and bend radius implications;

Ready to Lean More?

Click Here to Visit Raritan.Com and Learn About Our Power Solutions