Rack PowerRack Power

Intelligent Power Distribution in the Data Center

Agenda• Data Centre Power

– UPS– Power Distribution– Rack Mount PDU’s

• UPS– Power problems– Configuration– Capacity

• Power Distribution– Types of Distribution– Pros and Cons– Efficiency

Agenda Cont…• Power Density in Cabinet

– Single vs. Three Phase– 120V vs. 208V vs. 400V– High Density Applications (Blades, Switches)– Temperature Inside Cabinets

• Importance of PDU Design– Dimensions, Flexibility, Venting, Interface Location– Cord Retention, Obstruction– Alternate DesignsAlternate Designs

• Power Monitoring Software and Interface– How to save money with intelligent PDUs– Out of the box software– Integration to BMS and UPS Software (PDU doesn’t dictate what software to use)– PDU level and outlet level information, PUE Level 2 / 3, Accuracy – Switched PDUs – turn off servers, turn off outlets

Power Components in the Data CenterCenter

UPS• UPS• Floor Mount PDU• Distribution• Rack Mount PDURack Mount PDU

What is a UPS?• An uninterruptible power supply / source (UPS) or

battery/flywheel backup is an electrical apparatusbattery/flywheel backup, is an electrical apparatus that provides emergency power to a load when the input power source typically mains power failsinput power source, typically mains power, fails.

• A UPS is typically used to protect computers data• A UPS is typically used to protect computers, data centers, telecommunication equipment or other electrical equipment where an unexpected powerelectrical equipment where an unexpected power disruption could cause injuries, fatalities, serious business disruption or data lossbusiness disruption or data loss.

Most Important UPS Features

• High Efficiency• Service/Warranty / y• Battery Management• Scalable Design• Scalable Design• Redundancy Options• Remote Monitoring• Reduced Footprint and Weightp g

9 Electrical Power Disturbances

Voltage/Capacity

120V Si l Ph 300VA 3kVA• 120V Single Phase – 300VA - 3kVA• 240V Single Phase – 1500VA - 18kVA• 208V Three Phase – 5kVA - 225kVA• 480V Three Phase – 30kVA - 1MW480V Three Phase 30kVA 1MW• 600V Three Phase – 30kVA - 1MW

UPS Style

Th b i UPS d lThree basic UPS models

• 1) Off-line UPS• 2) Line Interactive UPS2) Line Interactive UPS• 3) Double Online Conversion

UPS Power Solutions(3) (5)Line- (9)True

Off-line Interactive On-LineYES

YES

Power Failure Power Sag Power Surges

YES

YES

YES

YES

YES

NO

NO

Power Surges Undervoltage Overvoltage

YES

YES

YES

YES

YES

YESNO

NO

NO

g Electrical Line Noise Frequency Variation

YES

NO

NO

YES

YES

YES

NO

NO

Switching Transient Harmonic Distortion

CLEAN POWER

NO

NO

YES

YES

??? CLEAN POWER ??? YES

OFF-LINE UPS

NORMAL MODE Utility Power

SurgeSuppressionEMI/RFIEMI/RFIFilters

InverterBatteryCharger UPS Load

Battery

ChargerOutlets

Power flows through the UPS

Battery & Inverter are OFF-LINEBattery & Inverter are OFF LINE

LINE-INTERACTIVE UPS

Utility NORMAL MODE

SurgeSuppressionEMI/RFI Fillters

Power BUCK

BOOST

NORMAL

InverterBattery

BOOST

UPS Load

Battery

InverterCharger Outlets

Power flows through the UPS

Battery & Inverter are OFF LINEBattery & Inverter are OFF LINE

ON-LINE UPS DOUBLE CONVERSION

NORMAL MODE BypassUtility

PowerSurgeSuppression InverterRectifier

OutputTo Load

EMI/RFIFilters DC Link

UPS Load

Battery

UPS Load Outlets

Power is converted to DC and reinverted to ACContinuous Clean, Filtered,

Regulated power to the load

What is Distribution?

H t f th UPS t th iti l l dHow power gets from the UPS to the critical load• Consists of:

– Floor mount PDU– Breaker outputs to individual cabinets– Rack/Cabinet mount PDU’s

Floor Distribution

T diti l t l hi f l• Traditional style – power whips from a panel board, run overhead or under floor to the

bi tcabinets.• Benefits

– power can be routed to exactly where it is needed.

• Cons– Expensive– Long cable runsong cable runs– Limited flexibility (adds moves changes)

Floor Mount PDU/RPP• Close coupled system – usually end of row distribution• PDU transforms voltage from 600 480 to 208/120V• PDU – transforms voltage from 600, 480 to 208/120V

– Usable distribution voltage to the cabinet– Built in panel boards for distribution to a specific area in the DC

• RPP – End of row distribution panel– Usable distribution voltage to the cabinet– Built in panel boards for distribution to a specific area in the DC

• Benefits Shorter cable runs to cabinets– Shorter cable runs, to cabinets

– Moderate flexibility

• ConsCons – Expensive, need an electrician for adds moves changes

Overhead Power Busway: Pros / ConsCons

• PROS– Lower voltage drop per feet

• CONS– More expensive material upfrontLower voltage drop per feet

(improved efficiency)– 40% fewer man hours to install

vs. cabling

p p– Requires plenty of space above racks

for installation (not always available)– Becomes costly if too many “turn”

(– Does not require panelboards– Feed directly from large floor

PDUs (300-500kVA)

components are used (i.e. tees, crosses, elbows)

– Monitoring can be challenging, but vendors are becoming creative and

– Delivers high density power (400A 3phase)

– Cleaner look, doesn’t block cooling better appearance

grelying more on PDU

– Once a vendor is chosen, system is not compatible with other busway in the

k t ( it t t i l d )cooling, better appearance– Most beneficial for relocation,

pods, containerized solutions

market (commitment to single vendor)

WHAT IS A PDU SUPPOSED TO DO?DO?

• Power servers reliably (last piece of the chain)P id t / t d t d it i• Provide accurate / easy to understand monitoring information

• Assist with capacity planning load balancing and PUEAssist with capacity planning, load balancing and PUE• Work effectively under high density loads

• WHAT IS NOT SUPPOSED TO DO? – Obstruct air flow and equipment accessObstruct air flow and equipment access– Fail under high temperature environment– Difficult to install and configureg

Where is all this power going to? • Lower power density racks

(< 5kW) can be powered by Si l Ph PDUSingle Phase PDUs

• For load densities > 5kW, customers should be looking at Three Phase power to cabinets

• Three Phase power is a must to power Blades, provide long term room for growth and allow for virtualizationvirtualization

• 415V can dramatically increase power density

Choosing the Right Voltage

120/208V Three Phase • Most available power (up to

28kW)

• Single 3PH circuit reduces bli t

• Higher power capacity (up to

cabling costs

• Metered PDUs facilitate load balancing

L b f i it

230V Single Phase Higher power capacity (up to 5kW)

• Less current draw• Lower losses • More efficient

• Lower number of circuits improves airflow

• Provides scalability for growth without bringing down power120V Single Phase

• Limited at 20A

• 30A impractical & expensive

More efficient • More security since requires

208V equipment• Locking input plugs • Most servers accept 120-

g g p

• Support 120V & 208V

• Allows for Blades, Virtualization and Cloud

g

• Widely available and preferred by small business

Most servers accept 0240V Computing

Maximizing Panel and Busway Power Usagewith Three Phase 30A PDUs

• 3PH 30A PDUsB t b l b t it

with Three Phase 30A PDUs

– Best balance between power capacity and number of circuits

– 98% power capacity from a 400A feed – 93% usage of poles or power drops

il blavailable

• Delivers 8.7kW

• Meets the need for most server racks (up to 2 Blades)

• Does NOT require exotic cooling solutions

• Lower cabling costs

• Local breakers provide additional protectionprotection

400/415V PDU

• Increase Data Center efficiency and deliver more powerIncrease Data Center efficiency and deliver more power • Reduce power losses and number of transformations• Proven power distribution worldwideProven power distribution worldwide• Becoming a new standard in many Data Centers

208V Three Phase 415V Three Phase

Power @ 30A (kW) 8.6 17.2

Output Voltage 208V 240V

Input Connection Line to Line Line to Neutral

Input Plug L15 / L21 / IEC L22 / IEC

Efficiency Gains None 6-8%

Powering Blade Servers / SwitchesSwitches

• 2 PDUs per cabinet• 2 PDUs per cabinet• Min 12 C19s or L6-20s

per PDU p• Each Blade requires 4-6

outlets each • Load Balancing is a must

since each outlet can have a different power prequirement

• 4 Blades = >20kW 5 6kW h i

BA• 5-6kW per chassis• 6 power supplies (3+3 redundancy )

Overall Approach

• PDUs designed to support Blade Servers and Switches i e CiscoPDUs designed to support Blade Servers and Switches, i.e. Cisco Catalyst 6500 and Nexus 7000

• Vertical and Horizontal PDUs can be used• Low / Mid / High Power Densities, depending on:

– Style of server/switch/– Number of servers/switches

• Propose 2 PDUs per rack or cabinet offering for redundancy• Locking C19 and C13s are used to replace L6 20• Locking C19 and C13s are used to replace L6-20

Recommendation

h h ( bl d / h )• 60-80A 208V Three Phase PDU (up to 4 blades/switches)– Input cord or Terminal Block

• 6 x 20A circuit Breakers• 6 x 20A circuit Breakers • 2-3 outlets per Breaker• Local and Remote load monitoringoca a d e ote oad o to g• Horizontals or Verticals

– Depends on type of cabinet, air constraints and mounting preference

Sample PDUs for Blades/Switches

I t• Inputs– 30A Single Phase– 30A and 60A Three Phase

2U Horizontals 0U Verticals

• Outputs– L6-30– Locking C13 and C19s

• Functionalityl d i i– Local and Remote Monitoring

• FormHorizontal OR Vertical– Horizontal OR Vertical

PDU DESIGN• Keys to a good PDU design

– Low profileLow profile– Flexible mounting

Interface location and accessibility– Interface location and accessibility – Cord retention features

High temperature operation– High temperature operation– Outlet distribution and density

I t d fl ibilit– Input cord flexibility – Chassis consistency– Custom Designs

Branch Breaker Area

• Low Profile UL489 Breaker Doghouse• Low Profile UL489 Breaker Doghouse• Molded to Number of Breakers• Clearly Labeled & DefinedClearly Labeled & Defined• Reduce Obstruction Inside Cabinet

2.8”

Cord Retention Feature

Cord Retention MechanismCord Retention Mechanism• Individual outlet retention

system• Eliminates need for cable bar • Built-in anchors on the PDU• Locking tethers attach to PDU• Locking tethers attach to PDU

anchor points• Tie wraps are required to

attach tethers to plugs • Tethers can be removed and

reused

Locking C19 / C13 Outlets

Release Button

- Same electrical property as standard C13/C19 outlets- No special input cord neededNo special input cord needed - Built-in button must be pressed to release cord- Prevents accidental OR intentional cord removal- Alternative to locking NEMA L6-20 outlets and other cable retention options- Saves installation time and provides cord security in high density cabinetsSaves installation time and provides cord security in high density cabinets

Higher Operational Temperature

Venting SystemI i li d• Improve convection cooling and airflow through vents positioned at the end and sides of the PDU

• Allows for higher tempAllows for higher temp environments (65C or 149F)

• Reduces failure due to component overheating

• Increasing server exhaust temperature(45-50C)

• Containment solutions generate more heat – Hot aisle containment– Vertical Exhaust Duct

C20 Input & Input Cord Retention

• IEC C20 is an international input for PDUs pwhich gives the customer the flexibility to plug in multiple power cords from 110-240V

• Power Cords– 110V Nema: 5-15, 5-20, L5-15, L5-20– 240V Nema: 6-15, 6-20, L6-15, L6-20– 240V IEC: C20, IEC 16A– 240V (others): German, British, Italian

plugs• Double Cord Retention system is provided to

k i t d f i lkeep input cords from coming loose

Alternative Designs• Hardwired Terminal Block inputs

– Permanent connection– Cost savings by not having to purchase mating connectors

• Dual Input PDU– 2 PDUs in 1 redundancy– Space saving

• Horizontals• Horizontals– 1U or 2U designs– Customers that cannot install vertical units

• Half Height– Mount up to 4 PDUs in cabinet

Ob t t l t f th bi t– Obstruct only part of the cabinet

Power Monitoring Software and InterfaceInterface

• How to save money with intelligent PDUs– Informed capacity decisions– Informed capacity decisions– Load balancing of phases / breakers– Reduce risks of overloading and outages

Daisy chaining– Daisy chaining

• Out of the box softwareI t ti t BMS d UPS S ft• Integration to BMS and UPS Software – PDU usually doesn’t dictate what software to be used

• PDU level and outlet level information for most• PDU level and outlet level information for most accurate PUE calculation S i h d PDU ff ff• Switched PDUs – turn off servers, turn off outlets, save power, increase security

PDU Local Interface Features

• Digital Local DisplayReset– Power, Amps, Voltage, PF

• Per Breaker or Phase

– IP Setup

• Buttons for Easy On-screen Menu

Large LCD Display

Status LED

Navigation• Temp/Humidity Sensors• Ethernet & Serial Communication

Ports

Easy to Navigate Soft Buttons

Ethernet Port Ports• IP Consolidation Feature

– Connect up to 20 PDUs

• USB PortEnvironmental

Sensor PortPDU Linking Ports US o t

– FW update & Data Download

• Rotates 180º to be viewed from either direction

USB Port

Ports

Serial Communication

PortPort

Product Metering Benefits

• Avoid outlet / breaker overloading • Avoid tripping breakers at PDU or upstream• Make informed capacity planning decisions• Balance loads across breakers or phases • Efficiently utilize power resources y p• Measure IT equipment power consumption to save

power and money• Calculate Green Grid’s PUE Level 2 and Level 3• Drive green data center initiativesg

IP Consolidation

• Setup, Control and Monitor lti l PDU i d tmultiple PDUs in your data

center with a Single IP Port • PDU Linking Feature allows you

to reduce the number of IPto reduce the number of IP connections in your data center, thus saving you time and money

• Link up to 30 PDUs• Only Master PDU requires

network connection• Linked units are connected

h h i l RS485through serial RS485• Auto PDU detection• PDU cloning feature

Why link multiple PDUs?

• Save MoneySave Money– Deploy fewer IP addresses– Reduce installation time IN OUT– Reduce Ethernet cable length

• Enjoy the following features– Outlet grouping across multiple PDUs (A/B) – Unified access to all linked PDUs

B bl l i f PDU h– Be able to clone settings from one PDU to another• Manage every PDU in the chain from a single screen

PDU Outlet Power Control • Remotely Control Individual Outlets

– On/Off, Cycle/ , y– Via https or SNMP– Local status LED

• Group Outlets from Single or Multiple PDUs– Control and Manage all of them as ONE

S O tl t• Sequence Outlets–In rush current mitigation on power up

• Outlet Level Scheduling• Outlet Level Scheduling

• Shut Off outlets to increase security or reduce power usage

Accuracy & Billing

Billi G d• Billing Grade– Data Centers can report and charge customers based

tion power consumption

• PDU must have:– +/- 1% Accuracy – Real Time Metering– Outlet Level Monitoring

Built-in Web Interface

• Single Monitoring Interface g g

• HTTP / SNMP v2/v3• IP v4 / v6

Fi U d• Firmware Upgrade• Alarm Settings

– PDU / Outlet / Temperature• Outlet Grouping• Data Logging & Exporting • Email NotificationEmail Notification• Admin and User Levels• Radius / LDAP Authentication

PUE Accuracy Categories

PUE Category 0 PUE Category 1 PUE Category 2 PUE Category 3g y g y g y g y

IT Energy Measurement

LocationUPS output UPS output Floor PDU output IT equipment input

(Rack PDU output)

Definition of IT Peak IT electricDefinition of IT Energy

Peak IT electric demand IT annual energy IT annual energy IT annual energy

Definition of Total Energy

Peak Total Electric demand (kW)

Total annual energy (kWh)

Total annual energy (kWh)

Total annual energy (kWh)

Accuracy Level Basic Intermediate High HighestAccuracy Level Basic Intermediate High Highest

• The closer to loads, the more accurate PUE calculations become

• Energy (kWh) is also more accurate than Power (kW) alone

• Annual measurement vs. peak demand increases accuracy

Main Takeawaysy

U d t d d it d d d ith• Understand power density and redundancy with three phase and high voltage PDUs

• Importance of PDU design installation time• Importance of PDU design installation time, equipment / air obstruction and increase mounting flexibilityflexibility

• How to save money and improve power monitoring and control with intelligent PDUs. g

Questions?

Thank You!