IntroductiontoDataCentres

Grant SaulsGrantSauls

CCDA CDNIDSCDCSNDSCDCDDCCAJNCIAERJNCISEProject+DCECCTT(FIA)CDS

CertifiedDataCenterDesignSpecialist

AGENDAAGENDA

9 59am 5pm

WhatandwhywehaveDataCentres

GeneralissuesfacingDataCentres

10CriticalcomponentsofaDataCentre

Datacentredesignguidelines

What is a Data Centre?WhatisaDataCentre?

ABuildingorportionofabuildingwhosei f i i hprimaryfunctionistohouseacomputerroom

anditssupportareas

.....Simplicity,Flexibility,scalability,modularity

Th f f ti l i t f d t Therearefourfunctionalrequirementsofadatacenter

1 Location i e A place to locate computer storage1. Locationi.e.Aplacetolocatecomputer,storageandnetworkingdevices.

2 Power i e Power needed to maintain the2. Poweri.e.Powerneededtomaintainthedevices

3 HVAC i e Temperature controlled environment3. HVACi.e.Temperaturecontrolledenvironmentwithintheparametersneed.

4 Structured cabling i e Connectivity provided to4. Structuredcablingi.e.Connectivityprovidedtootherdevicesbothinsideandout.

Data Centre Global MarketDataCentreGlobalMarket

$8billiondollarmarket2007 $15.6 billion expected in 2008$15.6billionexpectedin2008 50%spendingofI.TandFacilitiesManagersb d ill hibudgetswillgotothisarea

DataCentreexpectgrowthfor2009/2010willp g /be40%

Vi t li ti ? Virtualisation? Yes/No

Market drivers for Data CentresMarketdriversforDataCentres

DisasterBackup Data Protection SarbanesOxleyDataProtection Sarbanes Oxley GrowingFinancialsectorsi.e.Internetbanking Distancelearningforuniversitystudents Corporates requiring security storageCorporatesrequiringsecurity,storage,resilience

Growthforonlinebusiness

Issues facing Data CentresIssuesfacingDataCentres

ReliabilityandAvailability PowerPower RackSpace SiteLocation HeatHeat Cooling

ReliabilityReliability

Temperature55%

ElectronicsFailuredue

to

Vibration 20%Dust 6%

to...

High/LowHumidityHumidity 19%

POWERPOWERDX CoolingDXCoolingSystems

50%

CriticalL d

About9%ofapower stations

PowerConsumption

Loads36%

powerstationspowermakesittoaServer

pIssues...

UPS/BatteryLightning y

Charging11%

g g3%

Rack SpaceRackSpace

S i 3 2 b i ili i Serversrequire3+2basicutilities Power Data Control/KVM

CoolingRacking

18%ofRackspacegoesunusedduetopowerconsumptionandcooling.

Site LocationSiteLocation

S i i bl SeismicProblems GroundSubsidenceproblems Security/criminalproblems Availability of Electricity, water, sewage andAvailabilityofElectricity,water,sewageandtelecoms.

Proximity to railway lines airports chemical Proximitytorailwaylines,airports,chemicalstorageandmilitary

EMC Problems i e Mobile masts radar EMCProblemsi.e.Mobilemasts,radartransmitters

HEATHEAT

H t t f 55% f l t i f il Heataccountsfor55%ofelectronicfailure. Theremustbe1kwofaircon for1kwofheatinsideyourrack. Therearethreemethodsofheat

Conduction transferringheatthroughmetal Convection transferringheatthroughliquidsorgas Radiation transferring heat through electromagnetic wavesRadiation transferringheatthroughelectromagneticwaves

Itsimportantnottooverfillyourrack. For every eight server racks you should have one comms and one Foreveryeightserverracks,youshouldhaveonecomms andone

storagerack. RememberheatgenerationoccursattheChip,Server,Rackand

R l lRoomlevel A4kwrackisideali.e.Averageserveris370W

10criticalcomponentsofaDataCentres

k d l RackFundamentals CoolingFundamentals PowerDistributionFundamentals Generator FundamentalsGeneratorFundamentals AvailabilityandReliabilityFundamentalsPh i l I f t t M t PhysicalInfrastructureManagementFundamentalsi i h d FireProtectionMethods

GeneralDesign

Rack Location Unit ConceptRackLocationUnitConcept

Anonapplicationspecificworkingspace An area of the computer room floor i.e. 600 xAnareaofthecomputerroomfloori.e.600x1000mm

Ad d i di i i h Adequatepowerandairconditioningtothatlocationisessential

RackLocationwithinDataCenters

TIA942andASHRAErecommendlayingoutequipmentracksaccordingtotheHOTAISLE COLDAISLE,g ,

sevenpitchtilemodel

Rack Location condRackLocationcon d

Thi M d l th t i ThisModelmeansthatairconditioningunitspumpcoldairintothefloorvoid.

Ventsinthefloordelivercold air to the front of thecoldairtothefrontoftheracksfacingeachotherColdAisle

HotAirescapesthroughthebackoftherackintothe Hot Aisle where ittheHotAislewhereitreturnstheairtotheairconditioningunit

Rack Location CondRackLocationCon d

ThereareninedifferentAirFlowmodels Three different types of rack location designsThreedifferenttypesofracklocationdesigns

7tilepitch8 il i h 8tilepitch

9tilepitch

The 7 tile pitch is the biggest space saver The7tilepitchisthebiggestspacesaverwhenusinga4kwrack

Rack Location SafetyRackLocationSafety

EnsuretheFloorisstrongenough Ensure the floor is level and in good orderEnsurethefloorislevelandingoodorder Putheavyequipmentatthebottomofthe

krack.

Correctlyearthallthemetalworky

EIA310Disthestandardforracks,cabinets,Panelsandassociatedequipment.

Rack Location ClearanceRackLocationClearance

Aminimumof1mfrontclearancemustbeprovidedforinstallationofequipment.p q p

1.2mfrontclearanceisrecommended.A i i f 0 6 l b Aminimumof0.6mrearclearancemustbeprovidedforserviceaccess.

A1mRearclearanceisrecommended.

RackLocationDesignIssuestoAddress

SizeofRack Cableentry,toporbottomincludingcablesealingy, p g g CoolingMethodP S li (C t li d R k M t UPS) PowerSupplies(CentralisedorRackMountUPS)

Earth/Grounding CableManagement Rack Monitoring RackMonitoring FireSuppression

Data Center CoolingDataCenterCooling

Th li t f D t C t i t f CRAC ThecoolingsystemforaDataCenterconsistofCRAC(computerroomairconditioning)andforlargerDataCentresCRAH(computerroomairhandlingunit)( p g )

DataCenterhaveHOTSPOTSthattraditionalcoolingmethodshavenotbeenabletoaddress.

Thereare3typesofCoolingArchitectures Room,RackandRow.

The primary distinctions that affect the capability of Theprimarydistinctionsthataffectthecapabilityofcoolingsystemsarerootedinthedistributionsystems.

It is the configuration of these distributions systemsItistheconfigurationofthesedistributionssystemsthatdistinguishesthedifferenttypesofcoolingsystems.

Data Center Cooling CondDataCenterCoolingCon d Thereare3basicdistributionsystems

Flooded notrecommended1 3kWTh CRAC d h l d d i b lk i f h i h TheCRACandtheloadsdrawinbulkairfromtheroom,withoutanyspecialductinginbetweenthem.

LocallyDucted 4 8kWy Airisprovidedorreturnedviaductswhenhaveventslocated

neartotheloads Fully Ducted 5 15kW FullyDucted 5 15kW

Supplyorreturnairisdirectlyductedintooroutoftheloads

Basedonthisinformationthereare9combinationsofCoolingavailable.

Data Center Cooling CondDataCenterCoolingCon d Design Cooling ConsiderationsDesignCoolingConsiderations

Layoutracksinalternatingrows LocationofCRACUnits Crucial Quantityandlocationofvents Crucial SizingofDuctwork Proper internal configuration of racksProperinternalconfigurationofracks Ensureplumbingroutesareavailable

bl h d l l h Establishandplanpowersuppliestotheunits Decideonlevelsofredundancy

Data Center CoolingDataCenterCooling

T pical Challenges for Rack Room and Ro oriented Cooling TypicalChallengesforRack,RoomandRoworientedCoolingArchitectures Agility abilityofsystemtoadapttochange

P D it PowerDensity ExtensiveEngineeringinstallationsAdapt businessrequirements

System Availability SystemAvailability Eliminatehotspots redundancy Eliminate vertical temperature Eliminateverticaltemperature LiquidleaksHumanError

LifecycleCost Capitalinvestmentvs.Operationalefficiency Serviceability MeantimetorecoverManageability predictivefailureanalysisg y p y

Data Center Power DistributionDataCenterPowerDistribution

P i k t i t i il bilit i d t Poweriskeytomaintainavailabilityinadatacenter

US claim to lose $164 Billion a year due to power USclaimtolose$164Billionayearduetopowerdisruptions

Each Data Center must have a separate and EachDataCentermusthaveaseparateanddedicatedpowersourceandpowerinfrastructure

Power from the utility to the DC is either stepped PowerfromtheutilitytotheDCiseithersteppeddownorsteppedupatleast6timesbeforeitreachestheDCorUser.

Nominalvs.NormalVoltage

Data Center Rack Power DistributionDataCenterRackPowerDistribution

R k P S t t b d t bl t th RackPowerSystemsmustbeadaptabletothefollowing EquipmentEquipment OnDemand SafetyHazards Adverselyeffectthesystem

P tt h th t k i t Patternsshowthatrackpowerrequirementsareonanincreasefromyeartoyear.i.e.2004 2kWrack

2008 patterns have shown that the average rack power2008patternshaveshownthattheaveragerackpowerrequirementisat6kWperrack.

Advancementarebeingmadetowardsthe15kWrackgatarapidrate.

DataCenterRackPowerDistribution HereisabreakdownofEquipmentvs.kiloWattsq p

VeryLowloads PatchPanels Switches

Loadsin1kWrange16AmpEntry Level 1u severs EntryLevel1usevers

Loadsin23kWrange1620Ampcircuitsonly Mediumlevel1uservers Switches Majorityofrackopen

Loadsin5kWrange 20 30Ampsdependingonwhatsconnected Halffilledwith1uhighendServers Routers

Mix of technologies Mixoftechnologies

7kw+Range 30AmpCircuitsOnly BladeServers HPC3000 StorageDevices SwitchingGear ExtremelyRare

Data Center Power DistributionDataCenterPowerDistribution

VoltagecomingintotheDCmustbechecktoverifyifitistoohighortoolowasthiscany gimpacttheequipment.

Therefore Delta transformers must be ThereforeDeltatransformersmustbeinstalled.

Steppingupordown3phasepowermustusea delta transformer.adeltatransformer.

Data Center Power DistributionDataCenterPowerDistribution

TherearetwotypesofTransformers Delta Transformers use 3 phase power withDeltaTransformers use3phasepowerwithGround

W T f 3 h i h WyeTransformers use3phasepowerwithaNeutral

Themostcommontransformerconfigurationis a Delta to WyeisaDeltatoWye.

Data Center Power RequirementsDataCenterPowerRequirements

l l i i l l d Calculatecriticalload NamePlaterating worsecaserating Voltagerequirement SingleorThreePhaseg

UPSRequirementsmultiplying the incoming current in amps by the multiplyingtheincomingcurrentinampsbythevoltageofthedevicewithgiveyourVArating,then multiply the anticipated VA number by 0.67thenmultiplytheanticipatedVAnumberby0.67toestimatetheactualpower,inWattsandthendivideby1000todetermineforkWratingy g

Generator StatsGeneratorStats

AccordingtoContingencyPlanningResearch

powerrelatedeventssuchas;

bl k t d t f 31% f blackoutsandsurgesaccountfor31%ofcomputerdowntimeepisodeslastingmorethan12hours,

power failure and surges account for 45 3% of data loss and powerfailureandsurgesaccountfor45.3%ofdataloss,and accordingtoIDCpowerdisturbancesaccountforabout33%ofall

f lserverfailures.

ThereforestandbygeneratorsareaessentialcomponentinaDataCenter

Data Center Standby GeneratorsDataCenterStandbyGenerators

d f h f ll i Generatorsaremadeupofthefollowing: PrimeMover 4stroke Alternator Governor

MaintainRPM/ACoutputQuality IsochronousDesign SuperiorElectronicDesign

DistributionSystems Automatictransferswitch, SwitchingGear

PhysicalInfrastructureManagementBasics

C I CoreIssuesMaintainingSystemAvailability;andManaging problems and changesManagingproblemsandchanges

E ti l C t i f I f t t EssentialCategoriesforInfrastructureManagementare

Incident management Incidentmanagement Changemanagement Availability managementAvailabilitymanagement Capacitymanagement

PhysicalInfrastructureManagementBasics

Availability ManagementAvailabilityManagement

Systemicallyidentifyingavailabilityrequirements

andReliabilityrequirementsagainstactualperformance

Physical ChangeI id t t Infrastructure

ManagementBasics

Management

Focusesonthemethodsandproceduresfor

makingchangesintheDCInfrastructure

Incidentmanagement

ReturntonormalASAPwithlittleimpacton

Business

CapacityManagement

Providing IT Resources atProvidingITResourcesattherighttime;rightcostandmustbealignedwith

currentandfuturerequirements.

PhysicalinfrastructuremanagementBasics

Ph i l i f t t t i ll k d i Physicalinfrastructuremanagementrequiresallkeydevicesanddatapointstobemonitored.Theseincludeallthedevicesinthephysicalinfrastructurelayerandthesurroundingenvironment.

Best practices dictate that the following list of devices be Bestpracticesdictatethatthefollowinglistofdevicesbemonitoredattheracklevel:

Aminimumoftwotemperaturedatapointsp p Individualbranchcircuits Transferswitches Coolingdevices,and UPSsystems

DataCenterFireProtection,preventionandsuppression

43%ofbusinessthatcloseduetofire,neverreopenp

29%whodoreopenfailwithin3yearsTh f i i i l d Thereforeitisessentialasadatacentertoprepareforunseendisasters.

ThebestFireprotectionmethodisFirepreventionprevention

DataCenterFireProtection,preventionandsuppression

NFPA 75 i th t d d f t ti f t d t NFPA75isthestandardforprotectionofcomputerordataprocessingequipment.OnenotableadditiontoNFPA75thattookplacein1999,allowsdatacenters tocontinueto

fpowerelectronicequipmentuponactivationofaGaseousAgentTotalFloodingSystem,whichwewilldiscusslaterindetail.Thisexceptionwasmadefordatacenters thatmeetpthefollowingriskconsiderations:

Economiclossthatcouldresultfrom:L f f ti l f d Lossoffunctionorlossofrecords

Lossofequipmentvalue Loss of life Lossoflife andtheriskoffirethreattotheinstallation,tooccupants

orexposedpropertywithinthatinstallation

DataCenterFirePrevention

ItisimportanttomaintainthefollowingguidelinesinDataCenterforFirePreventiong DonotbuildaDCclosetoourbuildingAll furniture must be constructed in Metals except AllfurnituremustbeconstructedinMetalsexceptforchairs

D t ll ki i d DonotallowanysmokinginoraroundyourcommunicationandITfacilitieswithintheDC.

DCsshouldbevoidofanytrashreceptacles Noacousticalmaterialsuchasfoamorfabric

DataCenterFireProtection

Thereare3systemobjectivestoFireProtection FireDetection

Detecting the presence of fireDetectingthepresenceoffire Communication

C i t th i t f fi Communicatetheexistenceofafire Suppression

Containandextinguishthefire

DataCenterFireProtection

Thefourstagesofcombustionare: 1.Theincipientstageorprecombustionstage,p g p g , 2.Thevisiblesmokestage,3 The flaming fire stage and lastly 3.Theflamingfirestage,andlastly

4.Theintenseheatstage.

DataCenterDetectors

Th t t f S k d t t TherearetwotypesofSmokedetectors Airsamplingsmokedetectors

VESDA Highpoweredphotoelectric 200 7000sqm Usesalaserbeam

Intelligentspottypedetectors More sensitive Moresensitive Theyusealasertoscanparticles Individualaddressable

EPO EmergencyPowerOff

DataCenterFireSuppression

f i f 2typesofSuppressionagentsforDCs FireExtinguishers Totalfloodingfireextinguishingsystems

2commonlyusedagentsinDCsInert gas Inergen InertgasInergen

FluorinebasedcompoundsFE 200 FE200

FE227ea

DataCenterAvailabilityandReliabilityFundamentals93% f i h h l il bili f 10d 93%ofcompaniesthathavelostavailabilityfor10daysormorehavefiledforbankruptcyin1yearA il bilit d li bilit b ilt NCPI AvailabilityandreliabilityarebuiltonNCPI

NCPIarebuilton3corebusinessobjectivesI i Increaseinturnover

ReductionincostBetter utilization of assets Betterutilizationofassets

Availabilityisdeterminedbyasystemsreliabilityandits recovery timeitsrecoverytime.

Five9sofAvailability

DataCenterAvailabilityandReliabilityFundamentalsF t ff ti A il bilit d R li bilit FactorsaffectingAvailabilityandReliability ACPowerConditions

Sag / under voltageSag/undervoltage Swell/overvoltage transients

Cooling Cooling HotSpots

EquipmentFailure TemperatureSwings

NaturalDisasters Blackouts Blackouts

HumanError PoortrainingandDocumentationmanagement

Data CenterDataCenterGeneralDesignGuidelines

Data Centre StandardsDataCentreStandards

TIA942TelecommunicationsInfrastructurestandardforDataCentres,April2005p

EN501735InformationTechnology datacentres EuropecentresEurope

ANSIBICSI002DataCenterStandardcomplementarytoTIA942

ISO/IEC NP 24764 Information Technology ISO/IECNP24764 InformationTechnologyGenericcablingforDataCentres

Others Standards within Data CentresOthersStandardswithinDataCentres

TR / ERTR/ER

l i i i l d ATelecommunicationsRoomisanenclosedarchitecturalspaceforhousingt l i ti i t bltelecommunicationsequipmentcableterminationsandcrossconnectcabling.

AnEquipmentRoomisanenvironmentallycontrolledcentralizedspacefortelecom

iequipment DifferencebetweenTRandERsarethatERsserveabuildingorcampuswhileTRsserveafloorinbuilding.

TR / ER SizingTR/ERSizing

f 60 l b 3 2 TRsof460sqmorlessmustbe3mx2.4m TRsof929sqmandnotlessthan740sqmmustbe3mx3.4m

ERminimumsizemustbe3x5m Generally allow 9 sq m per work area Generallyallow9sqmperworkarea Multiplynoofworkareasby0.07sqm

000 5000sqm 5000/9=555x0.07=39sqmofERspace.

Ideal Computer Room SizeIdealComputerRoomSize

Theidealroomsizewouldbenotlargerthan600sqmq

TheyrequirelargeamountsofairconditioningCRAC / CRAH CRAC/CRAH(ComputerRoomAirConditioning)Calculationsbecomedifficultwhentheroomisbiggerthan600sqm

Gas Fire Suppression quantities will be huge GasFireSuppressionquantitieswillbehuge.

Requirements for a Data CenterRequirementsforaDataCenterSpace Required FunctionSpaceRequired Function

ComputerRoom/ServerRoom HousecomputerRacksandCommunicationsEquipment

ControlRoom allcontrolandmonitoringfunctionsareconcenrated

GeneralOfficeArea OfficeareawhereITstaffcanwork

EntranceFacility wherealltheexternalcommunicationscablingenterthebuilding

FireGasSuppressionStore StoragespacedependsonwhichgasisusedInertgase orHalocarbons

ElectricalSwitchRoom Externalpowercablesenterthebuildingandformsademarcationpoint

UPSandBatteryRoom Loadsof100kVAitisrecommendedtohaveaseperateUPSroom

GeneratorRoom houseastandbyDieselGenerator

Requirement for Data center condRequirementforDatacentercon d

S i iSpaceRequirement Function

OilStore House Dieselfuelforgenerator

Storage and Build Area To store and unpack equipment to buildStorageandBuildArea Tostoreandunpackequipment tobuild

DeliveryandLoadingArea Adjacent areatoallowheavyequipmenttobeshippedintobuilding

PlanningandMeetingRoom Toholdmeetings

Internalstafffacilities Male/female/disabled toiletsandshowers

Electrical Substation Due to power needs a separate substationElectricalSubstation Duetopowerneeds aseparatesubstationmayberequired

AirConditioningCondensers Asecureareaisrequireddependingonth i i ttheaircon requirements

Externalstafffacilities Parkingspace,bicycle andsmokeshelter

Facilities RequirementsFacilitiesRequirements

i i d i h RoomDimensionsandHeight FloorStrengthg ConnectionofServices External Services ExternalServices Access,LoadandDDA Decor LightingLighting FireRegulations

Data Center Room HeightDataCenterRoomHeight

MinimumCeilingheightmustbe2.6mfromthefinishedfloortoanyobstructionsuchasysprinklers

SlabtoSlabmustbe2.9mi.e. 400mmunderFloor 2 1m racks2.1mracks 400mmairreturnpathforCRAC

Data Center Floor StrengthDataCenterFloorStrength

i i di ib d fl l di i Minimumdistributedfloorloadingcapacitymustbe7.2kPA

Therecommendedloadingcapacityis12kPA The floor must have minimum of 1 2kPAThefloormusthaveminimumof1.2kPAhangingcapacityforsupportingloadsthataresuspended from the bottom of the floorsuspendedfromthebottomofthefloor.

Therecommendedhangingcapacityofthefl i 2 4kPAflooris2.4kPA

KPA/1000=N/persqmeter

Data Center LightingDataCenterLighting

Aminimumof500lux inthehoriziontal planeand200lux intheverticalplan.p

Thismustbemeasured1mabovethefinishedfloor in the middle of all aisles betweenfloorinthemiddleofallaislesbetweencabinets.

Data Center AccessDataCenterAccess

Doorsmustbeaminimumof1mwideand2.13mhighwithoutdoorsills,hingedtoopeng g poutward

Door must be fitted with locks and have either Doormustbefittedwithlocksandhaveeithercenterpostsorremovablecenterpoststof l f lfacilitateaccessforlargeequipment

DDA Ramps to be not less than 1.12 and allDDA Rampstobenotlessthan1.12andallpathwaystobe900wide

Data Center DecorDataCenterDecor

Floors,wallsandceilingmustbe Sealed Paintedand,Constructed out of material to minimize dust Constructedoutofmaterialtominimizedust.

FloorsmusthaveantistaticpropertiesasperIEC6100042

Data Center Fire RegulationsDataCenterFireRegulations

FirePlanandRiskassessment Emergency Lighting and SignageEmergencyLightingandSignage DoorandEmergencyexits

MUST ALL BE CLEARLYMARKEDMUSTALLBECLEARLYMARKED

Data Center Connection ServicesDataCenterConnectionServices

El t i it S l ElectricitySupply Whofrom? WhatCapacity?p y Wheredoesitenterthebuilding? Istheremorethanone?T l i ti Telecommunications Whofrom? Whatcapacity/typep y / yp Wheredoesitenterthebuilding Istheremorethanone.G t d i d Gas,water,drainageandsewage

Nootherserviceshouldcrossthecomputerroomspace

Data Center External servicesDataCenterExternalservices

S it bl l ti f t db t Suitablelocationforstandbygenerators Whatnoise? Whatweight? WhatSecurity WhatFuelStorage Whatproximity

Suitablelocationforexternalairconditioningcomponents What DX condenser units?WhatDXcondenserunits? Whatcentralchiller system? Whatweight?

What security? Whatsecurity? Whatpowersupplies? Whatproximity

ENDEND