1|P a g e

KeyInformationandNumbers–CERNDataCentre Asof1stDec2017_V1

GeneralInformationWebsites

Latestnumbers:http://monit-grafana-displays.cern.ch:3000/dashboard/db/it-overviewHowtousetheDataCentrevisitpoint:http://www.cern.ch/information-technology/about/computer-centre/visits/visitpoint

Keyfactsandnumbers

MeyrinDataCentre

WignerExtension TOTAL

Servers 11300 3500 14800=>15K Processorcores 170000 56000 226000=>230K

Disks 61800 29700

91500=>90K(units)(about200PBcapacitybutdataiscopiedtwiceondisks)

TapeCartridges 28800=>29K(units)(about400PBcapacity)

Shortoverview

TheCERNdatacentreistheheartofCERN’sentirescientific,administrative,andcomputinginfrastructure.Allservices,includingemail,scientificdatamanagementandvideoconferencinguseequipmentbasedinthedatacentre.AremoteextensionoftheCERNdatacentreishostedattheWignerResearchCentreforPhysicsinBudapest,Hungary,1800kmaway.ItprovidestheextracomputingpowerrequiredtocoverCERN’sneeds.The230000processorcoresand15000serversrun24/7.On29June2017,theCERNDataCentrepassedthemilestoneof200petabytesofdatapermanentlyarchivedinitstapelibraries.NewrecordinNovember2017:12.3petabytesofdatawerewrittenontapethatmonth.Usuallymorethan2PBarebeingaccessed(readandwrite)everyday.TheLHCdataareaggregatedintheCERNDataCentre,whereinitialdatareconstructionisperformed,andacopyisarchivedtolong-termtapestorage.Anothercopyissenttoseverallarge-scaledatacentresaroundtheworld.Subsequentlyhundredsofthousandsof

2|P a g e

computersfromaroundtheworldcomeintoaction:harnessedinadistributedcomputingservice,theyformtheWorldwideLHCComputingGrid(WLCG).

STORAGE:Websites:

www.cern.ch/eosMagnetictapesareusedasthemainlong-termstoragemedium.TheyarestoredintapelibrarieswhereroboticarmswhichWehavemanytaperobotstoensureefficientstorageandretrievalofdata.

- TheamountofdatarecordedontapeatCERNissteadilyincreasingovertime,withthistrendacceleratingovertime(+40%datastoredin2016comparedtowhathadbeenaccumulatedbytheendof2015).

- In2016,morethan49petabytesofLHCdatawererecordedattheCERNDataCentre(withastriking11petabytesinthemonthofJulyalone).Intotal,73PBofdatawererecordedontapes(fromLHCandnonLHCexperiments,etc.).

- Asoftoday,about28PBhavebeenstoredontapesin2017(fromallsources).- Accessingtapedataisrelativelyslow,about1-3minutesfromtapebeinglocated,mounted,

readanddatasent.Oftenphysicistsneedtoaccessthelatestdataimmediately,soitisalsomadeavailableondiskservers,whereaccesstimeissignificantlyfaster.

- Therearecurrently90,000disksintheCERNdatacentrewithacapacityofabout200PB- http://home.cern/about/updates/2017/07/cern-data-centre-passes-200-petabyte-

milestone:On29June2017,theCERNDataCentrepassedthemilestoneof200petabytesofdatapermanentlyarchivedinitstapelibraries.Wheredothesedatacomefrom?ParticlescollideintheLargeHadronCollider(LHC)detectorsapproximately1billiontimespersecond,generatingaboutonepetabyteofcollisiondatapersecond.However,suchquantitiesofdataareimpossibleforcurrentcomputingsystemstorecordandtheyarehencefilteredbytheexperiments,keepingonlythemost“interesting”ones.ThefilteredLHCdataarethenaggregatedintheCERNDataCentre,whereinitialdatareconstructionisperformed,andwhereacopyisarchivedtolong-termtapestorage.Evenafterthedrasticdatareductionperformedbytheexperiments,theCERNDataCentreprocessesonaverageonepetabyteofdataperday.

- Asoftoday,thereare214PBontapesandthetotalcapacityontapesreaches400PB(thiscanbeextendedasmuchasneededthough).

- Asoftoday,wehave145PBondisksandatotalcapacityof200PBondisks.However,asdisksarenotreliable(athousandtimeslessthantapes,30disksfaileachweekinthedata

3|P a g e

centre),wealwayscopytwicethedataondisksothatifadiskfailswedonotlosedata=>soonly72.5PBof‘real’dataisondisksatthemomentandthe‘real’capacityisabout100PB.

- TheCERNstoragesystem,EOS,wascreatedfortheextremeLHCcomputingrequirements.In2016,EOSinstancesatCERNapproachedonebillionfiles,matchingtheexceptionalperformancesoftheLHCmachineandexperiments.

ELECTRICITY:- 2.7megawattcomputingpowerconsumption(coolingnotincludedandrepresentsroughly

oneadditionalmegawatt)fromamaximumof3.5megawatt- DatacentreprotectedbyUPS(UninterruptablePowerSupply)=>Allowstostartdiesel

generatorsforcriticalsystemsandprovidestimetoshutdownnon-criticalsystems- Theelectricalinfrastructureisavitalelementofthedatacentre.Strategiesforincreasing

powerefficiencyarepermanentlyinvestigatedtobeabletomaximisethecomputingpowerservingtheCERN’sinfrastructureandscientificprogrammewhilststayingwithinthe3.5megawattelectricalcapacityenvelopeavailableinthedatacentre.

- Inthecaseofamajorelectricalcut,UninterruptiblePowerSupplies(UPS)providetimeforallthenon-criticalsystemstobeproperlyshutdownandacombinationofUPSsystemsanddieselgeneratorsensurethatthecriticalserviceskeepworking.

COOLING:- About1.4megawattisdedicatedtothecoolingofthedatacentre.- MainMachineroom:chilledairviasilverducts,intofalsefloorandthenintoclosedserver

aisles.Wealsousewater-cooledracksinsomeareas(basement:‘vault’).Efficientcoolingisakeyelementofthedatacentre.Whentheoutsidetemperatureislow,aircanbeusedtocooltheservers,otherwisechillersystemsareusedtocoolthedatacentreair.Coldairisdistributedviathesilverductsonthesidesofthedatacentreroom.Itthengoesunderthefalsefloorandintotheclosedserveraislesthroughtheperforatedfloortiles,tobedrawnfinallythroughtheserverstocoolthem.Someserversarewater-cooledusingactiveorpassiveheatexchangersintheirreardoors,providingahighercoolingcapacityperrack.

WLCG–WorldwideLHCComputingGridWebsites:

www.cern.ch/wlcgwww.cern.ch/wlcg-public

Keyfactsandnumbers:●Morethan170datacentresin42countrieswithmorethan600,000cpucores●CERNprovidesabout20%oftheWLCGresources●Allowsmorethan10,000physiciststoaccessLHCdata●>250,000jobsrunconcurrentlyontheGrid●In2016,globaltransferrateshaveregularlyexceeded35GB/s

Shortoverview:

4|P a g e

Today,theWorldwideLHCComputingGrid(WLCG)combinesthecomputingresourcesofmorethan600,000coresfromover170sitesin42countries,producingamassivedistributedcomputinginfrastructurethatprovidesmorethan10000physicistsaroundtheworldwithnearreal-timeaccesstoLHCdata,andthepowertoprocessit.TheWorldwideLHCComputingGridrunsover2,000,000tasksperdayandin2016,globaltransferrateshaveregularlyexceeded35GB/s.Thesenumberswillincreaseastimegoesonandascomputingresourcesandnewtechnologiesbecomeevermoreavailableacrosstheworld.CERNprovidesabout20%oftheresourcesofWLCG.

NETWORK:Webpage:

https://netstat.cern.ch/monitoring/network-statistics/ Keyfactsandnumbers:

- 35,000kmofopticalfibreneededtotransferlargeamountofLHCdatafromtheLHCexperimentstotheMeyrindatacentre.

- NetworkconnectionsforLHCdata:

- Three100GigabitpersecondcircuitsconnecttheMeyrinDataCentreanditsextensionin

Hungary(cf.below).

5|P a g e

- In2016,thedatatransferratesaroundtheglobealsoreachednewpeakrates–between30

and40gigabytespersecondcontinuousrates,aroundafactoroftwohigherthanhadbeentypicalduringRun1.

DataPreservation:Cf.computingchapteroftheCERNannualreport2016:https://e-publishing.cern.ch/index.php/Annual_Report/article/view/496/354

OpenSource:

Cf.computingchapteroftheCERNannualreport2016:https://e-publishing.cern.ch/index.php/Annual_Report/article/view/496/354

LHC@home:www.cern.ch/lhcathomeVolunteercomputingfortheLHC

CERNopenlab:www.cern.ch/openlabCERNopenlabisauniquepublic-privatepartnershipthatacceleratesthedevelopmentofcutting-edgesolutionsfortheworldwideLHCcommunityandwiderscientificresearch.ThroughCERNopenlab,CERNcollaborateswithleadingICTcompaniesandresearchinstitutes.EUprojects:www.cern.ch/information-technology/about/projects/eu/eu-funded-projectsListofthecurrentEUprojectshttp://information-technology.web.cern.ch/about/projects/eu/current/projectsUNOSAT:https://unitar.org/unosat/CERN’spowerfulITinfrastructureisusefulforfieldsotherthanfundamentalresearch.For15years,UNOSAThasbeenusingtheLaboratory’scomputingcentreinfrastructureforthepurposesofitshumanitarianwork.

6|P a g e

Toknowmore:http://home.cern/about/updates/2016/10/unosat-15-years-humanitarian-mappinghttp://cds.cern.ch/record/2223516?ln=enTHEPAST:

• Workinprogress,aguideforguidesaboutthehistoryoftheDataCentreisbeingprepared.• Thefirstsectionofthisvideoisgivesabriefoverviewofthehistoryofthedatacentre(but

therestofthevideoisnotanymoreuptodateintermsofnumbers,etc.):CERNITin8minutes(2013)https://cds.cern.ch/record/1604210

THEFUTURE:

• Workinprogress(thissectionwillbefilledatalaterstage)Thankyoutoallthepeoplewhohaveproducedmaterialthathasbeenincluded/copypastedinthisdocument.PleasedonothesitatetomodifyasappropriatethisdocumentwiththetrackchangesfeatureonandtosenditbacktoMélissaGaillardsothatthemodificationsaretakenintoaccountinthenextversion(thisdocumentwillbeupdatedonamonthlybasisandmadeavailableontheITwebsiteandaspartoftheITMMminutesonamonthlybasis).