CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Options for Expanding CERN’s Computing Capacity Without A

New Building

Medium term plans to handle growth of CERN’s computing needs

HEPiX NERSC

Alan Silverman

(with input by T.Cass, L.Pregernig and W.Salter)29th October 2009

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Outline

• Background• Options – hosting, containers, new

centre• Features of Hosting• Features of Containers• IT’s plans• Current status

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Limits of Building 513 Capacity

• Building 513 arriving to maximum capacity for power and cooling while demands to expand LCG and infrastructure services continue to grow.

• Limit probably now reached with respect to “critical” systems – systems whose loss would mean major disruption (corporate data and physics meta-data, mail servers, network devices, etc).

• Plans being made to accommodate more critical services but short term measures needed for next round of database servers among remaining 2009 purchases.

IT Department – 29 October 2009 – HEPiX NERSC- 3

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Long Term – A New Building?• Plans and budget exist for a Tier 0 centre – a major

new building, probably on Prevessin site.• Uncertainty created by -

– Uncertain physics needs in advance of real LHC data– Ongoing debate of cost-effectiveness of one-time new building

versus incrementally adding containers– Unknown costing of different options – Our unfamiliarity with these options

• Offer from Norway to host a remote centre -– costing unclear – remote operations support also unclear

• Prefer to take time to understand these various options before coming to a decision.

IT Department – 29 October 2009 – HEPiX NERSC- 4

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Short and Medium term

• Short-term: Price Enquiry issued to host offsite some “critical” systems to be acquired in the next round of purchases.

• Medium-term: Call for Tender to purchase (or rent) a number of Containers to install on the CERN site.

• Should be enough for the coming 2-5 years to give us time -– to get some experience of the Container option– to decide long term future – if a new building is chosen, to get it designed and built.

IT Department – 29 October 2009 – HEPiX NERSC- 5

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Hosting – choices - 1

• Rent space and use our own racks or rent space in their racks?

• If their racks, does our choice of PC fit with the specifications we select from our PC suppliers?

• Do they support the power rating we need per rack?

• Options for cooling the racks• Do we rent space now for possible expansion

or add this later? (Risk of renting non-contiguous spaces – does this even matter?)

IT Department – 29 October– HEPiX- 6

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Hosting – choices - 2

• Dual power supply? UPS cover? • Networking facilities in the Hosting Centre and

between the Hosting Centre and CERN?• Local (near CERN) so we can install our own

systems and manage them; or remote in which case what service is available? Affect on cost (Geneva prices)?

• Any software licences needed which may be tied to CERN site?

• Maintenance contracts for PCs still valid?• Data protection laws if personnel or financial data

stored offsite?

IT Department – 29 October– HEPiX- 7

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Status of Hosting Plans - 1

• Stay close to CERN so that we can get some experience with minimal risk.

• Supplier identified after Price Enquiry.• One attractive supplier did not bid because his

current installation is full and it would have cost too much for him/us to equip the next part of his building in time for our needs.

• Contract under negotiation with target of February start.

• Data protection laws in Switzerland are rather solid (fall-out from Swiss banking)

• Need 10GB network to CERN – under negotiation

IT Department – 29 October– HEPiX- 8

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Status of Hosting Plans - 2

• We have decided to use our own racks (because they are cheaper and we can use our manpower for installation)

• We will probably only rent space for immediate use and take the risk of extra space if we need it later being in a distant part of the hosting site??

• No software licence issue but some critical servers can never be off-site for this reason.

• Since the hosting site is in Geneva, no maintenance issue this time.

IT Department – 29 October– HEPiX- 9

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Container Features

• Size typically 3 by 12 metres.• Can be stacked but then you need infrastructure to

work on upper boxes.• Can be in open air (e.g. at SLAC) but easier if

indoors (e.g. Microsoft’s new supercentre in Chicago –

space for 56 containers times 2000 servers per container) -– safer access when raining/snowing – covered area to unpack equipment to install.

• If indoors, need crane or access for mobile crane to install the boxes.

• No special access needs on Building, already on the Containers themselves.

IT Department – 29 October 2009 – HEPiX NERSC- 10

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Container Features - 2

• Need access to -– power – water (even if we buy chillers we need to fill

them)– technical galleries (for the power and high-speed

networking).

• Need 400V, 3 phase power -– typically up to 15-20 racks times 15-20Kw/rack =

~300Kw per unit for computers (depends on PCs installed)

– 150Kw per unit for cooling.

IT Department – 29 October 2009 – HEPiX NERSC- 11

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

IT’s Plans for containers

• Initially buy (or rent?) 1 unit for “critical power” systems. Improves IT’s Business Continuity - different power source from 513, distance from 513.

• Plus 1 unit for LCG physics.• Plan on 1 additional unit for physics per year for up

to 4 years.• Two options (in fact decided on second):-

– Buy/rent containers and use Site Services for all power and cooling (then need power and cooling redundancy for critical power container plus UPS for physics units)

– Buy/rent a complete service which implies an extra unit to provide safe power and external chillers for the containers.

IT Department – 29 October 2009 – HEPiX NERSC- 12

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

IT’s needs for containers

• Ideally a building (Prevessin site? Meyrin West Area?) with capacity as needed

• Prevessin site offers better “business continuity”. This was chosen.

• Open ground (recommended to be concreted) with foreseen capacity for 4 years growth; ideally one area but not essential. This is the likely choice.

IT Department – 29 October 2009 – HEPiX NERSC- 13

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Status of Container Plans - 1

• Outdoor site (with a nearby small building) identified for initial set of containers. Near CCC so good networking close by.

• EN agree to supply 400V power from a nearby 18KV line and a basic water supply but no cooling facilities (so we need to acquire the chillers as well as backup diesels and UPS)

• CERN’s complicated purchase procedure has started; target installation Oct 2010

IT Department – 29 October 2009 – HEPiX NERSC- 14

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Status of Container Plans - 2

• Initial installation is planned to be –• One container for critical services – needs UPS in

addition to dual power feeds;• One container for physics – needs only dual power

feeds;• Each container will need a second container for

diesel backup and chillers (and UPS for critical services container).

• Possibility to add one physics container (probably plus services container) per year for 2-3 years.

• Open question – buy or rent? What is the rental amortisation period?

IT Department – 29 October– HEPiX- 15