Extreme Scaling vs. Energy Eﬃciency – A Challenge for Computer … · 2015-11-25 · n 2nd LRZ...

1

ExtremeScalingvs.EnergyEfficiency–AChallengeforComputerScience

DieterKranzlmüllerMunichNetworkManagementTeamLudwig-Maximilians-UniversitätMünchen(LMU)&LeibnizSupercompuFngCentre(LRZ)oftheBavarianAcademyofSciencesandHumaniFes

LeibnizSupercompu?ngCentreoftheBavarianAcademyofSciencesandHumani?es

D.Kranzlmüller ACOMP2015 2

Cuboidcontaining

compuFngsystems72x36x36meters

InsFtuteBuilding

InsFtuteBuilding

LectureHalls

VisualisaFonCentre

With156employees+38extrastaffformorethan100.000studentsandformorethan30.000employeesincluding8.500scienFsts

2


n ComputerCentreforallMunichUniversiFes


ITServiceProvider:• MunichScienFficNetwork(MWN)• Webservers• e-Learning• E-Mail• Groupware• Specialequipment:

•  VirtualRealityLaboratory•  VideoConference•  Scannersforslidesandlarge

documents•  Largescaleploders

ITCompetenceCentre:• Hotlineandsupport• ConsulFng(security,networking,scienFfccompuFng,…)• Courses(textediFng,imageprocessing,UNIX,Linux,HPC,…)

D.Kranzlmüller

TheMunichScien?ficNetwork(MWN)

ACOMP2015 4

3


n RegionalComputerCentreforallBavarianUniversiFes

n ComputerCentreforallMunichUniversiFes


VirtualReality&Visualiza?onCentre(LRZ)


4

ExamplesfromtheV2C



n  NaFonalSupercompuFngCentre

n  RegionalComputerCentreforallBavarianUniversiFes

n  ComputerCentreforallMunichUniversiFes


5

GaussCentreforSupercompu?ng(GCS)

n  CombinaFonofthe3GermannaFonalsupercompuFngcenters:–  JohnvonNeumannInsFtuteforCompuFng(NIC),Jülich–  HighPerformanceCompuFngCenterStudgart(HLRS)–  LeibnizSupercompuFngCentre(LRZ),Garchingn.Munich

n  Foundedon13.April2007

n  HosFngmemberofPRACE(PartnershipforAdvancedCompuFnginEurope)


PRACEResearchInfrastructureCreated

n  Establishmentofthelegalframework–  PRACEAISBLcreatedwithseatinBrusselsinApril

(AssociaFonInternaFonaleSansButLucraFf)–  20membersrepresenFng20Europeancountries–  InauguraFoninBarcelonaonJune9

n  Fundingsecuredfor2010-2015–  400Million€fromFrance,Germany,Italy,Spain

ProvidedasTier-0servicesonTCObasis–  Fundingdecisionfor100Million€inTheNetherlands

expectedsoon–  70+Million€fromECFP7forpreparatoryandimplementaFon

GrantsINFSO-RI-211528and261557Complementedby~60Million€fromPRACEmembers


6

PRACETier-0Systems

n  Curie@GENCI:BullCluster,1.7PFlop/s

n  FERMI@CINECA:IBMBG/Q,2.1PFlop/s

n  Hermit@HLRS:CrayXE6,1Pflop/s

n  JUQUEEN@FZJ:IBMBlueGene/Q,5.9PFlop/s

n  MareNostrum@BSC:IBMSystemXiDataPlex,1PFlop/s

n  SuperMUC@LRZ:IBMSystemXiDataPlex,3.2PFlop/s



n  EuropeanSupercompuFngCentre

n  NaFonalSupercompuFngCentre

n  RegionalComputerCentreforallBavarianUniversiFes

n  ComputerCentreforallMunichUniversiFes


SGI UV

SGI Altix

Linux Clusters

SuperMUC

Linux Hosting and Housing

7

SuperMUC@LRZ


Video: SuperMUC rendered on SuperMUC by LRZ

hdp://youtu.be/OlAS6iiqWrQ

Top500SupercomputerList(June2012)


www.top500.org

8

LRZSupercomputers

D.Kranzlmüller 15

SuperMUCPhaseII

ACOMP2015

SuperMUC Phase 1 + 2


9

InFrontof6.4PFlop/s


ChallengesinProgrammingandUsingtheseSupercomputers

D.Kranzlmüller ACOMP201518

10

SuperMUC and its predecessors


10m



10m

22m

11m

11



10m

22m

11m

LRZBuildingExtension


Picture: Ernst A. Graf

Picture: Horst-Dieter Steinhöfer

Figure: Herzog+Partner für StBAM2 (staatl. Hochbauamt München 2)

12

I/O

nodes

NAS 80 Gbit/s

18 Thin node islands (each >8000 cores)

1 Fat node island (8200 cores) è SuperMIG

$HOME 1.5 PB / 10 GB/s

Snapshots/Replika 1.5 PB

(separate fire section)

non blocking

non blocking

pruned tree (4:1)

SB-EP 16 cores/node

2 GB/core

WM-EX 40cores/node 6.4 GB/core

10 PB

200 GB/s

GPFS for $WORK

$SCRATCH

Internet Achive and Backup ~ 30 PB

Desaster Recovery Site

Compute nodes Compute nodes

SuperMUCArchitecture


PowerConsump?onatLRZ


0

5.000

10.000

15.000

20.000

25.000

30.000

35.000

Stro

mve

rbra

uch

in M

Wh

13

CoolingSuperMUC


SuperMUCPhase1&2@LRZ


14

LRZ Application Mix

q  Computational Fluid Dynamics: Optimisation of turbines and wings, noise reduction, air conditioning in trains

q  Fusion: Plasma in a future fusion reactor (ITER) q  Astrophysics: Origin and evolution of stars and galaxies q  Solid State Physics: Superconductivity, surface properties q  Geophysics: Earth quake scenarios q  Material Science: Semiconductors q  Chemistry: Catalytic reactions q  Medicine and Medical Engineering: Blood flow, aneurysms, air

conditioning of operating theatres q  Biophysics: Properties of viruses, genome analysis q  Climate research: Currents in oceans


Increasing numbers

Date System Flop/s Cores

2000 HLRB-I 2 Tflop/s 1512

2006 HLRB-II 62 Tflop/s 9728

2012 SuperMUC 3200 Tflop/s 155656

2015 SuperMUC Phase II 3.2 + 3.2 Pflop/s 229960


15

SuperMUC Jobsize 2015 (in Cores)

ACOMP2015D.Kranzlmüller 29

ChallengesonExtremeScaleSystems

n  Size:numberofcores>100.000

n  Complexity/Heterogeneity

n  Reliability/Resilience

n  EnergyconsumpFonaspartofTotalCostofOwnership(TCO)–  ExecutecodeswithopFmalpowerconsumpFon

(orwithinacertainpowerband)èFrequencyscaling–  OpFmizeforenergy-to-soluFon

èAllowmorecodeswithingivenbudget–  Improvedperformance

è(inmostcases)improvedenergy-to-soluFon


16

1st LRZ Extreme Scale Workshop

n  July2013:1stLRZExtremeScaleWorkshop

n  ParFcipants:–  15internaFonalprojects

n  Prerequisites:–  Successfulrunon4islands(32768cores)

n  ParFcipaFngGroups(Sotwarepackages):–  LAMMPS,VERTEX,GADGET,WaLBerla,BQCD,Gromacs,APES,SeisSol,CIAO

n  Successfulresults(>64000Cores):–  InvitedtoparFcipateinPARCOConference(Sept.2013)includinga

publicaFonoftheirapproach


1st LRZ Extreme Scale Workshop

n  Regular SuperMUC operation –  4 Islands maximum –  Batch scheduling system

n  Entire SuperMUC reserved 2,5 days for challenge: –  0,5 Days for testing –  2 Days for executing –  16 (of 19) Islands available

n  Consumed computing time for all groups: –  1 hour of runtime = 130.000 CPU hours –  1 year in total


17

Results (Sustained TFlop/s on 128000 cores)


Name MPI #cores Description TFlop/s/island TFlop/smaxLinpack IBM 128000 TOP500 161 2560Vertex IBM 128000 PlasmaPhysics 15 245GROMACS IBM,Intel 64000 MolecularModelling 40 110Seissol IBM 64000 Geophysics 31 95waLBerla IBM 128000 LatticeBoltzmann 5.6 90LAMMPS IBM 128000 MolecularModelling 5.6 90APES IBM 64000 CFD 6 47BQCD Intel 128000 QuantumPhysics 10 27

Results

n  5 Software packages were running on max 16 islands: –  LAMMPS –  VERTEX –  GADGET –  WaLBerla –  BQCD

n  VERTEX reached 245 TFlop/s on 16 islands (A. Marek)


18

Extreme Scaling Continued

n  Lessons learned è Stability and scalability

n  LRZ Extreme Scale Benchmark Suite (LESS) will be available in two versions: public and internal

n  All teams will have the opportunity to run performance benchmarks after upcoming SuperMUC maintenances

n  2nd LRZ Extreme Scaling Workshop è 2-5 June 2014 –  Full system production runs on 18 islands with sustained Pflop/s

(4h SeisSol, 7h Gadget) –  4 existing + 6 additional full system applications –  High I/O bandwidth in user space possible (66 GB/s of 200 GB/s max) –  Important goal: minimize energy*runtime (3-15 W/core)

n  Extreme Scale-Out with new SuperMUC Phase 2


ExtremeScale-OutSuperMUCPhase2

n  12May–12June2015(30days)n  SelectedGroupofEarlyUsers

n  NightlyOperaFon:generalqueuemax3islandsn  DayFmeOperaFon:specialqueuemax6islands(fullsystem)

n  Totalavailable:63,432,000corehoursn  Totalused:43,758,430corehours(UFlisaFon:68.98%)

Lessonslearned(2015):n  PreparaFoniseverythingn  FindingHeisenbugsisdifficultn  MPIisatitslimitsn  Hybrid(MPI+OpenMP)isthewaytogon  I/Olibrariesgewngevenmoreimportant


19

PartnershipIni?a?veComputa?onalSciencesπCS

n  IndividualizedservicesforselectedscienFficgroups–flagshiprole–  Dedicatedpoint-of-contact–  Individualsupportandguidanceandtargetedtraining&educaFon–  PlanningdependabilityforusecasespecificopFmizedITinfrastructures–  EarlyaccesstolatestITinfrastructure(hard-andsotware)developments

andspecificaFonoffuturerequirements–  AccesstoITcompetencenetworkandexperFseatCSandMathdepartments

n  Partnercontribu?on–  EmbeddingITexpertsinusergroups–  Jointresearchprojects(includingfunding)–  ScienFficpartnership–equalfooFng–jointpublicaFons

n  LRZbenefits–  Understandingthe(currentandfuture)needsandrequirementsofthe

respecFvescienFficdomain–  Developingfutureservicesforallusergroups–  ThemaFcfocusing:EnvironmentalCompu?ng

D. Kranzlmüller ACOMP2015 37

SeisSol-NumericalSimula?onofSeismicWavePhenomena


Picture:AlexBreuer(TUM)/ChrisFanPelFes(LMU)

Dr.ChrisFanPelFes,DepartmentofEarthandEnvironmentalSciences(LMU)Prof.MichaelBader,DepartmentofInformaFcs(TUM)

1,42Petaflop/son147.456CoresofSuperMUC(44,5%ofPeakPerformance)

hdp://www.uni-muenchen.de/informaFonen_fuer/presse/presseinformaFonen/2014/pelFes_seisol.html

20

ExtremeScaling-Conclusions

n  Thenumberofcomputecores,thecomplexity(andheterogeneity)issteadilyincreasing

n  Usersneedtopossibilitytoreliablyexecute(andopFmize)theircodesonthefullsizemachineswithmorethan100.000cores

n  TheExtremeScalingWorkshopSeries@LRZoffersanumberofincenFvesforusersèNextWorkshopSpring2016

n  ThelessonslearnedfromtheExtremeScalingWorkshopareveryvaluablefortheoperaFonofthecenter

n  TheLRZPartnershipIniFaFveComputaFonalScience(piCS)triestoimproveusersupport

hdp://www.sciencedirect.com/science/arFcle/pii/S1877050914003433



Extreme Scaling vs. Energy Efficiency - A Challenge for Computer Science

Dieter Kranzlmüller

[email protected]

Extreme Scaling vs. Energy Eﬃciency – A Challenge for Computer … · 2015-11-25 · n 2nd LRZ...

Documents

Transcript of Extreme Scaling vs. Energy Eﬃciency – A Challenge for Computer … · 2015-11-25 · n 2nd LRZ...