Exploiting private and commercial clouds to generate on ...€¦ · • Introduction to DODAS ......
12
EOSC-hub receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 777536. eosc-hub.eu @EOSC_eu Daniele Spiga INFN ([email protected] ) on behalf of CMS Collaboration and DODAS Team. Exploiting private and commercial clouds to generate on-demand CMS computing facilities with DODAS
Transcript of Exploiting private and commercial clouds to generate on ...€¦ · • Introduction to DODAS ......
EOSC-hubreceivesfundingfromtheEuropeanUnion’sHorizon2020researchandinnovationprogrammeundergrantagreementNo.777536.
eosc-hub.eu@EOSC_eu
DanieleSpigaINFN([email protected])onbehalfofCMSCollaborationandDODASTeam.
Exploitingprivateandcommercialcloudstogenerateon-demandCMScomputingfacilities
withDODAS
CHEP2018Sofia,Bulgaria 2
Outline
DanieleSpiga
• IntroductiontoDODAS– ArchitecturalPillars&HighLevelOverview
• OpportunisticcomputingwithDODASinCMS– PublicandPrivateclouds
• Dataprocessing:Cachingmechanism• MachineLearningasaService:anewusecase• Conclusions
CHEP2018Sofia,Bulgaria 3
DynamicOnDemandAnalysisService:DODAS
PlatformasaServicetoolwhichaimistoguaranteedeploymentofcomplexandintricatesetupon“anycloudprovider”withalmostzeroeffort. Allowstoinstantiateon-demandcontainerbasedclustertoexecutesoftwareapplications
DODASisaThematicServiceunderEOSC-hubHorizon2020EUproject.
InitiallydevelopedasanINDIGO-DataCloudusecasesbasedonCMSworkflow.
DODASinanutshell
DanieleSpiga
Opportunisticcomputing:Temporarilyavailableforaspecifictask,userorgroup.
ExtensionofexistingFacilityManagepeaksofusage
BatchSystemasaServiceHTCondorbatchSystem
ExploitationofMachineLearningasaService
MulticloudclusterdeploymentHarvestdispersedresources(PaaSlevelfederation)
CHEP2018Sofia,Bulgaria 4
Thepillars
DanieleSpiga
ResourcesAbstractionTOSCAtodescribesoftwareapplicationsanddependenciesInfastructureManagerasconnectorwithunderlyingIaaSes
AutomationAnsibleforsoftwareandapplicationsetupMesos/MarathontomanageresourceandorchestrateCluestoautomatehorizontalscalability
Multi-cloudsupportINDIGO-PaaSOrchestratortodealwithmultipleheterogeneousCloudinfrastructures
Seetalk:
Beyon
dX.509:Tok
en-based
Authe
nticationan
dAu
thorizationforH
EP
Federatedauthentication
INDIGO-IdentityAccessManagementtomanageJWT,OpenIDConnect,SAML2.0,LDAP,Local(Username/Passwd);Identityharmonizationetc
CHEP2018Sofia,Bulgaria 5
Puttingeverythingtogether
DanieleSpiga
5
HomeIdP
AuthN
DODASUser
IAMPaaSOrchestrator
SubmitTOSCA
IM
PrivateCloud PublicCloud
Marathon
Provsioning,Setup,Configurat
ion Provsioning,Setup,Configuration
MasterMasterMasterSlave
Executor
Executor
ExecutorSoftware
ApplicationSoftwareApplication
Slave
Executor
Executor
Executor
MarathonMasterMasterMaster
Slave
Executor
Executor
Executor
Slave
Executor
Executor
Executor
DataAnalysts
SoftwareApplication SoftwareApplication
SoftwareApplication
SoftwareApplication
PaaS
IaaS
CHEP2018Sofia,Bulgaria 6
AndhowallofthisintegrateswithCMSHTCondorGlobalpool
DanieleSpiga
CMSPhysicists
SquidProxy
CVMFS
CVMFS
CVMFS
Auto-RegisterandGETjobs
Master
LoadBalancer
SlaveSlave
Slave
HTCondor
HTCondorHTCondor
HTCondor
HTCondor
CMSDistributedStorages
ü CompletelytransparenttoCMSphysicistsü Seamlesslyintegratingtheglobalinfrastructure
CertCache
DATAI/O
DODASephemeralsite
TokenTranslation
X.509
OpenIDConnect
SeeTalk:
ExploringGlid
einW
MSan
dHT
Cond
or
scalab
ilityfron
tiersfo
ranexpa
nding
CMSGloba
lPoo
l
CHEP2018Sofia,Bulgaria 7DanieleSpiga
DODASintheRealLife@CMS
ADODASgeneratedCMS-
Site
T3_IT_Opportunistic_hnsci DODASwithintop6CMSTier2sfor10days
-FrancescoBrivioINFN-ChiaraAmendolaLLR
CMSusersanalysis:hh—>2b2tauà SkimmingminAOD(Data&MC)to
produceflat-ntuple
CHEP2018Sofia,Bulgaria 8DanieleSpiga
-Elasticityandself-healing-Handling“specialrequirements”highmemoryjobs-Stabilityoverdays/weeks(120kjobs)
àWorkdoneinconjunctionwithThankstoAndreaChierici
ResourcesManagementwithDODAS
CHEP2018Sofia,Bulgaria 9
DataIngestion:CachingStrategy
DanieleSpiga
• DeploymentofacachelayerbetweenWorkerNodesandremoteGridStorageElements.
• CachelayerimplementedwithXRootDDataCache(XCache)• SeamlesslyintegratingtheCMSruntime
environment
• SupportinganOn-DemandCachingdeploymentmodel• TOSCA(plusAnsible) DODASgenerated
CMS-SiteCMSAAAXRootD
Federation
àDevelopmentdoneinthecontextof SeeTalk:Advancementsindatamanagementservicesfordistributede-infrastructures:theeXtreme-DataCloudproject
CourtesyofDiegoCiangottini
CHEP2018Sofia,Bulgaria 10
DODASonOpenStack@ICL:cloud.grid.hep.ph.ic.ac.uk- UseCases
§ ProducingCMSnanoAODformatforICSMPAnalyses§ ProducingGen-SimfromCMSgridpacks§ ProducingCMSgridpacks
MoreaboutDODASusage:PrivateCloud
DanieleSpiga
DODASadoptedtorunworkflowswithspecificMemoryrequests(upto32GB)
T3_U
K_Opp
ortunisitc_d
odas
CourtesyofRiccardoDiMaria
CHEP2018Sofia,Bulgaria 11
DODASasMLaaService
DanieleSpiga
EndUsers
SparkDispatcher
Master
LoadBalancer
SlaveSlave
Slave
SparkNode
SparkNodeSparkNode
SparkNode
SparkNode
SparkNode
DODASephemeralsite
Analysisof“DataCache”relatedmetadataflow- Toimprovecachinglayermanagement:SmartCache
1. Collectionofrawdata• Workloadmanagement• Datamanagement
2. DataenrichmentandreductionwithSparkjobs• StoringofoutputdatainHDFS
3. Analysisofstructureddata4. Prototypingofamodel
CourtesyofMaricaAntonacciandMircoTracolli
CHEP2018Sofia,Bulgaria 12
DODASseamlessintegratedintoexistingHTCondorGlobalPoolofCMS- SeveralusecasesbasedbothonHTCondorandBigData
SuccessfullytestedonmanyPublicandPrivateclouds
DODASisaThematicServiceundertheEOSC-hubProject,assuchitissupposedtobeadoptedalsooutsideCMS
§ AMSExperimentisalreadytesting/evaluatingDODAS
DODASexpertsareavailableforconsultancyandtraininginterestedcommunities.- [email protected] Additionalinfo:https://dodas.gitbook.io/dynamic-on-demand-analysis-service/
Conclusions
DanieleSpiga
