Affiliation Name E-mail UCAR/Unidata Mohan Ramamurthy ... · UCAR/Unidata Mohan Ramamurthy...
Transcript of Affiliation Name E-mail UCAR/Unidata Mohan Ramamurthy ... · UCAR/Unidata Mohan Ramamurthy...
1|P a g e
Affiliation Name E-mail
UCAR/Unidata MohanRamamurthy [email protected]
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
Datasystemsandservices,software/middlewareandtools;AlmostalldataandsoftwarefromUnidataaremadeavailablefreelyandopenlyanduseopensourcelicensing,sotheycanbereused.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
InadditiontoUnidata-developedsoftware,wealsoprovideexternallydevelopedsoftwaretoourusers.Suchtoolsareidentifiedbasedontheneedsoftheacademicusersanddeliberatedbyourgoverningcommittees.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
NetCDFisUnidata'smostwidelyusedsoftware.Thechallengeistoprovidesupporttoaverylargeanddiverseuserbaseinalmosteverycountryintheworldandallgeosciencedomainsand sectors. The Local DataManager and THREDDS Data Server applications also have adiverseusercommunity inbothoperationalandresearchsettings.Providingsupporttoaneverexpandingcommunityremainsanongoingchallenge.Anotherchallengestemsfromtherapid growth in the volume of data, so a push approachwill not not be sustainable. Theincreasingvolumeanddiversityofdata sources, coupledwith thegrowinguserbase,alsocreateschallengesinscalingandinteroperability.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Asstatedearlier,maintaininghighqualityofsupporttoagrowingandexpandinguserbaseinan era of shrinking or level budgets remains a challenge. There are also sociological andcultural challenges with changing technologies and adoption and use of new tools andservices. Migration to cloud platforms poses challenges in developing business and costrecoverymodels.
KeyRisks
2|P a g e
The lackofNSF-fundedoperational cloud facilities forhostingdataanddelivering servicesremains a key gap. Also, most CI facilities are operating independently without muchcollaborationandpartnership.Inadditiontosharingknowledgeandexpertise,adiscussiononhowthefacilitiescanshareotherresourcesandinfrastructurewouldbevaluable.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Unidata provides education and training, through workshops in Boulder and at differentuniversities,onaregularbasistostudentsandfacultyonitsproductsandservices.Inaddition,Unidatahostsseveralinternsandmentorsthemeverysummer.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
ExplodingdatavolumesandscalingofCI tomeet thegrowingneeds remainsa challenge.Cybersecurityisanotherchallengingarea.EntrainingandretainingprofessionalsintoscientificCIareasisachallengegiventhatgraduatingstudentsandprofessionalsarepaidmuchmorebytheITandsoftwareindustrythatisthriving.
Doyouhaveanyothersuggestionsfortheworkshop?
Clearly stated goals for theworkshop andmore in-depth discussions on important issues(ratherthanmanyoverviewpresentations)islikelytoleadtomeaningfuloutcomes.
3|P a g e
Affiliation Name E-mail
NEON TomGulbransen,Battelle [email protected]
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
3ingestionqueues,4transformationpipelines,2websites.Tailoredsounlikelytoreuse.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
6 external host partners for community distribution and limited data product creation.AeroNet,MG-Rast,SRA,BOLD,PhenoCam,AmeriFlux
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
Sensormessagingandcontrolchallengingatsitesinfrequentlyvisited.Ingestionqueueswhichcanaccommodatedozensofdatatypesandsources.APIswhichgreatlysimplypowerfuldataaccessandsharingoptions.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
ThefusionofclassicalITsystemsdevelopmentnowinntegralkyreliesoncodewrittenbynon-ITanalysts.Thevalueofthelatterwasunderestimatedinitially,andwillbeover-emphasizedgoingforwardduringcommunityengagement.
KeyRisks
Sensor unreliability is a risk addressed by engineering.User diversitywill create demandsbeyond the dev team capacity. Initial Ops period will reveal if/where/when/howcyberinfrastructuremayneedtoautomatemorechecksandeditsbility.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
4|P a g e
Lots of cyberinfrastructure recruitment and resultant learning curve climbing duringconstruction. Scientific cosers are being herded toward conventions to promote easierinteroperabilityandexpansionthroughexternalcontributionswhichcanbeevaluated.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Usercommunitytraceabilityandexpansionofuser'sdemands.
Doyouhaveanyothersuggestionsfortheworkshop?
Shareregistrantsinfo.
5|P a g e
Affiliation Name E-mail
Ocean ObservatoryInitiative(OOI)
Ivan Rodero, RutgersUniversity
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
TheinfrastructureoftheCIhasbeendevelopedin-housefollowingindustrybestpractices.Itincludes thedata lifecyclemanagement system,and thenetworkand systemarchitecturedistributed across two geographically distributed data centers. The customized softwarestack,includingcoredatamanagementsystemanduserinterfacehasbeenalsodeveloped.TheCIarchitectureandbestpracticesareavailabletoothertoreuse.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
TheOOICIusesanumberofexternalservicesandtools,includinganApacheserverforrawdata delivery, a THREEDS server for asynchronous data product delivery, Alfresco fordocument configurationmanagementand shipboarddatadelivery, andanumberof toolssuchRedmineandConfluencefordocumentationandconfigurationmanagement,gerritandJenkinsforcontinuousintegration,andphpBBforforums.Thesetoolswereselectedbasedonrequirementsandprioritizingopensourcesolutions,whenneeded.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1)On-demanddataproductdelivery:OOIprovidesuserswithagraphicaluserinterface(i.e.,OOINetdataportal)forplottinganddownloadingon-demanddataproducts.Theportalalsoprovidesaccesstolivevideoandotherdataproducts.2)Rawdataarchive:dataisavailablefordownloadin“raw”indicatesdataastheyarereceiveddirectlyfromtheinstrument,ininstrument-specificformat.3)Machine-to-machineAPI:aREFTfuluserinterfaceisavailabletoaccessOOICIprogrammaticallyusingauthenticationmechanisms.We’dliketosharethearchitectureoftheenterprise-levelinformationlifecyclemanagementsystem,includingnetworkingandmonitoringcomponentswhichuseindustrybestpractices.
6|P a g e
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
TwoofthemostimportantchallengesoftheOOICIare1)evolvingrequirements(e.g.,datarates, services), 2) and integration of new components (e.g., new instruments). There arelessonslearntrelatedtotheimplementationof industrybestpracticesforthedeploymentandoperationofaproduction-levelCI.
KeyRisks
OneofthehighestrisksfortheOOICIisrelatedtotheuncertaintiesforkeepingthefundinglevel for operating and maintaining the core infrastructure, the software stack andfundamentalservices.Forexample, the lackofexpandingthestorage infrastructure in thefutureisarisk.Amitigationstepwasincludingexpandabletape-basestorageinfrastructureintheinformationlifecyclemanagementsystem.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
CI-relatedworkforcedevelopmentisatdifferentlevels.Ontheonehand,technicalpersonnelare engaged with continuous training on the technologies involved in CI (e.g. Palo Altotraining,DellCompellent,ApacheCassandra,etc.).Ontheotherhand,OOIengagedwithNSF-fundedCTSCforthedevelopmentofacomprehensivecyber-securityplan.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
New CI requirements/challenges in the next 5-10 are related to the expansion of the CInetworkwithnewinstruments,increasingdataratesandevolvingdatadeliverymechanisms.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
7|P a g e
Affiliation Name E-mail
NationalNanotechnologyCoordinatedInfrastructure(NNCI)
Azad Naeemi, GeorgiaInstituteofTechnology
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
Institutedeveloped components include a self-service firewallmanagement, and a sharedaccess model where institute purchased equipment is provided to faculty who in returnprovidesharedaccesstotheirpurchasedhardware.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
WeareactivelyimplementingtheOpenScienceGrid,Globus,scienceDMZ,andperfSONARfile and networking components. In addition,we are implementing Ohio SupercomputingCenter’sPBSTools,OpenXDMoDfromtheUniversityatBuffalo.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1)Rapidlygrowingdatasources.Ourstoragesystemshavegrownexponentiallysince2009to8petabytes.2)Utilizationpatternsthataremanysmalljobs,i.e.highthroughputcomputing(HTC)vsthefewverylargemonolithicjobs(HPC).WeaimtofunnelthesetypesofworkloadstoOSG,andimplementhardwarededicatedtorunningOSGcomputation.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
KeyRisks
Notatthistime
8|P a g e
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Wehireundergraduatestudents,contributetoLinuxClusterInstituteworkshopsandareintheprocessofdeployinganinstructionalcluster.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
As a major technological research institution, the Georgia Institute of Technology, whichincludesacademicunitsandtheGeorgiaTechResearchInstitute(GTRI),hasdirectexperiencewithmanyofthecurrentandemergingresearchchallengesfacingtoday's
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
9|P a g e
Affiliation Name E-mail
NHERI Tim Cockerill, University ofTexas - Texas AdvancedComputingCenter
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
NearlyalloftheCIcomponentsaredevelopedin-housebyTACCandaremadeavailableasopensourceingithub.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
WeusetheDjangowebframeworkbasedonourpreviousexperienceswiththisandotherframeworks.Wealsohavea local implementationof theFedoraDigitalObjectRepositoryManagementSystemforourarchivingourpublisheddata.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
TheDataDepotisourmostusedCIcomponent.Ourusershavealreadyuploadedmorethan16TBofdatainadditiontothe40TBwetransitionedinfromthepredecessorprojectNEES.Weallowallfiletypesandweencourageouruserstouploadanyandalldatatheyneedtodotheirresearch-wefeelthatnotrestrictingtheusersiskeytotheiradoptionofourCI.WeworkedwithMathworkstoacquireaMATLABlicensethatenablesallacademicuserstoaccessMATLABviaourCI.TheengineeringcommunityareheavyMATLABusers,andthishasalsohelpedwithadoption.WeimplementedJupyterNotebooksandareprovidingtrainingonhowtousethemalongwithbasicPythonscriptingskills.WeareseeingprettystronguptakeofJupyter.Itrunsprettyfastinthecloud,andusersarefindingittobeascapableasMATLAB.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
10|P a g e
Challenge: operation of a tightly-coupled operation across hemispheresIt is preliminary to speak of lessons lesson learned, as LSST is in construction. However,accurateanddetailedmodeltoeffectivelycommunicate,coordinateandmaintaintheabilitytotraceCIfeaturestotherequirementsandbusinessneed.IsanareaoffocuswhichLSSTfeelswillhelpmeetthischallenge.
KeyRisks
Forthisproject,sincetheCIisallatTACC,thereisnotmuchrisk.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
WeprovideroughlymonthlytrainingwebinarswhicharerecordedandthenmadeavailablepersistentlyonYouTube.Wealsohavesummerprogramsforhighschoolstudents-thisyeartheybuiltaninstrumentedmodel,experimentedwiththatmodelonashaketable,andthenanalyzedtheirresultsusingourCI.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Performanceisthepriority,sincewebdatatransferandremoteuseofinteractivetoolslikeMATLAB are slower than on a local laptop. Also expanded simulation and dataanalysis/visualizationcapabilitiesonthewebportalsothatwecaptureallresearchersinthiscommunity.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
11|P a g e
Affiliation Name E-mail
LSST DonPetraivck,NCSA-UIUCJeffKantor,WilliamO'Mullane
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
R:LSSTisinconstruction,butthefollowingareunderway,LSSThasfundedthedevelopmentof a significant, high bandwidth network between Chile and the United States. LSST isdevelopingQSERV,aspatiallyshareddatabasewhichisanticipatedtorequire40PBofdiskprovisioning,over250nodeby2025.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
-LSSTUsesHT-CONDORforthebasisofitsproductionsystem.HT-Condorisastandardinthoughputcomputing,isusedinLHCandtheDarkEnergysurvey.HTCondorsupportsthevariousbatchusecasesidentifiedinLSST.LSSThashadacollaborativeengagementwithHTCondorformanyyears.LSSThasusedXSEDEandBlueWatersduringitspre-constructionphasefordemonstrationsoffeasibilityofitsproductionsystem,andhasusedsimulationdatageneratedontheOpenScienceGrid.–Theseweretheobviouschoicesduestoagencysupportandavailability.LSSThasbuiltuponauthenticationandauthorizationsystemworkthatisalsoinuseinLIGO.Thereasonisthatthesystemsupportsavarietyofauthenticationandauthorizationprotocol,andinteroperatedwithIncommon.NationaleducationandresearchidentityfederationsareseenasusefulsourceofidentityinformationforLSST,wheretheclassofallUSandallChileanprofessionalastronomershavedatarights.LSST’sMasterInformationSecurityPlanwasdevelopedinConsultationwiththeCTSC.CTSCwasselecteddueitisknowledgeofcontemporarysecuritystandards,asappliedtoNSFprojects.LSST’sscienceuserinterfaceisbasedontheFireflyToolKitdevelopedatIPACatCaltech.ThisisacommonlyusedadvancedtoolkitusedwithinOpticalAstronomy.Rucio,acomponentdevelopedatCERNfortheLHCisbeingevaluatedforinternalfilesynchronization,asisPegasusfortheproductionworkflows.Bothofthesecomponentswereselectedduetotheirusewithsimilarusecasesinotherexperiments.
12|P a g e
JupyterisafoundationalcomponenttosupportinternalqualityassessmentandtosupportexploitationofthedataattheUNandChileanLSSTDataAccessCenters.Jupyterisawell-supportedmethodofexposingaspectsofafacilityinastructuredwaytoalargegroupofusers.BROisuseforintrusiondetectionattheLSSTChileansites,andatNCSA.BROisselectedforusutilityinbeinganintrusiondetectionsystemwherelargevolumesofdataretransferredbetweensites,andsuetothebodyofexpertisewiththesystematNCSA
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
-LSSTUsesHT-CONDORforthebasisofitsproductionsystem.HT-Condorisastandardinthroughoutcomputing,isusedinLHCandtheDarkEnergysurvey.HTCondorsupportsthevariousbatchusecasesidentifiedinLSST.LSSThashadacollaborativeengagementwithHTCondorformanyyears.LSSThasusedXSEDEandBlueWatersduringitspre-constructionphasefordemonstrationsoffeasibilityofitsproductionsystem,andhasusedsimulationdatageneratedontheOpenScienceGrid.–Theseweretheobviouschoicesduestoagencysupportandavailability.LSSThasbuiltuponauthenticationandauthorizationsystemworkthatisalsoinuseinLIGO.Thereasonisthatthesystemsupportsavarietyofauthenticationandauthorizationprotocol,andinteroperatedwithIncommon.NationaleducationandresearchidentityfederationsareseenasusefulsourceofidentityinformationforLSST,wheretheclassofallUSandallChileanprofessionalastronomershavedatarights.LSST’sMasterInformationSecurityPlanwasdevelopedinConsultationwiththeCTSC.CTSCwasselecteddueitisknowledgeofcontemporarysecuritystandards,asappliedtoNSFprojects.LSST’sscienceuserinterfaceisbasedontheFireflyToolKitdevelopedatIPACatCaltech.ThisisacommonlyusedadvancedtoolkitusedwithinOpticalAstronomy.Rucio,acomponentdevelopedatCERNfortheLHCisbeingevaluatedforinternalfilesynchronization,asisPegasusfortheproductionworkflows.Bothofthesecomponentswereselectedduetotheirusewithsimilarusecasesinotherexperiments.JupyterisafoundationalcomponenttosupportinternalqualityassessmentandtosupportexploitationofthedataattheUNandChileanLSSTDataAccessCenters.Jupyterisawell-supportedmethodofexposingaspectsofafacilityinastructuredwaytoalargegroupofusers.BROisuseforintrusiondetectionattheLSSTChileansites,andatNCSA.BROisselectedforusutilityinbeinganintrusiondetectionsystemwherelargevolumesofdataretransferredbetweensites,andsuetothebodyofexpertisewiththesystematNCSA
13|P a g e
1)UpgradingthenorthsouthnetworkfromLaSerena,ChiletoNCSAinthecontextofaMREFCproject.2) Dealingwith the evolution of processors, in particular the reduction of the amount ofmemory per core, and the need to increase the level of threading in LSST Codes.3)Selectingthetechnologiesneededtosupportendusersinthedataaccesscenter.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Challenge:operationofatightly-coupledoperationacrosshemispheresItispreliminarytospeakoflessonslessonlearned,asLSSTisinconstruction.However,accurateanddetailedmodeltoeffectivelycommunicate,coordinateandmaintaintheabilitytotraceCIfeaturestotherequirementsandbusinessneed.IsanareaoffocuswhichLSSTfeelswillhelpmeetthischallenge.
KeyRisks
Changes incomputingplatformsovertheremainingperiodofconstructionandoperationsthrough2034areaconcern.LSSThasdataprocessingaccessandarchivefacilities inthreecontinents. Foreachcontinentthepaceofsustainablechangewillvary. Forexample,weexpectcloudcomputingtolaginSouthAmerica.Theresponsetothesechallengesincludesprovidingsoftwareisolationlayers,forexampleKubernetes,whichcanbedeployedinlocallyprovisioned or in commercial systems.Wecurrentlyusecouldservicesforsoftwarebuildandtest.TheEPOcomponentofLSSThasa very large clouddeployment component. Our baseline thinking allows for use of cloudservicesfordisasterrecovery,foropportunisticbulkcomputing,andforelasticexpansionoftheUSDataAccesscenters.Ourbaselinemayevolveasconstructionproceeds.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Projectstaffattendworkshopsandconferences.AtNCSAsignificantworkinCIisperformedbyNCSAstaff.NCSAhasaprogramofworktodeveloptheHPCworkforce,includingrespondingtoNSFcallsforproposalsfortrainingCyberInfrastructureProfessionals.Additionally,NCSAhasaprogramofresearchandsupportingitsinfrastructure,includingoperationalsecuritygroup,supportfortheLinuxClusterInstitute(LCI),whichtrainsInfrastructureprofessionals.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
14|P a g e
KeepingtheCIeffortsinChileandtheintheUScoordinatedandwithaliketechnologybase.ChangesinCItechnologiesandhowCIisabsorbedbytheproject.LSSThasobligationstoprovidecomputingfacilitiesinChile,whereforexamplecloudfunctionalityisnotequivalenttothefunctionalityavailableintheUS.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
15|P a g e
Affiliation Name E-mail
NationalOpticalAstronomyObservatory(NOAO)
SeanMcManus
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
data reduction pipeline (DEC Community Pipeline); TADA (Telescope Automatic DataArchiver);yesthesetoolsaremostlyopen-source
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
Scientific Linux, IBM General Parallel File System, Puppet, Foreman, Libvirt, Django. Thecriteriausedtoselecttoolsvaries.Forsomeopen-sourcetools,thereisminimalinvestmentneededtotrysomething,andthereforedoesn'trequireaformalselectionprocess.Forpaidsoftware contracts, there is obviouslymore vetting by internal IT staff,management, andprocurement.Aspartofnormalvettingwetrytolookatwhatisworking/notworkingforotherpeerorganizationsinsideandoutsideofAURA.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1) Mass storage: We require inexpensive storage on the multi-Petabyte scale to storeastronomydataproducts;2) Bandwidth: Reliable, fast bandwidth across continents is needed to move data fromtelescopetoarchive;3)Software:Thesoftwarestackmustmeetoperationalrequirementsbutalsobesustainableinsideflatorshrinkingbudgetenvelope.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Forsmalldepartments,itisdifficulttoachieveabalanceofexperienceversusmotivationandfamiliaritywithcuttingedgetools.Lowstaffturnovercanresultinstaffbeingsettledononeparticulartechnology,andlaggingbehindrecentdevelopmentsinIT.Ontheotherhand,it's
16|P a g e
notcost-effectivetoreacttothelatest/greatestthingthatcomesouteveryyear.Abalanceofnewversusproventoolsmustbemade.
KeyRisks
workforcereductionduetobudgets,evenasmallone,couldhavesignificantimpact.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Webudgetforcontinuingeducation,butwhetherornotstaffparticipateisvoluntary
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
transitionfromNOAO/LSST/GeminitoNCOA
Doyouhaveanyothersuggestionsfortheworkshop?
n/a
17|P a g e
Affiliation Name E-mail
LIGO StuartAnderson,Caltech [email protected]
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
Allofthefollowingin-houseCIcomponentsareavailableforreuse:*LIGODataReplicator(bulkdatatransfers)*MetadatadatabasesandtoolsdesignedforGWobservations*low-latencydatadistributiononlargeclusters*DataMonitoringTools*low-latencytransienteventalertsystem*NetworkDataServer*WebandMatlabbasedDataViewertools*GWDetectorstatusmonitoringservice*GWdetectionandparameterestimationpipelines*Libraryofgravitationalwavealgorithms*LIGOOpenScienceCenternotebooks*Jobaccountingsystem
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
*HTCondor/Pegasus/BOINC*OSG*Docker/Singularity/Shifter*CVMFS/StashCache/Xrootd/GridFTP*Shibboleth/Grouper/CILogon/Kerberos/LDAP/GSI*OracleHSM/ZFS/HDFS*GitHub/GitLab/Travis/Jenkins*JupyterHubThesetoolswherepredominantlyidentifiedbyfirstrecognizinganeedandthenchargingasmallgrouptoresearch(sometimesaself-forminggroup)toresearchwhatiscurrentlyavailable.Insomecasesthatgrouptakesasolutiontofullscaleprototype(builditandtheywillcome),andinothersthealternativesarepresentedtoaLIGOcomputingcommitteetoevaluatetheprosandconsfirst.andMatlabbasedDataViewertools*GWDetectorstatusmonitoringservice*GWdetectionandparameterestimationpipeline*Libraryofgravitationalwavealgorithms*LIGOOpenScienceCenternotebooks*Jobaccountingsystem
18|P a g e
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
*IdentityandAccessManagementwasachallengeduringtheearlyphasesofLIGO,leadingtosignificantlossinproductivityduetounnecessarybarrierstoefficientaccesstoneededinformationandsystems.IntegratingShibboleth,Grouper,InCommon,andCILogonintoLIGO'sCIhasbeenagamechanger.InvestinginI&AMearlyoninaprojectishighlyrecommended.*IntheearlyyearsofLIGOattemptstouseOSGtorunLIGOdataanalysistasksfailed.Inthelastfewyearsthishasbecomeamajorsuccess,inpartduetomorematuretoolsformanagingdataintensiveworkflows(e.g.,Pegasus,CVMFS,andcontainerization),andinpartduetomorematuregravitationalwavedataanalysispipelines.*LIGOinitiallyinvestedinahomegrownjobexecutionenvironmentthatattemptedtominimizetheamountofcodeneededtobedevelopedbyscientistsperformingsearchesforgravitationalwaves..However,thatprovedinpracticetobeinsufficientlyflexibleandthependulumswungovertoallowingscientiststodeveloparbitrarya.outexecutablesmanagedbyHTCondor.Inhindsite,theoptimumwouldhavebeensomewherein-between.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
*IntegratingCIwithinternationalcollaboratorsremainsasignificantchallenge..OSGhasrecentlyprovidedamajorbreakthroughforprovidingauniforminterfacetoplanandexecuteLIGOworkflowsoninternationalcomputingresources.However,internationalfederatedI&AMremainsasignificantchallengeforLIGO.*FindingtherightsetofCItosupportbothtightlycontrolledproductiondataanalysisandallowingcreativenewideasbedevelopedisachallenge.
KeyRisks
* Funding for CI experts that support scientific personnel to use existing CI*SustainabilityofCIandbeingabletoeffectivelyidentifynewCIthatwillbeavailableinthelong-termbeforeinvestinglimitedinternalresources.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
*Sendingstudentstosummerschoolsandsimilartrainingopportunities.*Sendingprofessionalstafftoconferencesandworkshops.*Invitingexternalexpertstoprovidetrainingatinternalscientificmeetings.
19|P a g e
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
*Inter-federationagreementsthatcomplywithinternationalprivacylawswhilestillreleasingenoughinformationtobeusefulforinternationalscientificcollaborations.*Trainingtheteachers.AsmostoftheworkforcecomesfromacademicresearchgroupshowdowetrainacademicfacultytobeabletotraintheirnewstudentstousemodernCI.*long-termstabilityofsoftwarepackaginganddistributionthatwillallowreproducibilityofscientificresultsonaninterestingtimescale.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
20|P a g e
Affiliation Name E-mail
LIGO AlbertLazzarini,Caltech
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
KeyRisks
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
PleaseseewhitepapersubmittedbyStuartAndersonforallattendeesfromLIGO
Doyouhaveanyothersuggestionsfortheworkshop?
21|P a g e
What is the appropriate scale and relationship among large NSF computing facilities,computingfacilitiesthatarepartofe.g.,physicslargefacilitiesandMRIresourcesprovidedtoindividualcollaborationinstitutions?DoesNSFhaveapolicyonthese?
22|P a g e
Affiliation Name E-mail
ARF JonC.Meyer,UCSanDiego
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
weareintheprocessofdevelopingdatadeliveryviamodernmessagequeueandwelcometheopportunitytocollaborateandhaveothersreuse.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
Some vendors' tools are used due the demand for certain types of data to be regularlyproducedduringaseagoingmission
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
Uninterrupted Internet connectivity. Research vessels at sea need consistent, reliablecommunicationpathstobeabletoproducescientificallyinterestingdatainneartorealtime.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
KeyRisks
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Somespecializedandgeneralcomputing-relatedtraining.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
23|P a g e
High-speed,realtimedeliveryofdatafromtheocean.Abilitytointeractwithfieldresearchersseamlesslyfrom
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
24|P a g e
Affiliation Name E-mail
Gemini Chris Morrison, GeminiObservatory
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
none(notethatwedonotincludesoftwareinourdefinitionofCI)
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
Googleappsforbusiness;Amazonwebservices;zoomconferencingservices.Identifiedinallcasesbyindustrysurveys&bestpractices;selectionviarequirementsanalysis,insomecasesusabilityanalyses,andvalueformoney.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
Challenges:1.Netappstorage.Largeimpactifthisredundantsystemfails.2.Backupstorageinfrastructure.Expensive,complexandrequiressignificantexpertise.3.Remoteaccessconnectivity.Bringsusermanagementandsecurityconcerns.Bestpractices:1.Geminiinfrastructurehassignificantredundancy,asaresultoflessonslearnedinpreviousfailures.2.Useofcloudservice(AWS)forlarge-scaledataarchivingandaccess.3.CIreplacementpolicyonequipmentatendofwarranty.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Challenges&gaps:seeabove.Lessonstoshare:Redundancy(storage,networking,VMclusters,connectivity).Lessonstolearninthemeeting:offsitestoragemethods&dataretention.
25|P a g e
KeyRisks
Dependencies:AccesstoGoogle(forbusinessapplications);AWS(forarchivestorage)-lowlikelihood,highimpactrisks.Mitigation:RedundantnetworklinksinHawaiiandChile.BackupplanforanextendedoutageofAWSwouldbetobringthearchiveinhousetemporarilyuntilservicerestored.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Enterprisespecialisttrainingcoursesandcertifications.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Challenge:IntegrationofGeminiCIintoalargerCenter,andaligningserviceswithotherProgramsinthatCenter.WedonotseesignificantchangesinthetechnicalchallengeforGeminiCI,asthetelescopeswillnotfundamentallychangethewaytheyoperateatnight.
Doyouhaveanyothersuggestionsfortheworkshop?
1.FutureroleofNSFincoordinatingorprovidingCIthroughgrantfunding.2.Large-scalesciencedatastorageandaccessviacloudservices-bestpractices.
26|P a g e
Affiliation Name E-mail
DKIST,NSO Steve Berukoff and EricCross,NSO
[email protected]@nso.edu
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
FortheDKISTtelescopeBuiltIn-House•InstrumentControlSystems•FacilityControlSystems•Telescope•Enclosure•Environmental•AdaptiveOptics,WavefrontControl•Coude•SafetySystems•AretheseusefultootherCIorganizations?Uncleariftheywouldbeusefulelsewhere.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
•OpenSourcesoftware;givenbudgetaryconstraintsDKISTCIisleveragingOpenSourcewhereapplicable.ThedeploymentofOpenSourceiscenteredwithintheInfrastructurelayers.•GlobusGridFTPwillbeustilizedtomovedatafromthetelescopeonMauitotheBoulderDataCenter.•CEPHobjectstorageforlong-termdatastorage
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
•ComplexityofDKISTInstrumentshasdrivenaflexiblebutcustomizableapproachtoinstrumentcontrols.•DatanetworkmanagementhasprovidedachallengetoDKIST.WehavenetworkInterconnectsbetweentheDKISTFacilityonMaui,theUniversityofHawaii,theUniversityofColorado,andalsoleveragingInternet2.•ComplexityofDKISTInstrumentshasdrivenaflexiblebutcustomizableapproachto
27|P a g e
instrumentcontrols.•DatanetworkmanagementhasprovidedachallengetoDKIST.WehavenetworkInterconnectsbetweentheDKISTFacilityonMaui,theUniversityofHawaii,theUniversityofColorado,andalsoleveragingInternet2.•ThecombinationofPetascaledatavolumeunderaveryconstrainedbudgetchallengestheabilityoftheCItosupportitscommunity.BestPractices•BecauseofthedistributednatureoftheprogramwithmultipleproductownersfollowingSystemsEngineeringpracticesfordevelopingeffectiverequirementsandinterfacecontrols.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
• Ensuring the end to end CI design from Facility Control, Data Acquisition and end-userdistributionisbuilt-intotheoveralldesignandbudget.
KeyRisks
•Operationalfundinglevelsshouldallowappropriatemaintenancetobecompletedwithappropriatepersonnel.•Long-Termoperationallifetimesmandateavoidanceofmonolithicarchitectures.Mitigation•AbilitytobuildinfrastructurebuildingblocksbydevelopingaroadmapforDIBBSawards.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
•Professionaldevelopmentconferences
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
•Ensurewecandeliverthescopethatweneedtosupportourcommunity.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
28|P a g e
Affiliation Name E-mail
ARF Suzanne Carbotte,ColumbiaUniversity
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
R2Rhasdevelopedanetwork file system for storageof data anddocuments; a relationaldatabaseforstorageofassociatedmetadata;aWebportalforsearch,browse,anddownload;scriptedtoolsfordatacataloging,archiving,processing,andassessment;andasuiteofWebservices for interoperability. Most are built on existing open-source software such asPostgreSQL,ApacheHTTP/Tomcat,MapServer,etc.SelectedtoolsfordataprocessinghavebeenreleasedinthepublicdomainviaGitHub.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
R2Ruses commercialprovisioning in selected cases forWeb servicehosting (Linode.com),domainservices(Site5.com),anddeepstorage(AmazonGlacier).
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1.R2R'snetworkfilesystemistheheartofitsdailyoperation,usedforbothinternalprocessingworkflowsandservingcontenttotheWeb.ThefilesystemisbuiltonasuiteofFibreChannelstoragearrays,switches,andLinuxservers.2.R2R's"NavManager"softwarepackageisusedroutinelytocreateasuiteofquality-controlledshiptracknavigationproducts,whicharereusedbydownstreamQAprocessesandWebservices.3.R2R's"LinkedData"serverdisseminatestheCruiseCataloginastandards-compliantformat,whichisharvestedbyothergeosciencedatarepositoriesaswellasbyglobalsearchindexessuchasGoogle.WhataspectsaboutthefacilityCIanditsoperationwouldyouliketoshareasbestpractices?Itisnotuncommontorevisitold(er)datapackages,inordertoextractadditionalinformationand/orrefinequalityassessment.Maintainingdatapackagesonspinningdiskfora5ormore-yearslidingwindowhasprovenadvantageous,andcanbesustainedusing(lessexpensive)HDDsratherthanSSDs.Everydigitalresourcepublishedonline(vessel,cruise,dataset,document,sample,person,
29|P a g e
award,etc)shouldhaveagloballyuniquepersistentidentifier.Thisenablesinteroperabilitywithotherrepositories,reliablecitation,andlinkingtothescientificliterature.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Thevolumeofenvironmentalsensordatabeingproducedbymodernresearchvessels,isincreasingfasterthanthediskstoragecapacitythatcanbedeployedwithaffordableenterprise-gradelocalequipment.Commercialprovisioningprovidesanaffordablesolutionfordeepstorage,butnotforlocaldataprocessingoregress.AcademicprovisioningviasystemslikeXSEDEisdifficultbecausetheresourcesaredisjointedandconstantlyevolving,andcarrytheriskofabruptterminationwhenthegrantperiodends.Datatransferisalsohamperedbylocalcampusnetworkbandwidth.Whileprogresshasbeenmadetowardstandardization,theUS.academicfleetstillproducesdatainaveryheterogeneousmanner.Eachcruiseisunique.Significantmanpowerisstillrequiredtostayabreastofchangingdirectorystructuresandfileformats,andtorecoverfromoperatorerrors.
KeyRisks
Maintaininglocalserver,storage,andnetworkinfrastructureremainsanongoingchallenge,especially with the increased need to providemonitoring, metrics, and network security.Commercial provisioning shifts resources from a local to a remote location, but does noteliminatetheneedforasystemadministratoranddoesnotreducecosts.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
R2RstaffattendannualcommunitymeetingssuchasESIP,RDA,andRVTEC,tostayabreastofemergingtechnologies.Juniorstaffworkintandemwithseniorstaff,receivingon-the-jobtraining.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Theabilitytostoreandmovelargevolumesofdataasenvironmentalsensorscontinuetoevolvefasterthanstorage/networkresources;thelackof"smart"self-documentingsensors;andthelackofdesignatedlong-termarchivesforsomedatatypesremainsignificantchallenges.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
30|P a g e
Affiliation Name E-mail
NationalCenterforAtmosphericResearch(NCAR)
AaronAndersen,UCAR
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
AnumberofcomponentsoftheCIweredevelopedinhouse.Afewconcreteexamplesinclude:-ResearchDataArchiveservices-publicinterfacecanbefoundat:https://rda.ucar.edu/-ParallelPythontoolsforpostproductionofNetCDFfilesandspecificallyclimatedata:https://www2.cisl.ucar.edu/tdd/asap/parallel-python-tools-post-processing-climate-data-SystemAccountingManager(SAM)onHPCsystemshttps://www2.cisl.ucar.edu/user-support/systems-accounting-manager(currentlyNCARspecific)-VAPORistheVisualizationandAnalysisPlatformforOcean,Atmosphere,andSolarResearchers.VAPORprovidesaninteractive3Dvisualizationenvironmentthatcanalsoproduceanimationsandstillframeimageshttps.://www.vapor.ucar.edu/-NCARCommandLanguage-NCLisaninterpretedlanguagedesignedspecificallyforscientificdataanalysisandvisualization.AlltoolswereprimarilydevelopedwiththeneedsoftheAtmosphericsciencecommunityinmind.AllcomponentsareavailableforreuseexceptforSAM.SAMcouldbecustomizedandutilizedbyothersbutwouldrequiresomegeneralizationorsitespecificcustomization.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
AgoodnumberofexternalCIcapabilitiesand/orexternallydevelopedtoolsareinuseatNCARwithintheComputingandInformationSystemsLab(CISL)..Highlightsinclude:-NCARDataSharingService-GlobusToolkit-https://www.globus.org/-NCARalsoutilizesXDMoDaspartofthesuiteoftoolsusedtomanagetheHPCresources-http://open.xdmod.org/WithintheNCARWyomingsupercomputingcentertwocommercialpackagesareinusetocontrol,manageandmonitorthefacility.-ThecoreofthefacilityutilizesBuildingAutomation,hardware,softwareandsensorsfromJohnsonControlsInc.basedontheMetasysBuildingAutomationSystemhttp://www.johnsoncontrols.com/buildings/building-management/building-automation-systems-bas-MorerecentlyNCARhasdeployedanadvancedsystemtoallowhigherfidelitysamplingof
31|P a g e
theelectricalinfrastructure.ThosecomponentswereprovidedbySchneiderElectricSoftwareLLC.undertheirWonderwarebrand.ThesetwocommercialpackageswerepurchasedutilizingaformalRFPprocessandwereevaluatedbyatechnicalteam,businessteamandpricingteam.Technicalrequirementsweredevelopedinpartnershipwithexternalengineeringfirms.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
ThethreemostusedCIcomponentsaretheHighPerformanceComputingsystems,HighPerformanceDiskStorage(GLADE)andthetapearchiveHPSS.TheHPCsystemsareregularlyseegreaterthan90%systemutilization.GLADEsimilarlyhasbeenexceptionallypopularprovidingcommonsharedspaceacrossHPC,dataanalysisandvisualizationplatforms.FinallytheHPSSbasedarchivesystemisstillthecornerstoneofdataarchivalatNCARandinsomerespectsistoopopular:-HPCsystemsutilizetestanddevelopmenthardwarethatismuchsmallerscalebutprovidescapabilitiestonotimpactproductionworkwhileupgrading,patchingoraddingnewtoolstotheuserenvironment.OncechangestothetestenvironmentsarestabletheteamscanthenupgradeorchangethelargeHPCenvironments.Herecomplexityandscaleprovidesignificantchallenges.-TheGLADEenvironmentistechnicallychallengingprovidingaverylarge(50PB)highperformanceInfiniBandstorageenvironment.Howeverthetechnicalchallengesareonlyonecomponentoftheenvironment,userretentionpoliciesandmanagementofquotasareequallyaschallenging.-HPSSpresentsamorefinancialchallenge.Historicalarchivalstoragepolicieswerepredicatedoncomputingbeingexpensivebutstoragebeingcheap.CurrentlythoseeconomicassumptionsarenolongervalidandCISLhasembarkedonmodificationstostoragepolicies.Thateffortistoonewbutmaybecomeabestpractice.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Weseehumancapitalaspossiblyoneofourmostchallengingareascurrently.ExpertiseinHPC,largedatastorageandITenvironmentsareinhighdemand.Weoftenfindrecruitingstaffachallengeespeciallywheresomeareaslikedataanalyticsanddatascienceareinsignificantdemandinthecommercialaswellasresearchsectors.Keepingpacewithsalariesinachallengingfederalenvironmentisprovingdifficult.ClosertothefacilityoperationlevelweareseeinghighlydynamicHPCenergyconsumptionbasedoncomputingworkloads.AllHPCvendorsareactivelypursuingpowersavingcapabilitiesallthewaydowntothechiplevel,turningdownclocksorcomponentsondemand.Overallthisisagoodthingascomputingsystemsofthepastwerenotoriously
32|P a g e
wasteful.However,computingcomponentsthatturnupanddownoncomputingtimescales(subseconds)maynotbeamatchfortraditionalbuildingautomationsystemsormorebroadlyutilityproviders.Largechangesinelectricaldemandinfluencemechanicalcoolingsystemsaswellasthecapacityoftheutility.TheNWSChasahighlyenergyefficientdesignthatadaptstothedemandsoftheCIhousedinthefacility.
KeyRisks
Workforcedevelopment,recruitingandretentionareasignificantrisk.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
NCARhasanumberofeffortsunderwayasweseeworkforcedevelopmentascritical.TheNWSChasbeenutilizedasateachinglaboratorywith7summerinternsoverthelast5yearsworkingwithinthefacility.Withinthattimeframe,3womenand2minoritystudentshavebeenthroughthree-monthintensivesummerinternships.AllbuttwoofthosestudentshaveremainedinfieldsengagedwithlargeCI.CISlalsomanagestheSummerInternshipsinParallelComputationalScience(SIParCS).ThegoaloftheSIParCSprogramistomakealong-term,positiveimpactonthequalityanddiversityoftheworkforceneededtouseandoperate21stcenturysupercomputers.Graduatestudentsandundergraduatestudents(whohavecompletedtheirsophomoreyearbysummer2017)gainsignificanthands-onexperienceinhigh-performancecomputingandrelatedfieldsthatuseHPCforscientificdiscoveryandmodeling.MorerecentlytheOperationsManagerattheNWSChasbeenengagedaspartofthestateofWyomingWorkforceDevelopmentCouncil.Wyominginparticularislookingtodevelopgreaterinroadsspecifictolargecomputingfacilitieswithmoretraditionaltrades,communitycollegesandnon-traditionalstudents.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
SpecifictomodelingandsimulationweseeahighlydisruptiveCIenvironmentwithsignificantcomputing architecture diversity on the horizon and new clear winners. Heterogeneouscomputing architectures are now commonplace but the complexity and scale remainchallenging.Thereisalsoanexplosionofdataanddataresourcesthathaslongbeenpromisedbutwearestartingtoseewithgreaterclarity.Newmethodssuchasmachinelearningoffersomepromisebuttherearemanypathsandoptions.NCARcertainlydoesn'thavethecapabilitytoexploreallpossiblepathsandwillneedtopartneracrossmanydisciplinestofindanswers.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
33|P a g e
Affiliation Name E-mail
IncorporatedResearchInstitutionsforSeismology(IRIS)
Tim Ahern, University ofWashington
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
Mostcomponentshavebeendevelopedinhouseoverthe30yearslifeoftheDMC.Ofcoursecommercial andopen source software systems are usedwhen appropriate such asDBMSsoftware.Muchofourinfrastructureissomewhatdomainspecificsuchasreceptionofrealtimedataandtoolsthatworkwithdomainspecificdata.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
We use commercial software for virtualization (VmWare), PostgreSql for DBMS software,commercial geolocation software. All external tools were acquired using IRIS purchasingguidelines,multiplebidsetc.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1)Webservices,methodstoabstract timeseriesandmetadataaccessboth internallyandexternally2)storageRAIDindexingschemetoimproveaccesstocommodityRAID3)Synchronizationofdataversionsacrossmultiplestoragesystems(1primaryand1secondaryateachoftheDMCandtheADC)
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Scalability.Access toseismologicaldatacanbeepisodicespeciallyafterearthquakes. Alsocertain preprocessing services can exceed our internal capabilities. The promise of cloudresourceshaspotentialbutnotyetrealized.
KeyRisks
34|P a g e
Lossofkeypersonnelandtheirknowledge.NSFbudgetsaremakingfacilitieslikeourmoreandmorevulnerable.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
BothNSFandcommerciallysponsoredtrainingcourses.Weparticipateastimeandfinancialresourcesallow
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Reducingthecosttomaintainourinfrastructureandfindingexternalresourcesperhapscloud,thatcanmeetourdemandsandfitourwayofdoingbusinessnottheirs.
Doyouhaveanyothersuggestionsfortheworkshop?
Nothingatthistime,notabletospendmuchtimeonthis.....
35|P a g e
Affiliation Name E-mail
UNAVCO FranBoler,UNAVCO [email protected]
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
EssentiallyallcomponentsofUNAVCO’sCIhavebeendevelopedinhouse.ThisincludesdatahandlingfordataarrivingatUNAVCOfrommultiplevarietiesfieldinstrumentationandfromavarietyofproviders,archiving,anddistributionfunctions.MostoftheCIthataidsindatahandling is not available for reuse since it is highly customized.An exception is theGNSSpreprocessing software tool called “teqc”, which is widely shared with the community.SelectedCIcomponentshavebeendevelopedinpartnershipwithotherinstitutionsandaresharedwiththemincludingSARwebservicesdevelopedviatheNASASSARAprojectissharedwith the Alaska Satellite Facility; and the Geodesy Seamless Archive Centers open sourcesoftware was developed with NASA ACCESS support by UNAVCO with UCSD and NASA’sCrustalDynamicsDataInformationSystems.GSACiswidelyshared.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
CertainproprietarysoftwareprovidedbysensormanufacturersforhandlingrawdataarepartofUNAVCO’sCI.Theseareprescribedwhenamanufacturerisselectedasasensorprovider.MuchofUNAVCO’sSARdatahandlinginfrastructureiscurrentlybeingmigratedtotheXSEDEcloud.Commercialcloudstorageisemployedasoneofourbackupstrategies.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
Thedatasystemsthatweoperate(softwareandhardware)thatreceive,handleanddeliverGNSSdatatoourexternalcustomerbasehavethelargestuserbaseandareused24/7.Wehavebeen“saved”manytimesoverbyhavingfailoversystemsatthereadyfortheinevitablehiccupsinsystems.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Agapislackofadequateresourcestokeepsoftwareandtoalesserextenthardwareuptodate.Functionalityisregularlyaddedthroughtimeasnewcomponentsoftwaresystems,andthis functionality is developed with technologies reflecting the era during which it was
36|P a g e
developed,withsomeattempttoseeintothefuture;thesecomponentstendtoremainpartof operational infrastructure (we call them legacy components, but they are still key toaccomplishing our tasks). All along the way technical debt is incurred, and of coursetechnologymovesahead.Thisisafurtherchallengetomovingcapabilitiestothecloud.Wearetryingtoslowlyandonatrialbasismovecomponentstothecloud.Legacycomponentsareafurtherriskas itbecomesincreasinglydifficulttofindprogrammerswithappropriateskillsetstomaintainthem.Thepriorityisalmostnevertorebuildtheseoldersystemsaslongastheycontinuetooperate.AnotherchallengeisthewidevarietyoftechnologiesinuseintheEarthSciencestomeetCIneedsofvariousdomains.Tryingtocoverallbases isnearlyimpossible;tryingtoidentifywhichtechnologieswillemergeasmostusefulisachallengeforall.TheEarthCubeinitiativeisclearlyexposing/highlightingthis.
KeyRisks
Keyrisksarerelatedtothetechnicaldebtdescribedinaprevioussection.Anotherkeyriskislooming retirement of staff members with decades of domain knowledge and in-depthknowledgeofourCIcomponents.Further,thereisstrongcompetitioninourgeographicareaforskilledCIworkers.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Wesendstaffmemberstotraining.Weengageinterns.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Makinguseof thecloud (withappropriate returnon investment).Continuing to trackandidentify trends in technologies and being able to respond nimbly.Managing functionalitydemandsunderresourceconstraints.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
37|P a g e
Affiliation Name E-mail
IceCube Gonzalo Merino,University of WisconsinMadison
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
1)Datamanagementsoftware,handlingdataarchive,transferfromthesouthpoleandreplicationtolongtermarchives.2)Softwareframeworktomanagedistributedworkloads.UsedtomanageandbookkeepalltheIceCubesimulationproduction.Inbothcases,otherscoulduse,butthisdoesnothappenyet.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
1)SouthPolebroadbandsatellitesSPTR,DSCSandSkynet.ProvidedbyNASA,throughUSAP.ThisistheonlyavailableservicefordailybulkdatatransferfromtheSouthPole.~100Gbytes/day.2)Tapestorageforlongtermdataarchive.ProvidedbycollaboratinginstitutionsNERSCandDESY-Zeuthen.Theseinstitutionsalreadyoperatelargescaleautomatedtapefacilitiesforseveralexperiments.Theserviceisofferedasin-kindcontributiontotheCollaboration.3)OpenScienceGrid.ProvidingaccesstomillionsofCPUhoursinopportunisticresources.Also,operatingcoreGridservicesthatprovideusaccesstoIceCubecollaboratingsitesinEuropeandCanada.WehavebeenparticipatinginOSGforseveralyears.Distributedcomputing,andinparticularopportunisticcomputing,representsabigadvantageinourfieldwherealotofthedataprocessingandanalysisispleasantlyparallel.4)XSEDE.PartoftheIceCubesimulationchainreliesonGPUs.WestartedrequestingallocationsinGPU-capableXSEDEresourcesin2016toenlargethecomputingcapacityavailableforIceCubeandincreasetheanalysispotential.5)Globusdatatransferservice(globus.org).Convenientdatatransferserviceusedtoschedule/steerdatatransfersfromUW-Madisontoarchivelocations:NERSCandDESY-Zeuthen.Selectedbecauseitprovidedtheneededfunctionality(integrity,retries,etc)currentlyatnocost.Also,interestedinongoingdevelopmentstointerfacemoreefficientlytheHPSStapesystematNERSCwithGlobus(fileintegrity,performance).
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
38|P a g e
1)MaindataprocessingclusteratUW-Madison.LargeCPUandGPUclustercoupledtoamulti-petabytefilesystem(Lustre)usedby~300researcherstoanalyzetheIceCubedata.Themostchallengingparttooperateisthestorage,includingmonitoring,accounting,etc.However,operatingourownLustreclusterseemstostillbethemostcosteffectivesolutionforoursize(~6Petabytesofdisk).2)User-friendlyscalable/elasticcomputinginfrastructure:OSGandHTCondorhaveprovidedgreatcapabilitiessofarinthisfront.However,westillseealotofroomforimprovementintheuserexperience:higherefficiency,easeofuse,interfacetocloudresources,etc.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Everytimewehavebeenabletoleverageexisting3rdpartyservicestobuildourinfrastructurearoundthem,wehaveseenbenefits indoingthat.Fromlargearchivestoragefacilities, todatatransferservices,toworkloadmanagementservices,ourlessonlearntisthatitseemsworthforustoinvestonhavingasolidinterfacewithexistingservicesratherthantryingtoreplicatethem,orreinventthewheel.
KeyRisks
Withtheuseofexternalservices,therecomesdependenciesandrisk.Mitigationstrategiesarethereforeanimportanttopic.Inourcase,severaloftheseexternalservicesarecomingfrom the academic ecosystem, so some coordination inside or between agencies couldaddresspartoftherisk.Partofitwouldbeensuringthatthosecommonservicesthatmanyresearchersdependon,aresustainable.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Assistingtovariousworkshopsandconferences inthefield:NSFcyberinfrastructure,OpenScienceGrid,NationalDataService...
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Understanding how to best adapt IceCube analysis code to new emerging computingarchitecturesandsoftwareframeworkssuchasmanycore,GPU,FPGA,machinelearninganddataanalyticsframeworks,etcandengagetheworkforcewiththerequiredskillsthatweneedtomakethishappen.Hiringandretainingthispersonnelisgettingincreasinglydifficultaswecompetehead-onwiththeITprivateindustry.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
39|P a g e
Affiliation Name E-mail
NSCL Andreas Stolz, MichiganStateUniversity
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
Dataacquisitionandanalysissoftwareframework(NSCLDAQ/SpecTcl/DDAS),availabletoothers.Controlssoftware(EPICS)development,availabletoothers.Businessprocesssoftware;customandcustomizedapplications.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
Dataacquisition(DAQ)andexperimentaldataanalysisonLinuxbasedinfrastructure.CommodityPCs/Servers.StorageusingcommodityhardwareandZFS/Linux.Thisiswidelyused,freelyavailablesoftwareandlowcost.DAQisdevelopedin-house.Analysisapplicationsaretypicalfreelyavailablephysicsapplications(GEANT,ROOT,etc.)Businessprocess:ERP(IFSsoftware),Sharepointworkflowsanddocumentmanagement.Engineeringsoftware?Solidworksetc.Networking/Internet–externalaccessprovidedbyMSU
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
Infrastructure–virtualization:Normalforenterpriseinfrastructure,butdoesrequireexpertiseforsupport.Sharepoint:Usedforbusinessprocesses,collaborationetc.Againrequiringdeveloperandadministratorexpertise.Security:Networkandsystemssecurityincludingtechnicalcontrolsthemselvesandtheworkloadaroundmaintaininganddocumentingsame.Adoptingconfigurationmanagementtoolsandtestingdeploymentprocesses.Systemconfiguration–maintainingstableoperationsalongwithongoingsoftwarechangesandsecurityupdates.
40|P a g e
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Securityisongoingchallenge.
KeyRisks
Mainrisksaresimilartoanyenterprise:securityanddisasterrecovery.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Participatinginrelevantworkshops.CISecuritytrainingforallusers.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Providingincreaseddataaccesstooutsidevisitorsandexperimentersinfaceofincreasingdatasetsizesandsecurityrestrictions.FutureDAQsystemsforFRIBexperiments.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
41|P a g e
Affiliation Name E-mail
InternationalOceanDiscoveryProgram(IODP)
Jim Rosser, Texas A&MUniversity
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
SeveralCIcomponentsaredevelopedandmaintainedin-house:instrumenthostdatauploaders,webservices,webscienceapplications,databases,businessapplications(procurement,inventory,crewtracking).Yes,theseareavailabletoothersforreuse,but,inmostcases,wouldrequireextensiveeffort.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
OurapproachistofocusonJRSOcorecompetenciesandleveragecommodityservicesfromotherorganizationswhenpossible.Forexample,TexasA&MUniversityprovidesmanysharedservicesthatweusetosupportJRSOoperations,includingemail;directoryservices;storageservices;webconferencing;videostreaming;softwaretraining;cloudstorage;financial,travelandHRmanagementsystems;cybersecurityassessmenttools;softwareprocurement;projectmanagementassistance,etc.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1.WAN(includingVSAT)operationsandsupport.SustaininghighlyavailableWANservicesisquitechallengingwhentheresearchvessel(JR)operatesglobally.2.OracleODAs.OracleODAssignificantlyincreasedJRSOdatabaseengineperformance.However,therehasbeenasteeplearningcurveforconfiguringandmaintainingthiscapability.3.Cybersecurity.MinimizingsecurityriskwhilesupportinginternationalcustomerswhobringmanydifferentpersonaldevicesonboardtheJRandexpectassuredaccesstotheship'sportfolioofsciencelabservices(e.g.,LAN,serverstorage,applicationanddatabaseservices).
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
42|P a g e
MinimizingsecurityriskwhilesupportinginternationalcustomerswhobringmanydifferentpersonaldevicesonboardtheJRandexpectassuredaccesstotheship'sportfolioofsciencelabservices(e.g.,LAN,serverstorage,applicationanddatabaseservices).
KeyRisks
Commerciallyavailabletoolsareincreasinglycloud-based(e.g.,AdobeCreativeSuite,macOSapps,etc.).OurmeagercommunicationbandwidthsupportingtheJRrulesthoseout.Yet,manysoftwarepublishersprovidenoalternative.Thisissueisprobablyuniquetofacilitiesoperatinginlowbandwidth,highlatencyenvironments,andprobablyalsoappliestoorganizations,suchasDoD,thatoperateisolatednetworks(SIPRNet,JWICS,etc).Thisisagrowingproblemthatcontinuestochallengeus.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Technologyspecifictrainingforallaspectsofinfrastructure,softwaredevelopmentanddatamanagement.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
BetterWANlinkfortheJR.Adoptionofautomation/configurationmanagementtools,suchasChef,Ansible,Salt,etc.Makingdatamorediscoverable.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
43|P a g e
Affiliation Name E-mail
CHESS Werner Sun, CornellUniversity
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
Ourhigh-availabilityclustersandComputeFarmweredevelopedusingcommodityhardwareandopen-sourcesoftware,assembledandconfiguredin-housetomeettherequirementsofourfacility.Theseconfigurationscouldbesharedwithotherfacilities.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
WeprovideCHESSuserswithremotedatadownloadcapabilitiesusingGlobus.WeselectedthistoolforitsexcellentperformanceandbecauseofitswidespreadadoptionintheNSFLargeFacilitycommunity.
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
High-availabilityLinuxserverclustersformthebackboneofourCI.Weusethemforourcentralfilesystems,coreinfrastructureservices,webanddatabaseservers,andhardwarecontrolsystems.Incommissioningtheseclusters,wegainedexperiencewithselectingfreeandopen-sourcesoftwareandcommodityhardwaresolutionswithoutsacrificingreliabilityandperformance.TheCHESSdataacquisitionsystemisacentralrepositorythatreceivesrawdatafrommultipleinputstreamsandprovidesaccessforofflineanalysisandprocessing.Wedevelopedbackup,archive,androtationprocedurestoensurediskaccesstotworun-cycles'worthofdataandtaperetrievalforallpreviousdata.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
44|P a g e
Wewouldbeinterestedinlearningaboutmethodsforprovisioningtemporaryaccountsandimplementingfine-grainedauthorizationforCHESSusers.
KeyRisks
Wefacean increasinglychallengingcybersecuritythreat landscape.Wearealwaysseekingwaystobalancesecuringourfacilitycontrolsystemswhilemaintainingusability,access,andproductivity.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Onlinetutorials,managerialandtechnicaltrainings.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
UpgradestothescientificcapabilitiesoftheCHESSfacilitywillresultinincreaseddatathroughputandvolumes,whichwilleventuallyexhaustasinglesystem'sabilitytobothserveasthedatastoreandtheaccesspoint.Wemayneedmultipleingressandseparateanalysissystems.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
45|P a g e
Affiliation Name E-mail
PSC/CMU
JamesA.Marsteller
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
KeyRisks
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Doyouhaveanyothersuggestionsfortheworkshop?
46|P a g e
47|P a g e
Affiliation Name E-mail
NationalRadioAstronomyObservatory(NRAO)
BrianGlendenning,NRAO
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
100%(basedonopensourcesoftware),yes
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
AmazonAWS(modest),NSFXSEDE(experimental);Convenience/capability(AWS),cost(XSEDE)
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1.TheCASAdatareductionpackageisalarge(2MSLOC)packagebothusedforinternaloperationsuseanddownloadedbyfacilityusers(2kdownloadsperyear).2.Our"pipelines"embedexpertknowledgeinapythonscriptingframeworkforautomatedscienceproduction.3.Ourcomputinginfrastructurehasmultiple"archive"storageclusters,withattachedLustreandcomputationalclustersfordataprocessing.Wehavetotakethelongview-wehaveusabledatafrom40yearsago,oursoftwarepackageslivefordecades.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Keepingsoftwarepackagesreasonablyhigh-performanceoverdecadesisanissueforus.
KeyRisks
48|P a g e
DurableagreementswithHPCfacilities,IaaSresearchclouds,Internationalcompatibilitywithuserauthenticationmechanismsetc.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Ph.D.student/Post-docengagementwithwritingresearchcodes.Summer/co-opstudents.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
Seefinalbulletpointsinwhitepaper.
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
49|P a g e
Affiliation Name E-mail
Ocean Networks Canada
Benoit Pirenne
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
The Oceans 2.0 was entirely developed in house, starting in 2005. The code is not in the public domain owing to the decision made by ONC to pursue commercial applications of the system.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
External tools include standard tools such as OS (Linux), Java, Javascript and attendant libraries; Oracle as an RDMS, Cassandra for non-relational data... ERDDAP was integrated to provide standard access to specific data types. Jira for supporting all aspect of the development, including time sheets and billing on a per project basis Confluence for internal and external documentation
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
Until recently, the challenging elements included: - Cassandra: performance issues with the tool and the complexity of the fine-tuning required , Java memory allocation issues, difficulty with profiling complex code to understand where memory and time are actually spent, despite having an advanced test environment
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
Continuously evolving the technology and the services available and getting the continued funding for the required manpower. Providing easy to use data discovery interfaces that will be addressing user needs in the face of growing instrumentation, observing locations and expanding time
KeyRisks
50|P a g e
Risksinclude-maintainingtheleveloffundingtoenablecontinuousimprovementstothefacility:aCIisneverover!Mitigationrequiresmakingmanagementandfundingagenciesunderstandthat.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
We have had large fractions of the team of 20+ software engineers attend classes in: - the Agile Scrum methodology - usability - Kaisen
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
- As the facility continues to grow, a continuous emphasis on verification of our scalability, and possible adaptation will be necessary. - The support of multiple clients, re-organizing into a multi-project based entity - Need to support critical customers (e..g, Public Safety) with defined SLAs
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
51|P a g e
Affiliation Name E-mail
Oregon State University, College of Earth, Ocean,
and Atmospheric Sciences, Regional Class Research
Vessel Program
Christopher Romsos
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
The most significant CI component built in-house is our "datapresence" system. In a nutshell, the datapresence system captures and archives data from resident (or visiting) sensors, replicates the information shoreside, and presents the information to both the shipboard and shoreside science parties for use/consumption. The datapresence system includes functionality for data quality assessment, flagging, alert and user notification. Other CI components developed in-house include several databases for project management including a risk-register database application. Yes, these components are available for others to use.
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
There is a high likelihood that the most if not all RCRVs shall be provisioned with satellite service through HiSeasNet at UCSD (https://hiseasnet.ucsd.edu/), though some UNOLS ships are experimenting with going out and negotiating their own contracts for satellite service opting (out of the HighSeasNet program in areas where better deals can be struck such as the Gulf of Mexico). We, the RCRV datapresence developers, are currently formalizing an MOU with Leidos Antarctic Support contractors to share components of our acquisition and visualization code. Part of this process includes choosing an open source license under which to distribute software. Lastly, we've incorporated data and map services (hosted locally aboard the ship) from the Marine Geoscience Datasystem at Lamont-Doherty Earth Observatory (LDEO) into our real-time displays for scientific situational awareness. Specifically, the Global Multi-Resolution Topography Data Synthesis provides our base layer for the map interface http://www.marine-geo.org/portals/gmrt/ Other sources of thematic background information for this interface are provided by NOAA Fisheries, Office of Coast Survey, USGS, and various academic sources.
52|P a g e
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
1) Ship to shore (and back) data replication over high latency, low bandwitdh satellite networks. This problem, akin to the Long Fat Network problem of high bandwidth-delay product, is the most challenging issue that we are working on. We've had good success in increasing our throughput by optimizing the TCP window and buffer sizes and are now looking at managed WAN optimizatoin solutions to provide this service. 2) Cybersecurity is another challenge for the project. The RCRVs shall be equipped with integrated monitoring control systems to cover everything from bridge to engine room systems. Securing these online systems is a priority and a challenge.
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
At this project phase (construction) we don't yet have lessons learned to share.
KeyRisks
Key risks include security and expertise. As indicated the RCRVs shall present a significant CI advancement from current. To mitigate each of these risks we have an operations plan that includes support and oversight (budget and personnel) from a Class Management Office. However, the level of expertise for the technical support personnel (Marine Technicians) that sail with the ships will have to rise. Evidence to support this expertise risk can be gleaned from organizations that have recently taken operations responsibility for new research vessels.
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
Ah, a perfect follow-up question. A key component of our operations plan during transition to operations and post-delivery under Class Management will be technology transfer and training for new operators. We expect much of this initial ' workforce development' to take the form of hands on work during transition but additional training will be made possible through the Class Management Office during operations. In addition to periodic training we have staff that shall travel to each vessel on a rotating schedule (multiple visits per year) to inspect sensor systems, perform calibrations and maintenance, as well as conduct specific training while on a site visit.
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
BYOD IoT sensors - We must keep abreast of security and integration issues these devices present. On-Prem IaaS and PaaS - These industry trends or options are attractive but difficult to implement under the current model of support and operations (see expertise risk above). Cybersecurity - Particularly as it applies to on-board integrated monitoring and control systems.
53|P a g e
Doyouhaveanyothersuggestionsfortheworkshop?
Notatthistime
54|P a g e
Affiliation Name E-mail
Florida International University
Julio Ibarra
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
N/A
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
N/A
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
N/A
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
N/A
KeyRisks
N/A
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
NA/
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
N/A
55|P a g e
Doyouhaveanyothersuggestionsfortheworkshop?
N/A
56|P a g e
Affiliation Name E-mail
2-Dimensional Crystal Consortium, Pennsylvania State University
Yuanxi Wang
What percentage of the facility CI was developed in-house versus by reusing existingsolutions?
N/A
WhatexternalCIcapabilitiesandservicesand/orexternallydevelopedtools(ifany)doesthefacilityuseandwhoprovidesthem?Howwerethesetoolsidentifiedandwhatcriteriawasusedtoselectthetools?
N/A
Listupto3ofyourmostandleastfavoriteCIcomponentswitha1sentenceexplanationforeach.Whataspectsabout the facilityCI and itsoperationwouldyou like to shareasbestpractices?
N/A
WhataspectsofthefacilityCIanditsoperationdoyouseeaschallenges/gaps?Arethereanypitfalls/mistakes you would like to share? What aspects would you be interested inoutsourcing?
N/A
KeyRisks
N/A
WhatCI-relatedworkforcedevelopmentactivitiesdoesyourfacilitiesengagein?
N/A
WhatdoyouseeasyourkeynewCIrequirementsandchallengesinthenext5-10years
N/A
Doyouhaveanyothersuggestionsfortheworkshop?
57|P a g e
N/A