Managing HPC Software Complexity with Spack · LLNL-PRES-806064 This work was performed under the...
Transcript of Managing HPC Software Complexity with Spack · LLNL-PRES-806064 This work was performed under the...
LLNL-PRES-806064This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
github.com/spack/spack
ManagingHPCSoftwareComplexitywithSpack
Supercomputing2017Full-dayTutorialNovember,2017Denver,Colorado
LLNL-PRES-8060642github.com/spack/spack
Materials: Downloadthelatestversionofslidesandhandoutsat:
http://spack.readthedocs.io
Slidesandhands-onscriptsareinthe“Tutorial”Sectionofthedocs.
§ Spack GitHubrepository: http://github.com/spack/spack
§ Spack Documentation: http://spack.readthedocs.io
TutorialMaterials
Tweet at us!@spackpm #SC17
LLNL-PRES-8060643github.com/spack/spack
ToddGamblinGregoryBeckerGregLeeMattLegendre
MassimilianoCulpo
AdamStewart
MarioMelara
TutorialPresenters
LLNL-PRES-8060644github.com/spack/spack
Thecomplexityoftheexascale ecosystemthreatensproductivity.
§ Everyapplicationhasitsownstackofdependencies.§ Developers,users,andfacilitiesdedicate(many)FTEstobuilding&porting.§ Oftentradereuseandusabilityforperformance.
80+ software packagesx5+ target architectures/platforms
Xeon Power KNLNVIDIA ARM Laptops?
x
Up to 7 compilersIntel GCC Clang XL
PGI Cray NAGx
= up to 1,260,000 combinations!
15+ applications
x10+ Programming Models
OpenMPI MPICH MVAPICH OpenMP CUDAOpenACC Dharma Legion RAJA Kokkos
2-3 versions of each package + external dependenciesx
We must make it easier to rely on others’ software!
LLNL-PRES-8060645github.com/spack/spack
§ Someone’shomedirectory?
§ LLNL?LANL?Sandia?ANL?LBL?TACC?— Environmentsatlarge-scalesitesareverydifferent.
§ WhichMPIimplementation?
§ Whichcompiler?
§ Whichdependencies?
§ Whichversionsofdependencies?— Manyapplicationsrequirespecificdependencyversions.
Whatisthe“production”environmentforHPC?
Real answer: there isn’t a single production environment or a standard way to build.Reusing someone else’s software is HARD.
LLNL-PRES-8060646github.com/spack/spack
§ HowtoinstallSpack:
§ Howtoinstallapackage:
§ HDF5anditsdependenciesareinstalledwithintheSpack directory.
§ Unliketypicalpackagemanagers,Spack canalsoinstallmanyvariantsofthesamebuild.— Differentcompilers— DifferentMPIimplementations— Differentbuildoptions
Spack isaflexiblepackagemanagerforHPC
GetSpack!http://github.com/spack/spack
$ git clone https://github.com/spack/spack$ . spack/share/spack/setup-env.sh
$ spack install hdf5
@spackpm
LLNL-PRES-8060647github.com/spack/spack
PeoplewhowanttouseordistributesoftwareforHPC!
1. EndUsersofHPCSoftware— InstallandrunHPCapplicationsandtools
2. HPCApplicationTeams— Managethird-partydependencylibraries
3. PackageDevelopers— Peoplewhowanttopackagetheirownsoftwarefordistribution
4. UsersupportteamsatHPCCenters— PeoplewhodeploysoftwareforusersatlargeHPCsites
WhocanuseSpack?
LLNL-PRES-8060648github.com/spack/spack
Spack isusedworldwide
Sessions at spack.readthedocs.io by location, since April 2017
LLNL-PRES-8060649github.com/spack/spack
§ InNovember2015,LLNLprovidedmostofthecontributionstoSpack
§ Sincethen,we’vegonefrom300toover2,100packages
§ Mostpackagesarefromexternalcontributors!
§ Manycontributionsincore,aswell.
§ WearecommittedtosustainingSpack’s opensourceecosystem!
ContributionstoSpack aregrowing!
LLNL-PRES-80606410github.com/spack/spack
§ 30+organizations140+contributorsSharingover1,500packages andgrowing
§ Otherusecases:— ARMusingforentirecompilerregressionsuite.— LIGO collaborationusingfordeployment— Intel usingSpack topackageMLsoftware— NERSC usingSpack onCori:Craysupport.— EPFL (Switzerland)contributingcorefeatures.— Fermi,CERN,BNL:highenergyphysics.— ANL usingSpack onproductionLinuxclusters.— NASA packaginganIcemodelcode.— ORNL workingwithusonSpack forCORAL.— Kitware:corefeatures,ParaView,Qt,UV-CDAT
support
TheSpack communitynowspansDOEandbeyond
3IBM Brand ElementsIBM Impact 2010 DESIGN GUIDELINES
IBM Brand Elements
IBM Logo
IBM has one of the world’s most recognizable logos. Maintaining its integrity is an essential part of protecting the brand. The 8-bar blue logois more than a signature to our ads or a name on our products andservices. To customers, it is a promise that represents trust, value, quality and advanced technology. In a world of confusing, competitiveads and product claims, the IBM logo is uniquely reassuring, conveyingauthenticity, reliability and quality. The IBM logo is present on every formof IBM communication.
Color of the IBM Logo
IBM has one logo, which may appear only in black, white, or IBM Blue.The logo should appear consistently and without alteration, as shownhere. On a white or light background, use the IBM logo in black or IBMBlue. On a black or dark background, use the IBM logo in white or IBMBlue. The color itself is an important symbol of IBM, and should bespecified for printed documents using Pantone 2718 as a spot colorwhenever possible. Use of this spot color is necessary when setting thelogo in IBM Blue on a colored background.
IBM Blue
PMS 2718CMYK C75 M43 Y0 K0RGB R75 G107 B175WEB 006699
1 white on black or dark background
2 black on white or light background
3 IBM blue on white or light background
4 IBM blue on black or dark background
1 2
3 4
LLNL-PRES-80606411github.com/spack/spack
§ Majornewfeatures:1. Binarypackagingsupport(we’llusethistoday)2. Aninterfaceforpassingbuildinfobetweenpackages(demoedtoday)3. Testdependencies4. Multi-valuedvariants5. Morecontrolovercompilerflags6. BetterhelpoutputforSpack’s manycommands7. Betterbuildoutputhandlinganderrordisplay8. Numerousotherbugfixesandspeedimprovements
§ Over2,100packages
§ Spack nowhasitsownGitHuborganization(https://github.com/spack)
Spack v0.11.0wasjustreleased
LLNL-PRES-80606412github.com/spack/spack
§ Features:— Build/testinfrastructuretobuildandvalidatepackagebuildsregularly— Hostinginfrastructuretomakeselectpackageconfigurationsavialable asbinaries— Supportforcreatingcustomisolated“environments”(cf.Conda,Nix,Virtualenv)— Betterdependencyresolution(concretization)andcompilersupport— Distributed-parallelbuilds
§ Collaborations:— WorkwithUSExascale Programcodeteamstocoordinatesoftwarereleases— WorkwithDOElabstobuildandtestsoftwareonlargesupercomputers— Continueworkingwithbroadercommunitytointegratecontributions
WhatisontheSpack roadmap?
LLNL-PRES-80606413github.com/spack/spack
Spack isnotthefirsttooltoautomatebuilds— Inspiredbycopiouspriorwork
1. “Functional”PackageManagers— Nix https://nixos.org/— GNUGuix https://www.gnu.org/s/guix/
2. Build-from-sourcePackageManagers— Homebrew http://brew.sh— MacPorts https://www.macports.org
OthertoolsintheHPCSpace:
§ Easybuild http://hpcugent.github.io/easybuild/— Aninstallation toolforHPC— FocusedonHPCsystemadministrators– differentpackagemodelfromSpack— Reliesonafixedsoftwarestack– hardertotweakrecipesforexperimentation
§ Hashdist https://hashdist.github.io
RelatedWork
LLNL-PRES-80606414github.com/spack/spack
1. Foreveryone: Welcome&Overview 8:30- 8:50
2. Foreveryone: BasicsofBuildingandLinking 8:50- 9:00
3. Foreveryone: Spack Basics(handson) 9:00- 9:45
4. Foreveryone: CoreSpack Concepts 9:45- 10:00
-- Break-- 10:00- 10:30
5. Foreveryone: Configuration(hands-on) 10:3011:00
6. Forpackagers: MakingyourownPackages(handson) 11:00– 12:00
-- TutorialsLunch(FourSeasonsBallroom)-- 12:00- 1:30
7. Forpackagers: AdvancedPackaging(buildsystems) 1:30– 2:15
8. Forpackagers: AdvancedPackaging(virtualdependencies:BLAS/LAPACK/MPI) 2:15-3:00
-- CoffeeBreak-- 3:30- 3:30
9. ForHPCcenters: DeployingSpack withLmod atHPCFacilities(hands-on) 3:30- 4:15
10. Foreveryone: BuildyourownpackageswithhelpfromtheSpack team! 4:15– 5:00
TutorialOverview(timesareestimates)
LLNL-PRES-80606415github.com/spack/spack
Pleasesubmityourfeedbackonthistutorialat:
Tutorialreviewsareonlinethisyear
http://bit.ly/sc17-evalYour reviews help us keep the tutorial going!
LLNL-PRES-80606417github.com/spack/spack
What’sapackagemanager?
§ Spack isapackagemanager— Doesnot areplaceCmake/Autotools— PackagesbuiltbySpack canhaveany
buildsystemtheywant
§ Spack managesdependencies— Drivespackage-levelbuildsystems— Ensuresconsistentbuilds
§ Determiningmagicconfigurelinestakestime— Spack isacacheofrecipes
• Manages package installation• Manages dependency relationships• Drives package-level build systems
Package Manager
• Cmake, Autotools• Handle library abstractions• Generate Makefiles, etc.
High Level Build
System
• Make, Ninja• Handles dependencies among commands in a single build
Low Level Build
System
LLNL-PRES-80606418github.com/spack/spack
§ Staticlibraries:libfoo.a— .afilesarearchivesof.ofiles(objectfiles)— Linkerincludesneededpartsofastaticlibraryintheoutputexecutable— Noneedtofinddependenciesatruntime– onlyatbuildtime.— Canleadtolargeexecutables— Oftenhardtobuildacompletelystaticexecutableonmodernsystems.
§ Sharedlibraries:libfoo.so (Linux),libfoo.dylib (MacOS)— Morecomplexbuildsemantics,typicallyhandledbythebuildsystem— Mustbefoundbyld.so ordyld (dynamiclinker)andloadedatruntime
• Cancauselotsofheadacheswithmultipleversions— 2mainways:
• LD_LIBRARY_PATH:environmentvariableconfiguredbyuserand/ormodulesystem• RPATH:pathsembeddedinexecutablesandlibraries,sothattheyknowwheretofindtheirowndependencies.
Staticvs.sharedlibraries
LLNL-PRES-80606419github.com/spack/spack
§ API:ApplicationProgrammingInterface— Sourcecodefunctionsandsymbolnamesexposedbyalibrary— IfAPIofadependencyisbackwardcompatible,sourcecodeneednotbechangedtouseit— May needtorecompilecodetouseanewversion.
§ ABI:ApplicationBinaryInterface— Callingconventions,registersemantics,exceptionhandling,etc.— Definedbyhowthecompilerbuildsalibrary
• BinariesgeneratedbydifferentcompilersaretypicallyABI-incompatible.— Mayalsoincludethingslikestandardruntimelibrariesandcompilerintrinsicfunctions— Mayalsoincludevaluesofhard-codedsymbols/constantsinheaders.
§ HPCcode,includingMPI,istypicallyAPI-compatiblebutnotABI-compatible.— Causesmanybuildproblems,especiallyfordynamicallyloadedlibraries— OftenneedtorebuildtogetaroundABIproblems— LeadstocombinatorialbuildsofsoftwareatHPCsites.
APIandABICompatibility
LLNL-PRES-80606420github.com/spack/spack
1. Make(usuallyGNUMake)— https://www.gnu.org/software/make/
2. GNUAutotools— Automake: https://www.gnu.org/software/automake/— Autoconf: https://www.gnu.org/software/autoconf/— Libtool: https://www.gnu.org/software/libtool/
3. CMake:— https://cmake.org
3majorbuildsystemstobeawareof
Spack has built-in support for these plus Waf, Perl, Python, and R
LLNL-PRES-80606421github.com/spack/spack
§ ManyprojectsopttowritetheirownMakefiles.— Canrangefromsimpletovery
complicated
§ Makedeclaressomestandardvariablesforvariouscompilers— ManyHPCprojectsdon’trespectthem— Nostandardinstallprefixconvention— Makefiles maynothaveinstalltarget
§ AutomatingbuildswithMakeusuallyrequireseditingfiles— Typicaltousesed/awk/someother
regularexpressiontoolonMakefile— Canalsousepatches
MakeandGNUMake
<edit Makefile>make PREFIX=/path/to/prefix
Typical build incantation
None. Typically must edit Makefiles.
Configure options
Environment variablesCC CFLAGS LDFLAGSCXX CXXFLAGS LIBSFC FFLAGS CPPF77 FFLAGS
LLNL-PRES-80606422github.com/spack/spack
§ Threepartsofautotools:— autoconf:generatesaportable
configurescripttoinspectbuildhost— automake:high-levelsyntaxfor
generatinglower-levelMakefiles.— libtool:abstractionforsharedlibraries
§ Typicalvariablesaresimilartomake
§ Muchmoreconsistencyamongautotools projects— Wideuseofstandardvariablesand
configureoptions— Standardinstalltarget,staging
conventions.
Autotools
./configure --prefix=/path/to/install_dirmakemake install
Typical build incantation
./configure \--prefix=/path/to/install_dir \--with-package=/path/to/dependency \--enable-foo \--disable-bar
Configure options
Useful environment variablesCC CFLAGS LDFLAGSCXX CXXFLAGS LIBSFC FFLAGS CPPF77 FFLAGS
Environment variables
LLNL-PRES-80606423github.com/spack/spack
§ Gainingpopularity
§ Arguablyeasiertouse(fordevelopers)thanautotools
§ Similarstandardoptionstoautotools— differentvariablenames— Moreconfigurationoptions— Abstractsplatform-specific
detailsofsharedlibraries
§ MostCMake projectsshouldbebuilt“outofsouce”— Separatebuilddirectoryfrom
sourcedirectory
CMake
mkdir BUILD && cd BUILDcmake –DCMAKE_INSTALL_PREFIX=/path/to/install_dir ..makemake install
Typical build incantation
cmake \–D CMAKE_INSTALL_PREFIX=/path/to/install_dir \-D ENABLE_FOO=yes \-D ENABLE_BAR=no \..
Configure options
Useful environment variablesCMAKE_C_COMPILER CMAKE_C_FLAGSCMAKE_CXX_COMPILER CMAKE_CXX_FLAGSCMAKE_Fortran_COMPILER CMAKE_Fortran_FLAGSCMAKE_SHARED_LINKER_FLAGS CMAKE_EXE_LINKER_FLAGSCMAKE_STATIC_LINKER_FLAGS
Common –D options
LLNL-PRES-80606425github.com/spack/spack
§ Eachexpressionisaspecforaparticularconfiguration— Eachclauseaddsaconstrainttothespec— Constraintsareoptional– specifyonlywhatyouneed.— Customizeinstallonthecommandline!
§ Specsyntaxisrecursive— Fullcontroloverthecombinatorialbuildspace
Spack providesaspecsyntaxtodescribecustomizedDAGconfigurations
$ spack install mpileaks unconstrained$ spack install [email protected] @ custom version$ spack install [email protected] %[email protected] % custom compiler$ spack install [email protected] %[email protected] +threads +/- build option$ spack install [email protected] cppflags="-O3 –g3" setting compiler flags$ spack install [email protected] os=CNL10 target=haswell setting target for X-compile$ spack install [email protected] ^[email protected] %[email protected] ^ dependency information
LLNL-PRES-80606426github.com/spack/spack
`spack list`showswhatpackagesareavailable
$ spack list==> 303 packages.activeharmony cgal fish gtkplus libgd mesa openmpi py-coverage py-pycparser qt tcladept-utils cgm flex harfbuzz libgpg-error metis openspeedshop py-cython py-pyelftools qthreads texinfoapex cityhash fltk hdf libjpeg-turbo Mitos openssl py-dateutil py-pygments R the_silver_searcherarpack cleverleaf flux hdf5 libjson-c mpc otf py-epydoc py-pylint ravel thriftasciidoc cloog fontconfig hwloc libmng mpe2 otf2 py-funcsigs py-pypar readline tkatk cmake freetype hypre libmonitor mpfr pango py-genders py-pyparsing rose tmuxatlas cmocka gasnet icu libNBC mpibash papi py-gnuplot py-pyqt rsync tmuxinatoratop coreutils gcc icu4c libpciaccess mpich paraver py-h5py py-pyside ruby trilinosautoconf cppcheck gdb ImageMagick libpng mpileaks paraview py-ipython py-pytables SAMRAI uncrustifyautomaded cram gdk-pixbuf isl libsodium mrnet parmetis py-libxml2 py-python-daemon samtools util-linux automake cscope geos jdk libtiff mumps parpack py-lockfile py-pytz scalasca valgrindbear cube gflags jemalloc libtool munge patchelf py-mako py-rpy2 scorep vimbib2xhtml curl ghostscript jpeg libunwind muster pcre py-matplotlib py-scientificpython scotch vtkbinutils czmq git judy libuuid mvapich2 pcre2 py-mock py-scikit-learn scr wgetbison damselfly glib julia libxcb nasm pdt py-mpi4py py-scipy silo wxboost dbus glm launchmon libxml2 ncdu petsc py-mx py-setuptools snappy wxpropgridbowtie2 docbook-xml global lcms libxshmfence ncurses pidx py-mysqldb1 py-shiboken sparsehash xcb-protoboxlib doxygen glog leveldb libxslt netcdf pixman py-nose py-sip spindle xerces-c bzip2 dri2proto glpk libarchive llvm netgauge pkg-config py-numexpr py-six spot xzcairo dtcmp gmp libcerf llvm-lld netlib-blas pmgr_collective py-numpy py-sphinx sqlite yasmcallpath dyninst gmsh libcircle lmdb netlib-lapack postgresql py-pandas py-sympy stat zeromqcblas eigen gnuplot libdrm lmod netlib-scalapack ppl py-pbr py-tappy sundials zlibcbtf elfutils gnutls libdwarf lua nettle protobuf py-periodictable py-twisted swig zshcbtf-argonavis elpa gperf libedit lwgrp ninja py-astropy py-pexpect py-urwid szipcbtf-krell expat gperftools libelf lwm2 ompss py-basemap py-pil py-virtualenv tarcbtf-lanl extrae graphlib libevent matio ompt-openmp py-biopython py-pillow py-yapf taskcereal exuberant-ctags graphviz libffi mbedtls opari2 py-blessings py-pmw python taskdcfitsio fftw gsl libgcrypt memaxes openblas py-cffi py-pychecker qhull tau
§ Spack hasover2,100packagesnow.
LLNL-PRES-80606427github.com/spack/spack
§ Alltheversionscoexist!— Multipleversionsof
samepackageareok.
§ Packagesareinstalledtoautomaticallyfindcorrectdependencies.
§ Binariesworkregardlessofuser’senvironment.
§ Spack alsogeneratesmodulefiles.— Don’thave tousethem.
`spack find`showswhatisinstalled
$ spack find==> 103 installed packages.-- linux-redhat6-x86_64 / [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]@3.9.1 [email protected] [email protected] [email protected] [email protected]@1.0 [email protected] [email protected] [email protected] [email protected]@2.14.0 [email protected] [email protected] [email protected] [email protected]@1.55.0 [email protected] [email protected] [email protected] [email protected]@1.14.0 [email protected] [email protected] [email protected] [email protected]@1.0.2 jpeg@9a [email protected] [email protected] [email protected]@8.1.2 libdwarf@20130729 [email protected] [email protected] [email protected]@8.1.2 [email protected] ocr@2015-02-16 [email protected] [email protected]@2.11.1 [email protected] [email protected] [email protected]@2.5.3 [email protected] [email protected] [email protected]@2.31.2 [email protected] [email protected] [email protected]
-- linux-redhat6-x86_64 / [email protected] [email protected] [email protected] [email protected] libdwarf@20130729 [email protected]@1.0.1 [email protected] [email protected] [email protected] [email protected]
-- linux-redhat6-x86_64 / [email protected] [email protected] [email protected] [email protected]
-- linux-redhat6-x86_64 / [email protected] [email protected] [email protected] libdwarf@20130729 [email protected] [email protected]
-- linux-redhat6-x86_64 / [email protected] [email protected] [email protected] libdwarf@20130729 [email protected]@1.55.0 [email protected] [email protected] [email protected]
LLNL-PRES-80606428github.com/spack/spack
Userscanquerythefulldependencyconfigurationofinstalledpackages.
§ Architecture,compiler,versions,andvariantsmaydifferbetweenbuilds.
$ spack find callpath==> 2 installed packages.-- linux-x86_64 / [email protected] ———————— -- linux-x86_64 / [email protected] [email protected] [email protected]
Expand dependencies with spack find -d
$ spack find -dl callpath==> 2 installed packages.-- linux-x86_64 / [email protected] ----------- -- linux-x86_64 / [email protected] -----------xv2clz2 [email protected] udltshs [email protected] ^[email protected] rfsu7fb ^[email protected] ^[email protected] ybet64y ^[email protected] ^[email protected] aa4ar6i ^[email protected] ^[email protected] tmnnge5 ^[email protected] ^[email protected] ybet64y ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^[email protected] aa4ar6i ^[email protected]
$ spack find -dl callpath==> 2 installed packages.-- linux-x86_64 / [email protected] ----------- -- linux-x86_64 / [email protected] -----------xv2clz2 [email protected] udltshs [email protected] ^[email protected] rfsu7fb ^[email protected] ^[email protected] ybet64y ^[email protected] ^[email protected] aa4ar6i ^[email protected] ^[email protected] tmnnge5 ^[email protected] ^[email protected] ybet64y ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^[email protected] ynpai3j ^[email protected] ^[email protected] ynpai3j ^[email protected] ^[email protected] aa4ar6i ^[email protected]
LLNL-PRES-80606429github.com/spack/spack
Spack managesinstalledcompilers
§ Compilersareautomaticallydetected— AutomaticdetectiondeterminedbyOS— Linux:PATH— Cray:`moduleavail`
§ Compilerscanbemanuallyadded— IncludingSpack-builtcompilers
compilers:- compiler:
modules: []operating_system: ubuntu14paths:cc: /usr/bin/gcc/4.9.3/gcccxx: /usr/bin/gcc/4.9.3/g++f77: /usr/bin/gcc/4.9.3/gfortranfc: /usr/bin/gcc/4.9.3/gfortran
spec: [email protected] compiler:
modules: []operating_system: ubuntu14paths:cc: /usr/bin/clang/6.0/clangcxx: /usr/bin/clang/6.0/clang++f77: nullfc: null
spec: [email protected] - compiler:
...
$ spack compilers==> Available compilers-- gcc [email protected] [email protected]
-- clang [email protected]
compilers.yaml
LLNL-PRES-80606431github.com/spack/spack
§ Traditionalbinarypackagemanagers— RPM,yum,APT,yast,etc.— Designedtomanageasinglestack.— Installone versionofeachpackageinasingleprefix(/usr).— Seamlessupgradestoastable,welltested stack
§ Portsystems— BSDPorts,portage,Macports,Homebrew,Gentoo,etc.— Minimalsupportforbuildsparameterizedbycompilers,dependencyversions.
§ VirtualMachinesandLinuxContainers(Docker)— Containersallowuserstobuildenvironmentsfordifferentapplications.— Doesnotsolvethebuildproblem(someonehastobuildtheimage)— Performance,security,andupgradeissuespreventwidespreadHPCdeployment.
Mostexistingtoolsdonotsupportcombinatorialversioning
LLNL-PRES-80606432github.com/spack/spack
§ Eachuniquedependencygraphisauniqueconfiguration.
§ Eachconfigurationinstalledinauniquedirectory.— Configurationsofthesamepackagecancoexist.
§ Hash ofentiredirectedacyclicgraph(DAG)isappendedtoeachprefix.
§ Installedpackagesautomaticallyfinddependencies— Spack embedsRPATHsinbinaries.— NoneedtousemodulesorsetLD_LIBRARY_PATH— Thingsworkthewayyoubuiltthem
Spack handlescombinatorialsoftwarecomplexity.
spack/opt/linux-x86_64/
gcc-4.7.2/mpileaks-1.1-0f54bf34cadk/
intel-14.1/hdf5-1.8.15-lkf14aq3nqiz/
bgq/xl-12.1/
hdf5-1-8.16-fqb3a15abrwx/...
mpileaks
mpi
callpath dyninst
libdwarf
libelf
InstallationLayout
DependencyDAG
Hash
LLNL-PRES-80606433github.com/spack/spack
mpileaks
mpi
callpath dyninst
libdwarf
libelf
§ Spack ensuresoneconfigurationofeachlibraryperDAG— EnsuresABIconsistency.— UserdoesnotneedtoknowDAGstructure;onlythedependencynames.
§ Spack canensurethatbuildsusethesamecompiler,oryoucanmix— WorkingonensuringABIcompatibilitywhencompilersaremixed.
Spack Specscanconstrainversionsofdependencies
$ spack install mpileaks %[email protected] ^[email protected]
LLNL-PRES-80606434github.com/spack/spack
Spack handlesABI-incompatible,versionedinterfaceslikeMPI
$ spack install mpileaks ^[email protected] $ spack install mpileaks ^[email protected]:
$ spack install mpileaks ^mpi@2
mpileaks
mpi
callpath dyninst
libdwarf
libelf
§ mpi isavirtualdependency
§ InstallthesamepackagebuiltwithtwodifferentMPIimplementations:
§ LetSpack chooseMPIimplementation,aslongasitprovidesMPI2interface:
LLNL-PRES-80606435github.com/spack/spack
Concretizationfillsinmissingconfigurationdetailswhentheuserisnotexplicit.
mpileaks ^[email protected]+debug ^[email protected] User input: abstract spec with some constraints
Concrete spec is fully constrainedand can be passed to install.
[email protected]%[email protected]=linux-ppc64
[email protected]%[email protected]=linux-ppc64
[email protected]%[email protected]+debug
=linux-ppc64
[email protected]%[email protected]=linux-ppc64
[email protected]%[email protected]=linux-ppc64
libdwarf@20130729%[email protected]=linux-ppc64
Abstract, normalized specwith some dependencies.
mpileaks
mpi
[email protected]+debug
dyninst
libdwarf
Norm
alize
Concretize Store
spec:- mpileaks:
arch: linux-x86_64compiler:name: gccversion: 4.9.2
dependencies:adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77mpich: aa4ar6ifj23yijqmdabeakpejcli72t3
hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujhvariants: {} version: '1.0'
- adept-utils:arch: linux-x86_64compiler:name: gccversion: 4.9.2
dependencies:boost: teesjv7ehpe5ksspjim5dk43a7qnowlqmpich: aa4ar6ifj23yijqmdabeakpejcli72t3
hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4variants: {} version: 1.0.1
- boost:arch: linux-x86_64compiler:name: gccversion: 4.9.2
dependencies: {} hash: teesjv7ehpe5ksspjim5dk43a7qnowlqvariants: {} version: 1.59.0
...
spec.yaml
Detailed provenance is storedwith the installed package
LLNL-PRES-80606436github.com/spack/spack
Use`spack spec`toseetheresultsofconcretization
$ spack spec mpileaksInput spec------------------------------
mpileaks
Normalized------------------------------
mpileaks^adept-utils
^[email protected]:^mpi
^callpath^dyninst
^libdwarf^libelf
Concretized------------------------------
[email protected]%[email protected] arch=darwin-elcapitan-x86_64^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]+atomic+chrono+date_time~debug+filesystem~graph~icu_support+iostreams+locale+log+math~mpi+multithreaded+program_options~python+random +regex+serialization+shared+signals+singlethreaded+system+test+thread+timer+wave arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]+sigsegv arch=darwin-elcapitan-x86_64^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64^[email protected]%[email protected]~stat_dysect arch=darwin-elcapitan-x86_64
^libdwarf@20160507%[email protected] arch=darwin-elcapitan-x86_64^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
LLNL-PRES-80606437github.com/spack/spack
§ Spack installspackagesinownprefix
§ Somepackagesneedtobeinstalledwithindirectorystructureofotherpackages— i.e.,Pythonmodulesinstalledin$prefix/lib/python-<version>/site-packages— Spack supportsthisviaextensions
ExtensionsandPythonSupport
class PyNumpy(Package):"""NumPy is the fundamental package for scientific computing with Python."""
homepage = "https://numpy.org"url = "https://pypi.python.org/packages/source/n/numpy/numpy-1.9.1.tar.gz"version('1.9.1', ' 78842b73560ec378142665e712ae4ad9’)
extends('python')
def install(self, spec, prefix):setup_py("install“, "--prefix={0}".format(prefix))
LLNL-PRES-80606438github.com/spack/spack
Spack extensions
spack/opt/linux-redhat6-x86_64/
gcc-4.7.2/python-2.7.12-/
python/py-numpy
py-numpy-1.10.4-oaxix36/py-numpy/
...
§ SymboliclinktoSpack installlocation
§ Automaticallyactivateforcorrectversionofdependency— ProvenanceinformationfromDAG— Activatealldependenciesthatareextensionsaswell
spack/opt/linux-redhat6-x86_64/
gcc-4.7.2/python-2.7.12-6y6vvaw/
python/py-numpy-1.10.4-oaxix36/
py-numpy/...
$ spack activate py-numpy @1.10.4
§ Examplesofextensionpackages:— pythonlibrariesareagoodexample— R,Lua,perl— Needtomaintaincombinatorialversioning
LLNL-PRES-80606439github.com/spack/spack
Extensionsmustbeactivatedintoextendee
$ spack extensions python==> [email protected]% [email protected] ~tk~ucs4 arch=linux-redhat7-x86_64-i25k4oi==> 118 extensions:…py-autopep8 py-docutils py-mako py-numpy py-py2neo …
==> 3 installed:-- linux-redhat7-x86_64 / [email protected] [email protected] [email protected] [email protected]
==> None activated.
$ spack load python $ python Python 2.7.12 (default, Aug 26 2016, 15:12:42)[GCC 4.9.3] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import numpyTraceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named numpy>>>
$ spack activate py-numpy==> Activated extension [email protected]% [email protected] +blas+lapackarch=linux-redhat7-x86_64-77im5ny for [email protected] ~tk~ucs4 % [email protected]$ spack extensions python==> [email protected]% [email protected] ~tk~ucs4 arch=linux-redhat7-x86_64-i25k4oi…==> 1 currently activated:-- linux-redhat7-x86_64 / [email protected] -----------------------------
$ python Python 2.7.12 (default, Aug 26 2016, 15:12:42)[GCC 4.9.3] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import numpy>>> x = numpy.zeros(10)>>>
§ Pythonunawareofnumpy installation
▪ activate symlinks entirenumpy prefixintoPythoninstallation▪ Canalternativelyloadextension
$ spack load python$ spack load py-numpy
LLNL-PRES-80606440github.com/spack/spack
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64+debug
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
libdwarf@[email protected]
arch=linux-redhat6-ppc64
Buildingagainstexternallyinstalledsoftware
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64+debug
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
[email protected]@4.7.3
arch=linux-redhat6-ppc64
libdwarf@[email protected]
arch=linux-redhat6-ppc64
/path/to/external/gcc/openmpi-2.0.0
packages:mpi:buildable: False
openmpi:buildable: Falsepaths:[email protected] %[email protected] arch=linux-redhat6-ppc64:/path/to/external/gcc/openmpi-2.0.0
[email protected] %[email protected] arch=linux-redhat6-ppc64:/path/to/external/gcc/openmpi-1.10.3
[email protected] %[email protected] arch=linux-redhat6-ppc64:/path/to/external/intel/openmpi-2.0.0
[email protected] %[email protected] arch=linux-redhat6-ppc64:/path/to/external/intel/openmpi-1.10.3
...
packages.yaml
A user registers external packages with Spack.
If a node in the DAG matches an registered external package, Spack prunes the DAG at that node and replaces the node with a reference to the external package.
mpileaks ^[email protected]+debug ^openmpi ^[email protected]
LLNL-PRES-80606441github.com/spack/spack
$ spack repo create /path/to/my_repo$ spack repo add my_repo$ spack repo list==> 2 package repositories.my_repo /path/to/my_repobuiltin spack/var/spack/repos/builtin
§ Somepackagescannotbereleasedpublicly
§ Someusershaveusecasesthatrequirebizarre custombuilds
§ PackagingissuesshouldnotpreventusersfromupdatingSpack— Solution:separaterepositories— Arepositoryissimplyadirectoryofpackagefiles
§ Spack supportsexternalrepositoriesthatcanbelayeredontopofthebuilt-inpackages
§ Custompackagescandependonbuilt-inpackages(orpackagesinotherrepositories)
Spack packagerepositories
my_repo var/spack/repos/builtin
Proprietary packages
Pathological build cases
“Standard” packages
LLNL-PRES-80606442github.com/spack/spack
§ UserCache— Userscreatea
“mirror”oftararchivesforpackages
— Removeinternetdependence
Fetchingsourcecodeforspack
Concretize
spack install mpileaks
Recursively install dependencies
Fetch package source
Build software
Load package from repository
User Cache
Spack Cache
The Internet§ Spack Cache
— Spack automaticallycachestararchivesforpreviouslyinstalledsoftware
§ TheInternet— Spack packagescanfindsourcefilesonline
LLNL-PRES-80606443github.com/spack/spack
Addingcustomcompilerflags
$ spack install hdf5 cflags='-O3 –g –fast –fpack-struct'
§ ThisinstallsHDF5withthespecifiedflags— FlagsareinjectedviaSpack’s compilerwrappers(discussedlater).
§ Flagsarepropagatedtodependenciesautomatically— FlagsareincludedintheDAGhash— Eachbuildwithdifferentflagsisconsideredadifferentversion
§ Thisprovidesaneasyharnessfordoingparameterstudiesfortuningcodes— Previouslyworkingwithlargecodeswasverytedious.
§ Supportscflags,cxxflags,fflags,cppflags,ldflags,andldlibs— AddedfromCLIorconfig file
LLNL-PRES-80606444github.com/spack/spack
Hands-onTime:Configuration
Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"
LLNL-PRES-80606446github.com/spack/spack
CreatingyourownSpackPackages
from spack import *
class Zlib(Package):"""A free, general-purpose, legally unencumbered lossless
data-compression library."""
homepage = "http://zlib.net"url = "http://zlib.net/zlib-1.2.8.tar.gz"
version('1.2.8', '44d667c142d7cda120332623eab69f40')
depends_on(‘cmake’, type=‘build’)
def install(self, spec, prefix):configure('--prefix={0}'.format(prefix))
make()make('install')
▪ Packageisarecipeforbuilding
▪ EachpackageisaPythonclass— Downloadinformation— Versions/Checksums— Buildoptions— Dependencies— Buildinstructions
▪ Packagecollectionsarerepos— Spackhasa“builtin”repoin
var/spack/repos/builtin
$REPO/packages/zlib/package.py
LLNL-PRES-80606447github.com/spack/spack
Spack packagesaretemplates forbuilds
def install(self, spec, prefix):config_opts=[‘--prefix=‘+prefix]
if '~shared' in self.spec:config_opts.append('--disable-shared')
else:config_opts.append('--enable-shared')
configure(config_opts)make()make('install')
▪ Eachpackagehasoneclass— zlib forIntelcompilerandzlib forGCCcompilerare
builtwiththesamerecipe.
▪ Canaddconditionallogicusingspecsyntax— Thinkofpackageastranslating aconcreteDAGto
buildinstructions.— Dependenciesarealreadybuilt— Nosearchingortesting;justdowhattheDAGsays
▪ Compilerwrappershandlemanydetailsautomatically.— Spack feedscompilerwrappersto(cc,c++,f90,…)
toautoconf,cmake,gmake,…— Wrappersselectcompilers,dependencies,and
optionsunderthehood.
package.py
LLNL-PRES-80606448github.com/spack/spack
Spackbuildseachpackageinitsowncompilationenvironment
SpackProcess
Setupenvironment
CC = spack/env/spack-cc SPACK_CC = /opt/ic-15.1/bin/iccCXX = spack/env/spack-c++ SPACK_CXX = /opt/ic-15.1/bin/icpcF77 = spack/env/spack-f77 SPACK_F77 = /opt/ic-15.1/bin/ifortFC = spack/env/spack-f90 SPACK_FC = /opt/ic-15.1/bin/ifort
PKG_CONFIG_PATH = ... PATH = spack/env:$PATHCMAKE_PREFIX_PATH = ...LIBRARY_PATH = ...
do_install()
Install dep1 Install dep2 Install package…
BuildProcess
Fork
install() configure make makeinstall
-I /dep1-prefix/include-L /dep1-prefix/lib-Wl,-rpath=/dep1-prefix/lib
Compilerwrappers(spack-cc, spack-c++, spack-f77, spack-f90)
icc icpc ifort
▪ Forkedbuildprocessisolatesenvironmentforeachbuild.Usescompilerwrappersto:— Addinclude,lib,andRPATHflags— Ensurethatdependenciesarefoundautomatically— LoadCraymodules(userightcompiler/systemdeps)
LLNL-PRES-80606449github.com/spack/spack
WritingPackages- VersionsandURLs
class Mvapich2(Package):homepage = "http://mvapich.cse.ohio-state.edu/"url = "http://mvapich.cse.ohio-state.edu/download/mvapich/mv2/mvapich2-2.2rc2.tar.gz"
version('2.2rc2', 'f9082ffc3b853ad1b908cf7f169aa878')version('2.2b', '5651e8b7a72d7c77ca68da48f3a5d108')version('2.2a', 'b8ceb4fc5f5a97add9b3ff1b9cbe39d2')version('2.1', '0095ceecb19bbb7fb262131cb9c2cdd6')
▪ PackagedownloadsarehashedwithMD5bydefault— AlsosupportsSHA-1,SHA-256,SHA-512— We’llbeswitchingtoSHA-256orhighersoon.
▪ DownloadURLscanbeautomaticallyextrapolatedfromURL.— ExtraoptionscanbeprovidedifSpack can’textrapolateURLs
▪ OptionscanalsobeprovidedtofetchfromVCSrepositories
$REPO/packages/mvapich/package.py
LLNL-PRES-80606450github.com/spack/spack
WritingPackages– VariantsandDependencies
class Petsc(Package):variant('mpi', default=True, description='Activates MPI support')variant('complex', default=False, description='Build with complex numbers')variant('hdf5', default=True, description='Activates support for HDF5 (only parallel)')
depends_on('blas')depends_on('[email protected]:2.7')
depends_on('mpi', when='+mpi')
▪ Variantsarenamed,havedefaultvaluesandhelptext
▪ Otherpackagescanbedependencies— whenclauseprovidesconditionaldependencies— Candependonspecificversionsorothervariants
$REPO/packages/petsc/package.py
LLNL-PRES-80606451github.com/spack/spack
WritingPackages– BuildRecipes
def install(self, spec, prefix):with working_dir("build", create=True):
cmake("..", *std_cmake_args)make()make("install")
@when('@:8.1')def install(self, spec, prefix):
configure("--prefix=" + prefix)make()make("install")
▪ Functionswrapcommonops— cmake,configure,patch,make,…— Executable andwhichfornewwrappers.
▪ Commandsexecutedincleanenvironment
▪ FullPythonfunctionality— Patchupsourcecode— Makefilesanddirectories— Calculateflags— …
$REPO/packages/dyninst/package.py
LLNL-PRES-80606452github.com/spack/spack
Createnewpackageswithspack create
class Zlib(Package):# FIXME: Add a proper url for your package's homepage here. homepage = "http://www.example.com"url = "http://zlib.net/zlib-1.2.8.tar.gz"version('1.2.8', '44d667c142d7cda120332623eab69f40')
def install(self, spec, prefix):# FIXME: Modify the cmake line to suit your build system here.
▪ spack create<url>willcreateaskeletonforapackage— Spack reasonsaboutURL,hash,version,buildrecipe.— GeneratesboilerplateforCmake,Makefile,autotools,Python,R,Waf,Perl
— Notintendedtocompletelywritethepackage,butgetsyou80%ofthewaythere.
▪ spack edit<package>forsubsequentchanges
$ spack create http://zlib.net/zlib-1.2.8.tar.gz
$REPO/packages/zlib/package.py
LLNL-PRES-80606453github.com/spack/spack
Hands-onTime:CreatingPackages
Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"
LLNL-PRES-80606454github.com/spack/spack
Hands-onTime:AdvancedPackaging
Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"
LLNL-PRES-80606455github.com/spack/spack
Hands-onTime:ModuleFiles
Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"
LLNL-PRES-80606456github.com/spack/spack
§ ContributingpackagestoSpack issimple— Makepackagesonyourownsystem— SendapullrequesttoSpack toletothersusethem— GitHubguidetopullrequests:
https://help.github.com/articles/creating-a-pull-request/— SeecontributorguideintheSpack repository— Spack islicensedunderLGPLv2.1
§ Wewantmorethanjustpackages— Newfeatures— Newdocumentation— Anycontributioncanhelpthecommunity!
§ Spack hasahelpfulonlinecommunity— TypicallyhappytorespondtoGitHubissues— ActivemailinglistonGoogleGroups
JointheSpack community!
@spackpm
github.com/spack/spack
We hope to make distributing & using HPC software easy!