and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben...
Transcript of and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben...
![Page 1: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/1.jpg)
reproducibleand user-controlled
software management in HPC
with GNU GuixRicardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin
BOSC 2017, Prague
![Page 2: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/2.jpg)
Userswant fresh software,
multiple variants, latest
tools, flexibility
System Adminsprefer mature software,
no variants, only apply
unavoidable updates
use stable software
for systems
manage user software
stack on their own
![Page 3: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/3.jpg)
There is no way to reproduce the
environment, even on the same machine
at a different point in time.
Due to a lack of isolation, the
environment will change or even break
when the host system changes.
ad-hoc
volatileNo safe upgrades or roll-backs.
No separation for different workflows.
Unportable.primitive
![Page 4: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/4.jpg)
?
![Page 5: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/5.jpg)
...but reproducibility is still out of reach
Installing software is easy now!
![Page 6: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/6.jpg)
...but reproducibility is still out of reach
Installing software is easy now!
![Page 7: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/7.jpg)
![Page 8: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/8.jpg)
![Page 9: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/9.jpg)
100% reproducibleWe have all the bits!
![Page 10: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/10.jpg)
100% reproducibleWe have all the bits!
100% statefulWe only have the bits!
![Page 11: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/11.jpg)
App bundles are convenient
don't compose well
are like giant staticallylinked binaries
lack means ofabstraction
![Page 12: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/12.jpg)
![Page 13: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/13.jpg)
Functional packaging headerssourcesbuild toolslibraries... cabba9e-emacs-24.5/
bin
emacs
lib
...
![Page 14: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/14.jpg)
Functional packaging
cabba9e-emacs-24.5/
bin
emacs
lib
...
dedbeef-emacs-24.5/
bin
emacs
lib
...
Different inputs?Different outputs.
Same inputs?Same output!
![Page 15: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/15.jpg)
coreutils-8.24
perl-5.22.1
tar-1.28gzip-1.6 bzip2-1.0.6 xz-5.2.2file-5.25 diffutils-3.3
patch-2.7.5
sed-4.2.2findutils-4.6.0
gawk-4.1.3
grep-2.22 coreutils-8.24
make-4.1
bash-4.3.42
ld-wrapper-0
binutils-2.25.1
gcc-4.9.3
glibc-2.22
glibc-utf8-locales-2.22
acl-2.2.52
gmp-6.1.0
libcap-2.24
glibc-utf8-locales-2.22
gcc-4.9.3
ld-wrapper-boot3-0
binutils-cross-boot0-2.25.1
make-boot0-4.1
diffutils-boot0-3.3
findutils-boot0-4.6.0file-boot0-5.25
bootstrap-binaries-0
ed-1.12
libsigsegv-2.10
perl-boot0-5.22.1 perl-5.22.1
acl-2.2.52
gmp-6.1.0
libcap-2.24
pkg-config-0.29
guile-2.0.11
bison-3.0.4
readline-6.3
ncurses-6.0
gcc-cross-boot0-wrapped-4.9.3
texinfo-6.0
bash-static-4.3.42
libstdc++-4.9.3zlib-1.2.8
perl-boot0-5.22.1
gettext-boot0-0.19.7
gcc-cross-boot0-4.9.3
glibc-bootstrap-0
gcc-bootstrap-0
linux-libre-headers-3.14.37
gzip-1.6
gettext-0.19.7
attr-2.4.47
m4-1.4.17
gzip-1.6
guile-bootstrap-2.0
binutils-bootstrap-0
gettext-0.19.7
attr-2.4.47
m4-1.4.17
gcc-cross-boot0-wrapped-4.9.3
glibc-intermediate-2.22
m4-1.4.17
expat-2.1.0
lzip-1.16
pkg-config-0.29 libffi-3.2.1
readline-6.3
libunistring-0.9.6 libltdl-2.4.6
libgc-7.4.2gmp-6.1.0
ncurses-6.0libatomic-ops-7.4.2 m4-1.4.17
expat-2.1.0
![Page 16: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/16.jpg)
guix gc --references /gnu/store/…-foo-0.9
/gnu/store/…-glibc-2.25/gnu/store/…-gcc-4.9.3-lib/gnu/store/…-bar-0.7b/gnu/store/…-baz-1.4.9/gnu/store/…-foo-0.9
![Page 17: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/17.jpg)
guix package -i baz --with-input=foo=bar
guix package -i baz --with-source=baz-1.0.tgz
![Page 18: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/18.jpg)
guix pack samtools bedtools
/gnu/store/…-pack.tar.gz
![Page 19: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/19.jpg)
guix pack -f docker samtools bedtools
/gnu/store/ …-docker-pack.tar.gz
LOL
![Page 20: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/20.jpg)
guix package --manifest=GeneNetwork
All you need is:
Guix version + package manifest
(+ source code)
![Page 21: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/21.jpg)
The level ofabstraction matters1.Guix enables reproducibleand safe experimentation2.Guix makes environmentsharing easy3.There are ways touse Guix without root access4.Like Conda?5. guix package -i conda
![Page 22: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/22.jpg)
#guix on irc.freenode.net
http://gnu.org/s/guix
Learn more!
Poster A-142
![Page 23: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/23.jpg)
/home/rekado/.guix-profile
/var/guix/...
guix-profile
42
/gnu/store
hscq14x...-profile
bin
samtools
bowtie2
rhrdst11m...-samtools-1.2
bin
samtools
include
lud0v1c...-bowtie-2.2.4
bin
bowtie2
...
![Page 24: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/24.jpg)
/home/rekado/.guix-profile
/var/guix/...
guix-profile
42
43
/gnu/store
hscq14x...-profile
bin
samtools
bowtie2
rhrdst11m...-samtools-1.2
bin
samtools
include
lud0v1c...-bowtie-2.2.4
bin
bowtie2
z3braf1sh...-profile
bin
samtools
...
![Page 25: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/25.jpg)
/home/rekado/.guix-profile
/var/guix/...
guix-profile
42
43
/gnu/store
hscq14x...-profile
bin
samtools
bowtie2
rhrdst11m...-samtools-1.2
bin
samtools
include
lud0v1c...-bowtie-2.2.4
bin
bowtie2
z3braf1sh...-profile
bin
samtools
...
![Page 26: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/26.jpg)
/home/rekado/.guix-profile
/var/guix/...
guix-profile
43
/gnu/store
hscq14x...-profile
bin
samtools
bowtie2
rhrdst11m...-samtools-1.2
bin
samtools
include
lud0v1c...-bowtie-2.2.4
bin
bowtie2
z3braf1sh...-profile
bin
samtools
...
![Page 27: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/27.jpg)
/home/rekado/.guix-profile
/var/guix/...
guix-profile
43
/gnu/store
rhrdst11m...-samtools-1.2
bin
samtools
include
lud0v1c...-bowtie-2.2.4
bin
bowtie2
z3braf1sh...-profile
bin
samtools
...
![Page 28: and user-controlled software management in HPC with GNU …Ricardo Wurmus + Pjotr Prins + Ben Woodcroft + Altuna Akalin BOSC 2017, Prague. Users want fresh software, multiple variants,](https://reader035.fdocuments.in/reader035/viewer/2022070911/5fac003ab230873139635727/html5/thumbnails/28.jpg)
/home/rekado/.guix-profile
/var/guix/...
guix-profile
43
/gnu/store
rhrdst11m...-samtools-1.2
bin
samtools
include
z3braf1sh...-profile
bin
samtools
...