Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by...
Transcript of Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by...
![Page 1: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/1.jpg)
![Page 2: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/2.jpg)
Singularity A brief history
• Invented by Greg Kurtzer at LBNL to address problems specific to HPC
• Development began in October 2015
• First release April 2016
• User feedback, major changes and help from the community leads to 2.0 in June 2016
• Greg founds Sylabs.io January 1, 2018; already at 28 employees, product and revenue!
• Sylabs developers completely overhaul code base, create a suite of cloud services, and release 3.0 October 8, 2018!
![Page 3: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/3.jpg)
Widely Adopted for Compute Needs
Singularity is the container runtime of choice for compute based workloads like
simulation and AI
As of Singularity 2.x…
● Millions container runs per day● Estimated user base of more than 25k● Installed on over 3 million sockets
HPCWire Readers and Editors Choice Awards:
● 2016: Top products to watch● 2017: Top products to watch● 2017: Best HPC Programming Tool/Tech
The Uptake of Singularity has been even faster than CentOS!
![Page 4: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/4.jpg)
Singularity is Designed Specifically for HPC
● Compatible with traditional HPC architecture and workflows
● Supports all known resource managers
● Optimized for compute jobs and performance
● Supports GPUs, IB, OmniPath, MPI, parallel file systems, production kernels, etc...
● Users are themselves within the container
● Allows users to “BYOE”
● Blocks privilege escalation within the container
![Page 5: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/5.jpg)
Simplicity
Singularity is designed to be simple. From its build syntax, to its image format, the runtime, and compatibility with OCI and Docker, Singularity facilitates easy and
intuitive onboarding.
SIF can squashfs, encapsulate and sign Docker and OCI image formats (encryption coming soon)
Docker, OCI, etc... Singularity
![Page 6: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/6.jpg)
Reproducibility and Mobility
Singularity uses binary and immutable images, encapsulating the entire runtime stack so it is always 100% reproducible.
Host 1 Host 2
SCP, FTP, HTTP,Sylabs Cloud
![Page 7: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/7.jpg)
Designed for HPC and up!
From your laptop, to HPC compute, to the cloud and edge, Singularity
facilitates extreme mobility of compute.
Singularity provides an onramp across all resources, enterprises, and clouds.
![Page 8: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/8.jpg)
Singularity Runtime Features
Accelerated GPUs
Designed For Performance
Supports GUIs Resource Manager Agnostic
Secure
Standards Compliant
Jobs and ServicesCompatible with HPC
![Page 9: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/9.jpg)
A year Update on Singularity
![Page 10: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/10.jpg)
Challenges using/converting to Go
● CGo Interface :
○ Forking in Go. Certain syscalls should not be called in a multi threaded application.
● Following Go Standards (packaging, internal)
● Vendoring… dep, go mod, vendor/
● Packaging
○ Custom packaging tool makeit.
○ RPM
○ upcoming….DEB
![Page 11: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/11.jpg)
Wins & Gains from converting to Go
● Integration with other container projects. Easy to use packages from Kubernetes, Docker/Moby, OCI etc. No need to rebuild in Python or C.
● Concurrency model of Go.
● Go is easy to learn for developers coming from other languages.
● We used CNI rather than creating our own networking stack.
![Page 12: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/12.jpg)
Comparing Go to previous Languages (Python etc)
● Large ecosystem of packages for backend and system projects, cloud APIs.
● go test - Easy standards and framework for setting up tests.
● Easy to be productive in Go. Well designed language.
● Go is opinionated (Good and Bad). Works well but you have to follow the standards. Easy to read other codebases.
![Page 13: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/13.jpg)
Singularity Container Format Features
Guaranteed Reproducible Archival Mobile Controls Compliant
![Page 14: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/14.jpg)
What’s new in 3.0ー Cloud Service
Container Library• Like GitHub/DockerHub, Manage Container on Cloud, User can Pull/Push。Also
support Private Container。
Remote Builder• Sylabs provide Remote Builder Service on Cloud to the user who don’t have
privilege or environment.
KeyStore Service
• Can manage GPG Compatible Key
![Page 15: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/15.jpg)
Container Libraryhttps://cloud.sylabs.io/library
• Pull/Push Images• Wild card search• Star• Download
![Page 16: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/16.jpg)
• Create token - https://cloud.sylabs.io/auth• singularity push ./def.sif library://jialipassion/default/demo1.sif:latest• singularity pull demo1.sif library://jialipassion/default/demo1.sif:latest• singularity search demo1
Container LibraryDemo
![Page 17: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/17.jpg)
Remote Buildhttps://cloud.sylabs.io/builder
• Live Output• 11GB Space
![Page 18: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/18.jpg)
• Create token - https://cloud.sylabs.io/auth
• singularity build --remote library://sylabsed/examples/lolcow.sif:1.0 lolcow.def
• Build Definition File How To• https://www.sylabs.io/guides/3.0/user-guide/definition_files.html
Remote BuilderDemo
![Page 19: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/19.jpg)
KeyServicehttps://cloud.sylabs.io/keystore
• GPG Compatiable• Push/Pull
![Page 20: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/20.jpg)
• Create token - https://cloud.sylabs.io/auth• singularity keys newpair• singularity keys list• singularity keys push• singularity keys search (name/email)
KeystoreWorkflow
![Page 21: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/21.jpg)
Container Image Integrity
Singularity container runtimes are immutable, cryptographically signed, and verifiable, ensuring absolute trust of the container environment.
Note: Singularity uses no tarballs and thus no unsigned intermediate data
![Page 22: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/22.jpg)
• Using CNI Plugins, implement Network Virtualization。 https://www.sylabs.io/guides/3.0/user-guide/networking.html
• --dns • nslookup sylabs.io | grep Server• sudo singularity exec --dns 8.8.8.8 ubuntu.sif nslookup sylabs.io | grep Server
• --hostname• --net
• hostname -I• sudo singularity exec --net my_container.sif hostname -I
• --network• bridge• ptp• ipvlan• macvlan• User can also define their own network type
• --network-args• sudo singularity instance start --writable-tmpfs --net --network-args "portmap=8080:80/tcp"
docker://nginx web2• sudo singularity exec instance://web2 nginx• curl localhost:8080
Network VirtualizationCNI https://github.com/containernetworking/cni
![Page 23: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/23.jpg)
• Singularity3.0 provide a lot of security functionalities for Linux Containers• --add-caps/--drop-caps
• Ping• --allow-setuid
• enable setUID (disabled by default)• --keep-privs
• When singularity.conf is being modified、can use this flag to run in full privileges
• --security• Can use the security functions from SELinux/AppArmor/seccomp• For example、using seccomp to black list some commands, you can do like
the manual in this document https://www.sylabs.io/guides/3.0/user-guide/security_options.html
Security
![Page 24: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/24.jpg)
Native Support for cgroups
• Can use cgroups options• Sample
[memory] limit = 524288000[cpu] share = 512 period = 100000 quota = 20000 cpus = “0-1” mems = “0-1”[blockIO] weight = 1000 leftWeight = 1000[[devices]] access = “rwm” allow = false
![Page 25: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/25.jpg)
Sign/Verify
• Can sign/verify with user key• singularity sign ubuntu.sif• singularity verify ubuntu.sif• (delete key: rm ~/.singularity/sypgp/*) singularity verify ubuntu.sif
![Page 26: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/26.jpg)
Runtime Plugins
• plugin list• plugin compile• plugin install• plugin unable• plugin uninstall• plugin inspect
![Page 27: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/27.jpg)
MacOS
![Page 28: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/28.jpg)
KubernetesSingularity-CRI
![Page 29: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/29.jpg)
Meta-SchedulingCloud native HPC?
![Page 30: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/30.jpg)
NomadCloud native HTC
![Page 31: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/31.jpg)
We are hiring!
• https://www.sylabs.io/about-us/we-are-hiring/
#SUG19
![Page 32: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/32.jpg)
@SylabsIO@CarlosEArango
![Page 33: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/33.jpg)
Support slides
![Page 34: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/34.jpg)
$ cat recipe.defBootstrap: yumOSVersion: 7MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/x86_64/Include: yum
%postyum -y install vim-minimal
%runscriptexec /usr/bin/vi “$@”$ singularity build centos-vim.sif recipe.def…
Building a Container with Singularity
![Page 35: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/35.jpg)
# You can use the container as follows...$ singularity shell centos-vim.sif$ singularity exec centos-vim.sif vim testfile.txt$ singularity run centos-vim.sif testfile.txt
# Singularity containers are also executable, so you can ‘run’ them directly$ ./centos-vim.sif testfile.txt
# You can move the container onto any other Linux system with Singularity installed, and use the container directly$ scp centos-vim.sif [email protected]:$ ssh [email protected]$ ./centos-vim.sif newtestfile.txt
Example Usage of Singularity
![Page 36: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/36.jpg)
$ singularity sign centos-vim.sif Signing image: centos-vim.sifNo OpenPGP signing keys found, autogenerate? [Y/n] yEnter your name (e.g., John Doe) : GregEnter your email address (e.g., [email protected]) : [email protected] optional comment (e.g., development keys) : demokeysGenerating Entity and OpenPGP Key Pair... DoneEnter encryption passphrase :…Uploaded key successfully!Enter key passphrase: Signature created and applied to centos-vim.sif$
Cryptographically signed containers
![Page 37: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/37.jpg)
$ singularity push centos-vim.sif library://gmk/demo/centos-vim:latestINFO: Now uploading centos-vim.sif to the library 108.16 MiB / 108.16 MiB [=============================] 100.00% 13.75 MiB/s 7sINFO: Setting tag latest$
Using the Sylabs Container Library
![Page 38: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/38.jpg)
$ singularity pull library://gmk/demo/centos-vim:latest 108.16 MiB / 108.16 MiB [=============================] 100.00% 35.00 MiB/s 3s$ singularity verify centos-vim_latest.sif Verifying image: centos-vim_latest.sifINFO: key missing, searching key server for KeyID: 58D8405A30E12DE6...INFO: key retreived successfully!Store new public key F56D95BD3AFAC6FA3423911A58D8405A30E12DE6? [Y/n] yData integrity checked, authentic and signed by:
Greg (demokeys) <[email protected]>, KeyID 58D8405A30E12DE6$
Pulling and Validating a Container
![Page 39: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/39.jpg)
# Building a Singularity container (SIF) from DockerHub$ singularity build python.sif docker://python:latest…# Running a shell directly from DockerHub$ singularity shell docker://ubuntu:latestSingularity ubuntu_latest.sif:~/demo> cat /etc/lsb-release DISTRIB_ID=UbuntuDISTRIB_RELEASE=18.04DISTRIB_CODENAME=bionicDISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"Singularity ubuntu_latest.sif:~/demo> exit$ singularity exec docker://centos:latest cat /etc/redhat-releaseCentOS Linux release 7.5.1804 (Core)
Working with Docker
![Page 40: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/40.jpg)
$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu pythonPython 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import tensorflow as tf>>> x1 = tf.constant([1,2,3,4])>>> x2 = tf.constant([5,6,7,8])>>> result = tf.multiply(x1, x2)>>> print(result)Tensor("Mul:0", shape=(4,), dtype=int32)>>> exit()$
Accessing the Host’s GPU With Tensorflow
![Page 41: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/41.jpg)
SINGULARITY IMAGES
• Singularity uses a recipe file to describe the contents of the container
• Containers created from prebuilt sources at DockerHub or SingularityHub, or your own repository
• Package, Deploy, Visualize with reproducible results
BootStrap: yumOSVersion: 7MirrorURL:<http…>Include: yum
# If you want updates then uncomment#UpdateURL:<http…>
%runscript echo "This is what happens when you run the container..."
%post echo "Hello from inside the container"
yum -y install vim-minimal
![Page 42: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/42.jpg)
Singularity was designed with a different set of goals, that fits elegantly with other
packaging platforms for Business Intelligence and Big Data Analytics
ECOSYSTEM - OPPORTUNITY
VIRTUAL MACHINES
ENTERPRISE PERFORMANCE COMPUTING
Deep Learning Data Science
Artificial IntelligenceCompute Driven Analytics
ScienceNon-services
MPI Matlab
TensorflowInfiniBand
OmniPath
MICRO-SERVICE DERIVED
![Page 43: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/43.jpg)
Singularity is compatible
43
• Scheduler and Resource Manager work together to orchestrate jobs in HPC
• Traditional container daemons operate outside context of the Resource Manager
• Singularity maintains original execution context
• Jobs running within Singularity containers are just like any other jobs!
ContainedProgram / Workflow
User Privilege and PID Context
Singularity
Scheduler
Container Daemon
runs as root
User’s shell / Batch Script
ContainedProgram / Workflow
HPC Resource Manager
with resource managers
![Page 44: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/44.jpg)
Singularity / Sylabs Cloud
• The container services platform for hybrid cloud solutions• Simplify workflows, leverage signed and encrypted containers• Designed for continuous integration development practices• Brought to you by the Singularity Image Format; a single file
container package• Sylabs Cloud, your one-stop container ecosystem
ECOSYSTEM
![Page 45: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/45.jpg)
Sylabs Cloud Value added ecosystem for Singularity
• Container Build Services• On-prem/Off-prem• Enable users to build images on secure systems
• Container Library• Object store backed service for container distribution and storage• Constant container scanning, notification, and management for DevSecOps
• Cryptographic Key Management• Singularity supports cryptographically signed containers• Trusted source of public keys• Maintainer keys are downloaded and tested on demand to validate containers
CONTAINER SERVICES
![Page 46: Singularity - HPC Knowledge Portal · 2019. 8. 23. · Singularity A brief history • Invented by Greg Kurtzer at LBNL to address problems specific to HPC • Development began](https://reader033.fdocuments.in/reader033/viewer/2022060902/609edb9ee728eb29d5203db4/html5/thumbnails/46.jpg)
Sylabs and SingularityPROStable, Secure, Production ready
• Fully committed to open source
• SingularityPRO is to Singularity as RHEL is to Fedora
• LTS for several releases
• Stability through backported bug fixes and features
• Security though advanced security patches
• Full access to Sylabs cloud infrastructure and option for on-prem hosting