UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller...
Transcript of UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller...
![Page 1: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/1.jpg)
![Page 2: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/2.jpg)
UNICORE in the Human Brain Project
Bernd Schuller ([email protected])Jülich Supercomputing Centre
Sept 29, 2015 2015 HBP SUMMIT - Information in this presentation should be held confidential
![Page 3: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/3.jpg)
HBP Development System
HBP Molecular Dynamics
Supercomputer
HBP Supercomputer
HBP Massive Data Analytics
Supercomputer
HBP Cloud Storage
HBP High Fidelity Visualisation Systems
PRACE network
InternetInternet
ded
ica
ted
Hardware infrastructure
HBP High Fidelity Visualisation Systems
ded
ica
ted
![Page 4: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/4.jpg)
HBP Supercomputer
Systems access?
ssh / scp
● Login/password or ssh key● qsub, qstat, runjob, mpirun, ...● Setting cores, nodes, memory, ...● /usr/local/apps/myapp/bin/myapp, …● ~/mydata/2011/job123/ergebnisse.txt, ...
![Page 5: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/5.jpg)
Systems access??
How can I ...
– … use multiple, heterogeneous systems seamlessly and securely
– … manage my job input data and results?
– … across systems? Workflows?
– ... integrate HPC/data resources into applications/portals?
![Page 6: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/6.jpg)
![Page 7: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/7.jpg)
12/4/15 HPC site
BSC
HPC site
CINECA
HPC site
CSCS
HPC site
JSC
S3 storage
KIT
2. access REST APIspass OIDC token
OIDC server
1. authenticate
returns OIDC token
Unity
3. pass OIDC token
returns signed SAML
3.1 validate OIDC
Collaboratory, other portals
and applications
REST APISOAP/WS API
![Page 8: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/8.jpg)
Single sign-on
– Unity– Bridges UNICORE to HBP OIDC infrastructure– Supports REST and Web clients– Support for SOAP/WS clients (legacy)
– User management– Users are granted resources (-> review process)– User IDs and groups are mirrored to HPC sites (LDAP)– Access via UNICORE is configured automatically
– Go get an HBP account and compute time!
12/4/15
![Page 9: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/9.jpg)
Running NEST - without UNICORE
12/4/15
– Login via ssh to JUQUEEN– Manage working directory, code, input params
– Create/submit LoadLeveler script
#job_name = slns_demo#...#@bg_size = 32#@wall_clock_limit = 00:10:00
module load python3/3.4.2export TMPDIR=$WORK/tmpexport PYTHONPATH=/homeb/slns/slns007/local/opt/...
runjob --ranks-per-node 1 --exp-env ... : /bgsys/.../python3 microcircuit.py
![Page 10: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/10.jpg)
Running NEST - using UNICORE
12/4/15
– Complexity is now on hidden by UNICORE
– User can use a UNICORE Application “NEST”
– User need only invoke the application and provide relevant data
ApplicationName: NEST,
Parameters: [ NESTCODE: microcircuit.py, PARAMETERS: parameters.py, ],
Imports: [ ... ],
Resources: { Nodes: 32, Runtime: 1200 }
![Page 11: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/11.jpg)
Running NEST - using UNICORE
12/4/15
– Admin defines UNICORE Application “NEST” for JUQUEEN
<idb:IDBApplication>
<idb:ApplicationName>NEST</idb:ApplicationName>
<jsdl:POSIXApplication> <jsdl:Executable>runjob --ranks-per-node 1 --exp-env ... : .../python3</jsdl:Executable> <jsdl:Argument Type="filename">$NESTCODE?</jsdl:Argument> <jsdl:Argument Type="filename"># $PARAMETERS?</jsdl:Argument> </jsdl:POSIXApplication>
<idb:PreCommand>#@environment = COPY_ALL</idb:PreCommand> <idb:PreCommand>module load python3/3.4.2</idb:PreCommand> <idb:PreCommand>export TMPDIR=$WORK/tmp</idb:PreCommand> <idb:PreCommand>export PYTHONPATH=/usr/local/...:$PYTHONPATH</idb:PreCommand>
<idb:PostCommand>find -name *gdf | xargs zip output.zip</idb:PostCommand>
</idb:IDBApplication>
![Page 12: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/12.jpg)
Outlook – Collaboratory integration
12/4/15
– Task framework– THE way to integrate scientific computations into the
Collaboratory– Autogenerated Web UI, provenance support etc– But: currently only uses local resources
– HPC support in the Collaboratory– Via UNICORE– OIDC support and REST API are available– Job submission and management is very easy– Data management needs to be better defined
![Page 13: UNICORE in the - COnnecting REpositories · UNICORE in the Human Brain Project Bernd Schuller (b.schuller@fz-juelich.de) ... pass OIDC token OIDC server 1. authenticate returns OIDC](https://reader035.fdocuments.in/reader035/viewer/2022062505/5ede2281ad6a402d66696d2b/html5/thumbnails/13.jpg)
Summary
– Secure and easy access to HPC compute and storage resources
– UNICORE provides compute and storage abstractions. Acts as integration layer for a unified view on the underlying resources
– Allows integration of HCP/Storage into custom applications using the REST API
– HBP single sign-on via OIDC supported
For more on UNICORE: http://www.unicore.eu
12/4/15