OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019...
Transcript of OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019...
![Page 1: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/1.jpg)
Thursday AM, Lecture 2Brian Lin
OSG
![Page 2: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/2.jpg)
OSG User School 2019 2
HTCondorFile Transfer
HTTPProxies
StashCache
More Data
LocalStorage
![Page 3: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/3.jpg)
OSG User School 2019 3
submit server
exec server
HTCondor
submit file
executable
dir/ input
output
(exec dir)/
executable
input
output
exec server
exec server
exec server
<10MB/file, 1GB total
<1GB/file and total
![Page 4: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/4.jpg)
OSG User School 2019
• An HTC best practice!
• split large input for better throughput and less per-job data
• eliminate unnecessary data• compress and combine files
4
![Page 5: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/5.jpg)
OSG User School 2019 5
exec server
file size method of deliverywords within executable or arguments?tiny – 100MB per file HTCondor file transfer (up to 1GB total per-job)100MB – 1GB, shared download from web server (local caching)1GB - 20GB, unique or shared
StashCache (regional replication)
10 GB - TBs shared file system (local copy, local execute servers)
![Page 6: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/6.jpg)
OSG User School 2019
• Place the file onto a local, proxy-configured web server• Have HTCondor download via HTTP address
6
submit server
exec server
proxy web server
![Page 7: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/7.jpg)
OSG User School 2019
• Place the file onto a proxy-configured web server• Have HTCondor download via HTTP address
7
submit server
exec server
proxy web server file
![Page 8: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/8.jpg)
OSG User School 2019
• Place the file onto a proxy-configured web server• Have HTCondor download via HTTP address
8
submit server
exec server
proxy web server file
proxy web cache
![Page 9: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/9.jpg)
OSG User School 2019
proxy web cache
• Place the file onto a proxy-configured web server• Have HTCondor download via HTTP address
9
submit server
exec server
proxy web server
HTCondor
file
![Page 10: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/10.jpg)
OSG User School 2019
proxy web cache
• Place the file onto a proxy-configured web server• Have HTCondor download via HTTP address
10
submit server
exec server
proxy web server
HTCondor
file file
![Page 11: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/11.jpg)
OSG User School 2019
proxy web cache
• Place the file onto a proxy-configured web server• Have HTCondor download via HTTP address
11
submit server
exec server
proxy web server
HTCondor
file file
exec server
exec server
![Page 12: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/12.jpg)
OSG User School 2019
• HTCondor submit file: transfer_input_files = http://host.univ.edu/path/to/shared.tar.gz
• Anywhere (in-executable, or test download)wget http://host.univ.edu/path/to/shared.tar.gz
− in-executable: make sure to delete after un-tar or at the end of the job!!! (HTCondor thinks it’s ‘new’)
12
![Page 13: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/13.jpg)
OSG User School 2019
• Managed per-VO• Max file size: 1 GB• Local caching at OSG sites− good for shared input files, only− perfect for software and common input− need to rename changed files!!!
• Files are downloadable by ANYONE who has the specific HTTP address − Will work on 100% of OSG sites, though not all sites will have
a local cache13
![Page 14: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/14.jpg)
OSG User School 2019
• place files in $HOME/stash/public• address: http://stash.osgconnect.net/~user/shared.tar.gz
proxy web cache
14
any HTC submit
exec server
HTCondor
exec server
exec server
Stash server filefile
![Page 15: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/15.jpg)
OSG User School 2019 15
exec server
file size method of deliverywords within executable or arguments?tiny – 100MB per file HTCondor file transfer (up to 1GB total per-job)100MB – 1GB, shared download from web server (local caching)1GB - 20GB, unique or shared
StashCache (regional replication)
10 GB - TBs shared file system (local copy, local execute servers)
![Page 16: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/16.jpg)
OSG User School 2019
• regionally-cached repository managed by OSG Connect
16
![Page 17: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/17.jpg)
OSG User School 2019
• Place files in /home/username/stash/public on osgconnect.net
regional cache
17
any OSG submit
exec server
“Stash” origin file
exec server
exec server
local server
osgconnect.net
/home/username/stash/public
![Page 18: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/18.jpg)
OSG User School 2019
• Use HTCondor transfer for other files
regional cache
18
exec server
“Stash” origin
HTCondor
file
exec server
exec server
file
any OSG submit
local server
osgconnect.net
/home/username/stash/public
![Page 19: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/19.jpg)
OSG User School 2019
• Download using stashcp command (available as an OASIS software module)
regional cache
19
exec server
“Stash” origin
HTCondor
file
exec server
exec server
filestashcp
any OSG submit
local server
login.osgconnect.net
/home/username/stash/public
![Page 20: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/20.jpg)
OSG User School 2019
• Require StashCashe sites in the submit file+WantsStashCache
• Require sites with OASIS modules (for stashcp)Requirements = <OTHER REQUIREMENTS> && (HAS_MODULES =?= true)
20
![Page 21: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/21.jpg)
OSG User School 2019
#!/bin/bash
# setup:
module load stashcache
stashcp /user/username/public/file.tar.gz ./
<untar, then remove the tarball>
<job commands>
<remove all files from StashCache>
# END
21
![Page 22: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/22.jpg)
OSG User School 2019
• Available at ~90% of OSG sites• Regional caches on very fast networks− Max file size: 10 GB− shared OR unique data
• Can copy multiple files totaling >10GB• Just like HTTP proxy, change name when update files
22
![Page 23: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/23.jpg)
OSG User School 2019 23
exec server
file size method of deliverywords within executable or arguments?tiny – 100MB per file HTCondor file transfer (up to 1GB total per-job)100MB – 1GB, shared download from web server (local caching)1GB - 20GB, unique or shared
StashCache (regional replication)
10 GB - TBs shared file system (local copy, local execute servers)
![Page 24: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/24.jpg)
OSG User School 2019
• Some distributed projects with LARGE, shared datasets may have project-specific repositories that exist only on certain sites− (e.g. CMS, ATLAS, LIGO?, FIFE?, others?)− Jobs will require specific sites with local copies and use
project-specific access methods • OASIS?− Best for lots of small files per job (e.g. software)− StashCache and web proxies better for fewer larger files per job
24
![Page 25: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/25.jpg)
OSG User School 2019
• For StashCache AND web proxies:
make sure to delete data when you no longer need it in the origin!!!
• StashCache and VO-managed web proxy servers do NOT have unlimited space!− Some may regularly clean old data for you. Check with local
support.
25
![Page 26: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/26.jpg)
OSG User School 2019
• Only use these options if you MUST!!− Each comes with limitations on site accessibility and/or job
performance, and extra data management concerns
26
file size method of deliverywords within executable or arguments?tiny – 100MB per file HTCondor file transfer (up to 1GB total per-job)100MB – 1GB, shared download from web server (local caching)1GB - 20GB, unique or shared
StashCache (regional replication)
10 GB - TBs shared file system (local copy, local execute servers)
![Page 27: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/27.jpg)
OSG User School 2019
• 2.1 Using a web proxy for shared input− place the blast database on the web proxy
• 2.2 StashCache for shared input− place the blast database in StashCache
• 2.3 StashCache for unique input− convert movie files
27
![Page 28: OSG Brian Lin Thursday AM, Lecture 2 · Thursday AM, Lecture 2 Brian Lin OSG. OSG User School 2019 2 HTCondor File Transfer HTTP Proxies StashCache More Data Local Storage. OSG User](https://reader033.fdocuments.in/reader033/viewer/2022041923/5e6cae4aef5f0e5e071e10a4/html5/thumbnails/28.jpg)
OSG User School 2019
• Next: Exercises 2.1-2.3• Later: Large output and shared
filesystems
28