Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe...
-
Upload
aleesha-carter -
Category
Documents
-
view
214 -
download
0
Transcript of Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe...
![Page 1: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/1.jpg)
Scaling NVO Services to the Teragrid
Roy Williams
Conrad Steenberg
Craig Miller
Matthew Graham
Joe Jacob
Julian Bunn
![Page 2: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/2.jpg)
Desired Characteristics of NVO Services• Service oriented architecture
• Services should be easily and quickly deployable and usable on workstations or supercomputers
• Services deployed, managed, and upgraded by their developers• Service developers/deployers are trusted users
• Service developer acts as a broker between computing customer and computer center
• Service users authenticated with “graduated security”• Easy to start, but great power is possible
• Asynchrony for compute intensive jobs• Jobs submitted to batch queue• Unique sessionID may be used to monitor job & return results
• From “clicking” to “scripting”• Services may be accessed by clicking on a web page or with scripted client codes• Authentication for web clicking comes from a certificate store or fat browser• Scripted access requires a certificate (strong or weak) straight from the client
• Services as workflow components• A service user may be another service (a computer, not a human!)
![Page 3: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/3.jpg)
A “Graduated Security” Model
Web form - anonymous access, small jobsSome science....
Get NVO weak certificate - access logged, but identity not verified
More science....
Full TeraGrid account - browser accessBig-iron computing....
Scripted accessPower user
Portal-Based
![Page 4: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/4.jpg)
Traditional Grid Security
client
Show us your Certificate!I will do exactly what you want.
![Page 5: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/5.jpg)
Graduated Security
clientMay I have your Request and your Certificate?
![Page 6: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/6.jpg)
This is a US driver’s licence. In the US it proves identity strongly. It is like a strong certificate.
This is a loyalty card where I buy food.(You can put a false address on the application.)It is like a weak certificate.
This is a $50 gift card at a bookstore.It does not prove my identity in any way.It is like an anonymous certificate.
CertificatesThe Virtual Observatory as a Virtual Organization
![Page 7: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/7.jpg)
"nesssi_strong_cert_max_time" : 216000,"nesssi_weak_cert_max_time" : 3600,"nesssi_anon_max_time" : 900,"nesssi_anon_user" : ”nvo","nesssi_weak_user" : "nvo",
Graduated security• No certificate gets 15 CPU-minutes from community account
•Just switch on Javascript
• Weak certificate gets 1 CPU-hour from community account
• In exchange for registering name/email
• Strong (gridmapped) certificate gets infinity from own account
• Get this one from TeraGrid HQ
![Page 8: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/8.jpg)
service implementationweb formspython APIgraduated security
Certificatesmultiple browserscertificate chainsroot certificatesproxy certificatesproxy certificate chains2nd level proxy chainssecure https redirectionteragrid security policecaltech security policeNCSA security policechown directory ownershipNFS root-squashingpubcookie
![Page 9: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/9.jpg)
Three Interfaces
• Commandline with Python & Java• Cert or proxy in wacko place like .globus or /tmp/u509
• Fat Browser• https: and browser managed PKCS12 certs
• Thin Browser• Web Proxy works dynamically with cert authority
![Page 10: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/10.jpg)
Commandline Portal
client nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
XML-RPC
proxy
open http
buildproxy
Certificate Authority
get certificate
![Page 11: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/11.jpg)
Fat Browser Portal
Browser nesssi
node
node
node
node
Teragridcluster
certificatepolicies
queue
select useraccount
sandboxstorage
JSON-RPC
certificate
open http
Certificate Authority
load certificate
![Page 12: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/12.jpg)
Web Portal
client
certificateservices
nesssiweb portal
nesssi
node
node
node
node
web form SOAP http queue
fetchproxy
select useraccount
sandboxstorage
open http
certificatepolicies
![Page 13: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/13.jpg)
Exercise: Running a Nesssi Service
see http://us-vo.org/nesssi
![Page 14: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/14.jpg)
SessionID and Sandbox
• Identify which job we are talking about• 32 character hex string eg cb28d0753a7fec9a485981f741d425ec
• Used to monitor a running jobsessionID = remoteserver.cutout.init()msg = remoteserver.cutout.monitor(sessionID)
• Used to form URL where results appear, eg• http://dtf-test1.sdsc.teragrid.org:8080
/clarens/shell/cb/cb28d0753a7fec9a485981f741d425ec/cutouts/index.htm
• If you lose the sessionID, you lose your job
![Page 15: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/15.jpg)
DPOSS Mosaic Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)
mosaic_loc = "-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0"
session = nesssiServer.dpossMosaic.mosaic(mosaic_loc)print "Your session ID is %s." % session
msg = dbsvr.dpossMosaic.monitor(session)print msg
Repeat the monitoring
![Page 16: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/16.jpg)
![Page 17: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/17.jpg)
nesssiServer.dpossMosaic.mosaic (“-ra 49.1 -dec 60.1 -rawidth 0.5 -decwidth 0.5 -filt f -bgcorr 0”)
![Page 18: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/18.jpg)
Cutout Service
nesssiServer=nesssi.client('https://envoy.cacr.caltech.edu:8443/clarens/',debug=0)sessionID = nesssiServer.cutout.init()print "Session id is ", sessionID
# Upload locations filenesssiServer.upload_file(“inputfile.xml”, ”inputfile.xml”)
# Arguments for service, surveys to use and cutout sizeargs = "-surveys PQ:gr,PQ:gi,PQ:z1,PQ:z2,SDSS:r,SDSS:i,SDSS:z,2MASS:k,2MASS:h "args += "-size 64"
# Run servicenesssiServer.cutout.run(sessionID, args)
![Page 19: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/19.jpg)
Cutout Monitoring
![Page 20: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/20.jpg)
cutouts from Palomar-Quest, SDSS, 2MASSof sources from Veron quasar catalog
![Page 21: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/21.jpg)
![Page 22: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/22.jpg)
![Page 23: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/23.jpg)
Synoptic Coaddition service
Palomar-Quest Survey
Coverage map
Max=18
![Page 24: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/24.jpg)
Making a Service
• Developer builds script• Keyword-value pairs on command line [+uploaded files]• Sandbox location given on cmdline -- all files staged there• Should make index.htm in sandbox for progress• Make Nesssi connector for init(), upload(), run(), monitor()
• Nesssi admin installs your service• Interview first• Symlink to code• Code is cached, restart server after edit• Developer gets right to restart server (running jobs not affected)
• Service instantiations farmed out to cluster with PBS
![Page 25: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/25.jpg)
Server side code
Application example:dposs.py -dir sandbox \
-ra 123 -dec 22.7 \ -rawidth 0.4 -decwidth 0.4 \ -filt j -bgcorr 1It should:
(1) Use keyword-value arguments and uploaded files(2) Read/Write results in the given sandbox directory(3) Write a progress file in sandbox/index.htm(4) Estimate limits for anon/weak/strong certs
Service code will be symlinked from server code directoryRequires sudo server restart to see the service
![Page 26: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/26.jpg)
Client-side Javascript
<input type="button" onclick="connect_nesssi('dposs')" name="Connect" value="Connect to Nesssi">
Service nameExpect to run remote services called:
dposs.init(), dposs.run(), dposs.monitor()
![Page 27: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/27.jpg)
Client-side Javascript
<form name="Parameters"><input name="ra" value="202.4682”Etc…
function getparams() {var params = "-ra " + document.Parameters.ra.value + " " + "-rawidth " + document.Parameters.rawidth.value + " " + "-dec " + document.Parameters.dec.value + " " + "-decwidth " + document.Parameters.decwidth.value + " " + "-filt " + filt + " " + "-bgcorr " + bgCorr; return params;}
The form for the user
Developer converts the form to a string
![Page 28: Scaling NVO Services to the Teragrid Roy Williams Conrad Steenberg Craig Miller Matthew Graham Joe Jacob Julian Bunn.](https://reader031.fdocuments.in/reader031/viewer/2022013004/56649e615503460f94b5c872/html5/thumbnails/28.jpg)
Nesssi Assets
• Graduated security• Anonymous, Registered, Known
• Multiple interfaces• Fat browser, Web proxy, Scripted
• Multiple implementations• cacr.caltech.edu and sdsc.teragrid.org
• Some useful services• Hyperatlas mosaic, Cutouts, Synoptic coaddition
• Teragrid acceptance of security model• Server runs a job as somebody else• Anonymous access to TeraGrid!!