A Sneak Peak of What’s New in Globus GridFTP
-
Upload
maxwell-jordan -
Category
Documents
-
view
40 -
download
1
description
Transcript of A Sneak Peak of What’s New in Globus GridFTP
![Page 1: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/1.jpg)
A Sneak Peak of What’s New in Globus GridFTP
John Bresnahan
Michael Link
Raj Kettimuthu (Presenting)
Argonne National Laboratory and
The University of Chicago
![Page 2: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/2.jpg)
GridFTP A secure, robust, fast, efficient,
standards based, widely accepted data transfer protocol
We supply a reference implementation: Server Client tools (globus-url-copy) Development Libraries
Independent implementations interoperate Fermi Lab has a home grown server that work with ours
Lots of people have developed clients independent of the Globus Project
![Page 3: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/3.jpg)
GridFTP
Two channel protocol like FTP Control Channel
Communication link (TCP) over which commands and responses flow
Low bandwidth; encrypted and integrity protected by default
Data Channel Communication link(s) over which the actual data of interest flows
High Bandwidth; authenticated by default; encryption and integrity protection optional
![Page 4: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/4.jpg)
GridFTP
DPISPI
DPISPI
CPI
![Page 5: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/5.jpg)
Striping
GridFTP offers a powerful feature called striped transfers (cluster-to-cluster transfers)
![Page 6: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/6.jpg)
Topics for discussion
Performance enhancement GridFTP over UDT
Ease of Use enhancements GridFTP over SSH GridFTP Where there’s FTP
Resource Management in GridFTP Future directions
![Page 7: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/7.jpg)
GridFTP over UDT
UDT is an application-level data transport protocol that uses UDP to transfer data
Implement its own reliability and congestion control mechanisms
Achieves good performance on high-bandwidth, high-delay networks where TCP has significant limitations
GridFTP uses Globus XIO interface to invoke network I/O operations
![Page 8: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/8.jpg)
GridFTP over UDT
XIO framework presents a standard open/close/read/write interface to many different protocol implementations including TCP, UDP, HTTP -- and now UDT
The protocol implementations are called drivers. A driver can be dynamically loaded and stacked by any
Globus XIO application.
Created an XIO driver for UDT reference implementation
Enabled GridFTP to use it as an alternate transport protocol
![Page 9: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/9.jpg)
GridFTP over UDT
Argonne to NZ Throughput in Mbit/s
Argonne to LA Throughput in Mbit/s
Iperf – 1 stream 19.7 74.5
Iperf – 8 streams 40.3 117.0
GridFTP mem TCP – 1 stream 16.4 63.8
GridFTP mem TCP – 8 streams 40.2 112.6
GridFTP disk TCP – 1 stream 16.3 59.6
GridFTP disk TCP – 8 streams 37.4 102.4
GridFTP mem UDT 179.3 396.6
GridFTP disk UDT 178.6 428.3
UDT mem 201.6 432.5
UDT disk 162.5 230.0
![Page 10: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/10.jpg)
Alternate security mechanism
GridFTP traditionally uses GSI for establishing secure connections
In some situations, preferable to use SSH security mechanism
Leverages the fact that an SSH client can remotely execute programs by forming a secure connection with SSHD
![Page 11: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/11.jpg)
GridFTP over SSH
sshd acts similar to inetd control channel is routed over ssh
globus-url-copy popens ssh ssh authenicates with sshd ssh/sshd remotely starts the GridFTP server as user
stdin/out becomes the control channel
![Page 12: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/12.jpg)
SSHFTP Interactions
sshdCPI
GridFTP Server
2811
Port 22
ROOT
USER
ssh Stdin/out
![Page 13: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/13.jpg)
GridFTP Where there’s FTP (GWFTP)
GridFTP has been in existence for some time and has proven to be quite robust and useful
Only few GridFTP clients available FTP has innumerable clients GUI Clients? Windows Clients?
![Page 14: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/14.jpg)
GWFTP
GWFTP - created to leverage the FTP clients
A proxy between FTP clients and GridFTP servers
Not secure from client to proxy Run on a trusted net (127.0.0.1) Data channel routed or direct If 3pt it is direct and secure If 2 party must route through proxy, or be insecure
![Page 15: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/15.jpg)
GWFTP (3pt)
DPISPI
DPISPIYourClient
FTP 959
(not secure)
GSI Credential
GSI Delegated
Credential
gwtftp GSI Credential
![Page 16: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/16.jpg)
GWFTP (2pt routed)
DPI
SPI
YourClient
FTP 959
(not secure)GSI Credential
gwtftp
GSI CredentialDPI
![Page 17: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/17.jpg)
GWFTP (2pt direct)
DPI
SPI
YourClient
FTP 959
(not secure)
No Security
gwtftpGSI Credential
DPI
![Page 18: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/18.jpg)
Resource management
Fork/Exec is safer service model sandboxes leaks/segfaults/security/etc
If 1 session dies service exists Transient state
We need permanent & shared state between sessions
![Page 19: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/19.jpg)
GFork
Server Host
GFork
Server
GridFTP
Plugin
GridFTP Server
Instance
Fork
GridFTP Server
InstanceGridFTP Server
Instance
State Sharing Link
Client
Inherited
Links
Control Channel Connections
Client
Client
![Page 20: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/20.jpg)
Dynamic Backends
Dynamic list of available backends (DPIs)
Frontend (SPI) listens for registration Backends register (and timeout) Select backend(s) to use for a transfer
Backend failure is not system failure
Resources can be provisioned to suit load
![Page 21: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/21.jpg)
Dynamic Backends
Frontend Host
GFork
Server
GridFTP
Plugin
Frontend
Instance
Fork Lookup available
backend
Registration
Control Connection
Backend Host
GFork
Server
GridFTP
Plugin
Backend
InstanceFork
![Page 22: A Sneak Peak of What’s New in Globus GridFTP](https://reader035.fdocuments.in/reader035/viewer/2022062422/5681367d550346895d9e0afe/html5/thumbnails/22.jpg)
Future directions
Resource Properties GridFTP server expose state via resource properties
Server load Connection limits
Act as a WS-MDS provider Firewall traversal
Simultaneous open Capability to make use of dynamic firewall port opening