Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center...

Hepix LAL April 2001

An alternative to ftp : bbftpAn alternative to ftp : bbftp

Gilles FarracheIn2p3 Computing Center

[email protected]

2Hepix LAL April 2001 Gilles Farrache

HistoryHistory

In 1999, the members of the French Babar collaboration decided to import from SLAC to Lyon a big part of the SLAC objectivity database

First approach : via tapes Needs a lot of manpower Problems with cartridge compatibility Delay between data production at SLAC and their availability in

Lyon quite long


HistoryHistory

Second approach : via the network using standard ftp

Security problems :Username and password flying unencryptedUsername and password present in .netrc file

Performances problems (Maximum bandwidth available at this time between

SLAC and Lyon was 6 Mbps) : Throughput performance on one ftp session : 80 Kbytes/s~Seven hours for one two-gigabytes file


HistoryHistory

First idea to improve ftp performances : Start n concurrent ftp

• Total throughput reaches 3.5 Mbps But you still need no network outage for seven hours

Second idea to improve ftp performances : Cut the file in n parts Start n concurrent ftp (one for each part) Rebuild the file But you double the disk space usage at each endpoint


GoalsGoals

Improve performances on single file transfer

Optimize disk space usage

Improve security


Performance improvementsPerformance improvements

Use several TCP streams to transfer a file

Use large TCP windows (RFC 1323)

Compress the data on the fly (using gzip algorithm)



The client opens a control connection to the server

Exchange control information

Each side starts as many processes as requested, each process will send a slice of the file.



100 MB file

0

10

20

30

40

50

60

70

90

100

80

0

10

20

30

40

50

60

70

90

100

80

compress uncompress

bbftp client process bbftp server process

100 MB file



Compression plus parallel streams lead to throughput between 7 and 20 Mbps on the 6 Mbps link (often 12 Mbps with a standard Babar file)

But with higher speed links (34 Mbps), compression often reduces performance due to lack of CPU (perhaps we have to use another algorithm)


Security improvementsSecurity improvements

Standard bbftp connection method Use RSA encryption methods to transfer username and password

SSH bbftp connection method (Tim Adye’s idea) Use ssh to remotely start the server



Standard bbftp connection method

client server

connection

RSA key pair generationsend public key

Crypt user name and password using the public key

send encrypted data

Decrypt user name and password using the private key

OK or BAD



Standard bbftp connection method without password

Start a bbftpcd client which will keep the username and password in memory

Start a bbftpc client which will talk with bbftpcd and will give it the command to execute

Limitation : bbftpcd and bbftpc have to run on the same computer



SSH bbftp connection method

client

SSHD start the bbftpd process

Start a ssh client like ssh -x -l username “bbftpd -s” host

SSH client SSH daemon bbftpd process

Use the encrypted connection as control connection



SSH bbftp connection method without password

If the sshd server side allows the RSA authentication, a connection without any password is possible


BBFTP state of the artBBFTP state of the art

Current version 2.0.0 patch03 Possibility to play with :

• TCP window size• Read (Write) buffer size• Compression• Automatic directory creation• File mode• File access time • Number of streams• Remote and local umask• .bbftprc file• RFIO



Current commands :• cd• get• put• lcd• mkdir• mget• mput



Control file example :• 1 setnbstream 20 • 2 setremoteumask 022 • 3 setoption nocreatedir • 4 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1 • 5 setoption createdir • 6 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1 • 7 setnbstream 5 • 8 setrecvwinsize 1024 • 9 setoption gzip • 10 put /home/babar/jon/f2 /home/babar/bbrdist/newfiles/f2



BBFTP home page : http://ccweb.in2p3.fr/bbftp/ BBFTP discussion list : [email protected]

Archives on : http://ccmail.in2p3.fr/archives/bbftp-l.html Current experiments or sites using it :

PHENIX

ALICE (project)

Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center...

Documents

Transcript of Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center...