Hi,

I have an application which uses an oracle database to track over 1,000,000 image and video files which need to be synchronised between 5 servers. Currently I'm using ssh and scp via perl's system() call but this causes several connections to be made for each file transfered, which is pretty ugly and not really quick (a lot of time is lost setting up each connection). The script has to be able to create and delete directory hierarchies, transfer and delete files and also handle permissions correctly. The applications that access the files use the same database to determine which files exist on a server, so it's also imperative that all error conditions are handled appropriately. rsync, rdist, rdiff and scp are all of no use in this scenario :-(

So I'm now trying Net::SFTP. First, to install it I ended up having to install 25 other modules and libraries (Math::*, Digest::*, Crypt::* ...). Finally, I'm able to at least use the psftp script included in the eg/ directory. It seems to work, but it can't transfer more than 120 KB/s. Manually using sftp for the same files over the same connection pumps 7MB/s (5 times faster, which is why I though Net::SFTP might be a good option).

Is the psftp script a representative example of Net::SFTP performance? My hope was to replace my existing code with Net::SFTP and manage these files and directories via a single connection to each server. Sadly, it doesn't look like I'm going to gain much by doing so.

I'm wondering if anyone has tried to glue perl with the OpenSSH's sftp code? (Does OpenSSH even have a library? All I can find is libssl). Would it then be enough to install just sftp (or have the sftp sources next to the Net::SFTP build directory for compiling)?

Does anyone have any ideas?

Thanks in advance,

Steve

Hi Salva,

thanks for the tip. I just tried version 0.90_04 (by hacking a copy of psftp) and it looks promising. put() seems to work fine, transferring about 6-7MB/s (very close to full speed!) ls() and get(), however, don't seem to do much (get() just gives me an empty file). I'm not sure how the path cleansing routines are supposed to work either.

Do you have an idea of roughly how much effort would be required to get Net::SFTP::Foreign stable? I noticed some differences to the Net::SFTP API (ie: ls() doesn't take a code reference for callbacks). Do you have an API design in mind, so that I might be able to work toward a common goal and send you patches?

I'm not sure if I'll be able to spend more than a day or two on it, but if the chances of success are good, I'd be happy to do a bit of mutual back-scratching ;-)

Thanks,

Steve

short update: just found the psftp script delivered with Net::SFTP::Foreign (which uses the ::Compat subclass).

ls() works now, but get() throws an error ('dontsave' is not recognized, Foreign.pm's get() only expects '_dont_save').

Steve