I've got two linodes, both in atlanta. They've both got internal IPs and run IPSEC on said internal IPs.

What would be the best way of moving data from linode to linode here? Namely largeish files, anywhere from 100MB to 1GB.

So far, I've just been using scp from linode to linode. This works, but it also means I'm eating the CPU doing IPSEC + SSH crypto. I also may be doing quite a bit more of copying from linode to linode in the future, and invoking scp either a lot of times or with a huge amount of arguments doesn't sound all that awesome.

Seeing as I have IPSEC from linode to linode, high security protocols are not required. I've got that down. I'm leaning towards just using NFS, but as a whole I don't really like NFS due to my experiences in the past with it. It's also looking like the only real option, as I really don't want to install and configure samba instead.

Just curious what you guys collective either (1) use yourselves or (2) can come up with. :)

Both linodes are running centos.

if you need to move things right now, you can use tar and netcat:

server1$ tar cf - path_to_files | nc -l -p 12345
server2$ nc server1 12345 | tar xf -

If you want a more permanent solution, set up rsyncd and tell it to listen only on the private IPs. rsyncd doesn't do encryption, and you get the bonus of changes-only transmission if you're routinely moving the same files around.

Even faster, use ssh:

For the first copy, use tar, it's fastest:
$ tar -cpf - -C /path/to/files . | ssh user@otherlinode "tar -xvf - -C /path/to/files"

For incremental updates, use rsync over ssh, no need for rsyncd:
$ rsync -avz /path/to/files/* user@otherlinode:/path/to/files/

Doh.

That'll teach me to read the whole post. You specifically didn't want ssh...

For the best raw file transfer speeds, I usually fall back to good 'ol FTP. lftp is a great command-line FTP client that's fully scriptable, and available in most every distro I've ever used.

If these files are largely the same each time, with say 20% of the file contents changing each sync, then you really need to setup and rsyncd on the source host. This will allow rsync to do binary deltas - ie if you have a 1GB file where only 10MB of the file has changed, it will only transfer 10MB.

HTH,

Justin

We did benchmarks about the "double encryption" stuff at my old job... and we discovered it really doesn't cause a slowdown at all.

I wouldn't fix it if it ain't broke.

_________________
Jay Faulkner
http://oldos.org

I have heard system administrators at my office rave about some data transfer program named SneakerNet. It seems they use SneakerNet all the time and it works well.

James

Sneakernet at the Newark datacenter has huge latency - you have to wait while caker puts on his sneakers and drives up there.

_________________
/ Peter

That and how old it gets moving data with long and nasty ssh commands, that's the real problem here.

And sneakernet from the midwest to Atlanta is amazingly poor latency :)

I can hate on NFS as much as anyone, but for reducing typing, it's the way to go :-). If you don't mind having to move files on the destination, rsyncd isn't a bad solution.
That said, mostly I just rsync via ssh. Mostly I do NOT use the -z option; on a fast net, it doesn't buy much, and a lot of big files are already compressed.

_________________
The irony is that Bill Gates claims to be making a stable operating system and Linus Torvalds claims to be trying to take over the world.
-- seen on the net

Never underestimate the bandwidth of a trunk full of disks (to update an old joke).

Latency may suck, but bandwidth is massive!

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)