Hi,

I'm evaluating the best/optimal solution to use Amazon S3:

- Duplicity
- s3fs
- s3cmd
- other

I'm testing Duplicity and it looks good!
Functional, secure, and fast ... but I would like to browse the files too , so I think I'll evaluate the others.

What do you prefer?

You might want to specify which "s3fs" you mean, there are at least 3

We've used Jungle Disk some, and it may have all the features you seek (although it's commercial software).

When considering the "Best S3" solution, I'd make a distinction between backups and online "cheaper space." Obviously duplicity's forte is backups. Perhaps s3fs-FuseOverAmazon would be stronger for space due to its local cache (and lack of a built-in backup mechanism--but you could write to it with duplicity or perhaps rdiff-backup).

Edit: Wow, a new version of duplicity released yesterday--great to see the project becoming more active again. http://duplicity.nongnu.org/

You're right!

s3fs = http://code.google.com/p/s3fs/wiki/FuseOverAmazon

At this time I was looking for opensource software.

I'm very pleasant with duplicity! Fast and Secure!
Yesterday I've tested s3fs and everything works smoothly and I can have the benefit of browsing files, but I lose security.

I use duplicity with my own wrapper for ease of use. I miss being able to simply browse through the backups but that's offset by the fact that the data is encrypted so it's safe from any possible prying eyes and you can do incremental backups. It also helps that you can restore a specific path from a backup.

At least for me, s3fs performance is pretty dismal....

$ time dd if=/dev/zero of=/s3/test.20m bs=1024 count=20480
20480+0 records in
20480+0 records out
20971520 bytes (21 MB) copied, 76.7342 s, 273 kB/s

real    1m17.972s
user    0m0.015s
sys     0m0.172s

$ time cp /s3/test.20m ~

real    1m0.458s
user    0m0.000s
sys     0m0.045s

So, 273KB/s upload, 346KB/s download. Not a problem for a backup solution, but a pain in the *** when I'm actually trying to read one of my files. 346KB/s doesn't even max out a DSL modem! No, I did not use a DSL modem for this test. It was run on my linode in Fremont which apparently has lots of bandwidth available to it. And I'm using the latest version of s3fs.

If anyone's got a nice little trick to improve this situation, please let me know.

If you were on Windows I would suggest CloudBerry Explorer freeware with its PowerShell command interface. line interface to work with Amazon S3 http://cloudberrylab.com/

Anyone serve s3fs (with cache) data from their docroot with apache or nginx? I tried this with no luck.Seems like it could be a great option as a global NAS and load balanced pool of web/app servers.

why not serve directly from s3 instead of going through your webserver ?
should save you bandwith & latency..

I like the Firefox S3 Organizer add on.

Jeff

I'm not just talking about static content. Php and entire your entire website directories so you can run the same exaxt code base from multpile web servers and not have to rsync and backup your data.

So your docroot can put to s3fs on each app server

Regardless of which provider(s) I were using to host my content, no matter how redundant their systems were, I'd still back everything up daily.

_________________
Educational resources for parents and teachers: http://www.ClassHelper.org

hybinet, did you ever manage to improve this ?

I'm looking at a similar setup with s3 (or fuse via rsync.net -- ouch, pricey), but ~ 300 kB/s will sink it before it's even begun

@mjrich

Nope. I found s3fs simply too slow to be a replacement for SSHFS or NFS. Too much latency, too small bandwidth, and too much REST overhead.

It ain't exactly cheap, either, when you factor in all the costs of upload, download, storage, and per-transaction fees.

@hybinet

Drat! Back to the other options then.