That's incorrect. EC2 does not use S3 as primary storage, it uses EBS, which is non-volatile. EBS is cheaper than S3, and unlike S3, is intended for use as a filesystem.

As expected, s3fs SUCKS and trying to use it to extend the storage of my Linode for my photo gallery just isn't working very well (aside from s3fs generally being unreliable and wonky, it also has the unfortunate behavior of fetching the entire file any time any part of it is asked for, and this interacts badly with the gallery software).

Can you give some more details on your method of remote disk storage? What vendor have you found cheap and reliable for this purpose? Have you tried SSHFS (which seems easier to set up than NFS), and can you compare it to NFS for this purpose?

sshfs is like s3fs in that it's a kind of hacky kludge that works OK for casual stuff, but not for serious stuff. It's *really* easy to use and set up, but it completely falls apart under any load, so it should be used for convenience, not as a replacement for SMB of NFS.

In my experience, sshfs has been much more reliable than s3fs. The code base is rock solid, and the performance is way better than s3fs. This probably has to do with the fact that SFTP, the underlying protocol for sshfs, was designed for filesystem-like access patterns. If I remember correctly, with SFTP/sshfs you don't have to transfer the whole file to access a few bytes or the metadata. S3 on the other hand is based on a RESTful protocol, which sucks for a filesystem.

However, as @Guspaz said, sshfs is nowhere near as robust as NFS or CIFS (Samba). Read-only access is generally OK as long as the latency is low, but try writing to the same file at the same time and the contention will get ridiculous. For example, it's possible to run a read-write SQLite database over NFS, but you don't want to do it on an sshfs volume.

It's probably OK if all you're doing is fetching old photos and sending them over the web once in a while. I'd run forum archives on an sshfs volume for months without a problem. Generating a few thumbnails might also be OK, depending on the latency and bandwidth between the hosts. But anything more and you should be looking at NFS or a local filesystem.

I don't know how Gallery organizes uploaded files, but if it's anything like /year/month/filename, it'll be very easy to transfer old photos to a remote server and keep newer photos locally, without changing any PHP code. Just mount the remote volume somewhere outside of the docroot, move old directories to the remote volume, and replace their original locations with symlinks. Do this every now and then as your local filesystem gets filled up. Gallery won't know anything is amiss. (In case it's pesky enough to complain about a symlink, just do a "mount --bind" instead.)

The primary benefit of SSHFS, of course, is that it works everywhere. Even the cheapest backup service or large-disk VPS host supports SSH. I know several reliable hosts, but since some of them are Linode's competitors, I won't post their names here. I'll PM you.

On the transparency idea...

Some magic could be thrown into the web server to redirect requests for relocated content to S3. That would cut down on the download-then-transmit tromboning and gallery2 hacking, at the expense of another brief HTTP request.

_________________

/* TODO: need to add signature to posts */

(my 1st linode post so bear with me)

what about linking assets to other storage-only services like dropbox, spideroak or ubuntuone? You can get 100gb for $10-$20. It won't be the fastest but you can't have it all.

Automate your Linode Apache so bigger file types are automatically fetched remotely, maybe add a cache on your node. Makes sense?

IMHO asking for speed + storage + cheap is asking for trouble. Cost-benefit looks ugly once you include your own wasted time.

-- p

Dropbox would probably flag your account for doing that. I presume the others would too. It's not the intended purpose of those services.

Dropbox will disable access to a file if it gets more than 10 GB/day of traffic if you have a free account, and 250GB/day if you have a paid account, but since you're not supposed to use it for web hosting like this, I'm not sure how they'd react.

I dunno; their business model is to charge for bandwidth and space. You're not eating significant CPU and as long as you're not doing anything illegal why would they care?

People have come up with some very creative uses for DropBox and though they're not officially sanctioned they've been there for a while.

SpiderOak would be in a tough spot to accuse you of specifics without admitting their "zero-knowledge" encryption is fluff... and lose their #1 differentiator over DropBox.

Anyway I came across this thread cause it's the first I saw when I joined the forums. It makes for interesting debate about possible architectures but with all due respect it's a lot of noise for stated goals that make little sense.

I can't image why you'd want to go through a web host for family photos unless you have many different personas (serial liar husband, Mormon, Muslim whose wives didn't get the memo?). Considering how few users would access the family photo album just buy the extra Linode space if it's all that important, otherwise set up a DMZed Synology at home. Definitely don't mention "'privacy" if Google, Yahoo or Amazon are part of the chain; they're doormats.

My point is that I don't enjoy getting ripped off but neither do I want the host I rely on to go under or go Mr Hyde because of the current retarded pricing race to the bottom. Anyone who's whining over pennies obviously never ran a business; what can you expect from 4$/mo plans other than -- at best -- a $4 monthly turd?

I don't want my host to "barely make it", to get bear-hug acquired like MySQL, OpenOffice, etc., -- I'll pay a fair price so they can grow and their products and service keep getting better.

I'm coming from WestHost which used to be an awesome boutique shop with great support, root access & gcc but turned into a pumpkin after they got acquired, a godaddy wannabe, which I doubt ranks very high among Linoders.

I'd blow my brains out if I computed how much time I wasted during that period.

-- p