notsoluckycharm: What are you using to mount S3 storage on your Linode?

You can use the FUSE module (s3fs) but the problem is S3 doesn't provide a real file system, just a basic file/folder structure, and uploading to S3 from outside of the EC2 cloud is very slow (sometimes as bad as 60 kB/s from my linode).

This probably is fairly easy to circumvent. Simply rent the smallest, cheapest EC2 node, which is about $9.61/mth base cost with no bandwidth or storage, and mount S3 from there. Then your Linode communicates with your EC2 storage node rather than S3 directly.

Or better yet, instead of mounting S3 on your EC2 instance, mount EBS volumes on your EC2 instance and then mount that on your linode. Unlike S3, EBS is actually intended to be mounted as a filesystem, since it's block storage. EBS is also generally cheaper than S3 because it's $0.10 per GB and no data transfer fees (since it's internal), only for IO.

Let's say you want to mount a 1TB filesystem on your Linode. Let's calculate the total costs, including an average of 10GB read and 10GB write per month, and a million IO per month.

EC2 reserved micro instance: $54 + 8760 * $0.007 = $115.32
1000GB EBS storage: $0.10 * 1000 * 12 = $1200
10GB read per month: (10-1) * $0.12 * 12 = $12.96
10GB write per month: 10 * $0.10 * 12 = $12.00
1 million IO per month: $0.10 * 12 = $1.20
Total annual cost: $1341.48
Total monthly cost: $111.79
Total cost per gigabyte month: $0.11179

So there you have it, you can get bulk block storage on your linode for under twelve cents per gigabyte.

Guspaz, it's a decent and workable solution, but it's still a pretty big pain. Not only is there more latency when accessing data (and slower transfer speed), but you have to pay for bandwidth on both EC2 (EC2 has free incoming bandwidth but not outgoing) and Linode.

It really would be nice if Linode would offer more options for disk space. DreamHost, for example, uses NFS shares for all storage, with the storage in a different machine than the one doing the work. Granted, most of what they do is shared hosting, but they also do it with their VPSes. Something similar could work for Linode.

Even inexpensive lower quality storage (no RAID, consumer hard drives) would be great. You can get a couple TB for less than $100 with consumer hardware. As long as it's made clear that backups aren't made of data, and that it could disappear at any time, this could be very useful for many people (although not as useful as good, backed-up storage, it'd be a start).

Obviously you haven't bought a SATA hard drive lately. You'd be lucky if you scored a 320G SATAII HD for much under $100 these days.

Thailand, flood, under water - Google it.

The point was simply that cheap consumer hardware could be used instead of expensive server-grade hard drives.

I don't know the current situation over at DreamHost, but they have had all sorts of fun problems with their NFS servers, and switched at least some users to local storage a couple years ago.

_________________
Matt Nordhoff (aka Peng on IRC)

Probably not without deploying an additional model of server chassis and stocking another type of hard drive on-site. And this isn't going to be a small-scale thing, either: 1 TB only provides 20 GB for each of 50 customers.

I'm not saying it's not possible, I'm just saying it's probably not going to happen with a couple NASes from NewEgg. Not with tens of thousands of users hitting it.

(As an aside, I'm currently doing a bit of a Windows-based project on EC2/EBS. Holy damn is it slow. Feels like NFS over a DSL line. Linode has spoiled me a bit.)

I bought two 3TB WD Green drives for $140 CAD maybe two weeks ago. I was trying to buy five, they only shipped two, now I'm stuck with them.

The same drives at the same store currently sell for $370 CAD.

Ah, here's a picture I was looking for earlier:


^--- 700 TB, or about enough for 35000 users at 20 GB each. We'll say that's sufficient.

WARNING! WARNING! WARNING!
HOOPYCAT IS APPLYING MATHEMATICS TO BUSINESS STUFF


That's about as dense as I could find in a recent DSW. Based on my math (each server brings in 50*$20/mo, 40 servers per rack, 3 racks), the physical space is worth $0.17/GB/mo, fully loaded. (I don't think this is an unreasonable proposition; if the proposal were to replace one server with a non-revenue-generating thing, like a shelf or something, it would have to justify the loss of $1000/mo in revenue.)

Tough to speculate about hardware cost, but the DSW I linked to started with $1,000,000 for 100 TB of Really Fscking Fast SAN, and the 700 TB array is used by a television network. So, while it's 7x as much space, the performance requirements are much, much lower. Only data I can find how much TB costs isn't useful, so we'll say that's $1,000,000.

To pay that back in one year, it'd have to bring in $0.12/GB/mo, again assuming 100% occupancy. So that's $0.29/GB/mo. If it's only half-full, we're looking at $0.58/GB/mo, before maintenance costs, service contracts, staff, support, platform integration, and profit.

This actually wasn't where I was expecting the math to go.

Something like the BackBlaze hardware might be more likeable. This has the advantage of being much less expensive hardware-wise, but is much less reliable (by design) and requires much more work to make it go. So, at $4000/mo worth of space and $8000 per 58 TB (rounding the price up and using the actual RAID6 capacity), the same math would give 2900 users at 20 GB each, at $0.07/GB/mo for physical space, $0.01/GB/mo for 1-year payback, or $0.08/GB/mo for 100% utilization (or $0.16/GB/mo at half-full). That's getting closer, but it's going to be harder to implement than just "plug into LAN and push button", and even before the other costs, it's going to cost more than S3.

My assumptions are dubious, my math probably faulty. But, no matter which way you cut it, it's going to be more than just hanging a couple 1.5 TB drives out of the back of a server with some duct tape. There are hundreds of servers handling tens of thousands of customers in every single datacenter (maybe not Tokyo yet). Just the network traffic alone will be crazy. If we assume --- no, nevermind, I'm not going to do any more math here.

Well, if the front panel lights did the "cylon" thing, that'd totally be worth it.

Hoopy... do you hate Linode and all of us?
No... really?
Recommending EMC stuff?
They're the Oracle of storage... same way "enterprisey", same way "popular because managers decide to use it", just as expensive... and just as painful.

_________________
rsk, providing useless advice on the Internet since 2005.

I needed to warm up with big numbers before diving into the smaller numbers. Scientific notation sometimes gives me a headache if I eat it too fast.

The highest density I've ever seen (and I'm not making comments about appropriateness or reliability) was the Sun Fire X4500 (and later X4540). Sun discontinued them, but they had rather insane density.

They were 4U boxes, with a decent server component (dual processor quad core Opteron 2000 series with four GigE ports and up to 64GB of RAM. But the interesting part was that they mounted 48 hard drives in a 4U chassis. At the time, drives were smaller, but if you used the largest drives on the market today (4TB) and put ten servers in a rack, you'd have just a hair shy of 2 petabytes per standard rack.

Of course, that line of servers was discontinued in 2010, RAID-oriented drives only go up to 3TB, and I remember people being concerned about cooling that many drives in that small a space. But the density those things achieved was insane.

For funsies, I just priced out a SoftLayer QuantaStor server. (SoftLayer is where Linode colos in Dallas, which makes me find them interesting.)

The 24-disk server, with baseline hardware (which probably isn't a good idea) and 22 2 TB SATA II disks (the other 2 disks are for the OS) at $60/month each, you get 44 TB of space for $2829/month, or $0.063/GB/month. Halving that for RAID, it's 22 TB or $0.126/GB.

Using 600 GB 15k SAS drives at $150/month apiece, that's $4809/month for almost 13 TB, which is $0.364/GB/month. Halving it for RAID, it's $0.729/GB/month. That's...not good, and explains a lot about Linode's current prices.

(IANA mathematician, and I assumed 1024 GB TBs and 100% usable space. Reality would be a bit worse.)

Edit: Spelling.

_________________
Matt Nordhoff (aka Peng on IRC)