I already asked this question in an email to support, but I'm too impatient to wait for an answer ... also I thought other people might benefit from my asking this publicly.

Are the servers running the Linode virtual servers backed up? Meaning, if one of the dual-Xeon systems dies, or one of the RAID 1 arrays is somehow irrepairable damaged, what is Linode.com's recovery policy? If these backups are made, how often are they made?

I am very interested in the Linode concept; I currently host my own server (which I have taken from RedHat 6.0 in 1998 up to RedHat 9 today) but reliability is not great due to ISP issues, power outages, etc. Also my backup strategy leaves alot to be desired. Linode sounds like it would solve most of my problems, and if linode.com is doing backups, then my backup problem would be solved as well.

Hi,

I dont think they do backup the servers as that is usual left up to the end user.

It may be something they will offer as an addon at some point. But if they did that I would assume they would just backup the file, so the only recovery options would be a full restore.

It would be a nice option to have.

But it could be a costly option if the backup was off site, copying gigs of data.

Adam

Thank you for your response. Actually Chris responded to my support question also, and noted that there is no backup facility for the Linodes, in his words, "as of yet". Maybe it's coming in the future?

It wouldn't be a big deal for me to set up a nightly rsync of a Linodes disk, to my home PC, and given that the amount of data that changes on my server on a daily basis is limited to a small number of emails, should not use too much bandwidth either. So recovering files should a Linode system crash and burn would certainly be possible using my own mechanisms.

But, I'm more concerned about the timeliness of system recovery. I am looking for a hosted system because I want to get away from the unreliability of my current setup. I have read comments from others which indicate that I'm not alone in this. Unreliability for me comes in a few flavors:

* Power issues - since my server is in a friend's apartment, and he sometimes loses power, my system can be down for a lengthy period of time due to this. The problem is exacerbated by the fact that the system he built his Linux firewall out of doesn't have a BIOS option to come on automatically after a reboot, so unless he is there to turn it on after a power outage, the downtime is lengthened. Because Linodes are hosted in a colocation facility, I would expect power issues to be practically zero. Actually I read a post on one of these boards which demonstrates that even they are not immune, having experienced a day-long power outage earlier this month. But I expect that is a very rare occurrance, certainly much rarer than what I currently experience.

* Network issues - my server is on a DSL line, and that line goes down sometimes. It was down for 2 days recently which is what prompted me to start searching for other solutions. Once again, Linode's colocation probably has this very well covered and I could feel comfortable that network outages to my linode system would be extremely rare.

* Hardware issues - I don't have any redundancy built into my server (it's an old K6 based PC, you can see it here: http://www.ischo.com/box). From what I can read about Linode's systems, at least they are using a mirrored RAID setup, so presumably the failure of a hard drive can be tolerated, although probably with some associated downtime given that it sounds like a pretty simple IDE RAID. But how quickly would a hardware failure at Linode be handled? Is there a person available 24 hours a day 7 days a week to handle hardware failures? It's not clear to me that there is, and without that, the downtime associated with hardware failures could approach the downtime that I would experience should my server fail, since I'm not necessarily around all of the time to fix it either.

The last issue is what I am most concerned about and it is why I asked about the backups. In my vision of the ideal service, there would be contingency plans in place for any conceivable kind of hardware failure, which would involve anywhere from no downtime at all to at most a couple of hours' downtime. This would typically require having someone available to deal with hardware issues at all times, and constant monitoring without any exceptions whatsoever.

I really like the idea of Linodes and the only small flaw I think I am seeing is the fact that hardware failures could lead to outages of an indeterminite amount of time. I'd like to be proven wrong, of course, and I hope that a little bit more about the means by which hardware reliability of Linodes are or can be assured, can be disseminated in this thread ...

I initially thought I wanted Linode-supplied backups, but after thinking it over I'd rather handle them myself. The reasons are:

1. I have full control of the frequency and level (e.g. full or incremental).

2. If Linode goes down for an extended period - for any reason, e.g. DoS attack, network failures, hardware failures, fire, flood, locusts, etc. - I have everything I need to move quickly to another server.

3. Whatever the cost of user backups is, which probably should include offsite storage, has to be passed on to us users somehow. I'd rather that the base subscription price were kept low and perhaps charge for user backups as an option.

It really is pretty easy to set up rsync over ssh to mirror server data onto a home machine. If your home machine runs Windows instead of a *nix variant, you can use the rsync in Cygwin (www.cygwin.com). Automate it with cron and you have flexible, accessible, and low maintenance backups that may be better than a server vendor could provide. A HOWTO web page on using rsync for keeping multiple backup snapshots with efficient use of disk space is at http://www.mikerubel.org/computers/rsync_snapshots/

Roy

Whoops, you posted while I was composing. It looks like you're more interested in disaster recovery plans and protocols than backups per se.

Roy

From what I understand about UML, the disks in the linodes are really just large files on the drives.

Since they are just large files, transfering them between servers should be fairly easy to do, even moving them to different providers is linode was down for a long time.

What I am wondering is if a disk was created of size 1 gig, does that create a file of 1 gig on the hard drives, or does it create the file smaller and then increment it up to a size of 1 gig.

Adam

Yes those are all good points. I wouldn't mind doing my own backups, although having them done by Linode would make things that much easier, although at a cost of both money and flexibility, as you have pointed out.

I guess I am more concerned about the larger issue of reliability of service. Backups are only one part of that puzzle, and I happily concede that they are the least important part for Linodes. But I'd like to hear more about hardware issues which cause downtime and how the administrators of the Linodes server systems expect to handle them in a timely manner. Thanks!

Sorry, I meant to put my name as Bryan in on that post. Just wanted to make sure that it was attributed correctly. That post was done by me, the original poster to this thread. Thanks!

Excellent thread guys, thanks.

Power and network should not ever be an issue. ThePlanet has an excellent facility, multiple redundant power and network connections, etc. They're staffed 24/7, response times are measured in minutes, and they stock hundreds of thousands of dollars worth of parts (SuperMicro systems, hard drives, procs, etc).

If there is a single hard drive failure, the systems should remain fine until we schedule the planned maintenance. It is minimal downtime to replace a hard drive.

If there is a critical system failure, say a server overheats and smokes the motherboard, then it's a different story. The procedure at that point would be to distribute the least loaded server's Linodes to other machines, and then swap out the drives into the now-free server. For example, the Linode 192 server we have only has 3 accounts on it. I'd consider that a warm-standby. We're constantly adding new machines, so the chances of having a server with no/not many clients on it are good.

My ideal setup (something we're working towards as we grow) would be a dedicated fileserver that would serve the Linode's filesystems over a gigabit/private network. The fileservers themselves would be mirrored (perhaps with DRBD or similar). The host servers wouldn't have any Linode filesystems on them at all -- everything accessed across the network.

This accomplishes a few things -- redundancy for the hosts (if one host goes down, Linodes could be "rebooted" from a different host server), and load balancing every reboot. When you boot your Linode, our system could determine the least-loaded host server (in your plan) and instruct it to launch your processes. Our system has been designed with this in mind from the start.

There's an older discussion here:
http://www.linode.com/forums/viewtopic.php?t=191

Regarding backups, if we get enough requests (and we're almost there), a dedicated backup server will be provisioned. Starting off, this would be for in-house backups of the Linode filesystems only -- serving the purpose of non-recoverable hard drive failures.

Beyond that, limited access for customers (say around 1/4 your current disk quota) would be made available for backups. We will charge for more backup functionality -- it still has to be worked out.

To answer Adam's question, the UML filesystems are just big files on the host server -- I've created them "sparse". An empty 1GB file on the host only takes up a few K. But, as you start to use it, the filesystems will occupy more and more of the 1GB (even if you deleted everything).

With the exception of host4 which has required two reboots in the past 6 weeks, all the other servers have 100% uptime.

-Chris

I remeber that old discussion it got quite in depth.

It does seem like a good idea, except for the network traffic!!

It would offer the perfect backup solution and full redundency, it just might take a while for the linodes to boot.

Adam

I *knew* that was coming The proof will be in the pudding, I guess.

The backup server(s) will be there, regardless of how the redundancy/clustering plays out.

-Chris

So Linodes can be migrated to other hardware in the event of a failure? That's impressive! So I guess that as long as one hard drive from the RAID array of a downed system is readable, then any Linodes on that system would have minimal downtime because they would just be migrated to another server. Cool.

Someone from Linode is on pager 24x7 to deal with issues like this as they arise, right? I understand that it's a relatively new hosting service so I don't expect problems to be solved absolutely immediately, but I want to feel confident that it's better than what I have now, where my system could be down for hours or days because I might at the office or on vacation or otherwise unavailable when my system goes down.

I feel like Chris really has alot of good ideas and will continue to improve the reliability and performace of Linode as the hosting service matures. I'm so ready to sign up! And I'm gonna tell my friends!

On the subject of user backups, you might be interested in rdiff-backup. I'm using it to back up my linode's filesystem incrementally, keeping a history so that I can restore to a particular point in time. (This is not possible just using rsync, since rsync does not keep a history of the incremental changes.) It uses the same algorithm as rsync, so it's very bandwidth and storage efficient, transmitting only the changed portion of files. And of course it runs over SSH, just like rsync.

The website is at: http://rdiff-backup.stanford.edu/