This morning, I noticed I was unable to access my website hosted on Linode - just received this message.

Quote:

We experienced a RAID card failure on your Linode's host, dallas653. Despite our best efforts, we were unable to recover your Linode disk images from dallas653 due to an internal issue and unfortunately all data was lost.

I'm seriously disappointed with Linode. The basic tenent of data security is to have content in at least two places. I guess I thought that there was some level of redundancy in their VPS service. It's all too common that RAID hardware fails or disks can't be rebuilt. Even without me purchasing their backup service - I thought they had some level of backup in case of their "oops".

Unfortunately, we just deployed a new website and used this server for both a development and production environment. If I'm lucky, I may be able to piecemeal the site back together from local sources, but this may not go so well. First, I have to rebuild my server.

Linode doesn't make any claims about backing up your data, unless you sign up for the Backup Service. The very existence of a Backup Service makes it pretty clear that if you aren't using it, you don't have backups.

RAID card failures that lead to data loss are pretty rare, but they do happen, Linode makes no secret of the fact that hardware is not magic, and as always it's your responsibility to do your due diligence.

Sounds like you had some bad luck, but based on your description it doesn't appear that Linode behaved poorly. This wasn't "their oops", this was a hardware issue.

Backups are good for the soul.

- Les

You should always have your own backups kept in a secure offline location (preferably multiple locations). If you don't you're just asking for trouble. For future reference I suggest duplicity.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Sorry you had to go through this.

I can count on one hand the number of times we've lost an entire host in the 12 years we've been doing this. It's rare, but it can and does happen.

-Chris

Nightly rsync's are your friend.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

I will admit, I'm pretty new to using a VPS - this Linode is my first. I'm now learning that cloud hosting is by design fragile and that nodes should be assumed to be disposable. I was initially under the impression that inside the datacenter, my data was on more than one piece of hardware.

I started with Linode because of good reviews, but will be switching my company over to Digital Ocean for a few reasons:

[*] Linode support was unable to offer me any useful compensation for the loss of my data - such as a period of free backup service.
[*] Digital Ocean has really quick droplet deployment and was able to get me a working LAMP setup in only a few minutes
[*] I am also able to get monthly service + backup on DO for less than my service alone on Linode.

On top of that, I will be responsible for making my own backups with Cron/Rsync, Crashplan, or another solution.

Hi radionick, it totally sucks that you lost your data due to the failure on the Linode host. But, as caker said earlier, this kind of situation is rare.

I wouldn't say VPS services are fragile as such - no more so than a dedicated server - there are similar sorts of hardware issues involve. I'm not a long time Linode user, but I have had a couple of VPSs here for >2 years now, and not a single instance of any sort of data loss. I don't consider that fragile. When I used to have dedicated servers, I'd be totally responsible for my hardware (inc RAID) and backups. With a VPS, I'm only responsible for backups. I do take 4 hourly incremental backups, to two separate locations.

I'm not going to tell you to do backups, you know that already, you just didn't get it setup on this box it appears.

Regarding Digital Ocean, I have no idea if they are good or bad. If you think their 'droplet' setup is easy, have you taken a look at Linode's stackscripts to compare?

It's good to know I'm in the minority here.

A few steps I've taken to make sure this doesn't happen again:

1) UpdraftPlus Wordpress Plugin to make automatic site backups to Dropbox
2) Cloudflare CDN to keep serving static copies of my site if there are any outages
3) Using Digital Ocean's VPS backup plan

For additional protection, I may backup the whole server using CrashPlan, but for now the server is only hosting a single website.

I was one of the people who got hit with this too. I was able to salvage most of the important stuff because I had all my code in offsite git repositories, but my databases are all gone.

I had full backups of all my stuff — but on the same disk. It has been on my to-do list to improve my backup "system" and get all my important files in three separate locations. I didn't get to setting up a true backup system in time. I screwed up. Because in 15+ years of running dedicated servers and VPSs, I'd had never had a full data loss before, and other things were always more pressing.

As Jamie Zawinski would say: RAID is not a backup solution.

As Saul Goodman might say: "Backups. They're like health insurance. You hope you never need them, but man, oh man, not having them? No!"

Back up everything, offsite. Set up Crashplan, do an rsync to your computer, set up git hooks to keep your database in version control and fully-backed up, or whatever it is you need to do. But don't delay.

Our production server has two tiers of backup schemes in place:

- Linode backups
- Nightly rsync stored as incremental ZFS snapshots

And if all else fails, we've at least got copies of the various code and content on the developer machines along with occasional manual backups. I think we're pretty good with the two main backup setups though. The first option would let us get back up and running within a few minutes, the second option within a few hours. This was prompted after we got bit ourselves, when our previous fly-by-night VPS operator basically disappeared without a trace. Rebuilding from dev copies of code/content and slightly out of date manual database snapshots got us back up and running, but it was painful, and we've taken the steps I've mentioned to ensure it never happens again.