Update in a minute...

The host's filesystem needed an fsck. Processes were getting stuck in "D" state when accessing parts of certain files.

The host is running through an fsck now. After that's done, I'll restart the Linodes. Maybe 10 more minutes or so.

-Chris

Looks like the problems were due to a failing hard drive. I'll be configuring migrations off this machine in the next day or two.

-Chris

Scary stuff, and not good news as Strandz now gets a reasonable number of visits a day. Please tell me that the chance of loosing data in these situations is zero?

Linodes use RAID 1 so the probability of data loss following failure of one drive is small but not zero. You should consider making more frequent backups until you have been migrated to a different host.

_________________
/ Peter

RAID safety sounds pretty good. I make frequent disk copies but that is more to safeguard me stuffing up the machine, which is increasingly unlikely as everything is installed and working, and something I won't be able to do as more and more (hopefully!) people visit/rely on the machine.

I do have a backup script but that only looks at configuration stuff (including debian packages of course) and other data. So if I lost the disk I'd have to rebuild the machine again (I don't know for sure that it being a debian machine this is easy, theoretically it should be), import db images, and generally remember everything again. It would take time.

Not sure I understand about "until I am migrated to a different host" (by pclissold). The machine is supposed to be becoming a 196 pretty soon. Are you saying that at the moment I'm somehow on a dodgy host and there is a way I can get to a better one? When I go onto a 196 will I be on a better one? I know I will be at the same data center.

thanks - Chris

I found this thread after noticing a ~45 minute outage on my linode last night (GMT).

How does changing hosts affect us ? Do I keep the same IP address ? What is the expected downtime ?

Thanks,
Graham.

In order to change the defective disk, the host will have to be taken out of service. Caker will arrange for all the Linodes on the affected machine to be migrated to another one. Your Linode Platform Manager grows a 'migrate' button that you click to begin the move at a time that suits you. Caker sets a time at which he will complete any remaining migrations and then take the host out of service. This is way better than having Linodes out of service during the repair. Caker's third post in this topic promises migrations "in the next day or two."

A Linode host always has Linodes of the same size on it. If you are upgrading to a bigger Linode, you will be moved from your current machine to one that hosts the appropriate size. This uses the same migration process as the move from the failing host.


Only the host number you ssh into to access LISH changes, IPs stay the same.

You have to shut down your Linode to migrate. Migration takes ten to twenty minutes. Then you reboot and you're good to go. Your disk image(s) are moved to the new machine and the configuration database is updated to reflect your new location.

_________________
/ Peter

So this means that:

1. for the time being, host40 is running off a single disk?

2. we have to have an additional outage (migration) before the problem is resolved?

Since the disk failure required a reboot and can't be solved by a hot swap, I presume the systems are using software RAID. Are future linode hosts going to have hardware RAID? It seems to me that this would be worth it, even if subscription prices went up.

Yes



And, yes.

The additional outage however will only occur when you tell it to. You aren't migrated until either you push the migrate button when caker gets it setup, or you wait too long to migrate, and your linode is forced to shutdown and migrate when caker is ready to switch out the drives on the host. You'll have plenty of notice to plan your downtime and do it at a more convenient time. It will be to your advantage to migrate sooner though so you can get back on a host with 2 working drives so you won't have to worry about the other one going out (which is not likely to happen, but the possibility is there).

I'm sorry, Tierra, I guess I didn't make my post clear. It wasn't a request for clarification it was a complaint. I guess I should have emailed linode staff directly instead of trying to rally for others who were affected.

I'm more than a little upset that this has been two months in a row with less-than-trivial failures (1. power outage, 2. high risk of data loss) -- both of which have occurred shortly after billing is completed such that customers forget about it by the time the next charge rolls around.

I think, however, that both are really things that nobody could control. A power outage that was caused by the DC, and a freak hdd failure. Yes, it's downtime, and downtime is bad -- but I also think it's imporant to remember that we have the advantage, through virtualization, of having all our data moved to a safe location in an incredibly short period of time.

_________________
Jay Faulkner
http://oldos.org

To those on host40:

Due to the recent hard drive failure on host40, Linode.com is providing an "upgrade" to a faster host server. To migrate off host40, please follow these instructions:

1) Please login to the LPM website
2) Shutdown your Linode
3) Press the Migrate button located off the Overview page
4) Submit a boot job into the queue.

The migration should take between 10-20 minutes per Linode. This migration only affects which host you access Lish. Your IP address(es), config profiles, disk images, etc. all move to the new host automatically.

You may perform the migration at your leisure. Linodes that have not been migrated by Thursday, May 19th will be migrated automatically some time that evening.

Thank you for your business,
Linode.com