I see, thanks. I must have disabled lassie for some reason. Did it work in this case?

Linode and Linode support are wonderful, but I can't believe power outages repeatedly knock out a world-class datacenter like HE.

From the November outage RFO:

"The Fremont facility is consulting with the UPS manufacturer to make sure the system is more robust in order to protect against similar failures in the future. We plan to follow up with them and ensure that the reliability of Linode's infrastructure meets our expectations."

Yep.

given this is 3 strikes for HE,

What steps will Linode be taking to ensure that its provider is fit to provide a Tier 1 service?

Worked for me too. I just spent an hour going through logs looking for the reason for re-start and came up blank. Came to the forums to ask a question and here's my answer

Being in this DC is starting to worry me

I had done some work on my Linode the night before and I usually sync backups to another machine at my place. Add to that I had turned off the machine for the night. Just my luck that the DC would go down and those same configuration files where lost
Oh well. That is what I get for not backing things up. Will have to rewrite them when I can again.

um, what? how were your configurations lost? the power went down, didn't see anything about any hosts blowing up.

I'm not defending HE here because they did screw up, but building redundant power is way harder than it sounds. UPS's fail as the worst times even though they pass weekly tests. Diesel generators fail even though they pass weekly tests. Switching equipment jams, things overheat. Idiots wire dual PSU servers on one power circuit then swear blind they used both. Racks of servers all start up at the same time because the BIOS random start-up delay wasn't set and management didn't pay 10 times more for staged start-up PDUs. Air conditioning gets a bit old and starts to draw more current than the specs say. And humans normally screw things up big time when they realize they don't actually have a procedure for the current situation and start panicing.

Everything that can go wrong will go wrong. Everything that can't go wrong will go wrong anyway. And every bit of equipment that solves one problem introduces another one.

I don't think it's actually possible to do better than one rackmount UPS per server sitting in the rack right next to that server. That's what EMC do with their storage arrays. It's pretty expensive and hard to manage but it's the only thing I've seen that actually works.

One rackmount UPS per power supply, not per server. Obviously with redundant power supplies.

_________________
rsk, providing useless advice on the Internet since 2005.

To paraphrase the late, great Douglas Adams... The only difference between something that can go wrong and something that go wrong is that when something that can't go wrong does go wrong, it's much harder to fix.

I don't get you, or the other dude.. haha.. if fremont can't handle a occasional storm, it might be time to look for another facility near Fremont that can. And that might sound harsh, but you'd think by now they'd have some sort of plan that doesn't involve downtime.

Additionally, I think I'd be less worried about it if it happened with all the linode centers, instead it seems like fremont gets this online issue pretty often. And apparently.. I'm not the only one who isn't cool with downtime.

If you can't stand downtime, then you have already bought additional server(s) in other physical location(s) and set up failover so that one facility going down (which happens because, obviously, nothing is perfect) doesn't screw you over. If you haven't set that up, then you don't actually care about downtime.

That's a very nice example of the "No True Scotsman" logical fallacy. You are claiming that people who "really" care about downtime would do the thing you suggest, and anyone who doesn't doesn't "really" care about downtime.

The logical fallacy is in the presumption that "caring about downtime" implies caring about downtime to the exclusion of all else. Obviously different people balance their needs differently and it's perfectly possible to care about downtime to the extent that you buy into a service with a reasonable belief that it will maintain a given level of uptime and then find later that it does not meet your expectations. This does not mean that you didn't care about downtime to begin with because you didn't choose the most extreme option for minimizing it; it just means that either your initial assessment of the service was wrong, either because you assumed too much, or because the service promised too much.

I am not sure what is the case here; but it does certainly seem that H.E. does not meet the same level up of uptime as other data centers. It seems to me that there has been a history of problems with H.E. exceeding that of other data centers, and I think it is reasonable to express concern about this.

The SANS NewsBytes had a great quote about the Amazon outage that fits perfectly for this situation.

s/cloud/Internet/ and I'm pretty much on the same page.

The Internet (and the various cloud-computing technologies within) exist on real equipment in a real world. There will be failures. It sucks when they happen, but they will, and usually not in the way you're expecting.

I'm not excusing this outage by any stretch, but, well, it'll happen again. Maybe not Fremont again (although I said that last time, didn't I?!), but electrical power is particularly tricky to do right. I'm personally a big fan of DC, but it's one of those IPv6-like chicken-and-egg problems, except actual real capex is involved and there are no dominant standards yet.

This is anticipated to change this year, though it would be unwise to get your hopes up for immediate adoption, and this is but just one failure mode of many.

(I do find it interesting that the most intricate and failure-prone utility spawns the biggest outrage when it breaks; if a failover router had blown a turboencabulator and seized the common-mode ambaphascient lunar wain shaft, taking out network reachability for a similar period of time, this thread probably would have not gone on this long.)

_________________

/* TODO: need to add signature to posts */