Linode Forum

[ArbitraryConstant]

A company having the people and spare hardware to recover from a truly spectacular outage like this isn't something I take for granted.

Cheers.

[pclissold]

+1. Judging from the time it took to come back up, my primary mail server must have been on one of the hosts that got damaged during the outage. Really satisfying to have the problem just fixed in a few hours using redundant spare gear, instead of having to wait for repairs or shipment of replacement gear, as has happened to me at other providers.

Well done, Linode team.

[JasonTokoph]

While its nice that the pseudo redundancy was there, I'm still disappointed that I had to learn of the outage by searching twitter for "linode". No email, phone call, or a minimum @linode tweet. Unless I'm blind, there isn't a link to the status page on the main linode.com site.

[bryantrv]

I use the rss feed from http://status.linode.com/
I have to say- every other host I have ever used would have had, at minimum, 48 hours of downtime from this. Having everything back up and running in 6 hours after a catastrophic failure is impressive.

[vonskippy]

JasonTokoph wrote: No phone call
Where's Miss Hathaway when you need her.

I think it's pretty naive to think that a low cost VPS service is going to call you personally with every hiccup.

Look into monitoring apps (locally run or outsourced).

[db3l]

JasonTokoph wrote: While its nice that the pseudo redundancy was there, I'm still disappointed that I had to learn of the outage by searching twitter for "linode". No email, phone call, or a minimum @linode tweet. Unless I'm blind, there isn't a link to the status page on the main linode.com site.
Yeah, with something like this where the affected Linodes should be pretty clearly identified, having an email go out to their owners seems like it would be a good feature, providing it was internally simple enough so it didn't distract staff from actually addressing the problem.

With that said, if you need to be aware of outages for your node, I'd strongly suggest doing some monitoring (either yourself or with any of the free and/or paid online services). There are a number of problems - especially network related - that can arise for which Linode can't necessarily know for sure which nodes are affected, not to mention more localized issues. I know for my Linodes that I need to stay up, I monitor all the services I care about myself, and it's always my own monitoring that triggers me to go look at status.linode.com to see if it's a more general problem.

I'd also suggest throwing in a boot time cron entry on your Linode to spit out a message to you any time it boots, just as a head's up.

-- David

[Jay3ld]

I hope they really look into why the UPS system failed. I was shocked to hear a power outage would take down a hosting company. I can understand now that I see the UPS failed. Still a shock that the UPS systems failed says to me that something failed along the lines of making sure they where tested to ensure they worked.

Glad to hear they had hot spares ready to go for servers and got things fixed up for clients.

[hoopycat]

Datacenters exist in the real world and are built by humans. They can, and will, fail in unanticipated ways. That's life. Don't be surprised when it happens, or think that it won't happen just because of a company's reputation, a facility's history, etc. It will.

From my perspective, the next few days/weeks are the interesting part... figuring out the causes, the contributing factors, and all that stuff. I live for the RFO. :-)

[JasonTokoph]

db3l wrote:
With that said, if you need to be aware of outages for your node, I'd strongly suggest doing some monitoring (either yourself or with any of the free and/or paid online services).

Don't get me wrong. I have monitors in place for all of my systems, Linodes or otherwise. These monitors kicked off alerts that I needed. The issue is that I had no idea why the alerts were being generated.

Also, while I didn't expect a phone call, I did list it along with other forms of communication as I didn't receive anything.

Its even worse that I have yet to receive any form of communication from Linode regarding the outage. Fine, use 100% of your resources to fix the issue, but once its fixed, affected customers should receive some sort of notification.

I'll be happy when a post mortem is in my inbox along with an apology for lack of communication.

[vonskippy]

JasonTokoph wrote: I'll be happy when a post mortem is in my inbox along with an apology for lack of communication.
You're kidding right?

They have a SLA, which they didn't break.

They have a Status page - that pretty much tells you what's going on.

They sell a UNMANAGED service.

Unless you have a few DOZEN HIGH END VPS's that were effected, I think you're vastly over reacting.

They were down, they fixed it, they reported it, it's a rare event, get over it already.

[db3l]

JasonTokoph wrote: Don't get me wrong. I have monitors in place for all of my systems, Linodes or otherwise. These monitors kicked off alerts that I needed. The issue is that I had no idea why the alerts were being generated.
Oh, sorry - I got the wrong impression from your earlier phrasing, which made it sound like you didn't have anything yourself to let you notice the outage even on your own Linodes.

For what it's worth, when my monitors fire, and after the obvious stuff like issues on the monitor's own connection or the network, first thing I do is check status.linode.com, and then if there is nothing there, get on IRC and lurk a bit or ask about it. I'm not a tweeter, so I rarely look there. Typically one of the two will give me the low down fairly quickly (IRC in particular gets busier during the larger outages, and at a minimum you can probably find out if you aren't alone), but I'm willing to accept some latency also since I may have actually noticed things myself first or the issue may be small enough not to prompt a status update.

Also, the good news is that now you know about status.linode.com for the future. I agree some additional links to it couldn't hurt, though it's a fairly common naming convention among providers in the last few years, so not a bad name to try by default. Also, depending on the outage, the main Linode site may not itself respond - e.g., if the recent outage had been in Texas, so depending on using a link from the main site won't always work (status.linode.com is hosted elsewhere).

Quote: I'll be happy when a post mortem is in my inbox along with an apology for lack of communication.
Wouldn't necessarily wait for the inbox, but I'm sure some summary information will be posted or made available when possible. For example http://www.linode.com/forums/viewtopic.php?p=24778#24778 when there was an outage affecting a decent number of machines in the Newark DC last year.

-- David

[quanin]

Does anyone even use email for service updates anymore?

[OverlordQ]

Jay3ld wrote: I hope they really look into why the UPS system failed. I was shocked to hear a power outage would take down a hosting company.

You must be new to the internet

[Internat]

email is useless as a status message medium.. esepcially given my frontend client access server for mail was one of the nodes that went down.. Such is life though :)

[pclissold]

bryantrv wrote: I use the rss feed from http://status.linode.com/

+1. This is hosted at typepad.com, in a different datacenter to all Linodes.