So, the RFO they posted wasn't enough for you?

In my experience, Linode is very open about outage causes, and owns up pretty clearly when it's their fault (exhibit A: the shared library update debacle last year).

re exhibit A: all the servers were rebooted in less time than it took fremont to come back online.

My host came up straight away, my linode took 6 hours to come up. There were lots of people on irc in the same boat. Linodes were being fixed one host at a time manually.

The dashboard slowed and then stopped working, all the graphs in all data centres were stopped. Caker showed a graph with a load average of 150 which i think was a database server. Linode did have problems with their system, it wasn't mentioned in the status updates, that's all I'm saying.

When I got home and saw my linode was down, I checked the status and saw the power outage. I logged into the dashboard and it said my linode was running. I logged into lish and it definitely wasn't running. So I thought that's why my linode was still down. lassie isn't going to boot a dead linode when it thinks it's running.

So I opened a ticket to say the dashboard was out of sync with reality, at that time I didn't know there were still major issues. I was letting them know of a possible bug. I got a boiler plate reply about a power outage at fremont. That didn't address my issue at all, my host was up, my linode was down, the dashboard said it was running.

So I went ahead and rebooted from the dashboard, and replied to the ticket explaining the my host was up but jobs weren't running and my linode wouldn't boot. I got a reply saying there was a large queue in the backend and jobs would take a while to run. 6 hours.

Hat tip to the staff for working their bums off getting everyone back online, answering tickets, etc. -1 for not being completely honest in their status updates.

Can someone from Linode comment on this, it is concerning that no one has followed up with chesty concise post.

Its good they recovered without loss, had redundant hardware etc.., but if there were software issues (false positives) or genuine delays between host coming online to all the guests running then this points to Linode issue rather than DC issue, unless im missing something?

I guess it be great to hear something like 'whilst the cause of this incident was obviously the storm and much of the downtime was out of our hands, we have to admit that this incidient did also highlight (albiet in a small way) some issues with our software that have been further improved and enhanced with regards to status reporting and/or queue running.'

... You're replying to a thread that hasn't had another post in two months. A bit late, don't you think?

It's still a relevant question, Guspaz.

_________________
Matt Nordhoff (aka Peng on IRC)

Like always, any time we encounter something that isn't optimal we improve on it. There's not much more to say.

-Chris

Impressively vague, caker. Still, knowing you guys, I suppose it's answer enough.

_________________
Matt Nordhoff (aka Peng on IRC)