I've got a few questions for caker & co. I'm curious:

- How long into the outage before the Linode staff became aware of it?

- How long after Linode staff became aware before TP had it fully fixed?

- Did TP notify Linode about the problem, or did Linode have to notify TP about it?

- Do Linode staff receive notification if systems are down? [Monitoring system in place?]

- How come there wasn't a static website page (at perhaps a different IP, temporarily) with basic info (e.g. "TP's down, no ETR")?

Just wondering because that helped to cut down on questions when we had the big TP datacenter outage a few months ago. Also might suggest setting /topic in #linode as well.

- What does TP plan to do about their inability to keep an handle on power issues?

Are they going to hire an electrician or power engineer to perform an one-time review of how their power is distributed? Any remedial training for all NOC employees on dos/donts of power handling?

I'll wait for more information past that, though I should say that 7 hours seems a little on the excessive side for a circuit trip.

(I could see it taking a while had it been multiple circuits, like for the entire data center, but not for a single circuit)

Should be the same way, IIRC. It's just a matter of stripping the passphrase from the RSA (or DSA) private key.

http://www.openssl.org/docs/apps/rsa.html

Look at the first example in the Examples section.

The tradeoff is your usual security vs convenience/availability issue.

I have never received an email regarding any down-time. As said by others, there should be a reliable method (perhaps a host on another network) from which linode.com should notify their customers of down-time, planned, or unforeseen.



Hmmm... what's the point if the linode.com web server goes down at the same time, as it did today?

C'mon, you guys need to sort this mess out (down-time and notification reliability). The rest of your setup works beautifully and up until these two recent extended outages I've been a very happy customer; however, how on earth can we profess to offer great service to our clients if our own hosting provider is letting us down?

Don't know what to tell you. I do get the Linode emails on planned maintenances. From my saved mail folder:

292 Mar 09 Linode Customer ( 6) Linode Host Reboot Schedule - Host 2, 6,
293 Mar 09 Linode Customer ( 29) Linode Host Reboot Schedule - Host 2, 6,
329 Feb 10 Christopher S. ( 65) Linode Host Reboot Schedule - Host 1,2,6,

It looks like Linode sent mail if my host was affected for an upcoming scheduled maintenance. They probably sent other messages that didn't include my host.

These were for planned work. I don't think I ever got any mail for unplanned outages, but that's ok since I usually check the forums announcements for that.

We notified ThePlanet. ThePlanet does ping monitoring of all our hosts, with an escalation procedure to contact me if problems occur. This is another thing I'm steaming over - after the downtime last week they removed all our monitoring. I'm still waiting on a reply from them with regards to that issue. We'll also be finding a third party monitoring solution to not have all our eggs in one basket.

Power was restored minutes after we contacted ThePlanet.



We were busy getting everything back online ASAP. But, an off-site status page is in the works.


*I* plan on un-doing the changes they made last week. Basically they overloaded two power circuts and left one lightly loaded. Not the most common sense approach to distributing the load.

We're following up on every aspect of this issue.

-Chris

Because I was kind of expecting it to be handled by a human. I certainly didn't expect the system to automatically delete accounts that still have credit left on them.

Perhaps you need a "Delete my account when my credit runs out" or a "Do not renew" function?

Nonetheless, what's done is done. Nothing on that system was irreplaceable as all it provided was tertiary services, but getting service alerts so often for it was getting old fast.

Thanks for everyone's comments. We're working to get this resolved.

-Chris

If you could make a post afterwards as to what kind of measures you are taking to fix these problems, I think that would be great for all the members of linode to know about. We know your workin hard just try to get this issue sorted out, a lot of people do not want to leave this great hosting service over a freekin power issue.

_________________
James Lenhart.

Here's what we've done thus far:

Shuffled a few machines around (non-Linode hosting boxes, so no Linode downtime was required). Power draw on the three feeds in that cabinet are now within an acceptible range. We'll continue to take amperage samples over the next week to see if further action is required.

We've also corrected the monitoring and escalation issues with ThePlanet. We will now get notified (pager, email) when/if any equipment goes offline.

I'll be following up on the these items: off-site monitoring service for internal use, and an off-site status page, making a formal complaint with ThePlanet regarding all the issues we've had of late.

-Chris

I'm just a little curious, my node on host4 was never affected by any power problems as result of ThePlanet. Is that because its hooked up to diffrent equipment, or are host's 1-8 just blessed?

_________________
-- Surferdude

_________________
James Lenhart.

I appreciate all you guys have done to try and limit the downtime. However, I do think at this time, it is appropriate, that you guys start deploying some boxes in another DC, and giving some of us the option to move. I don't expect you to move all your current servers, that's nothing short of silly, but I think a third DC option would be a good step, not to mention, very warranted.

I do realize this isn't your fault, and you guys are as frustrated as some of us, but, please, for the love of god, stop giving TP your new servers and more money.

_________________
Jay Faulkner
http://oldos.org

Naturally, the problem with that is the time it takes for caker to get a facility that can accomidate the needs of Linode. Not an easy task if you ask me....

_________________
-- Surferdude

This was already decided a few weeks ago, not just because of ThePlanet screw ups of late. The plan has been for while now to have an east-coast datacenter offering. Look for an announcement in the coming weeks.

-Chris

RackSpace, I believe, has a DC in RTP area. You should go with somewhere in RTP, so I can implement the "baseball bat" idea, should they go tp on us

_________________
Jay Faulkner

http://oldos.org