blackwd has a number of excellent points.

I also feel that the Linode staff did an excellent job!

A central communications point would be most welcome. I prefer the forum. IRC seems like a very transient form of communication. When an emergency occurs, a link to a stable and secure communications point would be most beneficial.

Jeff

Sure, forums are easier and persist conversations well. But did you actually try to use the forums during the outage? They were unusably slow/unresponsive. In other words, they weren't stable or reliable for communicating anything. IRC is decentralized and lightweight. Perhaps IRC is not the answer, but I don't think forums hosted on servers that could go down with others during an outage is the answer for reliable communications. Perhaps an email list with an offsite backup MX, but not onsite forums.

I'm gonna disagree with y'all and say this: a single centralized point of communication is bad. Case in point:

- Twitter has a less than stellar uptime. If Twitter is chosen as the central notification system, it'll be just our luck Twitter goes down during the next outage.

- IRC isn't an option for many of us, myself included. I can get away reading a forum while at my day job. IRC, not so much.

- Forums, well, we saw how slow they got during the outage.

- Email lists, they aren't quite as real time, and tend to be announce-only.

I think what it comes down to is that each communications medium has its pros and cons, and each user has different preferred communications mediums. In order to weather the next outage, Linode is really going to have to use a number of different communications mediums, in order to reach (and retain) the maximum number of customers possible.

_________________
Disclaimer: I am not an Linode staff member.

I think the best answer is a static Web page, hosted somewhere other than Linode, which can be used to get us the critical information that blackwd listed. It's unidirectional, true, but it would not go down under load. Other channels could be used to ask questions etc.

Either Linode needs to ensure that a 100% complete, timely, and consistent message is delivered to five different channels (frankly, impossible), or they need to have one official place to announce emergency information.

Oh yeah, you're doomed now that Ive got a linode. Ive got a bad case of bad-luck/byte-rot so the next outage will take down the whole intertubes. Infact you just watch! I bet all the electrons will catch fire and asteroids will impact exactly on the data centers.

I suggested an external site , thinking that's like raid-1 for websites .

You'd want to make it one that didnt share any common network carriers either (infact why even host it in the US?). The chances of linode and an external site going down are... well.. much higher now that Im here. *mooohahahaha* (bond villain type laugh!)



*nod* I agree. Im not in a position to hang out on irc much. I feel like a bit of a leech everytime i have to pop on. ;-(

I did hit irc during the outage, but if I had had answers to some basic questions I would probably just passed the info on to people I support and let the linode techs get on with it. As it was, I joined the channel and asked the same silly questions as everyone else.



I agree about different needs for different feeds for different folks, but...

The issue with this outage was that I couldnt get those answers without going to the IRC channel, so I guess we already had a "single centralized point of communication", the IRC channel. That was the _only_ place that had any really useful info.

I thought about it and answering those few questions would have satisfied me and I would have left them to get on with it. That has to make the irc channel quieter and let the techs get on with their work with less noise and nagging.

I figure that *some* other people might be like me and if they could get answers to the most 'vital' questions then a good chunk of us will just shrug, feel a little sad and wait till it's fixed. Sure, Id be sad, but I wouldnt feel clueless/helpless when someone asked me 'whats happening?' and 'how long?'.

It's really the not knowing that causes the pain. If you know it is going to be two hours, you head off and do something else. If it's going to be 5 mins you wait. Not knowing means you are stuck.

I can come to terms with the downtime, accidents happen. The larger part of the issue for me was the lack of communication. (Someone posted on the forum 'do you want them to post here or fix the problem?' and my immediate reply was 'post here _then_ fix the problem' and I still feel that way very very strongly.)

But on the other hand, by joining the irc i did get to pass on a virtual hug to the staff but I think they could have done with 1 less hug and 1 less irc user.

---

Just for a lighter finish to this post: I was once booting/installing/halting a lab of linux machines and trying to admin the print/proxy server. (Yes, two things at once, you can almost taste it....). Anyway, I typed 'halt' on a lab pc, but I was remotely connected to the print server (wrong window). To top it all off, I did the same thing again 5 mins later. I posted to the department both times saying 'what happened and why'. People had a good laugh, but I didnt get any bad karma from anyone (even though I took the net down twice).

People understand if you own your mistakes, but when they dont know whats going on, that is when you get the agro (atleast in my experience).

---

Oh my, another bloody epic post. One day I'll post something short to the forums. Really!

_________________
---
Monkeys for the win....
.... but Labradors for the snooze.