That totally sucked. So what if they were marked down, they *were* running just fine until being rebooted...

Darn, too bad I queued up a shutdown, a startup and then another shutdown...

My linode in Dallas was down and rebooted. But now for some reason my website is no longer responding, which is running magento.

So, I have tried to see if I can just access or bring up a file and still not able to get anything. Anyone else seeing this type of problem?

Once again, we sincerely apologize for these issues. All system administrators are continuing to work to restore full service for all affected customers. Support tickets are being processed as rapidly as possible. Thank you for your patience.

Aha! So you're the one to blame for all this.

I've been a customer for over 4 years, but never posted here before. Before this, my downtime was approximately 0 (perhaps one reboot the whole time) and a few slowdowns from the occasional DDos to the DC. That's quite remarkable for $20/month. This is definitely the worst outage I've seen here, if judging by metrics such as duration and number of hosts affected. One of my hosts is still down. But keep it in perspective people:

a) This is not a frequent occurrence.

b) I'm sure they like it even less than you do. Reputation is important to small business like this.

c) I'm sure they will take steps to make sure it doesn't happen again. However, the nature of a shared virtual hosting environment with geographic (e.g., timezone) diversity means that unscheduled problems (aren't they all?) will always happen at a bad time for somebody. And they have to be able to do scheduled non-reboot service to hosts without coordinating windows with 40 people. As for notification of a patch to a host, sure, they could do that. But it still wouldn't have stopped this. Would you rather them patch the running host or bring down 40 VMs unnecessarily? Clearly, they've done this kind of maintenance before and it worked fine except for this time. If it were isolated to one host no more than 40 people would notice, but this was widespread.

d) They are busy fixing the problem; posting "we're working on it" messages to the forums is counter-productive. They've said just that on IRC.

e) Linode.com clearly states one 9 of uptime. In my experience, it exceeds this SLA. But if you want five 9s, you're going to have to pay more for it. If you're running a revenue generating operation that's that important (e.g., amount of money at stake per minute downtime) then you have either a backup/business continuity plan involving another provider or are paying to colocate your own racks, and even then you still have a backup plan with another provider. In other words, you're spending more money somewhere.

f) Other providers (EC2?) have unscheduled downtime / outages / capacity problems all the time.

All this doesn't make it OK. It's frustrating to have a host go down. But I think it's hard to find as reliable a host for the money.

One other observation, my original linode on a UML host has been rock solid. Anecdotally, I think UML is more stable than Xen. But Xen is faster ... go figure.

What would you guess your ETA is, for solving this problem for all hosts? How many hosts have you fixed and how many are left? It's been five hours downtime for one of my linodes now, and it's still down.

The guy who did this (or the boss who ordered him, if that was the case) should get punished by being forced to work 4 weekend-days without pay, and linodes with more than one hour downtime should get one month of service credited to their account.

That way the technician/boss would think twice before 1. not notifying customers of the potentially risky upgrade well in advance, and 2. not testing the update on a few hosts before pushing the update onto all (?) of your hosts at once.
And the company as a whole would get a clear financial incentive not to repeat such foolishness in the future.

That said, I will not move to a different provider just because of todays downtime. I just won't recommend Linode as enthusiastically any more. You're still better than any other alternative I've heard of.

For the future:

Let's say you have 1000 hosts you wish to update. You can never be sure nothing will break. You should therefore test your update on a test-server. If that works you should try it on a live production server. If that works, you should try it on two additional production servers at once. If that works you should try it on 4 more servers at once. If that works, try it on 8, 16, 32 and so on. After 11 tests you would have upgraded 1024 servers. Testing to see if everything is ok before proceeding to update even more servers, eleven times, is a reasonable "waste of time" considering the impact one undiscovered error would have for so many people. If you do upgrades on just a few servers at a time (as suggested above in this message), any problems you miss, we customers will catch because we are so many people.

And please create a mailinglist anyone could join if we want updates on planned upgrades and potential problems, progress reports and so on. Maybe one list per datacenter? I only want to know about any updates you do in the Atlanda DC for example.

Anyway, good luck fixing today's problems. It's 03:26 in Sweden now and I'm hitting the sack.[/b]

My server looks like running. But I can't connect by SSH, even ping doesn't work. Already more than 5 hours

I have logged in by LISH and I see that my kernel is ubuntu, why is that hell - my server is running on Gentoo

If you are still experiencing issues, please join #linode-is-broken on irc.oftc.net and I'll help you first-come-first-serve. We're working through these one by one, and do appreciate your patience.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.

Well I came to this thread annoyed and looking for answers.

I didn't necessarily find any answers, but I'm now more annoyed at the customers posting in this thread than I am at Linode.

Thank God I am not the underling or child of any of the perfectionists posting in here. May He have mercy on anyone who has to live up to your standards.

To Linode: I've been with a lot of companies since 2002 and your reliability has been the best I've dealt with to date. Going forward, I would appreciate a little more emphasis on communication as we have people to communicate with as well. It is difficult to do that when we have little or no information to go on. Please consider that - but otherwise good work.

Another tool who clearly only uses his Linode server as an expensive shell account.

I love that the internet gives people like you a place to be "tough"

And you the place to preach to others about what is acceptable to everyone else.

C'mon folks, no need to get personal here.

I was down for a good part of the evening too, but to be fair, this is maybe the second real outage (other than brief network issues, etc) I've had in what... 5 years? That's a pretty good track record to me.

I don't like it either, but these things happen. That's simply the reality of it.

_________________
----
Ed/Bones.

Wow my Linode went down for around seven hours today. Honestly it's pretty bad when people who depend on your server being up are angry and are demanding and explanation. What's worse is not being able to provide an explanation.

Linode guarantees 99.9% uptime which means that there's about nine hours a year when you can expect it to go offline. That's fine. What is not okay is having a major outage (all three of my servers went down) in the middle of the day and more importantly, not being able to provide detailed information about the outage that I can pass on to my clients.

This whole situation could have been a lot smoother if some administrator had posted perhaps every half an hour with something along the lines of,

These nodes are down. We expect to bring these nodes up in the next half hour. We expect to bring these nodes up in the next hour. And we expect to bring these nodes up in the next two hours. We expect to have the entire system back online within eight hours.

Not only would I have an ETA to give to the people who depend on this server, but I also wouldn't have to check back every five minutes to ascertain whether or not my servers were operable.