I'm sorry but I too have to question doing this in the middle of the day US time. This has been quite the outage.

I love linode and have recommended it to so many people over the years, but when things like this happen, it not only makes you look bad, but it makes me look bad to everyone I have referred here.

That is not even taking into account the users of my own site...

I love Linode, but this is absolutely unacceptable. Not only did your unscheduled, un-communicated upgrade break some customers, your "solution" caused abrupt hard resets for a great many who were not broken, including myself.

This is not how production systems are run. Billing credit needs to be extended to all affected customers, and this needs to never happen again.

Maintenance that has any chance of causing breakage needs to be done in off-peak hours with at least 24-hours notice to affected customers, and in case of emergencies, emails need to be sent out at least 30 minutes in advance of taking actions that you KNOW will cause problems for customer not currently affected by the problem you're trying to solve.

See if you can guess where the trouble started




Unfortunately my nodes were rebooted one by one, over the course of 2 hours which meant for far more downtime than if they had all been restarted together.

I didn't notice being offline but I did notice extremely slow response times. It seems to be much faster now.

Looking at my stats, there's a huge gap from 21:00 to 0:00 - I'm guessing that's the outage, and I just wasn't here to notice.

My linode says "Up since Oct 28, 12:00 AM".

I think my linode is in Dallas.

-=-

It would be better to do these kinds of things 4 AM eastern (1 AM Pacific).

Fortunately our setup has linode mirrors in Atlanta, Dallas, Newark and SFO and our monitoring system takes any that are down out of the lineup automatically, but the timing (non-wee-hours-of-the-AM) was awkward:

Tue Oct 27 20:01:09 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:02:19 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:03:31 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:05:03 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:06:14 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:07:28 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:08:48 EDT 2009 ERROR: 66.220.1.164 http is down!
Tue Oct 27 20:09:46 EDT 2009 ERROR: 66.220.1.164 http is down!

Stuff Happens. Still, it would have been better to have this particular Stuff Happen at 3 or 4 AM when nobody would notice ...

My server is still down as well. Unfortunately guys Virtual Servers are still shared servers and no-matter what time they do maintenance I guess there would be complaints as they have customers all over the world.

RE: status updates I guess they are too busy fixing the problem (been there myself in an office environment everyone rings you to find out whats going on so much so you don't get much time to fix the problem till you go into the server room and lock the door!).

Here's hoping everything is up soon!

awesome

JobID: 1498922 - Host initiated restart
Job Entered 01/03/1974 11:00:00 PM Status In Queue
Host Start Date Host Finish Date
Host Duration waiting on host Host Message

Totally killed my linode, way after you guys already knew there was an issue, did you not stop?

Oh newbies.. take it easy. For me this is the first time that there has been a problem(at all, as relatively minor as it is) since I signed up, in 2007(my linode is on Freemont btw). And this isn't even a real problem so it seems, just some upgrade that went awry. Over two years up, I believe that fucking beats Amazon services.

Colo companies have the exact same problem with power and network maintenance, but the difference is they tell their customers in advance and they schedule it for the least possible impact. You can't get it perfect, but there are far better times to reboot a bunch of US-based servers than late afternoon/early evening US time.

Mine are both still down and have been for almost 4 hours.

All the boot requests are just sitting in the queue.

Guess my plans for tonight are canceled...

just out of curiosity, will linode be giving refunds for downtime? my host was down 30 minutes, so i won't get refunds, but i see there are people who have been affected for hours.

http://www.linode.com/faq.cfm#what-is-your-sla

What is your SLA?

We're not going to lie to you: server maintenance, upgrades, hardware and network issues, all affect a Linode in the same ways as any other provider. What we can boast about is our commitment to resolving these issues in the quickest fashion possible. Most customers will tell you the last time they rebooted was to take advantage of a plan upgrade. 99.9% uptime, or your lost time is refunded back to your account.

_________________
http://ultramookie.com/

Who are you to tell us to take it easy? Not all of us use our Linode's for expensive shell accounts. I have clients who are pissed and ZERO warning. You take it easy when this is costing you money.

I feel for you guys, 4 hours in a long time. It seems my linode was down for less than one. Everything's back to normal here.

Keep in mind Linode has an amazing track record compared to any of their competitors. Yes, the timing was terrible and there was no warning, but these things happen in this industry.

That being said, I am a bit irked as I haven't gotten any explanation on my current situation: my servers have come back up and I can SSH in but I cannot get a shell prompt and I am unable to reboot from the dashboard.

Colo customers also pay a *premium* for that level of redundancy/reliability.

I'm not saying that the linode folks don't deserve some ribbing for this, or that customers don't deserve some credit on their accounts. Rather, just trying to provide some perspective on the issue. If you need umpteen nines of reliability, then linode probably isn't the service for you. For the rest of us, it's pretty dang good. I've been a customer since 2004, and since that time, the number of issues like this can be counted on say, 2 fingers. Pretty damn good if you ask me. While any amount of downtime sucks, this afternoon's incident surely isn't going to turn me away from Linode. Once the dust settles, I'm sure that we'll get an explanation from caker, as well as some credit on our accounts is likely as well.