Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Forum locked  This topic is locked, you cannot edit posts or make further replies.
Author Message
 Post subject: Outage: fremont86
PostPosted: Mon Oct 12, 2009 11:11 pm 
Offline
Senior Member

Joined: Sat Mar 28, 2009 4:23 pm
Posts: 415
Website: http://jedsmith.org/
Location: Out of his depth and job-hopping without a clue about network security fundamentals
fremont86 ran into an issue with its RAID controller. We are rebooting the host, and Linodes will be coming back up shortly.

We apologize for the inconvenience.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 10:55 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Another reboot. We think we've identified the problem being a faulty drive which is causing the RAID unit trouble. We'll be replacing this shortly. Linodes should be coming back on line shortly.

We apologize for the inconvenience.

-Chris


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 11:25 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Additional problems have been encountered. We're working on a resolution. Updates in a bit.

-Chris


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 12:06 pm 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
We're working on configuring migrations off this machine now, which we will execute one by one. Please stand by for further updates.

Again, we apologize for this inconvenience.


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 12:18 pm 
Offline
Senior Member

Joined: Thu Dec 04, 2008 10:55 am
Posts: 57
Location: New Jersey
I'm getting some serious lag trying to SSH into my node on fremont87. Is this related?


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 12:37 pm 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
spearson wrote:
I'm getting some serious lag trying to SSH into my node on fremont87. Is this related?

No.

We're still working through the problems with fremont86. Hopefully I'll have some news shortly.

-Chris


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 1:56 pm 
Offline

Joined: Wed Oct 14, 2009 12:45 pm
Posts: 1
ICQ: 6001760
Website: http://www.sassley.com
Yahoo Messenger: bluesassley
AOL: bluesassley
Location: Minnesota, US
Any new news on what is going on?

Thanks,
Blue


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 2:13 pm 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Fremont86 is having issues with its RAID unit. We're working on a solution as quickly as we can. Hopefully an announcement soon.

-Chris


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 2:57 pm 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
All Linodes on fremont86 have been booted. The RAID unit is rebuilding.

This downtime stemmed from what appears to have been a faulty drive in the RAID10 unit, which somehow caused its pair (RAID10 is two RAID1s striped) to have errors, so the subunit started rebuilding itself, despite the source drive encountering ECC errors.

Things should continue along now, while the unit rebuilds itself. However, once that's done (in a day or so) we'll be rebooting this machine again. For that, we'll send out advanced notice via the ticket system.

Again, we're sorry for the downtime but unfortunately hardware problems happen. We're just happy that this resulted in some downtime instead of something catastrophic.

Regards,
-Chris


Top
   
 Post subject:
PostPosted: Wed Oct 14, 2009 3:36 pm 
Offline
Newbie

Joined: Wed Nov 07, 2007 12:49 pm
Posts: 2
Location: Chicago, IL
My Linode's back up. Thanks for the fast action, caker!


Top
   
 Post subject:
PostPosted: Thu Oct 15, 2009 3:01 am 
Offline

Joined: Thu Oct 15, 2009 2:49 am
Posts: 1
To keep the record straight, fremont86 was taken down for about 12 minutes again at around 01:04am eastern time. A message was broadcast to the tty right before the shutdown.

Would there be any more anticipated downtime due to this RAID issue? Or it is resolved?

Thanks.


Top
   
 Post subject:
PostPosted: Thu Oct 15, 2009 3:08 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
We opened tickets with everyone on this host notifying them of an imminent migration. That downtime you saw was your Linode's turn in the migration queue, being moved to another host, so -- your Linode is no longer on fremont86...

-Chris


Top
   
Display posts from previous:  Sort by  
Forum locked  This topic is locked, you cannot edit posts or make further replies.


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group