| Author |
Message |
jed
Joined: 28 Mar 2009
Posts: 394
Location: New Jersey
|
| Posted: Mon Oct 12, 2009 10:11 pm Post subject: Outage: fremont86 |
|
|
fremont86 ran into an issue with its RAID controller. We are rebooting the host, and Linodes will be coming back up shortly.
We apologize for the inconvenience. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Oct 14, 2009 9:55 am Post subject: |
|
|
Another reboot. We think we've identified the problem being a faulty drive which is causing the RAID unit trouble. We'll be replacing this shortly. Linodes should be coming back on line shortly.
We apologize for the inconvenience.
-Chris |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Oct 14, 2009 10:25 am Post subject: |
|
|
Additional problems have been encountered. We're working on a resolution. Updates in a bit.
-Chris |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Oct 14, 2009 11:06 am Post subject: |
|
|
We're working on configuring migrations off this machine now, which we will execute one by one. Please stand by for further updates.
Again, we apologize for this inconvenience. |
|
| Back to top |
|
spearson
Joined: 04 Dec 2008
Posts: 57
Location: New Jersey
|
| Posted: Wed Oct 14, 2009 11:18 am Post subject: |
|
|
| I'm getting some serious lag trying to SSH into my node on fremont87. Is this related? |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Oct 14, 2009 11:37 am Post subject: |
|
|
spearson wrote: I'm getting some serious lag trying to SSH into my node on fremont87. Is this related?
No.
We're still working through the problems with fremont86. Hopefully I'll have some news shortly.
-Chris |
|
| Back to top |
|
Blue Sassley
Joined: 14 Oct 2009
Posts: 1
Location: Minnesota, US
|
| Posted: Wed Oct 14, 2009 12:56 pm Post subject: |
|
|
Any new news on what is going on?
Thanks,
Blue |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Oct 14, 2009 1:13 pm Post subject: |
|
|
Fremont86 is having issues with its RAID unit. We're working on a solution as quickly as we can. Hopefully an announcement soon.
-Chris |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Oct 14, 2009 1:57 pm Post subject: |
|
|
All Linodes on fremont86 have been booted. The RAID unit is rebuilding.
This downtime stemmed from what appears to have been a faulty drive in the RAID10 unit, which somehow caused its pair (RAID10 is two RAID1s striped) to have errors, so the subunit started rebuilding itself, despite the source drive encountering ECC errors.
Things should continue along now, while the unit rebuilds itself. However, once that's done (in a day or so) we'll be rebooting this machine again. For that, we'll send out advanced notice via the ticket system.
Again, we're sorry for the downtime but unfortunately hardware problems happen. We're just happy that this resulted in some downtime instead of something catastrophic.
Regards,
-Chris |
|
| Back to top |
|
jongman
Joined: 07 Nov 2007
Posts: 2
Location: Chicago, IL
|
| Posted: Wed Oct 14, 2009 2:36 pm Post subject: |
|
|
| My Linode's back up. Thanks for the fast action, caker! |
|
| Back to top |
|
lapinski
Joined: 15 Oct 2009
Posts: 1
|
| Posted: Thu Oct 15, 2009 2:01 am Post subject: |
|
|
To keep the record straight, fremont86 was taken down for about 12 minutes again at around 01:04am eastern time. A message was broadcast to the tty right before the shutdown.
Would there be any more anticipated downtime due to this RAID issue? Or it is resolved?
Thanks. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 15, 2009 2:08 am Post subject: |
|
|
We opened tickets with everyone on this host notifying them of an imminent migration. That downtime you saw was your Linode's turn in the migration queue, being moved to another host, so -- your Linode is no longer on fremont86...
-Chris |
|
| Back to top |
|
| |
