| Author |
Message |
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 9:46 am Post subject: Newark, NJ Facility partial power loss |
|
|
We've been informed by the Newark, NJ facility that they encountered an issue with some of their main power breakers. The following hosts were affected:
newark33-newark72
newark121-newark182
They've told us it's safe to bring the machines back online, and we're working through those now.
Please stand by for an incredibly verbose amount of updates.
-Chris |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 9:54 am Post subject: |
|
|
About 40 hosts thus far have their Linode's booting. I'm working through a detailed list now. Updates shortly.
-Chris |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 9:57 am Post subject: |
|
|
| There's no need to issue reboots -- Linodes will return to their previous state. No data loss is expected. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 10:05 am Post subject: |
|
|
| Still working through the list of affected servers. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 10:14 am Post subject: |
|
|
| Only about a dozen servers remaining. Updates momentarily. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 10:20 am Post subject: |
|
|
| 8 machines remaining. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 10:29 am Post subject: |
|
|
All machines have been brought back on line, and the Linodes are booting now.
One problem machine remains: newark60, which we're working with the facility to bring back to life. Updates momentarily. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 10:47 am Post subject: |
|
|
All but the last 100 host jobs or so have been processed. If you're one of the remaining few, please be patient -- the hosts are busy processing everyone's jobs.
Newark60's drives are being moved as I type this. Updates on its situation momentarily. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 10:52 am Post subject: |
|
|
| newark60 has been moved to spare hardware, and its Linodes are booting now. |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Thu Oct 29, 2009 11:06 am Post subject: |
|
|
All jobs have been processed, and Newark is fully back on line. If you're still experiencing a problem with your Linode please don't hesitate to open a support ticket -- we're here to help.
-Chris |
|
| Back to top |
|
betelgeuse
Joined: 03 Nov 2009
Posts: 1
|
| Posted: Tue Nov 03, 2009 5:22 pm Post subject: Post-mortem on Newark outage |
|
|
Is there any official information as to what caused the October 29 outage at the New Jersey datacenter. I have customers calling and asking what happened. We need to respond to them. Also, is there actually backup power available or what? Is there a description of the datacenter's infrastructure?
Thank you |
|
| Back to top |
|
aculade
Joined: 12 Apr 2009
Posts: 3
|
| Posted: Wed Nov 04, 2009 10:30 am Post subject: |
|
|
| By the way...A+...much better and much appreciated |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2906
Location: Galloway, NJ
|
| Posted: Wed Nov 04, 2009 10:32 am Post subject: |
|
|
We're still waiting for the official RPO from the facility, but will post it here once we receive it.
-Chris |
|
| Back to top |
|
JshWright
Joined: 27 Oct 2008
Posts: 173
|
| Posted: Wed Nov 04, 2009 10:33 am Post subject: Re: Post-mortem on Newark outage |
|
|
betelgeuse wrote: Is there any official information as to what caused the October 29 outage at the New Jersey datacenter. I have customers calling and asking what happened. We need to respond to them. Also, is there actually backup power available or what? Is there a description of the datacenter's infrastructure?
Thank you
Linode's Newark servers are host in NAC's Cedar Knolls facility. You can likely find more information about the facility itself on NAC's website. http://www.nac.net/ |
|
| Back to top |
|
tasaro
Joined: 15 Apr 2003
Posts: 244
Location: Manahawkin, NJ
|
| Posted: Sat Nov 07, 2009 12:26 am Post subject: Official RFO |
|
|
We finally received an official statement from NAC:
Quote: The power outage was caused by an accidental tripping of a disconnect switch that feeds three customer distributions panels. Electricians were working overhead when a piece of conduit waiting to be installed fell sideways and struck the disconnect switch. The electricians immediately noticed that the disconnect switch went into the protect mode, inspected the switch, and re-energized it.
As a result of this, Net Access has installed protective devices on all disconnect switches to prevent them from accidentally being de-energized.
To answer a previous question: Yes, the facility has redundant power, however power to the building was not interrupted. This was equivalent to someone flipping a breaker that only affected a few of their customers. It was a human error -- no systems were at fault here. We are confident that NAC has put measures in place to prevent this from happening again.
Again, we apologize to everyone who was inconvenienced by this.
Regards,
-Tom |
|
| Back to top |
|
| |
