Linode Forum
Linode Community Forums 		 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Forum locked  This topic is locked, you cannot edit posts or make further replies.

Board index » Linode.com Related Forums » System and Network Status
  Previous topic | Next topic 
Author Message
caker
PostPosted: Thu Oct 29, 2009 10:46 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
We've been informed by the Newark, NJ facility that they encountered an issue with some of their main power breakers. The following hosts were affected:

newark33-newark72
newark121-newark182

They've told us it's safe to bring the machines back online, and we're working through those now.

Please stand by for an incredibly verbose amount of updates.

-Chris
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 10:54 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
About 40 hosts thus far have their Linode's booting. I'm working through a detailed list now. Updates shortly.

-Chris
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 10:57 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
There's no need to issue reboots -- Linodes will return to their previous state. No data loss is expected.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 11:05 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Still working through the list of affected servers.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 11:14 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
Only about a dozen servers remaining. Updates momentarily.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 11:20 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
8 machines remaining.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 11:29 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
All machines have been brought back on line, and the Linodes are booting now.

One problem machine remains: newark60, which we're working with the facility to bring back to life. Updates momentarily.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 11:47 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
All but the last 100 host jobs or so have been processed. If you're one of the remaining few, please be patient -- the hosts are busy processing everyone's jobs.

Newark60's drives are being moved as I type this. Updates on its situation momentarily.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 11:52 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
newark60 has been moved to spare hardware, and its Linodes are booting now.
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Thu Oct 29, 2009 12:06 pm 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
All jobs have been processed, and Newark is fully back on line. If you're still experiencing a problem with your Linode please don't hesitate to open a support ticket -- we're here to help.

-Chris
Top
   
Reply with quote  
betelgeuse
PostPosted: Tue Nov 03, 2009 6:22 pm 
Offline

Joined: Tue Nov 03, 2009 6:13 pm
Posts: 1
Is there any official information as to what caused the October 29 outage at the New Jersey datacenter. I have customers calling and asking what happened. We need to respond to them. Also, is there actually backup power available or what? Is there a description of the datacenter's infrastructure?

Thank you
Top
   
Reply with quote  
aculade
 Post subject:
PostPosted: Wed Nov 04, 2009 11:30 am 
Offline
Newbie

Joined: Sun Apr 12, 2009 11:32 pm
Posts: 3
By the way...A+...much better and much appreciated
Top
   
Reply with quote  
caker
 Post subject:
PostPosted: Wed Nov 04, 2009 11:32 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 6:24 pm
Posts: 3090
Website: http://www.linode.com/
Location: Galloway, NJ
We're still waiting for the official RPO from the facility, but will post it here once we receive it.

-Chris
Top
   
Reply with quote  
JshWright
PostPosted: Wed Nov 04, 2009 11:33 am 
Offline
Senior Member

Joined: Mon Oct 27, 2008 10:24 am
Posts: 173
Website: http://www.worshiproot.com
betelgeuse wrote:
Is there any official information as to what caused the October 29 outage at the New Jersey datacenter. I have customers calling and asking what happened. We need to respond to them. Also, is there actually backup power available or what? Is there a description of the datacenter's infrastructure?

Thank you


Linode's Newark servers are host in NAC's Cedar Knolls facility. You can likely find more information about the facility itself on NAC's website. http://www.nac.net/
Top
   
Reply with quote  
tasaro
 Post subject: Official RFO
PostPosted: Sat Nov 07, 2009 1:26 am 
Offline
Linode Staff
User avatar

Joined: Tue Apr 15, 2003 10:32 pm
Posts: 246
Location: NJ, USA
We finally received an official statement from NAC:

Quote:
The power outage was caused by an accidental tripping of a disconnect switch that feeds three customer distributions panels. Electricians were working overhead when a piece of conduit waiting to be installed fell sideways and struck the disconnect switch. The electricians immediately noticed that the disconnect switch went into the protect mode, inspected the switch, and re-energized it.

As a result of this, Net Access has installed protective devices on all disconnect switches to prevent them from accidentally being de-energized.


To answer a previous question: Yes, the facility has redundant power, however power to the building was not interrupted. This was equivalent to someone flipping a breaker that only affected a few of their customers. It was a human error -- no systems were at fault here. We are confident that NAC has put measures in place to prevent this from happening again.

Again, we apologize to everyone who was inconvenienced by this.

Regards,
-Tom
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Forum locked  This topic is locked, you cannot edit posts or make further replies.

Board index » Linode.com Related Forums » System and Network Status


Who is online

Users browsing this forum: No registered users and 0 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group