Whose site is that? Does he/she work for Linode?

Nope, a totally separate company to Linode. However, they use the same datacentre.

http://i.imgur.com/2VoyJ.jpg

(Bay Bridge)

It was one of the worst lightning storms in the Bay Area in recent memory.

Wow- at some point, lightning will win.

_________________
--
Chris Bryant

Well, considering the breakdown voltage of air is in the neighborhood of a few megavolts per meter, I would say lightning -- by definition -- always wins.

_________________

/* TODO: need to add signature to posts */

More importantly, a typical lightning strike delivers five terajoules (~ 1.4 MWh) of energy in a few microseconds - always a winning formula, whether you're toasting bread (~100,000 slices in a one kilowatt, two-slice toaster) or servers.

_________________
/ Peter

Could you reduce that to, say, 50,000 slices of toast and get 5,000 pigs worth of bacon out of it as well? I figure there's about a 10:1 ratio of slices of toast per pig of bacon, energy-wise, more or less. If so, I think Hurricane Electric owes us breakfast.

_________________

/* TODO: need to add signature to posts */

That works out at a fifth of a pig in each bacon sandwich. More of an HE hog roast than breakfast.

_________________
/ Peter

maybe im wrong.. but i thought battery backup on the raid card, just kept the data in the raids cache until power was restored and it could be written out properly?

that battery isnt providing power to the hard drive(s) so when they lose power the head will still crash..

_________________
ServerAdmin - www.our-lan.com
"Diplomacy is the art of saying nice doggy whilst looking for a really big stick"
"In my experiece, any attempt to make any system idiot proof will only challenge God to make a better idiot"

Actually, I doubt any drives nowadays have a head crash - at least not in the traditional meaning of the head just dropping to the surface - on power loss. Instead, the drive automatically parks the head first in a dedicated landing zone (CSS) or off-media mechanism (ramp). A spring and/or platter inertia is used to provide the energy to reposition the head during power loss so it never hits the data surface.

-- David

linode staff went above and beyond expectations in fixing all the downed linodes. but they didn't come clean about why a power outage took hours to recover from.

linode is very secretive and hardly ever owns up publicly when an outage is their fault.

obviously the power failed and hosts were damaged weren't linodes fault, but their backend issues that stopped linodes from booting was.

No complaints about the outage or the staff, just a complaint about the backend issues not being mentioned in the status updates.

Well, when a DC goes out, you can't boot up all the servers at the same time or you will trip a circuit breaker. Then, as the servers come up, you have to boot 40 VPSes each, serially. It takes a while. An hour or two is not a long time to recover a whole data centre of VPSes.

Between the status pages, the forums here and the conversations that take place in the IRC channel which are publicly accessible to even non-customers, I think you'd find that Linode is one of the most open companies in the business.

In Linode's shoes, I would also have waited a little bit before bringing servers back up. The datacenter had just lost power, which is not supposed to happen; clearly, Something Is Wrong, and the last thing you want is to lose power again with thousands of fscks running at once. So, indeed, I can't really blame them for waiting ~15-30 minutes before turning things back on.

(Especially given that there was no power protection at that point, and there was a thunderstorm in progress.)

_________________

/* TODO: need to add signature to posts */

I've been pleased with the speed of response, and especially with quick responses on trouble tickets. They've now posted a full explanation of the data center problems on the status blog, too. http://status.linode.com/2010/11/fremont-power-issues-rfo.html