You realize that the problem is HE, not Linode, right? And only one of Linode's datacenters is with HE... So you really ought to take your issue up with HE's engineers, as Linode's people are not the source of your woes.



If your nodes didn't reboot properly, it's due to either a hardware fail or your config. If a hardware issue, you don't have to do anything, because the DC folks have to fix the hardware, and then it's back to normal. If it's your config, then it's not a DC or Linode issue at all, it's that your config is borked.



If you're paying $2k per month, why don't you have geographic redundancy?

akeri - I agree with most of your points.

Unfortunately I have a relationship with Linode, not HE. So I can't really go chase around HE's engineers... I guess I should have expanded that I haven't seen sufficient resolution from Linode's point of view (really nothing other than "sorry, it won't happen again").

That's fine, but I think I'll move to a host which either runs their own datacenters or has a better relationship with the datacenter engineers.

Most of the linodes did reboot, but the 'Lassie' feature seems to not work 100% all the time. For the linodes that didn't reboot properly, all I had to do is simply login, assess the situation, realize that Lassie didn't do the reboot correctly, and issue a manual reboot. So the server configuration was fine, it was just the Linode architecture.

And I agree, I should have geo redundancy. But this wouldn't have solved the fact that linodes didn't reboot... even with redundancy I would have had this issue. But I take your comments as constructive, and thus emphasizes the point that I should have some sort of redundancy built in.

Thanks again for your suggestions.

I will definitely agree that I wouldn't trust the fremont datacenter for things that needed uptime.

My main point is that it's just fremont that has this issue. All the other datacenters have been rocksolid, or at least as close to that as anything technical can be.

i scheduled a migration to dallas as soon as my linode came up and the process was pretty painless. file support ticket, wait for them to setup the migration (took less than five minutes), shutdown linode, click the migrate button, and update dns.

_________________
http://ultramookie.com/

In my opinion, it seems Fremont is best suited for those who perform their own backups or have development environments. Place your live server in another datacenter and your backup server in Fremont. If your backup server goes down your customers don't notice and, with a few exceptions, you can probably handle some unscheduled downtime of your backup server.

Why keep your backup server in a location that you know to be unreliable, if you can have it elsewhere for exactly the same cost? After all these outages, I can't think of a single reason to keep a node in Fremont, other than latency across the Pacific Ocean. But if it's a backup server, latency doesn't matter.

I'm really seriously hoping that Linode gets a new datacenter in the West Coast and/or Asia-Pacific region.

We rely on the fremont facility for the lower latency to Australia.

I think Linode needs to make its position clear on what action they will be taking to ensure HE has its house in order.

If it continues to simply be business as usual - then we will migrate our VM's away from Linode. As this has been the third outage due to power at this particular facility.

This can only be attributed to a lack of investment by HE at Fremont. And Linode should seriously start assessing the collateral damage this is having to both itself and its customers!

Don't get me wrong, I am a huge fan. But business is business and I cannot justify to my clients why they should continue to use nodes at a facility that is simply sub-par.

Let me partially rephrase, Fremont is a good solution for backups or developments systems if other options aren't available. It wouldn't be my first choice, but if I had to use Fremont i would only use it for backups or development.

I think there is some consensus here: move ALL production environments away from Fremont as there have been multiple unexplained, uncontrollable, unfixable downtime events over the past year, and there is little-to-no information that would suggest otherwise.

Twitter shows the situation is a disaster with HE now: http://twitter.com/#!/search/henet

Not too much Linode can do here, other than move their own datacenters.

I'll probably keep a linode account for testing purposes (not even reliable enough for staging purposes)... currently in process of migrating 12 production and staging linodes (most >1024) off... what a way to spend a beautiful Sunday!

At least the outbound transfer speed at Linode is FAST!

Probably move the 15 production and testing ones later next week / next month...

One was mother nature. Huge lightning storm in area took down pretty much everything in the Bay Area. Pretty rare for this area though.


viewtopic.php?t=6301&postdays=0&postorder=asc&start=30

What about putting in their own rack UPS'? Where there's a will, there's a way...Someone just needs to get their wallet out.

That might help prevent damages to hardware, but no UPS will keep your server online when the rest of the datacenter (including the networking equipment) has no power.

Depends how long the power is off for. Routers and switches boot quickly. At each of the four power outages it's taken around two hours before most VPS' are restored. I suspect the power is coming back on a lot sooner than this, but we're waiting for Hosts/SANs/Linode stuff to come up.

One would hope, in 2011, that their core network equipment had reliable dual power sources, surely...

Well, not really encouraging.

A move to another DC seems easy, according to this thread.
But the data move (stopping initial server, move, start, DNS change, DNS cache lag time...) has its share of painfulness.

The real question is :
- do we have any information that shows there is something going on in order to prevent such problem in a near future?
- or should we better move?

If it is the latter I'd prefer to read some kind of official information that says "Fremont is a test platform." - at least we'd be warned.

I asked what their plans were to mitigate these on-going power issues at Fremont were, and the response was they do not currently have any or not in a position to disclose them.