Anyone else in the Fremont Datacenter hear what happened to HE.net about 10am HST?

All our Linodes were unreachable for +90 seconds (nagios monitor)...

for our Email stuff, not so bad (but did show up on our IMAP Proxy).

but for our VoIP PBXs, it caused calls to drop...

So we're just wondering if anyone in the community heard anything.

I am not sure exactly what happened, but I believe I also experienced the same issue. Maybe somebody could shed some light into this issue?

I checked the Linode Status page, but it has nothing on this issue (at least not so far).

Bump. I experienced this too on three different unrelated linodes I have hosted in Fremont. Could not connect with them through http, ssh, or by ping, but they appeared to still be up when looking at the dashboard. From the looks of it, it was probably some kind of network issue.

Fremont seems to have all the problems! As a new Linode user, I picked Fremont because it was close, but so far I've seen:

- Backups failing every day for over two weeks
- DNS Manager issues (reporting zones with errors)
- Loss of connectivity today

I'm wondering if it's time to migrate to another DC. Anyone? Thoughts?

thanks,
bruce

Just choose the nearest one for main auditory of your site(s):
http://maps.google.com/maps/ms?ie=UTF&m ... 7bfb8ef294

Close? How so?

The Internet is a big place, and unless you're running two-way video or voice, or playing FPS type games, whats a few more milliseconds give or take?

I think you're missing the point. The DC's are clearly NOT all the same. This Fremont location is building up a poor reputation for stability.

One would normally think to pick the closest one, and that's exactly what I did (Fremont is in my state), but now I'm thinking I should have read up the quality and performance of the various sites before choosing.

I'm now thinking Atlanta or Dallas. Anyone?

thanks,
bruce

Well, difference in ping between Britain and USA is very noticeable. I become client of Linode only because Linode has data center in London

I understood you. I put it just as а starting point

And I think Dallas, because... http://www.linode.com/IPv6/

About stability: http://status.linode.com/

Why does it seem that people post in the forums instead of contacting support when they have issues like this? It would be nice to hear from someone at Linode to see if it was a Linode or HE issue and with all of the recent issues in Fremont to see if Linode is doing anything, like how hard they're leaning on HE to straighten their stuff up.

Looking at the graphs I see a huge spike for incoming traffic at about that time (8am PST). I'm wondering if someone made a mistake and realized it very quickly after the fact.

Actually, I HAVE created tickets with Support for all of my issues. In fact, the Support folks probably cringe every time I create one.

I came to the forums here because on more than one occasion, the Support folks have told me to look for help here (witness my recent performance issue of about two weeks ago). I think there are some awesomely smart people here, and the more we can share this information and crowd-source solutions, the better.

But yes, Support should always be the first stop. And by all means, I would love to hear an official report from Linode on the Fremont DC. The backup problem is getting almost ridiculous, for one...

thanks,
bruce

we did create tickets with the support team and they acknowledged the issue as valid. They said it possibly involved all the fremont HE facility, so we were looking to the community for more information.

A single storage pool in Fremont has had problems over the past few weeks, which we've been working on to correct.

Part of the solution requires redistributing the data to more storage capacity that was to be added to the pool in question. We encountered some hardware issues which prolonged adding the additional capacity. This was unanticipated, has since been resolved, and now we're in the final stages of moving the data around (which takes a while!). I'd expect the backup service for those affected to resume normal (even better!) operation within the next day. We're working on it.

We have also been discussing our procedures and how we could have handled this situation better - including customer notifications and compensation for the backup service downtime - rest assured that we'll make it right. Again, we're working on it.

The DNS Manager 'check zone' issue is completely unrelated to Fremont and has since been resolved.

The connectivity issue that was seen today was caused by a BGP session failure, upstream from us. We're working with the Fremont facility to resolve this, but I don't have any additional information at this time.

-Chris

Actually we've been quiet impressed with Linode. We've tried to run VoIP on other platforms (Rackspace) and it didn't work out. mix of latency and the VEs processor.

Plus you really have to appreciate that linode lets you pool the bandwidth.

As for the backup, We use Amazon S3 from within our linode, works good and not all your eggs are in 1 basket.

As the the DNS , We just run our own - easy enough.

Besides this 90 second outage (which only really affects VoIP users)... www and email could always blame it on something else..lol. Linode has rocked for us.

We've all become specialist ... HE.net I'm sure gets paid very well to run BGP... (off topic, but have you seen the $1/GB, that's unreal!).

We did talk to linode, but they're only human. The community would have resources beyong Linode (like maybe someone is a XO customer and could verify the outage @ Fremont)....

Just like open source, Lots of minds are better than some.