1) It's not that much more time it takes when we're talking about the internet
2) They're the same cool and nerdy VPS provider that are transparent about things. They have never, to my knowlegde, preannounced things.
3) They will announce it as soon as possible.

_________________
lakridserne
Serverfruit - shared and managed VPS hosting, SSL certificates and domains
Awesome servers rented from Linode!

Strictly speaking, they did kinda pre-announce that Fremont would be getting the upgrades soon, and are not saying anything about why it hasn't happened yet. So, I would say that this isn't a great example of either transparency nor of avoiding premature announcements.

_________________

/* TODO: need to add signature to posts */

Here is the straight dope: We were getting everything ready for a move to Hurricane Electric's FMT2 facility, which is like two miles down the road from FMT1. This was all to be ready about a week after the upgrade announcement. FMT2 is a modern build-out, which we consider to be a great improvement over FMT1's aging power and space restrictions. The strategy involved us getting a 10G wave on fiber they have between buildings, which places our two distinct networks on the same layer2 network. Meaning no IP swaps, no disruption in IP failover support, private networking working as expected, etc. Essentially this has created one large LAN despite hardware and Linodes existing in both facilities. Over the course of 3 months or so, we would perform a few phases of migration queues + physically moving and upgrading hardware from FMT1 into FMT2, and then continuing with the next phase, rinse, repeat. Doing it in phases was largely a host-resource game.

This was all staged and ready to go, but we discovered this link wasn't as redundant as we had believed. We asked questions, and got answers from HE. There are scenarios where the wave between the two facilities could break, and that news put a stop to everything while we investigated how to implement absolute redundancy between the two facilities in the event the link failed. We have a solution, but want to greatly shorten the time we're relying on this bridged network, which means altering the FMT1 to FMT2 transition timeline. This is where we stand now.

Things could change, but we're now thinking this will be: a few weeks heads-up during which the upgrade-button would simultaneously upgrade you AND migrate you to FMT2. Those who don't perform the upgrade by the deadline(s) would have their host shut down, the host de-racked and physically moved to FMT2 and then powered back on. Of course we would have everything prepared in FMT2 to receive these machines and essentially it would be: a shutdown, unplug host, transport it to FMT2, rack it and plug it into already-ran cables, and then power it on. Logistics so far indicate the downtime for that scenario is minimal, 60-120 minutes, performed in the middle of the west-coast night. Over the course of a few days we'd be out of FMT1 completely.

We've already deployed an impressive amount of capacity in FMT2, however this shortened timeline and elimination of phases means upgrade+migrate-to-fmt2 availability will be on a first-come-first-served basis. We've also been working on redirecting host builds from other facilities to fmt2 to augment fmt2's host resources. However, we can't guarantee everyone that wants to move/upgrade early will be able to, under this scenario.

There are still a few loose ends, and of course this could all change, but that's where we are. The logistics of this type of endeavor are quite complicated as you can imagine, and we want to do this with the least amount of stress, downtime, and risk as possible. We hope to nail down the timeline and enable the upgrade button very soon.

-Chris

Thank you so much for the update.

I have hated being on Fremont (FMT1) all this time, and I cannot wait until I get to FMT2. It's helpful to know what's going on. You're the best!

Thanks!
Bruce

Thanks Chris,

This is exactly the kind of answer I was hoping we'd get (it's obvious there was some sort of delay) so knowing what's happening and having appropriate expectations is the next best thing. Also, your response shows that you/the Linode team truly are concerned about making this a smooth transition for us, although I know that the world wouldn't come to an end if my very small $20/mo Linode went down, it's nice to know that minimizing downtime and inconvenience is a priority.

I really wish that you guys at Linode would post this sort of thing more often. Not necessarily all the time, or in that much detail, but it's nice to see. It's kind of like UI feedback; for a task that takes a given number of seconds, compare the user experience to having a progress bar versus just locking up the UI. The user who gets the progress bar is going to feel like the machine is faster and more response, while the user who just gets a frozen screen for the duration is going to feel the machine is slow and unresponsive.

It's like that for this; getting some insight into the technical stuff behind the scenes may not be something that we customers need, but it's definitely appreciated and reinforces that Linode (the company) is doing something. It also tends to make customers more tolerant of stuff too.

So, whenever you guys can afford to share some insight into goings-on, it's appreciated.

Thank you for the update, Chris, as well as your apparent attention to the details. It's great to see the move to FMT2. Does the current plan still mean no IP swaps?

Thanks for the update! Would love to see more detail like this in the future as has been in the past.

If we create a new linode at Fremont, which facility is the linode being created in?

FMT1, until we enable the upgrade button - then all new Linodes will go to FMT2.

-Chris

Thank you SO MUCH, Chris.

That's the kinda thing I would have been happy to hear earlier on -- as I am now. Good luck with the move, glad to hear about servers there already.

That all makes perfect sense. FMT2 is awesome. I like their "dark room" -- it's like putting an amphitheater over a small town then shutting off all the lights.

Yes, we are taking about the Internet -- and not the web, which seems to be your perspective since you do shared hosting. For other applications where latency is critical (let's use a game server as an example), a difference of even 10ms ping could make all the difference.

The problem is that situations like this are often fluid. There's a tough balance to strike between enough information and too much. Had Chris been putting out the updates along the way, I could easily see people misinterpreting pieces of it and getting unrealistic expectations or simply not understanding pieces of it and jumping to conclusions. There's a reason that linode usually doesn't communicate things out until they're done, because then you don't have to deal with threads like this where people are saying "it's been two weeks, where is it!?!?". My day job has plenty of this kind of stuff, so I'm very sympathetic to Chris' position (of course, my linodes are in NJ, so it helps that I'm not personally affected as well).

I'm pleased to have some info on the Fremont RAM upgrade. I will admit to a little frustration at being in the dark for about a month. AS my company's CEO likes to say, "Our customers can handle good news. They can handle bad news. What they can't handle is no news." I kind of felt like that.

Thanks for the forthcoming upgrade and the news.

Thanks for the update, it's really very much appreciated.