What is the best (where a major component of "best" is "cheapest or cheap") way to handle failover between data centres?

I live in Japan. Most of my users are in Japan. I have a web-app that will see a major up-surge in use after a major Japanese disaster / typhoon / tsunami / earthquake.

I currently run the web app on a server in the Japan Data Centre to benefit from the low latency. I like low latency. Low latency is good! So under normal circs, I would like to keep it here..

I have to consider the possibility that the Japanese data centre will be knocked out in the event of a major quake in its vicinity. ("Major" in Japan being 8+ on the Richter scale. The 2011 quake was a 9.)

Now I don't doubt that the Linode Japan data centre is "prepared and ready". But colour me a wee bit skeptical, in the wake of the meltdown at Fukushima Dai Ichi. I have had too many energy companies condescending to me over the last 22 years about how safe their nuke power sites are to treat any data-centre's damn-fool claims with anything other than a large pinch of salt.

Just as the gods can always create a bigger idiot than one's idiot-proof interface can handle, Japanese earthquake gods can really, really, create a more catastrophic disaster than mere mortals can easily prepare for or any user agreement can guarantee against...

I need:

1: to be able to failover to a US data centre (lowest latency after Japan).

2: the site will be https.

3: I can lose a BIT of data.

4: I really need the transition to be as automatic as possible

5: I will have maybe 4 or 5 linodes in the Japan centre, I don't want to have to mirror that on the US site, in the beginning at least. It is too expensive. This is essentially a free site. So mirroring to 1 server, but able to spin up the others fairly quickly...

6: I need the whole thing to happen, for everyone, within about 10 minutes.

What is a good strategy / architecture?


Incidentally - what is the Linode service agreement? If I have 5 Linodes (running mysql, nginx, nginx&php and redis) drop of the net as the Japanese data centre has gone walkabout, will Linode front me 5 instances at a US centre?

Will Linode even tell me if their Japanese data centre is not available?

I appreciate that you are currently laughing at the very idea that the JP data centre could simply drop off the net.

I used to, too...

kind regards from sunny Fukuoka.

Nick

It is by no means trivial to do what you desire, but I can offer some tips to point you in the right direction.

1. You'll need to use DNS-based failover. When the Japan datacenter goes down, you'll need to update your DNS records so that your hostname resolves to the US IP address instead. Since you want the failover to happen within 10 minutes, you'll need to set a TTL on your DNS record no greater than 10 minutes. For this to happen automatically, you'll need the DNS to be updated dynamically. Conventional DNS servers like BIND support dynamic DNS; newer DNS services like Amazon's Route 53 have an API for making updates to DNS records. You'll need a monitoring process running somewhere that monitors the Japan datacenter and updates the DNS when it goes down. This is rather tricky to do correctly, since you want to avoid failing over unnecessarily if there's only a momentary outage or if you have a network partition (where both the Japan data center and the monitoring process are running but can't communicate with each other).

2. You'll need to continuously replicate your database from your Japan server to your US server. MySQL and PostgreSQL both have replication support builtin. To make life easier, you should put *everything* in a single database. For example, if you're saving stuff to the filesystem (such as images), use a database table instead, since it will be far easier to do replication if there's only one thing to replicate. You should use asynchronous replication instead of synchronous replication, since this provides better performance, with the downside that you'll lose any data that was being written to the database in the moments before Japan goes down (which you've said is OK). NOTE: replication between data centers is only feasible for databases which aren't changed too often. If your webapp is constantly modifying your database, the bandwidth requirements will be too great. It will saturate your bandwidth, causing the replication to fall behind, and/or run up your bandwidth bill from Linode.

3. When Japan goes down, you'll need to promote your US database to master so you can start writing to it. It is VERY VERY hard to do this automatically and correctly. You do not want to accidentally promote your US database to master while Japan is still up; this will cause the two databases to diverge. Remember, because of network partitions, just because your US server can't contact your Japan server doesn't mean that Japan is down.

4. In order to spin up additional Linodes on the fly, you should use configuration management (Puppet, Chef, etc.) to provision your servers.

I would strongly recommend against trying to do failover automatically, since it's very difficult for a computer program to correctly decide whether failover is appropriate or not. It's better to "wear a pager" (i.e. set up monitoring that will wake you up at night if necessary) and initiate the failover yourself if you decide it's necessary. Write a script to do the failover so it's quick. But, I could see why this wouldn't work for you, since a major disaster in Japan could prevent you from initiating the failover.

All that said, maybe you should just put your Linodes in the US. Latency between Fremont and Tokyo is ~127ms. For most types of webapps that's not going to be noticable. You can always serve your static files from a Linode in Japan (or a CDN) to get a slight speed boost for Japanese users.

With any failover plan, you need to decide how much time you can allow between SITE1 going down and SITE2 coming up, and how often SITE1 updates the data on SITE2.

Those factors will determine the best method to use.

Hard to offer more details without knowing what site you're trying to setup the failover (and what Distro/Apps/etc it uses, how often it's content/userbase changes, etc).

Easiest (and cheapest) is set Site2 to be independent. Setup rsync to mirror SITE1 to SITE2 nightly. Advertise on SITE1 the URL for SITE2 with the notice that if SITE1 goes down, your users are to check SITE2. Although easy, this method has several pros and cons - once again, hard to tell without knowing your SITE1 details.

I use this on a few sites, using DOMAIN.COM for SITE1 in DC1 and DOMAIN.NET for the backup SITE2 in DC2. Easy for the users to remember, plus I can do maintenance on SITE1 with little or no warning (I just wait for the latest sync to SITE2 to happen then pull the plug on SITE1 while I work on it). Main downside is that new users/vistors might be confused, but for the sites I use it on, we're willing to risk it for the ease of maintaining a Duct Tape and Lego Failover system.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

Thank you to both of you for your extensive comments and input.

I do take on board the suggestion that one not try to over-automate, but let a human make the final decision. I also hear what you are saying about keeping it as simple as possible and only replicating one thing. (In fact I probably won't keep images in the database, but the basic point you make is a good one.). I currently write a lot of logging stuff to the database that would not need to be replicated, so I really need to start thinking about putting that into a separate database and just syncing the necessary stuff. (I have not got into MYSQL replication yet - I wonder if it is possible to exclude certain tables when replicating...)

The latency thing is - mmm - vanity in part, I suppose. A highly ajaxy web-app just *feels* so much better with latency at 20ms than it does with latency at 120ms.

Just FYI I am running Centos 6.5. Just a basic LEMP stack, though most of the app runs client side in JS. Wondering whether I should get up to speed with Ubuntu, as Docker looks potentially useful, but seems slightly more mature (or slightly less bleeding edge) in Ubuntu then Centos.

Once again, thanks.

Nick