I have 2 linodes in Dallas. One's a web server, the other a mysql server, and they communicate over the private network. Today I noticed my site wasn't responding. For some reason, it's taking around 30 seconds to set up a mysql connection to the private address of the mysql server:

$ time mysql -h 192.168.130.XX -u xxxx -p
Enter password: 
... (ctrl-D)
mysql> Bye

real   0m30.632s
user   0m0.004s
sys   0m0.001s

$

Substituting the server's public IP address, it connects instantly. The same thing happens with ssh, so I'm pretty sure it's not a mysql issue. In both cases, once the connection is set up, it's fast as usual.

What the hell? Any ideas? I've tried to poke around and debug this, but I'm not sure where to start. My /etc/network/interfaces on both nodes seem fine, and rebooting both of them didn't help.

Edit: plain HTTP connects instantly over the private IP, though.

One possibility is the server (or client) is attempting to do a reverse DNS lookup on the private IP (which fails, because private IPs are private, and have no rDNS). Try disabling it in the mysql server by passing --skip-name-resolve as a command-line option, or add a hostname for your private IPs to /etc/hosts.

bdonlan: You're right, it was reverse DNS timing out. mysql and ssh were attempting to get the reverse of the private IP, and this took 30 seconds to fail. This same config had been working fine for several weeks, so I'm not sure if this reflects a change in behavior with the Linode resolvers. Either way, I fixed it by adding hostnames for both private IPs in /etc/hosts on both nodes.

Thanks to the irc crew for helping me figure it out!

This affected me as well today. Any idea what changed this afternoon to cause it?

blackhole-{1,2}.iana.org are repsonsible for the reverse of 192.168.*. Perhaps these servers were down, or there was some connectivity issue between them and linode. Are you in Dallas too?

That's funny... I was diagnosing a problem sending email from my phone today, and tracked it down to reverse DNS lookups taking so long that the client gave up and dropped the connection.

Those were attempts to do lookups on two of Sprint's network blocks, 173.96.0.0/11 and 99.200.0.0/13 .

Figuring that out was quite hair-pulling. Reverse DNS on those IPs is now quick again, so things are normal. Could this be related?