Hi,

We have two Linodes linked via a private LAN. One is Apache web server, the other MySQL database.

Database VPS is serving around 100-130 queries per second. The CPU load for both VPSs is rather low.

Yet recently web server is unable to connect to MySQL and reporting:

Can't connect to MySQL server on 'PrivateIP' (4)

(4) is OS error "Interrupted system call"

This happens for around 10 seconds twice each hour and affects 5-10 MySQL client connections only.

Has anyone maybe experience similar issue? Any suggestions how to troubleshoot / fix it?

Kind regards,
Chris

A long shot, but what kernel are you running?

-Chris

Hi, thanks for a reply.

I am running standard Ubuntu 8.04 2.6.24-24 Xen kernel:

$ uname -a
Linux us1 2.6.24-24-xen #1 SMP Fri Sep 18 18:14:17 UTC 2009 x86_64 GNU/Linux

Just did a flood ping test between Linodes and got strange results:

ping -f 192.168.162.72
PING mysql-rw (192.168.162.72) 56(84) bytes of data.
...Warning: time of day goes back (-1098us), taking countermeasures.
Warning: time of day goes back (-1041us), taking countermeasures.
..Warning: time of day goes back (-1043us), taking countermeasures.
..Warning: time of day goes back (-1003us), taking countermeasures.
.
--- mysql-rw ping statistics ---
693417 packets transmitted, 693413 received, 0% packet loss, time 181824ms
rtt min/avg/max/mdev = 0.000/0.216/9.079/0.112 ms, ipg/ewma 0.262/0.217 ms

Never seen "Warning: time of day goes back (-XXXus), taking countermeasures." before... any idea what that might be?

Also note: there is a bit of packet loss - 4 packets out of 693417 seems not much, but this would be roughly the percentage of broken MySQL connections we are seeing.

The time goes back message means the clocks on your two machines are out of sync, do you have ntpd running?

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Do you need to run a "non-linode" kernel? I assume you do, since you took the time to set it up...

Regarding the time errors... are you running ntp on both nodes?

Both Linodes run the following cron job:

15 */2 * * * root /usr/sbin/ntpdate 0.pool.ntp.org > /dev/null

so clocks should be in sync.

Don't cron ntpdate, that's the reason you're seeing jumps in time.

ntpd will gracefully adjust your time, ntpdate just smacks it to the current time, which might be in the past, which can _really_ muck things up.

Bottom line... never use ntpdate in a cron job, always run ntpd.

hmmm... the thing is we are running this set up for the last 30 days, and "Can't connect to MySQL server on '192.168.162.72' (4)" errors started showing up only in the last two days. Please see a log file here

I'm not promising the time issue is related to the MySQL issue, but it's an easy fix, and bad time can cause very unpredictable errors.

Fix the time issue install ntpd (apt-get install npt) then disable the cron.

Wait until the times sync then see if the error goes away.

Also have you changed any firewall settings lately?

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

We finally managed to solve the problem by switching to "Latest 2.6 Paravirt" kernel. No more "Can't connect to MySQL server on 'server_LAN_IP' (4)" errors

It's even more critical that you set up ntpd in that case. The paravirt kernels don't get time from the host.