If you're regularly using swap, you have a problem. It also hurts the performance of whatever you have on the server, so find a convenient time to reboot your server and see if you can control your swap usage.

I am using swap space, but I don't know why!

I used to run my email server on a co-located Mini-itx VIA PD10000 1GHz Motherboard with only 512MB of ram and that never ever touched swap space.

I would have thought that a linode was quicker than that?? But since moving that all to a Linode 540 its done nothing but swap!

...but, and here's the really annoying part....

i've only been swapping like crazy for the last 3 weeks and I've been using the server since the start of December as my mail email server!

I don't understand why the performacne has degraded so much in the past 3 weeks

Rich

The problem we've been discussing in this thread only hits from time to time, for a maximum of few hours. Not for three weeks straight. Hardware problems tend to get fixed pretty soon, so if it's been three weeks, something's definitely wrong with one of your programs.

A linode 540 obviously has more RAM than your old colo, and it's also many times more powerful. Also, it was fine until three weeks ago, right? Then try to find out what happened when the problems began. Did you change some configurations? Did you update your programs from your distro's repository? Did you suddenly get a lot of visitors to your web site or a lot of spam to your mail server? Perhaps someone hacked your server without your knowledge?

There are many diagnostic tools that can tell you which program is using the most CPU and RAM. top is the most simple of all; htop looks prettier; munin is a bit more sophisticated. You can get halfway towards a solution if you can pinpoint the bad guy.

Head hung in shame... I think its a config error!

I've been poking about and found that my grey-listing daemon on my mail server is configured to keep connections open to mysql.

It doesn't seem to re-use connections, but just keeps opening them up to the 100 limit set in gld.conf

Hopefully I've fixed it and my Linode can go back to being awesome!

ok, I spoke too soon. but I think I've found the reason.

using 'htop' i've found that clamd is using 311MB of ram! WTF

No wonder nothing else has room to work nicely without swapping all the time

ok, I've found a possible fix for my I/O trhashing.

If I change my "vm.swappiness" value from the default of 60 to 95 it seems to help?

I run this at boot time:

sysctl -q -w vm.swappiness=95

However, it totally fills my swap space up

top - 12:34:33 up 20 min,  1 user,  load average: 0.24, 0.14, 0.10
Tasks: 104 total,   1 running, 103 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  1.0%sy,  0.1%ni, 95.9%id,  1.9%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:    553176k total,   537280k used,    15896k free,     2480k buffers
Swap:   262136k total,   218204k used,    43932k free,   297812k cached

But it seems to have reduced my disk IO

I'll keep an eye on it.

Rich

Using up most of the swap space during normal system operation is probably not a great idea. If your load increases even slightly for whatever reason, your machine is going to quickly slow to a crawl.

If none of the programs you're running are using abnormal amounts of memory, I would suggest just getting a bigger linode.

I apprecieate that its not a good position to be in but its better than it was with regards to performance.



I've changed my vm.swappiness value to be 5 to test both ends of the scale and my swap space isn't used but my load average goes up to around 3!

Reverting vm.swappiness back to 95 results in a load average of 0.40!

Rich

I've edited this post as my fix was to create more indexes in MySQL

do not edit your vm.swappiness value unless you have a good reason to do so