Hi all,

My machine in host22 is having some troubles at random. Recently, on Mar 13 and today, it started to eat io_tokens and swap. I don't know why. On Mar 13 I opened a support ticket and it seems there is something misconfigured in my system. Rebooting the machine was a temporary fix, but it happened again today. Mgreb and Caker told me to tune some parameters and to hop into the IRC Channel. To make thing worse, my broadband company has decided that the best way to fix a billing problem with a customer is just to shutdown his ADSL, so I'm stuck with my good old 56k USRobotics friend

Anyway, I've managed to make some fixes to my configuration: I placed caker's my.cnf into my system (linode 160) and I changed this in httpd.conf:

<IfModule prefork.c>
    StartServers         5
    MinSpareServers      2
    MaxSpareServers     10
    MaxClients         30
    MaxRequestsPerChild  20
</IfModule>

I use php via suphp, but as far as I know there would be at most as many php threads as Apache's worker threads, and I think I've limited those down.

I use postfix/mysql/courier-pop3d/courier-imapd and I think I have a bottleneck right there. My system has some big email accounts, and when there are more than a few Mb on one of them, the first time a POP3 is done, my io_tokens go to 0. I don't know if there is something I can do to get better performance there, including replacing postfix for other system.

Apart from that problem on POP3ing big accounts, and obviously something that happens at random and I have to track down yet, the system seems to run smoothly and almost at low load and 400K io_tokens, as you can see at https://www.ferferhosting.com/mrtg .

Do you have any suggestion on what more should I try? Do you know if there is any way I could monitor io_tokens and know which process is eating them when the system goes down (i.e. a cron script)?

Thank you very much in advance!

shameless plug: http://www.ratemylinode.com/

That will monitor your io_tokens usage, and give you an idea of where you stand in comparison to a handfull of other people.

Check out the current rankings page: http://www.ratemylinode.com/rankings.html

Signing up right now. But, as far as I can see, that will do more or less the same as mrtg. Is there any command I can run to see how many io_tokens are eating each process?

Probally not, unless you did something to trace the execution of the process or had something that monitored the kernel's VM subsystem.

Either way, it probally isn't what you're looking for anyway. On linodes, the #1 cause of io_token usage is swap thrashing. Find your applications that are memory hogs. Find the ones that are using the most swap. Tune them waaaaay down. Lower than you'd ever dream on a standalone box. Then benchmark your applications, see if they continue to function. If they don't, raise the limits on your apps.

Apache, and MySQL are notorious for sucking up memory. Take a look at how often your applications write logs to disk, see if you can turn down the logging level if you don't really need it.

Syslogd has an option to not actually sync() the files to disk every time it wants to write something:

authpriv.* -/var/log/secure

Just add a - to the beginning of the full path to your log files in /etc/syslog.conf. There are lots of other tips that can be found on the Linode Wiki, search around or hop on IRC and I'm sure you'll find someone who can help.

I once had a problem on a server (standalone dual-processor p4 with ATA hard disks) where some users had some *huge* inboxes, where they enjoyed keeping all their incoming mail (IE never cleaned them up). This resulted in huge mbox files which totally killed the server when one of those users accessed the mailbox; this was mainly because in order to get the index for each mailbox, the server had to traverse the file, which was a problem with 400MB+ mailboxes.

The solution turned out to be migrating all the mailboxes to maildir format, this eases index building a lot since each message is contained in a single file; so instead of traversing a huge file, the system just stats small files and, should it be needed, inspects the first few lines of each file to find subjects, senders, recipients, etc. This simple technique basically spared us from having to acquire new hardware! and the best part is, I only had to replace the imap server (from uw to courier) and tweak the system's default procmail rc for sendmail a bit in order to have maildir-formatted mailboxes.

If you think large mailboxes are the source of your problem this would be a very viable solution.