Given that caker said he is close to switching the default UML kernel to 2.6 in http://www.linode.com/forums/viewtopic.php?t=1291, I gave 2.6.9 a shot yesterday. It still seems to have a problem with some of my daemons dying off - specifically vixie cron and clamd (clamav). They don't leave any evidence as to why they've died, and it seems to be just those two. I'm not running with NPTL enabled or compiled (Gentoo). Switch back to 2.4 and everything is stable. I've tried recompiling both, including switching versions, and the behavior is the same.

I have also observed the same behavior with older 2.6 linode kernels, but hadn't reported it due to their experimental nature.

Strange : I'm on gentoo without NPTL and 2.6.9 is ok for my Linode64.
I'm running vixie cron and clamd too. The only problem was with Bind 9 (segfaults randomly), now that I switched to tinydns, everything works fine.

The machine is loaded though : Postfix, PostgreSQL, Dspam, SQLgrey, Clamd, Apache2, Tinydns, Ulogd, Tpop3d...

$USE :
"berkdb crypt gdbm imlib libwww maildir mmap ncurses nls pam pcre perl pg-intdatetime pg-vacuumdelay png postgres procmail python readline ssl threads vhosts x86 zlib"

free :

             total       used       free     shared    buffers     cached
Mem:         59592      58632        960          0       5116      35120
-/+ buffers/cache:      18396      41196
Swap:       262136      22784     239352

pstree :

init-+-agetty
     |-apache2---3*[apache2---apache2---3*[apache2]]
     |-clamd---clamd
     |-cron
     |-devfsd
     |-events/0-+-aio/0
     |          |-kblockd/0
     |          |-khelper
     |          |-2*[pdflush]
     |          |-xfsdatad/0
     |          `-xfslogd/0
     |-fetchmail
     |-freshclam
     |-jfsCommit
     |-jfsIO
     |-jfsSync
     |-2*[kjournald]
     |-klogd
     |-ksoftirqd/0
     |-kswapd0
     |-master-+-pickup
     |        `-qmgr
     |-pg_autovacuum
     |-postmaster-+-postmaster---postmaster
     |            `-2*[postmaster]
     |-sqlgrey
     |-sshd---sshd---bash---pstree
     |-svscan-+-supervise---tinydns
     |        |-2*[supervise---multilog]
     |        `-supervise---tcpserver
     |-syslogd
     |-tpop3d
     |-ulogd
     `-xfsbufd

I wonder if it's the OOM killer.. I am on a linode 128, but I am running spamassassin so if I get a flood of mail it gets hairy. But I'd expect the OOM killer to leave something in the kernel log, and there's nothing.

If running multimple spamassassin runs triggers the oom killer, you should be able to serialize spamassassin work.

If for example you use a pipe from postfix to spamc, you can configure postfix to only pipe one message to spamc at a time.

Yep, I do have problems with my setup, and they do need fixing. I just haven't had time yet. But the OOM killer idea was just off the cuff, I am not very convinced that it is the cause.

I've been running 2.6.9-linode6 for a while now, and it's been pretty stable. Until yesterday.

At around 2:00pm yesterday the files in my /proc filesystem disappeared (even though it was still mounted). I don't know if this was the cause or the symptom of my problems, but the machine was running really slowly. And without /proc, I don't know how to tell what processes are running or how much memory is being used or anything. The logs in /var/log don't seem to show anything useful.

Anyone know how I can inspect the system without /proc if this happens again in the future?