Wondering if someone here has some suggestions on how help me solve this problem (or at least reign it in):

Every several weeks (time period is not always the same), my CPU usage and disk IO pegs out the meter. My system becomes un responsive, and the only way to get back control is to reboot.

Does anyone have any suggestions on process monitors that can keep an eye on runaway processes, kill them if they get too high for too long, and restart them?

Though this doesn't quite answer your question... if you're running Mysql as part of a CMS installation, lack of caching and/or slow queries are likely to have something to do with it. Simply enabling some sort of caching (and the slow query log) for Mysql could help.

I met 300% ~ 400% CPU useage randomly. I used Centos5.0 with Virtualmin before, and met that issue every 1 or 2 days. Later I switched to Ubuntu 8.04 LTS with VIrtualmin (cause installation script cannot find 5.0 iso mirror on Centos site now, they only provide mirror for 5.2). But I still met it every each 3 or 4 days.

I just bought the account 1 month before, type is Linode 360 located at Newark, NJ, USA

I came to post a similar question.

It's happened a couple times in the past but it just happened twice in the same week. CPU spikes up but I/O flatlines.

Checking my /var/log/messages I see a ton of these

Nov 13 01:02:58 li4-190 kernel: ''IN-internet':'IN=eth0 OUT= MAC=fe:fd:42:dc:01:be:00:0c:db:fc:8b:59:08:00 SRC=88.31.96.105 DST=66.220.1.190 LEN=52 TOS=0x08 PREC=0x00 TTL=55 ID=18 15 DF PROTO=TCP SPT=4242 DPT=445 WINDOW=59584 RES=0x00 SYN URGP=0

Same thing when it crashed on Nov 9.

Logging in to LISH just gives me a screen full of that.

Not running a high profile site. 10 hits a day honestly.

It's not quite a process monitor, but I would suggest installing Munin (http://munin.projects.linpro.no/). That way, when this next happens, you can see if there was anything that accompanied that, such as high levels of network traffic or disk activity.

Based on what you say so far, I'll bet you run out of RAM and started to chew up swap, which would explain both the CPU and disk usage. One way munin can help is it can show you RAM usage over time and you can see if the amount of used memory slowly went up over time until this happened.

Good luck!

Those are netfilter (firewall) messages. Destination port 445 is a filesharing port. You can configure it to log these things elsewhere so it doesn't spam your console so much.

I was having the what sounds like the same problem almost every day: a huge IO spike, and then the CPU is maxed out until I reboot. From watching top I could see that all the RAM and swap were consumed. OOM-killer unsuccessfully tries to save the system:
Code:

Out of Memory: Kill process 28567 (apache2) score 41314 and children.

It seems that Apache goes out of control creating new processes and eats up all the memory. A few days ago I lowered the MaxClients and KeepAliveTimeout options and it hasn't happened again since.

Hope this helps.

Instead of Apache, you might want to give nginx a try. It's much nicer to your memory.

http://nginx.net/
http://blog.kovyrin.net/2006/05/30/ngin ... cgi-howto/

i can only second that, nginx is great. i'm using it with fcgi-php, small memory footprint, fast, rock solid.

check this site for english documentation: http://wiki.codemongers.com/Main

I see one "Out of Memory" message in my logs (from Today), where mysqld is killed. As suggested, I have lowered some of the values for apache (MaxSpareServers, MaxRequestsPerServer, etc). Looks like I need to look into something similar for mysql as well.

Thanks for your help, everyone.

@hotgazpacho: mysqld may not be the culprit.

My understanding of how the OOM system works (someone correct me if I'm wrong) is that when the system runs out of memory, the kernel basically goes into to "triage mode" and decides that it has to kill off some processes in order to keep the machine running. It does this by killing off what it *perceives* as the worst offenders, which may not always be the case.

Case in point, if you're getting hammered with connections, Apache will go and create as many as 255 child processes to handle those connections, which will eat up your memory. However, each of those processes MAY be smaller than a single MySQL process. Going by memory usage alone, it would appear at first glance that MySQL is wasting the most memory, when it is really Apache (with all of its child processes) that is really the culprit here.

@dmuth Point Taken

Here's my new apache settings:

StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 128
MaxClients 128
MaxRequestsPerChild 500

It was:

StartServers 5
MinSpareServers 5
MaxSpareServers 15
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 5000

Does your linode have enough RAM to support 128 Apache instances?

Probably not. That's half the default, though.

Guess I need to be more aggressive, no?

Maybe. Remember what I said about installing Munin some replies back? One of the reasons for doing that is so that you can get metrics and baselines. i.e., in normal operation, what is typical resource usage?

That when, when things fail on your box, you won't know just the /cause/ of the fail, but /how bad of a cause it was/. Based on whatever activity you the next time this happens, it will give you a better picture of if 128 servers was way too many, or just a few servers too many.