Linode 768, Ubuntu 10.04.1

It hosts two web sites , the one with all the traffic is Wordpress running around 6000 hits/hour. I had been experiencing sporadic hangs system-wide and so did some tuning. I -

- turned on WP caching
- cut down the number of queries displayed on the WP / page
- attempted to move to apache2-mpm-worker, but rolled back to prefork (reinstalling phplib5 in the process)
- at apache2.conf, I took timeouts from 300 to 1 second

Every 4 hours or so, the Linode panics and crashes:




Apache /server-status


Apache Server Status for www.cantstopthebleeding.com

Server Version: Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.7 with Suhosin-Patch mod_ssl/2.2.14 OpenSSL/0.9.8k
Server Built: Nov 18 2010 21:17:19
Current Time: Saturday, 05-Feb-2011 04:53:39 UTC
Restart Time: Saturday, 05-Feb-2011 03:57:23 UTC
Parent Server Generation: 0
Server uptime: 56 minutes 16 seconds
11 requests currently being processed, 7 idle workers
KKKWKK____K___.....KK.K....K....................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

PID Key:
3383 in state: K , 3371 in state: K , 3323 in state: K
3388 in state: W , 3376 in state: K , 3389 in state: K
3390 in state: _ , 3391 in state: _ , 3392 in state: _
3375 in state: _ , 3360 in state: K , 3393 in state: _
3394 in state: _ , 3395 in state: _ , 3367 in state: K
3368 in state: K , 3370 in state: K , 3298 in state: K


vmstat 1 20


procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 161908 21384 250356 0 0 13 4 92 20 3 0 95 1
0 0 0 161684 21396 250344 0 0 0 312 1024 184 6 0 94 0
0 0 0 160568 21396 250356 0 0 0 0 227 42 0 0 99 0
1 0 0 158212 21404 250352 0 0 0 0 2110 245 11 2 87 0
0 0 0 159288 21436 250336 0 0 0 0 2927 509 15 2 83 0
0 0 0 159040 21436 250472 0 0 0 0 1683 267 9 1 90 0
0 0 0 159032 21436 250472 0 0 0 52 113 38 0 0 100 0
0 0 0 157032 21448 250472 0 0 0 104 222 44 0 0 99 0
0 0 0 157032 21448 250472 0 0 0 0 154 32 0 0 100 0
1 1 0 156328 21452 250468 0 0 4 0 1285 285 5 0 95 0
2 0 0 140572 21496 250464 0 0 40 0 1502 232 6 1 91 1
1 0 0 140076 21496 250484 0 0 28 56 2219 475 14 1 84 0
0 0 0 142928 21496 250516 0 0 0 0 341 53 2 0 98 0
0 0 0 142680 21516 250508 0 0 0 576 961 218 5 0 95 0
1 0 0 130892 21516 250576 0 0 0 0 545 81 2 0 98 0
0 0 0 129892 21520 250572 0 0 0 0 1734 357 10 1 89 0
0 0 0 130264 21520 250632 0 0 0 736 1188 184 6 0 94 0
0 0 0 130264 21520 250680 0 0 0 0 137 46 0 0 100 0
0 0 0 130264 21520 250680 0 0 0 0 51 32 0 0 100 0
0 0 0 145144 21528 250676 0 0 0 124 98 37 0 0 100 0


free -m


total used free shared buffers cached
Mem: 751 609 141 0 21 245
-/+ buffers/cache: 343 407
Swap: 255 0 255


ALSO:

I put a value of 30 into /proc/sys/kernel/panic but the most recent crash did not reboot after the panic. I also put 1 into /proc/sys/kernel/panic_on_oops ...anybody know why that isn't working?

-r

What is MaxClients set to in your apache2.conf? As a rough guess, for prefork I'd suggest starting at around 30 on a Linode 768 and adjusting as memory usage patterns indicate.

MaxClients is at 30. In which direction should adjustment go?

Fit your processes into your available RAM. Divide how much memory you have by how big your processes are. Round down. Keep monitoring.

I've had issues with WP and Joomla running under PHP with the Suhosin patch. I tried troubleshooting along the lines of what you did.

In the end I reinstalled PHP minus the patch and that seems to cure it. Can't say it will do for you but something to consider if everything else fails.

I can't remember the link I followed, but this looks like it's the correct direction

http://chrisblunt.com/2009/05/01/php-fi ... -packages/

Swapped out the kernel, crashes went away and nominal CPU usage showed up.

Went from

Latest 2.6 Stable 2.6.18.80-linode10

to

2.6.37-linode 30


Thanks caker and everybody else!

-r