I'm at my wits end, and need your advice.

At least once a week my server goes "down" and when I go in with Lish all I see is apache processes getting killed.

While the server is down my SWAP and IO is in the RED and no other metrics are out of normal.

Linode 1536
Ubuntu 10.10
2.6.32.16-linode28
512MB SWAP





apache2 -l
Compiled in modules:
core.c
mod_log_config.c
mod_logio.c
prefork.c
http_core.c
mod_so.c

apache2.conf
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 3
MaxSpareServers 5
ServerLimit 48
MaxClients 48
MaxRequestsPerChild 4000
</IfModule>

keepalive OFF

/etc/mysql/my.cnf
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
skip-innodb
table_cache = 128
query_cache_limit = 1M
query_cache_size = 16M
expire_logs_days = 10
max_binlog_size = 100M
max_allowed_packet = 16M
key_buffer = 16M

php.ini
memory_limit = 256M

Any recommendations appreciated!
[/img]

Check through the forum for Apache tuning tips - the odds are very good that at the point you run into problems you end up with more (and/or larger) Apache processes than your system memory can handle and you end up in a death spiral of swapping. There have been a number of threads in the past that address this issue, how you can start to break down your memory usage, and then adjustments you can make to work within your resources.

If you do nothing else, try dropping MaxClients significantly. You may slow down your peak request rate somewhat, but you should prevent the system from getting into such a bad state. Finding the best value is heavily application and load dependent (and also addressed more fully in past forum threads), but simply making that number much smaller is a quick 'n dirty first step to ensuring you stick within the resources you have available.

-- David

thanks db3l

I've been beating the linode forum and other website threads..

trying this.. trying that.. I can't seem to find the right mix of configurations..


I really wish someone would look at my my.cnf / php.ini and apache2.conf and tell me it looks decent or if they see a glaring issue...

Overall the system seems to be big enough for my LAMP load and the SWAP issue seems to be random with no rhyme or reason. I'll push the maxclient down some more and see.. but all my tweaks seem to do NO help at all.

MaxClients 48
PHP Memory Limit 256M

48 x 256M = 12G of RAM!!!

There's no way you can fit all of that in your mid-sized Linode. The only reason your Linode works at all is because very few PHP scripts use anywhere near 256M of RAM. But even if each script used only 32M of RAM (typical for heavyweight CMSs), you'd run out of memory just the same. Throw in a couple of image processing add-ons, and you're going to hit the swap.

You need to do two things: cut down MaxClients to half of the current value (24), and cut down the memory limit to no more than a quarter of the current value (64M). That should give you a relatively safe starting point. Then enable the error log in php.ini. If some scripts require more RAM, errors will show up in the log file. Adjust the memory limit to make sure that all your scripts work, or get rid of add-ons that consume more memory than they're worth. Every time you need to increase the memory limit, decrease MaxClients to compensate. If your server works fine without hitting swap for a few weeks, try increasing MaxClients a little bit. It takes trial and error to find the sweet spot. (Good call on disabling KeepAlive, but giving it a very low value such as 1-2 wouldn't hurt, either.)

By the way, you have two key_buffer lines in your my.cnf. Delete the second one. While you're at it, try increasing key_buffer a bit, and see if it helps your pages load faster. Since your site seems to get a significant amount of traffic, consider transitioning to the InnoDB storage engine. But that's for the long term, and you'll need to budget a bit more RAM if you want to use InnoDB. Right now, you should focus on getting your memory usage under control.

@hybinet

Now this is what I needed to here.. thank you for helping tie this together for me.

Many thanks for your time.

So you want want me to remove the "key_buffer" located under [isamchk] ?? Will do.. just for my brain, why?

#max_connections = 100
table_cache = 128
#thread_concurrency = 10
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8


[isamchk]
key_buffer = 16M

Oh, that was under a different section? Then don't delete it. In your first post it looked like you had the same line twice in the same section. (By the way, the one that you should try adjusting is the key_buffer line in the [mysqld] section.)