I've recently experienced my linode being unresponsive leading to timeouts to http requests and not responding when trying to login through ssh. I'm on a linode 64 and think that this is down to running out of memory and restrictions on disk access with io tokens (I keep seeing negative tokens when I do cat /proc/io_status). When I do a ps, I found that apache (1.3) uses up the most memory. My question is, can I solve this problem by streamlining Apache or is the only solution to buy more memory?

This is what I get from a ps aux | grep apache:

root       370  0.0  0.2 142652 148 ?        S    Nov25   0:01 /usr/sbin/apache
www-data  7102  0.0  3.8 146944 2284 ?       S    12:25   0:01 /usr/sbin/apache
www-data  7103  0.0 11.4 146732 6776 ?       S    12:25   0:04 /usr/sbin/apache
www-data  7120  0.0 12.8 147100 7648 ?       S    12:28   0:01 /usr/sbin/apache
www-data  7130  0.0 12.0 146708 7140 ?       S    12:30   0:02 /usr/sbin/apache
www-data  7137  0.0 12.9 147292 7668 ?       S    12:31   0:02 /usr/sbin/apache
www-data  7138  0.0 12.3 146980 7356 ?       S    12:31   0:01 /usr/sbin/apache
www-data  7151  0.0  8.3 147316 4960 ?       S    12:33   0:01 /usr/sbin/apache
www-data  7159  0.0 10.0 146416 5980 ?       S    12:33   0:02 /usr/sbin/apache
www-data  7179  0.0 13.8 147480 8212 ?       S    12:35   0:02 /usr/sbin/apache
www-data  7216  0.0  7.2 147768 4320 ?       S    12:44   0:02 /usr/sbin/apache
root      8826  0.0  0.7  1336  472 pts/0    S    17:38   0:00 grep apache

These are the modules that I am loading:

LoadModule config_log_module /usr/lib/apache/1.3/mod_log_config.so
LoadModule mime_magic_module /usr/lib/apache/1.3/mod_mime_magic.so
LoadModule mime_module /usr/lib/apache/1.3/mod_mime.so
LoadModule negotiation_module /usr/lib/apache/1.3/mod_negotiation.so
LoadModule status_module /usr/lib/apache/1.3/mod_status.so
LoadModule includes_module /usr/lib/apache/1.3/mod_include.so
LoadModule autoindex_module /usr/lib/apache/1.3/mod_autoindex.so
LoadModule dir_module /usr/lib/apache/1.3/mod_dir.so
LoadModule cgi_module /usr/lib/apache/1.3/mod_cgi.so
LoadModule userdir_module /usr/lib/apache/1.3/mod_userdir.so
LoadModule alias_module /usr/lib/apache/1.3/mod_alias.so
LoadModule rewrite_module /usr/lib/apache/1.3/mod_rewrite.so
LoadModule access_module /usr/lib/apache/1.3/mod_access.so
LoadModule auth_module /usr/lib/apache/1.3/mod_auth.so
LoadModule expires_module /usr/lib/apache/1.3/mod_expires.so
LoadModule unique_id_module /usr/lib/apache/1.3/mod_unique_id.so
LoadModule setenvif_module /usr/lib/apache/1.3/mod_setenvif.so
LoadModule perl_module /usr/lib/apache/1.3/mod_perl.so
LoadModule php4_module /usr/lib/apache/1.3/libphp4.so

and this is from free at low usage:

            total         used         free     shared    buffers     cached
Mem:         59356      56988       2368          0       3300       6456
-/+ buffers/cache:      47232      12124
Swap:       394232      59440     334792

If you can't remove any module from your config, I believe upgrading to 96M or 128M is the only practical solution for you.

I imagine you already tried to disable each of these modules one by one to make sure none are superfluous.

It looks like there is something wrong with your Apache. The virtual memory footprint is more than four times the size of mine. I use Apache 2 rather than 1.3 and have these modules loaded:

LoadModule access_module                 modules/mod_access.so
LoadModule auth_module                   modules/mod_auth.so
LoadModule auth_anon_module              modules/mod_auth_anon.so
LoadModule auth_dbm_module               modules/mod_auth_dbm.so
LoadModule auth_digest_module            modules/mod_auth_digest.so
LoadModule include_module                modules/mod_include.so
LoadModule log_config_module             modules/mod_log_config.so
LoadModule env_module                    modules/mod_env.so
LoadModule mime_magic_module             modules/mod_mime_magic.so
LoadModule cern_meta_module              modules/mod_cern_meta.so
LoadModule expires_module                modules/mod_expires.so
LoadModule headers_module                modules/mod_headers.so
LoadModule usertrack_module              modules/mod_usertrack.so
LoadModule unique_id_module              modules/mod_unique_id.so
LoadModule setenvif_module               modules/mod_setenvif.so
LoadModule proxy_module                  modules/mod_proxy.so
LoadModule proxy_connect_module          modules/mod_proxy_connect.so
LoadModule proxy_ftp_module              modules/mod_proxy_ftp.so
LoadModule proxy_http_module             modules/mod_proxy_http.so
LoadModule mime_module                   modules/mod_mime.so
LoadModule status_module                 modules/mod_status.so
LoadModule autoindex_module              modules/mod_autoindex.so
LoadModule asis_module                   modules/mod_asis.so
LoadModule info_module                   modules/mod_info.so
LoadModule cgi_module                    modules/mod_cgi.so
LoadModule cgid_module                   modules/mod_cgid.so
LoadModule vhost_alias_module            modules/mod_vhost_alias.so
LoadModule negotiation_module            modules/mod_negotiation.so
LoadModule dir_module                    modules/mod_dir.so
LoadModule imap_module                   modules/mod_imap.so
LoadModule actions_module                modules/mod_actions.so
LoadModule speling_module                modules/mod_speling.so
LoadModule userdir_module                modules/mod_userdir.so
LoadModule alias_module                  modules/mod_alias.so
LoadModule rewrite_module                modules/mod_rewrite.so
LoadModule file_cache_module             modules/mod_file_cache.so
LoadModule echo_module                   modules/mod_echo.so
LoadModule charset_lite_module           modules/mod_charset_lite.so
LoadModule cache_module                  modules/mod_cache.so
LoadModule disk_cache_module             modules/mod_disk_cache.so
LoadModule mem_cache_module              modules/mod_mem_cache.so
LoadModule ext_filter_module             modules/mod_ext_filter.so
LoadModule case_filter_module            modules/mod_case_filter.so
LoadModule case_filter_in_module         modules/mod_case_filter_in.so
LoadModule deflate_module                modules/mod_deflate.so
LoadModule logio_module                  modules/mod_logio.so

plus mod_PHP and mod_SSL are loaded from the startup command line.

My ps aux gives:

root     23494  0.0  0.4 24576  248 ?        Ss   Nov05   0:01 /usr/sbin/apache2 -k start -D SSL -D PHP4
apache   23500  0.0  0.0 23584    4 ?        S    Nov05   0:00 /usr/sbin/apache2 -k start -D SSL -D PHP4
apache   23502  0.0 13.2 33416 7904 ?        S    Nov05   0:16 /usr/sbin/apache2 -k start -D SSL -D PHP4
apache   23855  0.0 13.6 33856 8104 ?        S    Nov05   0:15 /usr/sbin/apache2 -k start -D SSL -D PHP4
apache   18797  0.0 19.3 32880 11516 ?       S    Nov09   0:01 /usr/sbin/apache2 -k start -D SSL -D PHP4
apache   21083  0.0 13.8 32700 8236 ?        S    Nov10   0:02 /usr/sbin/apache2 -k start -D SSL -D PHP4
apache    1167  0.0 14.1 27968 8424 ?        S    14:09   0:00 /usr/sbin/apache2 -k start -D SSL -D PHP4

so my Apache has a maximum resident set size just over 11MiB and maximum virtual memory size less than 34 MiB. Your Apache instances have virtual memory sizes approaching 150MiB - that's where your swapping problem is originating - Apache needs to swap to do just about anything.

Try restarting Apache and see what the memory usage is immediately after startup. If it is much lower, then something is leaking memory. If the VM size is high from the start, there is something strange about your config.

_________________
/ Peter

You use apache2 pclissold. stylehk uses apache1.
It may be that apache1 is more memory hungry than apache2 (I use apache2 myself with only a handful of modules and it nicely fits into my Linode64), but I didn't expect that.

Does anyone know of a good guide to upgrading Apache from 1.3 to 2 on Debian?

One thing to be careful of (and this is why I'm not running Apache 2) is that PHP only works with the prefork MPM.

In other words, if you use PHP with Apache 2, you still have to use the one-child-per-connection model, so you won't save much memory at all.

Do as pclissold suggested:



If it is a memory-leaking app, a workaround is to set the MaxRequestsPerChild directive in your httpd.conf to a non-zero value.

See: http://httpd.apache.org/docs/mod/core.html#maxrequestsperchild

Try starting with a value like 50 -- though you might need to adjust it depending on how much memory is being leaked with each request and how busy your site is. Nevertheless, it will stop the child processes from getting too large.

HTH,

Cliff

I posted the results from my Apache 2 installation to show how bad your memory usage figures were, not to suggest that Apache 2 was the answer to your problems. Even if Apache 1.3 is more memory hungry than Apache 2 (which I doubt), upgrading is most likely only going to get you an Apache 2 installation which eats memory. Memory leakage is the most likely explanation for your high memory usage and c1i77's MaxRequestsPerChild suggestion will mitigate the problem until you can track down the culprit.

_________________
/ Peter

Hi,

First of all, thank you so much for your help guys! I'm still a newbie, so thanks for bearing with me.

I decided against upgrading to Apache2 after reading your posts and realising it wasn't the answer. What I did instead was run 'free' and watched what happened to memory as I surfed my sites, started FTP'ing etc.

When I went to my usual websites, memory usage went up a little, but nothing too alarming (however, I was wondering at what point Apache releases memory as it never seemed to go back up).

Then when I used Squirrelmail webmail, free memory took a nosedive. I then tested to see if it was my Qmail installation by using Outlook Express to connect through IMAP but it had little effect on memory. I tried Squirrelmail again and it ate up memory again. So I guess I've found the culprit.

I'm not sure how to fix this memory leak in Squirrelmail. I'll probably have a look on their forums. If anyone has any ideas, please share! Thanks!

I just tried installing Ilohamail, and when I login to it memory usage spikes as well. But when I use the demo Ilohamail service from their servers to access my qmail server, it has little effect on memory.

Any ideas what could be causing this?

Have you tried to connect to your POP server manually, and seeing what happens to the memory then? It could be worth looking into.

Thats quite a lot to be running on a Linode 64.

Apache, PHP, qmail, courier-imap (I'm guessing your using Courier), ftpd, and you haven't mentioned it, but I'm guessing your also running MySQL.

For every page you load up in Squirrelmail, your using one of the apache processes, loading up an instance of PHP, and starting usually at least one instance of courier-imap to do whatever job is on the page (which I've found leaves an average footprint of greater than 1MB). That's just for one page, so that's why you see spikes when using Squirrelmail.

If your not running MySQL, you should still be able to pull this off without a hitch. Are you also running spam checkers? That's another service I've found you can't really pull off too easily on a Linode 64.

Hi Ciaran,

Yes, I tried to connect manually using Outlook Express and it does not have much effect on memory.

Hi Tierra,

Thanks for the explanation about Squirrelmail. It makes sense and I guess it is a lot to be running on a 64. It all basically works as long as its only me connecting to it and not doing anything major. BTW, I am running mysql as well, but I stopped using spamd.

I guess it may be time to upgrade to a 96 and I've logged a ticket with Caker. Do you think a Linode 96 will be sufficient to run everything smoothly? Would it be better to get another 64 and seperate the mail server?

With a Linode 64, it is possible to pull off Apache, MySQL, PHP, qmail, courier-imap, squirrelmail, and a couple other small things given you know your way around and have cleaned out Apache modules, optimized Apache server settings, MySQL server and buffer settings, as well as giving PHP a smaller footprint by compiling binaries with stuff stripped out that it builds in by default that you don't need, and I'm sure you can find other places to cut back. It just comes down to the question of whether or not you have a choice of upgrading, have the time to do optimizations, whether or not you really need all those services and their included features, etc. Kind of a money vs. time issue.

If you want spam filtering, definately get another Linode to handle mail separate though. There's not a lot you can optimize with that, and in general it's a big job.