From another thread...
I've just been looking into this myself, and it's frustrating me: I want to monitor Apache's and mysql's memory usage so I can do capacity planning. But I haven't found any way to figure out what is really being taken up, based on a report like this:

ps aux | grep apache

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root      1066  0.0  6.3 142304 3760 ?       S    Nov06   0:03 /usr/sbin/apache
www-data 16560  0.0  8.9 144104 5308 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16561  0.0  9.2 144672 5536 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16562  0.0  9.4 144320 5628 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16563  0.0  8.7 144108 5188 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16564  0.0  8.8 144112 5292 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16599  0.0 12.3 145868 7372 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16600  0.0  9.8 144372 5872 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16601  0.0  9.2 144020 5508 ?       S    06:38   0:00 /usr/sbin/apache
www-data 16602  0.0  9.0 144068 5372 ?       S    06:38   0:00 /usr/sbin/apache

(At least with mysql on my box, all of the processes report the same memory usage.)

I have no idea how much of all these figures is shared memory.

Should I really care? I'm thinking maybe I should just take an average of the memory stats for each thread, just to have something to look at over time (knowing it's not a real number). Then just monitor total system memory usage and make sure the linode isn't thrashing.

But that's not really a good way to predict when I'll need to get more RAM, or move mysql to another server to free up memory for Apache...

Anyone know a more exact method for doing this kind of planning? I've been all over google and usenet, and haven't found anything helpful. Is there another tool I should be using?

Thanks,

ged

First thing, you may want to disable unused Apache modules :
On my gentoo, with only mod_ssl, mod_auth_pgsql :

> ps aux
[...]
root      1543  0.0  5.4  6500 3268 ?        Ss   02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1549  0.0  6.3 14932 3780 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1550  0.0  6.4 15012 3836 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1551  0.0  6.4 15012 3836 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1559  0.0  6.3 14932 3780 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1561  0.0  6.4 15012 3836 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1562  0.0  6.4 15012 3836 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1563  0.0  6.4 15012 3836 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1565  0.0  6.3 14932 3780 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1566  0.0  6.3 14932 3780 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
apache    1567  0.0  6.3 14932 3780 ?        S    02:36   0:00 /usr/sbin/apache2 -k start -D SSL -D AUTH_PGSQL
[...]

The only LoadModule statements in my main apache2.conf are :

LoadModule access_module                 modules/mod_access.so
LoadModule auth_module                   modules/mod_auth.so
LoadModule log_config_module             modules/mod_log_config.so
LoadModule env_module                    modules/mod_env.so
LoadModule setenvif_module               modules/mod_setenvif.so
LoadModule mime_module                   modules/mod_mime.so
LoadModule cgi_module                    modules/mod_cgi.so
LoadModule dir_module                    modules/mod_dir.so
LoadModule alias_module                  modules/mod_alias.so

One method you could use is to use free, check the 'xxxx' value
in "-/+ buffers/cache: xxxx"
before launching apache2 and after (let it handle some requests).

You can see shared memory segments and their size using the command "ipcs -m". The "nattch" column shows the number of processes using the shared memory at that current moment, I believe. It doesn't tell you what processes each segment belongs to though, so the next thing to do is get some relevant process IDs by using "ipcs -m -i <shmid>", where <shmid> is the shmid you want to look at. I believe that "cpid" is the process that created it. I have no clue what "lpid" is - perhaps the last process to access it?

In any case, you can then translate the PID to a name (if it's still running) by doing something like "ps aux | grep <pid>".

There's probably some better means of doing it than this but that's all I know.

Thanks to you both. I definitely need to review my loaded modules, I'm certain I can get a savings there.

The ipcs command is really interesting (and an education in itself). I wish the numbers lined up better with top or ps, but at least I have a better idea of what's going on with memory.

Thanks!

Please note that ipcs only tells what is allocated using shm calls. This is mostly used by different processes needing a common memory pool to exchange/reuse data. Apache2 configured to use threads doesn't use any for example.

The amount of memory used by an app is difficult to determine because of the memory saving mechanisms buit in Unix :
- copy on write for forked processes,
- shared libraries.

I believe the most practical way is to substract the free output values giving free memory (couting buffers and cache in free memory) before and after an application is launched.

That's certainly been proven to me over the last couple of days.


That seems to be the only way. My concern though is that on my box where Apache lives with mysql, I can't measure how much both of them grow with use over hours or days.

Since you can't tell how much either is using, it will be hard to tell, in a memory crunch, which is the culprit.

What I really need to do, I think, is keep an eye on CPU utilization and swap stats - at least I'll know if I need to get more memory or move a service to a different box.

That's the price I guess for keeping all of your services on one box (not really a good idea).

Thanks again!

ged

Are you sure you are running apache2 and not apache 1.3? The latter uses processes and not threads.

Correct, I am using Apache 1.3. Does that change the discussion?

I.e. am I any better or worse off in trying to calculate memory or CPU usage?

There is also vmstat which--at the very least--will tell you what has happened on your system. If you run it as "vmstat -s" it will give you some useful statistics. Using vmstat, your favorite statistic graphing software (MRTG, Cacti, etc.) and a simple little script to feed the info (especially pages swapped in and pages swapped out) to the graphing app, you can get a good, long term view of what your box is doing with it's swap. If it's high and swapping lots of data all the time, then it's time to investigate what is causing the swapping and think about migrating.

There is, of course, a man page for vmstat that will give you much more info than I can.

I hope this was at least somewhat useful.

--James

PS -> If you don't yet have a favorite graphing tool, Cacti is very powerful and not too hard to set up. I steer away from MRTG now because the new systems (most of which are based on the RRD tools) are much easier to configure. The linux administration world, IMHO, owes a great debt of gratitude to the author of MRTG/RRDTools.

One nice tool is to use pmap, available from:

http://procps.sourceforge.net/

I think that's already a part of the default Debian Sarge (and probably Woody) releases?

For instance, you can use pmap like this:

# pmap -x `ps -C apache -o ppid|tail -1`

Keep in mind that some are shared libs which is loaded once, made available to multiple processes automatically through some virtual memory address mapping... and some of memory usage could be deleted stuff that's still charged against the total memory size of the process.

But pmap -x generally breaks it down such that you can see exactly what is using how much.

-Dan