You might need to work on tuning APC.

I can't see your apc.php today, but I can see from your phpinfo.php that your APC memory is only 30MB, the Ubuntu default.

Is your apc.php pretty ugly with full cache, lots of fragmentation and high cache clear count?

If so, you should increase to about apc_shm_size=80 in /etc/php5/conf.d/apc.ini and also increase the system shared memory size to about 96MB on Ubuntu using /etc/sysctl.conf setting of kernel.shmmax = 100663296

Keep increasing the size of the apc cache til you get no cache clears and no fragmentation.

followed all the advice here & things are looking so much better, thank you all.

here is the result of the same test now after 10 minutes load test:


My prefork MPM is now:
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 2
MaxRequestsPerChild 200
</IfModule>

Increased APC memory
php info: http://109.74.198.57/php.php
apc info: http://109.74.198.57/apc.php


Any other tips or suggested MySQL configurations?

I'm pretty sure that having MaxClients < StartServers is a waste of RAM on eternally-idle processes...
... but then, I use worker and I don't know for sure about prefork.

_________________
rsk, providing useless advice on the Internet since 2005.

Change to Nginx.

wow, must say that using Nginx as a reverse proxy has helped massively & was extremely easy to set-up.
followed this guide

It's even easier to just ditch Apache entirely and use Nginx and PHP via FastCGI.

Or apache-worker with fastcgi. Really, once you move PHP out to FCGI semipermanent processes, the choice of frontend webserver is a matter of preference. Apache may be more practical because of all the "assuming apache's rewrites" .htaccess files out there.

_________________
rsk, providing useless advice on the Internet since 2005.

Finally switched DNS to my Linode & while it hasn't fully populated yet, I notice I am dipping into Swap Memory, is there anything in the screen shot below I should be worried about?
Running MaxClients at 5, should I cut back to 1 or 2 while I tweak other settings?

16 KB isn't much of a dip, in the grand scheme of things.

_________________

/* TODO: need to add signature to posts */

Nah, 16K barely qualifies even as "dipping"; that's small enough to essentially just be statistical noise. I wouldn't even worry if that grew to be a few MB, unless you start to see the I/O wait % rise. It's really when your working set forces constant swapping that things fall over the cliff and you'll recognize that when it happens, having now been through it before :-)

As for dropping MaxClients, mostly a judgment call. Doing so will free up a little more memory for the kernel to use for filesystem caching which can help with performance as long as the fewer simultaneous clients don't bottleneck things first.

You may find over time that mysqld will grow in memory usage (some of which can also be tuned as needed) as your database grows.

But you look pretty reasonable at this point so if I were you I might let things settle in a little further and gather more data about what steady state looks like. Just spot check your resources in the near term, or better yet set up something like munin to monitor it to let you review it over time.

-- David

something is very wrong,

if you have php 5.3.3 turn off apache, and use php-fpm to manage your php connections, post a TOP image then.

why do you say something is very wrong?

you should be aiming to reduce the memory consumption of apache, it looks as if it has actually increased.

using php-fpm or fast-cgi will reduce total memory consumption as only once instance will run, rather than an instance per apache process, at least, that is my understanding.

i had a similar problem before switching to lighttpd/fast-cgi.

with apache + mod_php i was averaging ~ 384mb used, now ~ 170mb used (active).

If you'd have one instance of PHP, you could handle one PHP request at a time, serializing your server's performance down no nothing.

Facts:
1. wth mod_prefork+mod_php there's a separate large Apache process WITH integral php parser launched for every single http connection, no matter if it's for a php script or a simple image file. That's a huge hog, and you plainly have to cut MaxClients down to a number that'll accomodate all these processes in memory.
2. With fastcgi, you "detach" PHP from Apache. There's a "manager" process, that spawns $PHP_FCGI_CHILDREN sub-handlers, so it can handle that much parallel scripts. Your apache (or any other webserver) talks to it, but only if it actually needs to execute a php script - "plain" files are served by the webserver directly.
3. If you want to use fastcgi and PHP in Apache, you need to use mod_fastcgi - mod_fcgid doesn't pipeline requests, and assumes "one subprocess = one script at a time". The whole PHP tree IS one subprocess to Apache, as you spawn only the "manager". So, mod_fcgid thinks you can execute only one PHP parser at a time.
4. If you're moving to fastcgi, there's nothing (except other non-threadsafe apapche modules you may have, that is) to hold you onto prefork. You can switch to mpm-worker, set a nice number or threads, and enjoy your fast, memory-efficient, Apache setup.
</Facts>

I have an mpm-worker setup that has 50-250 threads active (split 25 per process; probably could go for 50 per process, but don't see any need), and 25 php slaves. That means I can handle up to 250 parallel connections, 25 of which can be executing a php script, and rest pulling html files and/or images - otherwise, request is queued until a thread and/or a parser is free. It all fits in about 200MB of RAM, and that's including the 64MB shared-memory APC cache.
This thing serves about 200k hits per day without a hitch, and with about 18% CPU load in the rush hours.

_________________
rsk, providing useless advice on the Internet since 2005.

Once you put nginx in front of apache, apache pre-fork isn't such a horrible choice anymore.

1. You configure nginx to serve static files directly, so apache doesn't have to use a php-loaded thread to serve static files.

2. You disable keepalives on apache so php-loaded threads don't have to wait around for the keepalive time and can die immediately.

The memory and performance difference between nginx+php-fpm and nginx+apache-prefork isn't much.

Now, I use nginx+php-fpm and recommend it highly. But using nginx in front of apache-prefork does solve most of apache-prefork's disadvantages.