OK, have reduced MaxClients down to 15 and turned off KeepAlives.

Linode unresponsive on a 10 user test.

here is a screenshot of top at the time:

Yep, the reason you're unresponsive is that you've run out of memory, used all your swap and are thrashing (68% wait, kswapd is your top CPU process). I wouldn't be entirely surprised if you had a kernel panic at some point under this load if the OOM had nothing to kill.

So the good news is that it's clear what is happening (and it's what has been suspected in this thread), just still not why. The question remains what is tying up so much memory under load, as the processes in the top display only account for a small portion. We're one step closer, but we still haven't really identified the primary culprit for the resource use. Once we get to that "ah ha" moment, it should be easier to then work forward to plan a reasonable working memory set.

The apache processes shown aren't enough, but there must also be some missing (if you have MaxClients of 15) - probably due to the default sort order. But even if all 15 were using the high value shown of 29MB I don't think you'd have so little free memory, so someone else is using up a big chunk. Maybe the database server. Or there's an anomalous apache case we can't currently see.

If I were you, I'd probably drop MaxClients to 1, sort the top display by memory usage, then try to hang the machine again. You might actually prevent a hang at a setting of 1 (since it should free up a few hundred MB of memory) but hopefully will still see the peak memory user in any case. If everything works with a setting of 1, that's useful data too, but then you can start increasing it until you do get a hang.

Having a taller window to get more process entries on the top display it couldn't hurt, but the key is likely to be the first few top memory users.

Oh, and I'd probably just also look at current process memory usage even when idle. If there's some process we haven't discussed yet sitting around tying up a large chunk of memory even in idle conditions, that would be useful to know.

-- David

PS: I think this data also continues to argue for not just upgrading the machine plan yet, since you're already clearly at least using 768MB (physical+swap) and at the moment there's no reason not to expect that your configuration wouldn't just grow to encompass any larger memory on a larger plan. You may end up with a larger plan as a better fit, but you still need to identify where the usage is coming from in order to determine that.

thanks again for all the help with this, here is a screenshot sorted by memory with MaxClients at to 15, will reduce to 1 and post again.

Switching MaxClients to 1 doesn't kill the server but it still gets extremely slow.

The benchmark testing gets slow, or the server gets slow? The former I'd expect due to the fact that MaxClients of 1 essentially serializes all requests through the single process. Given these stats (which don't show much in the way of CPU or I/O overhead) I'd probably expect the Linode itself (say via ssh) to be fairly responsive. You seem to have eliminated the memory stats from the top output, but I'm assuming you had some memory free.

An important fact to highlight is that at least you have a configuration (poor performing though it is) that doesn't ever completely keel over. That's real progress in terms of troubleshooting, and at least provides a stable starting point to work up from.

The fact that your single apache2 process is using 81MB of resident memory is a big deal. If that happens to more than a few of the possible processes under your older MaxClients of 15 you could easily explain the problems you were getting into. Unfortunately, the prior post doesn't actually seem sorted in memory order (still CPU) so can't say for sure - but even there you had a much higher average resident usage (39-40MB) than the prior summary.

So, my take away from this is that your stack is actually using quite a bit of memory for the apache processes on average (say 30-40MB), with some extreme peaks (80+MB). That may or may not be something you can optimize, but it certainly crys for starting with a very low MaxClients (just divide those resident sizes into the available memory). I'm not really a PHP guy, but seem to recall comments somewhere about some of the cache solutions needing a lot of memory - since that's your tightest resource at the moment, you might also experiment with disabling any caching for the heck of it.

Sans optimization, a next step would be to slowly raise MaxClients, testing with each change, and watching until you got tight on free memory (or see a spike in CPU or I/O, but I suspect memory will be the first resource you exhaust). Given these resident sizes I'm thinking you won't get much beyond 5-10. That can give you an inkling of best performance without any further tuning. Keep the process list sorted by memory so you can observe peaks in apache process sizes.

You might also consider making sure that MaxRequestsPerChild is low (but not 0), in case there's a memory leak in the application stack that let's the longer lived apache processes grow. Such an issue could also explain why limiting things to a single process resulted in an even larger size. Setting it to 1 will hurt performance, but ensure the smallest footprint in such cases since the process exits after each request forcing a resource cleanup. Note that a setting of 1 will probably also make it harder to catch all the processes in top - dropping the refresh interval to 1s can help but you'll still just be seeing snapshots at intervals that are large compared to the process creation/destruction interval.

To put these two parameters in perspective, consider a load test pummeling your server with thousands of requests. Apache is going to let MaxClients copies of itself be running simultaneously, each handling a request. Thus, your memory footprint will be MaxClients times the process size, which in turn depends on what that process does, such as your PHP code. Apache leaves a process around for MaxRequestsPerChild, handling multiple requests. So if your processing causes the apache process to grow by some amount per request, your peak usage will be something like MaxClients * (initial_apache_size + (MaxRequestsPerChild * per_request_growth)).

Your goal here is to get these parameters high enough for maximum throughput, yet low enough to not exhaust memory worst case. Even if you do have some leak/growth, letting at least a few requests get handled by a single child can help a lot with fixed overhead of starting up the process and PHP interpreter, while protecting against unbounded growth over time. So for example, if you have a choice between MaxClients of 10 and MaxRequestsPerChild of 1, versus MaxCients of only 5 but MaxRequestsPerChild of 5, the latter might actually perform better due to a slower rate of process creation/destruction.

That should let you settle in on a rough working set for the current Linode. Let's say that you can only get MaxClients to 5, and due to a leak with higher MaxRequestsPerChild you're stuck with that at 1 to help bound individual apache process sizes. Now you'll know the rough requests/s you can handle on a Linode 512 without ever dying, and judge if that's fast enough. Remember also that even low request/s numbers can yield very large daily page visit counts. An average of 1/s is still 86,400 per day though clearly you want instantaneous peak req/s rates to be higher to support that on average. But your original target of 5K page views a day (let's say over an 8 hour period, and each page needs 10 individual requests, so 50,000 http requests in 8 hours) could be met with about 1.75 req/s on average. And that's probably a pretty conservative estimate since the static parts of the page can be serviced much more rapidly (and with less memory) by apache so won't be anywhere near as slow as your PHP-backed content.

If you judge the rate insufficient, then yes, at that point increasing the Linode plan will let you continue to raise MaxClients (slowly) depending on how much more memory the plan has, and gain some parallelism - at least up until the point where CPU or non-swapping I/O overhead begins to dominate.

Or of course, at that point digging more deeply into your application stack to find out if there are bugs, bottlenecks or things that can be improved there becomes an option too. But at least you'll have a stable platform to attempt tuning on.

Geez, this got long again ... sorry about that.

-- David

Thank you so much & don't be sorry about a long reply.
Gives me a lot to look at.
Can't wait to move the site to a linode, support here is way too good.
Will report back tomorrow with some new tests based on your recs.

FYI, if you're running top, you can hit capital M to sort processes by memory usage.

Thanks again for all the help with this.
Just testing again with MaxClients set at 2.
After about 10 minutes of running a 10 user load test, this is the result sorted by memory.



If I offload the mail to gmail, do I need clamav running?


and a minute later sorted by CPU as I noticed I'm hitting a high CPU usage at times.

You are hitting 80 megs per apache. The math is pretty easy -- if you had MaxClients = 10, you would be using 800 megs. So you can probably sustain MaxClients of 5 or so (=400 megs).

I have not used Wordpress, but these processes seem huge. For my codeigniter sites, the processes typically max out at 30 megs, and they include a lot of caching etc. I also use nginx as a proxy, so that might help by offloading a lot of the work.

Are there plugins you can turn off in Wordpress? Even if not, I would guess MaxClients of 5 should handle all the traffic you ever need it to.

OK, get it now, thank you.

any recommendations for the other settings?

StartServers 2
MaxClients 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestsPerChild 0

I do have some plugins I will be removing & I'll hardcode a lot of the links to reduce database queries, just want to get these settings right first.

So you're running Apache, and Webmin, and Python, and Perl, and MySql, AND Postgres, and BIND, and a SMTP app, AND a pop3 app, and a Antivirus.....

Besides optimizing Apache (which is pretty much a given) might want to trim back some of the non-essentials.

This seems about as high as I would go without removing other tasks or finding out how to shrink your average apache size. You have a reasonable amount of memory for caching purposes, but not a ton. So before increasing MaxClients any further, you need to find a way to save memory elsewhere. Or if your request rate is ok with this configuration, you could declare a short term victory, pending further optimization.


If you don't have inbound files flowing through your machine (whether mail or otherwise) that you want to scan, absolutely, I'd get rid of it. This should let you add at least 1 to MaxClients.

-- David

PS: BTW, your screen shots don't seem to be working from wherever they are hosted at the moment. If you still have them yourself and have any way to perhaps post them in this thread as text, it might help ensure the context is preserved for future readers.

Hmm, I have to admit I had been assuming that you were using the pre-fork MPM (which I thought was more common with PHP) and not the worker MPM. If you're using the worker model (threaded), then the multiple request threads per child might account some for the larger average apache process size. It probably doesn't make that much difference if n requests are spread across n apache processes or n threads within a fewer number of processes, but you could also experiment with using the pre-fork model to see if you get better behavior and/or control over your process size.

In either case, if you still have very large apache processes, you could also try with a low (but not 0) setting for MaxRequestsPerChild. A value of 0 here lets the same apache process run forever, so if there's a leak of any sort in the application stack, it could contribute to the unusually large apache process sizes.

You're basically looking at an iterable approach at this point - stick with your working configuration as a base, and then tweak in various ways to see how it affects your processing rate, memory usage, and general performance. Only fiddle with one or two parameters at a time or else you won't be able to tell what change is affecting behavior. There isn't necessary a single "right" set of configuration parameters, though you ought to be able to determine what works best for your application stack and load.

I'll second what others have said too - watch your heavy resource (particularly memory) users other than apache too, and decide if you really need them, or if the resource they are using could be freed up for use by apache and your web processing stack.

-- David

Thanks again guys.
I'm setting up a new disk image to check mem usage without all the unneeded processes that got installed by installing webmin.
Forgot this would kill the images, will re-add them shortly.

Sorry, I am using pre-fork, copy & pasted wrong bit from config file.