If I were in your shoes, I'd hit your node with testing with 'ab' (or seige or any other load test tool) to get it to keel over under controlled circumstances. Given the current behavior you are experiencing, I think this should be easy to do :-)

Then drastically drop MaxClients and MaxRequestsPerChild, say to 1 each, and re-test. That should prevent the problem, but kill performance. Leave something like top running, then slowly increase MaxClients only and re-test, watching memory until you're just at the point of having a little for buffers. Then increase MaxRequestsPerChild slowly until performance stops increasing. Note that running multiple requests per child may result in each worker process using up more memory, so you may need to lower MaxClients when using a higher MaxRequestsPerChild to balance out memory vs. request performance.

At that point decide if your performance is good enough or you need to continue to other more performance oriented tuning steps.

If this isn't possible on the production box, spin up a test Linode (even if only for a day or two), clone your current box to it, and run the tests there.

It's a bit monotonous, but not terribly complicated, and it's really the best way to identify how much resource you can tune your stack to use.


Might help or might not. Personally I suspect you'd still need to solve your current root issue and that involving nginx is a later tuning stage for performance. The odds are that most of your memory usage isn't Apache per-se, but the Drupal part of the code stack. And you still need to support that.

Fronting apache with nginx for static content may cut significantly down on the memory needed to serve the static portion of your site, but that's also content that Apache itself serves quickly, so hard to say for sure how much that'll change the overall memory profile. This sort of change may help more with performance than memory.

You could offload Drupal to a completely separate server process, but then you'd probably find that it was that server process eating up the same memory for the same number of simultaneous requests as when it was internal to Apache.

-- David

How to get the server to keel over with ab?

I tried all these, but they don't crash the server:

# ab -kc 300 -t 60 http://www.....com/

# ab -kc 1000 -t 60 -r http://www.....com/

# ab -n 10000 -c 100 http://www.....com/

# ab -n 3000 -kc 150 http://www.....com/

One important item is to pick a good representative URL to use. I'm not sure from the redacted examples you give, but if you're querying a home page it may or may not use anywhere near the full application stack to generate the page. Is the page you are querying created by executing the Drupal code and hitting your database in a representative way? If in doubt, you could check your logging to try to pick out some representative URLs occurring prior to recent crashes.

For testing stuff like MaxClients, the concurrency (-c) is most useful since that's how many requests to do in parallel. As long as it's more than your MaxClients you are testing (maybe with a 20-25% fudge factor) it should be high enough. So I'd think even something around 100 should be more than high enough.

Definitely don't use -k since that issues multiple requests in a single connection and you're trying to test multiple independent connections.

I'd have thought your example with "-n 10000 -c 100" would probably have been a reasonable test. What did your resource stats look like during that test (e.g., memory stats in top)?

Is this with your Apache configuration the same way as the last crash? If so, and if you have a decent test URL then your scenario may be more complicated, either due to specific Drupal processing occasionally needing a lot more memory than when you run these tests, or due to some other unidentified task on the box using up the memory. I suppose at that point you could also ask yourself if there have been recent changes to the site (new features or anything) that might use different Drupal code recently. Maybe a specific type of request needs far more memory than others to process.

Though for any of the Drupal stuff, the general hammer of MaxClients/MaxRequestsPerChild is still a good approach even if you don't know precisely what is happening. If ab can't push you out of resource while tying up all of your MaxClients processes, drop your MaxRequestsPerChild pretty low (maybe single digits) for now. That'll protect you against a long-lived process accumulating memory and throwing off the typical case MaxClients.

-- David

Are you sure both parameters are actually set? If you had kernel.panic set your node should be rebooting in these scenarios and not just staying in a panic. You might want to check (via sysctl) that it actually has the value you think it has.

You could certainly turn off vm.panic_on_oom if you like - as I note it will just put back the default behavior of killing off processes to try to free up memory, which may or may not be all that better than a panic.

-- David

I can tel you for sure that this isn't a hardware issue with your node.
But I'm not really sure.
Perhaps if David can respond back to you, someone could help on this one other than myself.
How did you deploy your configuration?
Are you running plain Drupal/WordPress or something on Apache?
I've done this before on a Linode512, with no problem.
So I'm not sure what's going on.
More information on what your urnning, and or how you configured it primarily might be a good idea?
I wouldn't suggest, getting a loarger Linode.
Budgeting in if you can even aford a higher end Linode is important, and like David or whomever said earlier, it wouldn't solve your problem for sure.
It would only be a temporary fix.

_________________
Regards, --Keith.
Primary Email:
keithint1234@gmail.com
Skype: skypedude1234
Twitter: @keithint1234
Facebook:
http://facebook.com/keith.hinton1

Ok, I just crashed it using "-n 10000 -c 100" on a different URL and turning drupal caching off. The CPU maxed out, memory slowly went to 100% (took about a minute I think) and after it filled the swap, the server was done. Do you think this is what crashed the server naturally before? By a bot or something?

I will try lowering the apache values and slowly up them tomorrow morning. I'll post the results.

Thanks again for all the help David!

One more thing.

I've found another problem. I've suspected Piwik (an open source Google Analytics alternative) for a long time. I just inspected it and it seems like a live real-time visitor widget on the Piwik dashboard is eating away memory.

When I hit the page it eats some memory and after that it keeps eating (if you keep running the dashboard) until it hits 1024mb, the swap fills, and the server crashes.

Is this a Piwik problem, or just a problem with my server's configuration?

Hard to say the root cause, but if there is a leak in a code path, that can be exacerbated by a high MaxRequestsPerChild setting, since that leaves the process around. That would align with what you're seeing about memory growing on repeated requests.

In such a scenario, it's even more important to keep MaxRequestsPerChild low, since that will let Apache restart its worker processes more frequently, and prevent memory growth from getting out of control if a code path is leaking memory. Think of it this way - even if your Piwki code is leaking memory on each request, if you had MaxRequestsPerChild at 1 (minimum) then you could never leak more than 1 request worth - e.g., repeated use of the dashboard wouldn't grow anything. The downside is the overhead of restarting the processes, so a modest value can be of large performance benefit, but you can get most of the benefit with even such modest values (e.g., single or double digits) while minimizing the risk of a single growing process.

In terms of this crash scenario, and the other one you pointed out in your last post, it's not that critical if these are exactly the same as what your users are triggering. Nothing you can do with a stress test ought to be able to crash your box, so any failure that you can control is a great test to use when tuning. The hope is that your stress test will turn out to be worse than anything end users can generate, and while you can't guarantee that the odds favor that if you can find a configuration that can pass these tests you'll be fine with end users.

-- David

The first thing I did when installing Piwik was set it up as a crontab (that runs every hour), then I disabled it from running on pages to get better performance;

crontab -e
MAILTO="emailforerrors@mydomain.com"
5 * * * * su --shell=/bin/bash --session-command="/path/to/misc/cron/archive.sh > /dev/null" apache

I would highly recommend this for your site

Supposedly if you have alot of data, it parses through it all, and it says by "default" on high load servers to set the memory limit for php to 512mb.. yeah.. I'm not doing that! Another option is just to use a program that can parse with lower overhead.. like google analytics (google does the calc for you), and webalizer (directly reads your http log files).

I tried AWStats, but it was very slow using Perl.. I probably won't use it again.

I'll set up a cron job. Didn't do that yet since I don't think that's the problem.

I just set both MaxClients and MaxRequestsPerChild to 5 and the ab stress test didn't kill it. But when I went to the Piwik live dashboard widget, it slowly ate my memory again until it crashed the server. It took like 10-15 minutes.

I don't think that's normal, is it?

When I run the dashboard without the widget, there's no problem. The ab stress test didn't get the memory usage over 150mb.

Ok, so I did the tuning and settled on:

MaxClients 10
MaxRequestsPerChild 100

Stress tested it with ab, with caching off memory usage would stay between 500-600mb. Seems fair right? Upping the values didn't seem to help performance much anyway.

With caching on the memory usage wouldn't even go over 125mb, but that's tricky since it's just testing one page, which gets cached pretty well.

I think MaxRequestsPerChild was definitely the problem (plus MaxClients was set too high to begin with).

I'm running into big problems with Piwik though. I've enabled the cron and disabled archiving on the page. But every time I visit a Piwik dashboard page it keeps eating up memory. Even without the live stats.

Where normally the memory would be given back to the system. All the Piwik pages keep eating memory with every page visit until it's filled up and the system crashes. Can this be a mySQL tuning problem?

Ugh, it seems I can do the same thing on my site. When visiting a lot of different pages that haven't been cached before, it slowly eats all the memory. A lot slower than Piwik, but it happens..

When testing with ab it's just one page, so I guess it's different when visiting a lot of different pages. Is this still apache's problem? Shouldn't it release the memory after the page has been sent?

Or should I edit these values too?

StartServers 5 
MinSpareServers 5 
MaxSpareServers 10

Ok, so I guess the problem I described above is how apache is supposed to work.. It doesn't care about your memory?

I've now settled on even lower settings. It seems to make some images load pretty slow, but at least the server doesn't seem to be crashing (yet).

    StartServers          3
    MinSpareServers       2
    MaxSpareServers       3
    MaxClients           10
    MaxRequestsPerChild  30