Linode 720
Ubuntu 9.04

I'm running the latest Drupal install, and have a community site running at AionElysium.com. In general the CPU usage hovers around 20% - 25% but ever since a few weeks ago the server occasionally spikes to max apache clients and 400% CPU usage. I've already optimized apache, PHP, and MySQL quite awhile ago. I never had problems with this config until recently (few weeks now).

When this began happening I immediately uninstalled Drupal modules I had installed, but that made no difference. There are no errors in the Drupal error log, and nothing in the Apache error log (except an entry saying that the max clients has been reached).

Restarting apache usually solves the issue, but I have had a few instances where I've had to reboot the entire linode.

I am a web developer with a B.S. in computer science, so I would appreciate any help or even a nod in the right direction.

Other notes:

Network traffic at the times is normal
Disk IO peaks as well (I'm associating that with the Apache max clients thing)
The times that this happens are not predictable (happens at random times throughout the day)
I've tried disabling all cron jobs, but still experienced issues

Thanks in advance

Trying increasing it a little, and see if it spreads the outages some. Look in your access logs and see if you are getting a lot of traffic from a single IP.

_________________
If it ain't broke, you didn't tweak it enough. If it is broke, use more duct tape.
http://independentchaos.com

What does the memory usage look like when it happens? Are you hitting swap? (Install Munin to find out.)

Max clients is currently at 25. I increased it all the way to 50, but the problem still occurs, randomly as always, and generally at least once per day. memory usage is spiking along with the CPU. As I mentioned, my disk IO takes quite a leap. The strange thing is that despite the memory and CPU usage given by top, I'm able to easily and quickly log into the server and issue an Apache restart command. The server seems responsive and is by no means slow, it's only Apache that is going crazy.

I've installed the DDoS-Deflate thinking that this may be an attack, but the fact that I can simply restart Apache and everything is fine, makes that unlikely.

UPDATE: Wanted to add that I have the Xen swappiness set to 25 as recommended by the Linode wiki.

UPDATE 2: pparadis recommended switching the kernel config to 2.6.28.3-linode17 (which I've done) so I'll give the server 48 hours and post the results here. Thanks again for everyone's help with this.

Nope, problem not solved. Any more ideas?

Have you done this ?

If you're intent on sticking with Apache, have a look at Caker's config partway down on http://www.linode.com/forums/viewtopic.php?t=4460. Personally, I'd think seriously about switching to Nginx or Lighttpd.

Are you sure that you've tightened up Mysql sufficiently ?

50 max clients with Apache/mod_php on a Linode 720 is unsustainable. Each request takes up ~13MB of memory, so with 50 simultaneous requests you need ~650MB of memory for Apache alone. No wonder your server goes OOM and starts eating up swap (hence the high disk IO).

1. Try setting max clients to something low, such as 15-20.

2. If your site is the only one on this linode, ditch Apache and start using something a little more VPS-friendly, such as lighttpd or nginx. There are plenty of tutorials on the web to help you make the transition, especially with something as popular as Drupal.

Thanks for the help so far everyone. I have used lighttpd in the past, but I much prefer Apache and some of the Drupal modules I use require Apache (unless I want to do a good deal of hackery, which I do not).

My max clients is usually set at 25, I only set it higher for testing purposes. I've done quite a bit of reading over at 2bits and came to use the following settings. Everything was fine until a few weeks ago. In fact, everything is still fine. The site performs wonderfully up until the crashes. The crashes also do not coincide with peak traffic. So it's not like I'm getting traffic spikes that cause this.

MaxKeepAliveRequests 100

KeepAliveTimeout 5

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxClients 25
MaxRequestsPerChild 4000
</IfModule>

I've also installed the latest eaccelerator which helps a lot with memory usage.

MySQL isn't an issue. It's memory usage remains normal, and restarting only Apache fixes the issue.

I have installed Munin today.

You can still both have Lighttpd/Nginx and Apache installed, and let the static files served by the first and proxy php to Apache. At least you static file requests won't eat up as much memory as with Apache.

With the proxying mode You can even set a prepend file in php which eliminates some HTTP headers so it can be transparent.

Also try disabling unnecessary Apache modules, those might add memory usage for each Apache handled request.

BTW what Drupal modules do you use that require Apache? I am using Lighttpd for one of my Drupal sites served by Lighty and FastCGI and I had no problems so far.

I honestly don't remember which modules, but I was running lighttpd for awhile ( a few months ago) and the reason I switched to Apache was because of a couple modules. Going through the trouble of splitting up static pages would do little good because the majority of my traffic is from logged-in users.

As I said before, I have zero issues 99.999% of the time. Apache performs well, and the server runs fine. Then out of nowhere, CPU usage hits 100% (according to top) but the server continues to respond like normal. I mean, everything except Apache responds quickly, and doesn't behave like a server under max CPU load.

A simple Apache restart instantly solves the problem. So, I guess what I'm saying is that I would much rather "fix" the problem and the server running the way it was before than run away from it.

Here is the link to my Munin results. I'll post again when there's another crash: http://aionelysium.com/munin/localdomai ... omain.html

Even if traffic comes from logged-in users, a simple request to a css or image file involves a php-powered apache process costing around 10-30 MB of memory which might be overkill.

Try to check your access and error logs during CPU spikes.

Also enable munin plugins which gets info from apache /server-status pages to see apache processes and volume. (and of course enable server-status from 127.0.0.1 in apache.)

Your problem might be caused by a drupal module as well. Have you checked your drupal logs about any errors...

Please read my original post. No entries in the Apache error log and nothing in the Drupal logs as well. I'll look into the munin plugins tonight. The server generally runs with a good amount of free memory.

total used free shared buffers cached
Mem: 705 544 160 0 20 225
-/+ buffers/cache: 297 407
Swap: 511 4 507

AND as I said, the crashes do not coincide with peak traffic times. Out of the blue apache simply goes crazy. I'm also confused about the CPU usage being at 1005 (or 400% according to Linode charts). I log and can move about the server with no delay. It certainly doesn't behave like a server that's under full load. In fact, I'm also running TeamSpeak on the server, and it continues to run fine the entire time.

My bad I haven't read carefully your original post, sorry.
Let's find an another spike and by then you should be able to see what apache did via the munin plugins.

Just got another crash: http://aionelysium.com/munin/localdomai ... omain.html

Can't miss the massive CPU spike. I see that MySQL spiked as well, so I'll take a look at that. (http://aionelysium.com/munin/localdomai ... eries.html)

BTW I installed munin-plugins-extra package, but do not see anything particular to apache.

UPDATE: Nothing in the MySQL logs.

UPDATE 2: OK, had to manually enable the apache plugins. Done.

Apache plugins need access to http://localhost/server-status?auto otherwise you won't see any data. It uses the ExtendedStatus feature of Apache.