Eh, I'm out of my league. Searching Google for those errors revealed various fixes and bug reports, but only one instance where someone said they were leading to pages not loading. In that case the fix was to add "apc.slam_defense = Off;" to php.ini. I didn't see that directive in your posted php.ini, maybe worth some more digging into that error message.

I don't use APC. If it could be the source of your problem, is there a way to flip a switch and run without it to simplify things while you try to identify the source of the issue? Sites running slower than normal would be better than not running at all while you get this solved.

you are OOM'ing no doubt about it.

can you post a screen of HTOP or TOP with the memory usage ordered from highest to lowest.

I will try that. But I don't know, I had APC on for a long time, it's never happened before, and my traffic isn't even that heavy.

My free -m was actually showing that I had free memory, and a lot of it.

total used free shared buffers cached
Mem: 995 823 171 0 192 411
-/+ buffers/cache: 219 775
Swap: 255 0 255

I still had 775 >_<

still OOM?

Please check out www.swagly.com/1.png

any way my mysql is causing this?!

httpd is running around 80mb/instance, which is high.

I'm thinking it is probably something to do with a joomla plugin you have running.

you may have some benefit running nginx and reverse proxy the php to apache, or use lighttpd and fast cgi.

My joomla, sugarCRM, and socialengine websites don't get any traffic. That will also deplete my memory?

That's the thing about mod_php and mpm_prefork. Every interpreter must be equally prepared for any possible HTTP request at any time.

Is squid doing anything in the processing chain? I notice it's running.

_________________

/* TODO: need to add signature to posts */

Sorry, I actually dont know what squid is, or if I've installed it.

Any way to monitor it?

Just a dumb question, but this free output is taken from exactly the moment in time when your requests are hanging right? Memory status can change very rapidly.

If in fact you do have that much memory free, then it certainly doesn't seem like a system-wide OOM, but perhaps some component has its own limit that is being exceeded. I think I'd concur with the other suggestion about seeing about disabling apc for the time being (given that's who seems to be logging about the memory failures). I'm assuming those failures might result in something failing to execute properly, which could then hang a response.

I understand you've been running with apc it for a long time, but something clearly changed recently and at this point I think you're better off having as few components running as you can get away with until you can get a better handle on what is going on. Then you can add stuff back in.

In terms of your mysql question, certainly mysqld is going to need resources, but again if your free output above is representative, you're not hurting for memory right at the moment so I wouldn't worry about that too much. If you want you could make sure you have slow query logging enabled (log-slow-queries and log-query-time parameters), so at least you'll know if mysql is contributing significantly to processing time for requests.

Depending on how hard it is to simplify your configuration, you might even start considering using a second, temporary, Linode for testing. You could clone your current Linode to it (with a few tweaks for network configuration and host name), then use it for testing making it easier to hack and slash the configuration until you've got stability.

Of course, if your primary Linode is essentially unusable at the moment, not sure how much worse it can get using it to experiment in that case.

Bottom line I guess my best overall suggestion at this point is simplify things as much as you can, to as few components as possible in an attempt to achieve basic stability, even if you have to do so at a cost to overall req/s throughput. You've got to reach a baseline that is at least stable since right now I suspect you might have had more than one issue over time contributing to the behavior you have seen. For example, you were clearly OOMing at one point (given your console logs) but now seem to have enough memory, so something else is happening.

Once you have some sort of baseline, you can start tweaking parameters and adding back components while continuing to stress the system and watching stats (cpu, memory, I/O, log files, etc..) and hopefully once it becomes unstable again, have a decent chance at identifying the component or resource that is the root cause.

-- David

David,

I did the free -m when my apache was hanging, so I do think I probably wasn't OOMing?

Thank you so much for your feedback. I had just stopped squid from running, (thanks hoopycat) and I'm going to see if my apache is going to stall again.

I'll wait for it to stall again, and monitor it again and change the parameters again.

And I've adjusted Max Clients according to your suggestion, but they werent making a difference...and it seems like, it was almost always taking the same amount of time for my server to stall. After i restart apache2 it'd work for a day or so, then it'd stall again. weirdness.

Edward

Try to set

KeepAlive Off

in apache config.

After half a year of testing, my server still goes down after a month or so. I am able to log into ssh. I checked into htop and terminated most of the processes that are hogging my memory, but even so, I still cannot access any of my web pages. I've changed my maxclients to very low...to like 10, and my server is 1GB memory.

Please let me know what else I can do. This server hanging is becoming a nuance! Thanks!

@oz "KeepAlive Off" worked for me Thx!