Hello again.

So, I'm having some huge issues. I've been trying to tune my server with APC and Apache settings since the 1gb memory upgrade and I'm not having any luck. My server runs about 15 wordpress sites all running w3 total cache. All the sites are very low traffic (the one with the highest is about 200 visitors a day, the rest are like 40-50).
I keep getting [notice] child pid xxxxx exit signal Segmentation fault (11) errors and my APC fragmentation is super high (right now at 44%) with 99% hit rate. I changed the TTL to 0 and it skyrocketed to 90% fragmentation so I put it back 7200. I also occasionally get "server reached MaxClients setting, consider raising the MaxClients setting" errors so I tried doing that as well.

ANY help/recommendations/advice to help me tune this properly would be great. I was doing great on my 512 till I started messing around and I don't know enough to get this right.

Thanks so much guys!

------

Here are all my relevant settings.

Running:
Linode 1024
Debain 6 32bit
Apache/2.2.16
PHP 5.3.3-7+squeeze15

----

My current apc settings:

extension = apc.so
apc.enabled=1
apc.shm_segments=1
apc.shm_size=768M
apc.num_files_hint=4096
apc.user_entries_hint=4096
apc.ttl=7200
apc.use_request_time=1
apc.user_ttl=7200
apc.gc_ttl=0
apc.cache_by_default=1
apc.filters = ";"
;apc.mmap_file_mask=/apc.shm.XXXXXX
apc.file_update_protection=2
apc.enable_cli=0
apc.max_file_size=1M
apc.stat=1
apc.stat_ctime=0
apc.canonicalize=0
apc.write_lock=1
apc.report_autofilter=0
apc.slam_defense=0
apc.optimization = 0
----

Mem: 1000 953 47 0 44 613
-/+ buffers/cache: 294 705
Swap: 255 140 115

----
LockFile ${APACHE_LOCK_DIR}/accept.lock
PidFile ${APACHE_PID_FILE}

LimitInternalRecursion 20

TimeOut 10

KeepAlive on

MaxKeepAliveRequests 100

KeepAliveTimeout 3


# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 3
MaxSpareServers 6
ServerLimit 40
MaxClients 40
MaxRequestsPerChild 3000
</IfModule>

<IfModule mod_disk_cache>
CacheDirLevels 1
CacheDirLength 2
</IfModule>


AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

DeflateCompressionLevel 9

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio

LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate

---
Loaded Modules:
core_module (static)
log_config_module (static)
logio_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
alias_module (shared)
auth_basic_module (shared)
authn_file_module (shared)
authz_default_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cgi_module (shared)
cloudflare_module (shared)
deflate_module (shared)
dir_module (shared)
disk_cache_module (shared)
env_module (shared)
expires_module (shared)
headers_module (shared)
mime_module (shared)
negotiation_module (shared)
php5_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
ssl_module (shared)
status_module (shared)

You probably want to lower APC Memory. You're allowing APC to use 75% of all available RAM now. Assuming you also run MySQL/Apache on there, you won't have enough.
Longview might give you information about what causes the segfaults, are you sure it's not going oom?

I had dropped the memory down to 512 and was finding the fragmentation levels also raising but I might have change the TTL number at the same time so that could have accounted for it.

And yes I have mysql in there.

I have longview installed but I'm really have a hell of time sorting out how to read it properly. I do know my apache workers are using large chunks of ram per process (between 40 - 150+ depending) so going out of memory is very possible.

You have a MaxClients of 40, so using 40MB/Process * 40 = 1.6GB, you have 1GB. That is excluding APC cache and MySQL. You should probably simply lower these values and monitor your memory usage. You can use Longview to do it or something like htop will also show your memory usage pretty clearly.

You could also switch to nginx to get rid of Apache's huge memory usage, or MPM_Worker which is also better iirc, not sure on that. However I believe there are some security issues with using MPM_Worker and multiple websites and PHP.

Ah..I'll give that a shot. It was the error message "server reached MaxClients setting, consider raising the MaxClients setting" that made me raise it from 24 to begin with.

As for NGINX, I used it for a year on a different server and I got frustrated always trying to find workarounds to get certain site features to work plus I use Webmin which at the time didn't support nginx very well.

It's certainly something worth considering but in the meantime, I'd really like to work with what i got.
the mpm_worker idea is something worth looking into as well but I know nothing about it.

Here's a little bit about the PHP thread safety with MPM_Worker: http://stackoverflow.com/questions/1623 ... afe-in-php

A solution for this could be PHP-FPM, where you can create separate PHP pools. Though I'm not sure if this will fix your memory issues.

I don't know how webmin works with nginx, but I run Wordpress on nginx which works fine

Turning keep alives off can help the max clients limit problem.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

so I changed my apc memory down to 512 and i'm still getting 32% fragmentation and 90% hit rate. Also still getting the segmentation faults. I also dropped the MaxClients to 25.

my free -m reading was this just now:
Mem: 1000 973 26 0 43 652
-/+ buffers/cache: 277 723
Swap: 255 10 245

I guess the segmentation faults are problems, but is the fragmentation actually an issue? You're getting a 90% hit rate, which seems pretty good to me. Nine out of ten requests are served from the cache.

i thought fragmentation was bad in terms of efficiency and lead to errors?

i'm at 95% fragmentation now

If you are using APC for anything but opcode caching then fragmentation is pretty much a given. Just ignore it.

Also, depending on your needs, Wordpress-MU might be a better fit. There will only be a single copy of the PHP files which will cut WAY back on APC memory usage.

_________________
Jason Litka
Utter Ramblings

lower your APC TTL values and it may help with fragmentation

also you may want to consider updating to latest PHP using the DotDeb Repos and/or switching to nginx/php-fpm (which again you can get latest versions via dotdeb)

more info about dotdeb here: http://www.dotdeb.org/

WP-MU only helps reduce APC usage on the global/opcode cache size but is a worthwhile thought

you should also look to set apc.stat = 0 as well if these are the only php sites on here and you have w3tc setup with APC on all sites and running the latest version of w3tc (it will clear apc caches when you update plugins which is needed for apc.stat=0)

Ok I'm gonna try playing with some more settings and look through my logs to see what might change. I'm so confused about the APC settings and the whole fragmentation concept but my hit % is always in the high 90s and if that is what matters, I'll leave it be. I'll try changing the apc.stat setting. I only left it on 1 because I was told it was needed for w3tc to work properly but yes, I'm using the latest version so I'll give it a whirl.

I'll update with results.

yeah the apc.stat bug is with older versions that didnt clear the opcode caches on config updates (because w3tc stores config in a php file on disk)