Hi,

I have spent a week or two researching and setting up my server to run Apache with the Worker MPM and FCID. I am trying to optimize it to allow for the most concurrent connections possible. It has been a nightmare to find good info on the Worker MPM.

Server - 1GB RAM (With Apache turned off its only using about 150MB of RAM) I would like Apache to have a memory usage CAP of about 750MB - so that my server will never run out of RAM.

I have been running the server for about 2 years without any problems - but we have recently started streaming MP3`s and this requires more concurrent connections. The server has also had a few minor DDOS attacks - so I trimmed the settings down a ton to prevent the server from running out of memory - I also added some firewall rules to rate limit.

The set up I have now looks like its working well - but I am getting some Segmentation fault errors

[Sat Mar 23 03:19:50 2013] [notice] child pid 28351 exit signal Segmentation fault (11)
[Sat Mar 23 03:56:20 2013] [notice] child pid 29740 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd.worker: malloc(): memory corruption: 0xb83abdd8 ***

And some Out of Memory Errors

Out of memory during array extend.

I am also just afraid that I have trimmed things down too much!

This is my current set up, I would really appreciate some advice.

Apache Settings:

Timeout 30
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 2
#####################
# Spawn 2 child processes, spawning 25 threads for each child process.
# So, a pool of 50 threads is left up and sleeping, ready to serve incoming requests.
# If more requests will come in, apache will spawn new child processes, each one spawning 25 threads,
# enlarging the thread pool until the total number of threads become 50. In that case, apache begin
# to cleanly drop processes, trying to reach 25 threads.
# New processes and its threads are spawned in case of a large spike of requests, until 200 parallel
# client requests are reached, then apache will no longer accept new incoming connections.
# When the load calm down, and requests come back under 200 parallel connections, apache will continue
# to accept connections. After 25, 000 requests served by a child, q. 1000 per thread, the process
# get closed by the father to ensure no memory leak is fired.
<IfModule worker.c>
ServerLimit      16
StartServers         2
MaxClients       400
MinSpareThreads   25
MaxSpareThreads  50 
ThreadsPerChild    25
MaxRequestsPerChild  1000
ThreadLimit          64 
ThreadStackSize      1048576
</IfModule>
#####################

And then some settings in fcgid.conf

FcgidMinProcessesPerClass 0 
FcgidMaxProcessesPerClass 8 
FcgidMaxProcesses  25
FcgidIdleTimeout 60 
FcgidProcessLifeTime 120 
FcgidIdleScanInterval 30

Any ideas?

What are you hanging off the FCGID? PHP? Python? Vuvuzelas? Custom C program?
What's the config of that "thing", in regards to memory limits and spawning subprocesses?

(Keep in mind that Apache mod_fcgid does not pipeline requests, it assumes each of the FCGI processes it spawned can handle one request at a time. If you're using php-cgi* with the FCGI_CHILDREN option, you should use mod_fastcgi instead.)

* Never used php-fpm, heard the situation is different there though.

_________________
rsk, providing useless advice on the Internet since 2005.

Do you require Apache? While it's possible to configure Apache for decent memory use, it's a lot easier to configure lighttpd or nginx for large numbers of consistent connections.

Your config generally seems oriented at much more RAM than you have. For example, you've got FastCGI configured for 25 processes. If we assume 64MB of RAM each (this is closer to the worst case, but that's what you should plan for), that's 1600MB of RAM for PHP right there. Plus, as far as I can tell, you've got Apache configured for up to 16 processes, if we assume 20MB of RAM for that (I've no idea how much RAM Apache uses), that would give us another 320MB. If you've got a SQL server running on this box as well, double trouble.

So you've basically configured Apache/PHP for a server with somewhere between 2 and 3 gigs of RAM.

It sounds like most of your long-lived connections are direct to Apache, not PHP (streaming static MP3 files), so you can probably afford to drop the number of PHP processes way the heck down. Fitting in 750MB of RAM as a worst case is going to take some sacrifice, so you probably can't afford it to be much beyond 8-10 or so. Might need to drop the number of Apache processes too.

Thanks so much for the reply!

The few suggestions you gave really made alot of different information I read on the internet make sense - I was just not sure what was what.

I have since moved Serverlimit down to 8 Maxclients down to 200 and FCGID down to 10 - will keep an eye on it and see how it goes.

I am also considering adding nginx in from of Apache to handle the static content.

I will keep this updated with what I find and how I update things!

Thanks again!

It's no problem getting everything running in 1GB without resorting to nginx, lighttpd, varnish, etc. All those do is add unneeded complexity.

I've got a box running httpd.worker/event (switched to event a week or so ago), mod_fastcgi + PHP, APC, and MySQL, all in around 650MB. The box is configured for 24 dynamic connections (PHP & MySQL) and 300 static (JS, CSS, images, etc.).

Right now MySQL is running 105MB, httpd 176MB, PHP (including 96MB cache in APC) 273MB. The remaining "core" system services eat the other 100MB or so, leaving around 350MB for buffers/cache and MySQL expansion (my max memory usage with MySQL would be ~200MB @ 24 connections).

_________________
Jason Litka
Utter Ramblings

Unless you need some aspect of Apache (rewrite rules or something), it's better to replace it entirely with nginx (both static and dynamic) rather than a hybrid approach mixing apache/nginx. That just adds unnecessary complexity and memory overhead without any gain, so if possible it should be avoided.

PHP has a default memory limit of 64MB per script, so the fact you've got 24 PHP processes eating only 273MB of RAM is just luck. Get hit with some heavy load/scripting/usage and those PHP processes will try to consume more RAM than you've got.

In my opinion, it's better to set things up taking the worst case into account so that high load causes a graceful failure (some people get through, some people get load errors, the whole box keeps working) rather than a catastrophic failure (RAM usage shoots up so high the entire sever locks up due to swap-death, OOMs cause the kernel to start killing random processes, nobody gets through).

@kiteplans: kudos on lowering ThreadStackSize to a saner value, but I've missed the part where you've put MaxRequestsPerChild 1000... that's a good idea for prefork, but for worker use a very high number or 0 (unlimited).

This is my setup. All this PLUS MySQL is fitting in under half of Linode 512, leaving rest for cache.

apache.conf, mpm_worker section:

    StartServers          2
    ServerLimit          12
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          25
    ThreadsPerChild      25
    MaxClients          300
    MaxRequestsPerChild   0
    ThreadStackSize 1048576

Each of these 25-thread Apache worker processes averages around 8 (eight) MB RAM.
The master, and FCGI process manager add up to another 16 MB or so.

apache.conf, fastcgi section:

        FastCgiConfig \
                -idle-timeout 120 \
                -initial-env PHP_FCGI_CHILDREN=16 \
                -initial-env PHP_FCGI_MAX_REQUESTS=500 \
                -killInterval 100000 \
                -listen-queue-depth 300 \
                -maxClassProcesses 1 \
                -singleThreshold 0

Once again - don't use PHP_FCGI_CHILDREN with mod_fcgid, as it will think there's one parser and will execute one PHP script at a time. Use mod_fastcgi instead.
(And without PHP_FCGI_CHILDREN APC is much less useful because it can't share cache between parsers.)

Also, PHP_FCGI_MAX_REQUESTS is equivalent-ish of MaxRequestsPerChild in prefork+mod_php. PHP isn't good at memory management, so you want the parsers to respawn relatively often to keep memory leaks under control.

php.ini:

memory_limit = 32M
apc.shm_size = 64

Careful: I'm running a PHP-heavy site, so I'm running many parsers in parallel. They usually average to 16MB each.
But as I /need/ to allow the ability of uploading and thumbnailing large pics, the PHP memory limit is set to 32MB. Which means it might be possible to OOM (or at least swapthrash) it by starting a bunch of such image uploads in parallel.

These values would be safe for a Linode 1024, though, I think.

_________________
rsk, providing useless advice on the Internet since 2005.

I've generated heavy load after setting up that dev box, the system is rock stable when being pummeled and does not OOM at all. The highest I've ever seen memory usage during stress testing was a total of ~820MB and that was with 64 clients slapping what turned out to be the most complicated PHP script (as per memory_get_peak_usage). Of course, since only 24 can be serviced at a time, page load time averaged around 5 seconds (with a peak of ~6) instead of the typical 1.0-1.2s, but it still worked.

The point of this is that you need to know your content in order to do a proper job of server tuning.

_________________
Jason Litka
Utter Ramblings

Thank you to everyone for all the information! It has been so helpful! Got to love the Linode Community!

@rsk - I have taken your advice and set

MaxRequestsPerChild 0

I was also reading more online and found many people saying that you should set the Serverlimit setting to the amount of CPU`s you have - so I set

Serverlimit 8

I also had a situation where someone was trying to download every file on our site and I was lucky enough to be online to monitor the server while it was going down. The server performed really well and never peaked over 650MB memory usage - I was even able to set the mod_fcgid settings a bit higher and saw a great increase in the amount of user that the site could serve.

FcgidMinProcessesPerClass 0 
FcgidMaxProcessesPerClass 35 
FcgidMaxProcesses  40
FcgidIdleTimeout 60 
FcgidProcessLifeTime 120 
FcgidIdleScanInterval 30