DO THIS FIRST. APC is a php cache that is trivial to install and use. Try it before going through the trouble of switching web servers. You'll want it either way.

Until it starts corrupting it's cache randomly.

I installed APC and it seemed to help. However, my server still runs like crap. I still need to set the prefork_mpm maxclients to at least 100 to see decent performance.

The application itself is extremely simple. It's one of those stupid facebook like sites where people just type in a little blurb of text, it's saved in a mysql database, and then they like the page they made and it spreads like crazy. It's not resource intensive or anything at all.

I have whos.amung.us installed and I'm only seeing a peak of around 500 users on at once.

I bought more ram but I know something just flat out isn't right. This 720 had no problem reaching one million uniques in a day before, and now it's struggling to handle a hundred thousand.

Google isn't helping me much.

Apache is still misconfigured...

You may want to just give up and install lighttpd or nginx or some other web server, since they're configured sanely out of the box. Or perhaps switch to mpm_worker? I'm not that familiar with configuring Apache since I switched to lighttpd years ago.

From a database perspective, is your schema properly indexed? On large data sets, this can be the difference between 1ms per query and 1000ms per query.

You can't use mod_php with worker, as PHP is not threadsafe. You'd need to use mod_fastcgi, and while it's possible, it takes a bit of work to get it almost right, and it's impossible to get it completely right unless you hack mod_fastcgi's code to not kill subprocesses on SIGUSR1.

Yeah, mod_fastcgi is hell to get right. mod_fcgid is just a tiny bit better, but again much more hassle than just switching to nginx or lighttpd.

OP, did you try profiling your slow pages? Add little hooks here and there to measure the time taken to process each stage of page generation (request parsing, database access, templating, etc.) You could use microtime(true), or grab any open-source PHP profiling library. That could help you pinpoint the source of the slowness.

Also, check your MySQL tables using the CHECK command. Those beasts tend to get corrupted from time to time. Also try optimize and reindex.

Mod_fcgid doesn't pipeline requests, so your whole multi-slave APC-enabled php tree handles one request at a time. To use fcgid you'd need to use the "dumb" mode with no PHP_FCGI_CHILDREN, and some other opcache that's external to the process. (Hmm, does memcached cache php opcodes, or webpage data?)

It's probably a lot easier to get lighttpd working with fastcgi, since it more or less comes pre-configured. Unless I'm mistaken, it's just a single command to enable the fastcgi module, and I don't think it requires any configuration beyond that.

Turn KeepAlives off. Or at the very least, turn the timeout down to 1 or 2 seconds. You're seeing that behavior because connections are being opened and Apache has to sit there waiting for about 15 seconds after it finishes sending data because the client might send another request down the same connection - this ties up that whole Apache process so it can't serve other requests. Turning off KeepAlives causes Apache to serve the data to the client, then close the connection so it can move on to serving up the next client's request.

-James

I notice each of the Apache processes was using 14MB min.

If you run PHP but a lot of requests you receive are non-PHP (ie static files) you should be able to reduce this to about 3 or 4MB each for non-PHP requests by switching from mod_php to mod_fcgid and using PHP via FastCGI. Note that you actually have to disable the mod_php module though. You can then also switch your MPM from prefork to worker which can save a little overall overhead with Apache.

Takes a bit of setting up - maybe practice in a virtual machine. Ah yes, I notice that others above have complained about mod_fcgid being difficult to set up - note that you can solve it though, you just have to make sure PHP does not spawn any threads, and all that is handled by mod_fcgid. mod_fcgid is not compatible with PHP's behaviour of spawning fastcgi threads itself, mod_fcgid needs to do that.

Thank you for all the help everyone.

I stumbled upon the server-status page and found out that I was very easily hitting 50 open connections, all of which were in the keep alive state. I disabled keepalive and saw that number drop to between 2 and 25, but it's hanging in the upper single digits mostly.

It's too early in the night to be stress testing the new setup, but I think this has made a very big difference.

Each process is using a ton of memory, still. top is reporting about 2.8% usage and I have a 720 slice with 360 extra ram.



Yeah, I spent a long time making sure everything was properly indexed. Slow query log is empty (and I set the time to 1 second). The problem is not with pages rendering slowly, but actually being served. I have timer code at the start and end of every PHP page. If a page takes 5 seconds to load, it still reports being generated in about .01 seconds. I thought that was enough to rule out my application's performance.



James this advice was A+. however, when I look at server-status page, it looks like everyone is in a state of waiting for the connection. Is this normal and acceptable? I'm guessing it is because instead of waiting around, apache is serving a page and moving on as soon as it's done.



I have a nearly identical VM locally I test on. I think I will try some different servers and see what I can do. I'm trying to run this stupid Facebook like page thing and a pretty complex startup website all by myself. Instead of fixing bugs and improving things I spend most of my time fiddling with the server. Not very helpful and there's a lot more better things I should be doing instead of breaking my server more and more while users say the site is slow and sucky.