I'm not sure how to look into this problem to prevent it happening from the future.

I just launched a site on this server and it was running great for about 8 hours. When I woke up and tried to access the site it was unresponsive to both http requests and ssh.

I rebooted the server and it's back up.

What do I do to investigate the problem so it doesn't happen again?

After some investigation it looked like my site went down around 11am

I looked into the apache logs and found this

[Mon Jan 18 11:29:15 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:33:13 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:34:59 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:38:55 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:43:14 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:43:58 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:45:55 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:49:25 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:58:26 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:59:02 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 11:59:32 2010] [notice] child pid 16078 exit signal Segmentation fault (11)
[Mon Jan 18 11:59:39 2010] [notice] child pid 16066 exit signal Segmentation fault (11)
[Mon Jan 18 12:00:15 2010] [notice] child pid 16102 exit signal Segmentation fault (11)
[Mon Jan 18 12:00:41 2010] [notice] child pid 16226 exit signal Segmentation fault (11)
[Mon Jan 18 12:01:21 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:04:03 2010] [notice] child pid 16145 exit signal Segmentation fault (11)
[Mon Jan 18 12:06:42 2010] [notice] child pid 16184 exit signal Segmentation fault (11)
[Mon Jan 18 12:06:55 2010] [notice] child pid 16115 exit signal Segmentation fault (11)
[Mon Jan 18 12:08:53 2010] [notice] child pid 16140 exit signal Segmentation fault (11)
[Mon Jan 18 12:10:02 2010] [notice] child pid 16166 exit signal Segmentation fault (11)
[Mon Jan 18 12:10:47 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:13:19 2010] [notice] child pid 16133 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:29 2010] [notice] child pid 16124 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:29 2010] [notice] child pid 16185 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:29 2010] [notice] child pid 16191 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:29 2010] [notice] child pid 16193 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:52 2010] [notice] child pid 16087 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:53 2010] [notice] child pid 16098 exit signal Segmentation fault (11)
[Mon Jan 18 12:13:59 2010] [notice] child pid 16190 exit signal Segmentation fault (11)
[Mon Jan 18 12:14:19 2010] [notice] child pid 16121 exit signal Segmentation fault (11)
[Mon Jan 18 12:14:56 2010] [notice] child pid 16158 exit signal Segmentation fault (11)
[Mon Jan 18 12:15:02 2010] [notice] child pid 16159 exit signal Segmentation fault (11)
[Mon Jan 18 12:15:05 2010] [notice] child pid 16161 exit signal Segmentation fault (11)
[Mon Jan 18 12:15:06 2010] [notice] child pid 16174 exit signal Segmentation fault (11)
[Mon Jan 18 12:18:04 2010] [notice] child pid 16096 exit signal Segmentation fault (11)
[Mon Jan 18 12:18:33 2010] [notice] child pid 16113 exit signal Segmentation fault (11)
[Mon Jan 18 12:18:56 2010] [notice] child pid 16132 exit signal Segmentation fault (11)
[Mon Jan 18 12:18:57 2010] [notice] child pid 16194 exit signal Segmentation fault (11)
[Mon Jan 18 12:21:53 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:22:55 2010] [notice] child pid 16198 exit signal Segmentation fault (11)
[Mon Jan 18 12:30:12 2010] [notice] child pid 16186 exit signal Segmentation fault (11)
[Mon Jan 18 12:30:29 2010] [notice] child pid 16065 exit signal Segmentation fault (11)
[Mon Jan 18 12:30:42 2010] [notice] child pid 16237 exit signal Segmentation fault (11)
[Mon Jan 18 12:31:18 2010] [notice] child pid 16163 exit signal Segmentation fault (11)
[Mon Jan 18 12:31:26 2010] [notice] child pid 16225 exit signal Segmentation fault (11)
[Mon Jan 18 12:31:46 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:32:13 2010] [notice] child pid 16126 exit signal Segmentation fault (11)
[Mon Jan 18 12:32:13 2010] [notice] child pid 16131 exit signal Segmentation fault (11)
[Mon Jan 18 12:32:13 2010] [notice] child pid 16206 exit signal Segmentation fault (11)
[Mon Jan 18 12:32:44 2010] [notice] child pid 16082 exit signal Segmentation fault (11)
[Mon Jan 18 12:33:02 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Jan 18 12:33:10 2010] [notice] child pid 16297 exit signal Segmentation fault (11)
[Mon Jan 18 12:34:09 2010] [notice] child pid 16183 exit signal Segmentation fault (11)
[Mon Jan 18 12:37:38 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:40:15 2010] [notice] child pid 16213 exit signal Segmentation fault (11)
[Mon Jan 18 12:40:16 2010] [notice] child pid 16242 exit signal Segmentation fault (11)
[Mon Jan 18 12:40:43 2010] [notice] child pid 16245 exit signal Segmentation fault (11)
[Mon Jan 18 12:41:06 2010] [notice] child pid 16093 exit signal Segmentation fault (11)
[Mon Jan 18 12:41:20 2010] [notice] child pid 16241 exit signal Segmentation fault (11)
[Mon Jan 18 12:42:50 2010] [notice] child pid 16137 exit signal Segmentation fault (11)
[Mon Jan 18 12:42:51 2010] [notice] child pid 16236 exit signal Segmentation fault (11)
[Mon Jan 18 12:46:49 2010] [notice] child pid 16162 exit signal Segmentation fault (11)
[Mon Jan 18 12:48:51 2010] [notice] child pid 16277 exit signal Segmentation fault (11)
[Mon Jan 18 12:49:16 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:50:21 2010] [notice] child pid 16282 exit signal Segmentation fault (11)
[Mon Jan 18 12:52:27 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:52:52 2010] [notice] child pid 16250 exit signal Segmentation fault (11)
[Mon Jan 18 12:53:19 2010] [notice] child pid 16295 exit signal Segmentation fault (11)
[Mon Jan 18 12:53:47 2010] [notice] child pid 16280 exit signal Segmentation fault (11)
[Mon Jan 18 12:53:51 2010] [notice] child pid 16281 exit signal Segmentation fault (11)
[Mon Jan 18 12:54:09 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:56:57 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 12:57:25 2010] [notice] child pid 16243 exit signal Segmentation fault (11)
[Mon Jan 18 13:02:11 2010] [notice] child pid 16289 exit signal Segmentation fault (11)
[Mon Jan 18 13:02:18 2010] [notice] child pid 16157 exit signal Segmentation fault (11)
[Mon Jan 18 13:02:18 2010] [notice] child pid 16202 exit signal Segmentation fault (11)
[Mon Jan 18 13:03:10 2010] [notice] child pid 16315 exit signal Segmentation fault (11)
[Mon Jan 18 13:03:30 2010] [notice] child pid 16301 exit signal Segmentation fault (11)
[Mon Jan 18 13:03:40 2010] [notice] child pid 16164 exit signal Segmentation fault (11)
[Mon Jan 18 13:03:40 2010] [notice] child pid 16283 exit signal Segmentation fault (11)
[Mon Jan 18 13:04:13 2010] [notice] child pid 16292 exit signal Segmentation fault (11)
[Mon Jan 18 13:04:26 2010] [notice] child pid 16308 exit signal Segmentation fault (11)
[Mon Jan 18 13:04:52 2010] [notice] child pid 16064 exit signal Segmentation fault (11)
[Mon Jan 18 13:05:07 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Mon Jan 18 13:07:03 2010] [error] (12)Cannot allocate memory: fork: Unable to fork new process

So it seems like the memory ran out. How... do I fix this? I'm no admin guru but would love to learn how to optimize my server against this. Is there a process manager I can install that tries to reboot apache if it's using too much memory? Is that a solution?

For all I know it can be MySql causing the memory issue too – I'm not really sure.

Linode also sent me emails with the following: Your linode, has exceeded the notification threshold (90) for CPU Usage by averaging 349.3% for the last 2 hours.

Your Linode, has exceeded the notification threshold (1000) for disk io rate by averaging 7695.86 for the last 2 hours.

Odds are good you're running Apache and PHP with the default settings; this generally ends poorly, as the configuration assumes an infinitely-large computer of trivial I/O latency.

This page has some adjustments to try for both Apache and MySQL:

http://library.linode.com/troubleshooti ... networking

Yeah so I guess I'm trying to gauge what the default configuration assumes?

I tried the apache settings linode suggested, so far so good. I have not done it for MySQL yet as I wasn't sure if it made as much sense.

So far things are running pretty smoothly though. 300 users online with 188mb of memory left out of 360

Yeah, step one is probably to lower MaxClients in your Apache configuration. Monit is a simple monitoring program that can restart services if they consume too many resources or become unresponsive. Note that server monitoring is not a substitute for properly tuning Apache and MySQL. Munin is a somewhat more sophisticated system for monitoring your system and generating statistics.

How do you feel about the MySql setting tuning? Should I do that as well next to the suggested Apache low memory settings?

Honestly, I haven't really needed to change the MySQL config, so I haven't bothered. The danger with reducing its memory usage is that you will cause it to hit the disk more often. I don't know of any good tuning guides, but surely they must be out there.