Setup: I have a 2048 Linode and my site has been going along just fine for years. I upgraded from Debian 5 to Debian 7 Wheezy a month ago and it went fine. My site is mostly Apache, PHP, MySQL and is used mostly for phpBB, my own custom PHP-based site/blog, and serving files.

Problem: 3 times in the past 2 days the site stopped responding. CPU was idle, 1.8gb RAM free. Nothing interesting in the Apache, MySQL, or PHP logs. Restarting Apache and MySQL did not help. Rebooting fixed the problem for ~14 hours. syslog shows many, many of these:

localhost kernel: nf_conntrack: table full, dropping packet
localhost kernel: nf_conntrack: table full, dropping packet
localhost kernel: net_ratelimit: 8 callbacks suppressed
localhost kernel: nf_conntrack: table full, dropping packet

After some Google-fu, I found this informative page:
http://antmeetspenguin.blogspot.com/201 ... outer.html
Increasing nf_conntrack_max allowed the site to respond again, but it is possible that whatever the problem was will just cause it to hit the max again. Still, I edited /etc/sysctl.conf and added:

net.netfilter.nf_conntrack_max = 32768
net.netfilter.nf_conntrack_tcp_timeout_established = 86400

I also found this in my sysctl.conf:

net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768    61000

I'm guessing I put this in on some previous tuning a long time ago. I'm not sure if it is any good?

Linode support (those guys are great!) suggested leaving nf_conntrack_max at the default (16384). One theory is something is up with my iptables, possibly "some odd circular iptables entries". I had used arno's firewall script and also installed fail2ban. Here's my iptables:
http://pastebin.com/raw.php?i=hhjd55PE
Very sorry for the formatting, that was copied out of the Lish console while the site was not responding. I don't really know anything about iptables, so I don't know if all that stuff is reasonable. Support suggested rebooting then doing this to reset the iptables:

service fail2ban stop
service arno-iptables-firewall stop
iptables -F 
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

So I did that and the site is running fine, though fully open. I can give it a few days to see if it stops responding. Assuming the site is ok with the iptables reset, what do I do for a permanent solution? I don't want to bother Linode support further, since it's clearly not a problem on their end, so I was hoping maybe someone here could help. I considered paying for Linode Managed, but I feel it would be a waste once my site is working reliably again.

Support mentioned, "If it is working properly, then my guess is that it's arno's script. There's some configuration you can do with it but honestly I haven't seen a case where the script performs better than fail2ban for typical use." Does this mean I should not have used both arno and fail2ban? Is fail2ban alone enough to protect me? All I did with arno is open 6 ports, so I'm not doing anything fancy. I installed fail2ban because I thought it might protect from HTTP basic auth and SSH brute force.

Setup good passwords for apache and only use public key authentication for SSH.

You don't need any of this fail2ban/arno's script craziness.

You can allow SSH, web, NTP, DNS from anywhere, anything from 127/8, and block everything else. Once that works you may figure out what else needs to send or receive packets and add rules for that. Longview will help if you are not already using it, as will 'ss -nptl' or 'netstat -nptl'.

Don't forget SMTP, too, if you're running e-mail for your site on the same node.

Within 24 hours of running with empty iptables, the site stopped responding and was rebooted. I guess the issue isn't with my iptables config. I wonder if it's a SYN flood? Next time I know how to look for that.

Got Longview set up, really nice!

I kept fail2ban and redid my iptables manually, here's my script (no SMTP):
http://pastebin.com/raw.php?i=84g4ZNXE

I read up on sysctl settings and tried to pick values that meet my needs:
http://pastebin.com/raw.php?i=30jfF92x
Any feedback on these? What do you guys do for a 2048 Linode that is just PHP/MySQL/Apache?

I guess that is all I can do. I'll just have to give it some time and see how it goes. I signed up for Linode Managed, so at least they can reboot it when I'm asleep (it only hangs when I'm asleep...) and my users won't be too affected.

Also, here are my Apache settings:
http://pastebin.com/raw.php?i=NcGu8uVM

Made it thru the night (14 hours) and she is still responding. Had a CPU spike for about 1 hour 45 minutes:



Not sure why that happened, but the machine handled it. Seems it was mostly the Apache processes, here is a graph of the workers just at that time:



More charts showing requests Apache receive jumping from 1/s to about 3/s:



Guess this is fine? I have lots of free memory, do you think I should be using it? Using ab to bench I think Apache will use ~1gb max.

One more chart to show MySQL:



MySQL doesn't seem taxed at all. Here's my config:
http://pastebin.com/juFkJbEa
I haven't read about it much, so I'm not sure this is any good. AFAIK all my tables are MyISAM. Is the key_buffer under [isamchk] overwriting key_buffer_size set earlier?

Any feedback on my setup is highly appreciated!