Lately my site has been getting SYN Flooded. From what I've been reading SYN Cookies should be preventing my site from going down, but it isn't.

Just running "hping3 -i u1000 -S -p 80 <siteip>" is enough to take my site down.

Is it normal for a server with syn cookies enabled to go down with such a simple syn flood attack? If not, any tips on what I could be doing wrong?

I'm a newbie to administrating my own server, so any advice will be very welcome.

Are SYN cookies actually working? While this is happening, you shouldn't see the connections in SYN_RECV state on your server. (netstat -nt)

Also, you're trying the hping3 from somewhere that isn't behind a firewall or NAT, correct? I just tried to reproduce this here, but my NAT promptly went apepoop and stopped the flood. This made it look like my site was down from here, but only over IPv4... IPv6 from here and IPv4 from everywhere else was fine.

Not before my target syslog'd this, though:

/var/log/remote-2600:3c03::f03c:<redacted>.log:Oct 13 14:04:11 2600:3c03::f03c:<redacted> kernel: TCP: Possible SYN flooding on port 80. Sending cookies.

I'd bypass my stateful firewalling and try it over IPv6, but hping3 doesn't do IPv6 and I've got better things to do with my day

Oh, and this is perhaps related: stateful firewalling might defeat SYN cookies, as you can't have stateful firewalling without maintaining state and SYN cookies work by not maintaining state. I don't know if this is an actual problem or not, but it's worth checking the conntrack tables to see if they're blossoming.

Hm, then maybe it is not working, since the way in which I detected the syn flood attack was by running that command and seeing that a single ip was generating hundreds of SYN_RECV connections.

However kern.log is filled with messages like this:
kernel: TCP: Possible SYN flooding on port 8080. Sending cookies.

and cat /proc/sys/net/ipv4/tcp_syncookies results in 1

Any clues?

I'm using iptables NAT table to redirect incoming connections in port 80 to 8080

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             anywhere            tcp dpt:www redir ports 8080

Could this be breaking it?

Hmm... quite possibly. Been awhile since I've traced through the mangling workflow, but I suspect it might be gumming up the works. When you had the SYN flood issue, were the SYN_RECVs to port 80 or to port 8080?

Is there any way you can avoid doing that REDIRECT?

_________________

/* TODO: need to add signature to posts */

The syn recvs were to port 8080, or well, this is what netstat showed:

tcp        0      0 <myserverip>:8080     <perpetrator ip>:50549      SYN_RECV

I installed tomcat6 which by default listens on port 8080, I read somewhere that I should redirect port 80 to 8080 so that the http port can be used instead. Perhaps this is a bad practice? If there's some other better way to do this then the answer is yes I can avoid it, but I don't know how right now.

I just tried removing all iptables rules and Syn Flooding port 8080 directly, the site went down anyway.

Hrm. It's possible that Tomcat is choking before the SYN cookies kick in, as well. I'd probably have to do some packet captures and debugging to figure out what exactly is going on.

Binding Tomcat to localhost:8080 and then using nginx or something else as a proxy might help: nothing will then be passed to Tomcat unless it is an actual valid HTTP request. See http://library.linode.com/web-servers/n ... -balancing for some config examples and http://wiki.nginx.org/HttpProxyModule for further stuff. (I was hammering an nginx server when I tested this, and it did not blink.)

How are you verifying that the site is down when you're doing a SYN flood? Does it come back on its own after you stop, or does it need some jiggling?

_________________

/* TODO: need to add signature to posts */

I installed apache and made tomcat run under it, that fixed the problem.

I guess nginx would have also solved the problem too but I had started installing and configuring apache before you posted this so, I'll stick with it

Thanks a lot for the help hoopycat, I really appreciate it!

I guess the lesson learned today is that tomcat's http connector is not good for production servers.

You're welcome! Glad to know it's working.

Sorry, my bad, it is definitely not fixed
I'll keep on doing tests and reporting here.