Hi all; Long time Linode customer, but this is my first time posting here.

Quick background: I'm a self-taught sysadmin and have been managing my company's various servers for ~6 years now. I'm no expert, but am a capable learner and have been able to solve all the issues that have cropped up for us over that time. This particular issue has me at a loss though and I'd appreciate any advice from more experienced folks than I.

Starting ~3.5 days ago (Thursday @ 3:00AM PST) one of our linodes experienced a 37-fold increase in system utilization. That lasted for ~25 minutes, when usage dropped to ~4 times typical levels. During the first 25 minutes, CPU and memory were maxed or nearly so. PHP requests were failing with out-of memory errors and vanilla HTTP requests were also very slow or non-responsive. Once the initial spike subsided, the server has been handling responses without any significant issues.

After researching log files and monitoring services, I found two separate, but simultaneous causes:

1) The SMTP (postfix) server is being flooded with ~110 dropped connections from unique IP addresses every minute:

Oct 11 15:06:09 linode1 postfix/smtpd[31750]: connect from unknown[xxx.xxx.xxx.xxx]
...
Oct 11 15:06:15 linode1 postfix/smtpd[27562]: lost connection after UNKNOWN from unknown[xxx.xxx.xxx.xxx]
Oct 11 15:06:15 linode1 postfix/smtpd[27562]: disconnect from unknown[xxx.xxx.xxx.xxx]

2) A single apache vhost is seeing a flood of ~85 POST requests to the homepage each minute, again from unique IP addresses.

xxx.xxx.xxx.xxx - - [13/Oct/2013:03:35:23 -0400] "POST / HTTP/1.1" 200 15390 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

The number of requests was naturally significantly higher during the initial 25 minutes of the attack. In total, I've found ~70,000 unique IP addresses in both SMTP and HTTP logs.


So, it seems an attacker is spoofing IP addresses to flood SMTP and HTTP ports. To begin with they directed enough traffic to take down services, but without any action on my part, reduced the magnitude of their attack to a point that, though it persists, it is not having a significant impact on resources.


Two questions then: First, can anyone posit a good reason for why an attacker would intentionally persist with an ineffective DDOS attack against a webserver? The only semi-plausible answer I can think of is that this is a blackhat SEO technique to slow server response times enough to impact Google's QOS factor in Google's ranking algos, but I think I need to be wearing a tinfoil hat to seriously consider that possibility.

Second, and more importantly, is there anything I can do to negate their efforts locally? (Assuming I'm correct about the IP spoofing, all my reading on the topic suggests I don't have any options).

Many thanks for any insights!

Best,
David

Probably my best suggestion is to try to make throwing away the bad connections as efficient as possible. To be honest, hundreds of connections a minute (just a few a second) should not really be that heavy a load to handle if all you're doing is rejecting them.

You mention php running into memory errors - that may imply that your current apache configuration permits too many apache processes at peak. Assuming bad requests can be rejected fairly quickly, they really shouldn't interfere much - aside from potentially adding a little latency - with real php processing. So one possibility is that you ended up thrashing memory with too many apache workers which exacerbated the impact of the bad connections. Generally you want to make sure that at peak, your working set remains completely within the available memory of your Linode - there are numerous threads in the forum regarding apache tuning.

If that's tuned reasonably, then another possibility is that you're allowing the bad requests to make it through to your php scripts where they require significant processing to reject. If that's the case (e.g,. a random post to / on your server ends up with full php processing) I'd definitely see if you could lock that down within the web server, or exit out at the php layer as quickly as possible. What restrictions you might put in place to implement that likely depends on the application, but it could be as simple as a cookie check (e.g., it's probably unlikely posts are permitted from random anonymous requests).

Assuming that the bad requests can be rejected at the web server level, then there's still the question of how to do that most efficiently. If you are using mod_php (rather than, for example, php-fpm), that also implies that each apache worker is using more memory than needed, since each includes a php interpreter. So, for example, you could front your apache with a server like nginx which could handle rejecting bad requests (or even servicing requests for static resources) more efficiently - only delegating to apache proper requests that need to use it. Or using php-fpm can accomplish something similar since you separate management of php resources from web server resources, so for example the apache workers themselves become less resource intensive.

You might also want to make sure that your default web processing does not serve your main site, but only does so with the proper server name. Won't help if they're targeting your server by its proper name, but in many cases random attempts won't actually know the official web server name for the address they are targeting, so you can reject them sooner. In nginx, I use a "return 444" block as my default host, which doesn't even bother with niceties, but just abruptly closes the socket.

-- David

See this thread: viewtopic.php?f=19&t=10501

He started getting hit with the same type of attack around the same time you did.

You know now I come to think of it I've seen this kind of activity too (only on port 80 I don't have 25 open). What reminded me was the user agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)". I've not seen it to a scale where it'd cause any performance issues but I've seen a lot of 404 requests where that UA has been scanning for phpmyadmin etc. A quick google suggests that UA isn't legitimate, and if it is it'd be IE6 which no sane person should use any more. So you could block the UA at the Apache level so php wouldn't have to process the request.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

@db3l — Your second guess is correct; These requests are getting through to PHP (we're using Apache with a CGI wrapper), and in this specific case I can filter out the bad requests easily at the Apache config level since the homepage for this vhost doesn't have a legitimate reason to accept POST requests (That's actually what I was just sitting down to do. ) But, this is a multi-tenant server so that approach won't work everywhere (should work great for this specific attack though).

Great suggestion to return 444 on the default host (we just return a lean, branded static page now), but for various reasons it's not practical to use Nginx on this server (I use it elsewhere and love it). I may return a 403 though, as that sounds a bit better than sending out even a minimal static page.

@Main Street James — Excellent find. I hadn't come across that in my research, and there's loads of good info there too. Thank you.

@obs — Good tidbit, and I just did some digging myself based on your notes. I agree on the sane user note, but with >150 sites hosted on this server, I'm hesitantto take that blanket approach until performance is truely impacted.

Thanks again to each of you for the quick and thoughtful replies. Some really helpful comments.

I'm still curious what the purpose of this attack is (They're not actually attempting any exploits, just acting as an annoyance), but it sounds like one of those "who the hell knows" scenarios, so I'll just have to do what I can and be satisfied with bewilderment.

Note that the "return 444" is a nginx-ism. It doesn't actually return an HTTP code 444, but just shuts down the connection at the socket layer immediately without sending any data at all to the client. I don't think Apache has something equivalent, but certainly returning any reasonable HTTP error code would be roughly equivalent.

-- David

Just FYI, TCP-based protocols (which include both SMTP and HTTP) can't be IP-spoofed like UDP protocols can. This is due to the three-way handshake required to start up a TCP socket. These requests are making it all the way to apache (layer 7), and as such, *have* to be real fully-connected sockets.

So rather than a spoofed IP, these requests are likely coming from some sort of botnet.

_________________