I am getting hammered on ports 25 and 80. The attacker is most probably spoofing the IP address, and so far I count about 70,000 different IP addresses. Blocking by IP address is not going to work.

The port 80 attack hits http://hgriggs.com with a POST and the POST data is full of crap. I have captured samples with tcpdump and it looks like this:

POST / HTTP/1.1
Accept: */*
Accept-Language: en-us
Content-Type: application/octet-stream
Content-Length: 129
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: hgriggs.com
Connection: Keep-Alive
Cache-Control: no-cache

..].[.\.>UDp.^M..I..-v......].0...........Y.!O...`....;\.)N}....~...J...8.....Y1.....0.%V.....O..G...02.'.&..q.=2A_.... .D..^...8.

I used to have a Wordpress blog on http://hgriggs.com, but I have removed it, so every call gets a 404 or 503 now. At first, it brought the Linode to its knees and I had to reboot. Now that the blog is gone and the 404 or 503 is going back, Apache is handling the constant load adequately.

On port 25, they are probably targeting hgriggs.com. They connect and immediately blast crap at me. I increased sendmail's greet_pause to 15 seconds, so these connections are immediately dropped. However, sendmail occasionally gets overwhelmed and stops accepting connections for a brief while, then starts again. I am not losing legitimate email, but some is being delayed. The data being blasted through contains unfinished snippets of XML, control characters, NULLs, all sorts of crap. Again, the IP addresses are probably spoofed, and there are about 70,000 of them.

At peak times, I see about 40 hits per second, at slow times I see about 5 hits per second, and I don't see any times where there is 0. It's been going on for about 30 hours now, non-stop. I have reduced my logging levels to stop the rapid growth of log files. I have Apache and sendmail handling the load okay for the time being. The Linode is functioning fine, but my traffic is constantly being eroded. I probably won't ever hit my monthly allocation even if they never stop, but it's really irritating and I would like to block it somehow.

I am using iptables to block unwanted traffic. I have a small but simple iptables config that has worked well. However, this new burst of traffic is being ignored because it is coming in from so many IP addresses, that it all looks like legitimate traffic. Is there anything I can do with iptables to block this stuff? I can't see how to block it without blocking the legitmate traffic.

Any suggestions?

As a temporary measure, I set the DNS for hgriggs.com to 127.0.0.1. After four hours, and it propagated out, all port 80 traffic stopped, and almost all port 25 traffic stopped. This coincided with the 48 hour mark for the attack, so I don't know if they gave up or not. I'll set the DNS back to normal value tomorrow and see if the attack comes back. It's an interesting temporary measure.

The MX records for hgriggs.com stayed the same, so all my email continued to come in. I learnt a lot from this whole mess.

Strange. The good news is that the incoming traffic does not count against your quota. You could try to minimize how much outbound traffic Apache generates in response using some ErrorDocument and header trickery.

If this persists, I would suggest opening a ticket with Linode; they may be able to discern some pattern to the attacks, although as you noted with forged source IPs it's not going to be straightforward.

The first thing I did was to minimise the outbound traffic and remove all internal handling of it. So I took the blog down, and got Apache to send back a 404 in 195 bytes each time, and got sendmail to drop the connection immediately it didn't follow protocol. So that was good. I did open a ticket with Linode, but they said I should come to the Forums for suggestions.

I have learned a lot from this experience. I've done a lot of research, and tried a lot of things. The big thing I learned is that when the source IP addresses are spoofed, "you just have to wait it out". It's not over yet, as I have lost the usefulness of hgriggs.com. I've been moving more to subdomains anyway, so I'll move that final blog to a subdomain and keep the primary domain name as a catchall ready for events like this. I'll keep that in mind for future domain names.

I'm not sure why you think the IP addresses are spoofed - spoofing an IP on a TCP connection is very difficult, most likely really impossible unless linode is in on the attack More likely there are really 70,000 IPs out there spamming you.

My first thought was that it was a small botnet. But then it got to 80,000 different IP addresses. Why would a large botnet like that be interested in a tinny little domain like mine? In my research, most people said that it wouldn't be a botnet, it was most probably spoofing the IP addresses. What do I know? So either a botnet with 80,000 PCs whacking at me, or a couple of desktops whacking at me and spoofing the IP addresses, and the end result is the same. They say "you just have to wait it out". I don't care whether it's a botnet or a spoof, I just have to wait it out.

I can't easily block the traffic without blocking legitimate traffic, so sacrificing one domain name to save everything else is okay.

80,000 is a small botnet.

Think how TCP works; to open a session requires a three-way handshake and so the source machine needs to accept the response message. You can't do this by random spoofing; you need the address to route back to the attacker, which either means they already own those 80,000 IP addresses or else they've subverted the routing tables in some way.

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)

Rats. Why would they target me? I'm worthless. There is no benefit to it. It's just a stupid waste of a botnet.

Probably more detail than anyone cares about, but from a historical perspective, the three-way handshake itself didn't preclude such attacks, until changes were made to how the initial sequence number was chosen. The original TCP ISN algorithm was primarily intended to avoid colliding with prior connection packets that might still arrive, but provided relatively protection against prediction. So it has been possible in the past under various conditions to predict the sequence numbers and generate traffic a destination would accept even without being able to receive any of the packets the destination might send back.

RFC1948 introduced the concept of incorporating a cyptographic hash into the ISN algorithm to defeat this, while still continuing to try to ensure no overlap with prior sequences. It turns out a purely random approach didn't actually work very well.

Now, RFC1948 was back in 1996, but it was only informational, and the concept wasn't put on the standards track until RFC6528 in 2012 which officially updates RFC793 (TCP). So it wouldn't surprise me to see stacks still in use that might not implement anything more than the original ISN algorithm from 793.

Linux does have a protective ISN algorithm, but its initial RFC1948-based implementation opted for partial MD4 rather than MD5 for performance reasons, which opened it up to brute force attacks (probably not something usable for the level of connections in this case), increasingly so over time as machines and networks got faster. It wasn't until Aug of 2011 that a patch was made to switch to MD5. I think that showed up in kernels 3.1+

I should also mention that syn cookies come into play in this space as well since they impact the ISN selection to help deter syn flood attacks. But it's only fairly recently that they've been getting turned on by default in some distributions.

I guess the bottom line is that while current Linux TCP implementations should be relatively immune against spoofing attacks, traditionally it was certainly possible, and even in some cases up until fairly recently in more limited scenarios.

-- David

Perhaps they're going after an electronics retailer and are just bad spellers? Sadly, the resources necessary to run a botnet are so small these days that they aren't limited to high-value targets, and can afford to make stupid mistakes and target someone completely random occasionally.

That's pretty funny.

If it was a botnet (which seems most likely to me) there's a reason they got to 80,000+ machines. You and your data might not mean anything to the hacker, but each machine it breaks can add value to its network

Distributed attacks like this are becoming much more common, which is why (IMHO) protective layers like fail2ban are becoming useless.