We've been under DDoS attack since Friday. From Friday until around 10 hours ago, I thought it was something else, then ran a netstat command:
netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

Which returns almost 200 IPs trying to connect to the server (http://pastie.org/2410271), the top ones being:
9 86.111.75.8
10 195.245.97.181
10 95.30.133.238
11 178.125.237.157
14 31.43.82.236
14 95.135.19.200
15 178.124.84.155
16 92.112.165.39
18 78.30.228.147
112 2.134.153.23

All of these IPs are in random foreign countries. We get about 30,000 unique visitors per month, 99% in the US, so this must be a DDoS attack.

I have exhausted seemingly all options and I do not know what to do.
-Installed mod_security and mod_evasive.
-Added a rule to IPTABLE.
-Binded mysql to 127.0.0.1

This is my config for mod_evasive:
DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 20
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10
DOSLogDir /var/log/apache2/mod_evasive
DOSEmailNotify alex@rentabilities.com
DOSWhitelist 127.0.0.1

I even created a new linode, and changed our DNS. Right away, the attack started happening on the new linode. So that leads me to believe they're attacking our hostname.

I can't think of any reason anyone would want to attack us. Our hostname is rentabilities.com - we're a startup that makes it easy to rent anything (ala ebay for renting), and as far as I know we don't have any enemies

Has anyone experienced DDoS attacks before and found a way to solve it?

We're on a 4GB linode. Maybe resizing would help, but I somehow doubt it.

Any help is truly appreciated!

Have you taken a look at your Apache logs to see if the IPs are trying to access anything in particular? Maybe that would give a clue as to why they're attacking you.

Thanks for the reply. Good idea. Sample outputs:

2.134.153.23 - - [22/Aug/2011:01:46:37 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
2.134.153.23 - - [22/Aug/2011:01:46:39 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
2.134.153.23 - - [22/Aug/2011:01:46:39 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
2.134.153.23 - - [22/Aug/2011:01:46:43 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
2.134.153.23 - - [22/Aug/2011:01:46:42 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
2.134.153.23 - - [22/Aug/2011:01:46:43 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
2.134.153.23 - - [22/Aug/2011:01:46:43 -0400] "GET / HTTP/1.1" 500 760 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"

I don't see specific files being requested - will have to dig more in the AM.

Your site is currently unavailable to me here in the uk - not sure if that is intentional or not.

I raised this very issue a couple of days ago. I am a small business too and just woke up to this problem.

Please let us know how you get on. Have you had a demand for money or anything?

How this is able to go on is disgraceful and the dirty secret of doing business on the internet. It is run by the mafia's not the service providers is the truth.

Down here ( Sri Lanka ) too !

Thanks for checking. The site's meant to be accessible anywhere, and we're still under dos attack.

Do all the requests use that same Russian Firefox user agent string, or is there some other consistent discriminant? If so, you could put nginx as a proxy in front of Apache, and use it to drop the connections for the bogus requests before they make it to Apache. Google nginx 444.

Configured properly, nginx should be able to handle this type of load without breaking a sweat, especially if it just has to drop the connections as they come in.

What effect is it having on the Linode as a whole, and is your network traffic graph showing anything interesting? Is anything showing up in Apache's error log, specifically about MaxClients?

I'm suspecting it's "just" a whole bunch of connections eating up your available connections, such as what Slowloris would do against Apache+mpm-prefork. Still a bugger, but an important distinction for mitigation strategies.

Is Cloudflare helping things at all?

_________________

/* TODO: need to add signature to posts */

Thanks. I think the requests are coming from several bots (not just the Russian one):


220.255.2.171 - - [22/Aug/2011:12:12:05 -0400] "GET / HTTP/1.1" 200 0 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"
95.57.90.155 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"
31.162.226.245 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
178.123.54.251 - - [22/Aug/2011:12:12:05 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"
178.125.44.203 - - [22/Aug/2011:12:12:05 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"
79.124.194.249 - - [22/Aug/2011:12:11:05 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
79.124.194.249 - - [22/Aug/2011:12:11:05 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
2.54.13.48 - - [22/Aug/2011:12:11:06 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
78.30.228.147 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
178.178.42.124 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
223.207.129.136 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 13600 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
79.124.194.249 - - [22/Aug/2011:12:11:33 -0400] "GET / HTTP/1.1" 200 20480 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
95.30.133.238 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
81.90.238.41 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
93.116.90.49 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
178.90.25.164 - - [22/Aug/2011:12:11:47 -0400] "GET / HTTP/1.1" 200 27360 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
178.123.54.251 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33261 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"
92.133.73.130 - - [22/Aug/2011:12:12:07 -0400] "GET / HTTP/1.1" 200 21900 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"
178.89.47.254 - - [22/Aug/2011:12:12:06 -0400] "GET / HTTP/1.1" 200 33266 "-" "Opera/9.02 (Windows NT 5.1; U; ru)"


I'll check that out!

What's in the mod_evasive log? I'm surprised it's not catching those repetitive GET / requests.

Thanks - you're right.

I did some more digging and found that there wasn't any log entries being created for mod_evasive. So I competely uninstalled it, then re-installed it using this article as a guide:
http://www.linuxlog.org/?p=135

That seems to have at least got it logging and working. In the log I now see:

-rw-r--r-- 1 www-data www-data 5 Aug 22 04:15 dos-110.168.196.227
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-173.245.53.219
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-178.125.44.203
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-79.162.4.89
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-93.116.90.49
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.30.133.238
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.59.196.92

I'd expect this to grow to hundreds of connections in the next few minutes after starting Apache again.

Looks like that log is changing.


-rw-r--r-- 1 www-data www-data 5 Aug 22 04:15 dos-110.168.196.227
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-173.245.53.219
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:01 dos-173.9.107.126
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-178.125.44.203
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:00 dos-77.124.17.60
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-79.162.4.89
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-93.116.90.49
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.30.133.238
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.59.196.92

... and growing:


-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-109.251.20.66
-rw-r--r-- 1 www-data www-data 5 Aug 22 04:15 dos-110.168.196.227
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-118.8.26.167
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-173.245.53.219
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:01 dos-173.9.107.126
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-178.125.100.101
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-178.125.142.124
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-178.125.44.203
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-178.125.68.231
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-178.137.4.41
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-178.216.122.233
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-178.90.192.242
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-180.1.225.47
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-180.31.144.15
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-188.237.65.176
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-188.99.187.65
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-189.239.209.80
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-190.226.214.165
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-190.41.236.215
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-193.151.105.5
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-193.19.228.226
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-193.84.22.10
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-194.44.235.187
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-200.35.36.75
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-210.117.21.95
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-212.142.93.141
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-46.10.64.96
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-46.118.189.216
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-46.162.233.223
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-62.221.107.177
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-62.87.214.17
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-74.7.76.233
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-76.251.218.25
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-76.90.202.228
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-77.101.245.155
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:00 dos-77.124.17.60
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-77.124.67.27
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-77.196.114.94
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-78.94.204.151
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-79.162.4.89
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-79.39.116.172
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:03 dos-81.90.238.41
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-82.104.2.117
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-82.204.95.254
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-82.82.177.184
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-83.251.60.28
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-83.99.181.74
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-84.101.142.64
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-84.229.255.32
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-84.3.52.55
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-84.94.34.25
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-84.94.63.202
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-84.94.75.89
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-85.114.11.98
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-85.64.157.98
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:07 dos-87.20.222.21
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:05 dos-87.91.125.225
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-91.156.164.2
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-91.22.137.172
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-92.119.57.241
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-93.116.90.49
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-94.76.66.32
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:04 dos-95.170.191.6
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.30.133.238
-rw-r--r-- 1 www-data www-data 6 Aug 22 13:06 dos-95.57.70.25
-rw-r--r-- 1 www-data www-data 6 Aug 22 12:53 dos-95.59.196.92

mod_evasive turned out to be evasive, but not evasive enough.

A friend stepped in and helped me make two changes

1. Blocked common user agents.
We added these 4 lines to .htaccess:
BrowserMatch "ru" ddos
BrowserMatch "1.1.4322" ddos
Order Deny,Allow
Deny from env=ddos

2. Setup CloudFlare
http://cloudflare.com
In the Apache config, you have to add CloudFlare's IPs to the whitelist, for example:
DOSWhitelist 127.0.0.1
DOSWhitelist 204.93.240.0
DOSWhitelist 204.93.177.0
DOSWhitelist 199.27.128.0
DOSWhitelist 173.245.48.0
DOSWhitelist 103.22.200.0
DOSWhitelist 141.101.64.0

Will post back here as I progress... hopefully this helps someone else out in the future.

Looks like your site is back up and running

I'm having trouble with it, though. I searched for A/V equipment near Montreal, and got nothing. Then I clicked on the A/V section and saw a bunch of stuff. A projector listed for $249. I clicked on it, and it shows the projector is actually $429, and is "Available from 1 store near Montreal" with the store being in Melbourne, Australia. Which, being 17 thousand kilometers away on a planet where the maximum distance between any two locations is a bit over 20 thousand kilometers, isn't exactly down the street