Hi, everybody!

UPDATE:
Azathoth suggested that the attack maybe Slowloris.
After running a Slowloris attack test, I found that the characteristics were not match. So, these attack remains a mystery for me.
Waiting for more suggestions, everybody. Plz help!

BTW, the attacks were stopped about 20 hours ago.

-------------------- original post --------------------------

My linode is attacked now, and I need some help on this. Any advise is appreciate! Thanks in advance!

After doing some investigation and google staff, I can give some surface descriptions on the attack. The attack began two days ago. It's not stop right now.

1. Basic information

My linode is running CentOS 6 and LAMP server, hosts a wordpress blog and UseBB forum with very low normal traffic.

ps aux information is at the end of this post.

Now, iptables allows only port 80 and ssh connection( not default 22 port). Related part reads like this:

-A INPUT -i lo -j ACCEPT 
# bad ip
-A INPUT -s 65.30.63.120/32 -j DROP 
-A INPUT -s 176.9.84.46/32 -j DROP 
-A INPUT -s 61.147.110.15/32 -j DROP 
-A INPUT -s 117.25.148.110/32 -j DROP 
-A INPUT -s 222.186.36.63/32 -j DROP 
-A INPUT -s 222.214.216.194/32 -j DROP 
-A INPUT -s 218.61.18.253/32 -j DROP 
-A INPUT -s 77.75.77.17/32 -j DROP 
-A INPUT -s 119.84.74.8/32 -j DROP 
-A INPUT -s 60.169.75.161/32 -j DROP 
-A INPUT -s 31.222.129.165/32 -j DROP 
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -m connlimit --connlimit-above 20 --connlimit-mask 32 -j LOG --log-prefix "connlimit blocked: " --log-level 6 
-A INPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -m connlimit --connlimit-above 20 --connlimit-mask 32 -j REJECT --reject-with tcp-reset 
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m limit --limit 50/min --limit-burst 200 -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j LOG --log-prefix "HTTP limit blocked: " --log-level 6 
-A INPUT -j DROP 

2. Attack is in 2 types:

a. Some IPs keep sending request to my port 80.

If not block it with iptables, the

netstat -anp

shows like ( and they just hang there for hours)

tcp        0      0 106.187.50.90:80            174.122.6.252:62841         SYN_RECV    -
tcp        0      0 106.187.50.90:80            174.122.6.252:56394         SYN_RECV    -

and

tcpdump -n

shows like( but there are so many packages like this and could continue for hours)

23:28:33.931729 IP 174.122.6.252.62841 > 106.187.50.90.http: Flags [S], seq 0, win 8192, length 0

This type of attack could cause amount of incoming and outgoing traffic. After banning it in iptables, only incoming traffic remains.

b. Another type of attack is on port 443 (https)

Since port 443 is not open in iptables,

netstat -anpt

shows nothing about this type of attack. But

tcpdump -n

reads like( but there are so many packages like this and could continue for hours):

23:21:18.552302 IP 221.120.194.182.acr-nema > 106.187.50.90.https: Flags [S], seq 0, win 8192, length 0
23:21:18.556223 IP 221.120.194.182.mit-dov > 106.187.50.90.https: Flags [S], seq 0, win 8192, length 0
23:21:18.556239 IP 221.120.194.182.mit-dov > 106.187.50.90.https: Flags [S], seq 0, win 8192, length 0
23:21:18.556247 IP 221.120.194.182.mit-dov > 106.187.50.90.https: Flags [S], seq 0, win 8192, length 0
23:21:18.559609 IP 221.120.194.182.sixxsconfig > 106.187.50.90.https: Flags [S], seq 0, win 8192, length 0
23:21:18.559627 IP 221.120.194.182.sixxsconfig > 106.187.50.90.https: Flags [S], seq 0, win 8192, length 0

This type of attack cause amount of incoming traffic but almost no outgoing traffic. And if ping is allowed in iptables and the ip is not banned, it could cause the website on vps responses very slow.

I've banned several IPs doing the attack. After the ip is banned, another ip came.

Here is a traffic graph related:



ps aux:

# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2   2928  1448 ?        Ss   May22   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S    May22   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    May22   0:01 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    May22   0:11 [kworker/0:0]
root         5  0.0  0.0      0     0 ?        S    May22   0:00 [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    May22   0:00 [migration/0]
root         7  0.0  0.0      0     0 ?        S    May22   0:00 [migration/1]
root         8  0.0  0.0      0     0 ?        S    May22   0:00 [kworker/1:0]
root         9  0.0  0.0      0     0 ?        S    May22   0:00 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    May22   0:00 [migration/2]
root        11  0.0  0.0      0     0 ?        S    May22   0:00 [kworker/2:0]
root        12  0.0  0.0      0     0 ?        S    May22   0:00 [ksoftirqd/2]
root        13  0.0  0.0      0     0 ?        S    May22   0:00 [migration/3]
root        14  0.0  0.0      0     0 ?        S    May22   0:00 [kworker/3:0]
root        15  0.0  0.0      0     0 ?        S    May22   0:00 [ksoftirqd/3]
root        16  0.0  0.0      0     0 ?        S<   May22   0:00 [cpuset]
root        17  0.0  0.0      0     0 ?        S<   May22   0:00 [khelper]
root        18  0.0  0.0      0     0 ?        S    May22   0:00 [kworker/u:1]
root        22  0.0  0.0      0     0 ?        S    May22   0:00 [xenwatch]
root        23  0.0  0.0      0     0 ?        S    May22   0:00 [xenbus]
root       149  0.0  0.0      0     0 ?        S    May22   0:00 [sync_supers]
root       151  0.0  0.0      0     0 ?        S    May22   0:00 [bdi-default]
root       153  0.0  0.0      0     0 ?        S<   May22   0:00 [kblockd]
root       163  0.0  0.0      0     0 ?        S<   May22   0:00 [md]
root       247  0.0  0.0      0     0 ?        S<   May22   0:00 [rpciod]
root       249  0.0  0.0      0     0 ?        S    May22   0:02 [kworker/0:1]
root       280  0.0  0.0      0     0 ?        S    May22   0:00 [kswapd0]
root       281  0.0  0.0      0     0 ?        SN   May22   0:00 [ksmd]
root       282  0.0  0.0      0     0 ?        S    May22   0:00 [fsnotify_mark]
root       286  0.0  0.0      0     0 ?        S    May22   0:00 [ecryptfs-kthrea]
root       288  0.0  0.0      0     0 ?        S<   May22   0:00 [nfsiod]
root       291  0.0  0.0      0     0 ?        S    May22   0:00 [jfsIO]
root       292  0.0  0.0      0     0 ?        S    May22   0:00 [jfsCommit]
root       293  0.0  0.0      0     0 ?        S    May22   0:00 [jfsCommit]
root       294  0.0  0.0      0     0 ?        S    May22   0:00 [jfsCommit]
root       295  0.0  0.0      0     0 ?        S    May22   0:00 [jfsCommit]
root       296  0.0  0.0      0     0 ?        S    May22   0:00 [jfsSync]
root       297  0.0  0.0      0     0 ?        S<   May22   0:00 [xfs_mru_cache]
root       298  0.0  0.0      0     0 ?        S<   May22   0:00 [xfslogd]
root       299  0.0  0.0      0     0 ?        S<   May22   0:00 [xfsdatad]
root       300  0.0  0.0      0     0 ?        S<   May22   0:00 [xfsconvertd]
root       301  0.0  0.0      0     0 ?        S<   May22   0:00 [glock_workqueue]
root       302  0.0  0.0      0     0 ?        S<   May22   0:00 [delete_workqueu]
root       303  0.0  0.0      0     0 ?        S<   May22   0:00 [gfs_recovery]
root       304  0.0  0.0      0     0 ?        S<   May22   0:00 [crypto]
root       866  0.0  0.0      0     0 ?        S    May22   0:00 [khvcd]
root       980  0.0  0.0      0     0 ?        S<   May22   0:00 [kpsmoused]
root       981  0.0  0.0      0     0 ?        S    May22   0:05 [kworker/1:1]
root       984  0.0  0.0      0     0 ?        S    May22   0:05 [kworker/2:1]
root      1009  0.0  0.0      0     0 ?        S    May22   0:01 [kjournald]
root      1034  0.0  0.0      0     0 ?        S    May22   0:04 [kworker/3:1]
root      1042  0.0  0.0      0     0 ?        S    May22   0:00 [kauditd]
root      1082  0.0  0.1   2660   700 ?        S<s  May22   0:00 /sbin/udevd -d
root      1296  0.0  0.1   2656   732 ?        S<   May22   0:00 /sbin/udevd -d
root      1496  0.0  0.0      0     0 ?        S    May22   0:01 [flush-202:0]
root      1547  0.0  0.2  30540  1124 ?        Sl   May22   0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
root      1566  0.0  0.1   8536   932 ?        Ss   May22   0:00 /usr/sbin/sshd
ntp       1574  0.0  0.2   5176  1436 ?        Ss   May22   0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1611  0.0  0.2   5160  1324 ?        S    May22   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --bas
mysql     2037  0.1 10.5 311972 53560 ?        Sl   May22   2:23 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mys
root      2149  0.0  0.4  13116  2120 ?        Ss   May22   0:00 /usr/libexec/postfix/master
postfix   2158  0.0  0.4  13260  2168 ?        S    May22   0:00 qmgr -l -t fifo -u
root      2187  0.0  0.2   5908  1160 ?        Ss   May22   0:00 crond
root      2198  0.0  0.0   2936   364 ?        Ss   May22   0:00 /usr/sbin/atd
root      2214  0.0  0.1   2056   512 hvuser     Ss+  May22   0:00 /sbin/agetty /dev/hvuser 38400 vt100-nav
root      5166  0.0  0.6  11096  3240 ?        S    May23   0:00 sshd: user [priv]
user        5172  0.0  0.2  11236  1284 ?        S    May23   0:02 sshd: user@pts/0
user        5173  0.0  0.3   5164  1600 pts/0    Ss   May23   0:00 -bash
root      5192  0.0  0.2   5456  1488 pts/0    S    May23   0:00 su root
root      5193  0.0  0.3   5296  1680 pts/0    S+   May23   0:00 bash
root     10321  0.0  1.7  92624  8756 ?        Ss   May23   0:00 /usr/sbin/httpd
root     10323  0.0  0.1   5168   992 ?        S    May23   0:00 /usr/sbin/rotatelogs /var/log/httpd/%Y_%m_%d_error_log 86400 480
apache   10344  0.1  5.2  99852 26800 ?        S    May23   0:07 /usr/sbin/httpd
apache   10346  0.2  7.3 100604 37188 ?        S    May23   0:12 /usr/sbin/httpd
apache   10347  0.1  4.7  96304 23956 ?        S    May23   0:10 /usr/sbin/httpd
apache   10349  0.1  5.3 100272 26892 ?        S    May23   0:09 /usr/sbin/httpd
apache   10370  0.1  5.2 100096 26712 ?        S    May23   0:05 /usr/sbin/httpd
apache   10390  0.1  5.7  94952 28944 ?        S    00:07   0:05 /usr/sbin/httpd
root     10403  0.0  0.6  11096  3236 ?        S    00:40   0:00 sshd: user [priv]
user       10405  0.0  0.2  11236  1288 ?        R    00:40   0:00 sshd: user@pts/1
user       10406  0.0  0.3   5292  1644 pts/1    Ss   00:40   0:00 -bash
root     10426  0.0  0.2   5456  1484 pts/1    S    00:41   0:00 su root
root     10427  0.0  0.3   5296  1660 pts/1    S    00:41   0:00 bash
postfix  10438  0.0  0.5  13192  2596 ?        S    00:43   0:00 pickup -l -t fifo -u
apache   10444  0.1  3.8  94892 19764 ?        S    00:50   0:02 /usr/sbin/httpd
apache   10465  0.1  3.8  95096 19312 ?        S    01:03   0:00 /usr/sbin/httpd
root     10469  0.0  0.2   4924  1052 pts/1    R+   01:08   0:00 ps aux

From what I can see from your output, your server does not appear to be under attack. Your charts seem perfectly fine, transferring 200 kb/sec is nothing large. What exactly would suggest that this is an attack as opposed to just standard traffic?

Thanks for your concern, iWizardPro!

I consider those attacks because the connections are not for browsing the websites on this vps(there are no httpd log entries related with those IPs) and they are lasting for several hours. There are not too much contents provided by the websites on this vps demand such a long time or traffic to transfer. At short, the traffic caused by these IPs are not normal.

And usually, visits on these websites should NOT generate too much incoming traffic.

Sounds like Slowloris attack to me. Then again I never experienced it myself because I never had Apache in front, always behind nginx which is not susceptible to this attack.

Edit: This is why monitoring like with Munin would come very handy, you could corelate various metrics to see better what's going on, like if my assumption is correct, all Apache processes being busy with decrease of bandwidth used.

Hi, Azathoth, thank you very much. Your suggestions mean a lot for me. It takes me a long time to understand.

I thought Slowloris may be it, but I need run some tests to confirm, since I can find any request log about those IPs in my Apache logs. I'll let you know right after finishing the tests.

After that, I'll give Nginx a try.

Thanks a lot!

Also worth noting that modern web browsers (particularly Chrome) tend to predictively open connections. In other words, if Chrome thinks I'm going to click on a particular link, it will open a connection just in case. While that may not be the case here, it does happen.

_________________

/* TODO: need to add signature to posts */

Yes, they do. But I think it's kind of good thing, because it reduces our waiting time and does no or less harm to servers.
Obviously it's not the reason of my case. In my case, the connection seems not stop and could continue for hours.

I believe that setting up CloudFlare would also mitigate this problem, since CloudFlare sits between your web server and the internet. This is, after all, the original intended purpose of CloudFlare, before they realized it also sped things up

That might let you block the attack without having to change web servers.

Thank you, Guspaz!

CloudFlare is a great CDN service, but is not always working in my country.