Linode Forum

zlochko · Joined: 15 Jan 2012 Posts: 9

Hello everyone.
Please try to help me before my client helps him self with a sharp knife. :D

I'm on a linode512. Having Apache/PHP/MySql/Wordpress site with an ftp server (turned off) and postfix/dovecot combination.
Please look at what is going on with my server graphs the last few hours:

As you can see I've even tried to restart the server after restarting httpd & mysqld (yeah I know it's stupid, but I had a client yelling over the phone so I was looking for a quick fix)... obviously it didn't help.

Code: netstat -an | grep :80 gave me some unusual number of open connections, so I became suspicious of a DOS or DDOS attack and started googling. I found the following command which should give me the number of connections per IP address:

Code: netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

This is the output:

Code: 1 146.255.65.245
1 213.149.150.251
1 31.11.72.38
1 31.11.73.110
1 31.11.84.175
1 46.217.69.242
1 46.217.94.33
1 77.199.39.207
1 77.28.13.29
1 77.29.134.127
1 77.29.150.201
1 77.29.201.128
1 77.29.5.110
1 79.125.129.119
1 79.125.139.83
1 79.125.148.229
1 79.125.169.178
1 79.125.179.183
1 79.125.193.7
1 79.126.132.200
1 79.126.193.104
1 79.126.243.230
1 79.126.251.213
1 89.205.7.113
1 89.205.7.72
1 92.53.0.195
1 92.53.15.71
1 95.156.52.17
1 95.86.52.159
2 195.26.144.127
2 77.28.12.63
2 77.28.156.97
2 77.28.28.247
2 77.29.173.157
2 77.29.79.7
2 78.157.2.114
2 78.157.27.0
2 79.125.197.222
2 79.125.234.208
2 79.126.208.111
2 79.126.214.155
2 79.126.217.97
2 89.185.195.212
2 89.185.221.229
2 89.205.14.161
3 151.67.110.91
3 31.11.79.4
3 46.217.116.213
3 46.217.3.153
3 62.162.205.250
3 77.28.35.111
3 77.28.98.134
3 77.29.216.94
3 77.29.51.90
3 89.205.42.167
3 92.55.104.182
4 46.217.12.13
4 77.28.149.35
4 79.141.120.118
4 87.4.65.169
4 92.53.62.25
5 77.28.114.120
5 77.29.208.192
5 79.141.125.231
6 78.157.17.3
6 79.125.198.179
15 0.0.0.0
1816

Would you look at that LAST row. What is that number ?!?
Can I find out what that is??

Now, I know I can optimize the apache & mysql configuration up until a certain point. This is my current configuration which worked great up until what happend few hours ago:

Code:
thread_cache_size = 4
tmp_table_size = 32M
max_heap_table_size = 32M

max_write_lock_count = 1
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 16M

Code:
<IfModule prefork.c>
StartServers 3
MinSpareServers 3
MaxSpareServers 8
ServerLimit 12
MaxClients 12
MaxRequestsPerChild 1000
</IfModule>

I'm also logging slow MySQL queries for an hour now, and the file is still empty...

So at this point I really don't know what to do. And what are those 1816 which are apparently coming from nowhere...
Please..... help..............

drpks · Joined: 20 Dec 2010 Posts: 55

Those 1816 are IPv6 listen ports. You can check them with Code: netstat -anp | grep ':::' and see what processes use those ports.

zlochko · Joined: 15 Jan 2012 Posts: 9

This is what I get...

Code:
tcp 0 0 :::993 :::* LISTEN 1888/dovecot
tcp 0 0 :::995 :::* LISTEN 1888/dovecot
tcp 0 0 :::110 :::* LISTEN 1888/dovecot
tcp 0 0 :::143 :::* LISTEN 1888/dovecot
tcp 0 0 :::80 :::* LISTEN 7594/httpd
tcp 0 0 :::22 :::* LISTEN 1714/sshd
tcp 0 0 :::25 :::* LISTEN 1968/master
tcp 0 0 :::443 :::* LISTEN 7594/httpd
udp 0 0 ::1:123 :::* 1722/ntpd
udp 0 0 fe80::f03c:91ff:fedf:195:123 :::* 1722/ntpd
udp 0 0 :::123 :::* 1722/ntpd

drpks · Joined: 20 Dec 2010 Posts: 55

I get 12 "whitespace" and 12 lines(listen ports) with second netstat command. Strange.

Use "ps axu" to check what process/es is/are using CPU (you can paste output here)

zlochko · Joined: 15 Jan 2012 Posts: 9

here it is:

Code:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2924 936 ? Ss 13:21 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 13:21 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/u:0]
root 6 0.0 0.0 0 0 ? S 13:21 0:00 [migration/0]
root 7 0.0 0.0 0 0 ? S 13:21 0:00 [migration/1]
root 8 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/1:0]
root 9 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S 13:21 0:00 [migration/2]
root 11 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/2:0]
root 12 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/2]
root 13 0.0 0.0 0 0 ? S 13:21 0:00 [migration/3]
root 14 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/3:0]
root 15 0.0 0.0 0 0 ? S 13:21 0:00 [ksoftirqd/3]
root 16 0.0 0.0 0 0 ? S< 13:21 0:00 [khelper]
root 17 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/u:1]
root 21 0.0 0.0 0 0 ? S 13:21 0:00 [xenwatch]
root 22 0.0 0.0 0 0 ? S 13:21 0:00 [xenbus]
root 148 0.0 0.0 0 0 ? S 13:21 0:00 [sync_supers]
root 150 0.0 0.0 0 0 ? S 13:21 0:00 [bdi-default]
root 152 0.0 0.0 0 0 ? S< 13:21 0:00 [kblockd]
root 162 0.0 0.0 0 0 ? S< 13:21 0:00 [md]
root 246 0.0 0.0 0 0 ? S< 13:21 0:00 [rpciod]
root 247 0.0 0.0 0 0 ? S 13:21 0:01 [kworker/0:1]
root 279 0.0 0.0 0 0 ? S 13:21 0:02 [kswapd0]
root 280 0.0 0.0 0 0 ? SN 13:21 0:00 [ksmd]
root 281 0.0 0.0 0 0 ? S 13:21 0:00 [fsnotify_mark]
root 285 0.0 0.0 0 0 ? S 13:21 0:00 [ecryptfs-kthrea]
root 287 0.0 0.0 0 0 ? S< 13:21 0:00 [nfsiod]
root 290 0.0 0.0 0 0 ? S 13:21 0:00 [jfsIO]
root 291 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 292 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 293 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 294 0.0 0.0 0 0 ? S 13:21 0:00 [jfsCommit]
root 295 0.0 0.0 0 0 ? S 13:21 0:00 [jfsSync]
root 296 0.0 0.0 0 0 ? S< 13:21 0:00 [xfs_mru_cache]
root 297 0.0 0.0 0 0 ? S< 13:21 0:00 [xfslogd]
root 298 0.0 0.0 0 0 ? S< 13:21 0:00 [xfsdatad]
root 299 0.0 0.0 0 0 ? S< 13:21 0:00 [xfsconvertd]
root 300 0.0 0.0 0 0 ? S< 13:21 0:00 [glock_workqueue]
root 301 0.0 0.0 0 0 ? S< 13:21 0:00 [delete_workqueu]
root 302 0.0 0.0 0 0 ? S< 13:21 0:00 [gfs_recovery]
root 303 0.0 0.0 0 0 ? S< 13:21 0:00 [crypto]
root 865 0.0 0.0 0 0 ? S 13:21 0:00 [khvcd]
root 979 0.0 0.0 0 0 ? S< 13:21 0:00 [kpsmoused]
root 1002 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/2:1]
root 1007 0.0 0.0 0 0 ? S 13:21 0:01 [kjournald]
root 1029 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/1:1]
root 1036 0.0 0.0 0 0 ? S 13:21 0:00 [kauditd]
root 1045 0.0 0.0 0 0 ? S 13:21 0:00 [kworker/3:1]
root 1081 0.0 0.0 2656 248 ? S<s 13:21 0:00 /sbin/udevd -d
root 1300 0.0 0.0 2652 252 ? S< 13:22 0:00 /sbin/udevd -d
root 1518 0.0 0.0 0 0 ? S 13:22 0:01 [flush-202:0]
root 1576 0.0 0.0 2864 4 ? Ss 13:22 0:00 /sbin/dhclient -1 -q -cf /etc/dhcp/dhclient-eth0.conf -
root 1622 0.0 0.0 2092 428 ? Ss 13:22 0:00 /sbin/portreserve
root 1629 0.0 0.1 30668 816 ? Sl 13:22 0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
dbus 1641 0.0 0.0 3008 176 ? Ss 13:22 0:00 dbus-daemon --system
68 1665 0.0 0.1 6172 740 ? Ss 13:22 0:00 hald
root 1666 0.0 0.0 3820 492 ? S 13:22 0:00 hald-runner
root 1714 0.0 0.0 8512 460 ? Ss 13:22 0:00 /usr/sbin/sshd
ntp 1722 0.0 0.2 5168 1072 ? Ss 13:22 0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root 1888 0.0 0.1 2908 556 ? Ss 13:22 0:01 /usr/sbin/dovecot
dovecot 1890 0.0 0.1 2768 736 ? S 13:22 0:00 dovecot/anvil
root 1891 0.0 0.1 2760 784 ? S 13:22 0:00 dovecot/log
root 1968 0.0 0.3 12480 1860 ? Ss 13:22 0:00 /usr/libexec/postfix/master
postfix 1975 0.0 0.3 12716 1984 ? S 13:22 0:00 qmgr -l -t fifo -u
root 1987 0.0 0.1 5904 648 ? Ss 13:22 0:00 crond
root 1999 0.0 0.0 2932 204 ? Ss 13:22 0:00 /usr/sbin/atd
postfix 2007 0.0 0.4 12712 2280 ? S 13:22 0:00 tlsmgr -l -t unix -u
root 2023 0.0 0.3 16532 1684 ? Ss 13:22 0:00 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/use
root 2080 0.0 0.5 15916 2996 ? Ss 13:22 0:01 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webm
root 2091 0.0 0.0 2052 436 hvc0 Ss+ 13:22 0:00 /sbin/agetty /dev/hvc0 38400 vt100-nav
root 2156 0.0 0.4 11596 2252 ? S 13:24 0:00 sshd: root@pts/0
root 2158 0.0 0.3 5296 1540 pts/0 Ss 13:24 0:00 -bash
postfix 7553 0.0 0.3 12556 1856 ? S 15:02 0:00 pickup -l -t fifo -u
root 7594 0.0 1.2 49084 6216 ? Ss 15:02 0:00 /usr/sbin/httpd
root 8908 0.0 0.2 5160 1172 pts/0 S 15:30 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql -
mysql 9027 57.0 5.7 63800 29188 pts/0 Sl 15:30 32:27 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/m
root 10446 0.0 0.5 11588 2668 ? S 16:04 0:00 sshd: root@pts/2
root 10481 0.0 0.2 4976 1460 ? S 16:05 0:00 dovecot/config
dovecot 10482 0.0 0.3 8756 1996 ? S 16:05 0:00 dovecot/auth
root 10498 0.0 0.3 5296 1640 pts/2 Ss+ 16:05 0:00 -bash
root 10771 0.0 0.1 6352 836 ? S 16:09 0:00 dovecot/ssl-params
root 11195 0.0 0.4 8528 2424 ? S 16:16 0:00 dovecot/auth -w
apache 11343 16.2 5.3 59744 27476 ? R 16:19 1:21 /usr/sbin/httpd
apache 11345 18.9 5.3 58648 27432 ? D 16:19 1:34 /usr/sbin/httpd
apache 11346 16.6 5.3 59772 27388 ? S 16:19 1:23 /usr/sbin/httpd
apache 11362 17.3 5.4 59900 27852 ? R 16:19 1:25 /usr/sbin/httpd
apache 11381 14.2 5.3 59972 27420 ? S 16:19 1:06 /usr/sbin/httpd
apache 11426 14.7 5.4 59768 27664 ? S 16:20 1:01 /usr/sbin/httpd
apache 11449 16.2 5.3 59768 27268 ? S 16:20 1:02 /usr/sbin/httpd
apache 11537 17.4 5.2 59444 26696 ? S 16:21 0:59 /usr/sbin/httpd
apache 11553 11.5 5.2 59784 26692 ? S 16:22 0:36 /usr/sbin/httpd
apache 11564 16.6 5.5 59984 28404 ? S 16:22 0:52 /usr/sbin/httpd
root 11651 0.0 0.4 8528 2424 ? S 16:24 0:00 dovecot/auth -w
apache 11687 13.8 5.2 59760 26732 ? S 16:25 0:18 /usr/sbin/httpd
apache 11705 15.9 5.2 60528 26788 ? S 16:25 0:15 /usr/sbin/httpd
root 11827 17.0 0.0 0 0 ? Z 16:27 0:00 [miniserv.pl] <defunct>
root 11828 0.0 0.2 4928 1044 pts/0 R+ 16:27 0:00 ps axu

zlochko · Joined: 15 Jan 2012 Posts: 9

Now I am mind buffled...

Code:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
1 46.217.108.131
1 79.125.129.160
3 77.28.98.134
15 0.0.0.0
2238

2238?
And the processor is above 250%

sweh · Joined: 13 Apr 2004 Posts: 565

Instead of Code: cut -d: -f1 do Code: sed 's/:[0-9][0-9]*$//'

zlochko · Joined: 15 Jan 2012 Posts: 9

Code:
1 46.217.108.131
1 77.28.80.168
1 84.175.200.44
1 ::ffff:69.171.228.246
1 ::ffff:69.171.228.250
1 ::ffff:69.171.229.251
1 ::ffff:77.28.31.60
1 ::ffff:77.29.141.155
1 ::ffff:77.29.156.44
1 ::ffff:77.29.255.138
1 ::ffff:77.29.33.213
1 ::ffff:78.157.19.142
1 ::ffff:79.125.187.153
1 ::ffff:84.175.200.44
1 ::ffff:95.86.37.239
2 77.28.199.45
2 79.125.173.61
2 ::ffff:46.217.116.213
3 77.28.98.134
3 ::ffff:66.249.72.228
6 41.194.70.130
6 95.86.41.237
7 ::ffff:79.125.140.247
8 ::ffff:31.11.116.176
8 ::ffff:89.205.63.199
8 ::ffff:89.205.80.233
10 ::ffff:31.11.82.141
10 ::ffff:79.125.173.61
11 :::*
11 ::ffff:89.205.15.34
15 0.0.0.0:*
15 ::ffff:79.125.139.83
15 ::ffff:79.125.188.235
15 ::ffff:89.185.220.148
15 ::ffff:92.53.29.60
16 ::ffff:95.86.41.237
20 ::ffff:77.29.37.201
23 ::ffff:77.28.24.51
27 ::ffff:79.126.252.115
33 ::ffff:78.157.26.201
34 ::ffff:77.28.75.254
34 ::ffff:77.29.153.251
37 ::ffff:41.194.70.130
38 ::ffff:77.28.199.45
43 ::ffff:46.217.88.61
45 ::ffff:79.126.255.155
45 ::ffff:85.30.96.3
45 ::ffff:95.86.24.33
47 ::ffff:92.53.28.211
48 ::ffff:78.157.28.125
50 ::ffff:178.249.174.8
50 ::ffff:77.29.106.91
50 ::ffff:92.53.51.163
51 ::ffff:77.29.246.86
51 ::ffff:77.29.68.68
52 ::ffff:78.157.1.56
53 ::ffff:46.217.15.183
53 ::ffff:77.28.108.122
54 ::ffff:77.28.69.86
54 ::ffff:77.28.75.12
55 ::ffff:77.28.144.188
55 ::ffff:77.28.76.104
55 ::ffff:77.28.78.105
55 ::ffff:77.28.80.168
55 ::ffff:77.29.1.107
55 ::ffff:79.125.179.113
55 ::ffff:79.126.248.89
55 ::ffff:92.53.30.229
55 ::ffff:95.180.226.228
56 ::ffff:62.162.52.222
56 ::ffff:77.28.166.225
56 ::ffff:77.28.49.207
56 ::ffff:77.29.67.248
56 ::ffff:77.29.88.169
56 ::ffff:79.125.236.200
56 ::ffff:79.126.244.243
56 ::ffff:89.205.95.15
56 ::ffff:95.180.186.251
57 ::ffff:46.217.69.172
57 ::ffff:79.125.251.232
57 ::ffff:92.55.99.216
60 ::ffff:89.185.214.17
61 ::ffff:92.53.16.37
90 ::ffff:89.205.30.213

Is this a DDOS ???

hoopycat · Posted: Sun Jan 15, 2012 12:11 pm Post subject:

What's your site's log file (access.log) say? It doesn't look necessarily look like an attack, it could just be heavy traffic. It's difficult to tell the two apart, but based on what's spiking (traffic, PHP CPU, and MySQL CPU), my gut is saying it's probably legit traffic...

You're using some sort of caching plugin for WordPress, right? MySQL seems to be grinding more than it should.

obs · Joined: 07 Mar 2010 Posts: 1403 Location: Earth

Also have a go at running http://mysqltuner.pl/mysqltuner.pl to help optimise mysql

zlochko · Joined: 15 Jan 2012 Posts: 9

obs wrote: Also have a go at running http://mysqltuner.pl/mysqltuner.pl to help optimise mysql

That's how I got the configuration posted above. Here is what the script is advizing me:

Code:
-------- Performance Metrics -------------------------------------------------
[--] Up for: 1h 55m 27s (1M q [220.752 qps], 19K conn, TX: 3B, RX: 185M)
[--] Reads / Writes: 74% / 26%
[--] Total buffers: 56.0M global + 2.7M per thread (151 max threads)
[!!] Maximum possible memory usage: 461.8M (92% of installed RAM)
[OK] Slow queries: 0% (0/1M)
[OK] Highest usage of available connections: 12% (19/151)
[OK] Key buffer size / total MyISAM indexes: 8.0M/4.6M
[OK] Key buffer hit rate: 99.6% (4M cached / 17K reads)
[OK] Query cache efficiency: 93.6% (1M cached / 1M selects)
[!!] Query cache prunes per day: 60768
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 15K sorts)
[!!] Temporary tables created on disk: 43% (4K on disk / 10K total)
[OK] Thread cache hit rate: 87% (2K created / 19K connections)
[OK] Table cache hit rate: 31% (56 open / 177 opened)
[OK] Open file limit used: 8% (88/1K)
[!!] Table locks acquired immediately: 94%

-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours - recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Optimize queries and/or use InnoDB to reduce lock wait
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
query_cache_size (> 16M)
tmp_table_size (> 32M)
max_heap_table_size (> 32M)

Now for what is worth, I did not have any caching mechanism installed. Now I have W3 Total Cache.
This is what happend after I deployed the caching configuration:

But to be honest I am not sure which really helped?
Is it the plugin? Or is it the inbound trafic which also went down...?

I remember having only 4-5 connected users no more then two hours ago and the CPU was well over 200%...

hybinet · Joined: 02 May 2008 Posts: 1058

Looks like traffic went down, too. It's been a few hours since your last post, so your graphs will tell a better story.

Grab Apache logs from the heavy-traffic period. See if there's a set of specific URLs that most of the requests are pointing at. Also check for a referer, since this could have been caused by a popular link on Digg/Slashdot/Reddit/etc. If there's an easily recognized pattern, you can tell your client what really happened.

Next step, OPTIMIZE EVERYTHING. Install APC. Install a caching plugin (which you already did) and make it use the most aggressive caching method available. Consider switching PHP to FastCGI and/or changing your web server to nginx so that requests for static files don't clog up your server. Tune MySQL. Get a bigger linode if mysqltuner keeps complaining that there isn't enough RAM.

zlochko · Joined: 15 Jan 2012 Posts: 9

First of all, I would like to thank each and everyone of you who tried to help me.

I've considered the last advice and did the following:
- removed eaccelerator
- installed APC
- installed nginx as a proxy above the apache server
- installed w3 total cache with page cache via opcode, minify via opcode, object cache via opcode and browser cache too.
...and I'll be waiting for some 20 more hours to see what will the mysqltune.pl script advise but I can see that prunes are already down to 0 and before they were 60000+

From the first look of it the site feels considerably faster.
I'll be waiting for the next busy period to determine how will the configuration cope with the traffic and post the results here.

Again, thank you very much! :)

zlochko · Joined: 15 Jan 2012 Posts: 9

Yes, yes, now we are talking! Thank you drpks, sweh, hoopycat, obs & hybinet!

170 concurent connections from 40 different IP addresses at the time this screenshot was taken.

CPU and outgoing traffic WAY to high