Hello everyone.
Please try to help me before my client helps him self with a sharp knife.

I'm on a linode512. Having Apache/PHP/MySql/Wordpress site with an ftp server (turned off) and postfix/dovecot combination.
Please look at what is going on with my server graphs the last few hours:



As you can see I've even tried to restart the server after restarting httpd & mysqld (yeah I know it's stupid, but I had a client yelling over the phone so I was looking for a quick fix)... obviously it didn't help.

netstat -an | grep :80

gave me some unusual number of open connections, so I became suspicious of a DOS or DDOS attack and started googling. I found the following command which should give me the number of connections per IP address:

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

This is the output:

      1 146.255.65.245
      1 213.149.150.251
      1 31.11.72.38
      1 31.11.73.110
      1 31.11.84.175
      1 46.217.69.242
      1 46.217.94.33
      1 77.199.39.207
      1 77.28.13.29
      1 77.29.134.127
      1 77.29.150.201
      1 77.29.201.128
      1 77.29.5.110
      1 79.125.129.119
      1 79.125.139.83
      1 79.125.148.229
      1 79.125.169.178
      1 79.125.179.183
      1 79.125.193.7
      1 79.126.132.200
      1 79.126.193.104
      1 79.126.243.230
      1 79.126.251.213
      1 89.205.7.113
      1 89.205.7.72
      1 92.53.0.195
      1 92.53.15.71
      1 95.156.52.17
      1 95.86.52.159
      2 195.26.144.127
      2 77.28.12.63
      2 77.28.156.97
      2 77.28.28.247
      2 77.29.173.157
      2 77.29.79.7
      2 78.157.2.114
      2 78.157.27.0
      2 79.125.197.222
      2 79.125.234.208
      2 79.126.208.111
      2 79.126.214.155
      2 79.126.217.97
      2 89.185.195.212
      2 89.185.221.229
      2 89.205.14.161
      3 151.67.110.91
      3 31.11.79.4
      3 46.217.116.213
      3 46.217.3.153
      3 62.162.205.250
      3 77.28.35.111
      3 77.28.98.134
      3 77.29.216.94
      3 77.29.51.90
      3 89.205.42.167
      3 92.55.104.182
      4 46.217.12.13
      4 77.28.149.35
      4 79.141.120.118
      4 87.4.65.169
      4 92.53.62.25
      5 77.28.114.120
      5 77.29.208.192
      5 79.141.125.231
      6 78.157.17.3
      6 79.125.198.179
     15 0.0.0.0
   1816 

Would you look at that LAST row. What is that number ?!?
Can I find out what that is??

Now, I know I can optimize the apache & mysql configuration up until a certain point. This is my current configuration which worked great up until what happend few hours ago:

thread_cache_size = 4
tmp_table_size = 32M
max_heap_table_size = 32M

max_write_lock_count = 1
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 16M

<IfModule prefork.c>
StartServers       3
MinSpareServers    3
MaxSpareServers   8
ServerLimit      12
MaxClients       12
MaxRequestsPerChild  1000
</IfModule>

I'm also logging slow MySQL queries for an hour now, and the file is still empty...

So at this point I really don't know what to do. And what are those 1816 which are apparently coming from nowhere...
Please..... help..............

Those 1816 are IPv6 listen ports. You can check them with

netstat -anp | grep ':::'

and see what processes use those ports.

This is what I get...

tcp        0      0 :::993                      :::*                        LISTEN      1888/dovecot        
tcp        0      0 :::995                      :::*                        LISTEN      1888/dovecot        
tcp        0      0 :::110                      :::*                        LISTEN      1888/dovecot        
tcp        0      0 :::143                      :::*                        LISTEN      1888/dovecot        
tcp        0      0 :::80                       :::*                        LISTEN      7594/httpd          
tcp        0      0 :::22                       :::*                        LISTEN      1714/sshd           
tcp        0      0 :::25                       :::*                        LISTEN      1968/master         
tcp        0      0 :::443                      :::*                        LISTEN      7594/httpd          
udp        0      0 ::1:123                     :::*                                    1722/ntpd           
udp        0      0 fe80::f03c:91ff:fedf:195:123 :::*                                    1722/ntpd           
udp        0      0 :::123                      :::*                                    1722/ntpd   

I get 12 "whitespace" and 12 lines(listen ports) with second netstat command. Strange.

Use "ps axu" to check what process/es is/are using CPU (you can paste output here)

here it is:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1   2924   936 ?        Ss   13:21   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S    13:21   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    13:21   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/0:0]
root         5  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    13:21   0:00 [migration/0]
root         7  0.0  0.0      0     0 ?        S    13:21   0:00 [migration/1]
root         8  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/1:0]
root         9  0.0  0.0      0     0 ?        S    13:21   0:00 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    13:21   0:00 [migration/2]
root        11  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/2:0]
root        12  0.0  0.0      0     0 ?        S    13:21   0:00 [ksoftirqd/2]
root        13  0.0  0.0      0     0 ?        S    13:21   0:00 [migration/3]
root        14  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/3:0]
root        15  0.0  0.0      0     0 ?        S    13:21   0:00 [ksoftirqd/3]
root        16  0.0  0.0      0     0 ?        S<   13:21   0:00 [khelper]
root        17  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/u:1]
root        21  0.0  0.0      0     0 ?        S    13:21   0:00 [xenwatch]
root        22  0.0  0.0      0     0 ?        S    13:21   0:00 [xenbus]
root       148  0.0  0.0      0     0 ?        S    13:21   0:00 [sync_supers]
root       150  0.0  0.0      0     0 ?        S    13:21   0:00 [bdi-default]
root       152  0.0  0.0      0     0 ?        S<   13:21   0:00 [kblockd]
root       162  0.0  0.0      0     0 ?        S<   13:21   0:00 [md]
root       246  0.0  0.0      0     0 ?        S<   13:21   0:00 [rpciod]
root       247  0.0  0.0      0     0 ?        S    13:21   0:01 [kworker/0:1]
root       279  0.0  0.0      0     0 ?        S    13:21   0:02 [kswapd0]
root       280  0.0  0.0      0     0 ?        SN   13:21   0:00 [ksmd]
root       281  0.0  0.0      0     0 ?        S    13:21   0:00 [fsnotify_mark]
root       285  0.0  0.0      0     0 ?        S    13:21   0:00 [ecryptfs-kthrea]
root       287  0.0  0.0      0     0 ?        S<   13:21   0:00 [nfsiod]
root       290  0.0  0.0      0     0 ?        S    13:21   0:00 [jfsIO]
root       291  0.0  0.0      0     0 ?        S    13:21   0:00 [jfsCommit]
root       292  0.0  0.0      0     0 ?        S    13:21   0:00 [jfsCommit]
root       293  0.0  0.0      0     0 ?        S    13:21   0:00 [jfsCommit]
root       294  0.0  0.0      0     0 ?        S    13:21   0:00 [jfsCommit]
root       295  0.0  0.0      0     0 ?        S    13:21   0:00 [jfsSync]
root       296  0.0  0.0      0     0 ?        S<   13:21   0:00 [xfs_mru_cache]
root       297  0.0  0.0      0     0 ?        S<   13:21   0:00 [xfslogd]
root       298  0.0  0.0      0     0 ?        S<   13:21   0:00 [xfsdatad]
root       299  0.0  0.0      0     0 ?        S<   13:21   0:00 [xfsconvertd]
root       300  0.0  0.0      0     0 ?        S<   13:21   0:00 [glock_workqueue]
root       301  0.0  0.0      0     0 ?        S<   13:21   0:00 [delete_workqueu]
root       302  0.0  0.0      0     0 ?        S<   13:21   0:00 [gfs_recovery]
root       303  0.0  0.0      0     0 ?        S<   13:21   0:00 [crypto]
root       865  0.0  0.0      0     0 ?        S    13:21   0:00 [khvcd]
root       979  0.0  0.0      0     0 ?        S<   13:21   0:00 [kpsmoused]
root      1002  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/2:1]
root      1007  0.0  0.0      0     0 ?        S    13:21   0:01 [kjournald]
root      1029  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/1:1]
root      1036  0.0  0.0      0     0 ?        S    13:21   0:00 [kauditd]
root      1045  0.0  0.0      0     0 ?        S    13:21   0:00 [kworker/3:1]
root      1081  0.0  0.0   2656   248 ?        S<s  13:21   0:00 /sbin/udevd -d
root      1300  0.0  0.0   2652   252 ?        S<   13:22   0:00 /sbin/udevd -d
root      1518  0.0  0.0      0     0 ?        S    13:22   0:01 [flush-202:0]
root      1576  0.0  0.0   2864     4 ?        Ss   13:22   0:00 /sbin/dhclient -1 -q -cf /etc/dhcp/dhclient-eth0.conf -
root      1622  0.0  0.0   2092   428 ?        Ss   13:22   0:00 /sbin/portreserve
root      1629  0.0  0.1  30668   816 ?        Sl   13:22   0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
dbus      1641  0.0  0.0   3008   176 ?        Ss   13:22   0:00 dbus-daemon --system
68        1665  0.0  0.1   6172   740 ?        Ss   13:22   0:00 hald
root      1666  0.0  0.0   3820   492 ?        S    13:22   0:00 hald-runner
root      1714  0.0  0.0   8512   460 ?        Ss   13:22   0:00 /usr/sbin/sshd
ntp       1722  0.0  0.2   5168  1072 ?        Ss   13:22   0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1888  0.0  0.1   2908   556 ?        Ss   13:22   0:01 /usr/sbin/dovecot
dovecot   1890  0.0  0.1   2768   736 ?        S    13:22   0:00 dovecot/anvil
root      1891  0.0  0.1   2760   784 ?        S    13:22   0:00 dovecot/log
root      1968  0.0  0.3  12480  1860 ?        Ss   13:22   0:00 /usr/libexec/postfix/master
postfix   1975  0.0  0.3  12716  1984 ?        S    13:22   0:00 qmgr -l -t fifo -u
root      1987  0.0  0.1   5904   648 ?        Ss   13:22   0:00 crond
root      1999  0.0  0.0   2932   204 ?        Ss   13:22   0:00 /usr/sbin/atd
postfix   2007  0.0  0.4  12712  2280 ?        S    13:22   0:00 tlsmgr -l -t unix -u
root      2023  0.0  0.3  16532  1684 ?        Ss   13:22   0:00 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/use
root      2080  0.0  0.5  15916  2996 ?        Ss   13:22   0:01 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webm
root      2091  0.0  0.0   2052   436 hvc0     Ss+  13:22   0:00 /sbin/agetty /dev/hvc0 38400 vt100-nav
root      2156  0.0  0.4  11596  2252 ?        S    13:24   0:00 sshd: root@pts/0 
root      2158  0.0  0.3   5296  1540 pts/0    Ss   13:24   0:00 -bash
postfix   7553  0.0  0.3  12556  1856 ?        S    15:02   0:00 pickup -l -t fifo -u
root      7594  0.0  1.2  49084  6216 ?        Ss   15:02   0:00 /usr/sbin/httpd
root      8908  0.0  0.2   5160  1172 pts/0    S    15:30   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql -
mysql     9027 57.0  5.7  63800 29188 pts/0    Sl   15:30  32:27 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/m
root     10446  0.0  0.5  11588  2668 ?        S    16:04   0:00 sshd: root@pts/2 
root     10481  0.0  0.2   4976  1460 ?        S    16:05   0:00 dovecot/config
dovecot  10482  0.0  0.3   8756  1996 ?        S    16:05   0:00 dovecot/auth
root     10498  0.0  0.3   5296  1640 pts/2    Ss+  16:05   0:00 -bash
root     10771  0.0  0.1   6352   836 ?        S    16:09   0:00 dovecot/ssl-params
root     11195  0.0  0.4   8528  2424 ?        S    16:16   0:00 dovecot/auth -w
apache   11343 16.2  5.3  59744 27476 ?        R    16:19   1:21 /usr/sbin/httpd
apache   11345 18.9  5.3  58648 27432 ?        D    16:19   1:34 /usr/sbin/httpd
apache   11346 16.6  5.3  59772 27388 ?        S    16:19   1:23 /usr/sbin/httpd
apache   11362 17.3  5.4  59900 27852 ?        R    16:19   1:25 /usr/sbin/httpd
apache   11381 14.2  5.3  59972 27420 ?        S    16:19   1:06 /usr/sbin/httpd
apache   11426 14.7  5.4  59768 27664 ?        S    16:20   1:01 /usr/sbin/httpd
apache   11449 16.2  5.3  59768 27268 ?        S    16:20   1:02 /usr/sbin/httpd
apache   11537 17.4  5.2  59444 26696 ?        S    16:21   0:59 /usr/sbin/httpd
apache   11553 11.5  5.2  59784 26692 ?        S    16:22   0:36 /usr/sbin/httpd
apache   11564 16.6  5.5  59984 28404 ?        S    16:22   0:52 /usr/sbin/httpd
root     11651  0.0  0.4   8528  2424 ?        S    16:24   0:00 dovecot/auth -w
apache   11687 13.8  5.2  59760 26732 ?        S    16:25   0:18 /usr/sbin/httpd
apache   11705 15.9  5.2  60528 26788 ?        S    16:25   0:15 /usr/sbin/httpd
root     11827 17.0  0.0      0     0 ?        Z    16:27   0:00 [miniserv.pl] <defunct>
root     11828  0.0  0.2   4928  1044 pts/0    R+   16:27   0:00 ps axu

Now I am mind buffled...

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
      1 46.217.108.131
      1 79.125.129.160
      3 77.28.98.134
     15 0.0.0.0
   2238 

2238?
And the processor is above 250%

Instead of

cut -d: -f1

do

sed 's/:[0-9][0-9]*$//'

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)

      1 46.217.108.131
      1 77.28.80.168
      1 84.175.200.44
      1 ::ffff:69.171.228.246
      1 ::ffff:69.171.228.250
      1 ::ffff:69.171.229.251
      1 ::ffff:77.28.31.60
      1 ::ffff:77.29.141.155
      1 ::ffff:77.29.156.44
      1 ::ffff:77.29.255.138
      1 ::ffff:77.29.33.213
      1 ::ffff:78.157.19.142
      1 ::ffff:79.125.187.153
      1 ::ffff:84.175.200.44
      1 ::ffff:95.86.37.239
      2 77.28.199.45
      2 79.125.173.61
      2 ::ffff:46.217.116.213
      3 77.28.98.134
      3 ::ffff:66.249.72.228
      6 41.194.70.130
      6 95.86.41.237
      7 ::ffff:79.125.140.247
      8 ::ffff:31.11.116.176
      8 ::ffff:89.205.63.199
      8 ::ffff:89.205.80.233
     10 ::ffff:31.11.82.141
     10 ::ffff:79.125.173.61
     11 :::*
     11 ::ffff:89.205.15.34
     15 0.0.0.0:*
     15 ::ffff:79.125.139.83
     15 ::ffff:79.125.188.235
     15 ::ffff:89.185.220.148
     15 ::ffff:92.53.29.60
     16 ::ffff:95.86.41.237
     20 ::ffff:77.29.37.201
     23 ::ffff:77.28.24.51
     27 ::ffff:79.126.252.115
     33 ::ffff:78.157.26.201
     34 ::ffff:77.28.75.254
     34 ::ffff:77.29.153.251
     37 ::ffff:41.194.70.130
     38 ::ffff:77.28.199.45
     43 ::ffff:46.217.88.61
     45 ::ffff:79.126.255.155
     45 ::ffff:85.30.96.3
     45 ::ffff:95.86.24.33
     47 ::ffff:92.53.28.211
     48 ::ffff:78.157.28.125
     50 ::ffff:178.249.174.8
     50 ::ffff:77.29.106.91
     50 ::ffff:92.53.51.163
     51 ::ffff:77.29.246.86
     51 ::ffff:77.29.68.68
     52 ::ffff:78.157.1.56
     53 ::ffff:46.217.15.183
     53 ::ffff:77.28.108.122
     54 ::ffff:77.28.69.86
     54 ::ffff:77.28.75.12
     55 ::ffff:77.28.144.188
     55 ::ffff:77.28.76.104
     55 ::ffff:77.28.78.105
     55 ::ffff:77.28.80.168
     55 ::ffff:77.29.1.107
     55 ::ffff:79.125.179.113
     55 ::ffff:79.126.248.89
     55 ::ffff:92.53.30.229
     55 ::ffff:95.180.226.228
     56 ::ffff:62.162.52.222
     56 ::ffff:77.28.166.225
     56 ::ffff:77.28.49.207
     56 ::ffff:77.29.67.248
     56 ::ffff:77.29.88.169
     56 ::ffff:79.125.236.200
     56 ::ffff:79.126.244.243
     56 ::ffff:89.205.95.15
     56 ::ffff:95.180.186.251
     57 ::ffff:46.217.69.172
     57 ::ffff:79.125.251.232
     57 ::ffff:92.55.99.216
     60 ::ffff:89.185.214.17
     61 ::ffff:92.53.16.37
     90 ::ffff:89.205.30.213

Is this a DDOS ???

What's your site's log file (access.log) say? It doesn't look necessarily look like an attack, it could just be heavy traffic. It's difficult to tell the two apart, but based on what's spiking (traffic, PHP CPU, and MySQL CPU), my gut is saying it's probably legit traffic...

You're using some sort of caching plugin for WordPress, right? MySQL seems to be grinding more than it should.

_________________

/* TODO: need to add signature to posts */

Also have a go at running http://mysqltuner.pl/mysqltuner.pl to help optimise mysql

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

That's how I got the configuration posted above. Here is what the script is advizing me:

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1h 55m 27s (1M q [220.752 qps], 19K conn, TX: 3B, RX: 185M)
[--] Reads / Writes: 74% / 26%
[--] Total buffers: 56.0M global + 2.7M per thread (151 max threads)
[!!] Maximum possible memory usage: 461.8M (92% of installed RAM)
[OK] Slow queries: 0% (0/1M)
[OK] Highest usage of available connections: 12% (19/151)
[OK] Key buffer size / total MyISAM indexes: 8.0M/4.6M
[OK] Key buffer hit rate: 99.6% (4M cached / 17K reads)
[OK] Query cache efficiency: 93.6% (1M cached / 1M selects)
[!!] Query cache prunes per day: 60768
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 15K sorts)
[!!] Temporary tables created on disk: 43% (4K on disk / 10K total)
[OK] Thread cache hit rate: 87% (2K created / 19K connections)
[OK] Table cache hit rate: 31% (56 open / 177 opened)
[OK] Open file limit used: 8% (88/1K)
[!!] Table locks acquired immediately: 94%

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Optimize queries and/or use InnoDB to reduce lock wait
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (> 16M)
    tmp_table_size (> 32M)
    max_heap_table_size (> 32M)

Now for what is worth, I did not have any caching mechanism installed. Now I have W3 Total Cache.
This is what happend after I deployed the caching configuration:



But to be honest I am not sure which really helped?
Is it the plugin? Or is it the inbound trafic which also went down...?

I remember having only 4-5 connected users no more then two hours ago and the CPU was well over 200%...

Looks like traffic went down, too. It's been a few hours since your last post, so your graphs will tell a better story.

Grab Apache logs from the heavy-traffic period. See if there's a set of specific URLs that most of the requests are pointing at. Also check for a referer, since this could have been caused by a popular link on Digg/Slashdot/Reddit/etc. If there's an easily recognized pattern, you can tell your client what really happened.

Next step, OPTIMIZE EVERYTHING. Install APC. Install a caching plugin (which you already did) and make it use the most aggressive caching method available. Consider switching PHP to FastCGI and/or changing your web server to nginx so that requests for static files don't clog up your server. Tune MySQL. Get a bigger linode if mysqltuner keeps complaining that there isn't enough RAM.

First of all, I would like to thank each and everyone of you who tried to help me.

I've considered the last advice and did the following:
- removed eaccelerator
- installed APC
- installed nginx as a proxy above the apache server
- installed w3 total cache with page cache via opcode, minify via opcode, object cache via opcode and browser cache too.
...and I'll be waiting for some 20 more hours to see what will the mysqltune.pl script advise but I can see that prunes are already down to 0 and before they were 60000+

From the first look of it the site feels considerably faster.
I'll be waiting for the next busy period to determine how will the configuration cope with the traffic and post the results here.

Again, thank you very much!

Yes, yes, now we are talking! Thank you drpks, sweh, hoopycat, obs & hybinet!

170 concurent connections from 40 different IP addresses at the time this screenshot was taken.