So I've setup a Ubuntu 12.04 node running Nginx, PHP-FPM, MySQL etc. to serve about 30 small to medium websites. Yesterday I got an alert saying my outbound traffic was high and not knowing what to do I power cycled it and sure enough it went back to normal. So how would you kind people suggest I analyse what's actually causing that so I can tell the difference between more visitors coming to my sites and someone hacking in and turning it into a spambotnetserver? Looking at the access logs is difficult to do when you don't know which site is responsible and they're split up per site. Is there a way to tell which IP port or process was seeing all the traffic to know if it's actually nginx?

For reference about 20GB of data went out over 2 hours before I rebooted.

Rebooting it may have made tracking it down even harder. The only way to see where all the traffic is going is to catch it in the act using a tool like tcpdump.

It's possible that this was legitimate traffic. However, would you expect your site to generate such a large amount of outbound traffic? On the other hand, it's quite possible that some malicious code was uploaded to your Linode by someone. I'd check recent logins, as well as look for any suspicious files or folders especially in directories where PHP would have write access.

-Tim

_________________
'If debugging is the process of removing bugs, then programming must be the process of putting them in.' //Edsger Dijkstra
'Nothing is withheld from us which we have conceived to do.' | 'Do things that have never been done.' //Russell Kirsch

Thanks thechman - I found in the auth.log repeated attempts to login with root ssh from one IP around that time so I've blocked that IP with apf now.

You should make sure that root logins via ssh are disabled entirely and use SSH Key authentication with a passphrase to login.

Lots of inbound login attempts still wont explain your traffic spike though.

It's happening again now. I'm looking at tcpdump but not really sure what to look for. Any help?

I did disable root login via ssh and use passphrase only. I'm currently looking at NetHogs but the traffic seems to have returned to normal.

If you can catch it again you may want to try running this command and then placing it in a pastebin:

tcpdump -nc 1000 -i eth0 not port 22

We might be able to see exactly what it is.

-Tim

_________________
'If debugging is the process of removing bugs, then programming must be the process of putting them in.' //Edsger Dijkstra
'Nothing is withheld from us which we have conceived to do.' | 'Do things that have never been done.' //Russell Kirsch

Ok, so I saw increased traffic again and captured this: http://paste2.org/p/2052798
I'm not 100% sure it was happening while that was taken as it seems to have stopped again.

I assume that "176.58.103.165" is your Linode's IP? It looks like your Linode is being used as a drone to attack other systems on the Internet. My personal recommended is to redeploy, and move all of your data over as well as audit it while you're moving it. You'll need to determine the attack vector to prevent it from happening again.

-Tim

_________________
'If debugging is the process of removing bugs, then programming must be the process of putting them in.' //Edsger Dijkstra
'Nothing is withheld from us which we have conceived to do.' | 'Do things that have never been done.' //Russell Kirsch

Ok, I found some processes that pointed me to an out of data wordpress install. I've killed the processes, changed the passwords and upgraded wordpress. So far so good. Thanks for your help.