My linode is just a sandbox right now for me to play with things, there shouldn't really be any other traffic than myself visiting. For the first few months my bandwidth IO graphs reflected that, it would be at zero except when I visited or a cron fired off an email.

2 days ago I started getting a sustained 1kb/s in, and 1 out. Now it's up to 1kb/s in and 2 out.

I only have a few scattered search engine spiders in my nginx log files, certainly not enough for sustained traffic 24/7. I noticed more ssh hacking attempts, but I've added a few ip addresses to my firewall and most are getting blocked.

How can I figure out this spike in outbound network traffic?

So far I've viewed netstat -a output (which didn't have anything malicious), and I've checked on the logs to see what's getting updated:

ls -alt /var/log

Am I being paranoid? Want to make sure my box isn't crapping out spam emails or something worse.

Try installing ntop, it has a nice web interface and monitors your traffic you can see what hosts are doing what.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

ntop shows network use, but iotop shows IO, that might be more direct.

iotop is for disk usage, they want network usage hence ntop

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Thanks, I took your advice and installed ntop. I opened up port 3000 just for my ip address, hopefully that's good enough in addition to ntop having a password.

Now I need to learn how to understand all of it's output, I'm off to tutorial land

FYI ntop runs as a daemon in the backgroud (or at least on ubuntu/debian) if you don't want to have it starting up when you boot your linode then remove it with

update-rc.d -f ntop remove

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Oops, didn't catch that, I saw "i/o" and thought disk "i/o", not network activity. I retract my former comment.

Aggressive port scanners? If inbound traffic roughly equals outbound my first guess is firewall reject packets. Try changing default policy to drop (no response to invalid inbound) and see if something changes.

That sounds like a fantastic idea! I'm using ufw on ubuntu 11.04 as a front end to iptables. The default is deny from all, and here is the ruleset:

# ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere                   DENY IN     91.207.234.129
[ 2] Anywhere                   DENY IN     216.176.62.137
[ 3] Anywhere                   DENY IN     222.143.26.249
[ 4] 22                         ALLOW IN    Anywhere
[ 5] 80                         ALLOW IN    Anywhere
[ 6] Apache Full                ALLOW IN    Anywhere
[ 7] 3000                       ALLOW IN    <redacted>

My understanding was these rules and the ufw default of deny all should prevent port scanning network traffice, is there a way for me to check? I'll research how to properly block port scanning, as always any pointers are greatly appreciated. (the open port 3000 is for ntop, recently installed to help me find out my network IO culprit)

run

iptables -L -n -v

It'll output a load of text (which you can put here).

Look for lines with DROP or REJECT in them, especially one starting "Chain INPUT"
If it says "DROP" then it's being dropped and no response is sent, if it says "REJECT" then it means it's sending notification back.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

There is one reject towards the bottom in the Chain ufw-user-limit section

Entire output:

# iptables -L -n -v
Chain INPUT (policy DROP 65 packets, 4400 bytes)
 pkts bytes target     prot opt in     out     source               destination
 991K  148M ufw-before-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 991K  148M ufw-before-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
13393  690K ufw-after-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
12014  621K ufw-after-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
12014  621K ufw-reject-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
12014  621K ufw-track-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ufw-before-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-before-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-after-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-after-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-reject-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination
 912K  189M ufw-before-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 912K  189M ufw-before-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 170K   13M ufw-after-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 170K   13M ufw-after-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 170K   13M ufw-reject-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 170K   13M ufw-track-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-after-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-after-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:137
    0     0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:138
    2    96 ufw-skip-to-policy-input  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:139
   10   496 ufw-skip-to-policy-input  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:445
    0     0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:67
    0     0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:68
    0     0 ufw-skip-to-policy-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW BLOCK] '

Chain ufw-after-logging-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
   71  4648 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW BLOCK] '

Chain ufw-after-logging-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-after-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ufw-user-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-before-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
   46  3885 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
 140K   23M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
   15   656 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           state INVALID
   15   656 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           state INVALID
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 3
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 4
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 11
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 12
    5   172 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 8
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp spt:67 dpt:68
19415 1163K ufw-not-local  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.251         udp dpt:5353
19415 1163K ufw-user-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-before-logging-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-logging-input (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-logging-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-output (1 references)
 pkts bytes target     prot opt in     out     source               destination
   46  3885 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
 137K   28M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
11930  867K ufw-user-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-logging-allow (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW ALLOW] '

Chain ufw-logging-deny (2 references)
 pkts bytes target     prot opt in     out     source               destination
   12   520 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state INVALID limit: avg 3/min burst 10
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 3/min burst 10 LOG flags 0 level 4 prefix `[UFW BLOCK] '

Chain ufw-not-local (1 references)
 pkts bytes target     prot opt in     out     source               destination
19415 1163K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           ADDRTYPE match dst-type LOCAL
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           ADDRTYPE match dst-type MULTICAST
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           ADDRTYPE match dst-type BROADCAST
    0     0 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 3/min burst 10
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-reject-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-reject-input (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-reject-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-skip-to-policy-forward (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-skip-to-policy-input (7 references)
 pkts bytes target     prot opt in     out     source               destination
   12   592 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-skip-to-policy-output (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-track-input (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-track-output (1 references)
 pkts bytes target     prot opt in     out     source               destination
   12   720 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW
11917  866K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW

Chain ufw-user-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       91.207.234.129       0.0.0.0/0
    0     0 DROP       all  --  *      *       216.176.62.137       0.0.0.0/0
 5619  337K DROP       all  --  *      *       222.143.26.249       0.0.0.0/0
 6110  367K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:22
   37  2092 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:80
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:80
    1    48 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport dports 80,443 /* 'dapp_Apache%20Full' */
  187  9724 ACCEPT     tcp  --  *      *       <redacted>        0.0.0.0/0           tcp dpt:3000
    0     0 ACCEPT     udp  --  *      *       <redacted>        0.0.0.0/0           udp dpt:3000

Chain ufw-user-limit (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 3/min burst 5 LOG flags 0 level 4 prefix `[UFW LIMIT BLOCK] '
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-user-logging-forward (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-logging-input (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-logging-output (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

It's not rejects causing your problems since by default you drop packets. Better keep looking!

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

You might need to use 'tcpdump' to capture the traffic over a time period to see what it actually is. 1 kb/sec on a five-minute average is slow enough that it's going to be difficult to measure it statistically. (It's a signal/noise problem... there's some amount of background noise on any network, and you're trying to see if there's a "signal" buried somewhere in there.)

My typical approach is to capture tcpdump to a file (tcpdump -n -i eth0 -s 0 -w foo.pcap), let it run for awhile, then use wireshark to analyze it on a local computer.

_________________

/* TODO: need to add signature to posts */

Figured it out, with the help of ntop!

Turns out it was a columbian IP address that was constantly accessing the site. I'm not sure exactly what they were doing yet (I'm still learning to interpret ntop info), but once I added their IP to the firewall the traffic IO graphs went back to normal (0 except for tiny bumps during cronjob emails to myself and whenever I access the site).

Thanks for your help guys!

Glad you figured it out, if you need help interpreting the ntop data just ask.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue