Checked on my linode today to see how it's going and was surpised to learn that i've consumed 500MB of bandwidth. Outbound traffic has been around 1.8MB, whie inbound is reported as 503MB. According to ifconfig, the numbers are 1.8MB outbound and 380.9MB in bound. TX packets are 36923 while RX packets are 9284497. The box has been up since i signed-up. ( 3days )

What's got me puzzled is how the bandwidth is getting consumed. The only services listening are apache ( 80 + 443 ) and ssh. The dhcp client is listening on port 68 and something was listening on 667, but i can't track down what and it no longer appears to be running after i stopped xinetd, although xinetd is not actually offering up any services, everything is disabled.

Apaches logs only mention a couple of hits, and certainly not enough to warrant somewhere between 300 and 500MB of traffic. The ssh logs only mention a handful of failed connections.

So i'm puzzled. If i did recieve over 9million packets, why didn't the box respond with somewhere around 9million packets. Does that imply 9million inbound syn packets ?

I'm concerned that at this rate, without the box actually doing anything, i'll consume my monthly bandwidth allocation.

I ran tcpdump src or dst host a.b.c.d -n -l on my node to see what packets were coming and going, and there are definately packets bound for my machine, but the usually also involve response which wouldn't explain the 500MB out , 2MB in imbalance. Most of it is ARP and HTTP between linode.com machines.

Did a little bit of investigating:

I ran ifconfig twice, one minute apart, then diff'd the output of the two :

5,6c5,6
< RX packets:9306054 errors:0 dropped:0 overruns:0 frame:0
< TX packets:37061 errors:0 dropped:0 overruns:0 carrier:0
---
> RX packets:9307948 errors:0 dropped:0 overruns:0 frame:0
> TX packets:37072 errors:0 dropped:0 overruns:0 carrier:0
8c8
< RX bytes:401100222 (382.5 Mb) TX bytes:2010823 (1.9 Mb)
---
> RX bytes:401178913 (382.5 Mb) TX bytes:2011773 (1.9 Mb)

So in one minute, i recieved 78000 bytes and 2000 packets.

During that one minute i also ran tcpdump capturing packets that were desinted for or from my IP and only collected 33 packets a handful of which was my ssh session.

I then ran tcpdump capturing all packets, and watched a steady stream of packets go past. This stream seemed to come and go. For a while there was a lot of HTTP activity, second time around alot of netbios scans and HSRPVo-hello packets being broadcast to 224.0.0.2

Since i wrote my original post apparently i've consumed another 3MB! At this rate i'm guessing that in one month i'd consume 5GB of bandwidth even if i wasn't running ANY listening services and never logged into the box. How does this compare with other linode stats ?

I've turned off all listening services, except for the DHCP client, and i'm now upto 514MB. hmmm ....

The control panel says I'm up to 1.53GB Total, that's 1.49GB Incoming and 41.2MB Outgoing.

IPTables shows a lot of traffic hit TCP port 135. What's interesting is that it is almost all coming from 64.2/16 through 64.7/16

$ cat /var/log/messages |grep "SRC" |wc -l
619
$ cat /var/log/messages |grep "DPT=135" |wc -l
600
$ cat /var/log/messages |grep "SRC=64\..*DPT=135" |wc -l
581

I'd assume this is MSBlaster (or related worms) coming from infected machines inside ThePlanet's datacenter. If I turn on logging for echo request (Nachi worm) it logs a lot of traffic from the same networks... looks like ThePlanet has a good number of infected windows servers.

Kinda related and probably a dumb question, but am I corrected that broadcast traffic does not count against the linode's incoming network quota? I'm silently dropping most of it but I do remember seeing a fair amount from dhcp, samba, etc.

kenny

Is blocked traffic counting toward our bandwidth usage?

There are a few issues:

A user on our network who is running smb/nbd and broadcasting UDP packets to everyone on our subnet. I've contacted them to ask them to quit it.

Second is the broadcast traffic problem; right now it is counted in the ip accounting. This is wrong. Ive logged this as a bug and we'll correct it.

Third issue is the network monitoring we do. Check "man nmap", the description for -sP when run as root. It has three methods it uses to see if a host is still up.

The correct solution is for us to update the bandwidth accounting to not include broadcast and local IP traffic.

Until we can get it fixed, don't worry so much about the extra bandwidth; if you go over I'll calculate that in manually.

-Chris

By "local IP traffic" I assume you mean traffic which does not leave the host cluster, i.e. from one linode to another. Is that correct?

Right -- anything that doesn't go through our gateways (out to the net) shouldn't be counted.

-Chris

Basically it does not cost you money if it does not leave the gateway, therefore there is no reason why we should be billed for it.

It does mean one good thing though.

If I was to have 2 linodes, 1 with apache and the other with mysql I would not be billed for traffic between the two linodes.

Adam

Thanks for the respones.

That idea of having two linodes ( one mysql + one apache ) is a bit of smart thinking. You could get a whole bunch of linodes and run your own cluster! The possibilities are endless... ;)

The only real advantage of having them seperate is for security issues.

But that would only really help if Chris would make the IPs assigned to a linode non-routable, so the data would never leave the internal network.

Adam

There would be a performance benefit from separate Linodes, as well. Even if they were on the same host -- since our hosts are dual processor, two Linodes can occupy processor time at the same time.


I'm still on my first cup of coffee, but how would this be different than using the route-able IP range? The traffic would still be switched anyway... Also assume that local-routed traffic isn't counted towards bandwidth usage.

-Chris

It is more that the node can not be accessed via the internet.

The traffic isnt that much of a problem, becuase as you say the internal traffic would not be counted.

It is more the security aspect of it.

Under a standard server set-up, if you wanted a DB server that can not be accecss by any method over the internet, you would set-up a seperate lan for internal traffic between the web servers and the database serves. The web servers been accessible over the internet via the external network.

Under the linode set-up, if the IPs where non-routable the linode could not be accesssed from the internet, but only via another linode or through the console access, thus increasing the security around the node.

Make any more sense?

Adam

Ok, I must have missed that we were talking about Linodes on a private LAN. Good idea

(caker runs for more coffee)

-Chris

Non-routing IPs with a good set of IPTables is just as good as a private LAN.

Private LANs only really come into effect when you have many of your own servers, either dedi or colo, much like what you have for the linnode servers.

Beside private lans came up under the central storage and distributions of linodes.

Non-routing IPs are the cheapest and quickest way of doing things, although private lans are just as useful, it also means that you can issue many more IPs than arin have allowed you to have.

Adam