Hi all,

Since last Wednesday, the traffic on my linode has suddenly doubled, without any reason I can think of.

The linode is used to host a few related web sites. The traffic has stayed stable (between 15 and 22 GB/day) for more than one year. It's now 45 MB/day and going up. Looking at the Apache2 stats, I can see that there is no increase in the number of HTTP requests that would explain this surge.

The only noticeable (but possibly unrelated) thing I have noticed in the linode traffic graphs is that there was an interruption in the traffic shown on the graph on last Wednesday just before the problem started. The host was up at the time of the interruption and had not been rebooted since several days before.

I opened a ticket and asked if there was a way to determine what is using the extra bandwidth (beyond the normal web site traffic), but was answered that Linode didn't monitor traffic, and suggested to ask you experts on the forum, so here I am!

Maybe the Linode has been compromised. There's a chance it's either performing outbound attacks, or is potentially serving files from a location unbeknownst to me. I looked at auth.log and didn't see anything suspicious... but that's not a proof.

Do you have any idea or suggestion? What do you think I could/should do from there?

Any help will be greatly appreciated.

Richard

netstat -Arn will show you all of your current connections. Also check ps aux for any odd processes.

It's possible that while the number of requests has not increased, that you're getting more requests for larger files (most likely media).

I looked but didn't notice any unknown port open or odd process, and there are almost no media files. I'm going to try blocking port 25 (SMTP) tomorrow for a while to see if traffic decreases - but I don't think this is the cause.

I have the monitoring tool ntop installed on my host, and it reports the following warning about a potential "wrong network mask":

Unexpected packets (e.g. traffic to closed port or connection reset)
Wrong network mask

ntop has detected an anomalyous situation with the network mask for a host. This occurs if ntop determines that the address is a broadcast address, but the actual packet destination is different.

Among other causes, ntop detects this problem when a host sends a packet to a broadcast address where the destination MAC address is not FF:FF:FF:FF:FF:FF. [b]This could simply indicate that the host is a bridge[/b].

The most likely cause of this is a misconfiguration, which SHOULD be fixed.

Using the wrong netmask is quite common on networks where the netmask has changed and some of the hosts still use the old netmask.

Most hosts use the netmask to determine the gateway router address, by setting the host portion of the address to 0x1 (i.e. the gateway for 192.168.1.1/24 is 192.168.1.1). If problems do occur, selecting the wrong gateway for non-local packets usually leads to apparent failure of the entire non-local network (support call: "The network is down"). It can also cause high packet loss, collisions, ttl expiration and other network problems.

Note: ntop defines the broadcast address as either zero (0.0.0.0) or an address which has a host part of 0. Perfectly normal. However, ntop determines the network and host portions for the monitored packet's address based on the actual configuration of ntop's own NIC. So if ntop's NIC has a different configuration it will tag traffic as having the wrong mask.

I'm currently looking with the Linode support to determine if it's simply a config problem, but if it's not - that is the host is used as a bridge - what do you think I should/could do to get out of this situation (else than changing job) ?

Richard

Sorry I can't be of much help there. ntop doesn't give me any errors on my linode, though I'm using the standard DHCP config it came with to configure my network connection.

What do you mean using it as a bridge?

Lastly, there's bandwithd, if there's a package for your distro, you cuold always install that quick to see what kind of traffic it is.

Update: The Linode supports told me that the ntop warning was kind of normal:


So this bridging is OK. Is another - bad - bridging possible beyond that? I don't know, I'm not a network specialist, could bridging be a technique to use my host as a relay?

I'll install bandwidthd, thanks for the tip.

Which is it, GB, or MB? If it's GB, then you're looking at going from 450-660 to 1350GB/mth, and that's something to get alarmed about since it means hefty overage charges, and should be investigated promptly.

If you're talking about MB/day, you're talking about inconsequential amounts of bandwidth, and shouldn't care; that's an increase of about 0.3% of your monthly cap on a 512.

Did you accidentally disable mod_deflate or the like?

Sorry, was a typo, I meant 45GB/day, which represents a sudden increase of more than 100%! And no, my server is still sending compressed pages.