Hi,

I have had a new linode VPS running smoothly for about a month now. Very happy so far. My only concern is that despite my website only getting several hits a day (less than 10) I have used up significant bandwidth.

Transfer/mo: 200 GB
Incoming: 1.74 GB
Outgoing: 2.26 GB
Total: 4.00 GB

My website is only a few 100KBs in size and contains no images, videos etc. I have only installed LAMP, Webmin, PHPMyAdmin and Sendmail. (I only recently installed Sendmail).

I seem to have almost constant incoming and outgoing traffic, but I cant work out for the life of me where its coming from and where its going. I am getting an almost constant 7Kb/s incoming and 9Kb/s outgoing. Despite having very few hits (according to Google Analytics).

My website is under development (http://www.elisle.com)

Two questions:
1. Is this normal?
2. What should I look at next?

Google Analytics does not see all traffic, particularly web crawlers and automated attacks. There's also a fair amount of "background noise" on the Internet.

That is a little high, though. What does your web server's log file say your traffic is like? How about other applications? Worse comes to worse, run tcpdump for a little while (tcpdump -s 0 -i eth0 -w some_filename.pcap), download the pcap file, and load it into wireshark on your local computer to see what all is going over the wire.

_________________

/* TODO: need to add signature to posts */

And what have you done to secure your setup?

What does your SENDMAIL logs show?

Probably search engines, robots scanning for vulnerabilities, script kiddies trying to guess your password, etc.

A few kilobits per second of unwelcome traffic is normal for any machine that is exposed to the whole wide world.

Thankyou all for the replies.

Well I have spent a few hours going through the logs. 99% of the outgoing traffic was ICMP based.

So I simply added an IP Firewall rule to drop all outgoing ICMP based packets. This has worked well and now my outgoing traffic is now negligible.

Dropping all outgoing ICMP packets will break some things such as ping requests. Are there any other issues I should look out for?

I have no idea how I am going to filter the incoming data as it is hard to identify any single source or attribute for the incoming data, other than UDP. I might have to outsource that to Cloudflare or something like that.

So what was generating all that outbound ICMP traffic?

If you didn't set it up that way, or mis-configure something that caused it - what is?

Just dropping the outbound traffic isn't solving the underlying problem.

I seem to be getting hit from lots of places with UDP traffic. The source IP addresses seem faked. So my server was responding to all these hits with an ICMP packet.

I agree this hasn't solved the problem and I am still trying to work out how to block rubbish packets coming in.

Blocking ICMP traffic breaks various things, such as path MTU discovery. It's recommended to let it pass. Blocking the inbound UDP traffic would be a good idea, though.

Could you post a hunk of the UDP traffic? I'm thinking there might be some pattern in there...

Also, keep in mind that this is apparently generating about ~2 GB of billable traffic per month, or $0.20/mo. It's also probably directed at the IP address you're using, so Cloudflare probably won't even see it. A cost-benefit analysis would be wise before throwing money/time/code at it.

_________________

/* TODO: need to add signature to posts */

Ok, well I just removed the ICMP rule.

As for blocking inbound UDP traffic, I actually did do that at first, but it slowed down Webmin to a crawl. Because I dont know enough about networking (I live on the application side of things) I decided to quickly remove it.

Below is a dump of log showing both incoming and outgoing:

Dec 31 18:00:06 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=222.124.45.209 DST=173.230.144.82 LEN=64 TOS=0x00 PREC=0x00 TTL=246 ID=13264 PROTO=UDP SPT=27739 DPT=53 LEN=44
Dec 31 18:00:06 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=222.124.45.209 LEN=92 TOS=0x00 PREC=0xC0 TTL=64 ID=40391 PROTO=ICMP TYPE=3 CODE=3 [SRC=222.124.45.209 DST=173.230.144.82 LEN=64 TOS=0x00 PREC=0x00 TTL=246 ID=13264 PROTO=UDP SPT=27739 DPT=53 LEN=44 ]
Dec 31 18:00:06 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=110.138.56.212 DST=173.230.144.82 LEN=65 TOS=0x00 PREC=0x00 TTL=246 ID=28 PROTO=UDP SPT=10227 DPT=53 LEN=45
Dec 31 18:00:06 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=110.138.56.212 LEN=93 TOS=0x00 PREC=0xC0 TTL=64 ID=61369 PROTO=ICMP TYPE=3 CODE=3 [SRC=110.138.56.212 DST=173.230.144.82 LEN=65 TOS=0x00 PREC=0x00 TTL=246 ID=28 PROTO=UDP SPT=10227 DPT=53 LEN=45 ]
Dec 31 18:00:06 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=180.251.180.210 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=241 ID=10991 PROTO=UDP SPT=10578 DPT=53 LEN=40
Dec 31 18:00:06 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=180.251.180.210 LEN=88 TOS=0x00 PREC=0xC0 TTL=64 ID=5598 PROTO=ICMP TYPE=3 CODE=3 [SRC=180.251.180.210 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=241 ID=10991 PROTO=UDP SPT=10578 DPT=53 LEN=40 ]
Dec 31 18:00:07 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=110.138.70.40 DST=173.230.144.82 LEN=62 TOS=0x00 PREC=0x00 TTL=246 ID=9303 PROTO=UDP SPT=14564 DPT=53 LEN=42
Dec 31 18:00:07 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=110.138.70.40 LEN=90 TOS=0x00 PREC=0xC0 TTL=64 ID=26831 PROTO=ICMP TYPE=3 CODE=3 [SRC=110.138.70.40 DST=173.230.144.82 LEN=62 TOS=0x00 PREC=0x00 TTL=246 ID=9303 PROTO=UDP SPT=14564 DPT=53 LEN=42 ]
Dec 31 18:00:07 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=110.138.4.76 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=40527 PROTO=UDP SPT=10430 DPT=53 LEN=40
Dec 31 18:00:07 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=110.138.4.76 LEN=88 TOS=0x00 PREC=0xC0 TTL=64 ID=11115 PROTO=ICMP TYPE=3 CODE=3 [SRC=110.138.4.76 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=40527 PROTO=UDP SPT=10430 DPT=53 LEN=40 ]
Dec 31 18:00:07 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=110.139.11.45 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=236 ID=4012 PROTO=UDP SPT=29714 DPT=53 LEN=40
Dec 31 18:00:07 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=110.139.11.45 LEN=88 TOS=0x00 PREC=0xC0 TTL=64 ID=37310 PROTO=ICMP TYPE=3 CODE=3 [SRC=110.139.11.45 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=236 ID=4012 PROTO=UDP SPT=29714 DPT=53 LEN=40 ]
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=125.166.113.218 DST=173.230.144.82 LEN=73 TOS=0x00 PREC=0x00 TTL=118 ID=4613 PROTO=UDP SPT=12860 DPT=53 LEN=53
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=125.166.113.218 LEN=101 TOS=0x00 PREC=0xC0 TTL=64 ID=64333 PROTO=ICMP TYPE=3 CODE=3 [SRC=125.166.113.218 DST=173.230.144.82 LEN=73 TOS=0x00 PREC=0x00 TTL=118 ID=4613 PROTO=UDP SPT=12860 DPT=53 LEN=53 ]
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=61.94.87.58 DST=173.230.144.82 LEN=65 TOS=0x00 PREC=0x00 TTL=237 ID=7114 PROTO=UDP SPT=11543 DPT=53 LEN=45
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=61.94.87.58 LEN=93 TOS=0x00 PREC=0xC0 TTL=64 ID=55064 PROTO=ICMP TYPE=3 CODE=3 [SRC=61.94.87.58 DST=173.230.144.82 LEN=65 TOS=0x00 PREC=0x00 TTL=237 ID=7114 PROTO=UDP SPT=11543 DPT=53 LEN=45 ]
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=110.138.4.76 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=40528 PROTO=UDP SPT=10431 DPT=53 LEN=40
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=110.138.4.76 LEN=88 TOS=0x00 PREC=0xC0 TTL=64 ID=11116 PROTO=ICMP TYPE=3 CODE=3 [SRC=110.138.4.76 DST=173.230.144.82 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=40528 PROTO=UDP SPT=10431 DPT=53 LEN=40 ]
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=f2:3c:91:df:8c:41:c8:4c:75:f5:d6:3f:08:00 SRC=10.252.85.38 DST=173.230.144.82 LEN=61 TOS=0x00 PREC=0x00 TTL=246 ID=5552 PROTO=UDP SPT=10557 DPT=53 LEN=41
Dec 31 18:00:08 li154-82 kernel: BANDWIDTH_OUT:IN= OUT=eth0 SRC=173.230.144.82 DST=10.252.85.38 LEN=89 TOS=0x00 PREC=0xC0 TTL=64 ID=59584 PROTO=ICMP TYPE=3 CODE=3 [SRC=10.252.85.38 DST=173.230.144.82 LEN=61 TOS=0x00 PREC=0x00 TTL=246 ID=5552 PROTO=UDP SPT=10557 DPT=53 LEN=41 ]

Besides this mostly being UDP traffic, the other common attribute for the bad packets seem to be that DPT=53. I have still yet to work out what DPT is. Of course I know it stands for Dynamic Packet Transport, but I don't understand anything more than that.

DPT = Destination Port

UDP 53 = DNS

They're probing to see if you have a poorly configured Name Server running (or a poorly secured CPanel install)

As a rule, you should BLOCK all traffic except the bare minimum that you need (i.e. on allow 80, 443 if you have HTTPS, 25 if you're running a SMTP server, and whatever port you moved SSH to, 10000 or whatever port you moved Webmin to, etc)

Ahh, cool thanks for that. Destination port makes a lot more sense

Woah.

Your IP address used to be dns-us1.powerdns.net. Looks like it changed some time back, before you got the IP address (and perhaps before Linode was assigned that block). Congratulations, you've got a raging case of afterglow.

_________________

/* TODO: need to add signature to posts */

How do you do historical WHOIS lookups?

Not only that, but dns-us1.powerdns.net was apparently targeted in some sort of attack less than 6 months ago.


There are several places where you can run historical WHOIS on domain names. I'm sure at least some of them can also do historical WHOIS on IP addresses. You may have to purchase a membership, though.

On a related note, services like this also account for quite a bit of "background noise" on bandwidth graphs.

Well this is very interesting and explains why I get more incoming packets than usual.

I plan on just changing my firewall configuration to block DNS requests.

I am curious that what kind of DNS requests I am getting. Are they legitimate requests, with users falling back to alternative DNS' or is it just the remnants of a DDoS attack?