Hi,

Linode: Ubuntu 11.10, Latest 32 bit (3.7.5-linode48); related trouble ticket [1621800]

I've recently run across issues with network usage on my linode (starting the second week of January).

I use my linode as a web server (nginx)/proxy (squid)/VPN. My noise floor usually hovers at about 100 kb/s and spikes with proxy/VPN use.
I am currently looking to what those spikes are, my best guess right now is 0.4 - 1.2 Mb/s @proxy and 2.0 - 4.0 Mb/s @VPN.

During the middle of January my noise floor jumped to 500 kb/s and has slowly risen to average around 750 kb/s.

I've had the linode since APR 2011 without any issues until an 'Emergency Migration' in January however I don't think it is related as that happened roughly a week after the increase in traffic. Since then I have had issues with my squid proxy randomly dying... the issues are probably related but I haven't been able to tie them together and no changes have been made to the proxy or its config since the beginning of October 2012.

I would appreciate any advice from the community! Below is a link to a screen shot with iftop and nethogs running ~10 minutes after a reboot.
The openVPN daemon was running, no VPN connection established. The squid daemon was running, no connection established. I am turning them off for testing.
It looks like the biggest offenders for bandwidth usage are linode resolvers (I hope that doesn't count against me!) and sanvigario.com.br (which I have no clue what that is... and may soon be my first iptables entry).

Thanks!
- Roy


http://s18.postimage.org/9w09k28br/2013_02_23_at_10_48_29.png

Looking at the output it looks like you also are running a dns server/resolver on that node? Has it been (accidentally?) set to allow recursing or forwarding for any? If so it might be someone attempting to do DNS reflection attacks*. It would explain it going to the linode resolvers at least.

Would also make sure the proxy is not open to the world, likely to be abused.

* Sending DNS queries using UDP packets with spoofed source address of the site to attacked, resulting in larger responses than the size of the queries, often yielding substantial amplification to attack targets using less bandwidth. Though looking at the graphs they're not getting much out of it.

trippeh,

Thanks for the input!

The squid proxy is locked down (ntlm), so that shouldn't be an issues. Looking back through my access logs it looks normal.

I do have dnsmasq up to allow local resolution for my VPN, I'll turn it off to see if it makes a difference and dig into my settings to see what my defaults are.

v/r
- Roy

So... it looks like someone is trying to use my DNS to try and attack isc.org, what they have against bind I don't know.

k0te@th3dojo:~$ sudo tcpdump -i eth0 -c 10 -n port domain
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
04:57:26.657315 IP 5.135.198.162.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.702758 IP 198.148.80.140.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.704518 IP 5.135.100.90.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.709182 IP 178.32.36.49.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.778372 IP 192.211.51.109.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.822753 IP 198.148.80.140.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.828310 IP 5.135.100.90.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.866726 IP 192.211.51.27.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:26.943084 IP 198.148.80.140.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
04:57:27.063139 IP 198.148.80.140.25345 > 66.228.38.104.53: 10809+ [1au] ANY? isc.org. (36)
10 packets captured

Looks like a multiple offending IPs, all coming from port 25345... Regardless, modified dnsmasq conf to only listen to queries from myself (intended behavior, be it from squid or the VPN).

k0te@th3dojo:~$ more /etc/dnsmasq.conf
# "/usr/sbin/dnsmasq --help" or "man 8 dnsmasq" for details.

#basic config

listen-address=127.0.0.1
port=53
bind-interfaces
user=dnsmasq
#pid-file=/var/run/dnsmasq.pid

#advanced config

domain-needed
bogus-priv
no-hosts
dns-forward-max=10 #max concurrent queries
cache-size=100 #150 default, 0 off
#no-negcache
neg-ttl=3600
#resolv-file= #default is /etc/resolv.conf
#no-poll #force resolv-file use

So... now I'm just receiving lots of random DNS lookups and not sending anything. Good, now the fun part is going to be figuring out what to do about all this extraneous traffic.

- Roy