Hi all,

I'm currently getting nailed but the following requests

tcpdump output

blah blah blah... 952+ [1au] ANY? ripe.net. (38)

I'm dumping all of the traffic but it's currently up to 1.5Mb/s on my linode. I've talked to linode about but they have said there is nothing they can do about it and won't provide me with a new IP address to mitigate the attack. I don't even have the option to purchase the new IP since now they know the reason I want it is no other than mitigating the attack.

Anyway this has been going on now for about a week and I really can't use the linode in this state. I'm not sure what to do and feel like linode has given me no options either.

Was wondering if anyone here has had the same problem and could offer up some solutions.

Also, I've had a firewall in place and just to be safe I shutdown and rebuilt a new box.

Oh and I've sent off an email to the apparent offenders domain to let them know of the attack, though these packets are probably forged.

Thanks again for the help!

If you really wanted a new IP address you could create a new Linode, copy your disk image over from your current Linode then delete your current Linode.


Edit: Not that that's really what you should do to fix this...

Thanks, but I've paid in advance for this linode so that isn't an option. Or am I mistaken about that?

Edit: Yes exactly Right now I've moved all of my vhosts to another linode and am just monitoring. The thing that sucks is in my opinion the linode is unusable...

Just wanted to let you guys know that linode changed my IP (thank you linode!). I'm up and running with no more DNS noise.

Damn this attack is a total PITA.

If you paid in advanced and make a new box you can ask linode to remove the remaining time over to the new one.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

I'm curious, why did this attack make the node "unusable"?

_________________
Matt Nordhoff (aka Peng on IRC)

It's a 1.5Mbps attack, if you're dropping the traffic it would have absolutely no impact on your Linode except to accrue roughly $48/mth worth of bandwidth usage, which isn't terribly much, all things considered.

Inbound transfer is free. If you're dropping it, it costs you nothing. Heck, it probably even improves Linode's ratios.

_________________
Matt Nordhoff (aka Peng on IRC)

Good point

So, yeah, I'll revise my statement to "A 1.5 Mbps attack that is being dropped by your firewall should have no impact whatsoever on your linode"

At the time of my writing it was 1.5Mb/s but it was as high as 7Mb/s. Yes I was dropping it, but that was also eating up CPU cycles of which I saw as much at 15%. All of that is right off of the top of the linode (L768).

I didn't want to continue using the linode when there was an active attack against it so that made it unusable to me. I was totally patient though as I had moved critical sites over to another linode and was hoping it was just stop, but that didn't happen.

Would you mind explaining your comment a bit more (or point me in the right direction) about how it would improve Linode's ratios?

(Sorry for the delayed respond...)

And thanks a lot for the comments I appreciate it!

OK, but 15% of 1 core is nothing. Even 15% of your overall CPU power of 4 cores -- i.e. 60% of 1 core -- shouldn't cause problems. (Well, at 60% I'd start worrying about the networking stack a bit, especially if the packets go through conntrack.)


It was largely a joke. Settlement-free peering agreements between ISPs -- when they connect each others' networks for no money -- often place great importance on their traffic ratios, requiring that they exchange a relatively equal amount of traffic. Linode is probably pretty unequal, since a lot of their traffic is probably web stuff, which tends to use more outbound traffic. (HTTP request: 1-2 KB. Response: Anything, but frequently hundreds of KB.)

_________________
Matt Nordhoff (aka Peng on IRC)

In this case I was dropping these packets and not tracking them. I guess I'm (incorrectly?) hung up on the fact that regardless what the numbers are legitimate traffic would be competing with this DNS noise and that just doesn't sit right with me. I suppose the counter argument to that is there is plenty of network noise, but it doesn't come in the form of several thousand packets a second .

So would you have just written this one off? Now you got me thinking that I was over reacting, but I do want to make sure I have a realistic view in case this happens again. I was really lucky to have space elsewhere to move sites around but that won't always be the case.

What other approaches would you have taken to mitigate the attack if any? Like I said I contacted abuse departments and whatnot (but that is really a waste of time bc the packets were likely forged), but other than that there didn't seem to me much else I could do.



Thanks for the explanation!

There's a difference between "don't want to use because the packets don't sit right with me" and "the linode is unusable". Let's be clear, the linode is perfectly usable (for anything but DNS serving). If I said "I don't want to use my linode because it's a full moon and also a Tuesday", that doesn't somehow make my linode unusable (unless it's a werelinode, but that's another issue). It just means that I haven't restocked on silver USB keys recently.

Basically, you don't pay for inbound traffic, so if you just drop the inbound DNS traffic, there is no impact on your linode. You've got four cores to play with (so effectively 400%), so 15% usage isn't a problem unless you're maxing out all four cores. The only scenario where you might see some impact is if you're trying to run a DNS server, but there's not really any reason why you would since Linode provides free DNS servers both for resolution and hosting.

Edit: apologies for the vagueness of my "unusable" statement please let me clarify below...



What I said was

In other words legitimate traffic would be competing with the traffic from the attack and that's what didn't sit right with me. Does the linode work? Well yes I said that in my inital post, but I followed that I'm not going to use a machine for business purposes while it's under attack thereby making it unusable to me.

I really don't think the previous statement is unreasonable. I apologize if I made it sound like linode's service is not totally awesome because it is. I've been a very happy linode customer for years, but this has never happened to me before which is why I was asking the community for help/perspective.

So you would have just ignored the attack, or noticed it an just said "whatever..." based on your statements below?




BTW it's freaking awesome that we don't have to pay for inbound traffic. I really would have been screwed otherwise. I was also saved by the fact that I was alerted when the inbound connections got to a certain rate...

Thanks again for the help!

You might want to define "attack" a bit better.

There are attacks, and then there are ATTACKS.

Eating up a bit of inbound pipe and a few clock cycles isn't really worth getting your cyberpanties in a bunch over.

If people stopped using systems for every little attack, a few port scans would shut down the internet.

You did what you thought best, but probably need to learn how to mitigate such things in the future instead of packing up shop and moving across the street the first time your shop wall gets tagged with a bit of graffiti.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.