Linode Forum

myriad · Joined: 06 Sep 2010 Posts: 4

I found out when I sat down at my computer this morning that my IP has been null-routed for 24 hours because we had a DOS attack on our UDP ports in the early hours of this morning. On further investigation it appears that the attack was targeting the second public IP on our Linode (we have 2 IP's).

The IP in question that was under attack had NO dns records or any services linked to it other than an old test installation of Shoutcast (which I think I disabled, but until I can reach my server again I can't say for sure). It appears to be a totally random act as I am only running the most innocuous of websites (no IRC or anything like that) but now my clients are MAJORLY PISSED that their websites will be out of commission for the next 24 hours. And are demanding that I move them to another more reliable service.

My question is twofold:
1. What did I do to cause this? Is my configuration somehow messed up that I set myself up for this? I have been running servers for 15 years, and this is the first time anything like this has ever happened to me.

2. Is it reasonable to ask Linode to activate the other (un-targeted IP) so I can get my servers up and running?

Guspaz · Joined: 26 May 2009 Posts: 1147 Location: Montreal, QC

I think it's reasonable to null-route only the affected IP... Have you tried talking to Linode about this?

myriad · Joined: 06 Sep 2010 Posts: 4

Yes. After about 7 hours downtime they finally agreed to restore my other IP (thank god).

myriad · Joined: 06 Sep 2010 Posts: 4

Yes. After about 7 hours downtime they finally agreed to restore my other IP (thank god). So Guspaz, what do you think about question 1? Is there any way to prevent this from happening again or is it just the luck of the draw?

Guspaz · Joined: 26 May 2009 Posts: 1147 Location: Montreal, QC

Well, there are things that you do to make your linode more DoS resistant, but nothing you can do to make it DoS proof. There's not much of a window between attacks of a size that you can mitigate yourself and attacks big enough to cause problems for other nodes on the host, so in the end there's not much you can do. Setting up a high-availability type setup can help, in that taking out one of your linodes would not take out the other, but then they can just attack both at the same time.

vonskippy · Joined: 27 Dec 2009 Posts: 469 Location: Colorado, USA

myriad wrote: but now my clients are MAJORLY PISSED that their websites will be out of commission for the next 24 hours.
And how much are these "clients" willing to pay for a 100% guaranteed uptime hosting service?

Use this experience to double check the wording on your service level agreement that you have with your clients and make sure that acts of God and/or the Internet are not covered.

myriad · Joined: 06 Sep 2010 Posts: 4

Vonskippy don't get me wrong, I am in no way promising them uninterrupted up time, no one can do that. I am just trying to make sure that I can prevent this from happening again. I totally respect you guys and that is why I am asking for your expert advice. My client was understanding about the need to null-route the IP, but not so understanding of the 24 hour penalty.

I am going to try to set up another Linode for auto-failover and that is going to be (another) learning curve as I have never set one up before. Off to the forums for some tips....

Guspaz · Joined: 26 May 2009 Posts: 1147 Location: Montreal, QC

It's impossible to completely survive a DDoS attack if the attacker is determined enough. Script kiddies have taken out Microsoft, Amazon, Wal-Mart, companies with a lot more bandwidth than you can afford.

There are hosts out there that specialize in DDoS mitigation. Linode is not one of them. These hosts try to survive DDoS attacks by throwing massive amounts of bandwidth and hardware at the problem, but even they aren't invincible.

The best you can try to do is limit the damage that they're likely to cause, not the damage that they could cause.

bezerker · Joined: 27 Jun 2005 Posts: 43

Pretty much what Guspaz says. If someone wants you down, you will be down.

That said, look further into why you were being attacked. What types of sites do your clients run? What software do they run if they allowed to on the box? (IRC eggdrop bots used to be a common target for example.)

Very rarely do DOS's happen for no reason. Something was done to piss someone off. I've seen it as simple as someone dislikes your site and wants it offline. Or someone banned a player from their minecraft server, etc.

But as he said, if someone wants your server down... he will take it down. There are things you can do to prevent it, but if it was enough traffic to take down the host, it's completely out of your hands.

linodeTail · Joined: 12 Oct 2009 Posts: 15

I've been getting a ton off dos attacks on my server. After installing "ddos deflate" it's stopped all of them so far. I get emails to my phone daily every time they try again (they were opening 1000s of connections) and then they are banned for a few hours.

As long as they are using a few IPs you're okay.

But if they are running a distributed attack even ddos deflate won't help.

details here: http://zedomax.com/blog/2009/08/17/web-server-hack-how-to-use-ddos-deflate-to-protect-against-dos-flooding/

glg · Joined: 09 Jan 2009 Posts: 505

linodeTail wrote: But if they are running a distributed attack even ddos deflate won't help.

so, basically, ddos deflate isn't named properly? (the first d is for distributed...)

jebblue · Joined: 23 May 2010 Posts: 112

Would iptables rate limiting mitigate a DOS or DDOS attack?

hybinet · Joined: 02 May 2008 Posts: 1058

glg wrote: so, basically, ddos deflate isn't named properly? (the first d is for distributed...)
It's named properly. It only deflates (reduces the severity of) the attack. It doesn't stop it, especially if the attack is too big to deflate.

vonskippy wrote: acts of God and/or the Internet
I worship thee, almighty Internet! Thou knowst everything, thou art everywhere... :roll:

pclissold · Joined: 24 Oct 2003 Posts: 877 Location: Netherlands

jebblue wrote: Would iptables rate limiting mitigate a DOS or DDOS attack?
It will mitigate a small attack by limiting the use of resources on your server. Once the attack is big enough to flood the connection to your box, you're dead in the water.

bezerker · Joined: 27 Jun 2005 Posts: 43

jebblue wrote: Would iptables rate limiting mitigate a DOS or DDOS attack?

Yes, but you're also limiting traffic on that port as well. So if say they are hitting port 80, you'll be limiting legit traffic as well as dos traffic. If however, they are hitting port say, 25, and you want your website to still work you can rate limit or even block traffic to port 25 and it may help.

However, remember the reason most attacks work if you filter them locally is because theyre either tying up system resources or the return traffic of your client communicating back is enough to kill the connection/system. If the incoming rate of data is > your pipe size, no matter what, local firewall running or not, you'll be down.

DOS attack grief!