Yikes, I just did an iptables -F and I think I locked myself out of my VPS. Any way to get back in?
| Linode Forum https://forum.linode.com/ |
|
| DOS Attack, help! https://forum.linode.com/viewtopic.php?f=19&t=5541 |
Page 1 of 1 |
| Author: | Acejam [ Tue May 04, 2010 11:17 am ] |
| Post subject: | DOS Attack, help! |
Hey guys, Not sure why, but I'm currently getting a "heavy" DOS attack from a few IP address. At the very basic level, I'm trying to flush my IPtables firewall, and then simply blocking the IP addresses in question. For the time being to get things stable, is there anything else that I can do? I'm using this simple command: iptables -A INPUT -s IP-ADDRESS -j DROP The problem is Apache is constantly crashing, due to the numerous HTTP GET requets. (1,000's a second) This causes the 720MB slice to lock up, causing SSH to lock up as well. I was actually on a 360 plan and thought I had a traffic spike at first. Now, I'm certain it's a DOS attack. (not sure if DDOS though) Any other tips? I'm running Ubuntu 8.04 x86 with Apache2, PHP, and MySQL. I'm running a simple Wordpress blog, which gets about 2,000 unique's a day. Thanks, Ace |
|
| Author: | Acejam [ Tue May 04, 2010 11:20 am ] |
| Post subject: | |
Yikes, I just did an iptables -F and I think I locked myself out of my VPS. Any way to get back in?
|
|
| Author: | jed [ Tue May 04, 2010 11:20 am ] |
| Post subject: | Re: DOS Attack, help! |
Acejam wrote: I'm using this simple command: iptables -A INPUT -s IP-ADDRESS -j DROP Dropping the traffic in netfilter isn't going to alleviate the DoS if it's designed to saturate your network. We tend to notice DoS attacks of that variety, however. Acejam wrote: The problem is Apache is constantly crashing, due to the numerous HTTP GET requets. (1,000's a second) This causes the 720MB slice to lock up, causing SSH to lock up as well. What is MaxClients at in httpd.conf (don't forget Ubuntu spreads its config over multiple files, use grep -r if you can't find it)? Are keepalives on? Acejam wrote: Yikes, I just did an iptables -F and I think I locked myself out of my VPS. Any way to get back in? :x
Yes, LISH. |
|
| Author: | Acejam [ Tue May 04, 2010 12:48 pm ] |
| Post subject: | |
I used LISH and managed to get iptables "working" again. However, I noticed that it tends to only work OK in IE. With Firefox, I was unable to type/input anything. On another note, my apache2.conf settings are as follows: (using prefork) Timeout 300 KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 15 StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 0 Pretty sure this is my default apache2 config from Ubuntu 8.04 Server x86. I tried following the Linode Troubleshooting low-memory settings when I was still on my 360 node last night, but they didn't seem to help much. For now, I've been having to turn off my VPS. Then during bootup I have to quickly kill Apache via SSH or else I can't SSH into the server. (yes I know I can disable auto-start etc)\ Any other thoughts? I'm going to send a notification to the owner of the IP in question as well. Thanks, Ace |
|
| Author: | jed [ Tue May 04, 2010 1:16 pm ] |
| Post subject: | |
Cut MaxClients by a factor of 10, from 150 to the 10-15 range. Turn off Keepalives, then reload Apache. You're probably starving Apache for memory, since the distribution defaults expect a machine with a lot of physical memory -- 150 MaxClients, particularly with PHP or something else riding along on Apache's worker processes, is far too many for most Linodes. |
|
| Author: | Acejam [ Tue May 04, 2010 2:54 pm ] |
| Post subject: | |
Cool. I've turned KeepAlives to off, and I've set MaxClients to 13. I have also sent out emails to the 3 IP addresses in question, and have received responses back from 1 of them. (I looked up the final hop/IP block owners and emailed their abuse departments) With the lower MaxClients setting, the VPS is more responsive now, but still a bit laggy. However, my SSH connection is no longer breaking/timing out, which is good. |
|
| Author: | Guspaz [ Wed May 05, 2010 10:20 am ] |
| Post subject: | |
The thing is, requests beyond the maxclients will get queued. So you want to set maxclients to a value that is just high enough to consume all available resources on the system, but not higher. Any higher and you're slowing down individual requests to handle others, and they might as well wait in queue. Another thing you might consider, if you're dealing with heavy load, is a web server such as lighttpd or nginx; they tend to handle such load better. Alternatively, Apache with fastcgi can achieve many of the benefits by decoupling PHP from the web server, although I don't think the setup is as easy under most distros (setting up PHP under lighttpd, which is fastcgi by default, can be done in three commands; install, symlink fastcgi module, reload lighttpd config). |
|
| Author: | Acejam [ Wed May 05, 2010 5:13 pm ] |
| Post subject: | |
Cool, thanks for the input guys! I heard back from the other 2 Abuse teams. They "took care" of the situation, and it looks like DOS requests have stopped. However, it might also look like this because I block their IP's via iptables. (I actually blocked the entire /24, since they tried to change their origin IP) The server load has returned back to normal now. I've gotten 5k hits so far today, and I'm only hovering around 0.05 load. As I stated in my OP, initially I thought I had a traffic spike, so I resized my Linode to a 720. Do you think I should bump back down to a 360? |
|
| Page 1 of 1 | All times are UTC-04:00 |
| Powered by phpBB® Forum Software © phpBB Group http://www.phpbb.com/ |
|