Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Sun Jan 09, 2011 11:39 pm 
Offline
Senior Member

Joined: Thu Nov 19, 2009 4:55 pm
Posts: 52
Quote:
tail -f -n 0 error.log
[Sun Jan 09 19:34:48 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:49 2011] [error] [client 88.90.106.161] Invalid URI in request \xf2\xa8\xe4\v5E\x19\x84\xc0\x04\xff\xaa
[Sun Jan 09 19:34:50 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:51 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:51 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:52 2011] [error] [client 88.90.106.161] Invalid URI in request #\xebM&\x0c\xb8\x8f\xa9\xe9-Ht\xe8S\xae\xf4\x90"-\xf7I\xf2
[Sun Jan 09 19:34:52 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:52 2011] [error] [client 88.90.106.161] Invalid URI in request Z\xff\x81Y\xacR\xee\x1eu\xed}\x94\xeb=\xb5\xb3z\x10\xfe\xd3g\xa7dc\xb3\x0c\x98\xddB\xa67\xf9<
[Sun Jan 09 19:34:53 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:53 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] Invalid URI in request \xc3x \x01>\x9a@M\x97\x17r
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:54 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:55 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:56 2011] [error] [client 88.90.106.161] Invalid URI in request \xf2\xa8\xe4\v5E\x19\x84\xc0\x04\xff\xaa
[Sun Jan 09 19:34:56 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:57 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:34:57 2011] [error] [client 88.90.106.161] Invalid URI in request #\xebM&\x0c\xb8\x8f\xa9\xe9-Ht\xe8S\xae\xf4\x90"-\xf7I\xf2
[Sun Jan 09 19:34:58 2011] [error] [client 88.90.106.161] request failed: error reading the headers
[Sun Jan 09 19:35:00 2011] [error] [client 88.90.106.161] request failed: error reading the headers


Happens really fast and brings down the server to a halt. Many apache instances and I have to restart apache for it to work. It'll stop working within 3 seconds.

I've blocked the IP for now. Something similar happened 3 days ago from a similar ip.

Both seem to be from same provider (norway).
I sent email to abuse-mailbox: abuse@telenor.net but no response.


Top
   
 Post subject:
PostPosted: Mon Jan 10, 2011 3:44 pm 
Offline
Junior Member

Joined: Sun Nov 07, 2010 10:36 pm
Posts: 33
Website: http://www.rent-matcher.com
It sounds like it COULD be an attack, but you can never be 100% positive. It could be a poorly written crawler/script or it could be malicious. Either way, blocking the IP as you did is probably your best course of action.


Top
   
 Post subject:
PostPosted: Mon Jan 10, 2011 3:47 pm 
Offline
Senior Member

Joined: Tue Jan 22, 2008 2:10 am
Posts: 103
It may be an attack of some sort, but, as it's only coming from one IP, it's not a DDoS - a Distributed Denial of Service attack, by definition, comes from multiple IPs.


Top
   
 Post subject:
PostPosted: Mon Jan 10, 2011 5:46 pm 
Offline
Senior Member
User avatar

Joined: Sun Dec 27, 2009 11:12 pm
Posts: 1038
Location: Colorado, USA
Unless you have family/friends/business in Norway - just ban (drop) that IP and then you have no worries.


Top
   
 Post subject:
PostPosted: Mon Jan 10, 2011 9:31 pm 
Offline
Senior Member

Joined: Thu Nov 19, 2009 4:55 pm
Posts: 52
Happened again today from: 83.109.195.192

Quote:
tail -f -n 0 error.log
[Mon Jan 10 17:29:14 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:16 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:17 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:18 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:19 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:20 2011] [error] [client 83.109.195.192] request failed: error reading the headers
[Mon Jan 10 17:29:22 2011] [error] [client 83.109.195.192] request failed: error reading the headers


Is there anything I can do to prevent it from taking down Apache? Some type of measure to throttle a specific ip if it's requesting too much in a short amount of time? Should I tweak my Apache settings? This is a problem if I'm not around the current second.

Should I setup monin to restart apache automatically? Anyone have some tips/solutions so this doesn't take down the server for a hour+ until I get back to computer?

So in summary, server seems to be getting hit same time every night from a different IP but all related to this "norway isp" telenor.


Top
   
 Post subject:
PostPosted: Mon Jan 10, 2011 10:23 pm 
Offline
Senior Member

Joined: Wed May 13, 2009 1:18 am
Posts: 681
Even if you get hit with a ton of these it really shouldn't take down Apache. At worst it should just make your site slow while they are happening, but it certainly shouldn't interfere with basic server function or access (e.g., over ssh).

If the machine is keeling over it probably means you have too aggressive a configuration for Apache (e.g., MaxClients too high or equivalent) for your Linode size. It may not have been an issue until now if you never received enough simultaneous requests.

-- David


Top
   
 Post subject:
PostPosted: Mon Jan 10, 2011 11:37 pm 
Offline
Senior Member

Joined: Thu Nov 19, 2009 4:55 pm
Posts: 52
db3l wrote:
Even if you get hit with a ton of these it really shouldn't take down Apache. At worst it should just make your site slow while they are happening, but it certainly shouldn't interfere with basic server function or access (e.g., over ssh).

If the machine is keeling over it probably means you have too aggressive a configuration for Apache (e.g., MaxClients too high or equivalent) for your Linode size. It may not have been an issue until now if you never received enough simultaneous requests.

-- David


David, thanks. What you say makes sense and I'm trying to test against this but I can't seem to recreate the situation.

This is my config:

Quote:
<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 3
MaxSpareServers 6
# ServerLimit 12
MaxClients 40
MaxRequestsPerChild 3000
</IfModule>


I'm on 756 MB of memory.

I would change the MaxClients to 20 but I don't want to change it arbitrarily without testing it to see if it makes a difference.

I tried to use a "ab" benchmark to see if it would bring the server down but no luck. The server kept responding promptly. Should I leave it on longer? Any ideas?

What would be a good ab benchmark test?

I tried doing the following:

ab -n 5000 -c 10

Which actually ended up closing my local wifi/internet connection. I originally tried -n 1000 -c 5. No luck on slowing down server.

Any advice would be greatly appreciated! Thank you!


Top
   
 Post subject:
PostPosted: Tue Jan 11, 2011 11:22 am 
Offline
Senior Member

Joined: Fri May 02, 2008 8:44 pm
Posts: 1121
ab -c 10 would only make 10 simultaneous connections. Since you have MaxClients set to 40, it would take ab -c 40 to stress test your server properly. You'll probably run out of memory on the server if you're using mod_php.


Top
   
 Post subject:
PostPosted: Tue Jan 11, 2011 7:05 pm 
Offline
Senior Member

Joined: Wed May 13, 2009 1:18 am
Posts: 681
arachn1d wrote:
David, thanks. What you say makes sense and I'm trying to test against this but I can't seem to recreate the situation.

Well, it sounds like you have an almost nightly occurrence via external sources, so even if you can't generate the necessary load manually, tweak and then just wait a day :-)

Though as hybinet points out, the crucial point is how many simultaneous requests you initiate, not really the total number, since that's what causes all the Apache clients to be instantiated, and likely overloading your memory. You ought to be able to generate quite a few simultaneous connection attempts even over a slow client link.

It depends on your application stack, but MaxClients of 40 feels too high for a 768. Personally I'd have no worries about dropping it to 20 and then watching behavior. Odds are your normal usage won't be affected at all.

There are a number of Apache tuning threads here in the forums you might reference for other ways to benchmark your system. One at viewtopic.php?t=6272 for example that covers some of the steps to take to monitor and test changes.

BTW, I had similar log patterns on an Apache server of minte in the past, and I've been wracking my brain trying to remember what I identified it as, as it didn't turn out to be malicious. Finally remembered...

In my case it ended up being incoming BitTorrent traffic, where a client behind the same firewall as the server had ended up choosing the same (non-standard) port as was being forwarded to Apache. So when the client stopped, for a period of time thereafter, I got incoming probes from BitTorrent clients looking for my peer, but now landing on Apache.

In general, your logs just say that some client using a binary protocol is connecting to the port Apache is listening on. So your case could be completely different, or could in fact be malicious, but perhaps this will help you think of other possibilities, especially if there's any chance that Apache is listening on a port that may be getting other uses, or proxy's or whatever.

-- David


Top
   
 Post subject:
PostPosted: Tue Jan 11, 2011 8:17 pm 
Offline
Senior Member

Joined: Fri May 02, 2008 8:44 pm
Posts: 1121
The error message seems to suggest that whoever is sending those requests isn't even bothering to write correct HTTP headers. Maybe it's not talking HTTP at all, as db3l says.

In that case, you could set up nginx on port 80 and have it proxy all requests to Apache (maybe except some static files, which are better served by nginx). Ideally, nginx will see that the request is not valid HTTP, and drop the connection without passing it on to Apache. Also, a few hundred additional connections on nginx will hardly use any resources, while a few dozen additional connections on Apache can easily bring a VPS to its knees.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group