Linode Forum
Linode Community Forums 		 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic

Board index » Linux Community Forums » Performance and Tuning
  Previous topic | Next topic 
Author Message
phowell32
PostPosted: Sat May 26, 2012 3:16 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
I have a Linode 512 in Atlanta and it's got Apache on Ubunto 10.4 and have had it for over a month now with no issues. I run basically 5 Wordpress sites off it, with very little traffic (about 700 visitors per day combined).

Starting late last night I started to get CPU Usage alerts through email. Also, high bandwidth alerts. My websites are crawling and most of the time will not even load up. See screenshot of graphs:

Image

I ran "ps aux" to see the processes and MYSQL is always taking less than 5% of the memory and no CPU so that's not the issue. I constantly see "www-data" processes with high CPU rates (see screenshot below). I've been using the kill command to kill the processes to keep the sites up.

Image

Any idea what I can do to find out what is causing all of the bandwidth usage and high CPU? My apache config:

Quote:
<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 3
MaxSpareServers 6
MaxClients 20
MaxRequestsPerChild 3000
</IfModule>


I have experimented with MaxClients at 32, and 15 and it seems to be doing best at around 20. Keepalive is OFF.
Top
   
Reply with quote  
phowell32
 Post subject:
PostPosted: Sat May 26, 2012 3:56 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
I found some old MX records for Google Gmail in my DNS Manager for one of my websites that I was trying to setup and gave up on several weeks ago. I took them out and I haven't had any speed issues since. Crossing fingers that was the only issue.

*update* nope that didn't help - having issue again and just noticed another "www-data" process taking up over 10% of the CPU.
Top
   
Reply with quote  
Azathoth
 Post subject:
PostPosted: Sat May 26, 2012 4:13 pm 
Offline
Senior Member

Joined: Mon Dec 07, 2009 6:46 am
Posts: 331
What do your access logs say?
Top
   
Reply with quote  
phowell32
 Post subject:
PostPosted: Sat May 26, 2012 4:59 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
Azathoth wrote:
What do your access logs say?


Hi Azathoth, thanks for the reply.

I'm a Linode newbie I admit. So when you say access logs, do you mean for each website (access.log)? They are huge, but I really don't see anything out of the ordinary when looking at them.
Top
   
Reply with quote  
Nuvini
 Post subject:
PostPosted: Sat May 26, 2012 5:21 pm 
Offline
Senior Member

Joined: Fri Feb 17, 2012 8:20 pm
Posts: 365
phowell32 wrote:
Azathoth wrote:
What do your access logs say?


Hi Azathoth, thanks for the reply.

I'm a Linode newbie I admit. So when you say access logs, do you mean for each website (access.log)? They are huge, but I really don't see anything out of the ordinary when looking at them.


Another thing you might try is iftop (apt-get install iftop) and then when the bandwidth goes up if you run it with sudo it should show you the IPs that are causing it. Might be able to narrow it down with the access.log files.
Top
   
Reply with quote  
Azathoth
 Post subject:
PostPosted: Sat May 26, 2012 5:44 pm 
Offline
Senior Member

Joined: Mon Dec 07, 2009 6:46 am
Posts: 331
phowell32 wrote:
They are huge, but I really don't see anything out of the ordinary when looking at them.


Nothing out of the ordinary during the peak bandwidth times? There must be something out of the ordinary if your site suddenly jumps from virtually nothing to 50 Mbps, and according to your process list sample there are 4 very active Apache processes.

Edit: What I'm trying to say is, could be some rogue script or wp module doing something nasty. This looks a lot like a situation I had not too long ago, a site had a "send to friend" feature which allowed basically any kind of email be sent anywhere. When the spammers got hold of that, they hammered the server with thousands of POSTs (which was out of the ordinary in the logs). 40k emails sent in under an hour.
Top
   
Reply with quote  
phowell32
 Post subject:
PostPosted: Sat May 26, 2012 6:20 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
Nuvini wrote:
phowell32 wrote:
Azathoth wrote:
What do your access logs say?


Hi Azathoth, thanks for the reply.

I'm a Linode newbie I admit. So when you say access logs, do you mean for each website (access.log)? They are huge, but I really don't see anything out of the ordinary when looking at them.


Another thing you might try is iftop (apt-get install iftop) and then when the bandwidth goes up if you run it with sudo it should show you the IPs that are causing it. Might be able to narrow it down with the access.log files.


Ok yes, iftop shows a ton of bandwidth from a few sources, it just doesn't stop. I'm going through the access logs for each site to see if I can find those IP's in the log. Will update once I find something.
Top
   
Reply with quote  
phowell32
 Post subject:
PostPosted: Sat May 26, 2012 6:34 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
Azathoth wrote:
phowell32 wrote:
They are huge, but I really don't see anything out of the ordinary when looking at them.


Nothing out of the ordinary during the peak bandwidth times? There must be something out of the ordinary if your site suddenly jumps from virtually nothing to 50 Mbps, and according to your process list sample there are 4 very active Apache processes.

Edit: What I'm trying to say is, could be some rogue script or wp module doing something nasty. This looks a lot like a situation I had not too long ago, a site had a "send to friend" feature which allowed basically any kind of email be sent anywhere. When the spammers got hold of that, they hammered the server with thousands of POSTs (which was out of the ordinary in the logs). 40k emails sent in under an hour.


Here are my firewall rules which should be blocking email:



Quote:
*filter

# Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

# Allow SSH connections
#
# The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT
Top
   
Reply with quote  
Azathoth
 Post subject:
PostPosted: Sat May 26, 2012 7:01 pm 
Offline
Senior Member

Joined: Mon Dec 07, 2009 6:46 am
Posts: 331
You could also grep out only the traffic during those peak load times from your access logs into a separate file, then run webalizer on that file, just be careful where you point its output and/or incremental (you don't need it for this), so you don't overwrite your regular stats.

Here's an example that extracts 4pm traffic of May 25th, and produces stat graphs in /var/www/incident_stats:

Code:
grep "25/May/2012:16" access.log > incident.log
webalizer -i -b -o /var/www/incident_stats incident.log


phowell32 wrote:
Here are my firewall rules which should be blocking email:


Nothing about email in those rules, and I meant outgoing not incoming.
Top
   
Reply with quote  
phowell32
 Post subject:
PostPosted: Sat May 26, 2012 9:03 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
I know this may sound weird but while watching "iftop" I notice I have much less traffic when I'm not connected via Putty as Root. I connected via LISH Ajax Console and there is much less traffic but when I connect via Putty with root I see a ton more and also the sites start to slow down.

I removed a bunch of old wp plugins on my sites and the traffic seems to have died down once again (doesn't mean it won't come roaring back though). I'll just watch it tonight and see if anything happens. I also turned KeepAlive back ON because I read that it saves your CPU at the cost of memory. Since CPU was getting destroyed I thought I would retry it.

Thanks for everyone's help!
Top
   
Reply with quote  
phowell32
 Post subject:
PostPosted: Wed May 30, 2012 5:45 pm 
Offline
Senior Newbie

Joined: Sat May 26, 2012 2:17 pm
Posts: 7
I wanted to give an update to this issue and I think I have found what was causing the problem.

While reviewing the accesslogs at a certain time when CPU and bandwidth went nuts I noticed one of my sites was getting a lot of access to plugins and every post over and over and over again. Also, there were a few old videos on the site that I completely forgot about and those were also being accessed. I'm assuming it was a bot because it all happens within a few seconds. I deleted the videos and moved the site off my Linode to another host.

From that point going forward I no longer had anymore issues.

Lesson learned, if you have issues with bandwidth, review your access logs!

Thanks to everyone for your help!
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Post new topic  Reply to topic

Board index » Linux Community Forums » Performance and Tuning


Who is online

Users browsing this forum: No registered users and 2 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group