Linode Forum
Linode Community Forums 		 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic

Board index » Linux Community Forums » Performance and Tuning
  Previous topic | Next topic 
Author Message
Moses
PostPosted: Fri Apr 23, 2010 8:50 am 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
Hello All,

I've just switched to Apache2 due to fastcgi issueshowever, I'm finding Apache runs well for a while, then seems to stall. There's nothing showing up in the error log, and my SSH session is responsive, just loading the site takes ages.

I've noticed when the usage slows down the load seems reasonably high, around .5. Looking in htop it seems Apache is the main culprit for processor time.

I've tried a heap of different apache settings, but I'm really just guessing, I'd be very appreciative for any suggestions.

Ubuntu 8.04 LTS
Running Wordpress and SMF forum, around 4,000 visits per day.
Linode 360 with 540MB RAM (that was a stop gap to hope it stayed up during a promo we ran), considering upgrading to a Lindoe 540

Quote:
dpkg -l | grep apache
ii apache2 2.2.8-1ubuntu0.15 Next generation, scalable, extendable web se
ii apache2-mpm-prefork 2.2.8-1ubuntu0.15 Traditional model for Apache HTTPD
ii apache2-utils 2.2.8-1ubuntu0.15 utility programs for webservers
ii apache2.2-common 2.2.8-1ubuntu0.15 Next generation, scalable, extendable web se
ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
rc libapache2-mod-rpaf 0.5-3 module for Apache2 which takes the last IP f

dpkg -l | grep php
ii libapache2-mod-php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
ii php-pear 5.2.4-2ubuntu5.10 PEAR - PHP Extension and Application Reposit
ii php5 5.2.4-2ubuntu5.10 server-side, HTML-embedded scripting languag
ii php5-cli 5.2.4-2ubuntu5.10 command-line interpreter for the php5 script
ii php5-common 5.2.4-2ubuntu5.10 Common files for packages built from the php
ii php5-curl 5.2.4-2ubuntu5.10 CURL module for php5
ii php5-gd 5.2.4-2ubuntu5.10 GD module for php5
ii php5-mcrypt 5.2.3-0ubuntu1 MCrypt module for php5
ii php5-mysql 5.2.4-2ubuntu5.10 MySQL module for php5
ii php5-suhosin 0.9.22-1 advanced protection module for php5
ii php5-xcache 1.2.1-3 Fast, stable PHP opcode cacher
ii phpmyadmin 4:2.11.3-1ubuntu1.3 Administrate MySQL over the WWW




Extract from /etc/apache2/apache.conf
Quote:
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 30

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 50

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 3
MaxSpareServers 8
MaxClients 10
MaxRequestsPerChild 1000
</IfModule>




Any suggestions for the next step? I was considering moving to the worker MPM, but now I'm thinking nginx to proxy static files??

Thanks,
Moses.
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Fri Apr 23, 2010 9:05 am 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
Here's some munin charts

Image

Image

Image

Image

Image

Image
Top
   
Reply with quote  
sleddog
 Post subject:
PostPosted: Fri Apr 23, 2010 10:12 am 
Offline
Senior Member

Joined: Sun Aug 31, 2008 4:29 pm
Posts: 177
It's possible that you're hitting the MaxClients limit (10). If you are then Apache will log messages to the error log. Make sure you're checking the main Apache error log and not a domain-specific error log. MaxClient messages will appear in the main error log, probably in /etc/apache2/logs.
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Fri Apr 23, 2010 10:29 am 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
Thanks sleddog, didn't realise there was another error.log , this one is at /var/log/apache2/error.log .

I have been getting a few MaxClients warnings, though wont raising this eat into more RAM? I've lifted it to 15 for the moment, will see how that goes overnight. I guess going to nginx will mean less RAM for each apache process, hence more clients can run...

Quote:
top - 00:24:20 up 1 day, 1:05, 1 user, load average: 0.35, 0.18, 0.15
Tasks: 95 total, 3 running, 92 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.0%us, 0.4%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 553184k total, 516256k used, 36928k free, 50916k buffers
Swap: 393208k total, 52k used, 393156k free, 288804k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21144 mysql 15 0 73620 49m 4728 S 2 9.1 5:53.37 mysqld
12354 www-data 16 0 46524 24m 15m R 20 4.5 0:02.01 apache2
12296 www-data 15 0 46284 24m 15m S 0 4.5 0:02.40 apache2
12355 www-data 15 0 46904 24m 14m S 0 4.5 0:01.38 apache2
12285 www-data 15 0 46132 24m 15m S 0 4.5 0:03.16 apache2
11605 www-data 16 0 45620 23m 15m R 5 4.4 0:06.41 apache2
9432 www-data 15 0 44144 22m 15m S 0 4.2 0:18.23 apache2
11714 www-data 15 0 44324 22m 15m S 0 4.2 0:04.59 apache2
12348 www-data 15 0 44152 20m 13m S 0 3.8 0:01.12 apache2
27480 root 18 0 40476 7400 4268 S 0 1.3 0:00.04 apache2
2574 root 15 0 6816 4956 1544 S 0 0.9 0:00.20 munin-node
2685 root 18 0 23728 4488 1776 S 0 0.8 0:00.28 fail2ban-server
22795 moses 15 0 4424 2988 1252 S 0 0.5 0:00.09 bash




I also note there are a bunch of xcache errors in this log, that was cause I had the xcache settings in /etc/php5/apache2/conf.d/xcache.ini as well as /etc/php5/apache2/php.ini
As I was trying to load xcache twice, it took the first settings with just 16MB RAM. I've increased this to 64MB and am hoping this will take some load of apache and mysql??

Quote:
[Fri Apr 23 12:03:37 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured -- resuming normal operations
[Fri Apr 23 12:05:29 2010] [notice] caught SIGWINCH, shutting down gracefully
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 12:05:31 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured -- resuming normal operations
[Fri Apr 23 12:05:36 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Fri Apr 23 12:16:23 2010] [notice] Graceful restart requested, doing restart
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 12:16:23 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured -- resuming normal operations
[Fri Apr 23 21:14:37 2010] [notice] Graceful restart requested, doing restart
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 21:14:37 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured -- resuming normal operations
[Fri Apr 23 21:51:39 2010] [notice] caught SIGWINCH, shutting down gracefully
PHP Warning: Module 'XCache' already loaded in Unknown on line 0
[Fri Apr 23 21:51:41 2010] [notice] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch configured -- resuming normal operations
[Fri Apr 23 21:51:45 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting


Thanks again for looking at this,
Moses.
Top
   
Reply with quote  
sleddog
 Post subject:
PostPosted: Fri Apr 23, 2010 12:44 pm 
Offline
Senior Member

Joined: Sun Aug 31, 2008 4:29 pm
Posts: 177
I think you did the right thing by making a small increase in MaxClients. Just monitor the error log and your memory usage for a couple days to see how it goes. Every Apache process is not using an additional 15M of memory -- a good part of that is shared memory between all Apache processes.

If you're still getting MaxClients errors try increasing the setting to 20...

Also, have you done any mysql optimization such as enable thread, query and table caches...?
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Fri Apr 23, 2010 7:20 pm 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
I had made a few changes to mysql, current excerpt is

Quote:
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 1M
thread_stack = 64K
thread_cache_size = 8
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query-cache-type = 1
query-cache-size = 32M
query_cache_limit = 1M
#query_cache_size = 20M



I'm thinking of removing some #'s and changing it to (will do some research first)
Quote:
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 1M
thread_stack = 64K
thread_cache_size = 8
#max_connections = 100
table_cache = 64
thread_concurrency = 10
#
# * Query Cache Configuration
#
query-cache-type = 1
query-cache-size = 32M
query_cache_limit = 1M
query_cache_size = 20M
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Fri Apr 23, 2010 8:50 pm 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
I re-enabled xcache yesterday afternoon, and this has led to a return of suhosin's canary errors in my site specific log

Quote:
[Sat Apr 24 10:27:04 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/index.php')
[Sat Apr 24 10:27:07 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
[Sat Apr 24 10:27:07 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
[Sat Apr 24 10:27:08 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
[Sat Apr 24 10:27:08 2010] [error] [client 23.23.23.23] ALERT - canary mismatch on efree() - heap overflow detected (attacker '23.23.23.23', file '/home/public_html/greenandgoldrugby.com/public/wp-admin/index.php')
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Fri Apr 23, 2010 8:57 pm 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
Read this

http://www.suspekt.org/2008/10/12/suhos ... -detected/

I seem to have this nagging feeling somewhere that xcache and suhosin don't agree with each other but for the life of me I can't remember why.

Have you tried APC?
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Fri Apr 23, 2010 10:03 pm 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
G'Day obs,
Thanks for that link, I'm pretty sure I've read it before, though just took the advice of it's last comment at set
suhosin.session.encrypt=Off

I had a look at APC but didn't have a lot of luck installing it, got sort of confused around pecl etc.. then xcache was a simple apt-get install so went with that
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Fri Apr 23, 2010 10:59 pm 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
apt-get install php-apc

No need for pecl

Although if you do want to use pecl it's

pecl install apc

if you're using php 5.3 it's

pecl install apc-beta

just make sure you add extension=apc.so to your php.ini
Top
   
Reply with quote  
hoopycat
 Post subject:
PostPosted: Sat Apr 24, 2010 12:11 pm 
Offline
Senior Member
User avatar

Joined: Sat Aug 30, 2008 1:55 pm
Posts: 1739
Location: Rochester, New York
Not sure if this has been tried already (I've got slow intertubes right now), but try setting KeepAlives off, or set MaxKeepAliveRequests down to a low number, significantly low than MaxClients; otherwise, you'll have a lot of idle connections just sitting there waiting for time to pass.
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Wed Apr 28, 2010 12:40 am 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
sleddog wrote:
It's possible that you're hitting the MaxClients limit (10). If you are then Apache will log messages to the error log. Make sure you're checking the main Apache error log and not a domain-specific error log. MaxClient messages will appear in the main error log, probably in /etc/apache2/logs.

I noticed that I was hitting the MaxClient at 15 as well, so upped it to 20. Then I went into swap memory so dropped it to 18. Seems better there, but hit swap memory again later today so I've reduced MaxRequestsPerChild from 1000 to 250, hopefully this will keep the 18 apache processes growing too large before respawning.

I'll add the nginx proxy this weekend and that should help a lot to keep apache in check, will go to APC at the same time..
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Fri Apr 30, 2010 11:39 pm 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
seems the apc package is not in the repository for 8.04 LTS

I tried with pecl however that failed, so ended up installing 3.1.13.p1 from source. Seems pretty good and I really like the stats page it gives
Top
   
Reply with quote  
obs
 Post subject:
PostPosted: Sat May 01, 2010 4:46 am 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
Ah didn't check the 8.04 since I use 9.10, but yeah it's a nice add-on and for php 5.3 you need 3.1 which is "beta" but I've had no problems with it.
You might want to look at upgrading to 10.4 LTS in a month or so.
Top
   
Reply with quote  
Moses
 Post subject:
PostPosted: Sun May 02, 2010 9:32 pm 
Offline
Senior Member
User avatar

Joined: Tue Dec 01, 2009 10:31 pm
Posts: 57
Well after getting APC to install once on the dev site I'm clueless how to get it working again!

I can install it from source, but when I reload apache get an error about

Quote:
PHP Warning: PHP Startup: apc: Unable to initialize module\nModule compiled with module API=20090626, debug=0, thread-safety=0\nPHP compiled with module API=20060613, debug=0, thread-safety=0\nThese options need to match\n in Unknown on line 0


I believe the error is coming from conflicting PHP versions?

My PHP Extension is 20060613
and the release of APC i'm trying to install (3.1.3p1) is 20090626

Any suggestions?
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Post new topic  Reply to topic

Board index » Linux Community Forums » Performance and Tuning


Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group