Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Wed Apr 14, 2010 10:10 pm 
Offline
Newbie

Joined: Thu Apr 08, 2010 7:41 am
Posts: 4
I found strange problem on my server: after 10 days of working the server stopped processing requests, CPU loading is 400%!!!

Image

How can I solve this problem? Or how can I make my server to reboot if CPU is loaded more than 40%? Because after reloading the server works well again.


Top
   
 Post subject:
PostPosted: Wed Apr 14, 2010 10:56 pm 
Offline
Senior Member
User avatar

Joined: Sat Aug 30, 2008 1:55 pm
Posts: 1739
Location: Rochester, New York
When that happens, your best bet is to look at the console and figure out what broke:

http://library.linode.com/linode-manager/using-lish-the-linode-shell

My guess is that the kernel locked up. It's rare, but it happens. The "logview" command in lish should still have the information from before the last reboot.


Top
   
 Post subject:
PostPosted: Tue May 11, 2010 3:11 pm 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
I'm getting this same exact problem.

Running CentOS 5.x
Virtualmin
Apache
MysSql

and running 3 Wordpress sites with 830MB of RAM

I can't find anything in logs until the server is locked up and reports its out of swap and memory. No large spikes of traffic or other cpu load leading up to this.

Code:
 free m
             total       used       free     shared    buffers     cached
Mem:        829644     666756     162888          0      15676     161228
-/+ buffers/cache:     489852     339792
Swap:       262136          0     262136


I then have to reboot the linode in order to solve the problem.


Top
   
 Post subject:
PostPosted: Tue May 11, 2010 3:16 pm 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
They're are not high traffic sites either. Avg cpu usage is < 5% avg network is 5kb/s combined.


Top
   
 Post subject:
PostPosted: Tue May 11, 2010 7:34 pm 
Offline
Senior Member

Joined: Sat Mar 28, 2009 4:23 pm
Posts: 415
Website: http://jedsmith.org/
Location: Out of his depth and job-hopping without a clue about network security fundamentals
You'll want to glance at this: http://library.linode.com/troubleshooti ... ory_issues

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.


Top
   
 Post subject:
PostPosted: Tue May 11, 2010 8:33 pm 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
My concern is, I have a dedicated Dell 1850. 2GB RAM 10k scsi drives. 100 sites, all dynamic, 300gb traffic a month. Same setup as this VPS - CentOS, Virtualmin, and no tweaks. Runs without issue. This VPS has 4 sites that barely see any traffic and it randomly croaks. Maybe I'm overestimating what a VPS should be capable of handling?

Here are the stats from my VPS in accordance to that article.

Code:

iostat -d -x 2 5   
Linux 2.6.18.8-x86_64-linode10 ()    05/12/2010

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.14     1.82  0.54  1.08    15.97    23.26    24.22     0.02   11.08   3.01   0.49
xvdb              0.00     0.00  0.00  0.00     0.02     0.00    26.22     0.00    9.39   9.11   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00     0.00  0.00  0.50     0.00     4.00     8.00     0.00    0.00   0.00   0.00
xvdb              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdb              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdb              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvdb              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00


Code:
httpd -V | grep 'MPM'
Server MPM:     Prefork
 -D APACHE_MPM_DIR="server/mpm/prefork"


Code:

# Timeout: The number of seconds before receives and sends time out.
#
Timeout 30

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serve
<IfModule prefork.c>
StartServers       5
MinSpareServers    5
MaxSpareServers   10
ServerLimit      128
MaxClients       128
MaxRequestsPerChild  2000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2
MaxClients           3
MinSpareThreads      3
MaxSpareThreads      7
ThreadsPerChild      3
MaxRequestsPerChild  200
</IfModule>


after an http restart:
Code:

free -m
             total       used       free     shared    buffers     cached
Mem:           810        361        449          0         21        150
-/+ buffers/cache:        189        620
Swap:          255          0        255


Code:
uname -ra
Linux li125-138 2.6.18.8-x86_64-linode10 #1 SMP Tue Nov 10 16:29:17 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux


Code:

cat /etc/my.cnf
[mysqld]
port            = 3306
socket          = /var/lib/mysql/mysql.sock
skip-locking
key_buffer = 16K
max_allowed_packet = 1M
table_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K

# For low memory, Berkeley DB should not be used so keep skip-bdb uncommented unless required
skip-bdb

# For low memory, InnoDB should not be used so keep skip-innodb uncommented unless required
skip-innodb

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql/
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50


[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 8M
sort_buffer_size = 8M

[myisamchk]
key_buffer = 8M
sort_buffer_size = 8M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid


Top
   
 Post subject:
PostPosted: Tue May 11, 2010 8:55 pm 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
these suckers get pretty large - pretty quick

Code:

%MEM %CPU   RSS    VSZ COMMAND
 6.6  0.1 55444 326360 /usr/sbin/httpd
 6.5  0.0 54536 325416 /usr/sbin/httpd
 6.5  0.0 54356 325352 /usr/sbin/httpd
 6.5  0.0 54264 325160 /usr/sbin/httpd
 6.5  0.0 54124 325100 /usr/sbin/httpd
 6.4  0.0 53628 324528 /usr/sbin/httpd
 1.8  0.0 15664 282452 /usr/sbin/httpd


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 10:59 am 
Offline
Senior Member
User avatar

Joined: Tue May 26, 2009 3:29 pm
Posts: 1691
Location: Montreal, QC
mattm wrote:
these suckers get pretty large - pretty quick

Code:

%MEM %CPU   RSS    VSZ COMMAND
 6.6  0.1 55444 326360 /usr/sbin/httpd
 6.5  0.0 54536 325416 /usr/sbin/httpd
 6.5  0.0 54356 325352 /usr/sbin/httpd
 6.5  0.0 54264 325160 /usr/sbin/httpd
 6.5  0.0 54124 325100 /usr/sbin/httpd
 6.4  0.0 53628 324528 /usr/sbin/httpd
 1.8  0.0 15664 282452 /usr/sbin/httpd


Apache is a notorious memory hog. You'll want to either tweak your Apache config or switch to a web server with a sane default setup.


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 11:06 am 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
Yes, that's what I read all over the forums. I think it (may) have been one of your threads where you said (or someone) said "tweak" apache use WP-Supercache and you would have no problem serving 15k hits a day on a 360 node.

I posted my worker process settings up top on this thread - maybe it's too little? These sites barely push any traffic and then out of no where CPU and IO spikes 400 times the norm then the server locks up.


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 11:06 am 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
I've since purchased more memory from ~800 to ~1100


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 11:12 am 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
Image

Image


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 11:42 am 
Offline
Senior Member
User avatar

Joined: Fri Dec 11, 2009 7:09 pm
Posts: 168
You are using prefork- not worker,
Quote:
httpd -V | grep 'MPM'
Server MPM: Prefork
-D APACHE_MPM_DIR="server/mpm/prefork"
so you need to tweak the prefork values-

Quote:
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serve
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 128
MaxClients 128
MaxRequestsPerChild 2000
</IfModule
>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 3
MinSpareThreads 3
MaxSpareThreads 7
ThreadsPerChild 3
MaxRequestsPerChild 200
</IfModule>



This is mine on a 360-
Quote:
<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 3
MaxSpareServers 6
Serverlimit 24
MaxClients 24
MaxRequestsPerChild 3000
</IfModule>


Seems to be working pretty well- I hit the max clients, but don't get in to the swap much (though I am still tweaking).

_________________
--
Chris Bryant


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 4:10 pm 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
thanks for adding yours. What kind of traffic are you getting to your 360?

What is the best method to monitor these processes, esp during normal traffic hours?


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 4:38 pm 
Offline
Senior Member
User avatar

Joined: Fri Dec 11, 2009 7:09 pm
Posts: 168
I have 3 sites on it- 1 static and 2 wordpress. The static site gets around 6-700 "visits" per day, and the wordpress sites each get around 200.

I have had it crash and burn, but now I use monit and Munin- Munin simply reports, but I have monit set to restart apache if it gets to memory hungry, and have made the changes outlined in http://www.linode.com/wiki/index.php/Rebooting_on_OOM if the whole thing borks- which it has once.

When I get time I'm going to get another 360 and set it up with worker (or nginx), and move everything over.

I'm way far from a guru- though I've used Linux at home for nearly 10 years now, I'm still a newbie when it comes to server administration :).

_________________
--

Chris Bryant


Top
   
 Post subject:
PostPosted: Wed May 12, 2010 4:51 pm 
Offline
Junior Member

Joined: Sun Mar 21, 2010 11:19 pm
Posts: 45
Nice - thanks for your input.

Code:
 free -m
             total       used       free     shared    buffers     cached
Mem:          1170        816        353          0         33        211
-/+ buffers/cache:        571        598
Swap:          255          0        255


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group