Hi, I got up in the morning and noticed that my Linode was down for 6 hours. Linode support has told me that my Linode kernel panicked as a result of running out of memory.



I did this and rebooted my Linode.



I'm relatively new to Linux (CentOS) and I would like to ask for some help identifying what the cause of the problem is.

free -m yields this output:

             total       used       free     shared    buffers     cached
Mem:          1004        407        597          0         10        156
-/+ buffers/cache:        239        764
Swap:          255          0        255

ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r | less
yields this output:

%MEM %CPU   RSS    VSZ COMMAND
 2.8  6.7 29496 125076 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
 1.9  0.2 20520  35180 /usr/sbin/httpd
 1.9  0.1 20516  35164 /usr/sbin/httpd
 1.8  0.3 18848  33388 /usr/sbin/httpd
 1.8  0.2 19152  33848 /usr/sbin/httpd
 1.8  0.2 18980  33632 /usr/sbin/httpd
 1.8  0.2 18840  33856 /usr/sbin/httpd
 1.8  0.2 18648  33360 /usr/sbin/httpd
 1.8  0.1 19420  34104 /usr/sbin/httpd
 1.8  0.1 19180  33888 /usr/sbin/httpd
 1.8  0.1 18860  33880 /usr/sbin/httpd
 1.8  0.1 18608  33644 /usr/sbin/httpd
 1.8  0.0 18820  33576 /usr/sbin/httpd
 1.7  0.1 18456  33104 /usr/sbin/httpd
 1.7  0.1 18384  33364 /usr/sbin/httpd
 1.7  0.1 18372  33360 /usr/sbin/httpd
 1.7  0.1 18324  33344 /usr/sbin/httpd
 0.8  0.0  9068  23928 /usr/sbin/httpd
 0.4  0.0  4548   4552 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 0.3  0.0  3248  10100 sshd: root@ttyp0

Any help would be appreciated.

From what you posted, your server looks OK for now.

What are you running on the Linode, e.g. WordPress, Drupal, phpBB, etc? Does any of that involve image processing, like generating thumbnails from large photos?

Is there anything else running on the server, e.g. dovecot?

Please post your httpd.conf and my.cnf. Those are the usual suspects.

This is my my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
#old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysqld]
# someother settings
socket=/var/lib/mysql/mysql.sock

#added these lines for foreign characters
[client]
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
[mysqld]
default-character-set = utf8
skip-character-set-client-handshake
character-set-server = utf8
collation-server = utf8_general_ci

init-connect = "SET collation_connection = utf8_general_ci"
init-connect = "SET NAMES utf8"

This is a part of my httpd.conf

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

I am running Wordpress and some custom made sites with Perl. I don't use any image processing at all.

I do a lot of MySQL database operations but I'm not sure if that's the cause.

Any thoughts on my configurations?

You seem to have left out a huge chunk of my.cnf. Where's the part with key_buffer and all those memory usage settings?

But never mind, here's your most likely culprit.

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
</IfModule>

That should be changed to something like:

<IfModule prefork.c>
StartServers       4
MinSpareServers    4
MaxSpareServers    8
ServerLimit       24
MaxClients        24
</IfModule>

Restart Apache, and you'll probably not see another OOM for a long time.

Thank you so much. I applied these changes and hope the OOM issue is completely gone.

My suggestion is start monitoring the instance so you know it's not responding. Something like nagios will work.

_________________
Larry Ludwig
Empowering Media
Managed Cloud Services and Managed VPS