I have a linode 512 with cent OS. I have installed nginx, mysql, php on this. I am running a drupal website on this. Whenever my website gets moderate traffic, the linode starts behaving strange. I get an alert email saying "Linode Alert - disk io rate" and then problem starts -

1) MySql stops accepting connections. Although the MySql remains running and listening on its TCP port. But it does not accept any database connections. It gives the error as "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)". As the mysql still running, the monit does not help. I have to manually restarts the MySql

2)Sometime the linode becomes unaccessible. I cant even shh to it. I have to reboot it from linode.com web interface.


So here are my questions:
1) How to solve the problem that I am facing specifically the MySql problem.

2) What is the expected behavior if my linode exceed the threshold IO? My expectation is that the IO will become slow rather than the IO failure which leads the MySql to behave bad.

3) What are the general guidelines to make highly available and reliable web servers using linode. I am planning to build such a website but now fearing to use linode because of the above mentioned experience.

Please help.....

You're looking at this backwards. What you are experiencing is not Linode throttling your disk IO, which chokes mysql.

That email you are getting is just a warning which notifies you of what your node is doing.

If you dig into your logs and configs, you're going to discover that some problem between nginx/your site/mysql is causing your server to eat up all its RAM, causing it to push into swap space, which is thrashing your disk IO and generating the email alert.

Also, the Linode Library has guides for setting up HA.

You're probably swapping a lot.

How are you running php? Can you show your mysql configuration?

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

It sounds like you're running out of memory.

The high IO rate isn't the root problem, it's a symptom. Your node is running out of memory and hitting your swap space very hard (which obviously involves a lot of disk io). There isn't any "threshold" as far as your node is concerned, that's the just threshold at which you get notified. You can adjust that threshold up and down in the control panel.

The issues you're experiencing have nothing to do with Linode in particular, any Linux server under the same conditions (your configuration, load, and available resources) would be acting the same way.

I'd look at reducing your memory usage. How are you running PHP? If you're using php-fpm, what's your max_children setting set at? My guess is you're spinning up too many php fcgi processes.

You should also look into optimizing MySQL. mysqltuner.pl is a great script to get you started there. Run it, research the settings it suggests you change, and then tweak them appropriately.

Thanks for the reply.
I am using php-fpm

I am putting few lines from configuration files which may be useful for diagnosing problem

MySql config file
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

php.ini
[Pdo]
;pdo_odbc.connection_pooling=strict
;pdo_odbc.db2_instance_name

[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=

[MySQL]
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off

php-fpm.conf
listen = 127.0.0.1:9000
listen.backlog = -1
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.max_children = 50
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500


I guess tht pm.max_children may be the root cause.
By the way, Do you know any open source (or free) load test tool so that I can test my website before making it public.


Thanks again for your valuable suggestions.

Are you sure you can manage your server? Read guides.
http://library.linode.com/

I am a newbie as web-admin. I guess there is no harm in asking a question for getting help.

drop max children to 4.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

No harm in reading these guides too, be sure. It's very useful.

A mistake often made is when people don't close their mysql connection in their php code after the page has parsed, thus creating additional connections everytime and eventually makes mysql deny additionnal connections/run out of memory.