Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Mon Jul 02, 2012 5:07 am 
Offline
Newbie

Joined: Sun Jul 01, 2012 1:10 pm
Posts: 3
I have a nginx + php+fpm Linode server with apc and running wordpress and w3 total cache.

The server was performing well (for a month without downtime) until yesterday where the CPU suddenly spiked, memory maxed out (1024MB) and load average of 20, 19.6, 18.4 (quad core). I have to restart the php-fpm every 5 minutes to keep it running.

After some meddling and troubleshooting, I have managed to identify the issue with the wordpress database. It is probably due to a corrputed database. The frontend was working fine without any issue, but everytime i save post in the backend, the CPU spike, php5-fpm process took over and crash the server. The problem is, whatever i do, it keeps returning a 504 gateway error.

When I save post in wp: 504 gateway error
when i import database in phpmyadmin: 504 gateway error

I have made plenty of changes to the nginx.conf, php.ini, http://www.conf and my.cnf and increase all the possible timeout and it doesn't seem to solve the 504 gateway error problem.

The situation now is that the site is still live, but I can't touch anything in the WordPress backend as it will cause the cpu to spike and server crash.

What actually went wrong? Can anyone help?

Below is my settings:

nginx.conf
Code:
# Generic startup file.
user damien;
worker_processes  4;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

# Keeps the logs free of messages about not being able to bind().
#daemon     off;

events {
        worker_connections  1024;
}

http {
#       rewrite_log on;

        include mime.types;
        default_type       application/octet-stream;
        access_log         /var/log/nginx/access.log;
        sendfile           on;
#       tcp_nopush         on;
        keepalive_timeout  0;
#       tcp_nodelay        on;
#       gzip               on;
        client_max_body_size 200M;
        client_body_buffer_size 128k;
        client_header_buffer_size 256k;
        client_header_timeout 300;
        client_body_timeout 300;
        send_timeout 300;
        index index.php index.html index.htm;

        # Upstream to abstract backend connection(s) for PHP.
        upstream php {
#               server unix:/tmp/php-fpm.sock;
                server 127.0.0.1:9000;
        }
        server {
                location ~* wp-config.php {
                    deny all;
                }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
                location ~ \.php$ {
                    try_files $uri =404;
                    fastcgi_split_path_info ^(.+\.php)(/.+)$;
                    fastcgi_pass 127.0.0.1:9000;
                # With php5-fpm:
                #fastcgi_pass unix:/var/run/php5-fpm.sock;
                    fastcgi_index index.php;
                    include fastcgi_params;
                    fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                    fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
                    fastcgi_read_timeout 1800;
                    fastcgi_send_timeout 1800;
                    fastcgi_connect_timeout 1800;
                    fastcgi_buffer_size 128k;
                    fastcgi_buffers 4 256k;
                    fastcgi_busy_buffers_size 256k;
                    fastcgi_temp_file_write_size 256k;
                    fastcgi_intercept_errors on;
                }

                location ~ /\. {
                     access_log off;
                     log_not_found off;
                     deny all;
                }

                if ($host ~* ^www\.(.*)) {
                    set $remove_www $1;
                    rewrite ^(.*)$ http://$remove_www$1 permanent;
                }

        }

        include sites-enabled/*;
}


http://www.conf
Code:
pm = dynamic

pm.max_children = 35

pm.start_servers = 10

pm.min_spare_servers = 2

pm.max_spare_servers = 32

;pm.process_idle_timeout = 10s;
 
pm.max_requests = 400


my.conf
Code:
[client]
port      = 3306
socket      = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket      = /var/run/mysqld/mysqld.sock
nice      = 0

[mysqld]
user      = mysql
pid-file   = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port      = 3306
basedir      = /usr
datadir      = /var/lib/mysql
tmpdir      = /tmp
lc-messages-dir   = /usr/share/mysql
skip-external-locking
log-error    = /var/log/mysql.err

bind-address      = 127.0.0.1

key_buffer      = 256M
max_allowed_packet   = 16M
thread_stack      = 192K
thread_cache_size       = 20

myisam-recover         = BACKUP
max_connections        = 80
table_cache            = 512
thread_concurrency     = 10

query_cache_limit   = 8M
query_cache_size        = 64M

myisam_sort_buffer_size = 32M
sort_buffer_size = 1M
read_buffer_size = 1M
join_buffer_size = 5M

log_slow_queries   = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

# * InnoDB
#
#skip-innodb
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

max_heap_table_size  = 80M
tmp_table_size       = 80M


[mysqldump]
quick
quote-names
max_allowed_packet   = 16M

[mysql]
#no-auto-rehash   # faster start of mysql but no tab completition

[isamchk]
key_buffer      = 16M

[myisamchk]
key_buffer = 64M
sort_buffer_size = 64M
read_buffer = 16M
write_buffer = 16M


Top
   
PostPosted: Mon Jul 02, 2012 5:32 am 
Offline
Senior Member
User avatar

Joined: Sat Aug 30, 2008 1:55 pm
Posts: 1739
Location: Rochester, New York
Would you say it started at around midnight UTC on July 1? If so, reboot. You've hit a bug related to leap-second handling.

_________________
Code:
/* TODO: need to add signature to posts */


Top
   
PostPosted: Mon Jul 02, 2012 5:48 am 
Offline
Newbie

Joined: Sun Jul 01, 2012 1:10 pm
Posts: 3
Yes, the server problem start occurring at around UTC midnight, but I have already reboot two times and the same problem occur. What's next?


Top
   
PostPosted: Tue Jul 03, 2012 1:36 pm 
Offline
Sysop

Joined: Sat Nov 27, 2010 3:32 am
Posts: 180
Website: https://blog.timheckman.net/
Location: San Francisco, CA
Is it the PHP process that seems to use CPU, or MySQL? Have you watched how things progress with something like htop?

-Tim

_________________
'If debugging is the process of removing bugs, then programming must be the process of putting them in.' //Edsger Dijkstra
'Nothing is withheld from us which we have conceived to do.' | 'Do things that have never been done.' //Russell Kirsch


Top
   
PostPosted: Wed Jul 18, 2012 8:19 pm 
Offline
Newbie

Joined: Sun Jul 01, 2012 1:10 pm
Posts: 3
Things seem to be working fine now.

Both the mysql and php are causing issue. Even after I fix the leap second issue, the problem still persists.

What I did is to remove the php-apc and reinstall it. Additionally I also installed memcached and everything is back to normal and is running even faster. Seems that the php-apc module is the one causing trouble.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group