Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Tue Jun 15, 2010 9:42 pm 
Offline
Senior Newbie

Joined: Sat Feb 20, 2010 12:43 am
Posts: 16
Hi, been having some very random issues with my site where the website will just display 'Database connection issue' too many connections. and a 'top' command shows mysql at 100% of CPU.

A mysql restart restores it fine but it keeps happening and I'm not sure why.

Which error logs are best to check on this , I tried var/log/mysql.log but theres nothing in it.

Thanks for any help


Top
   
 Post subject:
PostPosted: Wed Jun 16, 2010 9:43 am 
Offline
Senior Newbie

Joined: Sat Feb 20, 2010 12:43 am
Posts: 16
Also noticed from the graphs that whenever mysql goes to 100% (and the site goes down) then also the network ussage has a large spike. Are these some kind of http requests looing or something?

Thanks


Top
   
 Post subject:
PostPosted: Wed Jun 16, 2010 10:54 am 
Offline
Senior Member

Joined: Fri May 02, 2008 8:44 pm
Posts: 1121
Are you sure it's not just a large number of people visiting your site, or a half-assed ddos attempt? A large number of concurrent requests to database-heavy pages could cause the "too many connections" issue. Better check your web server logs for any unusual activity.


Top
   
 Post subject:
PostPosted: Wed Jun 16, 2010 6:03 pm 
Offline
Senior Newbie

Joined: Sat Feb 20, 2010 12:43 am
Posts: 16
Hi, no the access logs that my site generates show googlebots etc but nothing amazing.
I have Livezilla on it so can see who's coming in or out and only running one site and traffic has been probably even lower than normal (has been since Googles mayday changes).


Top
   
 Post subject:
PostPosted: Fri Jun 18, 2010 8:46 am 
Offline
Junior Member

Joined: Mon Feb 22, 2010 9:40 pm
Posts: 37
Maybe try using the slow query log and/or mytop?


Top
   
 Post subject:
PostPosted: Fri Jun 18, 2010 8:54 am 
Offline
Senior Member
User avatar

Joined: Tue Nov 24, 2009 1:59 pm
Posts: 362
IF that doesn't help, try running a full query log for a while (careful to not fill the disk - I personally directed the log to a fifo that was cat'd in a remote terminal with local logging enabled). Had similar problem, and slow log did not help because it was actually a flurry of quick* queries.
(*That is, a few dozen thousand full scans on a ~100MB table... unfortunately, each took less than 2 sec, so no slow log entries.)


Top
   
 Post subject:
PostPosted: Sun Jun 20, 2010 9:34 am 
Offline
Senior Newbie

Joined: Sat Feb 20, 2010 12:43 am
Posts: 16
Hi, thanks for the suggestions I have tried some of them and noted the last query that occurred when the site last went down just recently, I set the slow query log to 8 seconds and it only recorded one thing which was a standard query of which I also got in the browser when I reset mysql.


It does not indicate the issue as such so I am wondering if the last comment is more relevant as it is an issue where all the databse connections are being used for some reason, I will check that next.


Top
   
 Post subject:
PostPosted: Wed Jun 23, 2010 11:57 am 
Offline
Senior Newbie

Joined: Wed Jun 23, 2010 11:50 am
Posts: 6
WLM: brfsa@hotmail.com
Yahoo Messenger: brfsa
increase your mysql threads and threads cache if your mysql is using too much CPU.

high CPU is a sign of excessive forking.

For a Linode 768 I have:
# Threads:
thread_concurrency = 4
thread_cache_size = 6

_________________
Linux Sysadmin
Ruby on Rails Developer


Top
   
 Post subject:
PostPosted: Wed Jun 23, 2010 12:07 pm 
Offline
Senior Member
User avatar

Joined: Tue Nov 24, 2009 1:59 pm
Posts: 362
Quote:
high CPU is a sign of excessive forking.

Erm, what?

Quote:
thread_concurrency = 4

Interesting...

http://dev.mysql.com/doc/refman/5.1/en/ ... oncurrency
Quote:
This variable is specific to Solaris systems, for which mysqld invokes the thr_setconcurrency() with the variable value. This function enables applications to give the threads system a hint about the desired number of threads that should be run at the same time.


MySQL runs a thread per connection, unless you enable the single thread mode, which's extremely inefficient and used only for debug purposes.


Top
   
 Post subject:
PostPosted: Wed Jun 23, 2010 12:11 pm 
Offline
Senior Member

Joined: Sat Mar 28, 2009 4:23 pm
Posts: 415
Website: http://jedsmith.org/
Location: Out of his depth and job-hopping without a clue about network security fundamentals
brfsa wrote:
high CPU is a sign of excessive forking.

There's quite a few more likely things, depending on what "high CPU" really means.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.


Top
   
 Post subject:
PostPosted: Sun Jun 27, 2010 8:28 am 
Offline
Senior Newbie

Joined: Wed Jun 23, 2010 11:50 am
Posts: 6
WLM: brfsa@hotmail.com
Yahoo Messenger: brfsa
jed wrote:
brfsa wrote:
high CPU is a sign of excessive forking.

There's quite a few more likely things, depending on what "high CPU" really means.


TRUE, CPU usage is divided in 3 categories: user, system, iowait.
I had experience that mysql was taking on all IOwait, after an extensive mysql profiling we had to tune my.cnf to match the work load (joins, query cache, threads cache, sort_buffer etc)
have a look at the mysql sample config files under /usr/share/doc/mysql-x.x.xx/support-files/ {my-medium.cnf my-large.cnf my-huge.cnf}

Some tools you can use to find out your CPU usage are mpstat (found on sysstat package) and iostat

also: http://www.cyberciti.biz/tips/how-do-i- ... ation.html

_________________
Linux Sysadmin

Ruby on Rails Developer


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group