Hi all, like the topic says I am a new Linode user. New user to managing my own LAMP server in general actually, I was essentially booted from my current shared host a couple weeks ago so I just moved over a few of my sites to Linode over the past couple days.

I'm extremely out of my comfort zone managing this server but trying to do my best. I essentially have two issues:

1) My IO Rate/Swap IO graph is way out of control. Apparently the default warning level is 1000 but I am averaging about 20 times that. Here is the actual graph: http://img819.imageshack.us/img819/9307 ... graph1.png

What exactly do these numbers represent? Are these as out of control as they seem? Are they are high level common fixes for them that I can make to either my code or the server configuration itself?

2) I've been told by my shared host in the past that I am using too much CPU. I assumed this was just BS but now that I see a CPU graph they may have been correct. If I just put my sites up the % hovers been 10 and 30. However I also have some crawlers that I set to run in the background every 15 minutes. When they are on the CPU spikes to 80-100%. Here is the graph: http://img515.imageshack.us/img515/3036 ... graph2.png -- you can probably guess when the crons were running.

Is 10-30% throughout the day too high? Are there any general tips to keep it down?

For the crons that are spiking the CPU, they are basically crawling other websites, copying information into a local MySQL database and copying thumbnails over to the webserver. I will be switching from hosting the thumbnails myself to S3 in the next couple days time permitting, unsure if this will make a major difference or not. I am thinking to combat this I am going to spend some more on linode. I'm wondering what you think would be the best option:

1) Upgrade the current linode from 512 to 768 [or even higher potentially]
2) Buy a second linode 512 and run all the crons on there and keep current one to handle MySQL & webserver
3) Buy a second linode 512 and move MySQL there, keep current one for webserver, not sure where crons would go -- probably on the one with the least load

I don't want to spend a bunch of money and I'm probably capping it at ~$40/mo on Linode. Which of the three is my best bet moving forward?

I should also note that despite these graphs and everything else my sites seem to be running fine.
_________________
Andrew G Johnson
http://www.andrewgjohnson.com

To fix the swap issue check out this http://library.linode.com/troubleshooti ... networking especially the apache bit.

The cpu isn't anything you need to worry about, you can go as far as 400% before all your cpu power is being used since you have 4 cores, so 100% = 1 core.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Ok cool will check out that link.

I had my crons on a bit yesterday [outside range on that graph] and it spiked as high as ~120% and I freaked out. Do you see any issue in keeping things the way they are -- meaning crons on, CPU jumping between 10 and 100 percent all day?

Also I'm still going to need to still get a second linode or upgrade my current one to make sure I'm under the bandwidth limit and will some wiggle room. Do you suggest me just upgrading my current one or going to 2 [one for MySQL, one for webserver]? I would imagine upgrading the current one would be the easiest for me to deal with.

_________________
Andrew G Johnson
http://www.andrewgjohnson.com

Yes, your I/O is out of control. CPU is also quite high, but I suspect that most of it is due to heavy I/O. Get the I/O under control and CPU usage will probably go down, too.


According to the tutorial:
- ServerLimit 24
- MaxClients 24

Even that can be too high, especially if you use one of those modern, heavyweight CMS's like WordPress. Reduce them both to 16 and see if it helps reduce CPU usage and disk I/O.

Do this, install APC, and run mysqltuner.pl to optimize your MySQL settings, before you do anything as dramatic as upgrading to a larger linode or buying another linode. A properly optimized Linode 512 can pump out an awful lot of web pages.

Hmm it seems like that linode library article and mysqltuner.pl are giving me conflicting advice. One good thing is apparently I have a join somewhere without an index so I will hunt that down but other than that I am still kind of stuck. Making those changes didn't lower my IO graph at all, in fact it's even higher now.

I installed and implemented memcache yesterday [as opposed to APC.] I had a cache system previously that just wrote/read files but have replaced that with memcache since I now am on a VPS rather than a shared host.

I served something between 5000 and 6000 pageviews yesterday. I don't think that is very high but let me know. Once I move all my sites over I'll be peaking somewhere between 7000 and 8000 pageviews a day.



I am still wondering about this, I don't think it's a big leap because as I mentioned I am probably going to be hitting my monthly bandwidth limit. Either upgrading my current linode or getting a second one [I believe linode pools all your bandwidth together right?] would solve that so I was wondering which would likely give me a better performance boost.

_________________
Andrew G Johnson

http://www.andrewgjohnson.com

Also I tried those commands they show at the beginning of the linode library article and all I see are a huge amount of numbers that I don't know how to interpret :/

_________________
Andrew G Johnson

http://www.andrewgjohnson.com

Just realized I didn't answer your question/comment. I'm not running wordpress. I have a SMF forum that I run that gets something around 1000 pageviews/day but the rest of my sites are custom written software by me. Not really sure how to rank how good/resource-intensive they are.

_________________
Andrew G Johnson

http://www.andrewgjohnson.com

These three commands should provide a good snapshot of what's going on system-wide right about now:

free -m
ps aux
vmstat 1 30

(the vmstat command will run for 30 seconds, then exit)

Post the output here and everyone will opine on it.

From a prioritization standpoint, getting your swap I/O under control is probably #1. That slows everything down.

Next, your disk I/O is rather high. Figuring out whether it is reading or writing is key, as is figuring out exactly what is doing the reading or writing. The vmstat command will help a bit with this, but iotop is sometimes useful. (Most of the time, though, it doesn't tell me anything I don't already know. Whoop-de-flippin'-doo, mysqld is doing most of the I/O on my database server...)

CPU is CPU. There are lots of them in each host, and the most you can use at one time are four. The kicker with CPU is that you can only crunch data so fast, so if you're using a lot of CPU, your site is going to be slow.

If you can split up your tasks between multiple servers such that they're all sharing the I/O burden equally, life can be pretty good. I usually end up having a set of database servers, a set of web servers, and then some number of servers for other services (solr, memcached, postfix, munin/nagios/etc, SVN, ...). Make a block diagram of your operations, with lines denoting dependency, and little symbols indicating "does a lot of IO", "uses CPU like a drunken squirrel", "needs 4 GB of RAM", etc. You don't need to deploy your system such that you have no more than one block per server, but since each block should be able to talk to the other blocks it depends on over IP, you could.

(Beware the dreaded Distributed Single Point of Failure.)

_________________

/* TODO: need to add signature to posts */

Every article has specific performance goals. The MySQL portion of the article is meant to minimize memory consumption as much as possible. This might not be a good idea when you're trying to reduce CPU and I/O. Follow mysqltuner.pl, because it knows more about your specific setup than any general-purpose guide ever can. Besides, mysqltuner.pl will also tell you if your settings use excessive memory.


That could be the reason for your high CPU and I/O. An unindexed query needs to scan the entire table, which obviously causes a lot of disk access. Turn on slow query logs and you'll be able to find the offending query in a few hours.


Memcache is not a replacement for APC. Memcache reduces database load by storing query results in memory. APC makes PHP scripts run faster by storing the compiled opcode in memory. Use them both, but watch your memory usage. (You don't need to do anything to take advantage of APC. Just install it and forget about it.)


That's very low. A well-optimized Linode 512 running an off-the-shelf blog or forum software (including SMF) can serve thousands of dynamic pages every few minutes, let alone every day.


If you're hitting your bandwidth limit, yes, both options would cost you the same. But there's little reason to get two linodes unless your linode is already too big to upgrade or you need geographical redundancy. Just upgrade. It's easier to maintain one server than it is to maintain multiple servers and make them interoperate all the time.

Thanks for all the tips so far guys.

RE: APC, thanks I didn't realize that. I thought it was just a key/value store didn't realize it did this automatic file cache stuff. I'm looking at the graph/reporting stuff they include and it seems like it's caching a whole lot.

RE: Pageviews, I realized that analytics isn't counting the googlebot crawler [and possibly other crawlers] so my number is a lot lower than it should be but I agree, I didn't think my numbers were so high as to be problematic.

RE: Upgrading, I went ahead and boosted to the Linode 768. Figured made more sense to just do it now rather than have everything tuned for 512 and get some more RAM afterwards.

Here is the output of those commands:

free -m

root@agjco:~# free -m
             total       used       free     shared    buffers     cached
Mem:           750        433        316          0         21        289
-/+ buffers/cache:        121        628
Swap:          255         11        244

ps aux

root@agjco:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2728   756 ?        Ss   06:35   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S    06:35   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    06:35   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/0:0]
root         5  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    06:35   0:00 [migration/0]
root         7  0.0  0.0      0     0 ?        S    06:35   0:00 [migration/1]
root         8  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/1:0]
root         9  0.0  0.0      0     0 ?        S    06:35   0:00 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    06:35   0:00 [migration/2]
root        11  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/2:0]
root        12  0.0  0.0      0     0 ?        S    06:35   0:00 [ksoftirqd/2]
root        13  0.0  0.0      0     0 ?        S    06:35   0:00 [migration/3]
root        14  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/3:0]
root        15  0.0  0.0      0     0 ?        S    06:35   0:00 [ksoftirqd/3]
root        16  0.0  0.0      0     0 ?        S<   06:35   0:00 [khelper]
root        17  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/u:1]
root        21  0.0  0.0      0     0 ?        S    06:35   0:00 [xenwatch]
root        22  0.0  0.0      0     0 ?        S    06:35   0:00 [xenbus]
root       148  0.0  0.0      0     0 ?        S    06:35   0:00 [sync_supers]
root       150  0.0  0.0      0     0 ?        S    06:35   0:00 [bdi-default]
root       152  0.0  0.0      0     0 ?        S<   06:35   0:00 [kblockd]
root       162  0.0  0.0      0     0 ?        S<   06:35   0:00 [md]
root       246  0.0  0.0      0     0 ?        S<   06:35   0:00 [rpciod]
root       248  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/0:1]
root       279  0.0  0.0      0     0 ?        S    06:35   0:00 [kswapd0]
root       280  0.0  0.0      0     0 ?        SN   06:35   0:00 [ksmd]
root       281  0.0  0.0      0     0 ?        S    06:35   0:00 [fsnotify_mark]
root       285  0.0  0.0      0     0 ?        S    06:35   0:00 [ecryptfs-kthr]
root       287  0.0  0.0      0     0 ?        S<   06:35   0:00 [nfsiod]
root       290  0.0  0.0      0     0 ?        S    06:35   0:00 [jfsIO]
root       291  0.0  0.0      0     0 ?        S    06:35   0:00 [jfsCommit]
root       292  0.0  0.0      0     0 ?        S    06:35   0:00 [jfsCommit]
root       293  0.0  0.0      0     0 ?        S    06:35   0:00 [jfsCommit]
root       294  0.0  0.0      0     0 ?        S    06:35   0:00 [jfsCommit]
root       295  0.0  0.0      0     0 ?        S    06:35   0:00 [jfsSync]
root       296  0.0  0.0      0     0 ?        S<   06:35   0:00 [xfs_mru_cache]
root       297  0.0  0.0      0     0 ?        S<   06:35   0:00 [xfslogd]
root       298  0.0  0.0      0     0 ?        S<   06:35   0:00 [xfsdatad]
root       299  0.0  0.0      0     0 ?        S<   06:35   0:00 [xfsconvertd]
root       300  0.0  0.0      0     0 ?        S<   06:35   0:00 [glock_workque]
root       301  0.0  0.0      0     0 ?        S<   06:35   0:00 [delete_workqu]
root       302  0.0  0.0      0     0 ?        S<   06:35   0:00 [gfs_recovery]
root       303  0.0  0.0      0     0 ?        S<   06:35   0:00 [crypto]
root       865  0.0  0.0      0     0 ?        S    06:35   0:00 [khvcd]
root       979  0.0  0.0      0     0 ?        S<   06:35   0:00 [kpsmoused]
root       980  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/1:1]
root       990  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/2:1]
root      1008  0.0  0.0      0     0 ?        S    06:35   0:00 [kjournald]
root      1012  0.0  0.0      0     0 ?        S    06:35   0:00 [kworker/3:1]
root      1033  0.0  0.0   2368   136 ?        S    06:35   0:00 upstart-udev-br
root      1035  0.0  0.0   2368   260 ?        S<s  06:35   0:00 udevd --daemon
root      1202  0.0  0.0   2364     0 ?        S<   06:35   0:00 udevd --daemon
root      1204  0.0  0.0   2364     0 ?        S<   06:35   0:00 udevd --daemon
syslog    1927  0.0  0.0  28452   444 ?        Sl   06:35   0:00 rsyslogd -c4
root      1945  0.0  0.0   2428   568 ?        Ss   06:35   0:00 cron
mysql     1957 10.2  2.2  40852 17224 ?        Ssl  06:35   6:08 /usr/sbin/mysql
nobody    1969  0.1  3.3  70592 25436 ?        Sl   06:35   0:03 /usr/bin/memcac
root      2010  0.0  0.0   2288    12 ?        Ss   06:35   0:00 dhclient3 -e IF
root      2063  0.0  0.6  67776  5232 ?        Ss   06:35   0:01 /usr/sbin/apach
root      2078  0.0  0.0   1840   404 hvc0     Ss+  06:35   0:00 /sbin/getty -8
root      2080  0.0  0.0   3600   616 ?        Ss   06:35   0:00 /usr/sbin/ntpd
ntpd      2081  0.0  0.0   3496   532 ?        S    06:35   0:00 /usr/sbin/ntpd
root      2089  0.0  0.1   5600   800 ?        Ss   06:35   0:00 /usr/sbin/sshd
root      2124  0.2  0.0      0     0 ?        D    06:35   0:09 [flush-202:0]
root      4184  0.0  0.3   8408  2460 ?        Rs   07:29   0:00 sshd: root@pts/
root      4242  0.0  0.2   4664  1696 pts/0    Ss   07:29   0:00 -bash
www-data  4302  0.4  1.6  68892 12976 ?        S    07:29   0:01 /usr/sbin/apach
www-data  4421  0.8  1.9  70276 15120 ?        S    07:30   0:02 /usr/sbin/apach
www-data  4457  0.6  1.6  69120 12932 ?        S    07:32   0:00 /usr/sbin/apach
www-data  4466  0.7  1.7  69328 13136 ?        S    07:32   0:00 /usr/sbin/apach
www-data  4475  0.7  1.7  69724 13308 ?        S    07:33   0:00 /usr/sbin/apach
www-data  4483  1.3  1.8  69748 13844 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4484  0.8  1.6  69344 12928 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4485  1.1  1.2  69752  9912 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4486  0.1  1.1  69012  8460 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4492  0.0  0.6  68384  5192 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4493  0.0  0.6  68120  4876 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4494  0.0  0.6  68120  4876 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4495  1.5  1.3  69856 10084 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4496  0.5  1.1  68976  8884 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4497  0.9  1.5  69344 11820 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4498  0.7  1.2  69776  9732 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4499  2.0  1.1  69248  9184 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4500  0.7  1.1  69328  9172 ?        S    07:34   0:00 /usr/sbin/apach
www-data  4501  0.2  0.9  68584  7016 ?        S    07:34   0:00 /usr/sbin/apach
root      4502  1.0  0.1   2764  1052 pts/0    R+   07:34   0:00 ps aux

vmstat 1 30

root@agjco:~# vmstat 1 30
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0  11448 336148  23016 300460    2    3    46   518   82   28  2  1 93  4
 0  0  11440 335900  23024 300532    0    0    24   120  452  223  2  0 97  1
 0  0  11440 335432  23032 300900    0    0   356     0  997  375  3  0 93  4
 0  0  11432 335420  23032 300876    0    0     0     0  303  148  1  0 99  0
 0  0  11432 335276  23036 301064    0    0   208     0  489  193  1  0 96  3
 0  0  11432 334880  23036 301220    0    0   188     0  358  115  0  0 99  1
 0  0  11432 334500  23036 301260    0    0     0     0   71   23  0  0 100  0
 0  0  11432 334500  23048 301260    8    0     8   100  322  154  1  0 96  3
 0  1  11428 332020  23048 301260    4    0    96     0  699  411  0  0 96  4
 0  0  11428 331400  23048 301468    0    0   116     0  315   86  2  0 96  2
 0  0  11428 331408  23048 301468    0    0     0     0   41   19  0  0 100  0
 1  0  11428 330644  23048 301616    0    0   148     0  500  234  1  0 98  1
 0  0  11428 329412  23060 301752    8    0   176   104  694  259  2  0 96  2
 0  0  11428 332232  23060 301788    0    0     0     0  316  101  1  0 99  0
 0  1  11424 330536  23072 301840    0    0    68     0 1129  465  2  0 94  3
 0  0  11424 330528  23076 301844    0    0     4     0  123   34  0  0 99  0
 0  0  11424 330504  23172 301896    0    0   196     0 1077  547  1  0 93  5
 0  0  11416 333432  23184 302024    0    0    76   132  807  374  1  0 97  1
 0  0  11416 333456  23184 302020    0    0     0     0   73   15  0  0 100  0
 0  0  11416 333476  23184 302016    0    0     0     0   76   22  0  0 100  0
 0  0  11416 333476  23184 302016    0    0     0     0   61   21  0  0 100  0
 0  0  11416 335516  23184 302016    0    0     0     0  108   24  0  0 100  0
 0  0  11416 337748  23184 302016    0    0     0     0  134   23  0  0 100  0
 0  0  11416 337196  23200 302448    0    0   404    92 1453  594  4  2 91  4
 0  0  11416 337008  23200 302632    0    0   212     0  319  143  1  0 99  0
 0  1  11416 336504  23200 302832    0    0   192     0  711  340  1  0 98  1
 0  0  11416 336340  23200 302872    0    0    28     0  656  223  3  0 96  1
 0  0  11416 336340  23200 302852    0    0     0   300  277   73  0  0 100  0
 0  0  11416 336340  23212 302848    0    0     0   140  462  173  1  0 99  0
 0  0  11416 336328  23212 302852    0    0     0     0  142   46  0  0 100  0

I still need to hunt down the indexless join but I suspect it's a smaller part of the problem. mysqltuner says there are currently 16 joins without indexes. Since I cache almost all of my MySQL results I wouldn't expect this to be a query that is run very often. That being said I'm going to be spending some time later today fixing it.

_________________
Andrew G Johnson

http://www.andrewgjohnson.com

To help with mysql debugging add this into your my.cnf

[mysqld]
slow_query_log = 1
slow_query_log_file     = /var/log/mysql/mysql-slow.log
long_query_time = 1
log-queries-not-using-indexes
log-output=TABLE

It'll create a table called slow_log in the mysql database which logs queries that take longer than 1 second or don't use indexes, you can then use your favourite mysql client to search for the queries in that table.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Hmm something seems to be up with those last two lines when I add them [either together or either by itself] MySQL breaks. What I mean is I go in with nano to edit my.cnf, save it, then run "service mysql restart" which usually takes maximum 5 seconds. With those lines it hangs for upto a minute or two then claims its back but all my sites can't connect and I can't connect using my MySQL editor locally either.

Did you maybe make a typo? Will I still get some kind of output with the other lines?

_________________
Andrew G Johnson

http://www.andrewgjohnson.com

That's copied from my stackscript so there's no typo, what does your mysql error log say /var/log/mysql/error.log

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Nothing jumps out at me but here's 8am onward:

120116  8:00:39 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:00:41 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:00:41 [Note] Plugin 'FEDERATED' is disabled.
120116  8:00:41 [Note] Plugin 'InnoDB' is disabled.
120116  8:00:41 [Note] Event Scheduler: Loaded 0 events
120116  8:00:41 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:01:19 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:01:19 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:01:19 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:01:19 [Note] Plugin 'FEDERATED' is disabled.
120116  8:01:19 [Note] Plugin 'InnoDB' is disabled.
120116  8:01:19 [Note] Event Scheduler: Loaded 0 events
120116  8:01:19 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:02:31 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:02:31 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:02:33 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:02:33 [Note] Plugin 'FEDERATED' is disabled.
120116  8:02:33 [Note] Plugin 'InnoDB' is disabled.
120116  8:02:33 [Note] Event Scheduler: Loaded 0 events
120116  8:02:33 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:04:52 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:04:52 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:04:52 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:04:52 [Note] Plugin 'FEDERATED' is disabled.
120116  8:04:52 [Note] Plugin 'InnoDB' is disabled.
120116  8:04:52 [Note] Event Scheduler: Loaded 0 events
120116  8:04:52 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:05:32 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:05:32 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:05:32 [ERROR] /usr/sbin/mysqld: Sort aborted
120116  8:05:35 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:05:36 [Note] Plugin 'FEDERATED' is disabled.
120116  8:05:36 [Note] Plugin 'InnoDB' is disabled.
120116  8:05:36 [Note] Event Scheduler: Loaded 0 events
120116  8:05:36 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:06:10 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:06:10 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:06:10 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:06:10 [Note] Plugin 'FEDERATED' is disabled.
120116  8:06:10 [Note] Plugin 'InnoDB' is disabled.
120116  8:06:10 [Note] Event Scheduler: Loaded 0 events
120116  8:06:10 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:06:45 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:06:45 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:06:45 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:06:45 [Note] Plugin 'FEDERATED' is disabled.
120116  8:06:45 [Note] Plugin 'InnoDB' is disabled.
120116  8:06:45 [Note] Event Scheduler: Loaded 0 events
120116  8:06:45 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:09:32 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:09:32 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:09:32 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:12:42 [Note] Plugin 'FEDERATED' is disabled.
120116  8:12:42 [Note] Plugin 'InnoDB' is disabled.
120116  8:12:42 [Note] Event Scheduler: Loaded 0 events
120116  8:12:42 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:13:28 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:13:28 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:13:28 [ERROR] /usr/sbin/mysqld: Sort aborted
120116  8:13:31 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:13:31 [Note] Plugin 'FEDERATED' is disabled.
120116  8:13:31 [Note] Plugin 'InnoDB' is disabled.
120116  8:13:31 [Note] Event Scheduler: Loaded 0 events
120116  8:13:31 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:14:18 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:14:18 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:14:18 [Note] /usr/sbin/mysqld: Shutdown complete

120116  8:20:55 [Note] Plugin 'FEDERATED' is disabled.
120116  8:20:55 [Note] Plugin 'InnoDB' is disabled.
120116  8:20:55 [Note] Event Scheduler: Loaded 0 events
120116  8:20:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120116  8:21:22 [Note] /usr/sbin/mysqld: Normal shutdown

120116  8:21:22 [Note] Event Scheduler: Purging the queue. 0 events
120116  8:21:22 [Note] /usr/sbin/mysqld: Shutdown complete
120116  8:22:26 [Note] Plugin 'FEDERATED' is disabled.
120116  8:22:26 [Note] Plugin 'InnoDB' is disabled.
120116  8:22:27 [Note] Event Scheduler: Loaded 0 events
120116  8:22:27 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.41-3ubuntu12.10-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

_________________
Andrew G Johnson

http://www.andrewgjohnson.com

Try stopping mysql and running

/usr/sbin/mysqld

from a root shell and see if it outputs any more information

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue