I need help troubleshooting this.... Maybe you can see a clue that I missed? Here are the facts, as I know them. Any help interpreting this would be greatly appreciated!

$ sudo /sbin/service mysqld status
mysqld dead but subsys locked

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 16255928 12518040 3077288 81% /

$ ps aux | grep mysqld
502 30323 0.0 0.0 284 56 ttyp0 D+ 17:07 0:00 grep mysqld

And below follows the last 100 lines of /var/logs/mysqld.log:


110214 15:50:29 mysqld started
110214 15:50:29 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
110214 15:50:29 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: Log scan progressed past the checkpoint lsn 3 3118450946
110214 15:50:30 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 3 3118451227
110214 15:50:31 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 9
9
InnoDB: Apply batch completed
110214 15:50:31 InnoDB: Started; log sequence number 3 3118451227
110214 15:50:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

Number of processes running now: 0
110215 16:41:01 mysqld restarted
110215 16:42:05 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
110215 16:42:05 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110215 16:42:05 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110215 16:42:08 InnoDB: Started; log sequence number 3 3180794096
110215 16:42:08 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

Number of processes running now: 0
110215 16:52:48 mysqld restarted
110215 16:52:59 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
110215 16:52:59 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110215 16:53:05 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110215 16:53:15 mysqld ended

110215 17:09:32 mysqld started
110215 17:09:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
110215 17:09:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110215 17:09:33 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110215 17:09:34 InnoDB: Started; log sequence number 3 3181544202
110215 17:09:34 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

If you are restarting it because it is not running, then it is possible you are running out of memory and it is getting killed. Take a look at your lish console and/or your syslogs, you should find something there about OOM.

--
Travis

$ sudo /sbin/service mysqld status 
mysqld dead but subsys locked 

$ df 
Filesystem 1K-blocks Used Available Use% Mounted on 
/dev/xvda 16255928 12518040 3077288 81% / 

$ ps aux | grep mysqld 
502 30323 0.0 0.0 284 56 ttyp0 D+ 17:07 0:00 grep mysqld 

It doesn't appear to be OOM. Any other ideas?

Your ps shows it's not running, which would be the case if it OOM'd...

I misread my own ps ... thought the grep process was the mysqld process.

But in the log file I see no mention of oom (or am I overlooking that too ..)

Would it appear there?

OOM will appear in /var/log/syslog

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue