I don't even understand how things went wrong, but here's what's happening.

I'm running linode 1gb, ubuntu+apache+mysql+php5, all installed and optimized per linode instructions.
Everything was working great till today, once I started to get these emails "notification threshold (1000) for disk io rate by averaging 5351.16".
Today I transferred my main, biggest site here, and 2 hours later, troubles started.

All sites are wordpress, and it can't connect to db (all sites), then mysql crashes, everything becomes slow (even terminal, it takes 5min to login via ssh and switch to root).

When I run service mysql status, I get this: post-start process 16000. I tried optimizing db's with: myisamchk --recover /var/lib/mysql/*/*.MYI, didn't help.
What could be the issue? I can't even start iotop now, takes forever...

Here's the screenshot once it's all slow:


and here's when it works (guess mysql is dead here):


Can someone help, I can't catch a breath today...

What do your error logs say? What is your RAM usage when this is happening?

The Swap column looks bad while you're having the problems. When you log in via SSH or LISH do you get Out of Memory (OOM) errors?

Currently it's stuck, it might take 20min to get all logs, for now - here's ram stat:

You're out of RAM and Swap Space.

While this is happening run "top". While top is running press shift+f (for a capital F) and you'll get a sort menu. Chose 'n' to sort by memory. When you're done press 'q' to quit.

Apache error.log:

[Sun Oct 27 06:31:19 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 04:40:29 2013] [notice] Graceful restart requested, doing restart
apache2: Could not reliably determine the server's fully qualified domain name, using 23.92.17.126 for ServerName
[Mon Oct 28 04:40:29 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 06:01:22 2013] [notice] Graceful restart requested, doing restart
apache2: Could not reliably determine the server's fully qualified domain name, using 23.92.17.126 for ServerName
[Mon Oct 28 06:01:22 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 06:01:56 2013] [notice] caught SIGTERM, shutting down
[Mon Oct 28 06:01:58 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 06:27:05 2013] [notice] caught SIGTERM, shutting down
[Mon Oct 28 06:27:06 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 08:36:46 2013] [notice] Graceful restart requested, doing restart
apache2: Could not reliably determine the server's fully qualified domain name, using 23.92.17.126 for ServerName
[Mon Oct 28 08:36:46 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 08:45:46 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 28 09:56:08 2013] [notice] caught SIGTERM, shutting down
[Mon Oct 28 10:01:32 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 09:58:16 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 28 12:01:01 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8 with Suhosin-Patch configured -- $
[Mon Oct 28 11:57:44 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting

Mysql:

131028 14:32:44 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecate$
131028 14:32:44 [Note] Plugin 'FEDERATED' is disabled.
131028 14:32:44 InnoDB: The InnoDB memory heap is disabled
131028 14:32:44 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131028 14:32:44 InnoDB: Compressed tables use zlib 1.2.3.4
131028 14:32:44 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
131028 14:32:44 InnoDB: Completed initialization of buffer pool
131028 14:32:44 InnoDB: Fatal error: cannot allocate memory for the buffer pool
131028 14:32:44 [ERROR] Plugin 'InnoDB' init function returned error.
131028 14:32:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131028 14:32:44 [ERROR] Unknown/unsupported storage engine: InnoDB
131028 14:32:44 [ERROR] Aborting

131028 14:32:44 [Note] /usr/sbin/mysqld: Shutdown complete

Any other log I should paste?

If these are production services (i.e. downtime is bad), then the quick fix is to get at bigger Linode. Then sort out your optimization.

_________________
Either provide enough details for people to help, or sit back and listen to the crickets chirp.
Security thru obscurity is a myth - and really really annoying.

Right now ssh is working fast (mysql down):



I don't think I need bigger linode, there's around 2k visits all together daily.

vonskippy is right. If you need these sites up and running quickly you should upsize your Linode, even if only for a week to get these optimizations worked out.

You may have too many of something (Apache workers, MySQL InnoDB memory pool, etc). There are several recent threads regarding Apache related resource usage. Perhaps the Wordpress forums have threads on common optimizations to run Wordpress on a 1GB VPS.

Maybe you're right, but once site was up - everything was running smoothly. I don't believe I got massive traffic all of the sudden.

You may not need a larger Linode in the long run with only 2k v/p/d. But until you figure out how to get this to run properly in less than 1GB of RAM upsizing may be a good solution.

I counted at least 22 apache processes and at least 17 kworker processes. You're also requesting 128.0M for the InnoDB buffer pool. Reducing the Apache & kworker processes, as well as the size of the InnoDB pool (unless you have a lot of DB content), will definately work to your advantage. If your Wordpress DB isn't using the InnoDB driver - which is a possibility - than you certainly don't want to allocate RAM for InnoDB.

Thanks for helping, I'll see what I will do.

pex23,
I forgot about this potential RAM issue with Wordpress. If you're running Wordpress it could be the 'Max RAM' setting isn't defined so it tries to take 256M. Look at this thread for more info.

which apache mpm are you using? If it's prefork, you probably have too high a MaxClients

You could try switching to an nginx/Apache reverse proxy setup. It makes quite a difference when it comes to Apache memory usage and hosting WordPress sites. I just posted something about it here:

viewtopic.php?f=12&t=10568&p=61207#p61207