Hey all,
I setup a 768 linode and host 1 site on it. It's a fairly typical site - blog-type content. I'm using MySQL, nginx, php-fpm and memcached.

So, I had originally set it up without memcached and right around 1500 visitors an hour, it started using a ton of disk IO.
- The site calls about 5 sql queries for each page render.
- I have nginx setup with 4 workers and 1024 worker connections
- php is config'd to use 6mb of ram per thread
- ulimit -n is set to 1024
- The VPS has 768mb of ram

To bring the IO usage down, I installed memcached and started setting up the queries to use it and cache queries for 10 minutes. Seemed to work great. Fast load times still and the IO usage went way down... until about 10 hours later when I was getting nginx's gateway timeout's. Checking the logs and free -m told me that it was out of memory. I restarted memcached and it's all fine and dandy again. Even with memcached, it locks up around the 1500/user/hour threshold.

These are my settings for memcached:
MAXCONN="1024"
CACHESIZE="64"

Here's the seemingly important bits of my /var/log/messages right before it locks up:
Mar 15 01:42:00 host kernel: HighMem free:208kB min:128kB low:208kB high:288kB active_anon:25520kB inactive_anon:25632kB active_file:480kB inact$
Mar 15 01:42:00 host kernel: lowmem_reserve[]: 0 0 0 0
Mar 15 01:42:00 host kernel: DMA: 98*4kB 19*8kB 3*16kB 1*32kB 1*64kB 1*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2864kB
Mar 15 01:42:00 host kernel: Normal: 1358*4kB 2*8kB 1*16kB 1*32kB 0*64kB 0*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7032kB
Mar 15 01:42:00 host kernel: HighMem: 26*4kB 5*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 208kB
Mar 15 01:42:00 host kernel: 1592 total pagecache pages
Mar 15 01:42:00 host kernel: 126 pages in swap cache
Mar 15 01:42:00 host kernel: Swap cache stats: add 4286470, delete 4286344, find 616656/770806
Mar 15 01:42:00 host kernel: Free swap = 0kB
Mar 15 01:42:00 host kernel: Total swap = 1048572kB
*snip*
Out of memory: Kill process ###

Is there any way to detect if it's about to run out of memory and flush memcached or restart it? Or am I going about this in the wrong way somehow?

Try 'ps axl' to confirm that the problem is memcached eating all the ram.

First: You probably don't want your swap to be 1 GB. Try lowering it to 256 MB or 128 MB, so that your system can take action against memory hogs before everything grinds to a halt.

Second: It's probably not memcached. You're probably swap-thrashing, which is probably due to either PHP running amok or bad MySQL settings. Try running mysqltuner.pl and posting what it reports. Also, your pool config for php-fpm might prove helpful.

_________________

/* TODO: need to add signature to posts */

So, it seemingly turned out to be php-fpm and some gratuitous default settings. I lowered the max_children to 30 (from 50) and lowered start_servers and max_servers to 5, with max_spare_servers set to 8, with max_requests set to 50. I also raised the OS' ulimit to around 4000. Seems to be solid at the moment with similar traffic. I also did a test from loadimpact.com and pushed 2000 simultaneous "virtual" users at it. It hovered around 10% CPU and still had about 400Mb of RAM free during that time. Thanks folks

max_requests of 50 is really low. Theoretically you shouldn't need it. Setting it to something in the 1000s would be better.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue