Linode Forum

mottalrd · Joined: 23 Dec 2011 Posts: 16

Hi everybody,
I am running on a Linode 768 with apache2, mysql and a wordpress with 20.000 pageviews/day

The number of MaxClients allowed by Apache is 20. If I move this number to 25 the server starts swapping.

My problem is that apache easily reaches the maximum number of clients, the TCP queue grows and the site starts behaving slowly...however increasing MaxClients increase the swap (and I understood the the server should never swap...)

can I do better? Or I just reached the limit?

Code:
[TIME: 06:19:04] Output: 6 requests currently being processed, 12 idle workers
[TIME: 06:24:04] Output: 20 requests currently being processed, 0 idle workers
[TIME: 06:29:06] Output: 12 requests currently being processed, 8 idle workers
[TIME: 06:34:06] Output: 20 requests currently being processed, 0 idle workers
[TIME: 06:39:17] Output: 8 requests currently being processed, 10 idle workers
[TIME: 06:44:17] Output: 20 requests currently being processed, 0 idle workers
[TIME: 06:49:23] Output: 20 requests currently being processed, 0 idle workers
[TIME: 06:54:25] Output: 20 requests currently being processed, 0 idle workers
[TIME: 06:59:33] Output: 20 requests currently being processed, 0 idle workers
[TIME: 07:04:35] Output: 20 requests currently being processed, 0 idle workers
[TIME: 07:09:39] Output: 20 requests currently being processed, 0 idle workers
[TIME: 07:14:51] Output: 20 requests currently being processed, 0 idle workers
[TIME: 07:19:52] Output: 20 requests currently being processed, 0 idle workers
[TIME: 07:25:08] Output: 20 requests currently being processed, 0 idle workers
[TIME: 07:30:14] Output: 17 requests currently being processed, 3 idle workers

Code:
total used free shared buffers cached
Mem: 750 716 34 0 20 323
-/+ buffers/cache: 371 378
Swap: 255 15 240

Code:
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
ServerLimit 20
MaxRequestsPerChild 0
</IfModule>

Code: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2736 764 ? Ss 01:18 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 01:18 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 01:18 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S 01:18 0:01 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/u:0]
root 6 0.0 0.0 0 0 ? S 01:18 0:00 [migration/0]
root 7 0.0 0.0 0 0 ? S 01:18 0:00 [migration/1]
root 8 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/1:0]
root 9 0.0 0.0 0 0 ? S 01:18 0:00 [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S 01:18 0:00 [migration/2]
root 11 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/2:0]
root 12 0.0 0.0 0 0 ? S 01:18 0:00 [ksoftirqd/2]
root 13 0.0 0.0 0 0 ? S 01:18 0:00 [migration/3]
root 14 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/3:0]
root 15 0.0 0.0 0 0 ? S 01:18 0:00 [ksoftirqd/3]
root 16 0.0 0.0 0 0 ? S< 01:18 0:00 [khelper]
root 17 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/u:1]
root 21 0.0 0.0 0 0 ? S 01:18 0:00 [xenwatch]
root 22 0.0 0.0 0 0 ? S 01:18 0:00 [xenbus]
root 148 0.0 0.0 0 0 ? S 01:18 0:00 [sync_supers]
root 150 0.0 0.0 0 0 ? S 01:18 0:00 [bdi-default]
root 152 0.0 0.0 0 0 ? S< 01:18 0:00 [kblockd]
root 162 0.0 0.0 0 0 ? S< 01:18 0:00 [md]
root 246 0.0 0.0 0 0 ? S< 01:18 0:00 [rpciod]
root 248 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/0:1]
root 279 0.0 0.0 0 0 ? S 01:18 0:06 [kswapd0]
root 280 0.0 0.0 0 0 ? SN 01:18 0:00 [ksmd]
root 281 0.0 0.0 0 0 ? S 01:18 0:00 [fsnotify_mark]
root 285 0.0 0.0 0 0 ? S 01:18 0:00 [ecryptfs-kthrea]
root 287 0.0 0.0 0 0 ? S< 01:18 0:00 [nfsiod]
root 290 0.0 0.0 0 0 ? S 01:18 0:00 [jfsIO]
root 291 0.0 0.0 0 0 ? S 01:18 0:00 [jfsCommit]
root 292 0.0 0.0 0 0 ? S 01:18 0:00 [jfsCommit]
root 293 0.0 0.0 0 0 ? S 01:18 0:00 [jfsCommit]
root 294 0.0 0.0 0 0 ? S 01:18 0:00 [jfsCommit]
root 295 0.0 0.0 0 0 ? S 01:18 0:00 [jfsSync]
root 296 0.0 0.0 0 0 ? S< 01:18 0:00 [xfs_mru_cache]
root 297 0.0 0.0 0 0 ? S< 01:18 0:00 [xfslogd]
root 298 0.0 0.0 0 0 ? S< 01:18 0:00 [xfsdatad]
root 299 0.0 0.0 0 0 ? S< 01:18 0:00 [xfsconvertd]
root 300 0.0 0.0 0 0 ? S< 01:18 0:00 [glock_workqueue]
root 301 0.0 0.0 0 0 ? S< 01:18 0:00 [delete_workqueu]
root 302 0.0 0.0 0 0 ? S< 01:18 0:00 [gfs_recovery]
root 303 0.0 0.0 0 0 ? S< 01:18 0:00 [crypto]
root 865 0.0 0.0 0 0 ? S 01:18 0:00 [khvcd]
root 979 0.0 0.0 0 0 ? S< 01:18 0:00 [kpsmoused]
root 980 0.0 0.0 0 0 ? S 01:18 0:01 [kworker/1:1]
root 1005 0.0 0.0 0 0 ? S 01:18 0:00 [kworker/2:1]
root 1008 0.0 0.0 0 0 ? S 01:18 0:00 [kjournald]
root 1012 0.0 0.0 0 0 ? S 01:18 0:01 [kworker/3:1]
root 1033 0.0 0.0 2368 0 ? S 01:18 0:00 upstart-udev-bridge --daemon
root 1035 0.0 0.0 2236 4 ? S<s 01:18 0:00 udevd --daemon
root 1333 0.0 0.0 2232 4 ? S< 01:18 0:00 udevd --daemon
root 1420 0.0 0.0 2360 4 ? S< 01:18 0:00 udevd --daemon
syslog 1911 0.0 0.0 28452 492 ? Sl 01:18 0:00 rsyslogd -c4
root 1941 0.0 0.0 2428 212 ? Ss 01:18 0:00 cron
mysql 1952 0.5 4.6 135824 35856 ? Ssl 01:18 2:00 /usr/sbin/mysqld
root 1995 0.0 0.0 2288 4 ? Ss 01:18 0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0
root 2049 0.0 0.0 3600 276 ? Ss 01:18 0:00 /usr/sbin/ntpd
ntpd 2050 0.0 0.0 3496 332 ? S 01:18 0:00 /usr/sbin/ntpd
root 2060 0.0 0.0 5600 308 ? Ss 01:18 0:00 /usr/sbin/sshd -D
root 2077 0.0 0.0 4088 260 ? Ss 01:18 0:00 /usr/sbin/vsftpd
root 2129 0.0 0.0 1840 60 hvc0 Ss+ 01:18 0:00 /sbin/getty -8 38400 hvc0
root 2138 0.0 0.0 0 0 ? S 01:18 0:00 [flush-202:0]
root 6051 0.0 0.3 8408 2760 ? Ss 05:27 0:00 sshd: finalsayan [priv]
1000 6109 0.0 0.1 8540 1392 ? S 05:27 0:00 sshd: finalsayan@pts/0
1000 6110 0.0 0.2 4752 2048 pts/0 Ss 05:27 0:00 -bash
1000 6303 0.0 0.0 1884 512 pts/0 S 05:33 0:00 /bin/sh ./apache_status_monitor.sh
root 6833 0.0 0.6 35808 5008 ? Ss 06:12 0:00 /usr/sbin/apache2 -k start
root 6845 0.0 0.1 5084 1384 ? Sl 06:12 0:00 /usr/lib/phusion_passenger/ApplicationPoolServerExecutable 0 /usr/lib/phusion_passenger/passenger-spawn-server /usr/bin/ruby /tmp/passenger.6833
root 6846 0.3 0.5 14428 4308 ? Sl 06:12 0:14 Passenger spawn server
www-data 7335 3.8 2.5 49972 19416 ? S 06:47 1:33 /usr/sbin/apache2 -k start
www-data 7344 2.7 2.5 49972 19372 ? S 06:49 1:04 /usr/sbin/apache2 -k start
www-data 7416 3.2 2.7 50944 21348 ? S 07:02 0:50 /usr/sbin/apache2 -k start
www-data 7438 3.2 2.7 50724 21100 ? S 07:05 0:44 /usr/sbin/apache2 -k start
www-data 7439 2.1 2.7 52912 21488 ? R 07:05 0:29 /usr/sbin/apache2 -k start
www-data 7464 3.5 2.4 48508 18900 ? S 07:10 0:38 /usr/sbin/apache2 -k start
www-data 7465 3.3 2.5 49972 19256 ? S 07:10 0:35 /usr/sbin/apache2 -k start
www-data 7471 2.8 2.7 51636 20948 ? S 07:11 0:27 /usr/sbin/apache2 -k start
www-data 7495 3.4 2.4 48120 18460 ? S 07:14 0:28 /usr/sbin/apache2 -k start
www-data 7496 3.6 2.6 50864 20588 ? R 07:14 0:30 /usr/sbin/apache2 -k start
www-data 7497 2.9 2.4 48508 18960 ? S 07:14 0:24 /usr/sbin/apache2 -k start
www-data 7522 4.9 2.7 50952 21248 ? R 07:15 0:37 /usr/sbin/apache2 -k start
www-data 7526 3.8 2.6 49916 20204 ? S 07:15 0:29 /usr/sbin/apache2 -k start
www-data 7527 2.1 2.8 51496 21564 ? R 07:15 0:16 /usr/sbin/apache2 -k start
www-data 7535 3.5 2.7 50988 21340 ? S 07:15 0:26 /usr/sbin/apache2 -k start
www-data 7536 4.6 2.5 49980 19324 ? S 07:15 0:34 /usr/sbin/apache2 -k start
www-data 7552 3.4 2.6 50176 20512 ? S 07:18 0:19 /usr/sbin/apache2 -k start
www-data 7553 4.7 2.4 49564 18884 ? S 07:18 0:27 /usr/sbin/apache2 -k start
www-data 7554 2.9 2.7 51304 21120 ? R 07:18 0:17 /usr/sbin/apache2 -k start
www-data 7555 5.0 2.4 48508 18928 ? S 07:18 0:29 /usr/sbin/apache2 -k start
1000 7678 0.0 0.0 3288 468 pts/0 S 07:25 0:00 sleep 300
1000 7758 0.0 0.1 2764 1052 pts/0 R+ 07:28 0:00 ps aux

Guspaz · Joined: 26 May 2009 Posts: 1150 Location: Montreal, QC

You're actually only using 371 MB of RAM, add another 15 for what got swapped out, and you're at 386 MB used. So you've got some room to increase it.

There are more efficient ways to run Apache. If you don't want to migrate to lighttpd or nginx, you can try setting up Apache to run mpm_worker instead, as it's much more efficient (no longer need to load PHP into memory to serve static requests). It can be fairly involved, though, because it also means reconfiguring PHP to use FPM or fastcgi or something similar rather than mod_php.

hybinet · Joined: 02 May 2008 Posts: 1058

As @Guspaz said, using a bit of swap is OK. Thrashing swap is bad. Unless you're using an image gallery plugin, it should be safe to increase MaxClients a little bit.

Also,
- What's your KeepAlive setting? It should be very low, like 2-5.
- Are you using a caching plugin with WordPress?

caker · Joined: 15 Apr 2003 Posts: 2907 Location: Galloway, NJ

MaxClients of 20 and KeepAlives OFF can serve an insane amount of traffic. Turn off KeepAlives and see how it goes.

-Chris

hoopycat · Posted: Fri Jan 06, 2012 1:51 pm Post subject:

Also, give mysqltuner.pl a spin to optimize your MySQL memory usage.

My usual MySQL advice is to move as much as possible (i.e. everything but the mysql.* database) to InnoDB and crank the buffer pool; I find it's easier to tune for InnoDB, and not having to deal with writes locking entire tables can make some requests finish faster. This probably won't help your immediate problems, though.

(I'm a bit of a database snob. And yes, Wordpress is happy as a clam with InnoDB.)

hoopycat · Posted: Fri Jan 06, 2012 2:06 pm Post subject:

caker wrote: MaxClients of 20 and KeepAlives OFF can serve an insane amount of traffic. Turn off KeepAlives and see how it goes.
+1. Keepalives are right up there with MaxClients on the list of tragic mpm_prefork defaults. KeepAliveTimeout 1 gives me this for a somewhat busy vBulletin 3 forum (Alexa top 100,000, baby!), a handful of Wordpresses, and the rest of our PHP flotilla:

It's a 2 GB instance, so we've got mod_prefork at MaxClients 75, but we could crank that down quite a ways and not feel a thing. Perhaps the most telling thing is that we have 1.02 GB of completely unused (no buffers, no cache, no nothin') RAM on there. I should try setting KeepAliveTimeout back to the default sometime just to see what'd happen... :-)

(Also, we're 64 MB into swap. Linux memory management is fun.) -rt

mottalrd · Joined: 23 Dec 2011 Posts: 16

hybinet wrote: As @Guspaz said, using a bit of swap is OK. Thrashing swap is bad. Unless you're using an image gallery plugin, it should be safe to increase MaxClients a little bit.

Thank you Guspaz and hybinet. Yes, I also have a small forum and gallery3 installed. How could I distinguish between system "normal" swap and trashing? I have seen my swap going up when I increased the MaxClients directive but I cannot know what is the reason (is it because of wordpress? is it because of the gallery3 subsection?)

hybinet wrote: - What's your KeepAlive setting? It should be very low, like 2-5.

It was On with the default Timeout setting. I turned it Off following your suggestions, tomorrow I'll let you know :)

hybinet wrote: - Are you using a caching plugin with WordPress?
Sure, W3 Total Cache

hoopycat wrote: My usual MySQL advice is to move as much as possible (i.e. everything but the mysql.* database) to InnoDB and crank the buffer pool; I find it's easier to tune for InnoDB, and not having to deal with writes locking entire tables can make some requests finish faster. This probably won't help your immediate problems, though.
When I have to evaluate this solution? Right now it seems to me that mysql is not taking too much memory. Together with wordpress I have a small forum and gallery3

hybinet · Joined: 02 May 2008 Posts: 1058

mottalrd wrote: How could I distinguish between system "normal" swap and trashing?
In normal situations, the kernel moves unused data to swap, and just leaves it sitting there until it is needed. So you're using swap but not much data is being transferred in and out of swap. On the other hand, when you're thrashing, you transfer a lot of data repeatedly in and out of swap. So the amount of used swap is less important than the amount of data that is being transferred in and out of swap at any given time.

Run "vmstat 1" (without quores), let it print a few lines, and hit Ctrl+C to halt. Check the "si" and "so" columns. They represent the number of blocks per second that are swapped in and out, respectively. Lower is better, but you're probably OK as long as they're not in the thousands.

Alternatively, you could just check the I/O graph on your dashboard.

mottalrd · Joined: 23 Dec 2011 Posts: 16

Quote: Run "vmstat 1" (without quores), let it print a few lines, and hit Ctrl+C to halt. Check the "si" and "so" columns. They represent the number of blocks per second that are swapped in and out, respectively. Lower is better, but you're probably OK as long as they're not in the thousands.

Amazing. Thank you

Quote: On the other hand, when you're thrashing, you transfer a lot of data repeatedly in and out of swap. So the amount of used swap is less important than the amount of data that is being transferred in and out of swap at any given time.

This starts happening only when I am hitting the RAM limit, or this could happen also because of same insane I/O application?

mottalrd · Joined: 23 Dec 2011 Posts: 16

with keepAlive=Off I have much more idle workers, sounds good! :)

Code:
[TIME: 05:30:06] Output: 7 requests currently being processed, 9 idle workers
[TIME: 05:35:06] Output: 3 requests currently being processed, 14 idle workers
[TIME: 05:40:06] Output: 2 requests currently being processed, 9 idle workers
[TIME: 05:45:06] Output: 2 requests currently being processed, 11 idle workers
[TIME: 05:50:06] Output: 2 requests currently being processed, 9 idle workers
[TIME: 05:55:06] Output: 1 requests currently being processed, 9 idle workers
[TIME: 06:00:07] Output: 9 requests currently being processed, 6 idle workers

however the server is swapping with some peaks even if I have a lot of free memory in RAM. I suspect this may be related of some intensive I/O (maybe http downlods of site contents), is it possible?

Code:
total used free shared buffers cached
Mem: 750 650 100 0 27 401
-/+ buffers/cache: 221 529
Swap: 255 17 238

Code:
top - 06:18:43 up 1 day, 4:59, 1 user, load average: 0.43, 0.36, 0.44
Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 768512k total, 665116k used, 103396k free, 28248k buffers
Swap: 262140k total, 18148k used, 243992k free, 396048k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1952 mysql 20 0 134m 37m 2716 S 0 5.0 10:57.56 mysqld
17736 www-data 20 0 50684 20m 3620 S 0 2.7 0:02.99 apache2
17741 www-data 20 0 50200 19m 3612 S 0 2.6 0:00.59 apache2
17713 www-data 20 0 49692 19m 3748 S 0 2.6 0:10.05 apache2
17735 www-data 20 0 49688 19m 3620 S 0 2.6 0:03.48 apache2
17720 www-data 20 0 49064 18m 3704 S 0 2.5 0:02.83 apache2
17724 www-data 20 0 49972 18m 4044 S 0 2.5 0:04.57 apache2
17679 www-data 20 0 49564 18m 4200 S 0 2.5 0:13.81 apache2
17716 www-data 20 0 49552 18m 4120 S 0 2.4 0:06.37 apache2
17726 www-data 20 0 46816 16m 3748 S 0 2.2 0:05.82 apache2
17699 www-data 20 0 46808 16m 3756 S 0 2.2 0:09.06 apache2
17740 www-data 20 0 46816 16m 3568 S 0 2.2 0:01.18 apache2
11829 root 20 0 35808 4468 4304 S 0 0.6 0:09.39 apache2

hybinet · Joined: 02 May 2008 Posts: 1058

mottalrd wrote: however the server is swapping with some peaks even if I have a lot of free memory in RAM. I suspect this may be related of some intensive I/O (maybe http downlods of site contents), is it possible?
Wait until you get another peak like that, and run iotop to find out which processes are hitting the disk. (Install it if needed.) If it's MySQL, see MySQL tuning tips offered by other users above.

Can I do better with my linode?