Linode Forum

[nehalem]

Ever since switching to the Latest 3.0 kernel (3.0.4-linode38) and rebooting I notice that my server is being a bit more swappy:



As you could have guessed from the graph, the reboot happened around 6 days ago.

Can anyone say if this "aggressive" memory management is a feature of the new kernel. Does anyone have a similar issue?

free -m says:

Code:              total       used       free     shared    buffers     cached
Mem:           497        436         60          0         48        284
-/+ buffers/cache:        104        392
Swap:          255         26        229


I hadn't touched anything in my configuration when the swapping issue started happening.

Any ideas? Thanks!

[bryantrv]

I see the same thing- you can see where I rebooted from latest legacy 2.6.18.8-linode22 to latest:

No other changes made, plenty of free memory near identical to yours).

[nehalem]

Wow, I don't like the look of that at all.

Can an admin confirm if this is an issue to be concerned with? This kernel definitely seems to have some sort of issue.

In the meantime, according to my web server benchmarks, performance does not seem to have suffered at the current moment. Which is good :)

[Guspaz]

You can always just adjust the swappiness yourself, put a new value in /proc/sys/vm/swappiness

It's possible somebody changed the default, or the meaning of the values. Either way, trivial to change it yourself.

[hybinet]

nehalem wrote: This kernel definitely seems to have some sort of issue.
We're talking about an average of single-digit blocks per second. That's only a few kilobytes. No need for alarm here, though I'm curious about the change, too.

[nehalem]

Quote: We're talking about an average of single-digit blocks per second

I'd prefer an average of no blocks per second like it was for 99% of the time with the older kernel.

My last server was up for 3 months with no swapping. I'm afraid I can't be comfortable if this new kernel hammers the disk every 2 hours or so. I've seen IO usage go up into the 2000 range while it swaps and I don't like that at all. :(

[bryantrv]

FWIW, here is the Munin graph of memory for the same switch between leacy and latest :



What Munin calls inactive and active memory basically switch values.
FWIW, I do have these added lines in sysctl.conf (Debian Squeeze i386).

[hoopycat]

A lot has changed in Linux memory management over the past half-decade, for sure. Keep in mind that the 2.6.18 series is very very old and you will notice some (hopefully beneficial) behavioral differences in newer kernels.

Yes, more swapping seems to happen, apparently to minimize overall I/O. No, this shouldn't be a problem, and it shouldn't be thrashing more than a handful of blocks at a time.

I did a quick sampling of Linux machines I have access to (physical/virtual, desktop/server, across various providers, spanning 2.6.32 to 3.0.4), and all are at least a little bit into swap, irrespective of free memory. So, I'd say it's normal!

[bryantrv]

I don't worry about it actually using swap, I just wish ti wouldn't hit the disk so hard when there is plenty of ram available. If I get anywhere near actually using all the memory, it will OOM- even when there is some still unused.

[nehalem]

This is what my IO looks like for the last 24 hours:



If it stays that way tomorrow when the server is getting hammered then I'll be a very happy person.

Note: I have not made any changes to swappiness etc. I was waiting to see if it would settle down after it got used to its job of serving web pages :P

Will post back on any changes...

[mnordhoff]

bryantrv wrote: I don't worry about it actually using swap, I just wish ti wouldn't hit the disk so hard when there is plenty of ram available. If I get anywhere near actually using all the memory, it will OOM- even when there is some still unused.
? In what way is it hitting the disk "so hard"? Edit: The Munin graph you posted earlier shows it averaging less than 1 page per second, with a peak of 44. I don't know what it means by a "page", but whatever it is, it's extremely low.

[nehalem]

Quote: shows it averaging less than 1 page per second

The averages are quite misleading. By looking at the averages you don't see the peaks which are in the thousands.

That aside, I got tired of the "unnecessary" swapping and adjusted the swappiness. A value of 40 gives me this:

Code:  free -m
             total       used       free     shared    buffers     cached
Mem:           497        445         51          0         28        282
-/+ buffers/cache:        134        362
Swap:          255          0        255


That's just sweet if you ask me.

Quote: I don't worry about it actually using swap, I just wish ti wouldn't hit the disk so hard when there is plenty of ram available. If I get anywhere near actually using all the memory, it will OOM- even when there is some still unused.

Just adjust your swappiness and forget about it :P

[kenn]

I have the same problem.

We have a lot of servers - while 2.6.39.1-linode34 instances run just fine without swapping, 3.0.18-linode43 instances are REALLY swappy, to the point where sometimes they just hang up completely. Like this.



Notice the blank period - that's when the node was completely down.

Scary, huh? Fortunately for us, it's just one in a large cluster (that runs the same app code) so HAProxy just stop routing requests to that node, and no harm is actually done, except the slightly less overall capacity.

But it's a PITA nonetheless - in that case, we have to manually reboot from Linode Manager, cause even SSH is not responding.

I'm scared if it ever happens on our database nodes.

vm.swappiness is set to 0 on all machines, but it's not helping.

[nehalem]

I set my swappiness to 20 and it was perfect. If you're still swapping maybe you genuinely need more memory.

[kenn]

nehalem wrote: I set my swappiness to 20 and it was perfect. If you're still swapping maybe you genuinely need more memory.

That's not a logical answer.

The point is, it has been totally fine with the same amount of memory with 2.6. Plus, as you can see in the graph, memory used by app is about 60% and free + cache (= de facto free) is never less than 40% of total RAM.

What I need to know is, what makes the difference between 2.6 and 3.0, and if that's no clear, what exactly amount is "genuinely necessary memory" to determine if it would make more sense costwise to just stick with 2.6 for now.