This came up in a separate topic, but started me wondering about what might have happened to reserved memory in recent Linode kernels. I've generally run my Linodes with 2.6.18, but have been doing some testing recently with paravirt kernels - mostly for IPv6 filtering support - and am a little discouraged by some of the numbers I'm seeing, as it seems moving to the later kernels significantly reduces available memory.

It seems like there's a rather significant growth in the amount of memory being reserved by the paravirt kernels, certainly compared to the latest stable 2.6.18, but notably quite significant between somewhat close recent paravirt releases.

For example, some comparisons of dmesg output (all on Linode 512s):

2.6.18.8-linode22
    Memory: 511908k/532480k available (3989k kernel code, 12360k reserved, 1102k data, 224k init, 0k highmem)

2.6.38.3-linode32
    Memory: 509416k/532480k available (5378k kernel code, 22616k reserved, 1570k data, 424k init, 0k highmem)

2.6.39-linode33 #5
    Memory: 480264k/4202496k available (5700k kernel code, 43576k reserved, 1666k data, 412k init, 0k highmem)

Now I realize some growth over time might be expected, and the kernel/data increases seem reasonable, but the jump in reserved memory (especially between 2.6.38 and 2.6.39) seems excessive. Does anyone know what changed in that jump that could suddenly need almost twice as much reserved memory? I do know that vm.min_free_bytes was tweaked a few times for 2.6.39 (at least in part leading to build #5), but between my 2.6.18 and 2.6.39 nodes it's only changing from 2918K to 4096K so that's relatively minor.

I'm not that familiar with kernel memory management, but based on what I've been able to find, I don't think (but would love to be corrected) the kernel is going to use that memory for caching or apps, but only hold it ready for internal structures. So it seems to me that moving from latest stable to latest paravirt effectively decreases my working memory on a Linode 512 by ~30MB or 5% (almost 4% of which is just a change from 2.6.38 to 2.6.39).

One thing that I found interesting was the total memory shown in only the 2.6.39 log ... I guess it has different visibility into the host environment. But if the kernel is basing any calculations on that different figure, perhaps it explains some of the increase?

Can anyone shed better light on what might be going on and/or if this increase in reserved memory does in fact have a real impact on system memory usage. Any thoughts on how to get some of it back (assuming it is in fact largely being wasted)?

-- David

Over the same time period (since 2.6.18 was introduced), Linode has increased the amount of RAM included with a $20/mth linode from 100MB to 512MB. So, if RAM usage of the kernel has gone up by 30MB, but available RAM has gone up by 412MB, you're way ahead here.

To be honest, I don't think you really have a justifiable complaint. In fact, just recently we all got a free ~40% RAM upgrade, bringing the base linode from 360 to 512MB of RAM.

Hmm, not sure how you read my post as a complaint?

I was asking a technical question in regards to reserved memory growth. That's why I put it in the performance/tuning forum, and not, for example, the bugs forum. Though I suppose in honestly I do hold some hope it'll turn out to be something that can be better tuned with the later kernels.

Independent of Linode size, it appears that switching between the latest stable vs. paravirt kernels - or perhaps more notable, just between recent paravirt kernels - takes a chunk of memory away (I think, given my understanding so far of the behavior of kernel reserved memory) that is larger than I would have expected. I'm trying to understand why, and if there's a way to minimize it.

-- David

I found this discussion that shows that using the mem= kernel command line option reduces the reserved memory. But they are talking about kernel 3.0.0-rc2, so it may not apply to our Linodes.

I would try it out myself, but I'm not setup with pvgrub for running my own kernels.

Interesting, and nice find, in terms of a clear reference to Xen and reserved growth. The 3.0 kernel is really just the next usual release after 2.6.39 so I could see something actually changing in 2.6.39 but only happening to get noticed/focused on in 3.0.

In particular, the memory map analysis shown near the top of the thread seems to match very closely to my Linode 512 (except for the one usable RAM map line changing from 256MB to 512MB).

Here's the difference between 2.6.38 and 2.6.39 on my Linode 512s:

2.6.38.3-linode32
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000020800000 (usable)

2.6.39-linode33 #5
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000020000000 (usable)
 Xen: 0000000100000000 - 0000000100800000 (usable)

So identical total usable memory, but 2.6.39 shows this big hole between the 512MB block and the final 8MB block, while they are consolidated with 2.6.38. Presumably that unreachable 4GB gap is what increased the calculation for tables in reserved space, and creates the larger physical memory displayed in dmesg. I suppose it's possible that either kernel would calculate reserved memory the same way given the difference in map, as I don't know who has more impact on that map - the host Xen or the kernel itself, in how it interfaces to the host?

I had tried adding mem=512 to the kernel startup options (since you can do that with standard kernels through the Linode Manager) but it had no effect. Although dmesg memory still showed the larger total physical memory value, so that would jive with it being some tables computed based on that maximum value and not the reachable memory.

As you say without a pv-grub setup (which I also don't use), the other kernel options can't really be tested. It would be nice if other kernel options could be specified through the manager; in this case the memmap approach looks like it might be a practical workaround.

Can anyone from Linode confirm how the Xen guest memory is currently set up, and if there's any difference between the two kernels or is this most likely a change in how the kernel itself uses the same information provided by the host?

-- David

I tried setting my max ram to 511MB, but there was no mem=511M argument in my kernel command line. I did have about 1 less meg of ram, so Linode must use a different method to limit ram.

Looks like things improve for 2.6.39.1 (32 bit):

Linux ur-mom 2.6.39.1-linode34 #1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux
Memory: 509012k/532480k available (5701k kernel code, 23020k reserved, 1656k data, 412k init, 0k highmem)

These are ready to go now.

http://www.linode.com/kernels/
http://www.linode.com/kernels/rss.xml

-Chris

Yay! Recovered about 28MB from 2.6.39 to 2.6.39.1. So 2.6.39.1 now appears to be only a 2-3MB penalty over 2.6.18, which seems much more reasonable.

Interestingly, the "BIOS provided memory map" in dmesg now shows the consolidated usable memory region again (like pre-2.6.39). I guess I would have thought changes to that had to mean Xen was providing different information to the guest, but I guess not - it must have something to do with how the kernel is interpreting whatever is being provided.

Edit: Ah, it may have been this patch: http://git.kernel.org/?p=linux/kernel/g ... a5aa2b72c4 to prevent pushing that extra range up above 4GB in the case of Xen 32-bit.

-- David

Unfortunately the x86-64 version still seems to suffer from the same or a similar kind of problem in 2.6.39.1.

$ uname -a
Linux foo 2.6.39.1-x86_64-linode19 #1 SMP Tue Jun 21 10:04:20 EDT 2011 x86_64 GNU/Linux
$ dmesg | grep ^Mem
Memory: 433024k/4202496k available (6041k kernel code, 3670464k absent, 99008k reserved, 5314k data, 668k init)
$ free -m
             total       used       free     shared    buffers     cached
Mem:           424         89        335          0          2         35
-/+ buffers/cache:         51        373
Swap:         1023          0       1023
$

As a reference, here's the free -m output from 2.6.38:

             total       used       free     shared    buffers     cached
Mem:           490        441         48          0         95        258
-/+ buffers/cache:         88        401
Swap:         1023          1       1022

(I know it serves little purpose in itself with x86-64 on the 512MB plan, but I want it for full compatibility with a separate system.)

Yeah, I believe Chris mentioned that (colorfully) in another thread.

The commit I found earlier is specifically only 32bit. It appears to be "fixing" the 32bit case against a prior 2.6.39 commit that introduced the behavior: http://git.kernel.org/?p=linux/kernel/g ... fc8c3f2b26

I'm not familiar enough with the kernel VM setup processing to know why the original change was needed, nor why it was only corrected for 32bit. But I don't see 64bit behavior changing without further upstream commits, which at this point would mean completely rolling back the original commit, and presumably it was made for some reason.

-- David

I see 3.0 kernels were just added by Linode (thanks!).
Any news on memory issue in 64-bit version?

It actually looks much better: