Hello,

Ever since I upgraded my Linode to Debian squeeze, I have been seeing regular kernel "page allocation failure" messages such as the following:

kernel: swapper: page allocation failure. order:4, mode:0x20
kernel: Pid: 0, comm: swapper Not tainted 2.6.38-linode31 #1
kernel: Call Trace:
kernel:  [<c01845c0>] ? __alloc_pages_nodemask+0x4f0/0x670
kernel:  [<c01aa259>] ? cache_alloc_refill+0x2f9/0x520
kernel:  [<c01aa534>] ? __kmalloc+0xb4/0xe0
kernel:  [<c04f43be>] ? pskb_expand_head+0x12e/0x200
kernel:  [<c010630b>] ? xen_restore_fl_direct_end+0x0/0x1
kernel:  [<c01a9666>] ? kmem_cache_free+0x46/0x120
kernel:  [<c04f490d>] ? __pskb_pull_tail+0x4d/0x2b0
kernel:  [<c05ee17d>] ? packet_rcv_spkt+0xfd/0x140
kernel:  [<c04fca7a>] ? dev_hard_start_xmit+0x26a/0x520
kernel:  [<c0510792>] ? sch_direct_xmit+0xb2/0x170
kernel:  [<c0534cc4>] ? nf_iterate+0x74/0xa0
kernel:  [<c0559ca0>] ? ip_finish_output+0x0/0x300
kernel:  [<c04fce51>] ? dev_queue_xmit+0x121/0x550
kernel:  [<c0559ca0>] ? ip_finish_output+0x0/0x300
kernel:  [<c0559dd4>] ? ip_finish_output+0x134/0x300
kernel:  [<c055a04a>] ? ip_output+0xaa/0xe0
kernel:  [<c0559ca0>] ? ip_finish_output+0x0/0x300
kernel:  [<c0558f68>] ? ip_local_out+0x18/0x20
kernel:  [<c0559667>] ? ip_queue_xmit+0x137/0x3a0
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c056c652>] ? tcp_transmit_skb+0x372/0x7e0
kernel:  [<c056ec18>] ? tcp_write_xmit+0x198/0x980
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c056f464>] ? __tcp_push_pending_frames+0x24/0x90
kernel:  [<c056a3f6>] ? tcp_rcv_established+0x146/0x840
kernel:  [<c0571e46>] ? tcp_v4_do_rcv+0xd6/0x230
kernel:  [<c01382c6>] ? local_bh_enable+0x16/0x80
kernel:  [<c057266c>] ? tcp_v4_rcv+0x6cc/0x7b0
kernel:  [<c0554e37>] ? ip_local_deliver_finish+0x97/0x220
kernel:  [<c0554da0>] ? ip_local_deliver_finish+0x0/0x220
kernel:  [<c05547a6>] ? ip_rcv_finish+0xf6/0x3c0
kernel:  [<c04facfd>] ? __netif_receive_skb+0x32d/0x510
kernel:  [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
kernel:  [<c04fbbb7>] ? netif_receive_skb+0x67/0x70
kernel:  [<c049ff05>] ? xennet_poll+0x7f5/0xc20
kernel:  [<c04fc2fa>] ? net_rx_action+0x9a/0x130
kernel:  [<c013810c>] ? __do_softirq+0x7c/0x130
kernel:  [<c0138090>] ? __do_softirq+0x0/0x130
kernel:  <IRQ>  [<c0138005>] ? irq_exit+0x65/0x70
kernel:  [<c0439f2d>] ? xen_evtchn_do_upcall+0x1d/0x30
kernel:  [<c0109487>] ? xen_do_upcall+0x7/0xc
kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1010
kernel:  [<c0105b8f>] ? xen_safe_halt+0xf/0x20
kernel:  [<c010f66f>] ? default_idle+0x2f/0x60
kernel:  [<c0107ed2>] ? cpu_idle+0x42/0x70
kernel:  [<c07ca8ac>] ? start_kernel+0x2da/0x2dfApr 12 06:38:43 li140-180 kernel:  [<c07ca410>] ? unknown_bootoption+0x0/0x190
kernel:  [<c07cdaa5>] ? xen_start_kernel+0x530/0x538
kernel: Mem-Info:
kernel: DMA per-cpu:
kernel: CPU    0: hi:    0, btch:   1 usd:   0
kernel: CPU    1: hi:    0, btch:   1 usd:   0
kernel: CPU    2: hi:    0, btch:   1 usd:   0
kernel: CPU    3: hi:    0, btch:   1 usd:   0
kernel: Normal per-cpu:
kernel: CPU    0: hi:  186, btch:  31 usd:  42
kernel: CPU    1: hi:  186, btch:  31 usd: 171
kernel: CPU    2: hi:  186, btch:  31 usd: 134
kernel: CPU    3: hi:  186, btch:  31 usd: 184
kernel: active_anon:16222 inactive_anon:27566 isolated_anon:0
kernel:  active_file:27915 inactive_file:31023 isolated_file:0
kernel:  unevictable:0 dirty:33 writeback:0 unstable:0
kernel:  free:9224 slab_reclaimable:9493 slab_unreclaimable:2108
kernel:  mapped:7298 shmem:4430 pagetables:611 bounce:0
kernel: DMA free:2084kB min:84kB low:104kB high:124kB active_anon:120kB inactive_anon:316kB active_file:2692kB inactive_file:2804kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:580kB shmem:408kB slab_reclaimable:72kB slab_unreclaimable:32kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 500 500 500
kernel: Normal free:34812kB min:2816kB low:3520kB high:4224kB active_anon:64768kB inactive_anon:109948kB active_file:108968kB inactive_file:121288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:132kB writeback:0kB mapped:28612kB shmem:17312kB slab_reclaimable:37900kB slab_unreclaimable:8400kB kernel_stack:1144kB pagetables:2444kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 0
kernel: DMA: 241*4kB 62*8kB 5*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2084kB
kernel: Normal: 2285*4kB 2697*8kB 225*16kB 17*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 34860kB
kernel: 78061 total pagecache pages
kernel: 14676 pages in swap cache
kernel: Swap cache stats: add 330524, delete 315848, find 3444837/3487305
kernel: Free swap  = 423812kB
kernel: Total swap = 524284kB
kernel: 133104 pages RAM
kernel: 0 pages HighMem
kernel: 5637 pages reserved
kernel: 67797 pages shared
kernel: 65096 pages non-shared

As indicated, this is using Linux 2.6.38-linode31.

The involved process is always "swapper", "apache2" or "ssh" and the problem most frequently occurs during a nightly script that rsyncs to a volume mounted over sshfs+encfs (rsync does not report any error).

This is the memory information that may be relevant:

# free -lm
             total       used       free     shared    buffers     cached
Mem:           497        426         71          0         38        166
Low:           497        426         71
High:            0          0          0
-/+ buffers/cache:        221        276
Swap:          511         77        434
# cat /proc/sys/vm/lowmem_reserve_ratio 
256   32   32
# cat /proc/zoneinfo 
Node 0, zone      DMA
  pages free     688
        min      21
        low      26
        high     31
        scanned  0
        spanned  4080
        present  3952
    nr_free_pages 688
    nr_inactive_anon 48
    nr_active_anon 83
    nr_inactive_file 578
    nr_active_file 584
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 18
    nr_mapped    150
    nr_file_pages 1279
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 9
    nr_slab_unreclaimable 0
    nr_page_table_pages 0
    nr_kernel_stack 21
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     102
    nr_dirtied   5617
    nr_written   5608
    nr_anon_transparent_hugepages 0
        protection: (0, 500, 500, 500)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
    cpu: 3
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 6
  all_unreclaimable: 0
  start_pfn:         16
  inactive_ratio:    1
Node 0, zone   Normal
  pages free     17443
        min      704
        low      880
        high     1056
        scanned  0
        spanned  129024
        present  128016
    nr_free_pages 17443
    nr_inactive_anon 33819
    nr_active_anon 18066
    nr_inactive_file 18962
    nr_active_file 27900
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 42007
    nr_mapped    8023
    nr_file_pages 61636
    nr_dirty     30
    nr_writeback 0
    nr_slab_reclaimable 2336
    nr_slab_unreclaimable 2213
    nr_page_table_pages 919
    nr_kernel_stack 171
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 232406
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     4470
    nr_dirtied   3852185
    nr_written   3961663
    nr_anon_transparent_hugepages 0
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 184
              high:  186
              batch: 31
  vm stats threshold: 18
    cpu: 1
              count: 176
              high:  186
              batch: 31
  vm stats threshold: 18
    cpu: 2
              count: 165
              high:  186
              batch: 31
  vm stats threshold: 18
    cpu: 3
              count: 164
              high:  186
              batch: 31
  vm stats threshold: 18
  all_unreclaimable: 0
  start_pfn:         4096
  inactive_ratio:    1

Any ideas what this is about?

Thanks,
Bruno

I'm seeing the same.

I was getting server freeze-ups with Centos, so I backed up all my data onto my PC, installed Debian, and this time went with the debs for Apache, MySQL, and PHP.

On CentOS, I had compiled nginx, MySQL, and PHP. I thought the fastcgi PHP processes for nginx were suddenly causing the freeze-ups, so I switched to Apache, but still got the freeze-ups. I increased the swap file from 256 MB to 512 MB, but still "ran out of memory". That's when I decided to go from Centos 5 to Debian 6.

With Debian, I'm still seeing the freeze-ups, with the "swapper: page allocation failure" error. I don't have enough information to wonder if it might be a hardware/ram error. Since the Linux OS and all the software used were replaced, I figure the cause has to be either something related to the data I migrated to the new install, or a hardware issue.

The freeze-ups for me usually end with something killing the MySQL process, which sometimes alleviates the freeze-up and sometimes doesn't. Because of this, I've run MySQL's MyISAM repair tool on my databases to see if that helps. (I have a MyBB forum installed, which gets a fair amount of use daily.) I did this about ten hours ago, so I should know in a few days if it's resolved the issues or not. I typically see two to three server freeze-ups a day, sometimes one being within an hour of rebooting. (Rebooting seems to be the only temporarily recovery solution for me.)

I'll have to remember to respond again later whether I do or do not see more freeze-ups.

Edit: I had three more freeze-ups later the day I wrote this. Guess it wasn't a problem with the database files!

I've had the same issue. I'm also running Debian 6.

I asked support and they recommended increasing the kernel param vm.min_free_kbytes to 16384. It was about 2900 before that.

I highly recommend you open a ticket with support and send them that dmesg output - don't just paste that setting

I'll try changing that setting, and I'll read up on it a bit so I can understand what's happening. After I put in the change, I'll be sure to observe any changes in the server behavior, and see if a ticket should be put in (as there were issues with "default" settings.)

Edit: I still had a freeze-up even with vm.min_free_kbytes set higher. I'll try what melon wrote below after I get home from work, and will report on my results. If things still don't work out, it'll be ticket time.

I'm having the very same issue, running Ubuntu Lucid. I ran the same laps with support. After all they advised booting 2.6.32.16-linode28. I rebooted this morning with this older kernel and I am seeing no failures so far, but I'll wait 1-2 days before reporting back.

I was also having this issue with ubuntu 10.04, support suggested I change vm.min_free_kbytes to a higher value, it was 2906. I think they had suggested 16384, that solved it, I dropped it down to 8192 and still not seeing any issue.

to check it: sysctl vm.min_free_kbytes
change it: sysctl vm.min_free_kbytes=16384
to keep the change through a reboot you'll need to edit /etc/sysctl.conf

The issue was also solved by increasing vm.min_free_kbytes and rebooting. Support suggested:



I had vm.min_free_kbytes = 2906 before.

I thought for certain I updated my /etc/sysctl.conf file, but was still getting freeze ups, and checking the file showed I didn't have the necessary line added. I've made certain I edited the file this time, so here's hoping the freeze-ups will be gone for me now!

Same issue here. Will try the sysctl vm.min_free_kbytes solution. Damn you, xen!

Is that bolded line a typo?

_________________
sleddog

I've also noticed that recent Linode kernels (32-bit at least) have some difference. The prior latest paravirt (2.6.39-linode33), at least as of changes included in build #3, increased the default vm.free_min_kbytes to 4096 (from ~2900 in the 2.6.38 kernels), while the current latest paravirt (2.6.39.1-linode34) seems to be back to an older 2906. I wonder if a local Linode patch wasn't migrated forward, or are even the recent issues only on Linodes running a 2.6.38 kernel?

-- David

This is pretty much an obsolete thread... the underlying problem was fixed in mainline 2.6.39, at least for 32-bit. Just reboot with Latest 2.6 Paravirt and you (should) be OK.

If you're running 64-bit... well, if you've always wanted your name in the kernel changelog, this is your chance.

_________________

/* TODO: need to add signature to posts */

Do you have a reference for this? I assume you mean the underlying problem that the increased setting was a workaround for, and I'm curious what that issue was.

Thanks.

-- David

viewtopic.php?t=7451
viewtopic.php?t=7308

Along with various chatter on IRC, a couple bits on http://www.linode.com/kernels/, etc.

_________________

/* TODO: need to add signature to posts */

Thanks.

It was the kernel listing that made me believe it was the change noted with 2.6.39-linode33 #3 ("increased min_free_kbytes ratio calculation") that was intended to address the problem, and I did see the default value increase with that kernel, and subsequent 2.6.39-linode33 builds. But that appears to have gone away in 2.6.39.1 with the default value going back to 2.6.38 levels.

The other threads do appear, judging from post dates, to indicate that 2.6.39.1 should also be ok, but I'm curious how that's true with the lower default value. I was wondering if perhaps there was an upstream change that might have obviated the need for a larger value, or could 2.6.39.1 have re-introduced the susceptibility and people just haven't rebooted into it as much yet to have it happen?

-- David