This morning I had some trouble with network connectivity to my server. I noticed the the following kernel stack trace in /var/log/messages. I will assume it's nothing and reboot for good measure but I thought I'd post it for posterity.

Mike

...
Jul 22 09:15:06 www ntpd[2880]: kernel time sync enabled 0001
Jul 22 09:49:15 www ntpd[2880]: kernel time sync enabled 4001
Jul 22 10:23:22 www ntpd[2880]: kernel time sync enabled 0001
Jul 22 11:44:31 www kernel: swapper: page allocation failure. order:5, mode:0x20
Jul 22 11:44:31 www kernel: Pid: 0, comm: swapper Not tainted 2.6.38.3-linode32 #1
Jul 22 11:44:31 www kernel: Call Trace:
Jul 22 11:44:31 www kernel: [<c0184720>] ? __alloc_pages_nodemask+0x4f0/0x670
Jul 22 11:44:31 www kernel: [<c01aa339>] ? cache_alloc_refill+0x2f9/0x520
Jul 22 11:44:31 www kernel: [<c01aa614>] ? __kmalloc+0xb4/0xe0
Jul 22 11:44:31 www kernel: [<c04f44fe>] ? pskb_expand_head+0x12e/0x200
Jul 22 11:44:31 www kernel: [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
Jul 22 11:44:31 www kernel: [<c04f4a4d>] ? __pskb_pull_tail+0x4d/0x2b0
Jul 22 11:44:31 www kernel: [<c010630b>] ? xen_restore_fl_direct_end+0x0/0x1
Jul 22 11:44:31 www kernel: [<c04fcbba>] ? dev_hard_start_xmit+0x26a/0x520
Jul 22 11:44:31 www kernel: [<c0510912>] ? sch_direct_xmit+0xb2/0x170
Jul 22 11:44:31 www kernel: [<c0534e44>] ? nf_iterate+0x74/0xa0
Jul 22 11:44:31 www kernel: [<c0559e20>] ? ip_finish_output+0x0/0x300
Jul 22 11:44:31 www kernel: [<c04fcf91>] ? dev_queue_xmit+0x121/0x550
Jul 22 11:44:31 www kernel: [<c0559e20>] ? ip_finish_output+0x0/0x300
Jul 22 11:44:31 www kernel: [<c0559f54>] ? ip_finish_output+0x134/0x300
Jul 22 11:44:31 www kernel: [<c055a1ca>] ? ip_output+0xaa/0xe0
Jul 22 11:44:31 www kernel: [<c0559e20>] ? ip_finish_output+0x0/0x300
Jul 22 11:44:31 www kernel: [<c05590e8>] ? ip_local_out+0x18/0x20
Jul 22 11:44:31 www kernel: [<c05597e7>] ? ip_queue_xmit+0x137/0x3a0
Jul 22 11:44:31 www kernel: [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
Jul 22 11:44:31 www kernel: [<c056c7d2>] ? tcp_transmit_skb+0x372/0x7e0
Jul 22 11:44:31 www kernel: [<c056ed98>] ? tcp_write_xmit+0x198/0x980
Jul 22 11:44:31 www kernel: [<c056f5e4>] ? __tcp_push_pending_frames+0x24/0x90
Jul 22 11:44:31 www kernel: [<c056a576>] ? tcp_rcv_established+0x146/0x840
Jul 22 11:44:31 www kernel: [<c0571fc6>] ? tcp_v4_do_rcv+0xd6/0x230
Jul 22 11:44:31 www kernel: [<c01382a6>] ? local_bh_enable+0x16/0x80
Jul 22 11:44:31 www kernel: [<c05727ec>] ? tcp_v4_rcv+0x6cc/0x7b0
Jul 22 11:44:31 www kernel: [<c0554fb7>] ? ip_local_deliver_finish+0x97/0x220
Jul 22 11:44:31 www kernel: [<c0554f20>] ? ip_local_deliver_finish+0x0/0x220
Jul 22 11:44:31 www kernel: [<c0554926>] ? ip_rcv_finish+0xf6/0x3c0
Jul 22 11:44:31 www kernel: [<c04fae3d>] ? __netif_receive_skb+0x32d/0x510
Jul 22 11:44:31 www kernel: [<c0105b37>] ? xen_force_evtchn_callback+0x17/0x30
Jul 22 11:44:31 www kernel: [<c04fbcf7>] ? netif_receive_skb+0x67/0x70
Jul 22 11:44:31 www kernel: [<c04a0015>] ? xennet_poll+0x7f5/0xc20
Jul 22 11:44:31 www kernel: [<c04fc43a>] ? net_rx_action+0x9a/0x130
Jul 22 11:44:31 www kernel: [<c01380ec>] ? __do_softirq+0x7c/0x130
Jul 22 11:44:31 www kernel: [<c0138070>] ? __do_softirq+0x0/0x130
Jul 22 11:44:31 www kernel: <IRQ> [<c0137fe5>] ? irq_exit+0x65/0x70
Jul 22 11:44:31 www kernel: [<c043a03d>] ? xen_evtchn_do_upcall+0x1d/0x30
Jul 22 11:44:31 www kernel: [<c0109487>] ? xen_do_upcall+0x7/0xc
Jul 22 11:44:31 www kernel: [<c01013a7>] ? hypercall_page+0x3a7/0x1010
Jul 22 11:44:31 www kernel: [<c0105b8f>] ? xen_safe_halt+0xf/0x20
Jul 22 11:44:31 www kernel: [<c010f66f>] ? default_idle+0x2f/0x60
Jul 22 11:44:31 www kernel: [<c0107ed2>] ? cpu_idle+0x42/0x70
Jul 22 11:44:31 www kernel: [<c07ca8ac>] ? start_kernel+0x2da/0x2df
Jul 22 11:44:31 www kernel: [<c07ca410>] ? unknown_bootoption+0x0/0x190
Jul 22 11:44:31 www kernel: [<c07cdaa5>] ? xen_start_kernel+0x530/0x538
Jul 22 11:44:31 www kernel: Mem-Info:
Jul 22 11:44:31 www kernel: DMA per-cpu:
Jul 22 11:44:31 www kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 22 11:44:31 www kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jul 22 11:44:31 www kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jul 22 11:44:31 www kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jul 22 11:44:31 www kernel: Normal per-cpu:
Jul 22 11:44:31 www kernel: CPU 0: hi: 186, btch: 31 usd: 150
Jul 22 11:44:31 www kernel: CPU 1: hi: 186, btch: 31 usd: 135
Jul 22 11:44:31 www kernel: CPU 2: hi: 186, btch: 31 usd: 108
Jul 22 11:44:31 www kernel: CPU 3: hi: 186, btch: 31 usd: 180
Jul 22 11:44:31 www kernel: active_anon:15618 inactive_anon:15212 isolated_anon:0
Jul 22 11:44:31 www kernel: active_file:49699 inactive_file:33152 isolated_file:0
Jul 22 11:44:31 www kernel: unevictable:1136 dirty:1 writeback:0 unstable:0
Jul 22 11:44:31 www kernel: free:3443 slab_reclaimable:3771 slab_unreclaimable:1830
Jul 22 11:44:31 www kernel: mapped:3869 shmem:105 pagetables:507 bounce:0
Jul 22 11:44:31 www kernel: DMA free:3044kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:140kB active_file:668kB inactive_file:2728kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:72kB shmem:0kB slab_reclaimable:128kB slab_unreclaimable:392kB kernel_stack:32kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 22 11:44:31 www kernel: lowmem_reserve[]: 0 500 500 500
Jul 22 11:44:31 www kernel: Normal free:10728kB min:2816kB low:3520kB high:4224kB active_anon:62472kB inactive_anon:60708kB active_file:198128kB inactive_file:129880kB unevictable:4544kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:4544kB dirty:4kB writeback:0kB mapped:15404kB shmem:420kB slab_reclaimable:14956kB slab_unreclaimable:6928kB kernel_stack:824kB pagetables:2028kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 22 11:44:31 www kernel: lowmem_reserve[]: 0 0 0 0
Jul 22 11:44:31 www kernel: DMA: 409*4kB 62*8kB 25*16kB 4*32kB 4*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3044kB
Jul 22 11:44:31 www kernel: Normal: 2540*4kB 65*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 10728kB
Jul 22 11:44:31 www kernel: 84794 total pagecache pages
Jul 22 11:44:31 www kernel: 966 pages in swap cache
Jul 22 11:44:31 www kernel: Swap cache stats: add 14067, delete 13101, find 200063/200381
Jul 22 11:44:31 www kernel: Free swap = 255592kB
Jul 22 11:44:31 www kernel: Total swap = 262140kB
Jul 22 11:44:31 www kernel: 133104 pages RAM
Jul 22 11:44:31 www kernel: 0 pages HighMem
Jul 22 11:44:31 www kernel: 5637 pages reserved
Jul 22 11:44:31 www kernel: 90512 pages shared
Jul 22 11:44:31 www kernel: 68737 pages non-shared
Jul 22 12:13:15 www kernel: device eth0 entered promiscuous mode
Jul 22 12:13:21 www kernel: device eth0 left promiscuous mode
Jul 22 12:13:23 www kernel: device eth0 entered promiscuous mode
Jul 22 12:13:29 www kernel: device eth0 left promiscuous mode
Jul 22 12:14:24 www kernel: device eth0 entered promiscuous mode
Jul 22 12:14:29 www kernel: device eth0 left promiscuous mode
Jul 22 12:25:09 www dhclient: DHCPREQUEST on eth0 to 64.22.84.11 port 67
Jul 22 12:25:09 www dhclient: DHCPACK from 64.22.84.11
Jul 22 12:25:09 www dhclient: bound to 173.230.137.181 -- renewal in 42564 seconds.
Jul 22 13:14:00 www ntpd[2880]: kernel time sync enabled 4001

I think that's been fixed in more recent kernels.

Thanks for the report. Yes, please always try to reproduce using the latest kernel.

-Chris

How does one use the latest kernel? My profile reads:

My CentOS 5.5 Profile (Latest 2.6 Paravirt (2.6.39.1-linode34))

even though the log fragement mentions 2.6.38.3-linode32.

Do I simply reboot?

I have not actually rebooted yet. I was going to wait until later tonight.

Mike

If you haven't changed the profile, then you probably haven't rebooted since 2.6.38.3-linode32 was latest. Answer is indeed to simply reboot.