Here is what I found on the Lish console. I'm on fremont37 and running the 2.6.25-linode9 kernel. Any ideas?

BUG: unable to handle kernel paging request at 889cbbea
IP: [<c0182b97>] prune_dcache+0x87/0x180
*pdpt = 000000022a452027
Oops: 0002 [#4] SMP
Modules linked in:

Pid: 122, comm: kswapd0 Tainted: G      D  (2.6.25-linode9 #1)
EIP: 0061:[<c0182b97>] EFLAGS: 00010202 CPU: 1
EIP is at prune_dcache+0x87/0x180
EAX: 889cbbea EBX: da6cb094 ECX: da6cb0bc EDX: c0615d4c
ESI: 00000068 EDI: 00000000 EBP: 00000001 ESP: ecdc1ef8
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process kswapd0 (pid: 122, ti=ecdc0000 task=ecd40eb0 task.ti=ecdc0000)
Stack: 00006f54 c0615d38 00000096 000290a5 c0182cca c0155c65 c12d4be0 c130d900
       ecdc1f18 00000020 00000000 c069f640 000000d0 00000000 00000180 00000001
       c0612680 c0613200 00000001 c01570dc 00000001 00000000 c06154fc 00000000
Call Trace:
 [<c0182cca>] shrink_dcache_memory+0x3a/0x40
 [<c0155c65>] shrink_slab+0x115/0x190
 [<c01570dc>] kswapd+0x2cc/0x470
 [<c0155ad0>] isolate_pages_global+0x0/0x80
 [<c0136330>] autoremove_wake_function+0x0/0x50
 [<c0156e10>] kswapd+0x0/0x470
 [<c0136134>] kthread+0x74/0x80
 [<c01360c0>] kthread+0x0/0x80
 [<c01077a7>] kernel_thread_helper+0x7/0x10
 =======================
Code: 8d 59 d8 3b 7b 50 0f 84 f4 00 00 00 40 8b 49 04 39 c2 75 e0 81 f9 4c 5d 61 c0 74 73 8d 59 d8 4
e 8b 41 04 8b 11 89 42 04 89 49 04 <89> 10 a1 50 5d 61 c0 89 09 0f 18 00 90 8d 43 08 ff 0d 24 5d 61
EIP: [<c0182b97>] prune_dcache+0x87/0x180 SS:ESP 0069:ecdc1ef8
---[ end trace 312d266a1134b5bd ]---

----------------------------------------------------------------------------------------------------

Well .. the pv_ops Xen kernels (those > 2.6.18) are still somewhat experimental. I'd give it a few more versions before running those in production.

I'll forward your crash dump along to the pv_ops guys.

-Chris

That's fine. I've been using that kernel for about a month now without problems till this morning.

Thanks!

Well, I just rebooted and it crashed again very quickly (approx 10 minutes) using the same kernel. I'll try the Latest 2.6 Series kernel. I can't get the full trace from Lish because of the terminal size but here is what I can see:

 [<c039407d>] notify_update+0x1d/0x30
 [<c039518d>] vt_console_print+0x1ed/0x2d0
 [<c0394fa0>] vt_console_print+0x0/0x2d0
 [<c0122dee>] __call_console_drivers+0x5e/0x70
 [<c051c69c>] _spin_unlock_irqrestore+0xc/0x10
 [<c014272d>] sys_futex+0x9d/0x130
 [<c0123830>] vprintk+0x1c0/0x390
 [<c012073f>] mm_release+0x7f/0x90
 [<c01248d2>] exit_mm+0x12/0xe0
 [<c012636c>] do_exit+0x14c/0x6d0
 [<c0123a1b>] printk+0x1b/0x20
 [<c010857f>] die+0x17f/0x180
 [<c0115624>] do_page_fault+0x564/0xa20
 [<c0151155>] __alloc_pages+0x55/0x370
 [<c010348a>] xen_set_pte_at+0x6a/0xf0
 [<c01150c0>] do_page_fault+0x0/0xa20
 [<c051c932>] error_code+0x72/0x78
 [<c016007b>] vma_link+0x6b/0x100
 [<c01500d8>] setup_per_zone_lowmem_reserve+0x48/0xf0
 [<c0103195>] xen_make_pte+0x45/0x50
 [<c01620da>] mprotect_fixup+0x3da/0x600
 [<c051c60a>] _spin_lock_irq+0xa/0x30
 [<c012c3e0>] run_timer_softirq+0x130/0x190
 [<c0162489>] sys_mprotect+0x189/0x240
 [<c0106bce>] syscall_call+0x7/0xb
 =======================
Code: b8 01 00 00 00 e9 d6 ff ff ff 8d b6 00 00 00 00 64 8b 15 10 e1 69 c0 b8 6c e1 69 c0 8b 0c 10 8
5 c9 75 08 c7 04 10 02 00 00 00 c3 <0f> 0b eb fe 90 b8 02 00 00 00 e9 a6 ff ff ff 8d b6 00 00 00 00
EIP: [<c0113a9b>] paravirt_enter_lazy_cpu+0x1b/0x20 SS:ESP 0069:d773bc0c
---[ end trace 6e3af9dcb4e20f12 ]---
Fixing recursive fault but reboot is needed!

----------------------------------------------------------------------------------------------------

I can't help but think something is wrong with fremont37.

My linode crashed yet again while running the 2.6.18.8-domU-linode7 kernel...

Call Trace:
 [<c0173edf>] prune_one_dentry+0x54/0x75
 [<c0174031>] prune_dcache+0x131/0x15b
 [<c0174094>] shrink_dcache_memory+0x39/0x3b
 [<c01439a0>] shrink_slab+0x111/0x186
 [<c012eb7b>] finish_wait+0x25/0x4b
 [<c0144d5b>] kswapd+0x2e9/0x3eb
 [<c012e960>] autoremove_wake_function+0x0/0x37
 [<c0144a72>] kswapd+0x0/0x3eb
 [<c012e89a>] kthread+0xde/0xe2
 [<c012e7bc>] kthread+0x0/0xe2
 [<c0102b75>] kernel_thread_helper+0x5/0xb
Code: 06 89 d8 ff d2 5b c3 89 da a1 00 de 58 c0 5b e9 4f 52 fe ff 0f 0b ae 00 26 6b 4c c0 eb c5 53 89 c3 85 c0 74 58 8b 80 9c 00 00 00 <8b> 40 20 83 bb 40 01 00 00 20 74 48 85 c0 74 0e 8b 50 14 85 d2
EIP: [<c01750f4>] iput+0xd/0x6b SS:ESP 0069:ec5dfee4
 <1>BUG: unable to handle kernel paging request at virtual address 009f4557
 printing eip:
c01750f4
21e7d000 -> *pde = 00000002:10fff027
083bc000 -> *pme = 00000000:00000000
Oops: 0000 [#2]
SMP
Modules linked in:
CPU:    0
EIP:    0061:[<c01750f4>]    Not tainted VLI
EFLAGS: 00210286   (2.6.18.8-domU-linode7 #1)
EIP is at iput+0xd/0x6b
eax: 009f4537   ebx: d110b0d4   ecx: d110b0ec   edx: d110b0ec
esi: 0000007f   edi: 00000000   ebp: ec38da3c   esp: d5c91c44
ds: 007b   es: 007b   ss: 0069
Process ruby (pid: 25356, ti=d5c90000 task=ecae0ab0 task.ti=d5c90000)
Stack: c5393b14 c0173edf c5393b14 0000007f c0174031 00000080 00000000 0000f230
       c169eac0 00000090 00027c99 c0174094 c01439a0 00000010 c017ea49 c1037020
       00000018 00000000 c05c6fe0 00000080 000201d2 00000000 0000000c 00000000
Call Trace:
 [<c0173edf>] prune_one_dentry+0x54/0x75
 [<c0174031>] prune_dcache+0x131/0x15b
 [<c0174094>] shrink_dcache_memory+0x39/0x3b
 [<c01439a0>] shrink_slab+0x111/0x186
 [<c017ea49>] mpage_bio_submit+0x19/0x1d
 [<c0144f94>] try_to_free_pages+0x137/0x1f3
 [<c0140ae7>] __alloc_pages+0x12e/0x2d2
 [<c0141f6b>] __do_page_cache_readahead+0x1f4/0x28a
 [<c01d073a>] ext3_get_block+0x0/0xc8
 [<c0114c65>] __wake_up+0x32/0x43
 [<c01421f5>] blockable_page_cache_readahead+0x53/0xbb
 [<c01422b8>] make_ahead_window+0x5b/0x9b
 [<c014244e>] page_cache_readahead+0x156/0x1c8
 [<c013d52c>] do_generic_mapping_read+0x40e/0x496
 [<c013de28>] __generic_file_aio_read+0x16e/0x243
 [<c013bc05>] file_read_actor+0x0/0xc7
 [<c010808e>] timer_interrupt+0x448/0x685
 [<c012351c>] __capable+0xc/0x1f
 [<c013e09f>] generic_file_aio_read+0x3e/0x4f
 [<c015db64>] do_sync_read+0xc1/0x11c
 [<c0131741>] hrtimer_run_queues+0xc1/0x193
 [<c012e960>] autoremove_wake_function+0x0/0x37
 [<c0120d86>] __do_softirq+0x8b/0x116
 [<c015daa3>] do_sync_read+0x0/0x11c
 [<c015df67>] vfs_read+0xa2/0x160
 [<c015ea2e>] sys_read+0x41/0x6a
 [<c0105137>] syscall_call+0x7/0xb
Code: 06 89 d8 ff d2 5b c3 89 da a1 00 de 58 c0 5b e9 4f 52 fe ff 0f 0b ae 00 26 6b 4c c0 eb c5 53 89 c3 85 c0 74 58 8b 80 9c 00 00 00 <8b> 40 20 83 bb 40 01 00 00 20 74 48 85 c0 74 0e 8b 50 14 85 d2
EIP: [<c01750f4>] iput+0xd/0x6b SS:ESP 0069:d5c91c44

My linode crashed again. Note that after the crash the system is halted and CPU utilization on the dashboard shows 100%.... and Lassie doesn't catch it / reboot it. Shouldn't my linode get rebooted automatically after a panic?

I had exactly the same thing happen to a client's linode. Lassie didn't reboot it, but I did get a warning that I'd gone 100% CPU. I saw a kernel trace like the ones posted here.

Crashed again. This time a bit different. I still have a login prompt via lish and when attempt to login I always get the following stack dump. After the dump is printed the login prompt comes back.

<1>BUG: unable to handle kernel paging request at virtual address 74732e31
printing eip:
c0174d30
13c6f000 -> *pde = 00000004:6f1a9007
25e55000 -> *pme = 00000000:00000000
Oops: 0000 [#236]
SMP
Modules linked in:
CPU: 3
EIP: 0061:[<c0174d30>] Not tainted VLI
EFLAGS: 00010202 (2.6.18.8-domU-linode7 #1)
EIP is at __d_lookup+0x5b/0xf0
eax: c1602228 ebx: 74732e31 ecx: 00000011 edx: 000056db
esi: d56d9e0c edi: d56d9f34 ebp: d56d9e18 esp: d56d9d94
ds: 007b es: 007b ss: 0069
Process imap (pid: 4950, ti=d56d8000 task=ebd81ab0 task.ti=d56d8000)
Stack: dfbcd594 32fdb346 ec537b94 ec537b94 d56d9e0c 00000000 0000000b d97dd011
d56d9e0c d56d9e0c d56d9f34 d56d9e18 c016b774 ec4eb840 c01dca50 00000001
dfbcd614 32fdb346 d56d9e0c d56d9f34 d97dd01c c016c04e d56d9f34 d97dd011
Call Trace:
[<c016b774>] do_lookup+0x1a/0x127
[<c01dca50>] ext3_permission+0x0/0xa
[<c016c04e>] __link_path_walk+0x7cd/0xeec
[<c016c7b2>] link_path_walk+0x45/0xcb
[<c015d00b>] get_unused_fd+0x57/0xbc
[<c016cbdf>] do_path_lookup+0xab/0x23b
[<c016d1d1>] __path_lookup_intent_open+0x48/0x83
[<c016d280>] path_lookup_open+0x20/0x25
[<c016d4ea>] open_namei+0x6e/0x65c
[<c015d5f3>] do_filp_open+0x25/0x40
[<c015d00b>] get_unused_fd+0x57/0xbc
[<c015d64c>] do_sys_open+0x3e/0xc7
[<c012351c>] __capable+0xc/0x1f
[<c015d712>] sys_open+0x1c/0x20
[<c0105137>] syscall_call+0x7/0xb
Code: 07 01 d0 89 c2 81 f2 01 00 37 9e 8b 0d f4 dd 58 c0 d3 ea 31 d0 23 05 f8 dd 58 c0 01 c0 01 c0 03 05 f0 dd 58 c0 8b 18 85 db 74 20 <8b> 03 0f 18 00 90 8d 6b f0 8b 54 24 04 3b 55 1c 75 08 8b 34 24
EIP: [<c0174d30>] __d_lookup+0x5b/0xf0 SS:ESP 0069:d56d9d94

I'm using the Latest 2.6 Series (2.6.18.8-linode10) kernel. Is there a recommended kernel to use with Xen?

Your crash reports are all from 2.6.18.8-domU-linode7. Let me know if -linode10 solves this for you.

Thanks,
-Chris

Oh ok. I see my latest reboot picked up -linode10. I'll let you know how it goes...

I haven't had any problem since running the new -linode10 kernel. Thank you.