|
Found this in /var/log/messages:
Nov 27 04:02:04 liXXX-XXX syslogd 1.4.1: restart.
Nov 27 21:14:37 liXXX-XXX kernel: httpd: page allocation failure. order:1, mode:0x20
Nov 27 21:14:37 liXXX-XXX kernel: Pid: 19979, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:14:37 liXXX-XXX kernel: Call Trace:
Nov 27 21:14:37 liXXX-XXX kernel: <IRQ> [<ffffffff810921b4>] ? __alloc_pages_nodemask+0x584/0x670
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff810b9ba3>] ? cache_alloc_refill+0x2e3/0x5c0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff810ba037>] ? kmem_cache_alloc+0xa7/0xb0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff8143c522>] ? sk_prot_alloc+0x42/0x130
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff8143c6b4>] ? sk_clone+0x14/0x290
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814b26f0>] ? inet_csk_clone+0x10/0xb0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814c9f21>] ? tcp_create_openreq_child+0x21/0x470
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814c85f3>] ? tcp_v4_syn_recv_sock+0x53/0x1d0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff81534c6d>] ? tcp_v6_syn_recv_sock+0x46d/0x580
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff815abc15>] ? _raw_spin_lock+0x5/0x10
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814c9d7e>] ? tcp_check_req+0x31e/0x4a0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814c7c27>] ? tcp_v4_do_rcv+0x1c7/0x2a0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff81040a22>] ? local_bh_enable+0x22/0xb0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814c843b>] ? tcp_v4_rcv+0x73b/0x8a0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814a896b>] ? ip_local_deliver_finish+0xcb/0x200
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814a83a1>] ? ip_rcv_finish+0x191/0x3b0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff814a8816>] ? ip_rcv+0x256/0x2e0
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff81448a98>] ? netif_receive_skb+0x78/0x90
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff813e744d>] ? xennet_poll+0x59d/0xe20
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:14:37 liXXX-XXX kernel: [<ffffffff81448c77>] ? net_rx_action+0x97/0x1b0
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff81040bee>] ? __do_softirq+0xae/0x140
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff813788ba>] ? unmask_evtchn+0x1a/0xf0
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff81009c7c>] ? call_softirq+0x1c/0x30
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff8100ba65>] ? do_softirq+0x65/0xa0
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff810408b5>] ? irq_exit+0x75/0x90
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff8137912e>] ? xen_evtchn_do_upcall+0x13e/0x200
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff81009cce>] ? xen_do_hypervisor_callback+0x1e/0x30
Nov 27 21:14:38 liXXX-XXX kernel: <EOI> [<ffffffff81026cf5>] ? ptep_clear_flush_young+0x35/0x80
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff810ac1f1>] ? page_referenced_one+0xc1/0x110
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff810ad24f>] ? page_referenced+0x20f/0x230
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff81096ea7>] ? shrink_page_list+0x197/0x750
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff81096c9a>] ? shrink_active_list+0x25a/0x2d0
Nov 27 21:14:38 liXXX-XXX kernel: [<ffffffff81095d0c>] ? isolate_lru_pages+0x8c/0x220
Nov 27 21:14:39 liXXX-XXX kernel: [<ffffffff81097aab>] ? shrink_zone+0x64b/0xc80
Nov 27 21:14:39 liXXX-XXX kernel: [<ffffffff815a98df>] ? io_schedule_timeout+0xaf/0xf0
Nov 27 21:14:40 liXXX-XXX kernel: [<ffffffff81052a50>] ? autoremove_wake_function+0x0/0x30
Nov 27 21:14:40 liXXX-XXX kernel: [<ffffffff81098cdd>] ? try_to_free_pages+0x1dd/0x3a0
Nov 27 21:14:40 liXXX-XXX kernel: [<ffffffff81091fe5>] ? __alloc_pages_nodemask+0x3b5/0x670
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff810a389c>] ? handle_mm_fault+0x93c/0xa90
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff81311bd0>] ? cap_file_lock+0x0/0x10
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff81311bd0>] ? cap_file_lock+0x0/0x10
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff81311bd0>] ? cap_file_lock+0x0/0x10
Nov 27 21:14:41 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:14:41 liXXX-XXX kernel: Mem-Info:
Nov 27 21:14:41 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:14:41 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:14:41 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:14:42 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:14:42 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:14:42 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:14:42 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 169
Nov 27 21:14:42 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 153
Nov 27 21:14:42 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 97
Nov 27 21:14:42 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 158
Nov 27 21:14:42 liXXX-XXX kernel: active_anon:52288 inactive_anon:52335 isolated_anon:607
Nov 27 21:14:42 liXXX-XXX kernel: active_file:2553 inactive_file:959 isolated_file:1
Nov 27 21:14:42 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Nov 27 21:14:43 liXXX-XXX kernel: free:1165 slab_reclaimable:2255 slab_unreclaimable:3630
Nov 27 21:14:43 liXXX-XXX kernel: mapped:4727 shmem:3919 pagetables:7329 bounce:0
Nov 27 21:14:43 liXXX-XXX kernel: DMA free:1988kB min:84kB low:104kB high:124kB active_anon:6096kB inactive_anon:6288kB active_file:64kB inactive_file:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:100kB slab_unreclaimable:92kB kernel_stack:160kB pagetables:148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 27 21:14:43 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:14:43 liXXX-XXX kernel: DMA32 free:2672kB min:2784kB low:3480kB high:4176kB active_anon:203056kB inactive_anon:203052kB active_file:10148kB inactive_file:3768kB unevictable:0kB isolated(anon):2428kB isolated(file):4kB present:500960kB mlocked:0kB dirty:0kB writeback:0kB mapped:18896kB shmem:15676kB slab_reclaimable:8920kB slab_unreclaimable:14428kB kernel_stack:1480kB pagetables:29168kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3200 all_unreclaimable? no
Nov 27 21:14:43 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:14:43 liXXX-XXX kernel: DMA: 1*4kB 0*8kB 4*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1988kB
Nov 27 21:14:43 liXXX-XXX kernel: DMA32: 668*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2672kB
Nov 27 21:14:43 liXXX-XXX kernel: 9437 total pagecache pages
Nov 27 21:14:43 liXXX-XXX kernel: 2006 pages in swap cache
Nov 27 21:14:43 liXXX-XXX kernel: Swap cache stats: add 18105, delete 16099, find 96926/97556
Nov 27 21:14:43 liXXX-XXX kernel: Free swap = 212148kB
Nov 27 21:14:43 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:14:43 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:14:43 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:14:43 liXXX-XXX kernel: 148048 pages shared
Nov 27 21:14:43 liXXX-XXX kernel: 117864 pages non-shared
Nov 27 21:14:43 liXXX-XXX kernel: httpd: page allocation failure. order:1, mode:0x20
Nov 27 21:14:43 liXXX-XXX kernel: Pid: 20125, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:14:43 liXXX-XXX kernel: Call Trace:
Nov 27 21:14:43 liXXX-XXX kernel: <IRQ> [<ffffffff810921b4>] ? __alloc_pages_nodemask+0x584/0x670
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff810b9ba3>] ? cache_alloc_refill+0x2e3/0x5c0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff810ba037>] ? kmem_cache_alloc+0xa7/0xb0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8143c522>] ? sk_prot_alloc+0x42/0x130
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8143c6b4>] ? sk_clone+0x14/0x290
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814b26f0>] ? inet_csk_clone+0x10/0xb0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814c9f21>] ? tcp_create_openreq_child+0x21/0x470
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814c85f3>] ? tcp_v4_syn_recv_sock+0x53/0x1d0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81534c6d>] ? tcp_v6_syn_recv_sock+0x46d/0x580
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff815abc15>] ? _raw_spin_lock+0x5/0x10
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814c9d7e>] ? tcp_check_req+0x31e/0x4a0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814c7c27>] ? tcp_v4_do_rcv+0x1c7/0x2a0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81040a22>] ? local_bh_enable+0x22/0xb0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814c843b>] ? tcp_v4_rcv+0x73b/0x8a0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814a896b>] ? ip_local_deliver_finish+0xcb/0x200
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814a83a1>] ? ip_rcv_finish+0x191/0x3b0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff814a8816>] ? ip_rcv+0x256/0x2e0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81448a98>] ? netif_receive_skb+0x78/0x90
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff813e744d>] ? xennet_poll+0x59d/0xe20
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81046bc9>] ? run_timer_softirq+0x1b9/0x260
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81448c77>] ? net_rx_action+0x97/0x1b0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81040bee>] ? __do_softirq+0xae/0x140
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff813788ba>] ? unmask_evtchn+0x1a/0xf0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81009c7c>] ? call_softirq+0x1c/0x30
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8100ba65>] ? do_softirq+0x65/0xa0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff810408b5>] ? irq_exit+0x75/0x90
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8137912e>] ? xen_evtchn_do_upcall+0x13e/0x200
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81009cce>] ? xen_do_hypervisor_callback+0x1e/0x30
Nov 27 21:14:43 liXXX-XXX kernel: <EOI> [<ffffffff8100140a>] ? hypercall_page+0x40a/0x1010
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8100140a>] ? hypercall_page+0x40a/0x1010
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81378ab9>] ? notify_remote_via_irq+0x29/0x70
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81006c92>] ? xen_spin_unlock_slow+0x32/0x70
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff810ad262>] ? page_referenced+0x222/0x230
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81096ea7>] ? shrink_page_list+0x197/0x750
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81096c9a>] ? shrink_active_list+0x25a/0x2d0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81092f6d>] ? determine_dirtyable_memory+0xd/0x20
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81095d0c>] ? isolate_lru_pages+0x8c/0x220
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81097aab>] ? shrink_zone+0x64b/0xc80
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81098cdd>] ? try_to_free_pages+0x1dd/0x3a0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81091fe5>] ? __alloc_pages_nodemask+0x3b5/0x670
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff810a0dea>] ? __do_fault+0x3da/0x4e0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff810a389c>] ? handle_mm_fault+0x93c/0xa90
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8100777d>] ? __switch_to+0x26d/0x350
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff8102fb25>] ? set_next_entity+0x35/0x70
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81034182>] ? finish_task_switch+0x52/0xd0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff81021903>] ? pvclock_clocksource_read+0x53/0xd0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff815a9cc4>] ? schedule+0x3a4/0x6e0
Nov 27 21:14:43 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:14:43 liXXX-XXX kernel: Mem-Info:
Nov 27 21:14:43 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:14:43 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:14:43 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:14:43 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:14:43 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:14:43 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:14:43 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 116
Nov 27 21:14:43 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 132
Nov 27 21:14:43 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 65
Nov 27 21:14:43 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 132
Nov 27 21:14:43 liXXX-XXX kernel: active_anon:52284 inactive_anon:52222 isolated_anon:855
Nov 27 21:14:43 liXXX-XXX kernel: active_file:2521 inactive_file:946 isolated_file:1
Nov 27 21:14:43 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:20 unstable:0
Nov 27 21:14:43 liXXX-XXX kernel: free:1196 slab_reclaimable:2255 slab_unreclaimable:3630
Nov 27 21:14:43 liXXX-XXX kernel: mapped:4699 shmem:3919 pagetables:7329 bounce:0
Nov 27 21:14:43 liXXX-XXX kernel: DMA free:1988kB min:84kB low:104kB high:124kB active_anon:6096kB inactive_anon:6288kB active_file:64kB inactive_file:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:100kB slab_unreclaimable:92kB kernel_stack:160kB pagetables:148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 27 21:14:44 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:14:45 liXXX-XXX kernel: DMA32 free:2796kB min:2784kB low:3480kB high:4176kB active_anon:203040kB inactive_anon:202600kB active_file:10020kB inactive_file:3716kB unevictable:0kB isolated(anon):3420kB isolated(file):4kB present:500960kB mlocked:0kB dirty:0kB writeback:80kB mapped:18784kB shmem:15676kB slab_reclaimable:8920kB slab_unreclaimable:14428kB kernel_stack:1480kB pagetables:29168kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1782 all_unreclaimable? no
Nov 27 21:14:45 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:14:45 liXXX-XXX kernel: DMA: 1*4kB 0*8kB 4*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1988kB
Nov 27 21:14:45 liXXX-XXX kernel: DMA32: 689*4kB 6*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2804kB
Nov 27 21:14:45 liXXX-XXX kernel: 9381 total pagecache pages
Nov 27 21:14:45 liXXX-XXX kernel: 2036 pages in swap cache
Nov 27 21:14:45 liXXX-XXX kernel: Swap cache stats: add 18158, delete 16122, find 96946/97581
Nov 27 21:14:45 liXXX-XXX kernel: Free swap = 212092kB
Nov 27 21:14:45 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:14:45 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:14:45 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:14:46 liXXX-XXX kernel: 147885 pages shared
Nov 27 21:14:46 liXXX-XXX kernel: 117974 pages non-shared
Nov 27 21:14:46 liXXX-XXX kernel: httpd: page allocation failure. order:1, mode:0x20
Nov 27 21:14:46 liXXX-XXX kernel: Pid: 20096, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:14:46 liXXX-XXX kernel: Call Trace:
Nov 27 21:14:47 liXXX-XXX kernel: <IRQ> [<ffffffff810921b4>] ? __alloc_pages_nodemask+0x584/0x670
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff810b9ba3>] ? cache_alloc_refill+0x2e3/0x5c0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff810ba037>] ? kmem_cache_alloc+0xa7/0xb0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff8143c522>] ? sk_prot_alloc+0x42/0x130
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff8143c6b4>] ? sk_clone+0x14/0x290
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814b26f0>] ? inet_csk_clone+0x10/0xb0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814c9f21>] ? tcp_create_openreq_child+0x21/0x470
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814c85f3>] ? tcp_v4_syn_recv_sock+0x53/0x1d0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81534c6d>] ? tcp_v6_syn_recv_sock+0x46d/0x580
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff815abc15>] ? _raw_spin_lock+0x5/0x10
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814c9d7e>] ? tcp_check_req+0x31e/0x4a0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814c7c27>] ? tcp_v4_do_rcv+0x1c7/0x2a0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81040a22>] ? local_bh_enable+0x22/0xb0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814c843b>] ? tcp_v4_rcv+0x73b/0x8a0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814a896b>] ? ip_local_deliver_finish+0xcb/0x200
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814a83a1>] ? ip_rcv_finish+0x191/0x3b0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff814a8816>] ? ip_rcv+0x256/0x2e0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81448a98>] ? netif_receive_skb+0x78/0x90
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff813e744d>] ? xennet_poll+0x59d/0xe20
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81448c77>] ? net_rx_action+0x97/0x1b0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81040bee>] ? __do_softirq+0xae/0x140
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff813788ba>] ? unmask_evtchn+0x1a/0xf0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81009c7c>] ? call_softirq+0x1c/0x30
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff8100ba65>] ? do_softirq+0x65/0xa0
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff810408b5>] ? irq_exit+0x75/0x90
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff8137912e>] ? xen_evtchn_do_upcall+0x13e/0x200
Nov 27 21:14:47 liXXX-XXX kernel: [<ffffffff81009cce>] ? xen_do_hypervisor_callback+0x1e/0x30
Nov 27 21:14:47 liXXX-XXX kernel: <EOI> Mem-Info:
Nov 27 21:14:47 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:14:47 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:14:47 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:14:47 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:14:48 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:14:48 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:14:48 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 117
Nov 27 21:14:48 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 91
Nov 27 21:14:48 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 65
Nov 27 21:14:48 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 132
Nov 27 21:14:48 liXXX-XXX kernel: active_anon:52289 inactive_anon:52253 isolated_anon:855
Nov 27 21:14:48 liXXX-XXX kernel: active_file:2521 inactive_file:946 isolated_file:1
Nov 27 21:14:48 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:20 unstable:0
Nov 27 21:14:48 liXXX-XXX kernel: free:1196 slab_reclaimable:2255 slab_unreclaimable:3630
Nov 27 21:14:48 liXXX-XXX kernel: mapped:4699 shmem:3919 pagetables:7329 bounce:0
Nov 27 21:14:48 liXXX-XXX kernel: DMA free:2072kB min:84kB low:104kB high:124kB active_anon:6096kB inactive_anon:6208kB active_file:64kB inactive_file:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:100kB slab_unreclaimable:92kB kernel_stack:160kB pagetables:148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 27 21:14:48 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:14:48 liXXX-XXX kernel: DMA32 free:4788kB min:2784kB low:3480kB high:4176kB active_anon:203020kB inactive_anon:203024kB active_file:9668kB inactive_file:3644kB unevictable:0kB isolated(anon):1284kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:3552kB mapped:18224kB shmem:15452kB slab_reclaimable:8844kB slab_unreclaimable:14428kB kernel_stack:1480kB pagetables:29168kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2149 all_unreclaimable? no
Nov 27 21:14:48 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:14:48 liXXX-XXX kernel: DMA: 26*4kB 1*8kB 4*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2096kB
Nov 27 21:14:48 liXXX-XXX kernel: DMA32: 1019*4kB 81*8kB 4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4788kB
Nov 27 21:14:48 liXXX-XXX kernel: 10193 total pagecache pages
Nov 27 21:14:48 liXXX-XXX kernel: 2966 pages in swap cache
Nov 27 21:14:48 liXXX-XXX kernel: Swap cache stats: add 19097, delete 16131, find 96955/97594
Nov 27 21:14:48 liXXX-XXX kernel: Free swap = 208500kB
Nov 27 21:14:48 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:14:48 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:14:48 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:14:48 liXXX-XXX kernel: 147376 pages shared
Nov 27 21:14:48 liXXX-XXX kernel: 117343 pages non-shared
Nov 27 21:14:57 liXXX-XXX kernel: mysqld invoked oom-killer: gfp_mask=0x4d0, order=2, oom_adj=0
Nov 27 21:14:58 liXXX-XXX kernel: Pid: 20264, comm: mysqld Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:14:58 liXXX-XXX kernel: Call Trace:
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff810b9ba3>] ? cache_alloc_refill+0x2e3/0x5c0
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff810b9f8b>] ? __kmalloc+0x10b/0x110
Nov 27 21:14:58 liXXX-XXX kernel: [<ffffffff81441831>] ? __alloc_skb+0x71/0x160
Nov 27 21:15:02 liXXX-XXX kernel: [<ffffffff8143be87>] ? sock_alloc_send_pskb+0x1e7/0x300
Nov 27 21:15:03 liXXX-XXX kernel: [<ffffffff8150e819>] ? unix_stream_sendmsg+0x2b9/0x3d0
Nov 27 21:15:10 liXXX-XXX kernel: [<ffffffff81437786>] ? sock_aio_write+0x186/0x1a0
Nov 27 21:15:15 liXXX-XXX kernel: [<ffffffff810be720>] ? do_sync_write+0xc0/0x100
Nov 27 21:15:22 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:23 liXXX-XXX kernel: [<ffffffff81021903>] ? pvclock_clocksource_read+0x53/0xd0
Nov 27 21:15:23 liXXX-XXX kernel: [<ffffffff810bf3bc>] ? vfs_write+0x18c/0x1a0
Nov 27 21:15:23 liXXX-XXX kernel: [<ffffffff810bf4d3>] ? sys_write+0x53/0xa0
Nov 27 21:15:23 liXXX-XXX kernel: [<ffffffff81008d82>] ? system_call_fastpath+0x16/0x1b
Nov 27 21:15:23 liXXX-XXX kernel: Mem-Info:
Nov 27 21:15:23 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:15:23 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:15:23 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 0
Nov 27 21:15:23 liXXX-XXX kernel: active_anon:48131 inactive_anon:48213 isolated_anon:168
Nov 27 21:15:23 liXXX-XXX kernel: active_file:202 inactive_file:540 isolated_file:25
Nov 27 21:15:23 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Nov 27 21:15:23 liXXX-XXX kernel: free:1619 slab_reclaimable:1643 slab_unreclaimable:5785
Nov 27 21:15:23 liXXX-XXX kernel: mapped:2608 shmem:2532 pagetables:16779 bounce:0
Nov 27 21:15:23 liXXX-XXX kernel: DMA free:2008kB min:84kB low:104kB high:124kB active_anon:5732kB inactive_anon:5908kB active_file:20kB inactive_file:112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:76kB shmem:20kB slab_reclaimable:68kB slab_unreclaimable:300kB kernel_stack:176kB pagetables:604kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:15:23 liXXX-XXX kernel: DMA32 free:4468kB min:2784kB low:3480kB high:4176kB active_anon:186792kB inactive_anon:186944kB active_file:788kB inactive_file:2048kB unevictable:0kB isolated(anon):672kB isolated(file):100kB present:500960kB mlocked:0kB dirty:0kB writeback:0kB mapped:10356kB shmem:10108kB slab_reclaimable:6504kB slab_unreclaimable:22840kB kernel_stack:2040kB pagetables:66512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:15:23 liXXX-XXX kernel: DMA: 2*4kB 4*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2008kB
Nov 27 21:15:23 liXXX-XXX kernel: DMA32: 847*4kB 119*8kB 6*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4468kB
Nov 27 21:15:23 liXXX-XXX kernel: 12398 total pagecache pages
Nov 27 21:15:23 liXXX-XXX kernel: 9101 pages in swap cache
Nov 27 21:15:23 liXXX-XXX kernel: Swap cache stats: add 151688, delete 142587, find 138910/147611
Nov 27 21:15:23 liXXX-XXX kernel: Free swap = 0kB
Nov 27 21:15:23 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:15:23 liXXX-XXX kernel: ctive_anon:5780kB active_file:16kB inactive_file:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:52kB mapped:24kB shmem:20kB slab_reclaimable:60kB slab_unreclaimable:336kB kernel_stack:176kB pagetables:588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? yes
Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:15:23 liXXX-XXX kernel: DMA32 free:2736kB min:2784kB low:3480kB high:4176kB active_anon:187752kB inactive_anon:187952kB active_file:1060kB inactive_file:2256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:188kB mapped:10668kB shmem:10084kB slab_reclaimable:5736kB slab_unreclaimable:22264kB kernel_stack:2024kB pagetables:64824kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5088 all_unreclaimable? yes
Nov 27 21:15:23 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:15:23 liXXX-XXX kernel: DMA: 0*4kB 10*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2016kB
Nov 27 21:15:23 liXXX-XXX kernel: DMA32: 606*4kB 35*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2736kB
Nov 27 21:15:23 liXXX-XXX kernel: 11583 total pagecache pages
Nov 27 21:15:23 liXXX-XXX kernel: 8108 pages in swap cache
Nov 27 21:15:23 liXXX-XXX kernel: Swap cache stats: add 169548, delete 161440, find 142115/152306
Nov 27 21:15:24 liXXX-XXX kernel: Free swap = 12kB
Nov 27 21:15:25 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:15:35 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:15:41 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:15:41 liXXX-XXX kernel: 165215 pages shared
Nov 27 21:15:41 liXXX-XXX kernel: 120616 pages non-shared
Nov 27 21:15:41 liXXX-XXX kernel: Out of memory: kill process 20075 (httpd) score 85070 or a child
Nov 27 21:15:41 liXXX-XXX kernel: Killed process 20075 (httpd) vsz:340280kB, anon-rss:8176kB, file-rss:7444kB
Nov 27 21:15:41 liXXX-XXX kernel: httpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Nov 27 21:15:41 liXXX-XXX kernel: Pid: 20228, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:15:41 liXXX-XXX kernel: Call Trace:
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff810a389c>] ? handle_mm_fault+0x93c/0xa90
Nov 27 21:15:41 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:42 liXXX-XXX kernel: [<ffffffff810038ec>] ? xen_mc_flush+0x10c/0x1d0
Nov 27 21:15:47 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:15:49 liXXX-XXX kernel: [<ffffffff81002806>] ? xen_write_msr_safe+0x76/0xa0
Nov 27 21:15:50 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:51 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108b3fc>] ? file_read_actor+0x6c/0x190
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108d6c7>] ? generic_file_aio_read+0x4d7/0x6b0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff810be820>] ? do_sync_read+0xc0/0x100
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff810a9ca5>] ? mmap_region+0x225/0x530
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff810bf5e8>] ? vfs_read+0xc8/0x1a0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff810bf7c3>] ? sys_read+0x53/0xa0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81008d82>] ? system_call_fastpath+0x16/0x1b
Nov 27 21:15:53 liXXX-XXX kernel: Mem-Info:
Nov 27 21:15:53 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 25
Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 30
Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 2
Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 77
Nov 27 21:15:53 liXXX-XXX kernel: active_anon:48649 inactive_anon:48757 isolated_anon:0
Nov 27 21:15:53 liXXX-XXX kernel: active_file:338 inactive_file:599 isolated_file:0
Nov 27 21:15:53 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:86 unstable:0
Nov 27 21:15:53 liXXX-XXX kernel: free:1192 slab_reclaimable:1428 slab_unreclaimable:5645
Nov 27 21:15:53 liXXX-XXX kernel: mapped:2422 shmem:2519 pagetables:16218 bounce:0
Nov 27 21:15:53 liXXX-XXX kernel: DMA free:2024kB min:84kB low:104kB high:124kB active_anon:5720kB inactive_anon:5828kB active_file:0kB inactive_file:104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:76kB shmem:16kB slab_reclaimable:60kB slab_unreclaimable:344kB kernel_stack:176kB pagetables:588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7425 all_unreclaimable? yes
Nov 27 21:15:53 liXXX-XXX kernel: l4>Mem-Info:
Nov 27 21:15:53 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 40
Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 18
Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 90
Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 178
Nov 27 21:15:53 liXXX-XXX kernel: active_anon:49216 inactive_anon:49275 isolated_anon:0
Nov 27 21:15:53 liXXX-XXX kernel: active_file:397 inactive_file:585 isolated_file:0
Nov 27 21:15:53 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:123 unstable:0
Nov 27 21:15:53 liXXX-XXX kernel: free:1203 slab_reclaimable:1305 slab_unreclaimable:5409
Nov 27 21:15:53 liXXX-XXX kernel: mapped:1876 shmem:2469 pagetables:15350 bounce:0
Nov 27 21:15:53 liXXX-XXX kernel: DMA free:2064kB min:84kB low:104kB high:124kB active_anon:5696kB inactive_anon:5584kB active_file:148kB inactive_file:212kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:88kB mapped:36kB shmem:8kB slab_reclaimable:52kB slab_unreclaimable:312kB kernel_stack:184kB pagetables:580kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:15:53 liXXX-XXX kernel: DMA32 free:2748kB min:2784kB low:3480kB high:4176kB active_anon:191168kB inactive_anon:191516kB active_file:1440kB inactive_file:2128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:404kB mapped:7468kB shmem:9868kB slab_reclaimable:5168kB slab_unreclaimable:21324kB kernel_stack:1768kB pagetables:60820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4814 all_unreclaimable? yes
Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:15:53 liXXX-XXX kernel: DMA: 15*4kB 6*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2076kB
Nov 27 21:15:53 liXXX-XXX kernel: DMA32: 687*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2748kB
Nov 27 21:15:53 liXXX-XXX kernel: 10788 total pagecache pages
Nov 27 21:15:53 liXXX-XXX kernel: 7347 pages in swap cache
Nov 27 21:15:53 liXXX-XXX kernel: Swap cache stats: add 203670, delete 196323, find 144761/157218
Nov 27 21:15:53 liXXX-XXX kernel: Free swap = 0kB
Nov 27 21:15:53 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:15:53 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:15:53 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:15:53 liXXX-XXX kernel: 115797 pages shared
Nov 27 21:15:53 liXXX-XXX kernel: 121701 pages non-shared
Nov 27 21:15:53 liXXX-XXX kernel: Out of memory: kill process 20140 (httpd) score 84271 or a child
Nov 27 21:15:53 liXXX-XXX kernel: Killed process 20140 (httpd) vsz:337084kB, anon-rss:3676kB, file-rss:4212kB
Nov 27 21:15:53 liXXX-XXX kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Nov 27 21:15:53 liXXX-XXX kernel: Pid: 20233, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:15:53 liXXX-XXX kernel: Call Trace:
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81093d8c>] ? __do_page_cache_readahead+0xdc/0x230
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81052a80>] ? wake_bit_function+0x0/0x30
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81093efc>] ? ra_submit+0x1c/0x30
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8108c74b>] ? filemap_fault+0x3db/0x3e0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81004b2e>] ? xen_set_pte_at+0x6e/0x100
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff810a0a65>] ? __do_fault+0x55/0x4e0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff810a3194>] ? handle_mm_fault+0x234/0xa90
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff815ac57a>] ? error_exit+0x2a/0x60
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff815ac0dd>] ? retint_restore_args+0x5/0x6
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:53 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:15:53 liXXX-XXX kernel: Mem-Info:
Nov 27 21:15:53 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:15:53 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:15:53 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 40
Nov 27 21:15:53 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 18
Nov 27 21:15:53 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 90
Nov 27 21:15:53 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 178
Nov 27 21:15:53 liXXX-XXX kernel: active_anon:49166 inactive_anon:49332 isolated_anon:0
Nov 27 21:15:53 liXXX-XXX kernel: active_file:397 inactive_file:604 isolated_file:0
Nov 27 21:15:53 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:123 unstable:0
Nov 27 21:15:53 liXXX-XXX kernel: free:1196 slab_reclaimable:1305 slab_unreclaimable:5409
Nov 27 21:15:53 liXXX-XXX kernel: mapped:1904 shmem:2469 pagetables:15350 bounce:0
Nov 27 21:15:53 liXXX-XXX kernel: DMA free:2036kB min:84kB low:104kB high:124kB active_anon:5496kB inactive_anon:5812kB active_file:148kB inactive_file:288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:88kB mapped:36kB shmem:8kB slab_reclaimable:52kB slab_unreclaimable:312kB kernel_stack:184kB pagetables:580kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:693 all_unreclaimable? yes
Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:15:53 liXXX-XXX kernel: DMA32 free:2748kB min:2784kB low:3480kB high:4176kB active_anon:191168kB inactive_anon:191516kB active_file:1440kB inactive_file:2240kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:404kB mapped:7580kB shmem:9868kB slab_reclaimable:5168kB slab_unreclaimable:21324kB kernel_stack:1768kB pagetables:60820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4814 all_unreclaimable? yes
Nov 27 21:15:53 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:15:53 liXXX-XXX kernel: DMA: 0*4kB 4*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
Nov 27 21:15:53 liXXX-XXX kernel: DMA32: 687*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2748kB
Nov 27 21:15:53 liXXX-XXX kernel: 10864 total pagecache pages
Nov 27 21:15:53 liXXX-XXX kernel: 7334 pages in swap cache
Nov 27 21:15:53 liXXX-XXX kernel: Swap cache stats: add 203674, delete 196340, find 144768/157239
Nov 27 21:15:54 liXXX-XXX kernel: Free swap = 52kB
Nov 27 21:15:54 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:15:54 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:15:54 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:15:54 liXXX-XXX kernel: 114524 pages shared
Nov 27 21:15:54 liXXX-XXX kernel: 121186 pages non-shared
Nov 27 21:15:54 liXXX-XXX kernel: Out of memory: kill process 20141 (httpd) score 84271 or a child
Nov 27 21:15:54 liXXX-XXX kernel: Killed process 20141 (httpd) vsz:337084kB, anon-rss:3904kB, file-rss:4420kB
Nov 27 21:15:54 liXXX-XXX kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Nov 27 21:15:54 liXXX-XXX kernel: Pid: 4346, comm: mysqld Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:15:54 liXXX-XXX kernel: Call Trace:
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81093d8c>] ? __do_page_cache_readahead+0xdc/0x230
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81093efc>] ? ra_submit+0x1c/0x30
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108c74b>] ? filemap_fault+0x3db/0x3e0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81004b2e>] ? xen_set_pte_at+0x6e/0x100
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff810a0a65>] ? __do_fault+0x55/0x4e0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81311f10>] ? cap_socket_accept+0x0/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff810a3194>] ? handle_mm_fault+0x234/0xa90
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81311f10>] ? cap_socket_accept+0x0/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff815abc15>] ? _raw_spin_lock+0x5/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81439254>] ? move_addr_to_user+0xa4/0xb0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81439bf5>] ? sys_getsockname+0xa5/0x100
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:15:54 liXXX-XXX kernel: Mem-Info:
Nov 27 21:15:54 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 31
Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 177
Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 23
Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 102
Nov 27 21:15:54 liXXX-XXX kernel: active_anon:49006 inactive_anon:49384 isolated_anon:0
Nov 27 21:15:54 liXXX-XXX kernel: active_file:383 inactive_file:1254 isolated_file:0
Nov 27 21:15:54 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:32 unstable:0
Nov 27 21:15:54 liXXX-XXX kernel: free:1176 slab_reclaimable:1299 slab_unreclaimable:5317
Nov 27 21:15:54 liXXX-XXX kernel: mapped:1975 shmem:2434 pagetables:14978 bounce:0
Nov 27 21:15:54 liXXX-XXX kernel: DMA free:2032kB min:84kB low:104kB high:124kB active_anon:4680kB inactive_anon:4964kB active_file:60kB inactive_file:1656kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:104kB mapped:52kB shmem:8kB slab_reclaimable:52kB slab_unreclaimable:312kB kernel_stack:176kB pagetables:500kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5600 all_unreclaimable? yes
Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:15:54 liXXX-XXX kernel: DMA32 free:2672kB min:2784kB low:3480kB high:4176kB active_anon:191344kB inactive_anon:192572kB active_file:1472kB inactive_file:3360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:24kB mapped:7848kB shmem:9728kB slab_reclaimable:5144kB slab_unreclaimable:20956kB kernel_stack:1736kB pagetables:59412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32613 all_unreclaimable? yes
Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:15:54 liXXX-XXX kernel: DMA: 6*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
Nov 27 21:15:54 liXXX-XXX kernel: DMA32: 662*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2672kB
Nov 27 21:15:54 liXXX-XXX kernel: 14343 total pagecache pages
Nov 27 21:15:54 liXXX-XXX kernel: 10243 pages in swap cache
Nov 27 21:15:54 liXXX-XXX kernel: Swap cache stats: add 219708, delete 209465, find 146269/160080
Nov 27 21:15:54 liXXX-XXX kernel: Free swap = 20kB
Nov 27 21:15:54 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:15:54 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:15:54 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:15:54 liXXX-XXX kernel: 109516 pages shared
Nov 27 21:15:54 liXXX-XXX kernel: 121651 pages non-shared
Nov 27 21:15:54 liXXX-XXX kernel: Out of memory: kill process 19955 (httpd) score 84242 or a child
Nov 27 21:15:54 liXXX-XXX kernel: Killed process 19955 (httpd) vsz:336968kB, anon-rss:3804kB, file-rss:6308kB
Nov 27 21:15:54 liXXX-XXX kernel: httpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Nov 27 21:15:54 liXXX-XXX kernel: Pid: 20208, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:15:54 liXXX-XXX kernel: Call Trace:
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81026d91>] ? ptep_set_access_flags+0x51/0x80
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff810b18ac>] ? read_swap_cache_async+0xfc/0x150
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff810b197f>] ? swapin_readahead+0x7f/0xb0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff810a370b>] ? handle_mm_fault+0x7ab/0xa90
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81311bd0>] ? cap_file_lock+0x0/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff81002dfb>] ? xen_clts+0x6b/0x80
Nov 27 21:15:54 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:15:54 liXXX-XXX kernel: Mem-Info:
Nov 27 21:15:54 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:15:54 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:15:54 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 29
Nov 27 21:15:54 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 177
Nov 27 21:15:54 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 23
Nov 27 21:15:54 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 102
Nov 27 21:15:54 liXXX-XXX kernel: active_anon:49006 inactive_anon:49468 isolated_anon:0
Nov 27 21:15:54 liXXX-XXX kernel: active_file:383 inactive_file:1314 isolated_file:0
Nov 27 21:15:54 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:2 unstable:0
Nov 27 21:15:54 liXXX-XXX kernel: free:1083 slab_reclaimable:1299 slab_unreclaimable:5317
Nov 27 21:15:54 liXXX-XXX kernel: mapped:2015 shmem:2434 pagetables:14978 bounce:0
Nov 27 21:15:54 liXXX-XXX kernel: DMA free:2032kB min:84kB low:104kB high:124kB active_anon:4680kB inactive_anon:4964kB active_file:60kB inactive_file:1896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:212kB shmem:8kB slab_reclaimable:52kB slab_unreclaimable:312kB kernel_stack:176kB pagetables:500kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5600 all_unreclaimable? yes
Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:15:54 liXXX-XXX kernel: DMA32 free:2300kB min:2784kB low:3480kB high:4176kB active_anon:191344kB inactive_anon:192908kB active_file:1472kB inactive_file:3360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:24kB mapped:7848kB shmem:9728kB slab_reclaimable:5144kB slab_unreclaimable:20956kB kernel_stack:1736kB pagetables:59412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32613 all_unreclaimable? yes
Nov 27 21:15:54 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:15:54 liXXX-XXX kernel: DMA: 6*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
Nov 27 21:15:54 liXXX-XXX kernel: DMA32: 575*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2300kB
Nov 27 21:15:54 liXXX-XXX kernel: 14449 total pagecache pages
Nov 27 21:15:54 liXXX-XXX kernel: 10288 pages in swap cache
Nov 27 21:15:54 liXXX-XXX kernel: Swap cache stats: add 219803, delete 209515, find 146301/160136
Nov 27 21:15:54 liXXX-XXX kernel: Free swap = 220kB
Nov 27 21:15:57 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:16:01 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:16:03 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:16:03 liXXX-XXX kernel: 109633 pages shared
Nov 27 21:16:06 liXXX-XXX kernel: 121706 pages non-shared
Nov 27 21:16:06 liXXX-XXX kernel: Out of memory: kill process 19979 (httpd) score 84242 or a child
Nov 27 21:16:06 liXXX-XXX kernel: Killed process 19979 (httpd) vsz:336968kB, anon-rss:4212kB, file-rss:6444kB
Nov 27 21:16:06 liXXX-XXX kernel: httpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Nov 27 21:16:06 liXXX-XXX kernel: Pid: 20208, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:16:06 liXXX-XXX kernel: Call Trace:
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81026d91>] ? ptep_set_access_flags+0x51/0x80
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff810b18ac>] ? read_swap_cache_async+0xfc/0x150
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff810b197f>] ? swapin_readahead+0x7f/0xb0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff810a370b>] ? handle_mm_fault+0x7ab/0xa90
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81311bd0>] ? cap_file_lock+0x0/0x10
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81002dfb>] ? xen_clts+0x6b/0x80
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:16:06 liXXX-XXX kernel: Mem-Info:
Nov 27 21:16:06 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 155
Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 177
Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 23
Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 70
Nov 27 21:16:06 liXXX-XXX kernel: active_anon:48978 inactive_anon:49384 isolated_anon:0
Nov 27 21:16:06 liXXX-XXX kernel: active_file:383 inactive_file:1342 isolated_file:0
Nov 27 21:16:06 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:2 unstable:0
Nov 27 21:16:06 liXXX-XXX kernel: free:1083 slab_reclaimable:1299 slab_unreclaimable:5317
Nov 27 21:16:06 liXXX-XXX kernel: mapped:2015 shmem:2434 pagetables:14978 bounce:0
Nov 27 21:16:06 liXXX-XXX kernel: DMA free:2032kB min:84kB low:104kB high:124kB active_anon:4680kB inactive_anon:4964kB active_file:60kB inactive_file:1896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:0kB mapped:212kB shmem:8kB slab_reclaimable:52kB slab_unreclaimable:312kB kernel_stack:176kB pagetables:500kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5600 all_unreclaimable? yes
Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:16:06 liXXX-XXX kernel: DMA32 free:2300kB min:2784kB low:3480kB high:4176kB active_anon:191232kB inactive_anon:192460kB active_file:1472kB inactive_file:3472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB mlocked:0kB dirty:0kB writeback:24kB mapped:7848kB shmem:9728kB slab_reclaimable:5144kB slab_unreclaimable:20956kB kernel_stack:1736kB pagetables:59412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32613 all_unreclaimable? yes
Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:16:06 liXXX-XXX kernel: DMA: 6*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
Nov 27 21:16:06 liXXX-XXX kernel: DMA32: 668*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2672kB
Nov 27 21:16:06 liXXX-XXX kernel: 14365 total pagecache pages
Nov 27 21:16:06 liXXX-XXX kernel: 10184 pages in swap cache
Nov 27 21:16:06 liXXX-XXX kernel: Swap cache stats: add 219803, delete 209619, find 146301/160136
Nov 27 21:16:06 liXXX-XXX kernel: Free swap = 1796kB
Nov 27 21:16:06 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:16:06 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:16:06 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:16:06 liXXX-XXX kernel: 107825 pages shared
Nov 27 21:16:06 liXXX-XXX kernel: 121158 pages non-shared
Nov 27 21:16:06 liXXX-XXX kernel: Out of memory: kill process 20032 (httpd) score 84238 or a child
Nov 27 21:16:06 liXXX-XXX kernel: Killed process 20032 (httpd) vsz:336952kB, anon-rss:6360kB, file-rss:6444kB
Nov 27 21:16:06 liXXX-XXX kernel: gam_server invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Nov 27 21:16:06 liXXX-XXX kernel: Pid: 8449, comm: gam_server Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:16:06 liXXX-XXX kernel: Call Trace:
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81093d8c>] ? __do_page_cache_readahead+0xdc/0x230
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81093efc>] ? ra_submit+0x1c/0x30
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8108c74b>] ? filemap_fault+0x3db/0x3e0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff810a0a65>] ? __do_fault+0x55/0x4e0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff815ac57a>] ? error_exit+0x2a/0x60
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81009cab>] ? xen_hypervisor_callback+0x1b/0x20
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff810a3194>] ? handle_mm_fault+0x234/0xa90
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81311b80>] ? cap_file_permission+0x0/0x10
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff81021903>] ? pvclock_clocksource_read+0x53/0xd0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff8105b3a9>] ? ktime_get_ts+0x69/0xd0
Nov 27 21:16:06 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:16:06 liXXX-XXX kernel: Mem-Info:
Nov 27 21:16:06 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 27 21:16:06 liXXX-XXX kernel: DMA32 per-cpu:
Nov 27 21:16:06 liXXX-XXX kernel: CPU 0: hi: 186, btch: 31 usd: 45
Nov 27 21:16:06 liXXX-XXX kernel: CPU 1: hi: 186, btch: 31 usd: 76
Nov 27 21:16:06 liXXX-XXX kernel: CPU 2: hi: 186, btch: 31 usd: 95
Nov 27 21:16:06 liXXX-XXX kernel: CPU 3: hi: 186, btch: 31 usd: 134
Nov 27 21:16:06 liXXX-XXX kernel: active_anon:48624 inactive_anon:49273 isolated_anon:798
Nov 27 21:16:06 liXXX-XXX kernel: active_file:311 inactive_file:1743 isolated_file:2
Nov 27 21:16:06 liXXX-XXX kernel: unevictable:0 dirty:0 writeback:176 unstable:0
Nov 27 21:16:06 liXXX-XXX kernel: free:1183 slab_reclaimable:1279 slab_unreclaimable:5273
Nov 27 21:16:06 liXXX-XXX kernel: mapped:2132 shmem:2389 pagetables:14434 bounce:0
Nov 27 21:16:06 liXXX-XXX kernel: DMA free:2020kB min:84kB low:104kB high:124kB active_anon:4160kB inactive_anon:4392kB active_file:0kB inactive_file:48kB unevictable:0kB isolated(anon):3072kB isolated(file):0kB present:15772kB mlocked:0kB dirty:0kB writeback:700kB mapped:8kB shmem:8kB slab_reclaimable:52kB slab_unreclaimable:312kB kernel_stack:168kB pagetables:516kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14 all_unreclaimable? yes
Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 489 489 489
Nov 27 21:16:06 liXXX-XXX kernel: DMA32 free:2712kB min:2784kB low:3480kB high:4176kB active_anon:190336kB inactive_anon:192700kB active_file:1244kB inactive_file:6924kB unevictable:0kB isolated(anon):120kB isolated(file):8kB present:500960kB mlocked:0kB dirty:0kB writeback:4kB mapped:8520kB shmem:9548kB slab_reclaimable:5064kB slab_unreclaimable:20780kB kernel_stack:1704kB pagetables:57220kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6178 all_unreclaimable? yes
Nov 27 21:16:06 liXXX-XXX kernel: lowmem_reserve[]: 0 0 0 0
Nov 27 21:16:06 liXXX-XXX kernel: DMA: 1*4kB 2*8kB 5*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2020kB
Nov 27 21:16:06 liXXX-XXX kernel: DMA32: 678*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2712kB
Nov 27 21:16:06 liXXX-XXX kernel: 19692 total pagecache pages
Nov 27 21:16:06 liXXX-XXX kernel: 15245 pages in swap cache
Nov 27 21:16:06 liXXX-XXX kernel: Swap cache stats: add 237334, delete 222089, find 148153/163593
Nov 27 21:16:06 liXXX-XXX kernel: Free swap = 0kB
Nov 27 21:16:06 liXXX-XXX kernel: Total swap = 262140kB
Nov 27 21:16:06 liXXX-XXX kernel: 131071 pages RAM
Nov 27 21:16:06 liXXX-XXX kernel: 4842 pages reserved
Nov 27 21:16:06 liXXX-XXX kernel: 105720 pages shared
Nov 27 21:16:06 liXXX-XXX kernel: 120659 pages non-shared
Nov 27 21:16:06 liXXX-XXX kernel: Out of memory: kill process 20051 (httpd) score 84238 or a child
Nov 27 21:16:06 liXXX-XXX kernel: Killed process 20051 (httpd) vsz:336952kB, anon-rss:6376kB, file-rss:6412kB
Nov 27 21:16:52 liXXX-XXX kernel: httpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Nov 27 21:16:52 liXXX-XXX kernel: Pid: 20199, comm: httpd Not tainted 2.6.35.4-x86_64-linode16 #1
Nov 27 21:16:52 liXXX-XXX kernel: Call Trace:
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff8108e182>] ? dump_header+0x72/0x1b0
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff8100643f>] ? xen_restore_fl_direct_end+0x0/0x1
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff815abca6>] ? _raw_spin_unlock_irqrestore+0x26/0x40
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff8108e351>] ? oom_kill_process+0x91/0x170
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff8108e99d>] ? __out_of_memory+0x11d/0x190
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff8108ea6e>] ? out_of_memory+0x5e/0xe0
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff81092283>] ? __alloc_pages_nodemask+0x653/0x670
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff810a389c>] ? handle_mm_fault+0x93c/0xa90
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff81005da9>] ? xen_force_evtchn_callback+0x9/0x10
Nov 27 21:16:52 liXXX-XXX kernel: [<ffffffff81006452>] ? check_events+0x12/0x20
Nov 27 21:17:02 liXXX-XXX kernel: [<ffffffff81311ba0>] ? cap_file_free_security+0x0/0x10
Nov 27 21:17:07 liXXX-XXX kernel: [<ffffffff8102354f>] ? do_page_fault+0x1af/0x3c0
Nov 27 21:17:07 liXXX-XXX kernel: [<ffffffff810a932b>] ? vma_merge+0x1eb/0x2e0
Nov 27 21:17:11 liXXX-XXX kernel: [<ffffffff810a9863>] ? do_brk+0x253/0x340
Nov 27 21:17:13 liXXX-XXX kernel: [<ffffffff810d18ac>] ? dput+0xbc/0x180
Nov 27 21:17:15 liXXX-XXX kernel: [<ffffffff815ac375>] ? page_fault+0x25/0x30
Nov 27 21:17:15 liXXX-XXX kernel: Mem-Info:
Nov 27 21:17:15 liXXX-XXX kernel: DMA per-cpu:
Nov 27 21:17:15 liXXX-XXX kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 27 21:17:15 liXXX-XXX kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 27 21:17:15 liXX
|