|
I am getting the error below from my log watch files... Any idea what they mean?
WARNING: Kernel Errors Present
[<c05f930e>] ? error_code+0x66/0x6c ...: 5 Time(s)
9 Time(s): 0 pages HighMem
9 Time(s): 0 pages in swap cache
1 Time(s): 18226 pages shared
1 Time(s): 19813 pages shared
1 Time(s): 19877 pages shared
1 Time(s): 20036 pages shared
1 Time(s): 20064 pages shared
1 Time(s): 20190 pages shared
1 Time(s): 20220 pages shared
1 Time(s): 20251 pages shared
1 Time(s): 20255 pages shared
9 Time(s): 2989 pages reserved
2 Time(s): 7829 total pagecache pages
1 Time(s): 7851 total pagecache pages
1 Time(s): 7905 total pagecache pages
1 Time(s): 7907 total pagecache pages
1 Time(s): 7944 total pagecache pages
1 Time(s): 8062 total pagecache pages
1 Time(s): 8144 total pagecache pages
1 Time(s): 83358 pages non-shared
1 Time(s): 83960 pages non-shared
1 Time(s): 84086 pages non-shared
1 Time(s): 8485 total pagecache pages
1 Time(s): 84959 pages non-shared
1 Time(s): 85415 pages non-shared
1 Time(s): 85642 pages non-shared
1 Time(s): 85707 pages non-shared
1 Time(s): 85739 pages non-shared
1 Time(s): 85775 pages non-shared
9 Time(s): 92160 pages RAM
9 Time(s): CPU 0: hi: 0, btch: 1 usd: 0
3 Time(s): CPU 0: hi: 90, btch: 15 usd: 0
1 Time(s): CPU 0: hi: 90, btch: 15 usd: 14
1 Time(s): CPU 0: hi: 90, btch: 15 usd: 17
1 Time(s): CPU 0: hi: 90, btch: 15 usd: 33
1 Time(s): CPU 0: hi: 90, btch: 15 usd: 51
1 Time(s): CPU 0: hi: 90, btch: 15 usd: 79
1 Time(s): CPU 0: hi: 90, btch: 15 usd: 86
9 Time(s): CPU 1: hi: 0, btch: 1 usd: 0
6 Time(s): CPU 1: hi: 90, btch: 15 usd: 0
1 Time(s): CPU 1: hi: 90, btch: 15 usd: 65
2 Time(s): CPU 1: hi: 90, btch: 15 usd: 83
9 Time(s): CPU 2: hi: 0, btch: 1 usd: 0
8 Time(s): CPU 2: hi: 90, btch: 15 usd: 0
1 Time(s): CPU 2: hi: 90, btch: 15 usd: 69
9 Time(s): CPU 3: hi: 0, btch: 1 usd: 0
8 Time(s): CPU 3: hi: 90, btch: 15 usd: 0
1 Time(s): CPU 3: hi: 90, btch: 15 usd: 58
9 Time(s): Call Trace:
1 Time(s): DMA free:1452kB min:104kB low:128kB high:156kB active_anon:3216kB inactive_anon:3088kB active_file:40kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:64kB slab_unreclaimable:12kB kernel_stack:32kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
1 Time(s): DMA free:1456kB min:104kB low:128kB high:156kB active_anon:3084kB inactive_anon:3316kB active_file:4kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:28kB slab_unreclaimable:12kB kernel_stack:8kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
1 Time(s): DMA free:1456kB min:104kB low:128kB high:156kB active_anon:3164kB inactive_anon:3088kB active_file:8kB inactive_file:116kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:0kB slab_reclaimable:44kB slab_unreclaimable:16kB kernel_stack:24kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2 Time(s): DMA free:1464kB min:104kB low:128kB high:156kB active_anon:3140kB inactive_anon:3172kB active_file:24kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:28kB slab_unreclaimable:12kB kernel_stack:32kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
1 Time(s): DMA free:1468kB min:104kB low:128kB high:156kB active_anon:2920kB inactive_anon:3240kB active_file:0kB inactive_file:200kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:32kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
1 Time(s): DMA free:1508kB min:104kB low:128kB high:156kB active_anon:3012kB inactive_anon:3236kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:12kB kernel_stack:8kB pagetables:100kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2 Time(s): DMA free:1644kB min:104kB low:128kB high:156kB active_anon:2972kB inactive_anon:3172kB active_file:16kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:28kB slab_unreclaimable:16kB kernel_stack:72kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
9 Time(s): DMA per-cpu:
2 Time(s): DMA: 362*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1464kB
1 Time(s): DMA: 363*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1452kB
1 Time(s): DMA: 363*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1468kB
2 Time(s): DMA: 364*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1456kB
1 Time(s): DMA: 365*4kB 6*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1508kB
2 Time(s): DMA: 399*4kB 6*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1644kB
9 Time(s): Free swap = 0kB
1 Time(s): Killed process 11485 (apache2)
1 Time(s): Killed process 12216 (apache2)
1 Time(s): Killed process 12329 (apache2)
1 Time(s): Killed process 12802 (apache2)
1 Time(s): Killed process 12821 (python)
1 Time(s): Killed process 13055 (apache2)
1 Time(s): Killed process 14200 (apache2)
1 Time(s): Killed process 14366 (apache2)
9 Time(s): Mem-Info:
1 Time(s): Normal free:2204kB min:2308kB low:2884kB high:3460kB active_anon:166036kB inactive_anon:166152kB active_file:28kB inactive_file:124kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:128kB shmem:31108kB slab_reclaimable:1712kB slab_unreclaimable:5360kB kernel_stack:776kB pagetables:1888kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
1 Time(s): Normal free:2276kB min:2308kB low:2884kB high:3460kB active_anon:165908kB inactive_anon:166004kB active_file:112kB inactive_file:1144kB unevictable:0kB isolated(anon):4kB isolated(file):120kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:308kB shmem:31108kB slab_reclaimable:1668kB slab_unreclaimable:5416kB kernel_stack:784kB pagetables:1812kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:432 all_unreclaimable? yes
1 Time(s): Normal free:2284kB min:2308kB low:2884kB high:3460kB active_anon:166128kB inactive_anon:166128kB active_file:64kB inactive_file:472kB unevictable:0kB isolated(anon):12kB isolated(file):116kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:116kB shmem:31108kB slab_reclaimable:1868kB slab_unreclaimable:5360kB kernel_stack:760kB pagetables:1824kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:321 all_unreclaimable? yes
1 Time(s): Normal free:2324kB min:2308kB low:2884kB high:3460kB active_anon:166036kB inactive_anon:166152kB active_file:140kB inactive_file:236kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:128kB shmem:31108kB slab_reclaimable:1712kB slab_unreclaimable:5360kB kernel_stack:776kB pagetables:1888kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
1 Time(s): Normal free:2668kB min:2308kB low:2884kB high:3460kB active_anon:165028kB inactive_anon:165820kB active_file:64kB inactive_file:56kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:204kB shmem:31108kB slab_reclaimable:1848kB slab_unreclaimable:5436kB kernel_stack:768kB pagetables:1824kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
1 Time(s): Normal free:5400kB min:2308kB low:2884kB high:3460kB active_anon:163116kB inactive_anon:164348kB active_file:128kB inactive_file:2580kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:616kB shmem:31108kB slab_reclaimable:1776kB slab_unreclaimable:5372kB kernel_stack:784kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
1 Time(s): Normal free:8936kB min:2308kB low:2884kB high:3460kB active_anon:160236kB inactive_anon:166036kB active_file:0kB inactive_file:124kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:184kB shmem:31108kB slab_reclaimable:1692kB slab_unreclaimable:5400kB kernel_stack:776kB pagetables:1920kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
1 Time(s): Normal free:9360kB min:2308kB low:2884kB high:3460kB active_anon:161744kB inactive_anon:163080kB active_file:112kB inactive_file:904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:164kB shmem:31108kB slab_reclaimable:1680kB slab_unreclaimable:5452kB kernel_stack:832kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
1 Time(s): Normal free:9552kB min:2308kB low:2884kB high:3460kB active_anon:159116kB inactive_anon:166036kB active_file:60kB inactive_file:4kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:349504kB mlocked:0kB dirty:0kB writeback:0kB mapped:184kB shmem:31108kB slab_reclaimable:1692kB slab_unreclaimable:5400kB kernel_stack:776kB pagetables:1920kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
9 Time(s): Normal per-cpu:
1 Time(s): Normal: 1150*4kB 33*8kB 30*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5344kB
1 Time(s): Normal: 2068*4kB 79*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8936kB
1 Time(s): Normal: 2076*4kB 135*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9416kB
1 Time(s): Normal: 2193*4kB 108*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9668kB
2 Time(s): Normal: 558*4kB 2*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2264kB
1 Time(s): Normal: 566*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2280kB
1 Time(s): Normal: 581*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2340kB
1 Time(s): Normal: 685*4kB 4*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2788kB
1 Time(s): Out of memory: kill process 11485 (apache2) score 25572 or a child
1 Time(s): Out of memory: kill process 12216 (apache2) score 25641 or a child
1 Time(s): Out of memory: kill process 12329 (apache2) score 25574 or a child
1 Time(s): Out of memory: kill process 12800 (apache2) score 25791 or a child
1 Time(s): Out of memory: kill process 12802 (apache2) score 25699 or a child
1 Time(s): Out of memory: kill process 13055 (apache2) score 25315 or a child
1 Time(s): Out of memory: kill process 14200 (apache2) score 25704 or a child
1 Time(s): Out of memory: kill process 14366 (apache2) score 22787 or a child
1 Time(s): Pid: 11485, comm: apache2 Not tainted 2.6.32-linode23 #1
1 Time(s): Pid: 12215, comm: apache2 Not tainted 2.6.32-linode23 #1
1 Time(s): Pid: 12552, comm: apache2 Not tainted 2.6.32-linode23 #1
1 Time(s): Pid: 12821, comm: python Not tainted 2.6.32-linode23 #1
2 Time(s): Pid: 12843, comm: maxlifetime Not tainted 2.6.32-linode23 #1
1 Time(s): Pid: 14366, comm: apache2 Not tainted 2.6.32-linode23 #1
1 Time(s): Pid: 1813, comm: fail2ban-server Not tainted 2.6.32-linode23 #1
1 Time(s): Pid: 2661, comm: mountall Not tainted 2.6.32-linode23 #1
9 Time(s): Swap cache stats: add 0, delete 0, find 0/0
9 Time(s): Total swap = 0kB
1 Time(s): [<c0102be6>] ? xen_clts+0x46/0x50
1 Time(s): [<c01035f7>] ? xen_mc_flush+0xf7/0x1b0
4 Time(s): [<c0104310>] ? xen_set_pte_at+0x80/0xf0
3 Time(s): [<c01052b7>] ? xen_force_evtchn_callback+0x17/0x30
3 Time(s): [<c0105b0f>] ? xen_restore_fl_direct_end+0x0/0x1
5 Time(s): [<c0105b18>] ? check_events+0x8/0xc
4 Time(s): [<c0106675>] ? sys_clone+0x35/0x50
4 Time(s): [<c0107de9>] ? syscall_call+0x7/0xb
2 Time(s): [<c010a3e7>] ? handle_irq+0x17/0x20
10 Time(s): [<c011b150>] ? do_page_fault+0x0/0x330
7 Time(s): [<c011b284>] ? do_page_fault+0x134/0x330
1 Time(s): [<c0129ae7>] ? finish_task_switch+0x47/0xa0
4 Time(s): [<c012de9a>] ? copy_process+0xba/0xfd0
4 Time(s): [<c012ee20>] ? do_fork+0x70/0x2e0
2 Time(s): [<c0134b91>] ? irq_exit+0x31/0x70
1 Time(s): [<c0142c30>] ? posix_ktime_get_ts+0x0/0x10
1 Time(s): [<c0143ffe>] ? __wake_up_bit+0x2e/0x40
2 Time(s): [<c0146fe0>] ? hrtimer_get_res+0x0/0x30
1 Time(s): [<c014cbd3>] ? ktime_get_ts+0xe3/0x100
5 Time(s): [<c01752b7>] ? filemap_fault+0x397/0x3a0
8 Time(s): [<c0176fea>] ? oom_kill_process+0x9a/0x280
8 Time(s): [<c017765c>] ? __out_of_memory+0xfc/0x160
8 Time(s): [<c0177714>] ? out_of_memory+0x54/0xb0
1 Time(s): [<c017a84c>] ? __alloc_pages_nodemask+0x49c/0x580
8 Time(s): [<c017a911>] ? __alloc_pages_nodemask+0x561/0x580
4 Time(s): [<c017a944>] ? __get_free_pages+0x14/0x30
5 Time(s): [<c017c1dd>] ? __do_page_cache_readahead+0xdd/0x1f0
5 Time(s): [<c017c317>] ? ra_submit+0x27/0x40
5 Time(s): [<c0187c6a>] ? __do_fault+0x5a/0x490
5 Time(s): [<c0189d77>] ? handle_mm_fault+0x167/0x990
1 Time(s): [<c018f609>] ? mmap_region+0x1d9/0x480
1 Time(s): [<c01b0449>] ? poll_select_copy_remaining+0xd9/0x120
1 Time(s): [<c01b1740>] ? sys_select+0x40/0xb0
2 Time(s): [<c0434d5c>] ? unmask_evtchn+0x1c/0xe0
4 Time(s): [<c0460034>] ? do_blkif_request+0x24/0x340
1 Time(s): [<c05f909e>] ? _spin_unlock_irqrestore+0x1e/0x30
1 Time(s): active_anon:40578 inactive_anon:42302 isolated_anon:1
1 Time(s): active_anon:40802 inactive_anon:42302 isolated_anon:1
1 Time(s): active_anon:41189 inactive_anon:41579 isolated_anon:0
1 Time(s): active_anon:41550 inactive_anon:41916 isolated_anon:1
1 Time(s): active_anon:42048 inactive_anon:42227 isolated_anon:0
1 Time(s): active_anon:42207 inactive_anon:42311 isolated_anon:1
2 Time(s): active_anon:42294 inactive_anon:42331 isolated_anon:0
1 Time(s): active_anon:42336 inactive_anon:42304 isolated_anon:3
1 Time(s): active_file:0 inactive_file:31 isolated_file:0
1 Time(s): active_file:13 inactive_file:43 isolated_file:0
1 Time(s): active_file:18 inactive_file:43 isolated_file:0
1 Time(s): active_file:19 inactive_file:1 isolated_file:0
1 Time(s): active_file:26 inactive_file:118 isolated_file:29
1 Time(s): active_file:28 inactive_file:336 isolated_file:30
1 Time(s): active_file:29 inactive_file:226 isolated_file:0
1 Time(s): active_file:33 inactive_file:648 isolated_file:0
1 Time(s): active_file:41 inactive_file:71 isolated_file:0
2 Time(s): apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
2 Time(s): apache2 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0
1 Time(s): fail2ban-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
1 Time(s): free:1031 slab_reclaimable:473 slab_unreclaimable:1363
1 Time(s): free:1714 slab_reclaimable:451 slab_unreclaimable:1346
1 Time(s): free:2645 slab_reclaimable:430 slab_unreclaimable:1354
1 Time(s): free:2717 slab_reclaimable:428 slab_unreclaimable:1366
1 Time(s): free:2769 slab_reclaimable:430 slab_unreclaimable:1354
1 Time(s): free:917 slab_reclaimable:435 slab_unreclaimable:1343
1 Time(s): free:934 slab_reclaimable:483 slab_unreclaimable:1343
1 Time(s): free:936 slab_reclaimable:425 slab_unreclaimable:1360
1 Time(s): free:947 slab_reclaimable:435 slab_unreclaimable:1343
9 Time(s): lowmem_reserve[]: 0 0 0 0
9 Time(s): lowmem_reserve[]: 0 341 341 341
1 Time(s): mapped:158 shmem:7777 pagetables:452 bounce:0
1 Time(s): mapped:33 shmem:7777 pagetables:459 bounce:0
2 Time(s): mapped:39 shmem:7777 pagetables:475 bounce:0
1 Time(s): mapped:42 shmem:7777 pagetables:480 bounce:0
2 Time(s): mapped:50 shmem:7777 pagetables:483 bounce:0
1 Time(s): mapped:60 shmem:7777 pagetables:459 bounce:0
1 Time(s): mapped:86 shmem:7777 pagetables:461 bounce:0
2 Time(s): maxlifetime invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0
1 Time(s): mountall invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
1 Time(s): python: page allocation failure. order:0, mode:0x201da
9 Time(s): unevictable:0 dirty:0 writeback:0 unstable:0
|