Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
 Post subject:
PostPosted: Mon Apr 30, 2012 8:10 pm 
Offline
Junior Member

Joined: Mon Sep 19, 2011 2:48 am
Posts: 28
As MotoHoss suggested above, I tried for my Linode 512..
Code:
vm.min_free_kbytes = 5120

And have not seen an error since.


Top
   
 Post subject:
PostPosted: Tue May 08, 2012 4:08 am 
Offline

Joined: Tue May 08, 2012 4:04 am
Posts: 1
I'm seeing this in London too, running 3.0.18-linode43. I've even tried increasing min_free_kbytes to 16384, but it does not appear to help.


Top
   
PostPosted: Sat Jun 30, 2012 8:34 am 
Offline
Newbie
User avatar

Joined: Fri Mar 02, 2012 5:25 am
Posts: 3
Website: http://edwinlee.proxyy.biz
Location: Singapore
Recently encounter this problem on my Linode 512

From syslog:
--
kernel: nginx: page allocation failure. order:2, mode:0x20
kernel: Pid: 5003, comm: nginx Not tainted 2.6.39.1-linode34 #1
--

Doing as suggest:

vm.min_free_kbytes=5120

Not getting the debugging error on syslog since. :)


Top
   
PostPosted: Sat Jun 30, 2012 12:14 pm 
Offline
Senior Member

Joined: Sat May 03, 2008 4:01 pm
Posts: 568
Website: http://www.mattnordhoff.com/
edwinlee:

2.6.39.1-linode34? That's really old, and subject to a serious security vulnerability. You should upgrade.

_________________
Matt Nordhoff (aka Peng on IRC)


Top
   
PostPosted: Wed Jul 04, 2012 2:25 am 
Offline
Sysop

Joined: Sat Nov 27, 2010 3:32 am
Posts: 180
Website: https://blog.timheckman.net/
Location: San Francisco, CA
Also, the problem you ran in to is most likely a bug that has since been fixed. But yeah, you should get off that kernel. :)

-Tim

_________________
'If debugging is the process of removing bugs, then programming must be the process of putting them in.' //Edsger Dijkstra
'Nothing is withheld from us which we have conceived to do.' | 'Do things that have never been done.' //Russell Kirsch


Top
   
PostPosted: Fri Jul 06, 2012 6:54 am 
Offline
Senior Member

Joined: Sat Nov 13, 2010 3:05 am
Posts: 91
Website: http://www.graq.co.uk
I'm on kernel 3.4.2 (vm.min_free_kbytes = 4096) and still getting paging errors.

My OSSEC emails are very spammy.


Top
   
PostPosted: Thu Jul 12, 2012 4:43 pm 
Offline

Joined: Thu Jul 12, 2012 4:40 pm
Posts: 1
I'm also running into very similar errors. I'm now running 3.4.2-linode44, but also had issues on 2.6 kernels. I'm reasonably sure it's triggered by my daily rsync backup jobs, which correlates with the network layer functions in the backtrace.

Code:
swapper/0: page allocation failure: order:5, mode:0x20
Pid: 0, comm: swapper/0 Not tainted 3.4.2-linode44 #1
Call Trace:
 [<c019b538>] ? warn_alloc_failed+0x98/0x100
 [<c019bfa8>] ? __alloc_pages_nodemask+0x4d8/0x6e0
 [<c01c3b41>] ? T.874+0x31/0xe0
 [<c01c3e31>] ? T.871+0x91/0x250
 [<c01c423e>] ? cache_alloc_refill+0x24e/0x290
 [<c01c433e>] ? __kmalloc+0xbe/0xd0
 [<c056232e>] ? pskb_expand_head+0x12e/0x240
 [<c01c32d2>] ? kmem_cache_free+0x42/0x60
 [<c05628cd>] ? __pskb_pull_tail+0x4d/0x2a0
 [<c05675cf>] ? netif_skb_features+0xaf/0xc0
 [<c056b9dd>] ? dev_hard_start_xmit+0x1ed/0x410
 [<c05a876c>] ? nf_iterate+0x6c/0x90
 [<c057fd7a>] ? sch_direct_xmit+0xba/0x180
 [<c05fb7f0>] ? ip_finish_output2+0x280/0x280
 [<c056bcff>] ? dev_queue_xmit+0xff/0x340
 [<c05fad58>] ? ip_local_out+0x18/0x20
 [<c060eb65>] ? tcp_transmit_skb+0x395/0x660
 [<c06114cd>] ? tcp_write_xmit+0x1dd/0x500
 [<c0611854>] ? __tcp_push_pending_frames+0x24/0x90
 [<c060d922>] ? tcp_rcv_established+0x3d2/0x610
 [<c05f6600>] ? ip_rcv+0x330/0x330
 [<c061413b>] ? tcp_v4_do_rcv+0xbb/0x190
 [<c06148cd>] ? tcp_v4_rcv+0x6bd/0x7a0
 [<c05f6697>] ? ip_local_deliver_finish+0x97/0x220
 [<c05f6600>] ? ip_rcv+0x330/0x330
 [<c05f6067>] ? ip_rcv_finish+0xd7/0x340
 [<c0568dc3>] ? __netif_receive_skb+0x2c3/0x350
 [<c056a79f>] ? netif_receive_skb+0x1f/0x70
 [<c0506b51>] ? handle_incoming_queue+0x1a1/0x270
 [<c0506e44>] ? xennet_poll+0x224/0x570
 [<c056afda>] ? net_rx_action+0xea/0x1a0
 [<c0131c4c>] ? __do_softirq+0x7c/0x110
 [<c0131bd0>] ? irq_enter+0x70/0x70
 <IRQ>  [<c0131a26>] ? irq_exit+0x66/0x90
 [<c049414d>] ? xen_evtchn_do_upcall+0x1d/0x30
 [<c06f3147>] ? xen_do_upcall+0x7/0xc
 [<c01013a7>] ? hypercall_page+0x3a7/0x1000
 [<c010603f>] ? xen_safe_halt+0xf/0x20
 [<c010fcac>] ? default_idle+0x1c/0x40
 [<c010feea>] ? cpu_idle+0x4a/0x80
 [<c089986e>] ? start_kernel+0x2e3/0x2e8
 [<c0899406>] ? kernel_init+0x127/0x127
 [<c089c813>] ? xen_start_kernel+0x520/0x528
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:  91
CPU    1: hi:  186, btch:  31 usd:  23
CPU    2: hi:  186, btch:  31 usd: 130
CPU    3: hi:  186, btch:  31 usd:  86
HighMem per-cpu:
CPU    0: hi:   18, btch:   3 usd:  10
CPU    1: hi:   18, btch:   3 usd:  17
CPU    2: hi:   18, btch:   3 usd:  14
CPU    3: hi:   18, btch:   3 usd:   4
active_anon:35793 inactive_anon:49055 isolated_anon:0
 active_file:41148 inactive_file:36416 isolated_file:0
 unevictable:0 dirty:437 writeback:0 unstable:0
 free:16325 slab_reclaimable:4754 slab_unreclaimable:2843
 mapped:29371 shmem:11542 pagetables:1298 bounce:0
DMA free:3904kB min:108kB low:132kB high:160kB active_anon:0kB inactive_anon:484kB active_file:1228kB inactive_file:1232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:2 [<c05f6600>] ? ip_rcv+0x330/0x330
 [<c05f6067>] ? ip_rcv_finish+0xd7/0x340
 [<c0568dc3>] ? __netif_receive_skb+0x2c3/0x350
 [<c056a79f>] ? netif_receive_skb+0x1f/0x70
 [<c0506b51>] ? handle_incoming_queue+0x1a1/0x270
 [<c0506e44>] ? xennet_poll+0x224/0x570
 [<c056afda>] ? net_rx_action+0xea/0x1a0
 [<c0131c4c>] ? __do_softirq+0x7c/0x110
 [<c0131bd0>] ? irq_enter+0x70/0x70
 <IRQ>  [<c0131a26>] ? irq_exit+0x66/0x90
 [<c049414d>] ? xen_evtchn_do_upcall+0x1d/0x30
 [<c06f3147>] ? xen_do_upcall+0x7/0xc
 [<c01013a7>] ? hypercall_page+0x3a7/0x1000
 [<c010603f>] ? xen_safe_halt+0xf/0x20
 [<c010fcac>] ? default_idle+0x1c/0x40
 [<c010feea>] ? cpu_idle+0x4a/0x80
 [<c089986e>] ? start_kernel+0x2e3/0x2e8
 [<c0899406>] ? kernel_init+0x127/0x127
 [<c089c813>] ? xen_start_kernel+0x520/0x528
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:  91
CPU    1: hi:  186, btch:  31 usd:  23
CPU    2: hi:  186, btch:  31 usd: 130
CPU    3: hi:  186, btch:  31 usd:  86
HighMem per-cpu:
CPU    0: hi:   18, btch:   3 usd:  10
CPU    1: hi:   18, btch:   3 usd:  17
CPU    2: hi:   18, btch:   3 usd:  14
CPU    3: hi:   18, btch:   3 usd:   4
active_anon:35793 inactive_anon:49055 isolated_anon:0
 active_file:41148 inactive_file:36416 isolated_file:0
 unevictable:0 dirty:437 writeback:0 unstable:0
 free:16325 slab_reclaimable:4754 slab_unreclaimable:2843
 mapped:29371 shmem:11542 pagetables:1298 bounce:0
DMA free:3904kB min:108kB low:132kB high:160kB active_anon:0kB inactive_anon:484kB active_file:1228kB inactive_file:1232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:100kB shmem:0kB slab_reclaimable:28kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 700 754 754
Normal free:61192kB min:5008kB low:6260kB high:7512kB active_anon:132192kB inactive_anon:183580kB active_file:157112kB inactive_file:135248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:717288kB mlocked:0kB dirty:1740kB writeback:0kB mapped:106336kB shmem:44056kB slab_reclaimable:18988kB slab_unreclaimable:11372kB kernel_stack:1864kB pagetables:5192kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 428 428
HighMem free:204kB min:128kB low:220kB high:316kB active_anon:10980kB inactive_anon:12156kB active_file:6252kB inactive_file:9184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:54868kB mlocked:0kB dirty:8kB writeback:0kB mapped:11048kB shmem:2112kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 684*4kB 112*8kB 17*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3904kB
Normal: 4854*4kB 3130*8kB 900*16kB 51*32kB 11*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61192kB
HighMem: 11*4kB 2*8kB 3*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 204kB
100731 total pagecache pages
11592 pages in swap cache
Swap cache stats: add 205947, delete 194355, find 1954619/1973411
Free swap  = 446976kB
Total swap = 524284kB
198640 pages RAM
13826 pages HighMem
6561 pages reserved
155807 pages shared
118099 pages non-shared
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
  cache: size-131072, object size: 131072, order: 5
  node 0: slabs: 0/0, objs: 0/0, free: 0


Top
   
PostPosted: Wed Jul 18, 2012 10:13 am 
Offline
Newbie
User avatar

Joined: Fri Mar 02, 2012 5:25 am
Posts: 3
Website: http://edwinlee.proxyy.biz
Location: Singapore
Hi,

mnordhoff:

Thanks for heads up. I have checked on the security issue relating to kernel version 2.6.39.
-----
Vulnerability Summary for CVE-2012-0056
The mem_write function in Linux kernel 2.6.39 and other versions, when ASLR is disabled, does not properly check permissions when writing to /proc/<pid>/mem, which allows local users to gain privileges by modifying process memory
-----

I have also run the exploit (mempodipper)- http://blog.zx2c4.com/749
on my linodes with a normal login account and did not gain root shell.

----
pentester@mercury:~$ whoami && id
pentester
uid=1012(pentester) gid=1011(pentester) groups=1011(pentester)
pentester@mercury:~$ ./mempodipper
===============================
= Mempodipper =
= by zx2c4 =
= Jan 21, 2012 =
===============================

[+] Waiting for transferred fd in parent.
[+] Executing child from child fork.
[+] Received fd at -1.
[-] recv_fd: Address already in use
pentester@mercury:~$ [+] Opening parent mem /proc/11146/mem in child.
[-] open: No such file or directory

pentester@mercury:~$ whoami && id
pentester
uid=1012(pentester) gid=1011(pentester) groups=1011(pentester)

----

Confirmed this vulnerability on Debian - http://security-tracker.debian.org/trac ... -2012-0056 (linux-2.6 source squeeze (not affected) )
----
pentester@mercury:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 6.0.5 (squeeze)
Release: 6.0.5
Codename: squeeze
----

This raise lower concerns for now as it's only locally exploitable and I am the only user accessible to shell, non logins and system accts are properly secured. Have additional security measures with filesystem quotas, shell limits, and OSSEC HIDS (syscheck, rootkits detect, file changes, log monitoring, alerts and responses). I am actively looking into the security of my linodes with nmap and nessus scans.


theckman:

Thanks, I am considering using newer kernels when I order for new linodes. :)


Top
   
 Post subject: e-papieros forum
PostPosted: Thu Jul 19, 2012 10:01 am 
Offline

Joined: Thu Jul 19, 2012 9:57 am
Posts: 1
Image

_________________
<SPAMMER>


Top
   
PostPosted: Thu Jan 03, 2013 9:11 am 
Offline
Senior Newbie

Joined: Thu Mar 15, 2012 8:54 am
Posts: 9
Website: http://aptivate.org/
We're experiencing this as well (it happened once and I'd like to ensure that it doesn't happen again). Has anyone found a definite fix for this?

I found a post that seems to imply that it's a problem with Linode's Xen configuration, not something that we can change ourselves. http://xen.1045712.n5.nabble.com/SLUB-allocation-error-on-3-0-3-4-1-1-td4795696.html


Top
   
PostPosted: Thu Jan 03, 2013 9:19 am 
Offline
Senior Newbie

Joined: Thu Mar 15, 2012 8:54 am
Posts: 9
Website: http://aptivate.org/
And I found a discussion, which links to a Redhat Bugzilla that I don't have permission to access, but seems to imply that the message is faily harmless, as it just means that the kernel didn't have enough free RAM to process a network packet, so it was dropped. This could happen even if you have free Swap, as the packet is received in interrupt context and it's not safe to swap memory out to disk from that context, so if there's not enough free RAM then the kernel's hands are tied. (at least that's my interpretation of the discussion)
http://thr3ads.net/centos/2012/10/2111457-swapper-page-allocation-failure.-order-1-mode-0x20

So is this just a scary message or does it have real consequences?


Top
   
PostPosted: Wed Jan 09, 2013 4:59 pm 
Offline
Senior Member
User avatar

Joined: Tue Nov 24, 2009 1:59 pm
Posts: 362
Sounds just about right.
"Real world consequence" is a dropped packet, which will get retransmitted like after any other tiny networking blip.
Increasing the vm_free_kbytes setting reserves a larger in-RAM buffer for various operations like this, reducing the chance you'll run into this error.

_________________
rsk, providing useless advice on the Internet since 2005.


Top
   
PostPosted: Sun Jan 20, 2013 6:21 pm 
Offline
Senior Member
User avatar

Joined: Tue Apr 13, 2004 6:54 pm
Posts: 833
I wonder if this will fix it... http://git.kernel.org/?p=linux/kernel/g ... dc06f9ee83

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)


Top
   
PostPosted: Mon Jan 21, 2013 6:46 pm 
Offline
Senior Member
User avatar

Joined: Wed Mar 17, 2004 4:11 pm
Posts: 554
Website: http://www.unixtastic.com
Location: Europe
I'm not sure if this got fixed by a new kernel but anyway the message appears to be harmless. As stated before it's the kernel trying and failing to allocate a chunk of memory.

Does someone with this problem want to try:
ethtool -K eth0 lro off


Top
   
PostPosted: Tue Jan 22, 2013 3:40 pm 
Offline
Senior Member
User avatar

Joined: Tue Apr 13, 2004 6:54 pm
Posts: 833
Did you look at the patch? It's in the IPv4 memory allocation routines for metrics, and if a kzalloc fails then it will fall back to a vzalloc - which kinda matches the symptoms we're seeing (always in the IPv4 stack, and memory allocation failures) :-)

_________________
Rgds

Stephen

(Linux user since kernel version 0.11)


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group