Linode Forum
Linode Community Forums
 FAQFAQ    SearchSearch    MembersMembers      Register Register 
 LoginLogin [ Anonymous ] 
Post new topic  Reply to topic
Author Message
PostPosted: Sat Jun 09, 2012 5:04 am 
Offline
Newbie

Joined: Sat Jun 09, 2012 4:48 am
Posts: 4
Hi,

Before I proceed, please understand that I don't know ABC of Linode optimization or it's behavior.

Since last 8 - 10 days, my server is getting down for 7-8 hours everyday. When I contacted the support team, they sent me the following logs saying that

Quote:
Upon inspection of your Linode, I have found that your Linode powered off as a result of a kernel panic. To further troubleshoot this issue, please check your logs and your software configuration.


Quote:
-----------------------------------------------------------------------------
BUG: unable to handle kernel paging request at ffff880082c02ee8
IP: [<ffffffff810b1927>] __free_one_page+0x258/0x291
PGD 1c94067 PUD fe1067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:

Pid: 22632, comm: php Not tainted 3.0.18-x86_64-linode24 #1
RIP: e030:[<ffffffff810b1927>] [<ffffffff810b1927>] __free_one_page+0x258/0x291
RSP: e02b:ffff88000da89b48 EFLAGS: 00010012
RAX: 0000000002c07780 RBX: ffff88007fffb760 RCX: ffff88007fffb788
RDX: 0000000002c07780 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff88000da89b98 R08: ffff88007fbf9e58 R09: 0000000000000006
R10: 0000000000000000 R11: ffff88000da89e70 R12: 0000000000000000
R13: ffff88007fffb700 R14: 0000000000000000 R15: ffff88007fffb760
FS: 0000000000000000(0000) GS:ffff88007ff9a000(0063) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff880082c02ee8 CR3: 000000000a4fb000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process php (pid: 22632, threadinfo ffff88000da88000, task ffff88000d9fc410)
Stack:
ffff88000da89c38 ffff88007fffb700 0000000000000000 002c077800000000
0000000000000002 0000000000000002 ffff88007ffb1738 0000000000000002
ffff88007fffb788 ffff88007fffb760 ffff88000da89c18 ffffffff810b3dda
Call Trace:
[<ffffffff810b3dda>] free_pcppages_bulk+0xd3/0x15b
[<ffffffff810b42e3>] free_hot_cold_page+0x13d/0x15e
[<ffffffff810b45b3>] __pagevec_free+0x78/0x91
[<ffffffff810b6914>] release_pages+0x12a/0x186
[<ffffffff810069a2>] ? check_events+0x12/0x20
[<ffffffff810d7d81>] free_pages_and_swap_cache+0x61/0x7b
[<ffffffff810ca65b>] tlb_flush_mmu+0x43/0x61
[<ffffffff810cad80>] tlb_finish_mmu+0xf/0x34
[<ffffffff810ce86d>] unmap_region+0xfd/0x10f
[<ffffffff810cf751>] do_munmap+0x294/0x2fe
[<ffffffff810cf7fc>] sys_munmap+0x41/0x58
[<ffffffff8164cde3>] ia32_do_call+0x13/0x13
Code: 28 70 49 8d 44 05 68 48 89 53 30 48 89 43 28 48 89 0a eb 32 48 63 55 cc 48 8d 4b 28 48 c1 e2 04 44 89 f0 48 6b c0 58 4
8 8d 04 02 <4a> 8b 54 28 68 48 89 4a 08 48 89 53 28 49 8d 54 05 68 48 89 53
RIP [<ffffffff810b1927>] __free_one_page+0x258/0x291
RSP <ffff88000da89b48>
CR2: ffff880082c02ee8
---[ end trace 2333cf58181067ed ]---
-----------------------------------------------------------------------------


Everyday it's getting powered off automatically! Please tell me how to fix this!

Thanks a lot in advance.


Top
   
PostPosted: Sat Jun 09, 2012 5:29 am 
Offline
Senior Member

Joined: Tue Feb 19, 2008 10:55 am
Posts: 164
looks like you ran out of memory.

99 times out of 100, it's apaches fault.

on my system, i have /etc/apache2/apache2.conf

and in that file i have


<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 3
MaxClients 10
MaxRequestsPerChild 1000
</IfModule>

MaxClients is the critical line, you probably have yours set to 150, chop a zero off it, say 15.

what might happen then is instead of crashing, your website becomes slow to load at peak times, and in your apache log you might get a message telling you to raise maxclients.
then you can look at how much memory you're using, and slowly raise maxclients if you have enough spare, if you don't have enough spare, you need t upgrade your linode.


Top
   
PostPosted: Sat Jun 09, 2012 5:35 am 
Offline
Newbie

Joined: Sat Jun 09, 2012 4:48 am
Posts: 4
I am on NGINX, not Apache.

Do I still have to do what you said?


chesty wrote:
looks like you ran out of memory.

99 times out of 100, it's apaches fault.

on my system, i have /etc/apache2/apache2.conf

and in that file i have


<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 3
MaxClients 10
MaxRequestsPerChild 1000
</IfModule>

MaxClients is the critical line, you probably have yours set to 150, chop a zero off it, say 15.

what might happen then is instead of crashing, your website becomes slow to load at peak times, and in your apache log you might get a message telling you to raise maxclients.
then you can look at how much memory you're using, and slowly raise maxclients if you have enough spare, if you don't have enough spare, you need t upgrade your linode.


Top
   
PostPosted: Sat Jun 09, 2012 5:39 am 
Offline
Senior Member

Joined: Tue Feb 19, 2008 10:55 am
Posts: 164
looks like you are the 1%

No need to tune apache.

how many php processes are you running?

maybe paste ps aux | grep php


Top
   
PostPosted: Sat Jun 09, 2012 8:55 am 
Offline
Newbie

Joined: Sat Jun 09, 2012 4:48 am
Posts: 4
^ I am on NGINX server and not Apache.


Top
   
PostPosted: Sat Jun 09, 2012 9:45 am 
Offline
Senior Member
User avatar

Joined: Fri Dec 11, 2009 7:09 pm
Posts: 168
I cannot help with the OOM, but you can enable Lassie on the dashboard under Settings -> Shutdown watchdog. This will reboot automagically if it shuts down.

_________________
--
Chris Bryant


Top
   
PostPosted: Sat Jun 09, 2012 10:39 am 
Offline
Senior Member

Joined: Mon Dec 07, 2009 6:46 am
Posts: 331
Soumya wrote:
^ I am on NGINX server and not Apache.


So you serve only static files?


If that's the case, install and run atop with continuous sampling to do postmortem analysis of what processes did to cause an OOM. Munin graphs can also help show the environment over time.


Top
   
PostPosted: Sat Jun 09, 2012 12:22 pm 
Offline
Senior Member

Joined: Sun Mar 07, 2010 7:47 pm
Posts: 1970
Website: http://www.rwky.net
Location: Earth
Azathoth wrote:
Soumya wrote:
^ I am on NGINX server and not Apache.


So you serve only static files?


If that's the case, install and run atop with continuous sampling to do postmortem analysis of what processes did to cause an OOM. Munin graphs can also help show the environment over time.



Quote:
Pid: 22632, comm: php Not tainted 3.0.18-x86_64-linode24 #1
looks like php's running.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue


Top
   
PostPosted: Sat Jun 09, 2012 6:09 pm 
Offline
Newbie

Joined: Sat Jun 09, 2012 4:48 am
Posts: 4
Since I am new to this and don't know how to fix the issue, I would like to take the help of someone who can do this for me. I am ready to compensate for your efforts. Please note that I am on NGINX and you should be familiar with this web server. Please let me know.


Top
   
PostPosted: Sat Jun 09, 2012 7:48 pm 
Offline
Senior Member

Joined: Tue May 03, 2011 11:55 am
Posts: 105
Is it happening at the same time every day? Could it be a cron or similar that is running and causing problems?


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
RSS

Powered by phpBB® Forum Software © phpBB Group