My linode (SwamiProject) is crashing periodically (once every week or so). When this happens, I can still ping the server address, but can't SSH in, access any web pages, etc. The system graphs on the Linode management page, show that the CPU spikes and then somewhat flatlines at 6% CPU usage, around when the crash is suspected to have occurred (it happened today 2012-03-29 at around 13:45 on the graph). The I/O and network throughput is at 0.

This last crash, I opened a LISH console and managed to capture the following output. I could not figure out how to scroll back in the text, but it looks like a Kernel oops to me. Unfortunately this excerpt is probably not all that useful, since it doesn't show the top of the call stack. I did not find this in any logs either after rebooting the system. The system is running Ubuntu 11.04.

Any help would be much appreciated!

[<c0109350>] ? do_coprocessor_segment_overrun+0x80/0x80
[<c010b7c1>] ? oops_end+0x71/0xa0
[<c01093cf>] ? do_invalid_op+0x7f/0x90
[<c01a9506>] ? swap_count_continued+0x176/0x180
[<c0101227>] ? hypercall_page+0x227/0x1000
[<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
[<c01062c4>] ? check_events+0x8/0xc
[<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
[<c0188d73>] ? free_hot_cold_page+0xd3/0x140
[<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
[<c0103fa5>] ? pte_pfn_to_mfn+0xb5/0xd0
[<c06903c6>] ? error_code+0x5a/0x60
[<c012007b>] ? try_preserve_large_page+0x7b/0x340
[<c0109350>] ? do_coprocessor_segment_overrun+0x80/0x80
[<c01a9506>] ? swap_count_continued+0x176/0x180
[<c01a9601>] ? swap_entry_free+0xf1/0x120
[<c01ab877>] ? free_swap_and_cache+0x27/0xd0
[<c019dd43>] ? zap_pte_range+0x1b3/0x470
[<c0188f3d>] ? __pagevec_free+0x1d/0x30
[<c019e111>] ? unmap_page_range+0x111/0x190
[<c019e2bb>] ? unmap_vmas+0x12b/0x1e0
[<c01a0481>] ? exit_mmap+0x91/0x140
[<c01324eb>] ? mmput+0x2b/0xc0
[<c0135ddf>] ? exit_mm+0xef/0x120
[<c068fb50>] ? _raw_spin_lock_irq+0x10/0x20
[<c01379d5>] ? do_exit+0x125/0x350
[<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
[<c0137c3c>] ? do_group_exit+0x3c/0xa0
[<c0137cb1>] ? sys_exit_group+0x11/0x20
[<c068fe51>] ? syscall_call+0x7/0xb
[<c0680000>] ? sctp_rcv_ootb+0x50/0xf0

It's a screen session, so you can ctrl+a, [ to enter copy mode, then scroll up. This kind of thing tends to happen when you run out of memory and so then the OOMKiller steps in and kills off some running processes. Assuming this is the case here, you'll want to evaluate your memory usage and optimize where appropriate, or consider upgrading.

Which kernel were you running?

_________________
Matt Nordhoff (aka Peng on IRC)

Its running kernel 2.6.39.1-linode34 which I see is deprecated. I forgot that the Kernel is selected from the configuration, rather than the distribution. So this may be the kernel I had from the beginning.

I'm going to go ahead and try the latest kernel and will also look into memory usage. Its probably related to apache. PHP can consume a lot of memory on one of the Drupal based websites I have, so it may be that someone is spidering that site and causing the crash.

Thanks for the tips.

Yes, Apache can be quite a hog. Try using nginx.

I used to use Apache until some time last year, when it started taking about 40 seconds to load a page, which was ridiculous and finally convinced me to switch. I've had no problems at all with nginx.

Fixed that for you.
It's totally not fair to compare "out of the box" (and definitely idiotic) config of Apache, with mpm-prefork, mod_php, and huge clients settings with tweaked (and you NEED to tweak it to make it work at all) nginx with its non-forking engine, and PHP as fastcgi.

When you'll set your Apache to use mpm-worker (or the poll-based one that's even more like Nginx) engine, and set up PHP as external fastcgi process (like you HAVE to do with nginx), the performance difference is much smaller. And the supportability and flexibility is much higher.

_________________
rsk, providing useless advice on the Internet since 2005.