Hi!

In the last few weeks, I've been experimenting some random "crashes" of my Linode 1024. They happen every 9-13 days, at the same hour, about 8:00 GMT, and they cause the CPU, network and HD graphs at my Linode Dashboard to show values close to zero. The server doesn't respond from "outside", and the only possible solution is to reboot it, and this takes about 3 minutes.

It runs Ubuntu 10.04, with LAMP, Tomcat 6, Postfix and Dovecot. All of them are the last versions available, the software is totally up to date. I've checked some "system" logs, like syslog, daemon.log, etc., but they don't show anything relevant, they just "log" things until the crash moment.

I have this server since June, but the problems started in August. In addition, from the moment of the server purchase, I haven't made any software installation or similar. Other than versions changes, the server has the same software that was installed in the first time.

Any ideas?

Thanks in advance!

Hello,

You need to do two things: One, make sure you're running the "Latest 3.0" kernel (Linode Manager -> Linode -> Config Profile -> Kernel drop-down, save, reboot). Second, when this happens, inspect your Lish console before you reboot to see if there has been a kernel BUG or OOM or something, or if your Linode is still 'alive' but something has taken networking down.

Hope that helps,
-Chris

Thanks for your reply. I've just changed the kernel to "Latest 3.0", and now all runs OK. However, about the Linode being "alive" or just the network, I must say that ALL the "system" logs that I've checked were "empty" for the few hours the server was crashed. Not just the network, it was the server, completely.

Should I check one particular log?

And let's suppose this happens again. When I access my server using Lish console at Manager, what should I watch?

Finally, in the case I can't access my Linode from Lish console as said, what could be the issue?

Check your MaxClients in apache. the default is for systems with many gigabytes of RAM and is not appropriate for a linode. That is the cause of the vast majority of threads like yours.

The console shows a lot of things that wouldn't be in any log file. For example, a panicked kernel can't write anything to a log file, but it can still spit out a message to the console before going down. Firewall messages often show up on the console, too.

A reboot will probably erase anything that was written to the console, though.

THanks to everyone!

Regarding Apache, I don't think it is a memory problem, because memory consumption is all the time around 30% of the total.

The next time this happens, if it does happen, I will access my server using Lish console and check for warning messages before rebooting, so that they aren't erased.

Lish's logview command (detach from the screen session showing your console itself) shows the most recent lines of both your current boot and the previous one, so even if you have rebooted, the information is still there. Unless you've rebooted twice, anyway.

_________________
Matt Nordhoff (aka Peng on IRC)

OK.

OK, the server has just crashed, and here is the output from Lish console as you described:

REMOVED. SEE http://lists.xensource.com/archives/htm ... 01172.html FOR INFORMATION

I can't login, just opening the console throws those messages. And I can't do anything. Note that I changed MAC and IP addresses, because that isn't important.

In addition, I'm not the first one with this problem on the VPS:

viewtopic.php?t=7514

The graphs of my dashboard say that CPU load before crash was 20%, network was about 400 kbps and IO around 3.3 kblocks per sec. And, as I mentioned, memory is always about 30% all the time.

Clearly, I'm not the only customer of Linode that has this error. It would be nice if Linode, as it's a serious company, investigates to solve the problem

Thanks in advance!

I dont think you quite understand what your being shown.
This is a kernel crash probably todo with your instance running out of memory. This isnt linode's fault.

Scroll up in that logview and im sure somewhere there will be out of memory error.

This means you need to tune your instance to not eat as much memory. Maybe run ab or a benchmarking utility agaisnt it to see if you can put enough pressure on it to cause the error to happen

_________________
ServerAdmin - www.our-lan.com
"Diplomacy is the art of saying nice doggy whilst looking for a really big stick"
"In my experiece, any attempt to make any system idiot proof will only challenge God to make a better idiot"

Also, a quick tip - to scroll up in the console where this is being shown, press Ctrl-A and release, then press Escape. You'll then be able to scroll up and down with the cursor keys and Page Up/Page Down. Press Escape again to go back to normal.

I'm sorry, I can't scroll up. How is that done? How can I use "logview"?

If I press Ctrl-A, it shows me a message saying "No other window". If I press Escape, then the message hides and I return to the normal console. I can't scroll with the mouse, or the Page Up/Down keys.

Well, I managed to show the contents of the Lish console:

REMOVED. SEE http://lists.xensource.com/archives/htm ... 01172.html FOR INFORMATION.

It seems like it's an Apache issue... Could anyone translate the above code to English? Is it a memory error? IO? CPU? It seems it's something related to the memory, the stack, bit I don't know it 100% sure.

Thanks in advance!

It appears it's NOT an Apache issue:

http://www.gossamer-threads.com/lists/xen/devel/216324

This user has exactly the SAME problem as me, and... HE/SHE USES LINODE! Apparently, they haven't reached a solution, as they are not able to reproduce the error.

And in most forums in which users report this error, it appears that the problem is related with hardware. However, this is strange, as Linode uses virtual environments...

Finally, it's quite remarkable that, as I described, I have used Linode since June, but problems started in August, and they occur every few days. The forum post I linked is of August.

Hope we find a solution.

I believe Linode might be a couple steps ahead of you, given that the xen-devel thread was started by a Linode engineer.

Might be a good idea to open a ticket, to the attention of psandin, especially if you're willing to be a test pilot.

_________________

/* TODO: need to add signature to posts */