Hi!

Such strange thing is happening first time to me. I have been using Linode for couple years now and have not experienced this type of behaviour. Guys from Linode had no idea what to suggest, so I'm hoping that here few of you can suggest where to look.

Situation is like this - server can run for days or for month without problems. And then suddenly it stops responding at all. http down, impossible to even connect through ssh. After reboot everything is fine. And then again it can work for weeks or gets frozen after few hours. Even munin graphs have gaps for those periods of downtime. So i would tell that whole system crashes.

What is running on server: Apache2, Nginx, Mysql, php5. Everything on Ubuntu 10.04.

Any idea where to look? Are there any log files that could be useful?

are you running mpm_prefork in apache (most likely yes). if so, what is your MaxClients set to? The default is way too high. What could be happening is you're getting a short traffic spike that is causing you to run out of memory.

Running out of memory? You can leave a lish session open running 'top' or something (sorted by memory perhaps) to see if that's the case, and also installing Munin can provide some useful graphs to try to see if there is some info to be gained from all the resource graphs.

Out of memory is not it. I just opened Munin grapsh and apps usage is 304MB on avarage, max 361. I have 1GB linode.

In front of everything is nginx who handles static content and then apache2 comes in to server php (i use itk module).

Traffic should not be problem. On some days i serve 500 000 hits and everything is smooth. Today traffic was not as high as few previous days. When server went down it was only at about 50% of its regular load.[/img]

What kernel are you running ? run

uname -a

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

2.6.39.1-linode34

The same apparent thing happened to me (I posted at viewtopic.php?t=7459&highlight= but had no answers). I logged in via Lish and the Linode was happily humming away, it had just lost networking. Mine has only done that once though, and it could be a different issue.

_________________
--
Chris Bryant

It should not be network lose, because Munin have gaps in graphs for time of downtime. So nothing for CPU, memory or any other usage. Like someone would have switched it off too. In case of lost network munin should still work....

Freezes where the whole system randomly locks up completely normally indicates a kernel issue check /var/log/messages or /var/log/syslog for any kernel related message.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

Checked logs you suggested. In messages nothing interesting.

Last time server went down around 15:55.

However syslog is a bit more interesting:

Interesting part is '^@' . If i open file with some regular editor (like Notepad2) i get 'null' there. If i open in gVIM i get bunch of '^@' characters. So something bad was there, but I gues no way to find out what exactly.

Interesting, the yyyy should be the year as well so something's pretty busted what distro are you running? Can you remember roughly when you last rebooted before this started happening?

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

With 'yyyy' i just replaced name of my linode. So that part is good. Strange part is only those 'null' (^@^@) characters.

I'm using 32bit Ubuntu 10.04 (with latest updates).

First time i noticed this was on July 11. Since then it happened few times. In first cases i noticed this in few minutes and was able to reboot. Today i did not notice that for ~30min and whole server was down for that time. So 30min downtime suggests, that it does not bounce back.

Ah right ok, try booting with 3.0.0-linode35 kernel and see if the problems goes away (don't be scared of the big 3.0 it's not that different from 2.6)

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue

OK. I'm now using "3.0.0-linode35". Will monitor situation and will post update if it will crash. I hope there will be no updates from me.

If 3.0 is stable for you, keep an eye on http://www.linode.com/kernels/ when the latest Paravirt becomes 3.0 you should change to that to make sure your kernel is kept up to date.

You can also post here viewtopic.php?t=7505 to let Linode know how the kernel's doing.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue