Hi,

My Linode has been working great, but sometimes the connection stops to respond for some seconds (approximately 10-30). I think that the Linode is not freezing, and that it is a networking problem (there aren't any errors in the Linode's logs). I tried to connect from other hosts and it didn't answer anything during the "connection freeze" period.

The Linode is located at the HE datacenter.

Any idea on what is happening?. It is not my connection, because IRC and my other SSH shells work perfectly during the freeze period.

Bye and thanks.

_________________
JTRH

I seem to get the same thing from time to time... on host 7

The Linode systems are bound by disk I/O (way before CPU), and tuning of the vm/IO system is important for making sure the majority of work-loads don't block other processes...

Host7, and the two new servers at HE (host9 and host10) are using the new Anticipatory I/O Scheduler, and I suspect I have some tuning to do...

Can you tell me if this happens at any time of the day, or is this just late at night/early morning when all the Linode cron jobs are going off?

Thanks,
-Chris

Hi,

I am going to put an eye on my Linode so I can note down the time when the problem happens.

Most of the time the connection has been working OK, with no problems in whole days.

Even I don't remember the exact time when the connection showed difficulties, they happened mostly in the afternoon and in the night (GMT -0400).

Bye.

_________________
JTRH

well just now i probably had the biggest moment of lag that lasted several minutes... started at about 830 PST.... my load average last i checked leaped to 14... everything hit a standstill for quite some time...

although overall, these delays seem to happen intermittantly throughout the day... certainly not as long and as drastic as the one i just experienced, but usually a higher than normal load average can appear at the same time...

Correction: Host7 is running 2.4.22. What I'm seeing on host7 is long write queues, with processes in "D" state while vmstat shows throughput (blocks in/out) basically stalling. Perhaps 2.4.22 isn't going to cut it.

I have my 'special' 2.4.21 (+ a bunch of patches/mods) that runs great on the other servers, so if I can't find a solution we can reboot (the very very friendly kind of reboot) host7 back to 2.4.21.

BTW, I'm migrating the remaining Linodes off host4, so that likely doesn't help. Four more to go. Let's see if it was just the migrations on host7 for a few days...

Thanks for the reports!
-Chris

I'm about to reboot my box now, but I just had this occur this morning...

Oct 20 14:18:20 li- kernel: Out of Memory: Killed process 32364 (httpd).
Oct 20 16:43:45 li- kernel: Out of Memory: Killed process 1078 (httpd).
Oct 20 17:25:36 li- kernel: Out of Memory: Killed process 32392 (httpd).
Oct 20 17:25:42 li- kernel: Out of Memory: Killed process 32390 (httpd).
Oct 21 08:15:04 li- kernel: Out of Memory: Killed process 5702 (spamassassin).
Oct 21 08:40:48 li- kernel: Out of Memory: Killed process 5802 (spamassassin).
Oct 21 09:03:03 li- kernel: Out of Memory: Killed process 15696 (mysqld).
Oct 21 09:03:03 li- kernel: Out of Memory: Killed process 15697 (mysqld).
Oct 21 09:03:03 li- kernel: Out of Memory: Killed process 15699 (mysqld).
Oct 21 09:03:03 li- kernel: Out of Memory: Killed process 350 (mysqld).
Oct 21 09:03:03 li- kernel: Out of Memory: Killed process 5919 (mysqld).
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away SQL=INSERT INTO sig_class (sig_class_na$
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away
Oct 21 09:03:20 li- snort-mysql: database: unable to write classification
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away SQL=INSERT INTO signature (sig_name,sig$
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away
Oct 21 09:03:20 li- snort-mysql: database: Problem inserting a new signature 'ICMP Source Quench'
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away SQL=INSERT INTO event (sid,cid,signatur$
Oct 21 09:03:20 li- snort-mysql: database: mysql_error: MySQL server has gone away SQL=ROLLBACK
Oct 21 09:06:52 li- kernel: Out of Memory: Killed process 5918 (spamassassin).
Oct 21 09:08:54 li- kernel: Out of Memory: Killed process 5925 (spamassassin).
Oct 21 09:09:10 li- kernel: Out of Memory: Killed process 5958 (spamassassin).

I'm about to reboot my box, this was happening last night as well, but only an httpd process was dieing every once in a while, this one was a bit more major...

Can the issues you are experiencing on host7 cause these memory issues as well? Or do I need to look into getting more...

The issue he is speaking of is related to disk IO. The issue here is the kernel not being able to allocate memory so it tries to free memory by killing processes starting with the biggest. I assume you are on a Linode 64, but how much swap space do you have setup? Swap is important even with lots of RAM. It makes the kernel happy.

128 at the moment

It may be worth, checking your memory usage of the course of a couple of days, to see if that is one of your problems.

If nessecary look into increasing the size of the swap partition or getting more memory.

Adam

after i rebooted the linode, things have been flying again all day... no problems so far...

may have just needed a good kick in the arse

i'll keep watch of things and wait a few days, happens again i may have to step up...

thnx guys

Is it possible that you are running something that has a memory leak in it?

Adam

the only things that have changed on my linode around the time this began happening is that i installed snort 2.0.2, moved to host 7, and started hosting two new sites that use sql (both run nuke)

however these sites don't get *that* much traffic to them... probably not more than 2 or 3 people on them at a time...

unless anyone knows of any leak issues in snort?

The out-of-memory (OOM) errors you're getting could be caused by lots of things. Like how many database connections / threads your new sites use. Or if it's re-using existing connections. Or if Apache is spawning too many threads. Or if someone is sending lots of emails into the queue, launching lots of (send)mail processes. Might be time to look at adding more swap space (either swap FILES, or actual swap disk images) or upgrading your Linode to give you some more head room.

Back on-topic, how's the performance today?

-Chris

vitre0us: chris wants to know how the site been today?
SirWicked: pretty good

straight from my user who complains the most