My Linode on host5 is really really slow right now ... my load (which I was able to see after waiting for about 4 minutes for SSH to log me in) was well over 4, and yet my Linode wasn't really doing that much. I did notice that it was doing an updatedb but that the CPU load for that was very low. Also, host5's load is listed as low. I am guessing that the problem is being caused by lots of disk activity due to cron jobs firing late at night. Do system logs show an unusual amount of activity at that time? Is there anything that can be done to prevent this? Thanks!

BTW, it got better in the time it took me to write this message. I saw the extreme slowness from about 1:20 am PST to about 1:30 am PST. But it may have started before that ...

Actuallly I think I might have been misreading the host load from my members page. It is possible that the host load was listed as high before and not low as I previously reported; it's certainly high right now in any case.

My Linode's load is still listed at from 2.5 to 3.5, but is much more responsive than before. It's now almost 1:40 am PST ...

This is a nightly occurrence on the first few hosts, since they have Linodes with distro images deployed before I removed from cron most of what causes the high disk contention (updatedb, etc). I did pause each Linode for a few moments to determine if it was a specific Linode thrashing.

I've been working on a 2.6 host kernel and am preparing to deploy it fairly soon. 2.6 has improvements to the I/O system, and I'm looking forward to getting it online.

-Chris

Well in the spirit of trying to help mitigate this problem through cooperation of the various Linodes on host5, I'm posting my /etc/crontab listings for hourly, weekly, daily, etc here. If you have a Linode on host5 could you do the same so that we can all try to find times that do not coincide?

04,09,14,19,24,29,34,39,44,49,54,59 * * * * root run-parts /etc/cron.fiveminutely
07 * * * * root run-parts /etc/cron.hourly
17 4 * * * root run-parts /etc/cron.daily
27 4 * * 1 root run-parts /etc/cron.weekly
47 4 2 * * root run-parts /etc/cron.monthly

Notes:

I made my own "cron.fiveminutely" to run stuff every five minutes. I use this for webalizer.
My hourly times are on seven minutes after the hour.
My daily times are at 4:17 am eastern time.
My weekly times are at 4:27 am eastern time on Monday morning.
My monthy times are at 4:47 am eastern time on the second day of each month.

I run backups at 10:04a, 10:12a, 2:04p, 2:12p, 6:04p, 6:12p, 10:04p, and 10:12p all Pacific Time. Unfortunately, these are initiated from my home machine which currently is not synced with a time server. I'll try to remedy that shortly.

Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed.

It is odd to have them in pairs like that, but there probably is a reason; maybe backing up a database before some operation is done to it and then backing it up afterwards too?

I don't think that the backups that this person is doing is putting an appreciable load on host5. I have never noticed performance problems, and no one else seems to be reporting them, at the times that this person is doing backups.

The real problem seems to be all of the updatedb cron jobs firing at the same time late at night (eastern time). Hopefully people who have left their updatedb cron jobs to their default times will see these messages and remedy this.

The paired backup operations go in different directions, server to home and home to server. I separated them just to preserve scheduling flexibility.

The backups use hard links and rsync, to maintain multiple snapshots of the other machine. They get issued every four hours because I don't want to lose more than four hours of data !

As bji suggested, this should not be a substantial load. They take no more than a couple minutes each.