Hi all. I'm sure this is cron, because it's so regular. As many others in my TZ are, I'm a bit of an owl, and tend to hit my stride around 1 in the morning. Unfortunately, I have to endure a whole mess of lag that lasts for about fifteen minutes ever night starting at about 1:10.

This is really gnarly. Aside from typing repeating characters in IRC, it makes it bloody hard to upload and edit my website, and email isn't much fun to send, either.

What's the job that's murdering the server every night, and is there any way to re-nice it so that it doesn't block IO as badly? Thanks!

(edit: I'm on host15)

As far as I am aware niceness only affects the CPU run queue, not disk IO.

Caker is certainly aware of this problem. Some time ago he mentioned added some random element to the crontab timing as part of the imaging process. I don't know if he ever did.

I have scripts that include the following code to sleep for a random time -
#Sleep for a random time between 0 and 512 seconds.
sleeptime=`dd if=/dev/urandom count=1 bs=1 2>/dev/null| od -l | awk 'NR == 1 {print $2 * 2}'`
sleep $sleeptime

If only we could get everyone to include that before all disk heavy cron jobs.

it's not the nightlies that bother me, it's the weeklies that are the real killers

I happened to be up last night about this time (3:00 Central) and noticed the same thing on Host1... Everything was dog slow. Actually I submitted a support ticket, and they said it was my own Linode. But I haven't made any kind of change recently that would cause me to suddenly hit a wall like that, so maybe it's the same thing you're seeing.

I guess it could be cron, although my crontab runs dailies at 6:25am Central. Maybe the other Linodes default to 3:00 Central.

I don't know if it's the crons on the individual linodes hitting at once, or a cron on the actual server hosting the nodes. Could be either. But however you cut it, it's gnarly to deal with.

Standard RedHat (ie Fedora) crontabs run "cron.daily" at 4:02 in whatever timezone the machine is set for. My linode is set for Eastern time, and so that would be 03:02 Central. cron.daily is where nasty things run:

00-logwatch 0anacron prelink slocate.cron yum.cron
00-makewhatis.cron logrotate rpm tmpwatch

The really nasty one here is slocate.cron because it does a find on your file system, and for this reason cacker has taken to automatically disabling this in cron; 10 linodes all running slocate at the same time will kill performance for every linode on the box. Even with the IO limiter

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)

Well, it happened again, and I noticed it hit at precisely 1:00 AM PDT. Aside from griping about it in #linode, I also got into my linode and had a look around. When I ran "top", the most CPU-intensive operation was top, and for memory, it was clamav. free -m looked normal, and I had a full batch of io_tokens available at my disposal.

Also tried a traceroute, and everything looked perfectly normal up to pos3-0.gsr12012.fmt.he.net, after which traceroute was blocked.

I really can't get around this being something the linode server itself is doing at 1 AM, or else I've got a bad neighbor on host15. It sure does stink when it happens, though.

I'm having the same problem on host38, except it happens several times a day. And it's been getting worse and worse.

Turns out updatedb was running, hard, and thrashing all the nodes. Nice when I'm the one contributing to my own problem...