Below is the email I sent to members on host3 and host5. Mostly this only applies to early members -- member's that deployed before I had a chance to update the template filesystems. It couldn't hurt to take a look at your cron times. Removing, or at least moving the updatedb cron time from daily to weekly would help.

------

Dear Linode Member,

In an effort to reduce the disk I/O load on your host server, I ask that you
execute the following command (as root) for your distribution:

Debian
mv /etc/cron.daily/find /etc/cron.weekly/

RedHat
mv /etc/cron.daily/slocate.cron /etc/cron.weekly/

Gentoo
mv /etc/cron.daily/slocate /etc/cron.weekly/

Mandrake:
mv /etc/cron.daily/slocate.cron /etc/cron.weekly/

Slackware
mv /etc/cron.daily/slocate /etc/cron.weekly/

Fedora
No change necessary

This moves the "updatedb" job from running daily to only running weekly. The
"updatedb" command re-indexes your entire filesystem, so that you can find files
quickly by using the "locate (filename)" command. If you don't often use the
"locate" command, moving the index update to weekly (or deleting the cron job
completely) will help reduce the load during the early morning hours. Currently,
the host slows to a crawl while many Linodes attempt to re-index their
filesystems.

This isn't as much of an issue on newer hosts, since I modified the template
filesystems and people have deployed with these files already moved.

If you make the change and it doesn't complain about file not found (indicating
it's already been moved), please shoot me an email and let me know.

Thank you,
-Chris

I would like to add that if you do not move your daily cron jobs to weekly as mentioned, or delete them, that you at least "randomize" the times that your daily cron jobs run. This is probably the best way to solve this problem as it allows you to install services as normal and not worry about whether they put cron jobs in daily or weekly, and yet help prevent huge loads on the Linode host at specific times. To do this, simply edit your /etc/crontab file, under the # run-parts entry.

Here is an example section from an /etc/crontab file:

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

These indicate that cron.hourly jobs run at 1 minute past the hour, cron.daily jobs run at 4:02 am every day, cron.weekly jobs run at 4:22 am every Monday, and cron.monthly jobs run at 4:42 am on the first day of each month.

An example random change would be:

# run-parts
16 * * * * root run-parts /etc/cron.hourly
25 6 * * * root run-parts /etc/cron.daily
45 6 * * 2 root run-parts /etc/cron.weekly
55 6 3 * * root run-parts /etc/cron.monthly

This moves the hourly jobs to the 16th minute of every hour, dailies to 6:25 am, weeklies to Wednesday at 6:45 am, and monthlies to 6:55 on the third day of each month.

If we all stagger our times randomly like this, we can go a long way towards eliminating some common performance bottlenecks on Linodes.

Thanks!

In addition to 'randomising' start times, I schedule maintenance cron jobs after hours or on weekends,

Just a reminder that each time you upgrade the package containing the "find" utility (in Debian, it's "findutil"), the cron job will likely be unpacked into /etc/cron.daily again, while the old cron job you have moved into /etc/cron.weekly will be left there as an orphan.

So remember to move /etc/cron.daily/(find|slocate(.cron)?) back to /etc/cron.weekly (overwriting the orphan) after each upgrade of the parent package.