Today I got a Linode Alert at 10.20 CET:

I received Linode Alerts before, but that was while I was doing performance tests. Today I wasn't. How do I figure out what happened?
My munin installation is incomplete (just default), because the munin site gives 500 errors when trying to download a plugin for almost 2 weeks now.

What do I see on munin graphs:
* munin no activity between 6 and 9 am
* a peak at connections through firewall just before 6 am
* a peak at postfix bytes throughput at half past 6
* fork rate, cpu usage, and interrupts peak between 6.30 and 9.00 am
* gaps in the charts for number of threads, process priority, vmstat, file table usage, memory usage around that time.

Don't think it is an outside job (DOS, nothing in awstats or log files looks unusual). I have created 2 custom cronjobs yesterday, but they run between 23.00 and 23.10. Around half past 6 is the time when logrotate is scheduled and logwatch sends its report around that time too.
I noticed that I am sending a lot of messages to myself (mail.log)

Jan 10 06:40:05 m41l postfix/local[29928]: 4D434B34F: to=<www-data@m41l.example.com>, orig_to=<www-data>, relay=local, delay=0.03, delays=0.02/0.01/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Jan 10 06:40:05 m41l postfix/qmgr[2377]: 4D434B34F: removed
Jan 10 06:50:05 m41l postfix/pickup[4029]: 17379B34F: uid=33 from=<www-data>
Jan 10 06:50:05 m41l postfix/cleanup[24869]: 17379B34F: message-id=<20120110055005.17379B34F@m41l.example.com>
Jan 10 06:50:05 t4d0rn4 postfix/qmgr[2377]: 17379B34F: from=<www-data@m41l.example.com>, size=886, nrcpt=1 (queue active)

Where m41l.example.com is the hostname of my server (modified). Half past 6 it seemed like there were a 1000 mails in queue. Don't know what for. Don't know where the mails for www-data@m41l.example.com or root@m41l.example.com go to; haven't set up any email addresses. Port 25 is blocked by firewall.

Where do I start looking? How do I check if there is any mail for root or www-data? Or how do I divert it to another email address? How do I check which programs are trying to send me email? Logwatch and custom cronjobs (using php mailer) work fine in sending me messages (to an outside email address).
Anyway, will see what happens tomorrow, bit puzzled right now.

The postfix mail queue was not the cause of the excessive CPU usage. The logrotate, logwatch, ... at 6.20 am today hardly registered on the CPU chart.
Still don't know what caused the surge.

If www-data shows up on your mail log more often than it should, there's a possibility that one of your PHP scripts is being exploited by spammers. Do you have a contact form or any other web-accessible script that sends mail? It's a bit weird that the mail is being sent to local accounts at your server, but spam bots aren't very clever.

In Debian-based distributions (including Ubuntu), Postfix stores local mail in /var/mail by default. I'm not sure about other distros, but I suspect it's the same.

I do have a contact form, but the 'To' address is static, so I should receive an email. Tried it out and it works.
Most of domains on server are parked without contact email, maybe I should create catch-all email addresses per domain (hope I can do this with Google Apps).

Had a look in /var/mail, it was very enlightening.
The mails to root are munin cronjobs that failed.
The mails to www-data are awstats cronjobs that reported an error.
Will have a look at them tomorrow.

edit /etc/aliases to something like this

postmaster:    root
root: youremail@address.com
www-data: root
munin: root

Then run

newaliases && service postfix restart

That will forward emails to your email address.

_________________
Paid support
How to ask for help
1. Give details of your problem
2. Post any errors
3. Post relevant logs.
4. Don't hide details i.e. your domain, it just makes things harder
5. Be polite or you'll be eaten by a grue