Earlier this month I was shocked and astounded to see that I'd used 1% of my bandwidth in 1 day. WTF? I don't get that much email and don't serve that many pages! Hmm... Now it's the 9th and I've only used 2%. OK... so what ahappened?

Wow, my personal web site has 1500 hits already. I migrated the site to the linode on 16th May and for that half of the month there was 800 hits total.

More research... Ah hah... 3 or 4 search engines decided to reindex my site at the beginning of the month, and I have 3 cgi scripts that generate 2300 links to data views...

I think I need a robots.txt file to prevent those cgi's from being indexed!

Let that be a lesson to me

_________________
Rgds
Stephen
(Linux user since kernel version 0.11)