The last several days I have noticed a marked decrease in my sites performance. I hadn't changed anything so for the life of me I couldn't figure out what was happening. I have a drupal site running nginx that while a very "heavy" site usually only gets low traffic but has readily handled all traffic spikes up to 2000 visitors a day with 1 second page loads. Suddenly I'm seeing page loads of over 7 seconds? In checking my logs I see a bunch of 404 errors so thinking that may be the problem I install the drupal fast_404 module and things improve but not back to the level they should be. I also notice most of my hits are from google bot.

Today I am stunned to find my access log is 17mb (just for today! They usually run under 500kb) with traffic almost entirely form googlebot. The IP addresses listed actually are google IP's. The pages trying to be accessed are almost entirely from equine/admin/* type links which are no longer in use on my site so are generating 404's but would have been inaccessible previously (access denied). I checked webmaster tools and almost all the 404 errors to my site for the last couple of days are to these links and I mean PAGES of them.

So my question is why is google spamming my site trying to find links that aren't there? The changes I made to my sites structure have been in place since March so it's not like I changed things last week. I did add equine/admin as denied on my robots.txt but is there anything else I should do?

Thanks everyone!

_________________
Color Genetics

Thought I'd drop in with an answer as no-one else had.

google follows links. If spammy links have been posted to your site by a bot then there's nothing you can do about it. 2000 visitors a day is not high traffic and any linode should easily be able to cater for this.

Are you using a caching plugin or better yet APC/Memcached? if not, i'd advise getting this set up. You could always set up apache/nginx to block the /enquine/admin/* pages, this would reduce your load as requests aren't going back to your database. You could also block bots from accessing your site via apache/nginx config files but this is NOT recommended.

_________________
My Website: http://www.paulthomson.me