So we have two linodes running squid being fed by a nodebalancer and the squids are caching off a 4 gig linode running mediawiki/mysql/php etc.

Everything runs great, squid takes 95 percent of the load of serving pages.

But once a day on most days, not at the same time, the server goes into OOM and thrashing and has to be re-booted.

Looking at the nodebalancer incoming connections we see a huge traffic spike about 600-700 connections/second and then an immediate return to our normal 80-100.

There is nothing in the squid logs or apache logs abnormal during these periods.

Any ideas? I want to try and attribute it to a run away process somewhere, but what could trigger a traffic spike on the nodebalancer?

Any help is greatly appreciated!

Maybe daily log rotate/daily crons cause everything to get backed up. Check if it is the same time.

First step: tune your configurations (or upgrade) so the server will never go into an OOM condition.