This is my mail stack:
- Postfix
- Dovecot
- SpamAssassin (via spamass-milter, which I consider essential because it ensures spam is rejected early)
- SPF
- DKIM (using dkim-milter, soon to replaced with OpenDKIM)
- Postgrey (but only on some servers; my informal observation is that most spam which grey-listing stops would have been caught by SpamAssassin anyways; YMMV)
- BATV
I don't check any DNSBLs from Postfix because SpamAssassin checks a bunch by default anyways.
Unfortunately, checking DKIM signatures doesn't stop any spam (it's more of an anti-forgery/anti-phishing solution). Since many legitimate senders don't use DKIM, you can't assume that mail without DKIM is spam. And since spammers can use DKIM, you can't assume that mail with DKIM
isn't spam. That said, I do recommend you
sign your outgoing mail with DKIM, since it helps with email delivery at places like gmail and Yahoo. You'd also be doing your part to help move the Internet forward towards a world where email forgery is impossible

. I recommend using OpenDKIM, which is a fork of the now-defunct dkim-milter (documentation for one should apply to the other).
100MB of RAM is about what my SpamAssassin stack is using too (and one of my 64 bit servers is using almost double that). It sucks but there's no alternative that's as comprehensive as SpamAssassin. Spamprobe is more efficient and I know people who speak very highly of it, but it only does Bayesian filtering. You could use Spamprobe for Bayesian filtering and check DNSBLs from Postfix, but you'd still miss out on many of SpamAssassin's other rules. The rumor is that Linode will be doubling all RAM soon, which would make 100MB much less painful.
I recently started using
BATV to block backscatter. Unfortunately BATV is somewhat abandoned (despite some big players using it, like Amazon, UPS, and USPS) and I had to write my own implementation to get it to work decently (which I plan to open source very soon). BATV works wonderfully, but you might not want to bother unless backscatter is a problem. I only cared when I started getting dozens of backscatters a day.
Regarding Google Apps: My SpamAssassin setup blocks most spam from reaching my INBOX. I'm on the same group email aliases as some gmail users, and gmail regularly fails to block spam sent to these aliases, whereas SpamAssassin does fine. And I don't have a problem with false positives. My SpamAssassin config is not very customized, but I do train the Bayesian filter regularly. I drag false negatives into an IMAP folder called "Learn_Spam" and an hourly cron job feeds it to sa-learn. I think this is a large part of why SpamAssassin works so well for me. Although it takes a long time for the Bayesian database to train, in the long run it gives you an unparalleled spam filter.