Filesystem can make a difference. If you are willing to experiment try making a small test partition with ReiserFS and see if that speeds up your compile (making sure you compile on the partition of course).
Also if, for some reason, you are re-compiling a lot of files that haven't changed you could see speedups with something like
ccache. I don't think this wouldn't help in a normal development process if you're using 'make' etc. but bringing it up just in case.
Otherwise my guess would be that you don't have enough RAM and the system is swapping.