Take a look with vmstat:
Code:
peter@fremont ~ $ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 140 8648 60452 73076 0 0 1 1 6 14 0 0 100 0
and pay attention to si and so (swapped in and swapped out). Depending on your usage, the kernel will swap out VM pages that have not been accessed recently in order to free up RAM for buffers. A little swapping in those circumstances is OK, but if those numbers start to climb - you're swapping active pages and your performance will suffer.