I am running a Linode 512 for a while now, and slowly I have migrated everything I had on other servers to this instance.
I am running a few php websites, with a single php-fpm process, a few ruby on rails sites with unicorn (use much memory), and mail services including spamassassin. Webserver is Nginx.

I don't have any problems, but when I look into my munin monitor graphs, I see that my swap space is much filled, but also that my ram is also a lot free.
I have read about linux using all the ram, and swapping stuff if it doesn't need it currently, but I am left wondering if it's unhealty.

Before i added a new Mono-ASP.Net fastcgi server today, stats were:
Memory free: 180M
Swap Usage: 200M

After it:
Memory free: 150M
Swap Usage 220M

(These stats are gathered from 'free -m" and then the -+ buffers/cache items.)

So, I now have more swap usage then free memory, but I still do have a lot of memory free...
I don't know if this means that I should add more memory, or that Linux is just efficient in using all available space and I should start worrying when my free memory goes below 50M or so.

Any insights?

Using swap space is not in and of itself a problem. The Linux kernel will move rarely-needed data there to make room for more disk caching. Transferring data in and out of swap is what kills your performance.

Look at the swap in/out graph for Munin to see if your machine is having to swap pages in and out. If so, you should reduce the number of running processes or increase the memory available.