Any recommendations for a Python-aware parallel computing model on the SMP Xenodes? I'm leaning towards Bulk Synchronous Parallel:

http://en.wikipedia.org/wiki/Bulk_synchronous_parallel

for two reasons:

1) If I ever have access to a non-SMP distributed computing cluster (i.e., another Xenode on a separate physical computer) I can switch from BSPlib to BSPonMPI

http://bsponmpi.sourceforge.net/

and won't have to change my code or computing model.


2) Konrad Hinsen's Scientific Python:

http://dirac.cnrs-orleans.fr/plone/soft ... /overview/

directly supports BSP for Python parallel computing:

http://dirac.cnrs-orleans.fr/plone/publ ... /HLSO_2006

as well as the more difficult-to-program MPI parallel paradigm.


James

[insert sound of crickets chirping in an empty field]

All the servers will be massively multicore in a few years. Am I the only one trying to prepare for parallel computing here? We already have 4-core SMP virtual servers at Linode *today*.

Forked multiprocess programming is simpler than parallel computing (and lends itself to different operations than pure number crunching), but I want to get the more difficult stuff out of the way first.

James

No, but perhaps you're the only one trying to do it within Python. I use MPI with C/C++.

I'd argue, though, that most people won't need to use an external library to take advantage of multiple cores. The languages and their run-times will evolve to support it. Either that or new languages will take their place.

Good point, I didn't think of that. We use C# for SMP parallel processing at my day job.

James

Well, BSP is out - simple tests ate *way* too much RAM.

I'm now checking "parallel python" at http://www.parallelpython.com - it is simple to program both SMP and clusters. Initial tests are very encouraging.

James