[SciPy-Dev] [gpaw-users] wrapper for Scalapack

[Bennet Fauber]

Ralf, and all,

>> There seems to be a profusion of tools for parallelization, so choosing
>> just one to use as a basis for scipy's parallelization could be really
>> frustrating for users who have a reason to need a different one.
>
> You're thinking about the relatively small fraction of power users here that
> would care (compared to the n_jobs=<number> trivial parallelization users),
> and my first thought is that addressing that use case comes with costs that
> are possibly not worth the effort.

You might consider separating that which can be done on one physical
machine from that which requires (or expects) many.

This was largely done by the R developers.  The 'snow' library used
rsh/ssh whereas the 'multicore' library used fork() and processors.
Steve Weston and company have the 'foreach' library that provides a
user interface to various backends that distribute the tasks
appropriately.  Only after many years of experience, they merged many
functions into 'parallel' which became part of the base R.

It would probably be good to try to coordinate efforts at
parallelizing within SciPy, if you choose to go that route, with those
who are trying to get this to work better at the program level, e.g.,
multiprocessing and ipyparallel.  Whatever gets done, it would be good
to have it work well with many of the ways that people are
implementing parallel computing.

As a cluster administrator and help desk person, I would also
encourage you to think about how this would play out in a shared
environment that is administered not by the scientist but by some
system administrator who may have different ideas about what can and
cannot be done with respect to intermachine communication and using
multiple processes (for example, is ssh blocked? are user jobs put
into cgroups to limit cores and memory?).

Just a couple of thoughts from the sidelines; hopefully not too far off topic.