Dear Scipy developers, We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy. The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future? We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome. Best regards, Marc Barbry
Just off the cuff I'd guess that the BLACS dependency would make it very difficult to support Scalapack in SciPy proper. On Wed, Oct 4, 2017 at 8:45 AM, marc <marc.barbry@mailoo.org> wrote:
Dear Scipy developers,
We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy.
The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack
Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future?
We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome.
Best regards, Marc Barbry _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
I guess this problem of dependency could be avoid with a custom installation. MPI code could be compile only if specify, in a similar fashion than the mkl library for Blas/Lapack wrappers. On 10/04/2017 04:59 PM, Joshua Wilson via gpaw-users wrote:
Just off the cuff I'd guess that the BLACS dependency would make it very difficult to support Scalapack in SciPy proper.
On Wed, Oct 4, 2017 at 8:45 AM, marc <marc.barbry@mailoo.org> wrote:
Dear Scipy developers,
We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy.
The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack
Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future?
We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome.
Best regards, Marc Barbry _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
gpaw-users mailing list gpaw-users@listserv.fysik.dtu.dk https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
On Thu, Oct 5, 2017 at 4:17 AM, marc <marc.barbry@mailoo.org> wrote:
I guess this problem of dependency could be avoid with a custom installation. MPI code could be compile only if specify, in a similar fashion than the mkl library for Blas/Lapack wrappers.
There's a major difference there: MKL is optional, but BLAS/LAPACK is not. We require a BLAS/LAPACK install, the implementations can be switched out. For ScaLAPACK it seems pretty clear we can't require it to be installed. Moreover, distributed computing is out of scope for SciPy. So developing your ScaLAPACK wrappers (which do seem very useful indeed) as a separate package looks like the better way to go. Cheers, Ralf
On 10/04/2017 04:59 PM, Joshua Wilson via gpaw-users wrote:
Just off the cuff I'd guess that the BLACS dependency would make it very difficult to support Scalapack in SciPy proper.
On Wed, Oct 4, 2017 at 8:45 AM, marc <marc.barbry@mailoo.org> wrote:
Dear Scipy developers,
We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy.
The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack
Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future?
We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome.
Best regards, Marc Barbry _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ gpaw-users mailing list gpaw-users@listserv.fysik.dtu.dk https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Thu, Oct 5, 2017 at 1:54 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Oct 5, 2017 at 4:17 AM, marc <marc.barbry@mailoo.org> wrote:
I guess this problem of dependency could be avoid with a custom installation. MPI code could be compile only if specify, in a similar fashion than the mkl library for Blas/Lapack wrappers.
There's a major difference there: MKL is optional, but BLAS/LAPACK is not. We require a BLAS/LAPACK install, the implementations can be switched out.
For ScaLAPACK it seems pretty clear we can't require it to be installed. Moreover, distributed computing is out of scope for SciPy. So developing your ScaLAPACK wrappers (which do seem very useful indeed) as a separate package looks like the better way to go.
But does distributed computing stay out of scope for SciPy after 1.0? As a long term plan towards 2.0? (Not that MPI based solutions will help much on Windows, AFAICS.) Josef
Cheers, Ralf
On 10/04/2017 04:59 PM, Joshua Wilson via gpaw-users wrote:
Just off the cuff I'd guess that the BLACS dependency would make it very difficult to support Scalapack in SciPy proper.
On Wed, Oct 4, 2017 at 8:45 AM, marc <marc.barbry@mailoo.org> wrote:
Dear Scipy developers,
We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy.
The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack
Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future?
We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome.
Best regards, Marc Barbry _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ gpaw-users mailing list gpaw-users@listserv.fysik.dtu.dk https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Fri, Oct 6, 2017 at 7:05 AM, <josef.pktd@gmail.com> wrote:
On Thu, Oct 5, 2017 at 1:54 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Oct 5, 2017 at 4:17 AM, marc <marc.barbry@mailoo.org> wrote:
I guess this problem of dependency could be avoid with a custom installation. MPI code could be compile only if specify, in a similar fashion than the mkl library for Blas/Lapack wrappers.
There's a major difference there: MKL is optional, but BLAS/LAPACK is not. We require a BLAS/LAPACK install, the implementations can be switched out.
For ScaLAPACK it seems pretty clear we can't require it to be installed. Moreover, distributed computing is out of scope for SciPy. So developing your ScaLAPACK wrappers (which do seem very useful indeed) as a separate package looks like the better way to go.
But does distributed computing stay out of scope for SciPy after 1.0? As a long term plan towards 2.0?
Such changes are worth discussing once in a while, usually sharpens the focus:) My first thoughts: - traditional stuff like MPI, BLACS, ScaLAPACK will likely always remain out of scope - we can consider new dependencies, but only if they do not make it harder to install SciPy - a few more likely changes would be to start allowing/supporting pandas data frames as inputs, broader use of simple (optional) parallelization with joblib or threading, and using dask under the hood. Ralf
(Not that MPI based solutions will help much on Windows, AFAICS.)
Josef
Cheers, Ralf
On 10/04/2017 04:59 PM, Joshua Wilson via gpaw-users wrote:
Just off the cuff I'd guess that the BLACS dependency would make it very difficult to support Scalapack in SciPy proper.
On Wed, Oct 4, 2017 at 8:45 AM, marc <marc.barbry@mailoo.org> wrote:
Dear Scipy developers,
We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy.
The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack
Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future?
We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome.
Best regards, Marc Barbry _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ gpaw-users mailing list gpaw-users@listserv.fysik.dtu.dk https://listserv.fysik.dtu.dk/mailman/listinfo/gpaw-users
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Thu, Oct 5, 2017 at 8:36 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Oct 6, 2017 at 7:05 AM, <josef.pktd@gmail.com> wrote:
But does distributed computing stay out of scope for SciPy after 1.0? As a long term plan towards 2.0?
Such changes are worth discussing once in a while, usually sharpens the focus:)
My first thoughts: - traditional stuff like MPI, BLACS, ScaLAPACK will likely always remain out of scope - we can consider new dependencies, but only if they do not make it harder to install SciPy - a few more likely changes would be to start allowing/supporting pandas data frames as inputs, broader use of simple (optional) parallelization with joblib or threading, and using dask under the hood.
There seems to be a profusion of tools for parallelization, so choosing just one to use as a basis for scipy's parallelization could be really frustrating for users who have a reason to need a different one. The exception, I would say, is the concurrent.futures interface. This is part of python (3), and it allows a limited but manageable and useful amount of parallelization. It is also an interface other tools can and do implement. For example, emcee is capable of taking advantage of parallelization, but that parallelization happens entirely in one place: a map is done to compute log-probabilities for a list of candidates. emcee is agnostic about how this map works; by default it can use python's built-in map, but emcee provides an "MPIPool" object that supplies a parallel map that uses MPI, python's ThreadPoolExecutor and ProcessPoolExecutor also provide such a parallel map, and (for example) dask provides an Executor interface that allows such a map across a collection of dask instances. So: I think it would be good if scipy could incorporate the use of Executors to achieve parallelism where that's available from the underlying algorithms. From the user's point of view, this just means one or two more optional arguments, in particular a "pool" argument from which futures are generated. In turn, it might make sense to implement a few new algorithms that can use parallelism effectively. The global optimizers spring to mind as candidates for this process, but in fact any local optimizer that needs a gradient but has to compute it numerically can probably benefit from computing the derivative in parallel. This sort of opportunistic parallelization is no substitute for something like Scalapack or PaGMO, dedicated distributed computing algorithms, but it is a way for scipy to allow easy parallelization where possible. Anne
On Fri, Oct 6, 2017 at 10:37 PM, Anne Archibald <peridot.faceted@gmail.com> wrote:
On Thu, Oct 5, 2017 at 8:36 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Oct 6, 2017 at 7:05 AM, <josef.pktd@gmail.com> wrote:
But does distributed computing stay out of scope for SciPy after 1.0? As a long term plan towards 2.0?
Such changes are worth discussing once in a while, usually sharpens the focus:)
My first thoughts: - traditional stuff like MPI, BLACS, ScaLAPACK will likely always remain out of scope - we can consider new dependencies, but only if they do not make it harder to install SciPy - a few more likely changes would be to start allowing/supporting pandas data frames as inputs, broader use of simple (optional) parallelization with joblib or threading, and using dask under the hood.
Interesting thoughts, thanks Anne. This thread is a bit stale by now, but I still wanted to record my thoughts on the topic.
There seems to be a profusion of tools for parallelization, so choosing just one to use as a basis for scipy's parallelization could be really frustrating for users who have a reason to need a different one.
You're thinking about the relatively small fraction of power users here that would care (compared to the n_jobs=<number> trivial parallelization users), and my first thought is that addressing that use case comes with costs that are possibly not worth the effort.
The exception, I would say, is the concurrent.futures interface.
In terms of user friendliness, I'd say concurrent.futures is pretty poor. This: with futures.ThreadPoolExecutor(max_workers=4) as executor: some_function(..., pool=executor.map) is much worse than: some_function(..., n_jobs=4) This is part of python (3), and it allows a limited but manageable and
useful amount of parallelization. It is also an interface other tools can and do implement. For example, emcee is capable of taking advantage of parallelization, but that parallelization happens entirely in one place: a map is done to compute log-probabilities for a list of candidates. emcee is agnostic about how this map works; by default it can use python's built-in map, but emcee provides an "MPIPool" object that supplies a parallel map that uses MPI, python's ThreadPoolExecutor and ProcessPoolExecutor also provide such a parallel map, and (for example) dask provides an Executor interface that allows such a map across a collection of dask instances.
So: I think it would be good if scipy could incorporate the use of Executors to achieve parallelism where that's available from the underlying algorithms. From the user's point of view, this just means one or two more optional arguments, in particular a "pool" argument from which futures are generated.
It'd have to be two I think, like in emcee which has `*threads=1*, *pool=None`. *I'd say `threads` (or `n_jobs` as in scikit-learn and spatial.cKDTree) is the must-have if we go for more parallel support and `pool` is the power user one that would be a tradeoff. It would enable users of MPI, dask, etc., but on the other hand it makes the API more verbose, is more work to support, and a lot harder to test. IIRC joblib also does a lot of work to make ndarrays (and other objects?) pickleable to make parallelization work for a wider range of functions that plain multiprocessing. So I'm undecided on whether a `pool` keyword would make sense in scipy. In turn, it might make sense to implement a few new algorithms that can use
parallelism effectively. The global optimizers spring to mind as candidates for this process, but in fact any local optimizer that needs a gradient but has to compute it numerically can probably benefit from computing the derivative in parallel.
Clustering functions would be another good candidate.
This sort of opportunistic parallelization is no substitute for something like Scalapack or PaGMO, dedicated distributed computing algorithms, but it is a way for scipy to allow easy parallelization where possible.
Agreed Ralf
Ralf, and all,
There seems to be a profusion of tools for parallelization, so choosing just one to use as a basis for scipy's parallelization could be really frustrating for users who have a reason to need a different one.
You're thinking about the relatively small fraction of power users here that would care (compared to the n_jobs=<number> trivial parallelization users), and my first thought is that addressing that use case comes with costs that are possibly not worth the effort.
You might consider separating that which can be done on one physical machine from that which requires (or expects) many. This was largely done by the R developers. The 'snow' library used rsh/ssh whereas the 'multicore' library used fork() and processors. Steve Weston and company have the 'foreach' library that provides a user interface to various backends that distribute the tasks appropriately. Only after many years of experience, they merged many functions into 'parallel' which became part of the base R. It would probably be good to try to coordinate efforts at parallelizing within SciPy, if you choose to go that route, with those who are trying to get this to work better at the program level, e.g., multiprocessing and ipyparallel. Whatever gets done, it would be good to have it work well with many of the ways that people are implementing parallel computing. As a cluster administrator and help desk person, I would also encourage you to think about how this would play out in a shared environment that is administered not by the scientist but by some system administrator who may have different ideas about what can and cannot be done with respect to intermachine communication and using multiple processes (for example, is ssh blocked? are user jobs put into cgroups to limit cores and memory?). Just a couple of thoughts from the sidelines; hopefully not too far off topic.
On Mon, Oct 30, 2017 at 1:14 AM, Bennet Fauber <bennet@umich.edu> wrote:
Ralf, and all,
There seems to be a profusion of tools for parallelization, so choosing just one to use as a basis for scipy's parallelization could be really frustrating for users who have a reason to need a different one.
You're thinking about the relatively small fraction of power users here that would care (compared to the n_jobs=<number> trivial parallelization users), and my first thought is that addressing that use case comes with costs that are possibly not worth the effort.
You might consider separating that which can be done on one physical machine from that which requires (or expects) many.
This was largely done by the R developers. The 'snow' library used rsh/ssh whereas the 'multicore' library used fork() and processors. Steve Weston and company have the 'foreach' library that provides a user interface to various backends that distribute the tasks appropriately. Only after many years of experience, they merged many functions into 'parallel' which became part of the base R.
Thanks, always interesting to know the history of how designs evolved in related libraries/languages.
It would probably be good to try to coordinate efforts at parallelizing within SciPy, if you choose to go that route, with those who are trying to get this to work better at the program level, e.g., multiprocessing and ipyparallel.
Multiprocessing is exactly what I was talking about (use directly or via joblib, which is built on top of it). Ipyparallel is squarely aimed at use from within Jupyter, so not very relevant in the context of a library. Actually the implementation isn't too interesting I think (scipy.spatial.cKDTree uses threading rather than multiprocessing for example); having an easy and uniform API is what matters. Whatever gets done, it would be good
to have it work well with many of the ways that people are implementing parallel computing.
As a cluster administrator and help desk person, I would also encourage you to think about how this would play out in a shared environment that is administered not by the scientist but by some system administrator who may have different ideas about what can and cannot be done with respect to intermachine communication and using multiple processes (for example, is ssh blocked? are user jobs put into cgroups to limit cores and memory?).
The one thing I can think of is the design of `n_jobs=-1`, which means "use as many cores as present". This could pick up the limit on cores if that is defined in a standard way for a given OS. This must have come up for scikit-learn before I'd think.
Just a couple of thoughts from the sidelines; hopefully not too far off topic.
Not at all. Thanks. Ralf
Dear all, I'm glad to see that the Scipy community look interested by this topic. As I pointed out with the first mail of this thread, we started an implementation of ScaLapack wrapper, you can find it at the following repository, https://gitlab.com/mbarbry/python-scalapack We try to use the same method that have been use in Scipy for the Blas/Lapack wrapper (using f2py). The wrapper is already functional, but the installation is not as straightforward that I would hope and only very few routines are implemented for the moment. So any help is more than welcome. Best regards, Marc On 10/29/2017 08:15 PM, Ralf Gommers wrote:
On Mon, Oct 30, 2017 at 1:14 AM, Bennet Fauber <bennet@umich.edu <mailto:bennet@umich.edu>> wrote:
Ralf, and all,
>> There seems to be a profusion of tools for parallelization, so choosing >> just one to use as a basis for scipy's parallelization could be really >> frustrating for users who have a reason to need a different one. > > You're thinking about the relatively small fraction of power users here that > would care (compared to the n_jobs=<number> trivial parallelization users), > and my first thought is that addressing that use case comes with costs that > are possibly not worth the effort.
You might consider separating that which can be done on one physical machine from that which requires (or expects) many.
This was largely done by the R developers. The 'snow' library used rsh/ssh whereas the 'multicore' library used fork() and processors. Steve Weston and company have the 'foreach' library that provides a user interface to various backends that distribute the tasks appropriately. Only after many years of experience, they merged many functions into 'parallel' which became part of the base R.
Thanks, always interesting to know the history of how designs evolved in related libraries/languages.
It would probably be good to try to coordinate efforts at parallelizing within SciPy, if you choose to go that route, with those who are trying to get this to work better at the program level, e.g., multiprocessing and ipyparallel.
Multiprocessing is exactly what I was talking about (use directly or via joblib, which is built on top of it). Ipyparallel is squarely aimed at use from within Jupyter, so not very relevant in the context of a library.
Actually the implementation isn't too interesting I think (scipy.spatial.cKDTree uses threading rather than multiprocessing for example); having an easy and uniform API is what matters.
Whatever gets done, it would be good to have it work well with many of the ways that people are implementing parallel computing.
As a cluster administrator and help desk person, I would also encourage you to think about how this would play out in a shared environment that is administered not by the scientist but by some system administrator who may have different ideas about what can and cannot be done with respect to intermachine communication and using multiple processes (for example, is ssh blocked? are user jobs put into cgroups to limit cores and memory?).
The one thing I can think of is the design of `n_jobs=-1`, which means "use as many cores as present". This could pick up the limit on cores if that is defined in a standard way for a given OS. This must have come up for scikit-learn before I'd think.
Just a couple of thoughts from the sidelines; hopefully not too far off topic.
Not at all. Thanks.
Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (6)
-
Anne Archibald -
Bennet Fauber -
josef.pktd＠gmail.com -
Joshua Wilson -
marc -
Ralf Gommers