reviving an old thread because there's movement on the PR ( https://github.com/scipy/scipy/pull/8259) and would be nice to get that merged. On Wed, Jan 10, 2018 at 2:34 AM, Gael Varoquaux < gael.varoquaux@normalesup.org> wrote:
5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small
joblib has a custom backend framework that can be used for such purpose (if I understnad you well):
https://pythonhosted.org/joblib/parallel.html#custom-backend-api-experimenta...
Updated link (status is still experimental): https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-exp...
There are currently a Yarn and a dask.distributed backend that are getting better and better.
There's also this JoblibPool that can be taken over: https://github.com/adrn/schwimmbad/blob/master/schwimmbad/jl.py#L14 Seems simpler than a backend still tagged experimental.
6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the Parallel object is `__call__`ed. This leads to significant overhead. One can use the Parallel object with a context manager, which allows reuse of the Pool, but I don't think that's do-able in the context of using the DifferentialEvolutionSolver (DES) object as an iterator:
This is evolving. However, the reason behind this is that Pool get corrupted and lead to deadlock. Olivier Grisel and Thomas Moreau are working on fixing this in the Python standard library (first PR merged recently)!
Anyone know the status of this? And can this issue be avoided by the new loky backend to joblib? Ralf
One of the vision of joblib is to provide very light mid-layer that can connect to multiprocessing and threading (though we are considering switching to concurrent.futures) as well as other backends. Hopefully this common language makes it easier to do things like embedding dask in numerical algorithms without a hard dependencies (yes we are working with the dask team on this).
Gaël _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev