[SciPy-Dev] General discussion on parallelisation

[Eric Larson]

>
> 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the
> Parallel object is `__call__`ed. This leads to significant overhead. One
> can use the Parallel object with a context manager, which allows reuse of
> the Pool, but I don't think that's do-able in the context of using the
> DifferentialEvolutionSolver (DES) object as an iterator:
>

If you can use `__iter__` instead of `__next__` in your
DifferentialEvolutionSolver class I think you can avoid this problem with
something along the lines of:

class DifferentialEvolutionSolver(object):
    ...
    def __iter__(self):
        with Parallel(...) ...:
            for it in ...:
                yield it

​
As for the map problem I don't know, but it's probably worth asking the
`joblib` people if there is a suitable solution or workaround.

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180108/131eaff4/attachment.html>