<sheepishly> Ok, I managed to search and AFAICT the main issue is that if a child process terminates then the master process can hang waiting for it to return (https://bugs.python.org/issue22393).

> This is merged and released since a while. Loky is now used in anger. So far, I think that people are happy with it. In particular it is more robust than multiprocessing.Pool (specifically, robust to segfault), and the improvements have been contributed upstream to Python concurrent.futures's process pool executor (available in Python 3.7).

So, reading the `concurrent.futures.ProcessPoolExecutor` documentation indicates that it is resistant to this issue. concurrent.futures is available in Python3, but wasn't ported to 2.7.

On Mon, 3 Sep 2018 at 19:30, Andrew Nelson <andyfaff@gmail.com> wrote:
SNIP
> In particular it is more robust than multiprocessing.Pool (specifically, robust to segfault), and the improvements have been contributed upstream to Python concurrent.futures's process pool executor (available in Python 3.7).

SNIP
> While simpler multiprocessing-based code will sometimes give less overhead compared to joblib, it will probably be brittle.

Is there anything I can read to learn a bit more about this? Is it mainly related to how easy things are to pickle?



--
_____________________________________
Dr. Andrew Nelson


_____________________________________