[scikit-learn] scikit-learn 1 - pytest - multiprocessing Pool - hangs?

[Olivier Grisel]

Maybe you can try to use faulthandler.dump_traceback_later
https://docs.python.org/3/library/faulthandler.html#faulthandler.dump_traceback_later
to get a traceback of all the threads of the main process.

But the fact that you are using the default `p =
multiprocessing.Pool()` makes me think that it might be related to the
lack of fork-safety of the OpenMP runtime library of GCC (libgomp)
[1]. There are several ways to check this:

- print the output of threadpoolctl.threadpool_info() before calling
the code that freezes to confirm (or not) that the libgomp runtime has
been loaded before creating the MP Pool.
- use multiprocessing Pool using a forkserver context instead of the
default fork context: multiprocessing.get_context("forkserver").Pool()
- alternatively, use loky.get_reusable_excutor() instead of
multiprocessing.Pool() (with a slightly different API)
- alternatively, use joblib that uses loky internally with an even
more different API.
- alternatively, recompile scikit-learn from source with clang instead
of gcc so as to link scikit-learn to llvm-openmp instead of gcc's
libgomp runtime. llvm-openmp is forksafe,
- alternatively, install scikit-learn from conda-forge (conda install
-c conda-forge scikit-learn) as the conda-forge distribution relinks
all OpenMP compiled extensions of its packaged libraries to
llvm-openmp transparently at install time, even if they were built
with GCC (maybe we should do that for our linux wheels).

[1] https://gcc.gnu.org/legacy-ml/gcc-patches/2014-02/msg00979.html

If that does not work or need more help, please feel free to open an
issue with a minimal reproducer and ping me on gitter or discord.

Le jeu. 9 déc. 2021 à 05:59, Norbert Preining <norbert at preining.info> a écrit :
>
> Dear all,
>
> I am trying to track down a strange behaviour in one of our (Fujitsu)
> library we are planning to open source. In preparation for that, I am
> trying to bring it into a state that it works with scikit-learn >= 1.
>
> But, some of our tests fail when running in parallel mode. But they
> only fail when running under pytest, but NOT when running under python.
>
> The library code contains
>
>         def fit(self, X, y=None):
>             ...
>             p = multiprocessing.Pool()
>             ret = _reduce(
>                 p.map(....))
>
> Now what happens is that with scikit-learn 1(.0.1), the code hangs
> forever. I adjusted the code also so that the pool definition is not in
> the fit function, but in the __init__ function, and saved into self, but
> that didn't help either.
>
> When interrupted, pytest gives:
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> /home/norbert/.pyenv/versions/3.9.6/lib/python3.9/threading.py:312: KeyboardInterrupt
> (to show a full traceback on KeyboardInterrupt use --full-trace)
> ================================================ 1 passed, 2 warnings in 273.84s (0:04:33) =================================================
> Exception ignored in: <function Pool.__del__ at 0x7ff72f31b9d0>
> Traceback (most recent call last):
>   File "/home/norbert/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 268, in __del__
>     self._change_notifier.put(None)
>   File "/home/norbert/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/queues.py", line 378, in put
>     self._writer.send_bytes(obj)
>   File "/home/norbert/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes
>     self._send_bytes(m[offset:offset + size])
>   File "/home/norbert/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
>     self._send(header + buf)
>   File "/home/norbert/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/connection.py", line 373, in _send
>     n = write(self._handle, buf)
>
>
> While when running under python testfile.py all goes well.
>
>
> I have tested the following combinations:
> * scikit-learn 0.23.*, python 3.8 and python 3.9 => works
> * scikit-learn 0.24.*, python 3.8 and python 3.9 => works
> * scikit-learn 1.0.1,  python 3.8 and python 3.9 => fails
>
> I don't really understand where scikit-learn comes into the play here,
> so I wanted to ask whether someone here has an idea.
>
> Thanks for any suggestion
>
>
> Norbert
>
> --
> PREINING Norbert                              https://www.preining.info
> Fujitsu Research  +  IFMGA Guide  +  TU Wien  +  TeX Live  + Debian Dev
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn



-- 
Olivier