[New-bugs-announce] [issue39445] h5py not playing nicely with subprocess and mpirun

[Rafael Laboissière]

New submission from Rafael Laboissière <rafael at laboissiere.net>:

* Preamble:

The problem reported hereafter possibly comes from the h5py module, which is not part of Python per se. This problem has been already reported to the h5py developers: https://github.com/h5py/h5py/issues/1467

and also against the Debian package python3-h5py: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946986

I apologize if this issue report is considered abusive.  Please, feel free to close it, if it is the case.

* The problem:

The combination of "import h5py", "subprocess.Popen", and "mpirun" is yielding a weird result. Consider these two scripts:

#############################################################
### File name: bugtest-without-h5py.py
import subprocess
simulationProc = subprocess.Popen("mpirun",
                                  shell=True,
                                  stdout=subprocess.PIPE,
                                  stderr=subprocess.PIPE)
(stdout, stderr) = simulationProc.communicate()
returnCode = simulationProc.wait()
print("stdout = ", stdout)
print("stderr = ", stderr)
print("return code = ", returnCode)
#############################################################

#############################################################
### File name: bugtest-with-h5py.py
import subprocess
import h5py
simulationProc = subprocess.Popen("mpirun",
                                  shell=True,
                                  stdout=subprocess.PIPE,
                                  stderr=subprocess.PIPE)
(stdout, stderr) = simulationProc.communicate()
returnCode = simulationProc.wait()
print("stdout = ", stdout)
print("stderr = ", stderr)
print("return code = ", returnCode)
#############################################################

The only difference between them is the line containing "import h5py" in the second.

Here is the result when running the first script:

   $ python3 bugtest-without-h5py.py
   stdout =  b''
   stderr =  b'--------------------------------------------------------------------------\nmpirun could not find anything to do.\n\nIt is possible that you forgot to specify how many processes to run\nvia the "-np" argument.\n--------------------------------------------------------------------------\n'
   return code =  1

and here is the result for the second script:

   $ python3 bugtest-with-h5py.py
   stdout =  b''
   stderr =  b''
   return code =  1

It seems that, when h5py is imported, the mpirun command is not even launched by subprocess.Popen, even though there is noting in this call that is related to h5py.

When "mpirun" is replaced by other commands (e.g. "date"), then the output is identical for both scripts, as it should be.

As I wrote in the preamble, this is possibly a problem with h5py.  I am reporting it here because the developers of the subprocess module may have an idea about the origin of the problem or give me a hint on how to debug the it.

The tests were done on a Debian bullseye system with the following versions:

h5py 2.10.0 (compiled with MPI support)
Python 3.7.6

----------
components: Library (Lib)
messages: 360638
nosy: Rafael Laboissière
priority: normal
severity: normal
status: open
title: h5py not playing nicely with subprocess and mpirun
type: behavior
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39445>
_______________________________________