[New-bugs-announce] [issue27406] subprocess.Popen() hangs in multi-threaded code

[Kristján Valur Jónsson]

New submission from Kristján Valur Jónsson:

On a quad-core raspberrypi, i have experienced that subprocess.Popen() sometimes does not return immediatelly, but only much later, when an unrelated process has exited.
Debugging the issue, I find the parent process hanging in 
                # Wait for exec to fail or succeed; possibly raising exception
                # Exception limited to 1M
                data = _eintr_retry_call(os.read, errpipe_read, 1048576)

This behaviour is consistent with the problem described in pipe_cloexec():


def pipe_cloexec(self):
            """Create a pipe with FDs set CLOEXEC."""
            # Pipes' FDs are set CLOEXEC by default because we don't want them
            # to be inherited by other subprocesses: the CLOEXEC flag is removed
            # from the child's FDs by _dup2(), between fork() and exec().
            # This is not atomic: we would need the pipe2() syscall for that.
            r, w = os.pipe()
            self._set_cloexec_flag(r)
            self._set_cloexec_flag(w)
            return r, w

In short:  It appears that occasionally the pipe FD is leaked to a different subprocess (started on a separate thread) before the cloexec flags can be set.  This causes the parent process to wait until that other instance of the file descriptor is closed, i.e. when that other unrelated process exits.

I currently have a workaround which involves using a threading.Lock() around the call.

This is not very nice, however.  Also, there is #Issue12196 which could be backported to 2.7 to address this problem.

----------
components: Interpreter Core
messages: 269432
nosy: kristjan.jonsson
priority: normal
severity: normal
status: open
title: subprocess.Popen() hangs in multi-threaded code
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27406>
_______________________________________