[New-bugs-announce] [issue28462] subprocess pipe can't see EOF from a child in case of a few children run with subprocess

[Vyacheslav Grigoryev]

New submission from Vyacheslav Grigoryev:

I'm creating a master stand-alone module on Python which should run some children via subprocess module. Working with children is done in separate worker threads. Additionally I need to receive real-time output from a child so in a worker thread I also create reader helper thread which reads on a pipe from the child.

All is working correctly while I have only one worker thread and run only one child. When the child finishes the reader helper thread gets EOF and exits. But when I have two worker threads and run two children, a pipe from early child doesn't see EOF until the second child is finished. Though they are completely unrelated.

Let's take a look on simplest example for reproducing the problem. There is a simplest child:
------------------------------
import time, sys

# first arg is an ID. Second arg is how long to work in seconds
sys.stdout.write("start slave %s\n" % sys.argv[1])
sys.stdout.flush()

time.sleep(int(sys.argv[2]))

sys.stdout.write("finish slave %s\n" % sys.argv[1])
sys.stdout.flush()
------------------------------

And there is a master module:
------------------------------
import subprocess, sys, os, threading, time

g_logLock = threading.Lock()

def log(msg):
    with g_logLock:
        t = time.time()
        print "%s.%03d %-5s %s" % \
            (time.strftime('%H:%M:%S', time.localtime(t)), int((t - t // 1) * 1000), threading.currentThread().name, msg)

def thread1Proc():
    def reader(stdout):
        while True:
            line = stdout.readline()
            if not line:
                break
            log('slave said: %s' % line.strip())
        log('finish slave reader thread')

    log('thread 1 started')
    timeToWork = '1'
    util = subprocess.Popen((sys.executable, 'slave.py', '1', timeToWork), stdout=subprocess.PIPE)
    readerT = threading.Thread(target=reader, args=(util.stdout,), name='t1-r')
    readerT.start()
    log('slave 1 returned %d' % util.wait())
    readerT.join()
    log('thread 1 finished')

def thread2Proc():
    log('thread 2 started')
    timeToWork = '3'
    util = subprocess.Popen((sys.executable, 'slave.py', '2', timeToWork))
    log('slave 2 returned %d' % util.wait())
    log('thread 2 finished')

#---------------------------
log('starting test')

threads = (threading.Thread(target=thread1Proc, name='t1'), threading.Thread(target=thread2Proc, name='t2'))
for t in threads:
    t.start()
for t in threads:
    t.join()

log('finished test')
------------------------------

Here is what I see on the output (note - slave 1 outputs to the master via pipe, while slave 2 outputs to a console because its output is not redirected):

>master.py
08:57:31.342 MainThread starting test
08:57:31.342 t1    thread 1 started
08:57:31.342 t2    thread 2 started
08:57:31.405 t1-r  slave said: start slave 1
start slave 2
08:57:32.420 t1-r  slave said: finish slave 1
08:57:32.420 t1    slave 1 returned 0
finish slave 2
08:57:34.415 t1-r  finish slave reader thread
08:57:34.415 t2    slave 2 returned 0
08:57:34.415 t1    thread 1 finished
08:57:34.431 t2    thread 2 finished
08:57:34.431 MainThread finished test

Here you can see that even if the slave 1 finishes at 32.420, its reader thread receives EOF and exits only when the slave 2 finishes also - at 34.415 (slave 1 works 1 second, slave 2 - 3 seconds).

Why the reader thread doesn't see EOF just in time?

The issue is reproduced in Python 2.7.12 x86 on Windows 7. On Linux Ubuntu 16.04 with system Python 2.7 all works as expected.

----------
components: Library (Lib)
messages: 278809
nosy: Vyacheslav Grigoryev
priority: normal
severity: normal
status: open
title: subprocess pipe can't see EOF from a child in case of a few children run with subprocess
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28462>
_______________________________________