[issue13812] multiprocessing package doesn't flush stderr on child exception

[Jon Brandvein]

Jon Brandvein <jon.brandvein at gmail.com> added the comment:

Regarding the patch: I'd also like to see sys.stdout.flush() and sys.stderr.flush() between "exitcode = self._boostrap()" and "exit(exitcode)" in /Lib/multiprocessing/forking.py :: main(). (The extra stderr flush would be for symmetry with Popen.__init__() for unix.)

The remainder of this post is what I wrote before seeing your patch.


Thank you for the explanation concerning libc and buffering.

As for os._exit, I can see the need under unix, where the child process has a stack inherited from the parent. Also, allowing the child to cleanup shared resources would be disastrous for the parent. (I was going to propose the idea of making a sys.exit()-like function that did not rely on the exception mechanism to unwind the stack. But it wouldn't be usable here for this reason.)

Under Windows, the child process's multiprocessing.forking.main() is launched either by directly calling main() on the command line, or from within freeze_support() in the user program. In the latter case, the user can be advised not to catch SystemExit around freeze_support(), just as they are already advised to make calling freeze_support() the first statement within "if __name__ == '__main__':". So I don't see any harm in switching to sys.exit() there.


I think that in general, the differences between multiprocessing's behavior under unix and windows are a weakness, as is the lack of a precise specification for this behavior.

At the same time, multiprocessing is a lot more convenient to use than the subprocess module when the child process is a Python program. In particular, I use multiprocessing for its support of passing pickled objects between parent and child. With subprocess, I don't think it's even possible to share a file descriptor between parent and child under Windows.

I'm wondering whether it would be desirable to make an option for the unix side of multiprocessing to behave more like the windows one, and invoke the Python interpreter from the beginning with no trace of the parent's stack. That is, it'd follow a more traditional fork()-then-exec*() pattern. Then you'd be able to have the interpreter destruct naturally under both platforms, close resources, etc.

Are there use cases that require just forking under unix? Is the performance significantly better without an exec*()?

Perhaps it would be better If I took this discussion to python-ideas.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13812>
_______________________________________