[New-bugs-announce] [issue32608] Incompatibilities with the socketserver and multiprocessing packages

[Michael Durso]

New submission from Michael Durso <rbprogrammer at gmail.com>:

This is more of an issue with socketserver rather than multiprocessing.  I could not find socketserver experts to add to the nosy list, but I've included the multiprocessing experts for completeness.

Under specific conditions, multiprocessing queues cease to work when modules like socketserver is used, more specifically when fork() is used.  This makes sense that things can break when fork() is used since a simple forked process has things like file descriptors and threads invalidated.  The socketserver.ForkingMixIn class, which has very reasonable use-cases, causes multiprocessing queues and threads to break.  A script that replicates and simulates the issue is attached.

The attached script sets up a multiprocessing queue used for logging across all sub-processes, then uses a socketserver.ForkingTCPServer to handle new client connection.  What's happening is the fork() is invalidating the thread in the QueueHandler in each new sub-process (for each new client conncetion).  It turns the QueueHandler into a dead object.

In this scenario, the only indication the problem exists is the fact that log messages are not actually logged.  But I think it's plausible that more than just log data could be lost.

When a non-modified version of CPython, the script should run and produce a log file.  However, logs from the client connection handler (MyClientHandler) will be lost and never sent to the logging handler that performs the actual file writing.  To exercise the proposed fix to CPython, build the changes to socketserver.py and then change the script's "USE_FIX" flag to "True" and run the script.

----------
components: Library (Lib)
files: multiprocessing_socketserver.py
messages: 310371
nosy: davin, pitrou, rbprogrammer
priority: normal
severity: normal
status: open
title: Incompatibilities with the socketserver and multiprocessing packages
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file47398/multiprocessing_socketserver.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32608>
_______________________________________