[pypy-issue] Issue #3060: Multiprocessing regression in PyPy 7.x (pypy/pypy)
Nathan Stocks
issues-reply at bitbucket.org
Tue Aug 27 18:16:42 EDT 2019
New issue 3060: Multiprocessing regression in PyPy 7.x
https://bitbucket.org/pypy/pypy/issues/3060/multiprocessing-regression-in-pypy-7x
Nathan Stocks:
I'm the maintainer of [Green](https://github.com/CleanCut/green) and I've run across what seems to be a very odd pypy3 bug in `multiprocessing/connection.py` specific to macOS. This doesn't happen on any other version of Python nor on any platform that I test, also this didn’t happen on PyPy 6.0 and earlier, so I don’t suspect the python module itself, per se.
OS: macOS Mojave 10.14.6
Pypy: `Python 3.6.1 (784b254d669919c872a505b807db8462b6140973, Jun 18 2019, 05:54:52)` / `[PyPy 7.1.1-beta0 with GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin`
Steps to reproduce:
1. Install green
* `pip_pypy3 install green`
1. Run a portion of green's self-tests:
* `/usr/local/share/pypy3/green -vvv green.test.test_runner`
I observe 3\(!!!\) different crashes, each of which seem to implicate `multiprocessing/connection.py`. As far as I can tell, which error you get is simply random chance, though the first type seems to occur most often for me.
Crash Type 1 seems to be the most common, an indefinite hang which requires pressing Ctrl-C to break out of:
```
$ /usr/local/share/pypy3/green -vvv green.test.test_runner
Green 2.16.1, Coverage 4.5.4, Python 3.6.1
green.test.test_runner
TestInitializerOrFinalizer
. Given a blank dotted function, calling the initializer/finalizer does nothing.
. An importable, callable object...crashes.
. Given an actual importable module and function, the function is run.
. An importable, but not-callable-object also raises an InitializerOrFinalizerError.
. Given an unimportable module, an InitializerOrFinalizerError is raised.
TestProcesses
. Bad syntax in a testfile is caught as a test error.
. run() can catch SIGINT while running a process.
. If tempfile.gettempdir() is used for dir, using same testfile name will not collide.
. args.processes = 0 causes auto-detection of number of processes.
. run() does not crash with empty suite and processes
. Running coverage in process mode doesn't crash
. Exceptions that escape the test framework get caught by poolRunner and reported as a failure. For example, the testtools implementation of TestCase unwisely (but deliberately) lets SystemExit exceptions through.
TestRun
. run() can use a GreenStream for output.
. A failing test case causes the whole run to report 'FAILED'
. failfast causes the testing to stop after the first failure.
. When we don't find any tests, we say so.
. run() can use sys.stdout as the stream.
. Raising a SystemExit gets caught and reported.
. verbose=3 causes version output, and an empty test case passes.
E setting warnings='always' doesn't crash
^CProcess DaemonlessPoolWorker-1:
Traceback (most recent call last):
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/process.py", line 252, in _bootstrap
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/util.py", line 288, in _exit_function
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/util.py", line 254, in _run_finalizers
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/util.py", line 186, in __call__
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/managers.py", line 615, in _finalize_manager
process.join(timeout=1.0)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/process.py", line 121, in join
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/popen_fork.py", line 48, in wait
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 911, in wait
ready = selector.select(timeout)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/selectors.py", line 376, in select
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/signals.py", line 36, in __call__
KeyboardInterrupt
Error in green.test.test_runner.TestRun.test_warnings
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/case.py", line 601, in run
testMethod()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/test/test_runner.py", line 182, in test_warnings
result = run(tests, self.stream, self.args)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/runner.py", line 87, in run
manager = multiprocessing.Manager()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/context.py", line 56, in Manager
m.start()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/managers.py", line 503, in start
reader, writer = connection.Pipe(duplex=False)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 512, in Pipe
fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
Warning: Some tests may not have been run.
Ran 20 tests in 10.948s using 8 processes
FAILED (errors=1, passes=19)
```
Here is Crash Type 2 talking about "bad message length":
```
$ /usr/local/share/pypy3/green -vvv green.test.test_runner
Green 2.16.1, Coverage 4.5.4, Python 3.6.1
green.test.test_runner
TestInitializerOrFinalizer
. Given a blank dotted function, calling the initializer/finalizer does nothing.
. An importable, callable object...crashes.
. Given an actual importable module and function, the function is run.
. An importable, but not-callable-object also raises an InitializerOrFinalizerError.
. Given an unimportable module, an InitializerOrFinalizerError is raised.
TestProcesses
. Bad syntax in a testfile is caught as a test error.
. run() can catch SIGINT while running a process.
. If tempfile.gettempdir() is used for dir, using same testfile name will not collide.
. args.processes = 0 causes auto-detection of number of processes.
. run() does not crash with empty suite and processes
. Running coverage in process mode doesn't crash
. Exceptions that escape the test framework get caught by poolRunner and reported as a failure. For example, the testtools implementation of TestCase unwisely (but deliberately) lets SystemExit exceptions through.
TestRun
. run() can use a GreenStream for output.
. A failing test case causes the whole run to report 'FAILED'
. failfast causes the testing to stop after the first failure.
. When we don't find any tests, we say so.
. run() can use sys.stdout as the stream.
. Raising a SystemExit gets caught and reported.
. verbose=3 causes version output, and an empty test case passes.
E setting warnings='always' doesn't crash
Error in green.test.test_runner.TestRun.test_warnings
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/case.py", line 601, in run
testMethod()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/test/test_runner.py", line 182, in test_warnings
result = run(tests, self.stream, self.args)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/runner.py", line 89, in run
for target in toParallelTargets(suite, args.targets)]
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/runner.py", line 89, in <listcomp>
for target in toParallelTargets(suite, args.targets)]
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/managers.py", line 662, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/managers.py", line 554, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 732, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 218, in recv_bytes
self._bad_message_length()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 151, in _bad_message_length
raise OSError("bad message length")
OSError: bad message length
Ran 20 tests in 5.613s using 8 processes
FAILED (errors=1, passes=19)
```
And finally Crash Type 3, talking about an "EOF Error":
```
$ /usr/local/share/pypy3/green -vvv green.test.test_runner
Green 2.16.1, Coverage 4.5.4, Python 3.6.1
green.test.test_runner
TestInitializerOrFinalizer
. Given a blank dotted function, calling the initializer/finalizer does nothing.
. An importable, callable object...crashes.
. Given an actual importable module and function, the function is run.
. An importable, but not-callable-object also raises an InitializerOrFinalizerError.
. Given an unimportable module, an InitializerOrFinalizerError is raised.
TestProcesses
. Bad syntax in a testfile is caught as a test error.
. run() can catch SIGINT while running a process.
. If tempfile.gettempdir() is used for dir, using same testfile name will not collide.
. args.processes = 0 causes auto-detection of number of processes.
. run() does not crash with empty suite and processes
. Running coverage in process mode doesn't crash
. Exceptions that escape the test framework get caught by poolRunner and reported as a failure. For example, the testtools implementation of TestCase unwisely (but deliberately) lets SystemExit exceptions through.
TestRun
. run() can use a GreenStream for output.
. A failing test case causes the whole run to report 'FAILED'
. failfast causes the testing to stop after the first failure.
. When we don't find any tests, we say so.
. run() can use sys.stdout as the stream.
. Raising a SystemExit gets caught and reported.
. verbose=3 causes version output, and an empty test case passes.
E setting warnings='always' doesn't crash
Error in green.test.test_runner.TestRun.test_warnings
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/unittest/case.py", line 601, in run
testMethod()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/test/test_runner.py", line 182, in test_warnings
result = run(tests, self.stream, self.args)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/site-packages/green/runner.py", line 87, in run
manager = multiprocessing.Manager()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/context.py", line 56, in Manager
m.start()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/managers.py", line 517, in start
self._address = reader.recv()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/local/Cellar/pypy3/7.1.1/libexec/lib-python/3/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
Ran 20 tests in 4.402s using 8 processes
FAILED (errors=1, passes=19)
```
More information about the pypy-issue
mailing list