[New-bugs-announce] [issue44153] Signaling an asyncio subprocess raises ProcessLookupError, depending on timing
report at bugs.python.org
Sun May 16 18:24:45 EDT 2021
New submission from Max Marrone <syntaxcoloring at gmail.com>:
Basic use of `asyncio.subprocess.Process.terminate()` can raise a `ProcessLookupError`, depending on the timing of the subprocess's exit.
I assume (but haven't checked) that this problem extends to `.kill()` and `.send_signal()`.
This breaks the expected POSIX semantics of signaling and waiting on a process. See the "Expected behavior" section.
# Test case
I've tested this on macOS 11.2.3 with Python 3.7.9 and Python 3.10.0a7, both installed via pyenv.
# Tested with:
# asyncio.ThreadedChildWatcher (3.10.0a7 only)
# asyncio.MultiLoopChildWatcher (3.10.0a7 only)
# asyncio.SafeChildWatcher (3.7.9 and 3.10.0a7)
# asyncio.FastChildWatcher (3.7.9 and 3.10.0a7)
# Not tested with asyncio.PidfdChildWatcher because I'm not on Linux.
WATCHER_CLASS = asyncio.FastChildWatcher
async def main():
# Dummy command that should be executable cross-platform.
process = await asyncio.subprocess.create_subprocess_exec(
for i in range(20):
# I think the problem is that the event loop opportunistically wait()s
# all outstanding subprocesses on its own. Do a bunch of separate
# sleep() calls to give it a bunch of chances to do this, for reliable
# I'm not sure if this is strictly necessary for the problem to happen.
# On my machine, the problem also happens with a single sleep(2.0).
process.terminate() # This unexpectedly errors with ProcessLookupError.
The `process.terminate()` call raises a `ProcessLookupError`:
Traceback (most recent call last):
File "kill_is_broken.py", line 29, in <module>
File "/Users/maxpm/.pyenv/versions/3.7.9/lib/python3.7/asyncio/runners.py", line 43, in run
File "/Users/maxpm/.pyenv/versions/3.7.9/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
File "kill_is_broken.py", line 24, in main
process.terminate() # This errors with ProcessLookupError.
File "/Users/maxpm/.pyenv/versions/3.7.9/lib/python3.7/asyncio/subprocess.py", line 131, in terminate
File "/Users/maxpm/.pyenv/versions/3.7.9/lib/python3.7/asyncio/base_subprocess.py", line 150, in terminate
File "/Users/maxpm/.pyenv/versions/3.7.9/lib/python3.7/asyncio/base_subprocess.py", line 143, in _check_proc
# Expected behavior and discussion
Normally, with POSIX semantics, the `wait()` syscall tells the operating system that we won't send any more signals to that process, and that it's safe for the operating system to recycle that process's PID. This comment from Jack O'Connor on another issue explains it well: https://bugs.python.org/issue40550#msg382427
So, I expect that on any given `asyncio.subprocess.Process`, if I call `.terminate()`, `.kill()`, or `.send_signal()` before I call `.wait()`, then:
* It should not raise a `ProcessLookupError`.
* The asyncio internals shouldn't do anything with a stale PID. (A stale PID is one that used to belong to our subprocess, but that we've since consumed through a `wait()` syscall, allowing the operating system to recycle it).
asyncio internals are mostly over my head. But I *think* the problem is that the event loop opportunistically calls the `wait()` syscall on our child processes. So, as implemented, there's a race condition. If the event loop's `wait()` syscall happens to come before my `.terminate()` call, my `.terminate()` call will raise a `ProcessLookupError`.
So, as a corollary to the expectations listed above, I think the implementation details should be either:
* Ideally, the asyncio internals should not call syscall `wait()` on a process until *I* call `wait()` on that process.
* Failing that, `.terminate()`, `.kill()` and `.send_signal()` should should no-op if the asyncio internals have already called `.wait()` on that process.
nosy: asvetlov, syntaxcoloring, yselivanov
title: Signaling an asyncio subprocess raises ProcessLookupError, depending on timing
versions: Python 3.10, Python 3.7
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce